Friday, November 27, 2020

Mysteries of the Department of Justice's ULTR for Firearm-toolmark Pattern Examinations

The Department of Justice's Uniform Language for Testimony and Reports (ULTR) for the Forensic Firearms/toolmarks Discipline – Pattern Examination" offers a ready response to motions to limit overclaiming or, to use the pedantic term, ultracrepidarianism, in expert testimony. Citing the DoJ policy, several federal district courts have indicated that they expect the government's expert witnesses to follow this directive (or something like it). \1/

Parts of the current version (with changes from the original) are reproduced in Box 1. \2/ This posting poses three questions about this guidance. Although the ULTR is a step in the right direction, it has a ways to go in articulating a clear and optimal policy.

Box 1. The ULTR

DEPARTMENT OF JUSTICE
UNIFORM LANGUAGE FOR TESTIMONY AND REPORTS
FOR THE FORENSIC FIREARMS/TOOLMARKS DISCIPLINE –
PATTERN MATCH EXAMINATION
...
III. Conclusions Regarding Forensic Pattern Examination of Firearms/Toolmarks Evidence for a Pattern Match

The An examiner may offer provide any of the following conclusions:
1.Source identification (i.e., identified)
2.Source exclusion (i.e., excluded)
3.Inconclusive
Source identification
‘Source identification’ is an examiner’s conclusion that two toolmarks originated from the same source. This conclusion is an examiner’s decision opinion that all observed class characteristics are in agreement and the quality and quantity of corresponding individual characteristics is such that the examiner would not expect to find that same combination of individual characteristics repeated in another source and has found insufficient disagreement of individual characteristics to conclude they originated from different sources.

The basis for a ‘source identification’ conclusion is an examiner’s decision opinion that the observed class characteristics and corresponding individual characteristics provide extremely strong support for the proposition that the two toolmarks came originated from the same source and extremely weak support for the proposition that the two toolmarks came originated from different sources.

A ‘source identification’ is the statement of an examiner’s opinion (an inductive inference2) that the probability that the two toolmarks were made by different sources is so small that it is negligible. A ‘source identification’ is not based upon a statistically-derived or verified measurement or an actual comparison to all firearms or toolmarks in the world.

Source exclusion
‘Source exclusion’ is an examiner’s conclusion that two toolmarks did not originate from the same source.

The basis for a ‘source exclusion’ conclusion is an examiner’s decision opinion that the observed class and/or individual characteristics provide extremely strong support for the proposition that the two toolmarks came from different sources and extremely weak or no support for the proposition that the two toolmarks came from the same source two toolmarks can be differentiated by their class characteristics and/or individual characteristics.

Inconclusive
‘Inconclusive’ is an examiner’s conclusion that all observed class characteristics are in agreement but there is insufficient quality and/or quantity of corresponding individual characteristics such that the examiner is unable to identify or exclude the two toolmarks as having originated from the same source.

The basis for an ‘inconclusive’ conclusion is an examiner’s decision opinion that there is an insufficient quality and/or quantity of individual characteristics to identify or exclude. Reasons for an ‘inconclusive’ conclusion include the presence of microscopic similarity that is insufficient to form the conclusion of ‘source identification;’ a lack of any observed microscopic similarity; or microscopic dissimilarity that is insufficient to form the conclusion of ‘source exclusion.’

IV. Qualifications and Limitations of Forensic Firearms/Toolmarks Discipline Examinations
• A conclusion provided during testimony or in a report is ultimately an examiner’s decision and is not based on a statistically-derived or verified measurement or comparison to all other firearms or toolmarks. Therefore, an An examiner shall not assert that two toolmarks originated from the same source to the exclusion of all other sources. This may wrongly imply that a ‘source identification’ conclusion is based upon a statistically-derived or verified measurement or an actual comparison to all other toolmarks in the world, rather than an examiner’s expert opinion.
○ assert that a ‘source identification’ or a ‘source exclusion’ conclusion is based on the ‘uniqueness’3 of an item of evidence.

○ use the terms ‘individualize’ or ‘individualization’ when describing a source conclusion.

○ assert that two toolmarks originated from the same source to the exclusion of all other sources.
• An examiner shall not assert that examinations conducted in the forensic firearms/toolmarks discipline are infallible or have a zero error rate.

• An examiner shall not provide a conclusion that includes a statistic or numerical degree of probability except when based on relevant and appropriate data.

• An examiner shall not cite the number of examinations conducted in the forensic firearms/toolmarks discipline performed in his or her career as a direct measure for the accuracy of a proffered conclusion provided. An examiner may cite the number of examinations conducted in the forensic firearms/toolmarks discipline performed in his or her career for the purpose of establishing, defending, or describing his or her qualifications or experience.

• An examiner shall not assert that two toolmarks originated from the same source with absolute or 100% certainty, or use the expressions ‘reasonable degree of scientific certainty,’ ‘reasonable scientific certainty,’ or similar assertions of reasonable certainty in either reports or testimony unless required to do so by a judge or applicable law.34


2 Inductive reasoning (inferential reasoning):
A mode or process of thinking that is part of the scientific method and complements deductive reasoning and logic. Inductive reasoning starts with a large body of evidence or data obtained by experiment or observation and extrapolates it to new situations. By the process of induction or inference, predictions about new situations are inferred or induced from the existing body of knowledge. In other words, an inference is a generalization, but one that is made in a logical and scientifically defensible manner. Oxford Dictionary of Forensic Science 130 (Oxford Univ. Press 2012).
3 As used in this document, the term ‘uniqueness’ means having the quality of being the only one of its kind.’ Oxford English Dictionary 804 (Oxford Univ. Press 2012).
34 See Memorandum from the Attorney General to Heads of Department Components (Sept. 9. 2016), https://www.justice.gov/opa/file/891366/download.

1

Are the two or three conclusions -- identification, exclusion, and inconclusive -- the only way the examiners are allowed to report their results?

In much of the world, examiners are discouraged from reaching only two conclusions--included vs. excluded (with the additional option of denominating the data as too limited to permit such a classification). They are urged to articulate how strongly the data support one classification over the other. Instead of pigeonholing, they might say, for example, that the data strongly support the same-source classification (because those data are far more probable for ammunition fired from the same gun than for ammunition discharged from different guns). 

The ULTR studiously avoids mentioning this mode of reporting. It states that examiners "may provide any of the following ... ." It does not state whether they also may choose not to -- and instead report only the degree of support for the same-source (or the different-source) hypothesis. Does the maxim of expressio unius est exclusio alterius apply? Department of Justice personnel are well aware of this widely favored alternative. They have attended meetings of statisticians at which straw polls overwhelmingly endorsed it over the Department's permitted conclusions. Yet, the ULTR does not list statements of support (essentially, likelihood ratios) as permissible. But neither are they found in the list of thou-shalt-nots in Part IV. \3/ Is the idea that if the examiners have a conclusion to offer, they must state it as one of the two or three categorical ones -- and that they may give a qualitative likelihood ratio if they want to?

2

Is the stated logic of a "source identification" internally coherent and intellectually defensible?

The ULTR explains that "[t]he basis for a 'source identification' is

an examiner’s opinion that the observed class characteristics and corresponding individual characteristics provide extremely strong support for the proposition that the two toolmarks originated from the same source and extremely weak support for the proposition that the two toolmarks originated from different sources.

Translated into likelihood language, the DoJ's "basis for a source identification" is the belief that the likelihood ratio is very large -- the numerator of L is close to one, and the denominator is close to zero (see Box 2).

On this understanding, a "source identification" is a statement about the strength of the evidence rather than a conclusion (in the sense of a decision about the source hypothesis). However, the next paragraph of the ULTR states that "[a] ‘source identification’ is the statement of an examiner’s opinion (an inductive inference2) that the probability that the two toolmarks were made by different sources is so small that it is negligible."

Box 2. A Technical Definition of Support
The questioned toolmarks and the known ones have some degree of observed similarity X with respect to relevant characteristics. Let Lik(S) be the examiner's judgment of the likelihood of the same-source hypothesis S. This likelihood is proportional to Prob(X | S), the probability of the observed degree of similarity X given the hypothesis (S). For simplicity, we may as well let the constant be 1. Let Lik(D) be the examiner's judgment of the likelihood of the different-source hypothesis (D). This likelihood is Prob(X | D). The support for S is the logarithm of the likelihood ratio L = Lik(S) / Lik(D) = Prob(X | S) / Prob(X | D). \4/

In this way, the ULTR jumps from a likelihood to a posterior probability. To assert that "the probability that the two toolmarks were made by different sources ... is negligible" is to say that Prob(D|X) is close to 0, and hence that Prob(S|X) is nearly 1. However, the likelihood ratio L = Lik(S) / Lik(D) is only one factor that affects Prob(D|X). Bayes' theorem establishes that

Odds(D|X) = Odds(D) / L.

Consequently, a very large L (great support for S) shrinks the odds in favor of S, but whether we end up with a "negligible" probability for D depends on the odds on D without considering the strength of the toolmark evidence. Because the expertise of toolmark analysts only extends to evaluating the toolmark evidence, it seems that the ULTR is asking them to step outside their legitimate  sphere of expertise by assessing, either explicitly or implicitly, the strength of the particular non-scientific evidence in the case.

There is a way to circumvent this objection. To defend a "source identification" as a judgment that Prob(D|X) is negligible, the examiner could contend that the likelihood ratio L is not just very large, as the ULTR's first definition required, but that it is so large that it swamps every probability that a judge or juror reasonably might entertain in any possible case before learning about the toolmarks. A nearly infinite L would permit an analyst to dismiss the posterior odds on D as negligible without attempting to estimate the odds on the basis of other evidence in the particular case (see Box 3).

Box 3. How large must L be to swamp all plausible prior odds?

Suppose that the smallest prior same-source probability in any conceivable case were p = 1/1,000,000. The prior odds on the different-source hypothesis would be approximately 1/p = 1,000,000. According to Bayes' rule, the posterior odds on D then would be about (1/p)/L = 1,000,000/L.

How large would the support L for S have to be to make D a "negligible" possibility? If "negligible" means a probability below, say 1/100,000, then the threshold value of L (call it L*) would be such that 1,000,000 / L* < 1/100,000 (approximately). Hence L* > 10^11. Are examiners are able to reliably tell whether the toolmarks are such that L > 100 billion?

One can use different numbers, of course, but whether the swamping defense of the ULTR really works to justify actual testimony as to "source identification" defined according to the ULTR is none too clear.

The ULTR seems slightly embarrassed with the characterization of a "source identification" as an "opinion" on the small size of a probability. Parenthetically it calls the opinion "an inductive inference," which sounds more impressive. But the footnote that is supposed to explain the the more elegant phrase only muddies the waters. It reads as follows:

Inductive reasoning (inferential reasoning): A mode or process of thinking that is part of the scientific method and complements deductive reasoning and logic. Inductive reasoning starts with a large body of evidence or data obtained by experiment or observation and extrapolates it to new situations. By the process of induction or inference, predictions about new situations are inferred or induced from the existing body of knowledge. In other words, an inference is a generalization, but one that is made in a logical and scientifically defensible manner. Oxford Dictionary of Forensic Science 130 (Oxford Univ. Press 2012) [sic]. \4/

The flaws in this definition are many. First, "inferential reasoning" is not equivalent to "inductive reasoning." Inference is reaching a conclusion from stated premises. The argument from the premises to the conclusion can be deductive or inductive. Deductive arguments are valid when the conclusion is true given that the premises are true. Inductive arguments are sound when the conclusion is sufficiently probable given that the premises are true. Second, inductive reasoning can be based on a small body of evidence as well as on a large body of evidence. In other words, deduction produces logical certainty, whereas induction can yield no more than probable truth. Third, an induction -- that is, the conclusion of an inductive argument -- need not be particularly scientific or "scientifically defensible." Fourth, an inductive conclusion is not necessarily "a generalization." An inductive argument, no less than a deductive one, can go from the general to the specific -- as is the case for an inference that two toolmarks were made by the same source. Presenting an experience-based opinion as the product of "the scientific method" by the fiat of a flawed definition of "inductive reasoning" is puffery.

3

If the examiner has correctly discerned matching "individual characteristics" (as the ULTR calls them), why cannot the examiner "assert that a ‘source identification’ ... is based on ... ‘uniqueness’" or that there has been an "individualization"?

The ULTR states that a "source identification" is based on an examination of "class characteristics" and "individual characteristics." Presumably, "individual characteristics" are ones that differ in every source and thus permit "individualization." The dictionary on which the ULTR relies defines "individualization" as "assigning a unique source for a given piece of physical evidence" (which it distinguishes from "identification"). But the ULTR enjoins an examiner from using "the terms ‘individualize’ or ‘individualization’ when describing a source conclusion," from asserting "that a ‘source identification’ or a ‘source exclusion’ conclusion is based on the ‘uniqueness’ of an item of evidence," and from stating "that two toolmarks originated from the same source to the exclusion of all other sources."

The stated reason to avoid these terms is that a source attribution "is not based on a statistically-derived or verified measurement or comparison to all other firearms or toolmarks." But who would think that an examiner who "assert[s] that two '[t]oolmarks originated from the same source to the exclusion of all other sources'" is announcing "an actual comparison to all other toolmarks in the world"? The examiner apparently is allowed to report a plethora of matching "individual characteristics" and to opine (or "inductively infer") that there is virtually no chance that the marks came from a different source. Allowing such testimony cuts the heart out of the rules against asserting "uniqueness" and claiming "individualization."

NOTES

  1. E.g., United States v. Hunt, 464 F.Supp.3d 1252 (W.D. Okla. 2020) (discussed on this blog Aug. 10, 2020).
  2. The original version was adopted on 7/24/2018. It was revised on 6/8/2020.
  3. Are numerical versions of subjective likelihood ratios prohibited by the injunction in Part IV that "[a]n examiner shall not provide a conclusion that includes a statistic or numerical degree of probability except when based on relevant and appropriate data"? Technically, a likelihood ratio is not a "degree of probability" or (arguably) a statistic, but it seems doubtful that the drafters of the ULTR chose their terminology with the niceties of statistical terminology in mind.
  4. A.W.F. Edwards, Likelihood 31 (rev. ed. 1992) (citing H. Jeffreys, Further Significance Tests, 32 Proc. Cambridge Phil. Soc'y 416 (1936)).
  5. The correct name of the dictionary is A Dictionary of Forensic Science, and its author is Suzanne Bell. The quotation in the ULTR omits the following part of the definition of "inductive inference": "A forensic example is fingerprints. Every person's fingerprints are unique, but this is an inference based on existing knowledge since the only way to prove it would be to take and study the fingerprints of every human being ever born."

Tuesday, November 24, 2020

Wikimedia v. NSA: It's Classified!

The National Security Agency (NSA) engages in systematic, warrantless "upstream" surveillance of Internet communications that travel in and out of the United States along a "backbone" of fiber optic cables. The ACLU and other organizations maintain that Upstream surveillance is manifestly unconstitutional. Whether or not that is correct, the government has stymied one Fourth Amendment challenge after another on the ground that plaintiffs lacked standing because they cannot prove that the surveillance entails intercepting, copying, and reviewing any of their communications. Of course, the reason plaintiffs have no direct evidence is that the government won't admit or deny it. Instead, the government has asserted that the surveillance program is a privileged state secret, classified its details, and resisted even in camera hearings in ordinary courts.

In Wikimedia Foundation v. National Security Agency, 857 F.3d 193 (4th Cir. 2017), however, the Court of Appeals for the Fourth circuit held that the Wikimedia Foundation, which operates Wikipedia, made "allegations sufficient to survive a facial challenge to standing." Id. at 193. The court concluded that Wikimedia's allegations were plausible enough to defeat a motion to dismiss the complaint because

Wikimedia alleges three key facts that are entitled to the presumption of truth. First, “[g]iven the relatively small number of international chokepoints,” the volume of Wikimedia's communications, and the geographical diversity of the people with whom it communicates, Wikimedia's “communications almost certainly traverse every international backbone link connecting the United States with the rest of the world.”

Second, “in order for the NSA to reliably obtain communications to, from, or about its targets in the way it has described, the government,” for technical reasons that Wikimedia goes into at length, “must be copying and reviewing all the international text-based communications that travel across a given link” upon which it has installed surveillance equipment. Because details about the collection process remain classified, Wikimedia can't precisely describe the technical means that the NSA employs. Instead, it spells out the technical rules of how the Internet works and concludes that, given that the NSA is conducting Upstream surveillance on a backbone link, the rules require that the NSA do so in a certain way. ...

Third, per the PCLOB [Privacy and Civil Liberties Oversight Board] Report and a purported NSA slide, “the NSA has confirmed that it conducts Upstream surveillance at more than one point along the [I]nternet backbone.” Together, these allegations are sufficient to make plausible the conclusion that the NSA is intercepting, copying, and reviewing at least some of Wikimedia's communications. To put it simply, Wikimedia has plausibly alleged that its communications travel all of the roads that a communication can take, and that the NSA seizes all of the communications along at least one of those roads. 

Id. at 210-11 (citations omitted).

The Fourth Circuit therefore vacated an order dismissing Wikimedia's complaint issued by Senior District Judge Thomas Selby Ellis III, the self-described "impatient" jurist who achieved later notoriety and collected ethics complaints (that were rejected last year) for his management of the trial of former Trump campaign manager Paul Manafort.

On remand, the government moved for summary judgment. Wikimedia Found. v. Nat'l Sec. Agency/Cent. Sec. Serv., 427 F.Supp.3d 582 (D. Md. 2019). Once more, the government argued that Wikimedia lacked standing to complain that the Upstream surveillance violated its Fourth Amendment rights. It suggested that the "plausible" inference that the NSA must be "intercepting, copying, and reviewing at least some of Wikimedia's communications” recognized by the Fourth Circuit was not so plausible after all. To support this conclusion, it submitted a declaration of Henning Schulzrinne, a Professor of Computer Science and Electrical Engineering at Columbia University. Dr. Schulzrinne described how companies carrying Internet traffic might filter transmissions before copying them by “mirroring” with “routers” or “switches” that could perform “blacklisting” or “whitelisting” if the NSA chose to give the companies information on its targets with which to create “access control lists.”

But Dr. Schulzrinne supplied no information and formed no opinion on whether it was at all likely that the NSA used the mirroring methods that he envisioned, and Wikimedia produced a series of expert reports from Scott Bradner, who had served as Harvard University’s Technology Security Officer and taught at that university. Bradner contended that the NSA could hardly be expected to give away the information on its targets and concluded that it is all but certain that the agency intercepted and opened at least one of Wikimedia's trillions of Internet communications.

The district court refused to conduct an evidentiary hearing on the factual issue. Instead, it disregarded the expert's opinion as inadmissible scientific evidence under Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993), because no one without access to classified information could "know what the NSA prioritizes in the Upstream surveillance program ... and therefore Mr. Bradner has no knowledge or information about it." Wikimedia, 427 F. Supp. 3d at 604–05 (footnotes omitted).

This reasoning resembles that from Judge Ellis's first opinion in this long-running case. In Wikimedia Found. v. Nat'l Sec. Agency, 143 F. Supp. 3d 344, 356 (D. Md. 2015), the judge characterized Wikimedia’s allegations as mere “suppositions and speculation, with no basis in fact, about how the NSA” operates and maintained that it was impossible for Wikimedia to prove its allegations “because the scope and scale of Upstream surveillance remain classified . . . .” Id. Rather than allow full consideration of the strength of the evidence that makes Wikimedia’s claim plausible, the district court restated its position that “Mr. Bradner has no [direct] knowledge or information” because that information is classified. Wikimedia, 427 F. Supp. 3d at 604–605.

In a pending appeal to the Fourth Circuit, Edward Imwinkelried, Michael Risinger, Rebecca Wexler, and I prepared a brief as amici curiae in support of Wikimedia. The brief expresses surprise at “the district court’s highly abbreviated analysis of Rule 702 and Daubert, as well as the court’s consequent decision to rule inadmissible opinions of the type that Wikimedia’s expert offered in this case.” It describes the applicable standard for excluding expert testimony. It then argues that the expert’s method of reasoning was sound and that its factual bases regarding the nature of Internet communications and surveillance technology, together with public information on the goals and needs of the NSA program, were sufficient to justify the receipt of the proposed testimony.

UPDATE (9/27/21): On 9/15/21, the Fourth Circuit affirmed the summary judgment order -- but not on the basis of Judge Ellis's theories about expert testimony. A divided panel reasoned that the suit had to be dismissed because the government had properly invoked the state secrets privilege and that because the government would have to disclose those secrets to defend itself, “further litigation would present an unjustifiable risk of disclosure.” Wikimedia Found. v. Nat'l Sec. Agency/Cent. Sec. Serv., 14 F.4th 276 (4th Cir. 2021).

Monday, September 28, 2020

Terminology Department: Significance

Inns of Court College of Advocacy, Guidance on the Preparation, Admission and Examination of Expert Evidence § 5.2 (3d ed. 2020)
Statisticians, for example, use what appear to be everyday words in specific technical senses. 'Significance' is an example. In everyday language it carries associations of importance, something with considerable meaning. In statistics it is a measure of the likelihood that a relationship between two or more variables is caused by something other than random chance.
Welcome to the ICCA

The Inns of Court College of Advocacy ... is the educational arm of the Council of the Inns of Court. The ICCA strives for ‘Academic and Professional Excellence for the Bar’. Led by the Dean, the ICCA has a team of highly experienced legal academics, educators and instructional designers. It also draws on the expertise of the profession across the Inns, Circuits, Specialist Bar Associations and the Judiciary to design and deliver bespoke training for student barristers and practitioners at all levels of seniority, both nationally, pan-profession and on an international scale.

How good is the barristers' definition of statistical significance? In statistics, an apparent association between variables is said to be significant when it is lies outside the range that one would expect to see in some large fraction of repeated, identically conducted studies in which the variables are in fact uncorrelated. Sir Ronald Fisher articulated the idea as follows:

[I]t is convenient to draw the line at about the level at which we can say: ‘Either there is something in the treatment, or a coincidence has occurred such as does not occur more than once in twenty trials.’ This level ... we may call the 5 per cent. point .... If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent. point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach that level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance. [1]

For Fisher, a "significant" result would occur by sheer coincidence no "more than once in twenty trials" (on average). 

Is such statistical significance the same as the barristers' "likelihood" that an observed "relationship ... is caused by something other than random chance"? One might object to the appearance of the term "likelihood" in the definition because it too is a technical term with a specialized meaning in statistics, but that is not the main problem. The venacular likelihood that X is the cause of extreme data (where X is anything other than random chance) is not a "level of significance" such as 5%, 2%, or 1%. These levels are conditional error probabilities: If the variables are uncorrelated and we use a given level to call the observed results significant, then, in the (very) long run, we will label coincidental results as significant no more than that level specifies. For example, if we always use a 0.01 level, we will call coincidences "significant" no more than 1% of the time (in the limit).

The probability (the venacular likelihood) "that a relationship between two or more variables is caused by something other than random chance" is quite different. [2, p.53] Everything else being equal, significant results are more likely to signal a true relationship than are nonsignificant results, but the significance level itself refers to the probability of data that are uncommon when there is no true relationship, and not to the probability that the apparent relationship is real. In symbols, Pr(relationship | extreme data) is not Pr(extreme data | relationship). Naively swapping the terms in the expressions for the conditional probabilities is known as the transposition fallacy. In regard to criminal cases involving statistical evidence, it often is called the "prosecutor's fallacy." Perhaps "barristers' fallacy" can be added to the list.

REFERENCES

  1. Ronald Fisher, The Arrangement of Field Experiments, 33 J. Ministry Agric. Gr. Brit 503-515, 504 (1926).
  2. David H. Kaye, Frequentist Methods for Statistical Inference, in Handbook of Forensic Statistics 39-72 (David Banks et al. eds. 2021).

ACKNOWLEDGMENT: Thanks to Geoff Morrison for alerting me to the ICCA definition.

Wednesday, August 26, 2020

Terminology Department: Defining Bias for Nonstatisticians

The Organization of Scientific Area Committees for Forensic Science (OSAC) is trying to develop definitions of common technical terms that can be used across most forensic-science subject areas. "Bias" is one of these ubiquitous terms, but its statistical meaning does not conform to the usual dictionary definitions, such as  "an inclination of temperament or outlook, especially: a personal and sometimes unreasoned judgment" \1/ or "the action of supporting or opposing a particular person or thing in an unfair way, because of allowing personal opinions to influence your judgment." \2/ 

I thought the following definition might be useful for forensic-science practitioners:

A systematic tendency for estimates or measurements to be above or below their true values. A study is said to be biased if its design is such that it systematically favors certain outcomes. An estimator of a population parameter is biased when the average value of the estimates (from an infinite number of samples) would not equal the value of the parameter. Bias arises from systematic as opposed to random error in the collection of units to be measured, the measurement of the units, or the process for estimating quantities based on the measurements.

It ties together some of the simplest definitions I have seen in textbooks and reference works on statistics -- namely:

Yadolah Dodge, The Concise Encyclopedia of Statistics 41 (2008): From a statistical point of view, the bias is defined as the difference between the expected value of a statistic and the true value of the corresponding parameter. Therefore, the bias is a measure of the systematic error of an estimator. If we calculate the mean of a large number of unbiased estimations, we will find the correct value. The bias indicates the distance of the estimator from the true value of the parameter. Comment: This is the definition for mathematical statistics.
B. S. Everitt & A. Skrondal, The Cambridge Dictionary of Statistics 45 (4th ed. 2010) (citing Altman, D.G. (1991) Practical Statistics for Medical Research, Chapman and Hall, London): In general terms, deviation of results or inferences from the truth, or processes leading to such deviation. More specifically, the extent to which the statistical method used in a study does not estimate the quantity thought to be estimated, or does not test the hypothesis to be tested. In estimation usually measured by the difference between the expected value of an estimator and the true value of the parameter. An estimator for which E(θ-hat) = θ is said to be unbiased. See also ascertainment bias, recall bias, selection bias and biased estimator. Comment: The general definition (first sentence) fails to differentiate between random and systematic deviations. The “more specific” definition in the next sentence is limited to the definition in mathematical statistics.
David H. Kaye & David A. Freedman, Reference Guide on Statistics, in Reference Manual on Scientific Evidence 283 (Federal Judicial Center & Nat’l Research Council eds., 3d ed. 2011): Also called systematic error. A systematic tendency for an estimate to be too high or too low. An estimate is unbiased if the bias is zero. (Bias does not mean prejudice, partiality, or discriminatory intent.) See nonsampling error. Comment: This one is intended to convey the essential idea to judges.
David H. Kaye, Frequentist Methods for Statistical Inference, in Handbook of Forensic Statistics 39, 44 (D. Banks, K. Kafadar, D. Kaye & M. Tackett eds. 2020): [A]n unbiased estimator t of [a parameter] θ will give estimates whose errors eventually should average out to zero. Error is simply the difference between the estimate and the true value. For an unbiased estimator, the expected value of the errors is E(t – θ) = 0. Comment: Yet another version of the definition of an unbiased estimator of a population or model parameter.
JCGM, International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM) (3d ed. 2012): measurement bias, bias -- estimate of a systematic measurement error Comment: The VIM misdefines bias as an estimate of bias.
David S. Moore & George P. McCabe, Introduction to the Practice of Statistics 232 (2d ed. 1993): Bias. The design of a study is biased if it systematically favors certain outcomes. In a causal study, bias can result from confounding. Or can it?

NOTES

  1. Merriam-Webster Dictionary (online).
  2. Cambridge Dictionary (online)

Saturday, August 22, 2020

Phrases for the Value or Weight of Evidence

A few statisticians asked me (independently) about the usage of the terms evidential value, evidentiary value, and probative value. For years, I thought the phrases all meant the same thing, but that is not true in some fields.

Evidential Value

Black’s Law Dictionary (which tends to have aged definitions) has this definition of evidential value: “Value of records given as or in support of evidence, based on the certainty of the records origins. The value here is not in the record content. This certainty is essential for authentic and adequate evidence of an entity’s actions, functioning, policies, and/or structure.”

Under this definition, "evidential value" pertains to a document's value merely as the container of information. The definition distinguishes between the provenance and authenticity of a document -- where did it come from and has it been altered? -- and the content of the document -- what statements or information does it contain? Likewise, archivists distinguish between "evidential value" and "informational value." The former, according to the Society of American Archivists "relates to the process of creation rather than the content (informational value) of the records."

Evidentiary Value

Lawyers use the phrases "evidentiary value" and "probative value" (or "probative force") as synonyms. For example, a 1932 note in the University of Pennsylvania Law Review on "Evidentiary Value of Finger-Prints" predicted that "the time is not far distant when courts must scrutinize and properly evaluate the probative force to be given to evidence that finger-prints found on the scene correspond with those of the accused." \1/

Forensic scientists use "evidentiary value" to denote the utility of examining objects for information on whether the objects have a common origin. A 2009 report of a committee of the National Academies complained that there was no standard threshold for deciding when bitemarks have "reached a threshold of evidentiary value." \2/ More generally, the phrase can denote the value of any expert analysis of effects as proof of the possible cause of those effects. \3/

Evidential Value

Unlike archivists, forensic scientists use the phrase “evidential value” interchangeably with "evidentiary value." It appears routinely in titles or articles and books such as "Evidential Value of Multivariate Physicochemical Data," \4/ "Enhancing the Evidential Value of Fingermarks Through Successful DNA Typing," \5/ and "Establishing the Evidential Value of a Small Quantity of Material Found at a Crime Scene." \6/

Probative Value

Lawyers use "probative value" to denote the degree to which an item of evidence proves the proposition it is offered to prove. Credible evidence that a defendant threatened to kill the deceased, whose death was caused by a poison, is probative of whether the death was accidental and whether defendant was the killer. With circumstantial evidence like this, various probability-based formulations have been proposed to express probative value quantitatively. \7/ One of the simplest is the likelihood ratio or Bayes factor (BF) favored by most forensic statisticians. \8/ Its logarithm has qualities that argue for using log(BF) to express the "weight" of an item of evidence. \9/

The rules of evidence require judges to exclude evidence when unfair prejudice, distraction, and undue consumption of time in presenting the evidence substantially outweigh the probative value of the evidence. \10/ In theory, judges do not exclude evidence just because they do not believe that the witness is telling the truth. The jury will take credibility into account in deciding the case. However, in ensuring that there is sufficient probative value to bother with the evidence, judges can hardly avoid being influenced by the trustworthiness of the source of the information. Moreover, the importance of the fact that the proposed testimony addresses and the availability of alternative, less prejudicial proof also can influence the decision to exclude evidence that is probative of a material fact. \11/

NOTES

  1. Note, Evidentiary Value of Finger-Prints, 80 U. Penn. L. Rev. 887 (1932).
  2. Comm. on Identifying the Needs of the Forensic Sci. Cmty. Nat'l Research Council, Strengthening Forensic Science in the United States: A Path Forward 176 (2009).
  3. Nicholas Dempsey & Soren Blau, Evaluating the Evidentiary Value of the Analysis of Skeletal Trauma in Forensic Research: A Review of Research and Practice, 307 Forensic Sci. Int'l (2020), https://doi.org/10.1016/j.forsciint.2020.110140. Still another usage of the term occurs in epistemology. See P. Gärdenfors, B. Hansson, N-E. Sahlin, Evidentiary Value: Philosophical, Judicial and Psychological Aspects of a Theory (1983); Dennis V. Lindley, Review, 35(3) Brit. J. Phil. Sci. 293-296 (1984) (criticizing this theory).
  4. Grzegorz Zadora, Agnieszka Martyna, Daniel Ramos & Colin Aitken, Statistical Analysis in Forensic Science: Evidential Value of Multivariate Physicochemical Data (2014).
  5. Zuhaib Subhani, Barbara Daniel & Nunzianda Frascione, DNA Profiles from Fingerprint Lifts—Enhancing the Evidential Value of Fingermarks Through Successful DNA Typing, 64(1) J. Forensic Sci. 201–06 (2019), https://doi.org/10.1111/1556-4029.13830.
  6. I.W. Evett, Establishing the Evidential Value of a Small Quantity of Material Found at a Crime Scene, 33(2) J. Forensic Sci. Soc’y 83-86 (1993).
  7. 1 McCormick on Evidence § 185 (R. Mosteller ed., 8th ed. 2020); David H. Kaye, Review-essay, Digging into the Foundations of Evidence Law, 116 Mich. L. Rev. 915-34 (2017), http://ssrn.com/abstract=2903618.
  8. See Anuradha Akmeemana, Peter Weis, Ruthmara Corzo, Daniel Ramos, Peter Zoon, Tatiana Trejos, Troy Ernst, Chip Pollock, Ela Bakowska, Cedric Neumann & Jose Almirall, Interpretation of Chemical Data from Glass Analysis for Forensic Purposes, J. Chemometrics (2020), DOI:10.1002/cem.3267.
  9. Fed. R. Evid. 403; Unif. R. Evid. 403; 1 McCormick, supra note 6, § 185.
  10. I. J. Good, Weight of Evidence: A Brief Survey, in 2 Bayesian Statistics 249–270 (Bernardo, J. M., M. H. DeGroot, D. V. Lindley & A. F. M. Smith eds., 1985); Irving John Good, Weight of Evidence and the Bayesian Likelihood Ratio, in The Use of Statistics in Forensic Science 85–106 (C. G. G. Aitken & David A. Stoney eds., 1991).
  11. 1 McCormick, supra note 6, § 185.