What's Uniqueness Got to Do with It?

Columbia University has announced that "AI Discovers That Not Every Fingerprint Is Unique"! The subtitle of the press release of January 10, 2024, boldly claims that

Columbia engineers have built a new AI that shatters a long-held belief in forensics–that fingerprints from different fingers of the same person are unique. It turns out they are similar, only we’ve been comparing fingerprints the wrong way!

Forensic Magazine immediately and uncritically rebroadcast (quoting verbatim without acknowledgment from the press release) the confused statements about uniqueness. According to the Columbia release and Forensic Magazine, "It’s a well-accepted fact in the forensics community that fingerprints of different fingers of the same person—or intra-person fingerprints—are unique and therefore unmatchable." Forensics Magazine adds that "Now, a new study shows an AI-based system has learned to correlate a person’s unique fingerprints with a high degree of accuracy."

Does this mean that the "well-accepted fact" and "long-held belief" in uniqueness been shattered or not? Clearly, not. The study is about similarity, not uniqueness. In fact, uniqueness has essentially nothing to do with it. I can classify equilateral triangles drawn on a flat surface as triangles rather than as other regular polygons whether or not the triangles are each different enough from one another (uniqueness within the set of triangles) that I notice these differences. To say that objects "are unique and therefore unmatchable" is a nonsequitur. A human genome is probably unique to that individual, but forensic geneticists know that six-locus STR profiles are "matchable" to those of other individuals in the population. A cold hit to a person who could not have been the source of the six-locus profile in the U.K. database occurred long ago (as was to be expected for the random-match probabilities of the genotypes).

Perhaps the myth that the study shatters is that it is impossible to distinguish fingerprints left by different fingers of the same individual X from fingerprints left by fingers of different individuals (not-X). But there is no obvious reason why this would be impossible even if every print is distinguishable from every other print (uniqueness).

The Columbia press release describes the study design this way:

[U]ndergraduate senior Gabe Guo ... who had no prior knowledge of forensics, found a public U.S. government database of some 60,000 fingerprints and fed them in pairs into an artificial intelligence-based system known as a deep contrastive network. Sometimes the pairs belonged to the same person (but different fingers), and sometimes they belonged to different people.

Over time, the AI system, which the team designed by modifying a state-of-the-art framework, got better at telling when seemingly unique fingerprints belonged to the same person and when they didn’t. The accuracy for a single pair reached 77%. When multiple pairs were presented, the accuracy shot significantly higher, potentially increasing current forensic efficiency by more than tenfold.

The press release reported the following odd facts about the authors' attempts to publish their study in a scientific journal:

Once the team verified their results, they quickly sent the findings to a well-established forensics journal, only to receive a rejection a few months later. The anonymous expert reviewer and editor concluded that “It is well known that every fingerprint is unique,” and therefore it would not be possible to detect similarities even if the fingerprints came from the same person.

The team ... fed their AI system even more data, and the system kept improving. Aware of the forensics community's skepticism, the team opted to submit their manuscript to a more general audience. The paper was rejected again, but [Professor Hod] Lipson ... appealed. “I don’t normally argue editorial decisions, but this finding was too important to ignore,” he said. “If this information tips the balance, then I imagine that cold cases could be revived, and even that innocent people could be acquitted.” ...

After more back and forth, the paper was finally accepted for publication by Science Advances. ... One of the sticking points was the following question: What alternative information was the AI actually using that has evaded decades of forensic analysis? ... “The AI was not using ... the patterns used in traditional fingerprint comparison,” said Guo ... . “Instead, it was using something else, related to the angles and curvatures of the swirls and loops in the center of the fingerprint.”

Proprietary fingerprint matching algorithms also do not arrive at matches the way human examiners do. They "see" different features in the patterns and tend to rank the top candidates for true matches in a database trawl differently than the human experts. Again, however, these facts about automated systems neither prove nor disprove claims of uniqueness. And, theoretical uniqueness has little or nothing to do with the actual probative value of assertions of matches by humans, automated systems, or both.

Although not directly applicable, the day after the publicity on the Guo et al. paper, I came across the following report on "Limitations of AI-based predictive models" in a weekly survey of papers in Science:

A central promise of artificial intelligence (AI) in health care is that large datasets can be mined to predict and identify the best course of care for future patients. Unfortunately, we do not know how these models would perform on new patients because they are rarely tested prospectively on truly independent patient samples. Chekroud et al. showed that machine learning models routinely achieve perfect performance in one dataset even when that dataset is a large international multisite clinical trial (see the Perspective by Petzschner). However, when that exact model was tested in truly independent clinical trials, performance fell to chance levels. Even when building what should be a more robust model by aggregating across a group of similar multisite trials, subsequent predictive performance remained poor. -- Science p. 164, 10.1126/science.adg8538; see also p. 149, 10.1126/science.adm9218

Note: This posting was last modified on 1/12/24 2:45 PM

Mysteries of the Department of Justice's ULTR for Firearm-toolmark Pattern Examinations

The Department of Justice's Uniform Language for Testimony and Reports (ULTR) for the Forensic Firearms/toolmarks Discipline – Pattern Examination" offers a ready response to motions to limit overclaiming or, to use the pedantic term, ultracrepidarianism, in expert testimony. Citing the DoJ policy, several federal district courts have indicated that they expect the government's expert witnesses to follow this directive (or something like it). \1/

Parts of the current version (with changes from the original) are reproduced in Box 1. \2/ This posting poses three questions about this guidance. Although the ULTR is a step in the right direction, it has a ways to go in articulating a clear and optimal policy.

Box 1. The ULTR

---

DEPARTMENT OF JUSTICE
UNIFORM LANGUAGE FOR TESTIMONY AND REPORTS
FOR THE FORENSIC FIREARMS/TOOLMARKS DISCIPLINE –
PATTERN MATCH EXAMINATION ...
III. Conclusions Regarding Forensic Pattern Examination of Firearms/Toolmarks Evidence for a Pattern Match

The An examiner may offer provide any of the following conclusions:

1.Source identification (i.e., identified)
2.Source exclusion (i.e., excluded)
3.Inconclusive

Source identification
‘Source identification’ is an examiner’s conclusion that two toolmarks originated from the same source. This conclusion is an examiner’s decision opinion that all observed class characteristics are in agreement and the quality and quantity of corresponding individual characteristics is such that the examiner would not expect to find that same combination of individual characteristics repeated in another source and has found insufficient disagreement of individual characteristics to conclude they originated from different sources.

The basis for a ‘source identification’ conclusion is an examiner’s decision opinion that the observed class characteristics and corresponding individual characteristics provide extremely strong support for the proposition that the two toolmarks came originated from the same source and extremely weak support for the proposition that the two toolmarks came originated from different sources.

A ‘source identification’ is the statement of an examiner’s opinion (an inductive inference²) that the probability that the two toolmarks were made by different sources is so small that it is negligible. A ‘source identification’ is not based upon a statistically-derived or verified measurement or an actual comparison to all firearms or toolmarks in the world.

Source exclusion
‘Source exclusion’ is an examiner’s conclusion that two toolmarks did not originate from the same source.

The basis for a ‘source exclusion’ conclusion is an examiner’s decision opinion that the observed class and/or individual characteristics provide extremely strong support for the proposition that the two toolmarks came from different sources and extremely weak or no support for the proposition that the two toolmarks came from the same source two toolmarks can be differentiated by their class characteristics and/or individual characteristics.

Inconclusive
‘Inconclusive’ is an examiner’s conclusion that all observed class characteristics are in agreement but there is insufficient quality and/or quantity of corresponding individual characteristics such that the examiner is unable to identify or exclude the two toolmarks as having originated from the same source.

The basis for an ‘inconclusive’ conclusion is an examiner’s decision opinion that there is an insufficient quality and/or quantity of individual characteristics to identify or exclude. Reasons for an ‘inconclusive’ conclusion include the presence of microscopic similarity that is insufficient to form the conclusion of ‘source identification;’ a lack of any observed microscopic similarity; or microscopic dissimilarity that is insufficient to form the conclusion of ‘source exclusion.’

IV. Qualifications and Limitations of Forensic Firearms/Toolmarks Discipline Examinations
• A conclusion provided during testimony or in a report is ultimately an examiner’s decision and is not based on a statistically-derived or verified measurement or comparison to all other firearms or toolmarks. Therefore, an An examiner shall not assert that two toolmarks originated from the same source to the exclusion of all other sources. This may wrongly imply that a ‘source identification’ conclusion is based upon a statistically-derived or verified measurement or an actual comparison to all other toolmarks in the world, rather than an examiner’s expert opinion.

○ assert that a ‘source identification’ or a ‘source exclusion’ conclusion is based on the ‘uniqueness’³ of an item of evidence.

○ use the terms ‘individualize’ or ‘individualization’ when describing a source conclusion.

○ assert that two toolmarks originated from the same source to the exclusion of all other sources.

• An examiner shall not assert that examinations conducted in the forensic firearms/toolmarks discipline are infallible or have a zero error rate.

• An examiner shall not provide a conclusion that includes a statistic or numerical degree of probability except when based on relevant and appropriate data.

• An examiner shall not cite the number of examinations conducted in the forensic firearms/toolmarks discipline performed in his or her career as a direct measure for the accuracy of a proffered conclusion provided. An examiner may cite the number of examinations conducted in the forensic firearms/toolmarks discipline performed in his or her career for the purpose of establishing, defending, or describing his or her qualifications or experience.

• An examiner shall not assert that two toolmarks originated from the same source with absolute or 100% certainty, or use the expressions ‘reasonable degree of scientific certainty,’ ‘reasonable scientific certainty,’ or similar assertions of reasonable certainty in either reports or testimony unless required to do so by a judge or applicable law.³⁴

---

² Inductive reasoning (inferential reasoning):

A mode or process of thinking that is part of the scientific method and complements deductive reasoning and logic. Inductive reasoning starts with a large body of evidence or data obtained by experiment or observation and extrapolates it to new situations. By the process of induction or inference, predictions about new situations are inferred or induced from the existing body of knowledge. In other words, an inference is a generalization, but one that is made in a logical and scientifically defensible manner. Oxford Dictionary of Forensic Science 130 (Oxford Univ. Press 2012).

³ As used in this document, the term ‘uniqueness’ means having the quality of being the only one of its kind.’ Oxford English Dictionary 804 (Oxford Univ. Press 2012).
³⁴ See Memorandum from the Attorney General to Heads of Department Components (Sept. 9. 2016), https://www.justice.gov/opa/file/891366/download.

1

Are the two or three conclusions -- identification, exclusion, and inconclusive -- the only way the examiners are allowed to report their results?

In much of the world, examiners are discouraged from reaching only two conclusions--included vs. excluded (with the additional option of denominating the data as too limited to permit such a classification). They are urged to articulate how strongly the data support one classification over the other. Instead of pigeonholing, they might say, for example, that the data strongly support the same-source classification (because those data are far more probable for ammunition fired from the same gun than for ammunition discharged from different guns). 

The ULTR studiously avoids mentioning this mode of reporting. It states that examiners "may provide any of the following ... ." It does not state whether they also may choose not to -- and instead report only the degree of support for the same-source (or the different-source) hypothesis. Does the maxim of expressio unius est exclusio alterius apply? Department of Justice personnel are well aware of this widely favored alternative. They have attended meetings of statisticians at which straw polls overwhelmingly endorsed it over the Department's permitted conclusions. Yet, the ULTR does not list statements of support (essentially, likelihood ratios) as permissible. But neither are they found in the list of thou-shalt-nots in Part IV. \3/ Is the idea that if the examiners have a conclusion to offer, they must state it as one of the two or three categorical ones -- and that they may give a qualitative likelihood ratio if they want to?

2

Is the stated logic of a "source identification" internally coherent and intellectually defensible?

The ULTR explains that "[t]he basis for a 'source identification' is

an examiner’s opinion that the observed class characteristics and corresponding individual characteristics provide extremely strong support for the proposition that the two toolmarks originated from the same source and extremely weak support for the proposition that the two toolmarks originated from different sources.

Translated into likelihood language, the DoJ's "basis for a source identification" is the belief that the likelihood ratio is very large -- the numerator of L is close to one, and the denominator is close to zero (see Box 2). On this understanding, a "source identification" is a statement about the strength of the evidence rather than a conclusion (in the sense of a decision about the source hypothesis). However, the next paragraph of the ULTR states that "[a] ‘source identification’ is the statement of an examiner’s opinion (an inductive inference2) that the probability that the two toolmarks were made by different sources is so small that it is negligible."

Box 2. A Technical Definition of Support The questioned toolmarks and the known ones have some degree of observed similarity X with respect to relevant characteristics. Let Lik(S) be the examiner's judgment of the likelihood of the same-source hypothesis S. This likelihood is proportional to Prob(X | S), the probability of the observed degree of similarity X given the hypothesis (S). For simplicity, we may as well let the constant be 1. Let Lik(D) be the examiner's judgment of the likelihood of the different-source hypothesis (D). This likelihood is Prob(X | D). The support for S is the logarithm of the likelihood ratio L = Lik(S) / Lik(D) = Prob(X | S) / Prob(X | D). \4/

In this way, the ULTR jumps from a likelihood to a posterior probability. To assert that "the probability that the two toolmarks were made by different sources ... is negligible" is to say that Prob(D|X) is close to 0, and hence that Prob(S|X) is nearly 1. However, the likelihood ratio L = Lik(S) / Lik(D) is only one factor that affects Prob(D|X). Bayes' theorem establishes that Odds(D|X) = Odds(D) / L. Consequently, a very large L (great support for S) shrinks the odds in favor of S, but whether we end up with a "negligible" probability for D depends on the odds on D without considering the strength of the toolmark evidence. Because the expertise of toolmark analysts only extends to evaluating the toolmark evidence, it seems that the ULTR is asking them to step outside their legitimate  sphere of expertise by assessing, either explicitly or implicitly, the strength of the particular non-scientific evidence in the case. There is a way to circumvent this objection. To defend a "source identification" as a judgment that Prob(D|X) is negligible, the examiner could contend that the likelihood ratio L is not just very large, as the ULTR's first definition required, but that it is so large that it swamps every probability that a judge or juror reasonably might entertain in any possible case before learning about the toolmarks. A nearly infinite L would permit an analyst to dismiss the posterior odds on D as negligible without attempting to estimate the odds on the basis of other evidence in the particular case (see Box 3).

Box 3. How large must L be to swamp all plausible prior odds? Suppose that the smallest prior same-source probability in any conceivable case were p = 1/1,000,000. The prior odds on the different-source hypothesis would be approximately 1/p = 1,000,000. According to Bayes' rule, the posterior odds on D then would be about (1/p)/L = 1,000,000/L. How large would the support L for S have to be to make D a "negligible" possibility? If "negligible" means a probability below, say 1/100,000, then the threshold value of L (call it L*) would be such that 1,000,000 / L* < 1/100,000 (approximately). Hence L* > 10^11. Are examiners are able to reliably tell whether the toolmarks are such that L > 100 billion? One can use different numbers, of course, but whether the swamping defense of the ULTR really works to justify actual testimony as to "source identification" defined according to the ULTR is none too clear.

The ULTR seems slightly embarrassed with the characterization of a "source identification" as an "opinion" on the small size of a probability. Parenthetically it calls the opinion "an inductive inference," which sounds more impressive. But the footnote that is supposed to explain the the more elegant phrase only muddies the waters. It reads as follows:

Inductive reasoning (inferential reasoning): A mode or process of thinking that is part of the scientific method and complements deductive reasoning and logic. Inductive reasoning starts with a large body of evidence or data obtained by experiment or observation and extrapolates it to new situations. By the process of induction or inference, predictions about new situations are inferred or induced from the existing body of knowledge. In other words, an inference is a generalization, but one that is made in a logical and scientifically defensible manner. Oxford Dictionary of Forensic Science 130 (Oxford Univ. Press 2012) [sic]. \4/

The flaws in this definition are many. First, "inferential reasoning" is not equivalent to "inductive reasoning." Inference is reaching a conclusion from stated premises. The argument from the premises to the conclusion can be deductive or inductive. Deductive arguments are valid when the conclusion is true given that the premises are true. Inductive arguments are sound when the conclusion is sufficiently probable given that the premises are true. Second, inductive reasoning can be based on a small body of evidence as well as on a large body of evidence. In other words, deduction produces logical certainty, whereas induction can yield no more than probable truth. Third, an induction -- that is, the conclusion of an inductive argument -- need not be particularly scientific or "scientifically defensible." Fourth, an inductive conclusion is not necessarily "a generalization." An inductive argument, no less than a deductive one, can go from the general to the specific -- as is the case for an inference that two toolmarks were made by the same source. Presenting an experience-based opinion as the product of "the scientific method" by the fiat of a flawed definition of "inductive reasoning" is puffery.

3

If the examiner has correctly discerned matching "individual characteristics" (as the ULTR calls them), why cannot the examiner "assert that a ‘source identification’ ... is based on ... ‘uniqueness’" or that there has been an "individualization"?

The ULTR states that a "source identification" is based on an examination of "class characteristics" and "individual characteristics." Presumably, "individual characteristics" are ones that differ in every source and thus permit "individualization." The dictionary on which the ULTR relies defines "individualization" as "assigning a unique source for a given piece of physical evidence" (which it distinguishes from "identification"). But the ULTR enjoins an examiner from using "the terms ‘individualize’ or ‘individualization’ when describing a source conclusion," from asserting "that a ‘source identification’ or a ‘source exclusion’ conclusion is based on the ‘uniqueness’ of an item of evidence," and from stating "that two toolmarks originated from the same source to the exclusion of all other sources."

The stated reason to avoid these terms is that a source attribution "is not based on a statistically-derived or verified measurement or comparison to all other firearms or toolmarks." But who would think that an examiner who "assert[s] that two '[t]oolmarks originated from the same source to the exclusion of all other sources'" is announcing "an actual comparison to all other toolmarks in the world"? The examiner apparently is allowed to report a plethora of matching "individual characteristics" and to opine (or "inductively infer") that there is virtually no chance that the marks came from a different source. Allowing such testimony cuts the heart out of the rules against asserting "uniqueness" and claiming "individualization."

NOTES

1. E.g., United States v. Hunt, 464 F.Supp.3d 1252 (W.D. Okla. 2020) (discussed on this blog Aug. 10, 2020).
2. The original version was adopted on 7/24/2018. It was revised on 6/8/2020.
3. Are numerical versions of subjective likelihood ratios prohibited by the injunction in Part IV that "[a]n examiner shall not provide a conclusion that includes a statistic or numerical degree of probability except when based on relevant and appropriate data"? Technically, a likelihood ratio is not a "degree of probability" or (arguably) a statistic, but it seems doubtful that the drafters of the ULTR chose their terminology with the niceties of statistical terminology in mind.
4. A.W.F. Edwards, Likelihood 31 (rev. ed. 1992) (citing H. Jeffreys, Further Significance Tests, 32 Proc. Cambridge Phil. Soc'y 416 (1936)).
5. The correct name of the dictionary is A Dictionary of Forensic Science, and its author is Suzanne Bell. The quotation in the ULTR omits the following part of the definition of "inductive inference": "A forensic example is fingerprints. Every person's fingerprints are unique, but this is an inference based on existing knowledge since the only way to prove it would be to take and study the fingerprints of every human being ever born."

More on Rule 704 and Source Attribution in Forensic Science

On July 17, I observed that

At a webinar on probabilistic genotyping organized by the FBI, the Department of Justice’s Senior Advisor on Forensic Science, Ted Hunt, summarized the rules of evidence that are most pertinent to scientific and expert testimony. In the course of a masterful survey, he suggested that Federal Rule of Evidence 704 somehow conflicts with the evidence-centric approach to evaluating laboratory results recommended by a subcommittee of the National Commission on Forensic Science, by the American Statistical Association, and by European forensic-science service providers.

I proceeded to analyze Rule 704 and its application to the conclusion-centric testimony used in US courts. In a comment posted on July 21 (and reproduced below), Mr. Hunt generously took the time to respond. Taken together, his webinar presentation, my posting on it, and his comment on this blog discuss three issues:

(1) What does Rule 704 reveal about the proper or permissible role of an expert witness?
(2) Are certain publications on the desirability of the evidence-centric approach deficient or misleading for failing to mention Rule 704?
(3) Does Rule 704 pertain to every source-attribution opinion?

As explained below, the short answers are "not much," "no," and "no," respectively. In addition, an essay on the first and third issues will appear in the Winter 2020 issue of Jurimetrics Journal.

I

Mr. Hunt commented that “I don’t disagree that Rule 704 ‘does not counsel in favor’ of admitting expert opinion on an ultimate issue. I never said or implied anything or the sort.” At the same time, he wrote that “the Federal Rules (and most state rules of evidence) affirmatively allow an expert to offer an opinion about a proposition.”

Listening to the talk, I thought I was hearing the germ of a legal argument that Rule 704 not only allows, but also affirmatively supports the admission of source-attribution opinions such as “I am sure that the defendant is the source of the latent print.” The recording of the webinar is on the Forensic Technology Center of Excellence website, I transcribed the two small parts that pertain to Rule 704. I think I have captured all the words on Rule 704 and source opinions; they are in Appendix A.

The talk begins with a reference to “lots of statements” on the role of the expert witness “that have been pushed out in the literature” and therefore merit “a deeper dive and look at the American judicial system and the role of the expert in it.” After reviewing the most important rules affecting expert and scientific testimony generally, Mr. Hunt quoted several statements from organizations to the effect that experts ought to explain the degree to which the scientific data support the relevant hypotheses (“propositions” about the source of the data) rather than opine on the truth of the hypotheses themselves. Various forensic scientists, psychologists, law professors, and others have been advocating this approach for decades.

Mr. Hunt cautioned the audience as follows:

However, in the US system there is Rule 704, which is a rule that you don't hear much about, and I think there are some people who would like to pretend that it doesn't exist, but it actually goes against that school of thought and allows an expert to opine on an ultimate contested proposition before the court, not only on the strength of the evidence for or against the proposition. Specifically an opinion is not objectionable just because it embraces an ultimate issue, and you can add proposition in there as well.

He concluded his exposition of Rule 704 with the observation "So in the DNA context this would allow a source-attribution statement" and then described a few cases that accepted DNA source conclusions.

To my ear, the suggestion that people are pretending that Rule 704 does not exist because "it actually goes against that school of thought" sounded like an argument that Rule 704 is an affirmative reason for a court to allow traditional source-attribution testimony. My criticism of this argument (whether or not it was Mr. Hunt's) had two components. First, sometimes source opinions are not opinions on "ultimate" issues or facts—things like whether the killer was insane or whether the deceased committed suicide. In those cases, the rule does not even come into play.

Second, to say that Rule 704(a) "allows" source attributions is merely to say that a previous and ill-defined common-law rule on opinion testimony is no longer a reason to exclude the testimony. Rule 704(a) just cancels out a categorical rule of exclusion, leaving the court with a big zero. The rule does not militate either for or against the idea that forensic scientists should avoid proffering source opinions. It merely means that the objecting party had better have some other reason for the court to exclude the opinion.

This much is true whether the issue or fact is called ultimate or intermediate. Parties opposing source-attribution opinions could argue for exclusion under the helpfulness requirement of Rule 702 or the balancing mandated by Rule 403. The documents cited by Mr. Hunt do not invoke these rules, just as they do not discuss Rule 704, because they are not making a legal argument.

In emphasizing that Rule 704 is merely a rule of non-exclusion, I hasten to add that I am not suggesting that Mr. Hunt misrepresented the rule as mandating admission of ultimate-opinion testimony. He did nothing of the kind. The undisputed legal principle is that Rule 704 is agnostic toward ultimate-opinions. As Mr. Hunt agreed, it does not counsel in favor of their admission. Once that much is understood, readers can decide for themselves whether the rule “actually goes against that school of thought” that asks criminalists to avoid expressing opinions on the truth of hypotheses.

II

To emphasize that he was not suggesting "that Federal Rule of Evidence 704 somehow conflicts with the evidence-centric approach to evaluating laboratory results," as I originally put it, Mr. Hunt explained that he only was trying to correct a possible misunderstanding of what he called the "role assignment" documents:

My comments about FRE 704 were directed to the claim of ‘role assignment’ by the authors of the cited sources. The documents I cited (ENFSI, Evett, NIFS, AFSP, NCFS, ASA) make statements such as ‘the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves.’ (NIFS).

I simply noted that this purported assignment of ‘roles’ is more limited than that which is permitted by the Federal Rules of Evidence. None of these documents concede (or even acknowledge) that the Federal Rules (and most state rules of evidence) affirmatively allow an expert to offer an opinion about a proposition, rather than limiting that opinion to the relative strength of the evidence in support of competing propositions.

Omitting this information may mislead some to believe that the ‘role’ assigned by these documents to an expert is instead assigned by legal rules, rather than a ‘best practices’ world view.

***

In conclusion, to the extent there is any ‘tension’ or ‘inconsistency’ between Rule 704 and the noted references, it was created by the referenced documents’ omission of a discipline-specific basis for their claim that ‘the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves.’ Each of the cited sources fail to make it clear that this limited ‘role’ is self-imposed, not legal in nature. That was the point of my observation.

Informing people that the law allows well founded source-attribution opinions is entirely reasonable. Only a few courts depart from that generalization. But the position papers and guides to evaluative reporting with which I am familiar surely are not deficient, deceptive, or misleading about this.

The document from which the quoted words, "role of the expert," come is the Introductory Guide to Evaluative Reporting prepared by a special committee created by the National Institute of Forensic Science. NIFS is a directorate within the Australian and New Zealand Policing Advisory Agency (ANZPAA). The "role assignment" language occurs within a box contrasting "traditional reporting" and "evaluation using a logical framework." I don't care for the phrase "logical framework" (to the extent it intimates that nothing else is logical), but the words "role of the expert" are appropriately used only to refer to the role of the expert within the stated alternative to the traditional approach. Let’s put the words back into the full sentence in which they occur (with the language that Mr. Hunt questioned in bold):

The fundamental principles of evaluative reporting or interpretation are that (i) the crime must be considered to have occurred within a framework of circumstances, (ii) that the findings must be considered in light of at least two competing propositions that will be guided by the case circumstances and (iii) that the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves. (P. 6).

The first page of the NIFS Guide explains that the purpose is to introduce readers to an alternative to traditional and continuing practice and to “provide[] managers with advice on implementing evaluative reporting where they consider it appropriate.” (P. 1, emphasis added). A "discipline-specific basis for their claim" that reporting likelihoods has cross-disciplinary advantages over elimination-versus-identification opinions would make little sense in a document designed to enable "many traditional comparative forensic disciplines" to produce reports and testimony in "ways that can be standardised between analysts within, and between, organisations and forensic disciplines." (P. 6).

Little would be gained by reviewing the details of the other documents Mr. Hunt feels should have "conceded" or "acknowledged" the existence of Rule 704. I cannot see how many readers of these documents could construe them as describing the current rules of U.S. evidence law (or that of other countries), 1/ but it is always good to to distinguish between legal and statistical or scientific principles.

Adhering to this distinction, there is a legal argument for the role that the documents recommend for the expert. Proponents of this “likelihood-ratio” or “Bayes-factor” mode of presentation believe that it nicely captures the role of a scientist supplying the legal factfinder with information based strictly on the physical or chemical features of the specimens that are compared. DNA experts do not normally testify that the defendant's DNA is present in the crime-scene DNA. Instead, they testify to probability of the measured quantities of identifying DNA fragments (or to the probability of the alleles inferred from these measurements) if the defendant's DNA is not present in the sample. They also may testify to the probability if it is, or to the ratio of these probabilities.

To reach the further conclusion that the defendant's DNA actually is in the questioned sample would require information outside of the fragment lengths measured in the laboratory. In Bayesian terms, any conclusion about the true source "necessarily rests, in part, on an implicit assessment or assumption about the prior odds that the suspect is the source," and “[m]aking those assessments or assumptions takes forensic scientists beyond their scientific expertise in ways that arguably usurp the role of legal fact-finders." 2/ The core of the argument is that experts exceed their legally assigned role of assessing the physical evidence when they venture beyond the scientific data to which the likelihood ratio pertains. "Experts are rarely in a good position to evaluate the prior odds that the items they are comparing have a common source, and arguably have no business doing so." 3/

That criminalists dealing with other types of evidence were implicitly opining on prior probabilities beyond their expertise was not obvious in the days when it was thought that every specimen—I'll use fingerprints as an example—is unique. Uniqueness of prints makes it sound as if the criminalist does not have to think about the population from which a latent print came in order to conclude that the print had to be the defendant's. But uniqueness of friction ridge skin is not a sufficient basis for a source attribution. It overlooks the fact that latent print examiners do not deal with three-dimensional skin. They compare impressions of a complete and clear set of prints to incomplete and often low quality impressions. The two-dimensional representations vary from one impression to the next. The only question the analyst can answer strictly on the basis of the features (and knowledge of their joint distribution in a relevant population) is how the probability of a latent print being as similar as it is to the exemplar print—when it is, in fact, from the finger that produced the exemplar print—compares to the probability of the latent print being so similar when it is from someone else's finger.

This "role-assignment" argument is not conclusive. What if experts legitimately can say that in every relevant population of plausible size, the probability is very high that only one finger could have produced the observed features? Or that the likelihood ratio is so huge that it swamps the prior probability for an individual randomly drawn from that population? Then the expert who indulges in source attribution (without specifying a prior probability for the judge or jury to consider) arguably would not exceed the role of the expert defined in Federal Rule of Evidence 702.

III

Moving back to the content of Rule 704, it seems that we have a sharp disagreement about the reach of the "ultimate issue" rule that Rule 704 (building on earlier case law and academic criticism) decisively rejected. The rule applies to any opinion from any witness—expert or lay—that "embraces an ultimate issue." But what does that phrase mean? I maintained that Rule 704 does not apply to all source attributions because "the identity of the source of a trace is not necessarily an ultimate issue." To make this point, I wrote as follows:

To use the example of latent-print identification given in the webinar, the traditional opinion that a named individual is the source of a print is not an opinion on an ultimate issue. Courts have long allowed examiners to testify that the print lifted from a gun comes from a specific finger. But this conclusion is not an opinion on whether the murder defendant is the one who pulled the trigger. The examiner’s source attribution bears on the ultimate issue of causing the death of a human being, but the examiner who reports that the prints were defendant's is not opining that the defendant not only touched the gun (or had prints planted on it) but also pulled the trigger. Indeed, the latent print examiner would have no scientific basis for such an opinion on an element of the crime of murder.

Mr. Hunt's comment insists that every source attribution is an ultimate-issue opinion to which Rule 704 applies. Here is his argument:

[Y]our blog states that ‘the identity of the source of a trace is not necessarily an ultimate issue.’ As evidence for that assertion, you used my latent print example, claiming that ‘the traditional opinion that a named individual is the source of a print is not an opinion on an ultimate issue.’

Your claim is directly contrary to the contents of the judicial opinion I cited in the questioned slide, which states in full: ‘[Defendant] . . . contends that the district court erred in allowing the expert to specifically testify that he had no doubt that the prints on the parking stub were [Defendant’s] because the expert was testifying to the "ultimate issue’ of whether it was Sherwood's print. This argument fails in light of Fed. R. Evid. 704, which permits expert testimony on the ultimate issue in a case.” United States v. Sherwood, 98 F.3d 402, 408 (9th Cir. 1996).

In Sherwood, the ultimate issue was not described as a connection between the latent print and activity-level behavior; rather, it was the expert’s testimony about the source of that print. Your description of an ultimate issue as limited to one in which (using your example), ‘the murder defendant is the one who pulled the trigger’ is a narrower description of the types of factual assertions entailed by Rule 704 than those described throughout case law. As such, your explanation of the Rule is not entirely correct.

For other examples of courts describing identification testimony as ‘ultimate’ in nature, see U.S. v. Rose, 731 F.2d 1337, 1346-47 (print match to shoe) United States v. Bice-Bey, 701 F.2d 1086 (4th Cir. 1883) (voice identification); United States v. Sellers, 566 F.2d 884 (4th Cir. 1977) (identification of suspect in a picture); United States v. Couser, 15 Fed. R. Evid. Serv. (CBC) 837 (4th Cir. Apr. 5, 1984) (comparison of suspect’s features to photograph). These cases clearly demonstrate that source identification testimony constitutes an opinion that embraces an ‘ultimate’ fact.

It is true that the Ninth Circuit in Sherwood referred to the source attribution as an "ultimate" opinion. I assume every other case he lists did the same. However, this use of authority is an instance of looking for law in all the wrong places. Every time a party objects, "Your honor, my identity as the source of the trace is an ultimate issue!", the court can summarily reject the objection by citing Rule 704's text. Why bother to try to trace the boundaries of the "the so-called 'ultimate issue' rule" (Fed. R. Evid. 704 Advisory Committee's Note) when the murky rule "is specifically abolished" (id.)? Regardless of the reach of Rule 704, the outcome is the same. If the expert opinion is outside the old rule, then it is "not objectionable" just because it embraces an ultimate issue. If it is covered by the old rule, it is "not objectionable" because the old rule is no longer a barrier.

So the fact that modern courts often assume—without analysis—that a source opinion is ultimate-opinion testimony tells us next to nothing about how the opinion would have fared under the defunct common-law exclusionary rule. To answer that arcane question, we must look to the old cases applying that rule to exclude opinion testimony.

Where should we begin? McCormick on Evidence § 12, at 80 (7th ed. 2013), documents the "general doctrine that witnesses may never give their opinions or conclusions on an ultimate fact in issue" with two cases: United States v. Spaulding, 293 U.S. 498 (1935), and State v. Carr, 196 N.C. 129, 144 S.E. 698 (1928). In Spaulding the "ultimate issue" was whether the plaintiff was totally disabled while an insurance policy was in force. He produced "medical opinions" that he "became totally and permanently disabled before his policy lapsed." The Supreme Court wrote "that question is not to be resolved by opinion evidence. It was the ultimate issue ... . The experts ought not to have been asked or allowed to state their conclusions on the whole case." 293 U.S. at 506.

Carr is a homicide case. The defense was that a man who died in bed of a gunshot wound had committed suicide. The state produced a medical doctor who testified that "I don't think it is possible for the deceased to have fired the gun and made the wound that I saw." 144 S.E. at 700. The North Carolina Supreme Court held that the trial court erred in admitting the opinion. It explained that

[T]he opinion or inference of the witness must not be an answer to the exact issue which the jury is to determine. When the witness testified that he did not think it possible for the deceased to have fired the gun and to have made the wound he necessarily testified in effect that in his opinion the deceased did not kill himself. True, the "exact issue" was whether the defendants are guilty, but if the deceased killed himself the conclusion that the defendants did not kill him would necessarily follow.

Id. These cases reveal that the essence of the ultimate-issue rule is that the witness may not opine on a fact that determines whether an element of the offense or complaint, or of the defense, is satisfied.

Other facts are intermediate. They are links in a chain of reasoning that terminates in the answer to one of these ultimate issues. In an action for a declaration of paternity, for example, an expert opinion that (based on DNA tests) the defendant is the father, falls under the former ultimate-issue rule. But suppose the government introduces an expert opinion that a murder defendant is the father of a fetus whose remains were found along with the mother's merely to show that he had a motive to kill the mother, who refused to have an abortion as he had demanded. Now the source conclusion is not an opinion on an ultimate issue. The jury can accept the source opinion but conclude that defendant, despite the motive, was not the killer. In contrast to Spaulding, the expert opinion does not decide "the whole case." Unlike Carr, the ultimate conclusion does not "necessarily follow."

On which side of the ultimate-intermediate line the examiner's opinion in Sherwood falls I cannot say. The opinion is too sparse. But the question is academic. One purpose of Rule 704 is to spare the courts from having to pin these labels on opinion testimony. (See Advisory Committee's Note, observing that the old rule was "difficult of application"). With the rule in place, they can just say "so what—the argument fails in light of Rule 704." But that does not mean that a source opinion is always an opinion on "an ultimate issue." That depends on how the opinion fits into the case.When the source opinion is near enough to the "ultimate" endpoint, Rule 704 does not say, "OK, let it in." In conjunction with Rule 702, it says "Well, yes, but there is no mechanical rule excluding it on that basis alone—just address how helpful it is for the jury to hear it."

When courts address the question of helpfulness, they just might conclude that instead of old-style source attributions, statements about the probabilities of the findings under different hypotheses would fulfill the role of helping the factfinder find the facts better that traditional expressions of an examiner's belief that the defendant is the source of the trace. Or they might not. Either way, no one is trying to deny that Rule 704 exists. Why would they when it sheds so little light on the choice between traditional conclusion-centric opinons (source attributions that, to a Bayesian, reflect an expert's personal prior probabilities and utilities) and evidence-centric ones (such as Bayes factors that enable experts to express the probative value of the findings without venturing into that nonscientific territory)?

NOTES

1. Australia's uniform Evidence Acts § 80(a), adopted by many of its jurisdictions, abolished the common-law rule against ultimate opinions ("[e]vidence of an opinion is not inadmissible only because it is about—a fact in issue or an ultimate issue."). New Zealand did the same in its 2006 Evidence Act ("(2) An opinion by an expert is not inadmissible simply because it is about—(a) an ultimate issue to be determined in a proceeding; or (b) a matter of common knowledge."). 
2. William C. Thompson, How Should Forensic Scientists Present Source Conclusions?, 48 Seton Hall L. Rev. 773, 791 (2018).
3. Id.; see also David H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence § 2.2.2, at 44 (2d ed. 2011) (notes omitted):

   But this form of opinion leaves the jury with an inappropriate choice. If the jury is to accept the opinion on a matter within the witness’s expertise, it must also accept the expert’s (presumably unarticulated) assessment of evidence that has nothing to do with the expertise and that falls within the jury’s ability to asses. Or, it must accept an unvalidated and implausible theory of individualization.

APPENDIX A

Ted R. Hunt, The Expert’s Role in the American Judicial System (Probabilistic Genotyping of Evidentiary DNA Typing Results – An Online Workshop Series, Module 8: Probabilistic Genotyping Summation and Special Topics, July 17th, 2019, available at https://forensiccoe.org/ (excerpt of all remarks on Rule 704 and source conclusion testimony)

---

*** My talk is entitled "the role of the expert" because there aren't special rules that just apply to DNA analysts and no one else, and the reason that I want to talk about it today is, in the forensic literature in the last number of years, lots of statements have been made about the role of the courts and the role of the experts, and those statements aren't specific to any judicial system. They're pretty much general statements that have been pushed out in the literature so I thought we'd take a deeper dive and look at the American judicial system and the role of the expert in it.

***

Now I mentioned before at the beginning of this talk there's a lot of literature on the role of the courts and the role of the expert in the forensic literature. I have some examples here. The European Network has its guidelines for evaluative reporting in forensic science. Number 3 here in this excerpt talks about the role of the forensic practitioner to consider the probability of the findings considering the propositions and not the probability of those propositions. Similarly, other documents in the community from Australia and New Zealand, from Dr. Evett, from the Association of Forensic Science Service Providers. All basically say the same thing, talking about the role of the expert to comment on the probability of the findings given those propositions and not the prepositions themselves.

This has gotten into the US forensic community to a certain extent. On the left is the National Commission [on Forensic Science] Views Document. I just want to be clear this document did not get voted to approval through the Commission, but nevertheless there is a final stand-alone version to that document that basically says the same thing: Forensic practitioners should not state that a specific individual or object is the source of forensic evidence and refers to the formal method of evaluative reporting as well. And basically the same people who wrote that document wrote the American Statistical Association document which, for the most part, says the same thing, discouraging the statements to the effect that a specific person or object is the source of evidence.

However, in the US system there is Rule 704, which is a rule that you don't hear much about, and I think there are some people who would like to pretend that it doesn't exist, but it actually goes against that school of thought and allows an expert to opine on an ultimate contested proposition before the court, not only on the strength of the evidence for or against the proposition. Specifically an opinion is not objectionable just because it embraces an ultimate issue, and you can add proposition in there as well.

I need to add that 704 does not lower the bar to admit otherwise inadmissible evidence. It will work in conjunction with the other rules of evidence. The only exception in the American system or the US federal system is that the expert may not opine on the defendant's mental state —an ultimate legal proposition before the court—or witness credibility. However, they can opine on an ultimate issue.

Just as an example, this is a latent fingerprint case from the 9th circuit in 1996 [United States v. Sherwood, 98 F.3d 402 (9th Cir. 1996)] and a defendant objected that an expert was specifically testifying that he had no doubt that the prints on the piece of evidence were the defendant’s. The defendant objected because the expert was testifying to an ultimate issue, and the court basically said, so what—the argument fails in light of 704, which permits an expert to opine on ultimate issues.

Like I said before, I’m trying to be very descriptive. I’m not being normative, and I’m not weighing in on the logic or the advisability of this. I'm simply trying to correctly state what I understand the law to be.

So in the DNA context this would allow a source attribution statement. ***

APPENDIX B

Comment posted to this blog, July 21, 2019

---

Professor Kaye,
     My comments about FRE 704 were directed to the claim of ‘role assignment’ by the authors of the cited sources. The documents I cited (ENFSI, Evett, NIFS, AFSP, NCFS, ASA) make statements such as ‘the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves.’ (NIFS).
     I simply noted that this purported assignment of ‘roles’ is more limited than that which is permitted by the Federal Rules of Evidence. None of these documents concede (or even acknowledge) that the Federal Rules (and most state rules of evidence) affirmatively allow an expert to offer an opinion about a proposition, rather than limiting that opinion to the relative strength of the evidence in support of competing propositions.
     Omitting this information may mislead some to believe that the ‘role’ assigned by these documents to an expert is instead assigned by legal rules, rather than a ‘best practices’ world view. I never said or implied that Rule 704 ‘stands in the way’ of an expert’s provision of a more limited opinion about the strength of the evidence relative to competing propositions.
     In addition, your blog states that ‘the identity of the source of a trace is not necessarily an ultimate issue.’ As evidence for that assertion, you used my latent print example, claiming that ‘the traditional opinion that a named individual is the source of a print is not an opinion on an ultimate issue.’
     Your claim is directly contrary to the contents of the judicial opinion I cited in the questioned slide, which states in full: ‘[Defendant] . . . contends that the district court erred in allowing the expert to specifically testify that he had no doubt that the prints on the parking stub were [Defendant’s] because the expert was testifying to the "ultimate issue’ of whether it was Sherwood's print. This argument fails in light of Fed. R. Evid. 704, which permits expert testimony on the ultimate issue in a case.” United States v. Sherwood, 98 F.3d 402, 408 (9th Cir. 1996).
      In Sherwood, the ultimate issue was not described as a connection between the latent print and activity-level behavior; rather, it was the expert’s testimony about the source of that print. Your description of an ultimate issue as limited to one in which (using your example), ‘the murder defendant is the one who pulled the trigger’ is a narrower description of the types of factual assertions entailed by Rule 704 than those described throughout case law. As such, your explanation of the Rule is not entirely correct.
     For other examples of courts describing identification testimony as ‘ultimate’ in nature, see U.S. v. Rose, 731 F.2d 1337, 1346-47 (print match to shoe) United States v. Bice-Bey, 701 F.2d 1086 (4th Cir. 1883) (voice identification); United States v. Sellers, 566 F.2d 884 (4th Cir. 1977) (identification of suspect in a picture); United States v. Couser, 15 Fed. R. Evid. Serv. (CBC) 837 (4th Cir. Apr. 5, 1984) (comparison of suspect’s features to photograph). These cases clearly demonstrate that source identification testimony constitutes an opinion that embraces an ‘ultimate’ fact.
     I don’t disagree that Rule 704 ‘does not counsel in favor’ of admitting expert opinion on an ultimate issue. I never said or implied anything or the sort.
    In conclusion, to the extent there is any ‘tension’ or ‘inconsistency’ between Rule 704 and the noted references, it was created by the referenced documents’ omission of a discipline-specific basis for their claim that ‘the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves.’ Each of the cited sources fail to make it clear that this limited ‘role’ is self-imposed, not legal in nature. That was the point of my observation.
     Thanks for your participation in the webinar series and for your interest in my talk.

Ted R. Hunt
Senior Advisor on Forensic Science
U.S. Department of Justice

Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 2)

As noted earlier this month, one of the more striking parts of the ProPublica articles on the FBI laboratory's image image analysis unit 1/ is its discovery of testimony, given more than sixteen-years ago, about the probability of finding certain features in photos of a shirt worn by a bank robber caught on film in a string of robberies. The government accused Wilbert McKreith of eight robberies and of and possession of firearms having been previously convicted of a felony. 2/

According to ProPublica, a ruling that Dr. Richard Vorder Bruegge’s “testimony met the Daubert standard” “enshrined the FBI unit’s techniques and testimony as reliable scientific evidence.” This characterization seems to overstate the importance of a single, unpublished pretrial ruling that followed a short, spur-of-the-moment hearing. 2/ Still, the testimony was central to McKreith’s conviction on the bank robbery counts. Dr. Vorder Bruegge’s conclusions fortified the testimony from ordinary witnesses that, among other things, McKreith and the bank robber wore two wrist watches at the same time; wore or had ski masks in south Florida; wore plaid shirts; drove a maroon, burgundy or red-colored car; and had the same general appearance.

Dr. Vorder Bruegge’s signal contribution came from comparing video images from bank cameras to items seized from the defendant’s home and to McKreith himself. Some of this testimony was not definitive. For example, Vorder Bruegge testified that enlargements of the photographs from several bank robberies lacked sufficient resolution for him to say “with a scientific certainty” what that they showed wristwatches. Instead, he stated that they were “consistent with these features being two watches on the left wrist of the bank robber.” Likewise, he testified that a bank photograph that showed a profile of the robber’s face lacked the resolution to reveal allegedly “individual identifying characteristics [such as moles, stars, chipped teeth, ear patterns or other facial minutiae],” but “the overall characteristics of the profile, which include the shape of the nose, mouse, and chin” displayed “similarities” “consistent with” McKreith’s being the bank robber.

Dr. Vorde Bruegge’s analysis of the pattern of a plaid Van Heusen shirt in photos from seven of the robberies produced more conclusive evidence. A condensed version of the testimony on direct examination on December 16, 2002, is in the shaded boxes that follow. Discussion is interspersed between the boxes. ProPublica posted the full transcript.of the testimony from Dec. 16 and 17.

"Individual Identifying Characteristics"

MR. [Roger] STEFIN [Assistant US Attorney]: Our next witness would be Richard Vorder Bruegge.
THE COURT: Okay, members of the jury, [o]ur next witness will be an expert witness. ....
[Defense counsel objected that the government first needed “to lay a sufficient predicate” and “request[ed] a Daubert ... hearing. The court held an impromptu hearing, with no written briefing, at which Dr. Vorder Bruegge affirmed that “the techniques [are] well recognized ... in forensics [a]nd in the scientific community at large.” The court overruled the defendant’s objection.]
Q. [by MR. STEPHIN] [H]ave you had the opportunity to view the eight by ten photographs that have been introduced in evidence in this particular case with respect to the bank robbery images in each of the eight bank robberies?
A. Yes, I have. ...
Q. ... Let’s talk about the shirt now. What is it about the manufacturing process of shirts that might enable you to identify features of that shirt that could be then compared with image analyses?
A. Well, any, any comparison analysis involves a comparison of first of all, the class characteristics. ... In a shirt like this, the class characteristics ... include such things as, does it have a pattern, which this shirt does — it has a plaid pattern or a checked pattern, if you will. ... It also is a button-down shirt — it's not a pull-over, and it's a long-sleeved shirt. ...
Once one has ... found ... class characteristics that match, one can move on to look at the individual identifying characteristics. ... [I]f I were trying to differentiate one person from another and reach a positive identification, then I would need individual identifying characteristics, such as moles, scars, freckle patterns, chipped teeth. These, these features enable you to really differentiate people down to saying this person is unique from all other people. [B]ecause this shirt is a patterned shirt, there are individual identifying characteristics ... based upon ... the way ... the pieces on the shirt are cut out and then sewn together.

The class-individual distinction is entrenched in forensic science, and there is a grain of truth in it, But it assumes what is to be proved -- that every "individual characteristic" (or some unspecified combination of them) makes an individual distinguishable from every other individual. "Class characteristics" are known to be generic, whereas "individual" ones might not be. Logically, both work the same way -- the presence or absence of a characteristic changes the probability of a common source for the specimens or images being compared. A less tendentious pair of terms would be "generic" and "randomly acquired or incorporated," keeping in mind that random characteristics are not necessarily specific to an individual.

"One in-35 Through Random Processes"

Q. So what can you conclude with respect to the pattern itself on this shirt?
A. [B]ecause we don't see the patterns matching across the seams, we can use that as a way to individualize this shirt relative to the other shirts that would be manufactured at the same time. ...
Q. [W]hat do you call this when you find an individual feature?
A. Individual identify characteristic. Basically, each seam ... can be considered on its own as an entire set of individual identifying characteristics. It's almost one individual identifying characteristic. ... [I]f we look at where the right yoke meets the right front panel, there's this ... thick dark that I've been talking about this comes down from the neck, and it terminates here about 2/3 of the way across this panel ... . Likewise, if we look at the left sleeve, I point out that ... very thin dark line here, which is coming up the right sleeve. It almost exactly meets the point where the yoke joins the front panel. ... Likewise, ... look at the collar itself. You'll see that the collar has a very dark line [that] just cuts the corner of the bottom hole on this side. That is paralleled on the other side because it's one piece. So that's another individuating characteristic for this shirt. ...
Q. All right. And did you take precise measurements of these types of lining up of the different — the thick lines or the thin lines in order to further identify, you know, the individual characteristics of this particular shirt?
A. I measured the width of these features on this shirt so that I could figure out [the] relationship ... between places meeting on one side with those meeting on the other side.
Q. And I think you said that the pattern repeats basically every three-and-one-half inches?
A. Yes. Basically, ... if we just repeat going from this dark line to the next dark line, it's three-and-a-half inches. ...
Q. The thick line or the thin line?
A. The thin line. Each ... of these features repeats every three-and-a-half inches. It's just that the thin line is the smallest feature that we can see very easily.
Q. So you were using the thin line as a point of reference in measuring out the three-and-one-half inches? A. Exactly. Recall that I said there's a curved surfaces here where the sleeve meets the other seams. That would complicate any type of repeat analysis because since it's not a straight line, that's going perpendicular to that feature, so it's actually going to be longer.
So basically, if we did have a single seam that we were trying to match this pattern to run across it, the chances of that very thin dark line matching up, matching up or at least touching the black line on the other side, would be one in 35 through random processes.
Q. Could you explain that, explain that a little bit, how you came up to the one in 35? ...
A. ... Let's use, let’s use the yoke and the sleeve. ...
Q. OK ... And the yoke, again, is this back panel ... ?
A. Right. I'm using the yoke in the back because they're almost aligned. ... You can see here that on the yoke the dark line is ... slightly below the seam here. And you have to go about half an inch down before you hit the black line on the other side.
Q. Okay. In other words, you can measure the distance between the black line on the yoke to the black line on the sleeve, and you measured, say ... a half an inch?
A. Yeah.
Q. All right. And so just taking that one point of reference, what would be the odds that two shirts coming out the same manufacturing plant being manufactured in this, in this method would end up having the same alignment of yoke and the sleeve, whereby you could have this ... half an inch offset between them just for this one point of identification?
A. Just for that one point, it would be a 1 in 35 chance.
Q. And how did you come up with the number that it's a 1 in 35 of probability that these two items would line up the same in more than one shirt?
A. Because the feature itself that I’m looking to align is 1/35th of the overall repeat length. And to see ... that one feature come up randomly happens to be one in 35 times.
Q. So one in every 35 shirts manufactured by the company using this patterned cloth, the exact same cloth, you could say that one in 35 probability-wise would come up with the exact same alignment between the yoke and the left sleeve?
A. That is correct.

Several things are going on here to give rise to a uniform probability distribution. First, the plaid design of the shirt comes from a repeated 3.5"-wide distinctive pattern that contains lines of at least two different thicknesses. Second, these lines can be offset from one another across the seams of the shirt. Third, Dr. Vorder Bruegge measures how large the offset is in 1/10th" strips on the 8x10" photographs. Finally, the exact offset -- and hence, which strip a corresponding line on the other side of a seam falls, is determined at random.Thus, the number (call it X) of 1/10th" strips that separate the starting point of every plaid block on the two different cuts of fabric that are sewn together along a seam can take on the value x = 0, 1, 2, ..., 34, with probability f(x) = 1/35 for every x. The following sketch of a repeating block showing only one line in the pattern may clarify what x stands for:

1/10" strip ---------| |
1/10" strip          | |
1/10" strip          | |---------} x=2 (2/10" offset)
1/10" strip          | |
...
1/10" strip ---------| |
1/10" strip          | |
1/10" strip          | |---------
1/10" strip          | |
...                   s
1/10" strip ---------|e|
1/10" strip          |a|
1/10" strip          |m|--------

The idea of assigning an equal probability to elementary events dates back to Bernouilli (1713) and Laplace (1814). The underlying principle of insufficient reason, indifference, or symmetry holds that if there is no reason to believe that any event is more likely to occur than any other event in a set of possible, mutually exhaustive events, then one should assume that all the events are equally probable. This principle offers one way to motivate or understand the axioms of probability theory. Although it has fallen out of favor, it is not without defenders. 4/

But Vorder Bruegge did not base his probability model on a metaphysical or philosophical principle. He gave an empirical justification. During the short Daubert inquiry, he testified that he visited "manufacturing plants, factories where articles of clothing [are] made" and that "in this particular instance, I visited ... cutting plants in Alabama where the patterned material is cut out, as well as manufacturing plants where the cut out pieces are sewn together so that I can see for myself how the process takes place." Furthermore, "I've also been to another plant, Guess plant in Southern California, where shirts were also manufactured, and found that they use the same manufacturing process at the Guess Factory as they do at the Arrow shirt factories in Alabama and ... Georgia." He learned that "[m]anufacturers do not make an effort, in general, to make these features align, because to do so would be prohibitively expensive" and that "there are also places where it is not possible ... to make them align because of the curvature such as along the arms and the sleeves."

In essence, he proposed that busy workers stitched the pre-cut pieces of fabric together without regard to how well their patterns aligned with one another. He elaborated:

Q. All right. Could you ... explain the manufacturing process ... as to how shirts of this nature would be made in a factory?
A. [Y]ou start off with a huge bolt of cloth that can be hundreds and hundreds of yards long. The cloth is then laid down at a cutting plant on a table that [is] maybe about a hundred yards long. Now, these huge bolts will be rolled out on the table and then rolled back onto itself until the entire roll is done. The another role is attached, ... and it continues until you have on the order of 500 plies of material. Now, because of the way that the material is rolled back in, this pattern is not going to line up from one ply to the next. ...
Once the fabric is laid out and you've got 500 plies on top, the manufacturers lay out, basically, tracing paper that has cutting patterns. Just like if you were sewing at home and making your own clothes, you would have a pattern to cut out. Only this is a hundred yards long [and] has been designed by engineers whose only job is to figure out how best to place each piece of this shirt in its closest proximity so they are wasting as little of that fabric as possible. ...
[T]hey slap it down and they will actually have people get on there with jigsaws and cut them out by hand. Or in some of the plants they will have computers that can manage the cutting out. Once all of those pieces are cut out, they [are] transferred over to the sewers ... [Y]ou've got people who do nothing all day but sew on sleeves onto shoulders or yolks on to the back.
Q. ... So there will be, like, thousands of pieces for the collar and thousands ... for the sleeves and so forth?
A. ... If you have 500 plies thick, you have 500 pieces in one particular area. ... [O]n this shirt we've got two pieces on the collar. If you look closely at this shirt, you'll see there's actually a piece here it appears on the outside and a piece on the inside. There's another piece that goes around the collar; so that's three pieces. ... [All together, there are] 16 pieces of this same pattern cloth on this shirt [that must be sewn together].
Q. What significance does that have in terms of determining whether or not, you know, identifying the uniqueness of a particular shirt as opposed to any of the other shirts that are manufactured by the plant using the same cloth coloring?
A. Well, as I mentioned before, every one of these pieces is cut out. The plies line up on every row. If, in the manufacturing process, they happened to take a piece from the top layer and try to stitch it to a piece from the next layer down, you're going to get a lot of randomness in this process. Because there is this offset, you're not going to get this pattern lining up across the seam in any consistent way from one shirt to the next. ... [I]n fact, you can't get a curved seam like this to have an alignment across it, because it isn't geometrically possible.
You look at the pockets on this shirt and you'll see that they line up. [T]he fact that one of these pockets lines up makes this shirt twice as expensive to manufacture as it would be if this pocket were on the bias. [T]hey have to make sure that this pocket is cut out and exactly the same orientation as the front panel. Furthermore, they have to actually have a human being take the time to physically line this pocket up when they're sewing ... .
Q. Is there any effort to line up any of the other pieces of cloth? In other words, lining up the lines from the back to the sleeves, or the yolk to the sleeves, or the yolk to the front panels —
A. No. No. And you can see that for yourself just by comparing the way the left sleeve doesn't align with the yoke in the same way that the right sleeve aligns with the yoke. ...

The uniform-probability model of the placement of the lines is not as "preposterous" and "outrageous" as the ProPublica article suggests. But no one can see for themselves that there is no "effort to line up any of the other pieces of cloth" from the mere fact that the plaid patterns are displaced where a "sleeve aligns with the yoke." That is like saying that a marksman is shooting entirely at random because he missed the bullseye. There is a random component, but if the the marksman is skilled, bullets are more likely to arrive near the center of the target than to be dispersed equally across it.

Our theory about the marksman firing at random would be more credible if we saw that he was wearing a blindfold and firing rapidly. Likewise, the theory in McKreith is that the workers won't "take the time to physically line [the plaid patterns] up when they are sewing." It could be a good theory, but how has it been validated? Might not some workers try to get the unit patterns to line up just a little bit as they join the pre-cut pieces of fabric? If that happened, smaller cross-seam offsets would be more probable than larger ones. We could test the theory empirically by inspecting shirts. If the uniform probability model is correct, we would expect to find pretty much the same number of 1/10" offsets (x = 0, 1, 2, and so on) at a given seam in a large sample of Van Heusen shirts with the same design. The FBI apparently had no such data to support the probability model.

To be sure, the probability model was empirically motivated — Dr. Vorde Bruegge informed himself about the manufacturing process by visiting several manufacturing plants. His sense of the process might be entirely correct. When interviewed, I was not told what model he had adopted and what the basis for it was. `But even if I had been asked to study the testimony before reacting, I still might have said that even a plausible model could be "terribly flawed" (or at least not well validated).

"650 Billion, Give or Take a Few Billion"

A characteristic that results from the manufacturing process that occurs with a probability of 1/35 does not "individualize" an item in the only-one-in-the-universe sense that criminalists use the term. It is a generic feature, and Dr. Vorder Bruegge did not claim otherwise. To arrive at the ultimate opinion took a few more steps. The first was to assume that the offset at each seam is statistically independent of the offset at every other seam (and combination of them).

Q. Now, would the same randomness apply to all the other features in pieces of cloth that go into the shirt?
A. Yes, they would.
Q. So if you were able to, for example, reach a measurement as far as the left sleeve is concerned, with the yoke, would the line up — would the line up be the same or would it be, again, random with respect to the right sleeve?
A. There's going to be a 1 in 35 chance that it's going to be the same on the right sleeve as it is on the left sleeve.
Q. All right. So if you were able to find, for example, from the photographs, two points of identification, whereby the photograph matches the shirt in one area, maybe the yoke to the sleeve on the left side, and then you're able to find a second point of comparison or identification, say on the right sleeve and the yoke, what would be the odds of two shirts randomly being manufactured coming from the factory that would match this particular shirt?
A. It would be ... 1 in 35. But to simplify things and to be conservative, I prefer to use one in 30. By saying one in 30, that's — each giving it a better chance of being the same, but it makes the math easier. Thirty times 30 is 900. So one in 900 chance that you're going to find another shirt that has the left sleeve aligned to the yoke the same way and the right sleeve aligned to the yoke in the same way.
Q. All right. Now let's say you ... have a good enough picture that you can make three points identification. ...
A. Well, ... if we had sleeve to yoke, yoke to back, and yoke to sleeve, then that’s 30 times 30 times 30, which is one in 27,000.
Q. So if you were able to make three items of — points of identification the odds would be one in 27,000 that there would be two shirts randomly made from the factory that would have those three points of identification exactly the same?
A. Correct. ...
Q. And were you able to actually make those types of identification with respect to the photographs depicted in the bank robbery surveillance photos with this shirt ... .?
A. Yes. I was.
Q. And what was the highest number of points of identification that you were able to match up with respect to any particular bank robbery photo — in the surveillance photographs with respect to this shirt?
A. Eight. ...
Q. So 30 to the eighth power [would] be the odds in which two shirts would be randomly manufactured by the company [with] all those eight points of identification lining up exactly the same?
A. That's correct.
Q. And does your pocket calculator actually print out all the numbers that would come out if you were to insert 30 to the 8th power?
A. No. It came out to be 6.5 x 10 to the 11th, which is basically 650 billion, give or take a few billion.

Some of ProPublica's criticism of this part of the testimony misses the mark. The one example of the "[m]any problems in the examiner’s testimony [that] went unnoticed, or were simply unknown, during trial" is that "Vorder Bruegge undercut the precision of his calculations when he admitted having rounded down the shirt measurements used in his calculations because 'it makes the math easier.'" But how could anyone not notice that he used the figure of 1/30 instead of 1/35? And why is that reduction in "the precision of the calculations" a problem for the defendant? It means that the joint probability is even smaller than the pocket calculator's output.

When ProPublica contacted me and quoted the 1/650,000,000,000 figure, my reaction was "How could you get that number"? Not knowing anything about the case, I assumed it was the product of frequency estimates for different kinds of characteristics, such as shirt size, color, style, pattern, imperfections, discolorations, and so on. I doubted that such frequencies were known with sufficient accuracy to justify giving a single  astronomically large denominator to a jury. 5/

Apparently, "Karen Kafadar, chair of the statistics department at the University of Virginia," had a similar reaction and was among the "seven statisticians and independent forensic scientists [who] told ProPublica that "[t]he statistics were also preposterous" because "[t]he features Vorder Bruegge matched might be common in plaid shirts, making them of little value for identifying the garments." Indeed, Dr. Kafadar inveighed "that the 1-in-650-billion claim 'makes about as much sense as the statement two plus two equals five.'"

However, while the proposition that "two plus two equals five" seems to violate mathematical logic, it is not illogical to argue for a uniform probability distribution of offsets and to multiply probabilities as Dr. Vorder Bruegge did. If the 1/10" measurements are all correct, if the offsets are uniformly distributed over such strips, and if the independence assumption for different seams holds, then the probability of an equal offset at every one of n corresponding seams is (1/35)ⁿ, just as he testified. The assumptions are part of a perfectly logical argument, and they are not inherently "preposterous." But they have not been the subject of any systematic study (that I know of).

The Probability of Uniqueness

With all that said, the testimony had an unusual virtue over the typical "individualization" thinking of its day. Dr. Vorde Bruegge followed Lord Kelvin's dictum that "when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be." Dr. Vorde Bruegge was explicit and numerical about the grounds for concluding that only one shirt was involved in the pictures:

Q. [D]id you, during your research, contact the ... Van Heusen Company or the factory to determine ... approximately how many shirts using this cloth and this design were made by them?
A. They do not keep records of every single shirt that they ever make, but they have people who recognize patterns and they also know what their typical runs are. [A]t most, they said they would have made about no more than 18,000 of these shirts. ...
Q. So seven points of identification with respect to the bank robbery surveillance photos and the shirt with the Commerce Union bank robbery?
A. Yes.
Q. Let's go to the next robbery. ...
A. That would be seven, I believe.
Q. So the odds of this being random with two shirts, 30 to the seventh power?
A. If we use 30.
Q. Being conservative?
A. Yes. ...
A. That's seven points of identification with respect to the Bank of America shirt?
A. Yes.
Q. [F]rom the ... Bank United robbery? ... Five positive points of comparison?
A. Right.
Q. Okay. South Trust ... Well it's either 30 to the 7th power or 30 to the 8th power or 30 to the 6th power? A. That's it for South Trust. ...
Q. So that would be five points of identification for the Union Bank photographs?
A. Yes.
Q. [I]s there a point that you were able to conclude that all the photographs in all the different bank robberies are depicting the same shirt?
A. Well, for me in this case, it only took three points. Because with one to the 30th chance each time ... having three points ... I've got 30 times 30 times 30, which is 27,000, which is half again as many as 18,000. [S]o for my opinion it's enough to go over to say that three of those are enough. The fact that I've got four, five, six, seven and eight, just makes me all the more certain.
Q. And what is your opinion with respect to this comparison analysis?
A. They're all the same shirt.
Q. All right. And all the shirts. In the bank robbery surveillance photographs, do you have an opinion as to whether or not they are the same shirt as Exhibit 11, which is the questioned shirt?
A. Government Exhibit 11, in my opinion, is the shirt worn by the bank robber in each of these seven bank robberies. ...

Right or wrong, the testimony is transparent about the threshold for concluding that a picture has the defendant's shirt in it. For 18,000 shirts, Dr. Vorder Bruegge asserted, it takes only three seams to conclude that there is but one shirt in existence. But is this a reasonable criterion for an inference of uniqueness?

It might seem that way. One might be tempted to reason as follows: The random-match probability for two seams is 1/900. For 18,000 shirts manufactured as described in the uniform probability model, we would expect to find about 18,000 shirts x 1/900 matches per shirt = 200 shirts with two matching seams. That is way too many to claim individuality. But for 3 matching seams, the expected number is 18,000 x 1/27,000 = 2/3. In other words, we would expect to find lots of two-seam matches, but not even one three-seam match. So it does not look like a second shirt is very likely for three or more matching seams.

But there is a problem here that did go unrecognized in the case and the article about it. I have called it the expected value fallacy. 6/ It consists of thinking that as long as the expected number of items in a population is less than 1, the probability that there really is less than one is very high. Life, or at least mathematics, is not this simple. Even when the expected number is a little less than 1, as it is here, the probability of at least one additional matching item is appreciable. In this case, it is about 3/10, as shown in Box 1.

BOX 1. THE PROBABILITY OF ANOTHER MATCHING SHIRT
The number y of items with a given characteristic that has a constant, small probability p of appearing in each item in a large population of size n is a Poisson random variable with the parameter λ = np. The probability of any y is f(y; λ) = λ^ye^-λ/y! For the three-seam match, λ = 2/3.The conditional probability that at least one more item in the population has the characteristic, given the known fact that one such item is present is [1 – f(0, λ) – f(1; λ)] / [1 – f(0; λ)] = 0.30.

In short, even ignoring any modeling and measurement uncertainty in McKreith, there is a 30% probability that another Van Heusen shirt that matches at three seams has been manufactured. This large a risk of an erroneous conclusion of individuality would not be acceptable under a 2000 FBI policy that uses similar reasoning in explaining when an analyst may testify that a DNA profile comes from a named individual.

Thus, it appears that Dr. Vorde Bruegge choose a relatively undemanding quantitative threshold for source attribution. We can say this only because he was unusually clear as to the probabilistic basis for his conclusion that only one shirt —  the defendant's —  appeared in all the pictures. It also should be noted that the final opinion would have been the same had he selected a threshold as high as five, for which the conditional probability of matching Van Heusen shirts of the same type would be only 0.0004, or 0.04%. Still, the prosecution introduced the three-seam testimony to make the matches at five and more seams appear more impressive than they were. Even if this abuse of statistical reasoning did not affect the outcome, it was unfortunate.

NOTES

1. Ryan Gabrielson, The FBI Says Its Photo Analysis Is Scientific Evidence. Scientists Disagree, Propublica, Jan. 17, 2019; Ryan Gabrielson, FBI Scientist’s Statements Linked Defendants to Crimes, Even When His Lab Results Didn’t, Propublica, Feb. 22, 2019.
2. United States v. McKreith, 140 Fed.Appx. 112, 2005 WL 1600471 (11th Cir. 2005) (per curiam).
3. Hans-Werner Sinn, A Rehabilitation of the Principle of Insufficient Reason, 94 Q. J. Econ. 493 (1980); Jon Williamson, Justifying the Principle of Indifference, 8 European J. Phil. Sci. 559 (2018).
4. An oral ruling in the midst of a trial rarely creates or enshrines a rule of law. The U.S. Court of Appeals for the Eleventh Circuit affirmed the conviction, but it did not consider its opinion important enough to release for publication, and McKreith did not raise the Daubert claim on appeal. Although the January article states that McKreith “exhausted his appeals, most of which attempted to dispute the FBI Lab findings,” the Westlaw database’s history of the case displays only one direct appeal and one petition for postconviction relief for ineffective assistance of counsel. In neither of these attacks did McKreith challenge the scientific basis of the FBI lab’s work, and no court seems to have cited the case to support admitting similar testimony.
5. The article states that "The statisticians who reviewed Vorder Bruegge’s materials for ProPublica said the examiner’s calculations cannot be correct. Vorder Bruegge’s statistic — 1 in 650 billion — is simply too astronomical to be true, said Kaye, the Penn State professor. There isn’t a database documenting features on plaid-shirt seams like there is for human DNA, making it impossible to determine the likelihood a different shirt would appear to match the robber’s shirt." I was not one of "[t]he statisticians who reviewed Vorder Bruegge’s materials for ProPublica," but I was (and am) of the view that estimating such small numbers on the basis of strong modeling assumptions alone is fraught with danger. The same thing can be said about DNA evidence. Decades ago, England's leading forensic statistician, Ian Evett, rhetorically asked me why American experts give such small probabilities in DNA matches instead of stopping at a number like one in a million.
6. David H. Kaye, The Expected Value Fallacy in State v. Wright, 51 Jurimetrics J. 1 (2011).

POSTINGS IN THIS SERIES

• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 1), Mar. 3, 2019
• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 2), Mar. 19, 2019.
• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 3), Mar. 20, 2019