The New York Court of Appeals Returns to Probabilistic Genotyping Software (Part III—Six Empirical Studies)

New York’s Court of Appeals returned to the contentious issue of “probabilistic genotyping software” (PGS) in People v. Wakefield, 2022 N.Y. Slip Op. 02771, 2022 WL 1217463 (N.Y. Apr. 26, 2022). As previously discussed, in People v. Williams, 147 N.E.3d 1131 (N.Y. 2020), a slim majority of the court had reasoned that the output of a computer program should not have been admitted without a full evidentiary hearing on the program's general acceptance within the scientific community.

In Wakefield, the Court of Appeals faced a different question for a more complex computer program. This time, the question was whether, after holding such a hearing, the trial court erred in finding that the more sophisticated program was generally accepted as a scientifically valid and reliable means of estimating “likelihood ratios” for DNA mixtures like the ones recovered in the case. The program, known as TrueAllele, is marketed by Cybergenetics, “a Pittsburgh-based bioinformation company [whose] computers translate DNA data into useful information.”

As discussed separately, the Wakefield court held that, in the circumstances of the case, the output of TrueAllele was admissible to associate the defendant with a murder. It emphasized “multiple validation studies ... demonstrat[ing] TrueAllele's reliability, by deriving reproducible and accurate results from the interpretation of known DNA samples.” 2022 WL 1217463 at *7. But the court did not describe the level of accuracy attained in any of the validation studies. That is surely something lawyers would want to know about, so I decided to read the “peer-reviewed publications in scientific journals” (id.) to which the court must have been referring.

The state introduced 31 exhibits at the evidentiary hearing in 2015. Nine were journal publications of some kind. Six of those described data collected to establish (or indirectly suggesting) that TrueAllele was accurate. Only three of them relied on “known DNA samples” as opposed to samples from casework. The synopses that follow do not describe all the parts of them, let alone all the findings from them. I merely pick out the parts that I found most interesting and most pertinent to the question of accuracy or error (two sides of the same coin).

The 2009 Cybergenetics Known-samples Study

The first study is M.W. Perlin & A. Sinelnikov, An Information Gap in DNA Evidence Interpretation. 4 PLoS ONE  e8327. This experiment used 40 laboratory-constructed two-contributor mixture samples (from two pairs of unrelated individuals) with varying mixture proportions and total DNA amounts (0.125 ng to 1 ng) to show that TrueAllele was much better at classifying a sample as containing a contributor’s DNA than was the cumulative probability of inclusion method (CPI) that employed peak-height thresholds for binary determinations of the presence of alleles. TrueAllele’s likelihood ratios (LRs) supported the hypothesis of inclusion in nearly every instance (LR>1).

However, the data could not reveal whether the level of positive support (log-LR) was accurate. Does a computed LR of 1,000,000 “really” indicate evidence that is five orders of magnitude more probative than a computed LR of 10? The “empirical evidence” from the study cannot answer this question. The best we can do is to verify that the computed LR increases as the quantity of DNA does. The uncertainty inherent in the PCR process is smaller for larger starting quantities, and this should be reflected in the magnitude of the LR.

The 2011 Cybergenetics–New York State Police Casework Study

The second study also used two-contributor mixtures, but these came from casework in which the alleles, as ascertained by conventional methods, did not exclude the defendant as a possible contributor. In Mark W. Perlin et al., Validating TrueAllele  DNA Mixture Interpretation, 56 J. Forensic Sci. 1430 (2011), researchers from Cybergenetics and the New York State Police laboratory selected “16 two-person mixture samples” that met certain criteria “from 40 adjudicated cases and one proficiency test conducted in” the New York laboratory. TrueAllele generated larger LRs than those from the manual analyses. That TrueAllele did not produce LRs < 1 (indicative of exclusions) for any defendant included by conventional  analysis is evidence of a low false-exclusion probability. The computed LRs are greater than 1 when they should be. But this empirical evidence does not directly address the question of whether the magnitude of the LRs themselves are as close or as far from 1 as they should be if they are to be understood as a Bayes' factor.

The 2013 Cybergenetics–New York State Police Casework Study

The third study is more extensive. In Mark W. Perlin et al., New York State TrueAllele ® Casework Validation Study, 58 J. Forensic Sci. 1458 (2013), Cybergenetics worked with the New York laboratory to reanalyze DNA mixtures with up to three contributors  from 39 adjudicated cases and two proficiency tests. “Whenever there was a human result, the computer’s genotype was concordant,” and TrueAllele “produced a match statistic on 81 mixture items ... , while human review reported a statistic on [only] 25 of these items.” 

This time Cybergenetics also tried to answer the question of how often TrueAllele produces false “matches” (LR>1) when it compares a known noncontributor’s sample to a mixed sample. It accomplished this by simulating false pairs of samples for TrueAllele to process. As the authors explained,

We compared each of the 87 matched mixture evidence genotypes with the (<87) reference genotypes from the other 40 cases. Each of these 7298 comparisons should generate a mismatch between the unrelated genotypes from different cases and hence a negative log(LR) value. A genotype inference method having good specificity should exhibit mismatch information values [log-LRs] that are negative in the same way that true matches are positive.

Id. at 1461. Thus, they derived two empirical distributions for likelihood ratios—one for the nonexcluded defendants in the cases (who we would expect to be actual sources)—and one for the unrelated individuals (who we would expect to be non-sources). The empirical distributions were well separated, and the log(LR) was always less than zero for the presumed non-sources. 

So TrueAllele seems to work well as a classifier (for distinguishing true-source pairs from false-source pairs) in these small-scale studies. But again, the question of whether the magnitudes of its LRs are highly accurate remains. With astronomically large LRs, it is hard to know the answer. Cf. David H. Kaye, Theona M. Vyvial & Dennis L. Young, Validating the Probability of Paternity, 31 Transfusion 823 (1991). \1/

The 2013 UCF–Cybergenetics Known-samples Study

The fourth study is J. Ballantyne, E.K. Hanson & M.W. Perlin, DNA Mixture Genotyping by Probabilistic Computer Interpretation of Binomially-sampled Laser Captured Cell Populations: Combining Quantitative Data for Greater Identification Information, 53 Sci. & Justice 103–114 (2013). It is not a validation study, but researchers from the University of Central Florida and Cybergenetics made two different two-person mixtures with equal quantities of DNA from each person. In such 50:50 mixtures, peak heights are expected to be similar, making it harder to fit the pattern of alleles into the pairs (single-locus genotypes) from each contributor than if there had been a major and minor contributor. So the team created ten small (20 cell) subsamples of each of the two mixed DNA samples by selecting cells at random. They analyzed these subsamples separately. They used TrueAllele to estimate the relative contributions (“mixture weights”) in the 20-cell samples, and found that when TrueAllele combined data from multiple subsamples, it assigned a 99% probability to the two contributor genotypes. The point of the study was to demonstrate the possibility of subdividing even small balanced samples to take advantage of peak height differences arising from imbalances in the even smaller subsamples.

The 2013 Cybergenetics–Virginia Department of Forensic Services Casework Study

The fifth study is more on point. In Mark W. Perlin et al., TrueAllele Casework on Virginia DNA Mixture Evidence: Computer and Manual Interpretation in 72 Reported Criminal Cases, 9 PLOS ONE e92837 (2014), researchers from Cybergenetics and the Virginia Department of Forensic Services compared TrueAllele with manual analysis on 111 selected casework samples. The set of criminal case mixtures paired with a nonexcluded defendant’s profile should produce large LRs. For ten pairs, TrueAllele failed to return “a reproducible positive match statistic.” Among the 101 remaining, presumably same-source pairs, the smallest LR was 18. Since the LR must be less than 1 to be deemed indicative of a noncontributor, in no instance did TrueAllele generate a falsely exonerating result.

But what about falsely incriminating LRs? This time, the researchers did not reassign the defendant’s profiles to other cases to produce false pairs. Rather, they generated 10,000 random STR genotypes (from population databases of alleles in Virginia) to simulate the STR profiles of non-sources of the mixtures from the criminal cases. They paired each of these non-source profiles with 101 genotypes that emerged from the unknown mixtures and calculated LR values. There were fewer than 1 in 20,000 LRs suggesting an association (LR > 1) among these mixture/non-source pairs; less than 1in 1,000,000 for LR > 1,000; and no false positives at all for LR > 6,054. In other words, TrueAllele produced an empirical distribution for false pairs that consisted almost entirely of LRs < 1 and that never had very large LRs. Again, it seems to be an excellent classifier.

The 2015 Cybergenetics–Kern Regional Crime Laboratory Known-samples Study

Finally, in M.W. Perlin et al., TrueAllele Genotype Identification on DNA Mixtures Containing up to Five Unknown Contributors, 60 J. Forensic Sci. 857 (2015), researchers from Cybergenetics and the Kern Regional Crime Laboratory in California obtained DNA samples from five known individuals. They constructed ten two-person mixtures by randomly selecting two of the five contributors and mixing their DNA in proportions picked at random. The researchers constructed ten 3-, 4-, and 5-person mixtures in the same manner. From each of these 4 × 10 mixtures, they created a 1 nanogram and a 200 picogram sample for STR analysis. TrueAlelle computed an LR for each of the genotypes that went into each analyzed sample (the alternative hypothesis being a random genotype).

Defining an exclusion as a LR < 1, TrueAllele rarely excluded true contributors to the 1 ng 2- or 3-contributor mixtures (no exclusions in 20 comparisons and 1 in 30, respectively), but with 4 and 5 contributors involved, the false-exclusion rates were 9/40 and 9/50, respectively. The false exclusions came from the more extreme mixtures. As long as at least 10% of the nanogram mixtures came from the lesser contributor, there were no false exclusions. The false-exclusion rates for the 200 pg samples were larger: 2/20, 4/30, 13/40, and 19/50. For these low-template mixtures, a greater proportion of the lesser contributor’s DNA (25%) had to be present to avoid false exclusions.

To assess false inclusions, 10,000 genotypes were randomly generated from each of three ethnic population allele databases. These noncontributor profiles were compared with the eight mixtures. For ethnic group and DNA mixture sample, the LRs fell well below LR=1, meaning that there were few false inclusions. For the high DNA levels (1 ng), the proportion of comparisons with misleading LRs (LR > 1 for the simulated noncontributors) were 0/600,000, 25/900,000, 186/1,200,000, and 1,301/1,500,000 for the 2-, 3-, 4-, and 5-person mixtures, respectively. The worst case (the most misleadingly high LR) occurred for the five-person mixture, where one LR was 1,592. For the low-template DNA mixtures, the corresponding false-inclusion proportions were 2/600,000, 53/900,000, 177/1,200,000, and 145/1,500,000. The worst outcome was an LR of 101 for a four-person mixture.

Apparently using “reliable” in its legal or nonstatistical sense (as in Daubert and Federal Rule of Evidence 702), the researchers concluded that “[t]his in-depth experimental study and statistical analysis establish the reliability of TrueAllele for the interpretation of DNA mixture evidence over a broad range of forensic casework conditions.” \2/ My sense of the studies as of the time of the hearing in Wakefield is that they show that within certain ranges (with regard to the quantity of DNA, the number of contributors, and the fractions from the multiple contributors), TrueAlelle’s likelihood ratios discriminate quite well between samples paired with true contributors and the same samples paired with unrelated noncontributors. \3/ Moreover, the program’s output behaves qualitatively as it should, generally producing smaller likelihood ratios for electrophoretic data that are more complex or more deviled by stochastic effects on peak heights and locations.

NOTES

1. In this early study, we compared the empirical LR distribution for parentage using presumably true and false mother-child-father trios derived from a set of civil paternity cases to the “paternity index,” a likelihood ratio computed with software applying simple genetic principles to the inheritance of HLA types. We found that the theoretical PI diverged from the empirical LR for PI > 80 or so.
2. Cf. David W. Bauer, Nasir Butt, Jennifer M. Hornyak & Mark W. Perlin, Validating TrueAllele Interpretation of DNA Mixtures Containing up to Ten Unknown Contributors, 65 J. Forensic Sci. 380, 380 (2020), doi: 10.1111/1556-4029.14204 (abstract concluding that “[t]he study found that TrueAllele is a reliable method for analyzing DNA mixtures containing up to ten unknown contributors
3. One might argue that the number of mixed samples collectively studied is too small. PCAST indicated that “there is relatively little published evidence” because “[i]n human molecular genetics, an experimental validation of an important diagnostic method would typically involve hundreds of distinct samples.” President's Council of Advisors on Sci. & Tech., Exec. Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods 81 (2016) 81 (notes omitted), https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf [https://perma.cc/R76Y-7VU]. The number of distinct samples (mixtures from different contributors) combining all the studies listed here seems closer to 100.

Mysteries of the Department of Justice's ULTR for Firearm-toolmark Pattern Examinations

The Department of Justice's Uniform Language for Testimony and Reports (ULTR) for the Forensic Firearms/toolmarks Discipline – Pattern Examination" offers a ready response to motions to limit overclaiming or, to use the pedantic term, ultracrepidarianism, in expert testimony. Citing the DoJ policy, several federal district courts have indicated that they expect the government's expert witnesses to follow this directive (or something like it). \1/

Parts of the current version (with changes from the original) are reproduced in Box 1. \2/ This posting poses three questions about this guidance. Although the ULTR is a step in the right direction, it has a ways to go in articulating a clear and optimal policy.

Box 1. The ULTR

---

DEPARTMENT OF JUSTICE
UNIFORM LANGUAGE FOR TESTIMONY AND REPORTS
FOR THE FORENSIC FIREARMS/TOOLMARKS DISCIPLINE –
PATTERN MATCH EXAMINATION ...
III. Conclusions Regarding Forensic Pattern Examination of Firearms/Toolmarks Evidence for a Pattern Match

The An examiner may offer provide any of the following conclusions:

1.Source identification (i.e., identified)
2.Source exclusion (i.e., excluded)
3.Inconclusive

Source identification
‘Source identification’ is an examiner’s conclusion that two toolmarks originated from the same source. This conclusion is an examiner’s decision opinion that all observed class characteristics are in agreement and the quality and quantity of corresponding individual characteristics is such that the examiner would not expect to find that same combination of individual characteristics repeated in another source and has found insufficient disagreement of individual characteristics to conclude they originated from different sources.

The basis for a ‘source identification’ conclusion is an examiner’s decision opinion that the observed class characteristics and corresponding individual characteristics provide extremely strong support for the proposition that the two toolmarks came originated from the same source and extremely weak support for the proposition that the two toolmarks came originated from different sources.

A ‘source identification’ is the statement of an examiner’s opinion (an inductive inference²) that the probability that the two toolmarks were made by different sources is so small that it is negligible. A ‘source identification’ is not based upon a statistically-derived or verified measurement or an actual comparison to all firearms or toolmarks in the world.

Source exclusion
‘Source exclusion’ is an examiner’s conclusion that two toolmarks did not originate from the same source.

The basis for a ‘source exclusion’ conclusion is an examiner’s decision opinion that the observed class and/or individual characteristics provide extremely strong support for the proposition that the two toolmarks came from different sources and extremely weak or no support for the proposition that the two toolmarks came from the same source two toolmarks can be differentiated by their class characteristics and/or individual characteristics.

Inconclusive
‘Inconclusive’ is an examiner’s conclusion that all observed class characteristics are in agreement but there is insufficient quality and/or quantity of corresponding individual characteristics such that the examiner is unable to identify or exclude the two toolmarks as having originated from the same source.

The basis for an ‘inconclusive’ conclusion is an examiner’s decision opinion that there is an insufficient quality and/or quantity of individual characteristics to identify or exclude. Reasons for an ‘inconclusive’ conclusion include the presence of microscopic similarity that is insufficient to form the conclusion of ‘source identification;’ a lack of any observed microscopic similarity; or microscopic dissimilarity that is insufficient to form the conclusion of ‘source exclusion.’

IV. Qualifications and Limitations of Forensic Firearms/Toolmarks Discipline Examinations
• A conclusion provided during testimony or in a report is ultimately an examiner’s decision and is not based on a statistically-derived or verified measurement or comparison to all other firearms or toolmarks. Therefore, an An examiner shall not assert that two toolmarks originated from the same source to the exclusion of all other sources. This may wrongly imply that a ‘source identification’ conclusion is based upon a statistically-derived or verified measurement or an actual comparison to all other toolmarks in the world, rather than an examiner’s expert opinion.

○ assert that a ‘source identification’ or a ‘source exclusion’ conclusion is based on the ‘uniqueness’³ of an item of evidence.

○ use the terms ‘individualize’ or ‘individualization’ when describing a source conclusion.

○ assert that two toolmarks originated from the same source to the exclusion of all other sources.

• An examiner shall not assert that examinations conducted in the forensic firearms/toolmarks discipline are infallible or have a zero error rate.

• An examiner shall not provide a conclusion that includes a statistic or numerical degree of probability except when based on relevant and appropriate data.

• An examiner shall not cite the number of examinations conducted in the forensic firearms/toolmarks discipline performed in his or her career as a direct measure for the accuracy of a proffered conclusion provided. An examiner may cite the number of examinations conducted in the forensic firearms/toolmarks discipline performed in his or her career for the purpose of establishing, defending, or describing his or her qualifications or experience.

• An examiner shall not assert that two toolmarks originated from the same source with absolute or 100% certainty, or use the expressions ‘reasonable degree of scientific certainty,’ ‘reasonable scientific certainty,’ or similar assertions of reasonable certainty in either reports or testimony unless required to do so by a judge or applicable law.³⁴

---

² Inductive reasoning (inferential reasoning):

A mode or process of thinking that is part of the scientific method and complements deductive reasoning and logic. Inductive reasoning starts with a large body of evidence or data obtained by experiment or observation and extrapolates it to new situations. By the process of induction or inference, predictions about new situations are inferred or induced from the existing body of knowledge. In other words, an inference is a generalization, but one that is made in a logical and scientifically defensible manner. Oxford Dictionary of Forensic Science 130 (Oxford Univ. Press 2012).

³ As used in this document, the term ‘uniqueness’ means having the quality of being the only one of its kind.’ Oxford English Dictionary 804 (Oxford Univ. Press 2012).
³⁴ See Memorandum from the Attorney General to Heads of Department Components (Sept. 9. 2016), https://www.justice.gov/opa/file/891366/download.

1

Are the two or three conclusions -- identification, exclusion, and inconclusive -- the only way the examiners are allowed to report their results?

In much of the world, examiners are discouraged from reaching only two conclusions--included vs. excluded (with the additional option of denominating the data as too limited to permit such a classification). They are urged to articulate how strongly the data support one classification over the other. Instead of pigeonholing, they might say, for example, that the data strongly support the same-source classification (because those data are far more probable for ammunition fired from the same gun than for ammunition discharged from different guns). 

The ULTR studiously avoids mentioning this mode of reporting. It states that examiners "may provide any of the following ... ." It does not state whether they also may choose not to -- and instead report only the degree of support for the same-source (or the different-source) hypothesis. Does the maxim of expressio unius est exclusio alterius apply? Department of Justice personnel are well aware of this widely favored alternative. They have attended meetings of statisticians at which straw polls overwhelmingly endorsed it over the Department's permitted conclusions. Yet, the ULTR does not list statements of support (essentially, likelihood ratios) as permissible. But neither are they found in the list of thou-shalt-nots in Part IV. \3/ Is the idea that if the examiners have a conclusion to offer, they must state it as one of the two or three categorical ones -- and that they may give a qualitative likelihood ratio if they want to?

2

Is the stated logic of a "source identification" internally coherent and intellectually defensible?

The ULTR explains that "[t]he basis for a 'source identification' is

an examiner’s opinion that the observed class characteristics and corresponding individual characteristics provide extremely strong support for the proposition that the two toolmarks originated from the same source and extremely weak support for the proposition that the two toolmarks originated from different sources.

Translated into likelihood language, the DoJ's "basis for a source identification" is the belief that the likelihood ratio is very large -- the numerator of L is close to one, and the denominator is close to zero (see Box 2). On this understanding, a "source identification" is a statement about the strength of the evidence rather than a conclusion (in the sense of a decision about the source hypothesis). However, the next paragraph of the ULTR states that "[a] ‘source identification’ is the statement of an examiner’s opinion (an inductive inference2) that the probability that the two toolmarks were made by different sources is so small that it is negligible."

Box 2. A Technical Definition of Support The questioned toolmarks and the known ones have some degree of observed similarity X with respect to relevant characteristics. Let Lik(S) be the examiner's judgment of the likelihood of the same-source hypothesis S. This likelihood is proportional to Prob(X | S), the probability of the observed degree of similarity X given the hypothesis (S). For simplicity, we may as well let the constant be 1. Let Lik(D) be the examiner's judgment of the likelihood of the different-source hypothesis (D). This likelihood is Prob(X | D). The support for S is the logarithm of the likelihood ratio L = Lik(S) / Lik(D) = Prob(X | S) / Prob(X | D). \4/

In this way, the ULTR jumps from a likelihood to a posterior probability. To assert that "the probability that the two toolmarks were made by different sources ... is negligible" is to say that Prob(D|X) is close to 0, and hence that Prob(S|X) is nearly 1. However, the likelihood ratio L = Lik(S) / Lik(D) is only one factor that affects Prob(D|X). Bayes' theorem establishes that Odds(D|X) = Odds(D) / L. Consequently, a very large L (great support for S) shrinks the odds in favor of S, but whether we end up with a "negligible" probability for D depends on the odds on D without considering the strength of the toolmark evidence. Because the expertise of toolmark analysts only extends to evaluating the toolmark evidence, it seems that the ULTR is asking them to step outside their legitimate  sphere of expertise by assessing, either explicitly or implicitly, the strength of the particular non-scientific evidence in the case. There is a way to circumvent this objection. To defend a "source identification" as a judgment that Prob(D|X) is negligible, the examiner could contend that the likelihood ratio L is not just very large, as the ULTR's first definition required, but that it is so large that it swamps every probability that a judge or juror reasonably might entertain in any possible case before learning about the toolmarks. A nearly infinite L would permit an analyst to dismiss the posterior odds on D as negligible without attempting to estimate the odds on the basis of other evidence in the particular case (see Box 3).

Box 3. How large must L be to swamp all plausible prior odds? Suppose that the smallest prior same-source probability in any conceivable case were p = 1/1,000,000. The prior odds on the different-source hypothesis would be approximately 1/p = 1,000,000. According to Bayes' rule, the posterior odds on D then would be about (1/p)/L = 1,000,000/L. How large would the support L for S have to be to make D a "negligible" possibility? If "negligible" means a probability below, say 1/100,000, then the threshold value of L (call it L*) would be such that 1,000,000 / L* < 1/100,000 (approximately). Hence L* > 10^11. Are examiners are able to reliably tell whether the toolmarks are such that L > 100 billion? One can use different numbers, of course, but whether the swamping defense of the ULTR really works to justify actual testimony as to "source identification" defined according to the ULTR is none too clear.

The ULTR seems slightly embarrassed with the characterization of a "source identification" as an "opinion" on the small size of a probability. Parenthetically it calls the opinion "an inductive inference," which sounds more impressive. But the footnote that is supposed to explain the the more elegant phrase only muddies the waters. It reads as follows:

Inductive reasoning (inferential reasoning): A mode or process of thinking that is part of the scientific method and complements deductive reasoning and logic. Inductive reasoning starts with a large body of evidence or data obtained by experiment or observation and extrapolates it to new situations. By the process of induction or inference, predictions about new situations are inferred or induced from the existing body of knowledge. In other words, an inference is a generalization, but one that is made in a logical and scientifically defensible manner. Oxford Dictionary of Forensic Science 130 (Oxford Univ. Press 2012) [sic]. \4/

The flaws in this definition are many. First, "inferential reasoning" is not equivalent to "inductive reasoning." Inference is reaching a conclusion from stated premises. The argument from the premises to the conclusion can be deductive or inductive. Deductive arguments are valid when the conclusion is true given that the premises are true. Inductive arguments are sound when the conclusion is sufficiently probable given that the premises are true. Second, inductive reasoning can be based on a small body of evidence as well as on a large body of evidence. In other words, deduction produces logical certainty, whereas induction can yield no more than probable truth. Third, an induction -- that is, the conclusion of an inductive argument -- need not be particularly scientific or "scientifically defensible." Fourth, an inductive conclusion is not necessarily "a generalization." An inductive argument, no less than a deductive one, can go from the general to the specific -- as is the case for an inference that two toolmarks were made by the same source. Presenting an experience-based opinion as the product of "the scientific method" by the fiat of a flawed definition of "inductive reasoning" is puffery.

3

If the examiner has correctly discerned matching "individual characteristics" (as the ULTR calls them), why cannot the examiner "assert that a ‘source identification’ ... is based on ... ‘uniqueness’" or that there has been an "individualization"?

The ULTR states that a "source identification" is based on an examination of "class characteristics" and "individual characteristics." Presumably, "individual characteristics" are ones that differ in every source and thus permit "individualization." The dictionary on which the ULTR relies defines "individualization" as "assigning a unique source for a given piece of physical evidence" (which it distinguishes from "identification"). But the ULTR enjoins an examiner from using "the terms ‘individualize’ or ‘individualization’ when describing a source conclusion," from asserting "that a ‘source identification’ or a ‘source exclusion’ conclusion is based on the ‘uniqueness’ of an item of evidence," and from stating "that two toolmarks originated from the same source to the exclusion of all other sources."

The stated reason to avoid these terms is that a source attribution "is not based on a statistically-derived or verified measurement or comparison to all other firearms or toolmarks." But who would think that an examiner who "assert[s] that two '[t]oolmarks originated from the same source to the exclusion of all other sources'" is announcing "an actual comparison to all other toolmarks in the world"? The examiner apparently is allowed to report a plethora of matching "individual characteristics" and to opine (or "inductively infer") that there is virtually no chance that the marks came from a different source. Allowing such testimony cuts the heart out of the rules against asserting "uniqueness" and claiming "individualization."

NOTES

1. E.g., United States v. Hunt, 464 F.Supp.3d 1252 (W.D. Okla. 2020) (discussed on this blog Aug. 10, 2020).
2. The original version was adopted on 7/24/2018. It was revised on 6/8/2020.
3. Are numerical versions of subjective likelihood ratios prohibited by the injunction in Part IV that "[a]n examiner shall not provide a conclusion that includes a statistic or numerical degree of probability except when based on relevant and appropriate data"? Technically, a likelihood ratio is not a "degree of probability" or (arguably) a statistic, but it seems doubtful that the drafters of the ULTR chose their terminology with the niceties of statistical terminology in mind.
4. A.W.F. Edwards, Likelihood 31 (rev. ed. 1992) (citing H. Jeffreys, Further Significance Tests, 32 Proc. Cambridge Phil. Soc'y 416 (1936)).
5. The correct name of the dictionary is A Dictionary of Forensic Science, and its author is Suzanne Bell. The quotation in the ULTR omits the following part of the definition of "inductive inference": "A forensic example is fingerprints. Every person's fingerprints are unique, but this is an inference based on existing knowledge since the only way to prove it would be to take and study the fingerprints of every human being ever born."

Are "Exclusions" Deductive and "Identifications" Merely Probabilistic?

Lately I have heard people say that “source exclusions” are the product of deductively valid reasoning, whereas “source identifications” are less certain. But the difference between such conclusions does not arise from the fact that one is deductive and the other is not. On reflection, "exclusions" are no less probabilistic in nature than "identifications." 

Presumably, the reason people may think that exclusions are deductions is that “exclusion” can be part of a deductive argument. For example,

(1) No human being with Type O blood will leave Type A blood at a crime scene.
(2) Defendant has Type O blood.
(3) The crime-scene bloodstain is Type A.
Therefore,
(4) Defendant is excluded as the source of the stain.

This argument is a formally valid deduction. If the premises (1)–(3) are true, the conclusion (4) must be true. There is, however, no guarantee that any or all the premises are true. Perhaps something very strange (but not logically impossible) happened to convert a Type O stain into a Type A one. Or perhaps the defendant or the stain was mistyped. These are not very likely events, but deductive logic does not make them true. So although (4) is certain to be true conditional on (1)–(3), we cannot be absolutely certain that (4) is in fact true. In the symbolism of probability statements, the fact that Pr[(4) | (1)&(2)&(3)] =1 does not ensure that Pr(4) = 1 unless Pr(1) = Pr(2) = Pr(3) = 1.

One might think that I have misstated the argument. Indeed, the argument ending in exclusion might be reframed as follows:

(1) Everything we know tells us that no human being with Type O blood will leave Type A blood at a crime scene.
(2) A blood test shows that defendant has Type O blood.
(3) A blood test shows that the crime-scene bloodstain is Type A.
Therefore,
(4) The blood test excludes defendant as the source of the blood stain.

This too is a deductively valid argument. The premises entail the conclusion, and the premises are even harder to dispute than the ones in the previous example. But if this is all that a criminalist means by an exclusion, then an exclusion is not actually a statement that the defendant is not the source of the trace. The conclusion in our second argument only asserts that the test has excluded the defendant. Unless the test never errs, it does not follow (deductively) that the defendant was not the source of the bloodstain. To appreciate the force of the "deduction," we need to study how often criminalists report exclusions when examining items from different sources as opposed to items from the same source.

The situation is the same for “identification.” This conclusion also can come at the end of a deductive argument. For example,

(1) Fingerprints from the same finger always match.
(2) Fingerprints from different fingers never match.
(3) The questioned and known fingerprints being compared match.
Therefore,
(4) The fingerprints being compared are from the same finger — an “identification.”

As with an exclusion, the argument from (1)–(3) to (4) is logically impeccable. If (1)–(3) are true, then so is (4). And, once more, because propositions (1)–(3) might not all be true, the truth of the “identification” is not absolutely certain.

Again, we can rephrase the argument in a (vain) effort to make it appear that the desired conclusion is purely deductive:

(1) Fingerprints from the same finger always match.
(2) Fingerprints from different fingers never match.
(3) The questioned and known fingerprints being compared match.
Therefore,
(4) I have identified the questioned print as coming from the finger that left the known print.

But again, the deductive argument does not get us to the conclusion that the known finger is the source of the questioned print. To ascertain the probative value of a positive source classification (an “identification”), we need to study the performance of criminalists making these source attributions. We need to study how often criminalists report “identification” when examining items from the same source as opposed to items from different sources. In the end, if there is a difference in the certainty we can attach to an exclusion as opposed to an identification, it does not emanate from the difference between inductive and deductive forms of argument. It results from the fact that the premises of some inductive arguments are more probably true than the premises of other inductive arguments.

Reference

1. Brian Skyrms, Choice and Chance: An Introduction to Inductive Logic (4th ed. 2000).