Does Qualitative Measurement Uncertainty Exist?

I have heard it said that forensic-science standards for interpreting the results of chemical or other tests need not discuss uncertainty in measurements of qualitative properties. For instance, ASTM International appropriately requires standards for test methods to include a section reporting on precision and bias as manifested in interlaboratory tests. Yet, it applies this requirement exclusively to quantitative measurements. Its 2021 style manual is unequivocal:

When a test method specifies that a test result is a nonnumerical report of success or failure or other categorization or classification based on criteria specified in the procedure, use a statement on precision and bias such as the following: “Precision and Bias—No information is presented about either the precision or bias of Test Method X0000 for measuring (insert here the name of the property) since the test result is nonquantitative" (ASTM 2020, § A21.5.4, pp. A3-A14).

Qualitative measurements are observation-statements such as the ink is blue, the friction ridge skin pattern includes loops, the bloodstain displays a cessation pattern, the blood group is type A, the glass fragments fit together perfectly, or the material contains cocaine. Likewise, the statements could be comparative: the recording of an unknown bell ringing sounds like it has a higher pitch than the ringing of a known bell; the hairs are microscopically indistinguishable; or the striations on the recovered bullet and the test bullet line up when viewed in the comparison microscope.

“Precision” is defined as “the closeness of agreement between test results obtained under prescribed conditions” (ibid. § A21.2.1, at A12). “A statement on precision allows potential users of the test method to assess in general terms its usefulness in proposed applications” and is mandatory (ibid. § A21.2, at A12). So how can it be that statements of precision and bias are not allowed for qualitative as opposed to quantitative findings? In both situations, the system that generates the findings could be noisy or skewed in its outcomes.

The only answer I have heard is that measurements cannot be qualitative because the word "measurement" is reserved for determining the magnitude of quantities such as length or mass. The values of these quantitative variables are basically isomorphic to the nonnegative real numbers. Counts, such as the number of alpha particles emitted in a given interval of time by radium atoms, also qualify as measurements because there is a quantitative, additive structure to them. The values of the variable are basically isomorphic to the natural numbers. Properties that only have names are described by nominal variables. Although numbers can assigned (1 for a match and 0 for a nonmatch, for example) these numbers are no more a measurement than a social security number is. In short, the argument is that because “measurements” do no not include qualitative judgments, classifications, decisions, identifications, or whatever one might call them, no statement of measurement uncertainty or error is possible, let alone required.

This argument is incredibly weak. To begin with, the definition of “measurement” is a highly contested concept. As one guide from NIST explains, a “much wider” conception of measurement than the one “contemplated in the current version of the International vocabulary of metrology (VIM)” has been developed in the metrology literature, and the measurand “may be ... qualitative (for example, the provenance of a glass fragment determined in a forensic investigation" (Possolo 2015). Broader conceptions of measurement have been the subject of many decades of writing in psychology and psychometrics (see, e.g., Humphry 2017; Mitchell 1990). Philosophers have been struggling to describe the scope and meaning of "measurement" at least since Aristotle (see, e.g., Tal 2015).

Second, even if one agrees with the definition in one NIST publication that “[m]easurement is [confined to] an experimental process that produces a value that can reasonably be attributed to a quantitative property of a phenomenon, body, or substance” (NIST 2019), some qualitative observations fit this definition. The color of a strip of litmus paper, for instance, can be understood as a value “that can reasonably be attributed to a quantitative property,” It is simply a crude measurement of pH.

Finally, the argument that there can be no measurement error for qualitative properties because those properties are not really “measured” is a semantic ploy that misses the point. The observations or estimates of nonquantitative properties as well as the individual measurements of quantitative properties are all subject to possible random and systematic error, and statements expressing the range of probable error for all measurements, observations, estimates, and classifications are essential. The need for these statements cannot be avoided for qualitative properties or judgments by the fiat of the VIM or some other dictionary. Even if “measurement” must be read in one particular, narrow, technical sense, “evaluation uncertainty” or “examination uncertainty” still must be reckoned with (Mari et al. 2020).

In sum, there is no excuse for ASTM and other organizations promulgating standards for forensic-science test methods to exempt any reported findings from required statements of uncertainty. Many statistics can be used to indicate how reliable (repeatable and reproducible) and valid (accurate) the test results may be (ibid.; Ellison & Gregory 1998; Pendrill & Petersson 2016). The qualitative-quantitative distinction affects the choice of the statistical method or expression but not the need to have one.

REFERENCES

• ASTM Int’l, Form and Style for ASTM Standards (2020), https://www.astm.org/FormStyle_for_ASTM_STDS.html.
• Stephen L. R. Ellison & Soumi Gregory, Perspective: Quantifying Uncertainty in Qualitative Analysis, Analyst 123, 1155-1161 (1998), https://doi.org/10.1039/A707970B
• Stephen M. Humphry, Psychological Measurement: Theory, Paradoxes, and Prototypes, 27(3) Theory & Psychology 407–418 (2017)
• L. Mari, C. Narduzzi, S. Trapmann, Foundations of Uncertainty in Evaluation of Nominal properties, 152 Measurement 107397 (2020), DOI:10.1016/j.measurement.2019.107397
• Joel Mitchell, An Introduction to the Logic of Psychological Measurement (1990)
• NIST, Statistical Engineering Division, Measurement Uncertainty, updated Nov. 15, 2019, https://www.nist.gov/itl/sed/topic-areas/measurement-uncertainty
• Leslie Pendrill & Niclas Petersson, Metrology of human-based and other qualitative measurements, 27(9) Measurement Sci. Technol. 27 094003 (2016)
• A. Possolo, Simple Guide for Evaluating and Expressing the Uncertainty of NIST
  Measurement Results (NIST Technical Note 1900), 2015, doi: 10.6028/NIST.TN.1900
• Eran Tal, Measurement in Science, in Stanford Encyclopedia of Philosophy (Edward N. Zalta ed. 2015), https://plato.stanford.edu/archives/fall2017/entries/measurement-science/

APPENDIX: ADDITIONAL PUBLICATIONS ON "QUALITATIVE MEASUREMENT"

1. Mary J. Allen & Wendy M. Yen, Introduction to Measurement Theory 2 (1979) ("In measurement, numbers are assigned systematically and can be of various forms. For example, labeling people with red hair "1" and people with brown hair "2" is a measurement. Since numbers are assigned to individuals in a systematic way and differences between scores represent differences in the property being measured (hair color).")
2. Peter-Th. Wilrich, The determination of precision of qualitative measurement methods by interlaboratory experiments, Accreditation and quality assurance, 15: 439-444 (2010)
3. Boris L. Milman, Identification of chemical compounds, Trends in Analytical Chemistry, 24:6, 2005 ("identification itself is considered as measurement on a qualitative scale")
4. NIST Expert Working Group on Human Factors in Latent Print Analysis, Latent Print Examination and Human Factors: Improving the Practice Through a Systems Approach, Gaithersburg: National Institute of Standards and Technology, David H. Kaye ed., 2012 (defining "measurement" broadly, to encompass categorical variables, including the examiner's judgment about the source of a print).
5. Lim, Yong Kwan, Kweon, Oh Joo, Lee, Mi-Kyung and Kim, Hye Ryoun. Assessing the measurement uncertainty of qualitative analysis in the clinical laboratory. Journal of Laboratory Medicine, vol. 44, no. 1, 2020, pp. 3-10. https://doi.org/10.1515/labmed-2019-0155 ("Measurement uncertainty is a parameter that is associated with the dispersion of measurements. Assessment of the measurement uncertainty is recommended in qualitative analyses in clinical laboratories; however, the measurement uncertainty of qualitative tests has been neglected despite the introduction of many adequate methods.")
6. Donald Richards, Simultaneous Quantitative and Qualitative Measurements in Drug-Metabolism Investigations, Pharmaceutical Technology 2013
7. Kadri Orro, Olga Smirnova, Jelena Arshavskaja, Kristiina Salk, Anne Meikas, Susan Pihelgas, Reet Rumvolt, Külli Kingo, Aram Kazarjan, Toomas Neuman & Pieter Spee, Development of TAP, a non-invasive test for -qualitative and quantitative measurements of biomarkers from the skin surface, Biomarker Research 2: 20 (2014)
8. J M Conly & K Stein, Quantitative and qualitative measurements of K vitamins in human intestinal contents, Am J Gastroenterol. 1992 Mar;87(3):311-316
9. Wenjia Meng, Qian Zheng, Gang Pan, Qualitative Measurements of Policy Discrepancy for Return-Based Deep Q-Network, IEEE Transactions on Neural Networks and Learning Systems 2020
10. Rudolf M. Verdaasdonk, Jovanie Razafindrakoto, Philip Green, Real time large scale air flow imaging for qualitative measurements in view of infection control in the OR (Conference Presentation) Proceedings Volume 10870, Design and Quality for Biomedical Technologies XII; 1087002 (2019) https://doi.org/10.1117/12.2511185
11. Rashis, Bernard, Witte, William G. & Hopko, Russell N., Qualitative Measurements of the Effective Heats of Ablation of Several Materials in Supersonic Air Jets at Stagnation Temperatures Up to 11,000 Degrees F, National Advisory Committee for Aeronautics, July 7, 1958
12. Lawrence F Cunningham and Clifford E Young, Quantitative and Qualitative Approaches, Journal of Public Transportation 1(4) (1997) ("The study also contrasts the results of quantitative and qualitative measurements and methodologies for assessing transportation service quality")
13. JM Conly, K Stein, Quantitative and qualitative measurements of K vitamins in human intestinal contents, American Journal of Gastroenterology, 1992
14. P Sinha, Workshop on Biologically Motivated Computer Vision, 2002 - Springer ("Our emphasis on the use of qualitative measurements renders the representations stable in the presence of sensor noise and significant changes in object appearance. We develop our ideas in the context of the task of face-detection under varying illumination")
15. D Michalski, S Liebig, E Thomae & A Hinz, Pain in Patients with Multiple Sclerosis: a Complex Assessment Including Quantitative and Qualitative Measurements, 40 J. Pain 219–225 (2011), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3160835/
16. Cécilia Merlen, Marie Verriele, Sabine Crunaire,Vincent Ricard, Pascal Kaluzny, Nadine Locoge, Quantitative or Only Qualitative Measurements of Sulfur Compounds in Ambient Air at Ppb Level? Uncertainties Assessment for Active Sampling with Tenax TA®, 132 Microchemical J. 143-153 (2017)
17. Tomomichi Suzuki, Jun Ichi Takeshita, Mayu Ogawa, Xiao-Nan Lu, Yoshikazu Ojima, Analysis of Measurement Precision Experiment with Categorical Variables, 13th International Workshop on Intelligent Statistical Quality Control 2019, Hong Kong ("Evaluating performance of a measurement method is essential in metrology. Concepts of repeatability and reproducibility are introduced in ISO5725-1 (1994) including how to run and analyse experiments (usually collaborative studies) to obtain these precision measures. ISO5725-2 (1994) describe precision evaluation in quantitative measurements but not in qualitative measurements. Some methods have been proposed for qualitative measurements cases such as Wilrich (2010), de Mast & van Wieringen (2010), Bashkansky, Gadrich & Kuselman (2012). Item response theory (Muraki, 1992) is another methodology that can be used to analyse qualitative data.").

What is Accuracy?

The Organization of Scientific Area Committees for Forensic Science (OSAC) has an online "lexicon" that collects definitions of terms as they appear in published standards. 1/ These may or may not be the same as definitions in textbooks or other authoritative sources. 2/ They may or may not be accurate. (Yet, the drafters of OSAC standards sometimes point to the existence of a definition in the compendium as if it were a conclusive reason to perpetuate it. 3/)

Speaking of "accurate," the word "accuracy" has five overlapping definitions in OSAC's lexicon:

• Closeness of agreement between a measured quantitiy [sic] value and a true quantity vlaue [sic] of a measurement.
• The degree of agreement between a test result or measurement and the accepted reference value.
• Closeness of agreement between a test result or measurement result and the true value. 1) In practice, the accepted reference value is substituted for the true value. 2) The term “accuracy,” when applied to a set of test or measurement results, involves a combination of random components and a common systematic error or bias component. 3) Accuracy refers to a combination of trueness and precision. [ISO 3534-2:2006].
• The closeness of agreement between a test result and the accepted reference value. 1) In practice, the accepted reference value is substituted for the true value. 2) The term "accuracy," when applied to a set of test or measurement results, involves a combination of random components and a common systematic error or bias component. 3) Accuracy refers to a combination of trueness and precision.
• Degree of conformity of a measure to a standard or true value.

Some of the definitions in the "lexicon" are designated "preferred terms." 4/ None of the definitions in the lexicon is marked preferred.

The main difficulty with the forensic scientists' set of definitions is that "accuracy" can refer to single measurements or estimates or to a process for making measurements or estimates. The longer definitions are confusing because they do not make it plain that "a combination of trueness and precision" applies to the accuracy of the process (or a large set of measurements from the process) and not so much to the accuracy of particular measurements.

"Precision" refers to the dispersion of repeated measurements under the same conditions. A precise estimate comes from a process that generates measurements that are typically tightly clustered around some value -- without regard to whether that value is the true one. A set of precise measurements -- ones that come from a process that tends to generate similar measurements when repeated -- may be far from the true value. Such measurements.(and the system that generates them) is statistically biased; these measurements have a systematic error component.

Conversely, an imprecise estimate -- one coming from a system that tends to produce widely divergent measurements -- may be essentially identical to the true value. Most other estimates from the same system would tend to stray farther from, the true value, but to say that an estimate that is spot on is not accurate sounds odd. The estimate may be unreliable (in the statistical sense of coming from a process that is highly variable), but it is practically 100% accurate (in this case). Even a generally inaccurate system may produce some accurate results.

The epistemological problem is that we should not rely on an unreliable system to ascertain the true value. For extremely imprecise point estimates, accuracy (in the sense of the absence of error and correspondence to the truth) becomes a matter of luck. It is unwise to act as if a particular measurement (or a small number of them) from an unreliable system adds much to our knowledge.

But the fact that the individual estimates provide little information is not well expressed by describing a result that is (luckily) correct as lacking accuracy.The investment analyst who said that a bitcoin will increase in value by 50% tomorrow is accurate if bitcoin's price did spike by approximately 50%. Nevertheless, this accurate prediction probably was unwarranted. Unless the analyst had a remarkable history of consistently predicting the ups and downs of bitcoin and an articulable and plausible basis for making the predictions, giving much credence to the prediction before the fact would have been unjustified.

Let's apply these elementary ideas to some forensic measurements. Suppose that analysts in a laboratory use an appropriate instrument to measure the refractive index of glass fragments. Most analysts are extremely proficient. Their measurements are both reliable (repeatability is high) and generally close to the true values. A smaller number of analysts are less proficient. Indeed, they are downright sloppy. They are not biased -- they err in both directions -- but the values they come up with are highly variable. An analyst from the proficient group obtains the value x for a particular fragment, and so does an analyst in the sloppy group.

Should we say that x is an accurate value when it comes from one of the former analysts and inaccurate when it comes from one of the latter? Some of the definitions from the standards suggest (or could be read as giving) one answer, whereas others suggest the opposite. It is far more straightforward to say that x is accurate (if it is close to the truth) in both cases.

To be sure, precision is a component of accuracy in the long run -- the imprecise analysts will tend to have lower accuracy (and higher error) rates. Their reports do not provide a sound basis for action. They are neither trustworthy nor statistically reliable. But it invites confusion to characterize every such report -- even ones that provide perfectly or approximately true values -- as inaccurate. When speaking of particular measurements, we simply need to distinguish between those that are wrong because they are far from the truth -- inaccurate -- and those that are accurate -- close to the truth either by good fortune or because of true knowledge. Systems that use luck to get the right answers are systematically inaccurate; properly functioning systems grounded on true knowledge are systematically accurate.

NOTES

1. "The OSAC Forensic Lexicon should be the primary resource for terminology and used when drafting and editing forensic science standards and other OSAC work products. It is continually updated with the latest work from OSAC units, as well as terms from newly published documentary standards and standards elevated to the OSAC Registry." OSAC Registry, https://lexicon.forensicosac.org/ (undated).
2. Cf. id. ("The terms and definitions in the OSAC Lexicon come from the published literature, including documentary standards, specialized dictionaries, Scientific Working Group (SWG) documents, books, journal articles, and technical reports. When a suitable definition can’t be located in any of these sources, an OSAC unit generates new or modifies existing definitions. Gradually terms are evaluated and harmonized by the OSAC to a single term. This process results in an OSAC Preferred Term."). 
3. E.g.,  Comment Adjudication, OSAC 2021-N-0001, Wildlife Forensics Method-Collection of Known DNA Samples from Domestic Mammals, Feb. 11, 2021, at cells L25 & L27 (OSAC Proposed Standard added to the Registry Apr. 6, 2021) (link to Excel spreadsheet at https://www.nist.gov/osac/public-documents).
4. Id. They should be called "preferred definitions" for terms, and terms that are not supposed to be used in standards anymore should be called "deprected terms," but I digress.

Last modified: 9/27/21 14:30 ET

"Quite High" Accuracy for Firearms-mark Comparisons

Court challenges to the validity of forensic identification of the gun that fired a bullet based on toolmark comparisons have increased since the President's Council of Advisors on Science and Technology (PCAST) issued a report in late 2016 stressing the limitations in the scientific research on the subject. A study from the Netherlands preprinted in 2019 adds to the research literature. The abstract reads (in part):

Forensic firearm examiners compare the features in cartridge cases to provide a judgment addressing the question about their source: do they originate from one and the same or from two different firearms? In this article, the validity and reliability of these judgments is studied and compared to the outcomes of a computer-based method. The ... true positive rates (sensitivity) and the true negative rates (specificity) of firearm examiners are quite high. ... The examiners are overconfident, giving judgments of evidential strength that are too high. The judgments of the examiners and the outcomes of the computer-based method are only moderately correlated. We suggest to implement performance feedback to reduce overconfidence, to improve the calibration of degree of support judgments, and to study the possibility of combining the judgments of examiners and the outcomes of computer-based methods to increase the overall validity.

Erwin J.A.T. Mattijssen, Cilia L.M. Witteman, Charles E.H. Berger, Nicolaas W. Brand & Reinoud D. Stoel, Validity and Reliability of Forensic Firearm Examiners. Forensic Sci. Int’l 2020, 307:110112. 

 

Despite the characterization of examiner sensitivity and specificity as "quite high," the observed specificity was only 0.89, which corresponds to a false-positive rate of 11%—much higher than the <2% estimate quoted in recent judicial opinions. But the false-positive proportions from different experiments are not as discordant as they might appear to be when naively juxtaposed. To appreciate the sensitivity and specificity reported in this experiment, we need to understand the way that the validity test was constructed.

Design of the Study

The researchers fired two bullets from two hundred 9 mm Luger Glock pistols seized in the Netherlands. These 400 test firings gave rise to true (same-source) and false (different-source) pairings of two-dimensional comparison images of the striation patterns on cartridge cases. Specifically, there were 400 cartridge cases from which the researchers made "measurements of the striations of the firing pin aperture shear marks" and prepared "digital images [of magnifications of] the striation patterns using oblique lighting, optimized to show as many of the striations as possible while avoiding overexposure." (They also produced three-dimensional data, but I won't discuss those here.)

They invited forensic firearm examiners from Europe, North America, South America, Asia and Oceania by e-mail to examine the images. Of the recipients, 112 participated, but only 77 completed the online questionnaire with 60 comparison images of striation patterns aligned side-by-side. (The 400 images gave rise to (400×399)/2 distinct pairs of images, of which 200 were same-source pairs. They could hardly ask the volunteers to study all these 79,800 pairs, so they used a computer program for matching such patterns to obtain 60 pairs that seemed to cover "the full range of comparison difficulty" but that overrepresented "difficult" pairs — an important choice that we'll talk about soon. Of the 60, 38 were same-source pairs, and 22 were different-source pairs.)

The examiners first evaluated the degree of similarity on a five-point scale. Then they were shown the 60 pairs again and asked (1) whether the comparison provides support that the striations in the cartridge cases are the result of firing the cartridge cases with one (same-source) or with two (different-source) Glock pistols; (2) for their sense of the degree of support for this conclusion; and (3) whether they would have provided an inconclusive conclusion in casework.

The degree of support was reported or placed on a six-point scale of "weak support" (likelihood ratio L = 2 to 10), "moderate support" (L = 10 to 100), "moderately strong support" (L = 100 to 1,000), "strong support" (L = 1,000 to 10,000), "very strong support" (L = 10,000 to 1,000,000), and "extremely strong support" (L > 1,000,000). The computerized system mentioned above also generated numerical likelihood ratios. (The proximity of the ratio to 1 was taken as a measure of difficulty.)

A Few of the Findings

For the 60 two-dimensional comparisons, the computer program and the human examiners performed as follows:

Table 1. Results for (computer | examiner) excluding pairs deemed inconclusive by examiners.

	SS pair	DS pair
SS outcome	(36 | 2365)	(10 | 95)
DS outcome	(2 | 74)	(12 | 784)
validity	sens = (.95 | .97) FNP = (.05 | .03)	spec = (.55 | .89) FPP = (.45 | .11)
Abbreviations: SS = same source; DS = difference source sens = sensitivity; spec = specificity FNP = false negative proportion; FPP = false positive proportion

Table 1 combines two of the tables in the article. The entry "36 | 2365," for example, denotes that the computer program correctly classified as same-source 36 of the 38 same-source pairs (95%), while the examinations of the 77 examiners correctly classified 2,365 pairs out of the 2,439 same-source comparisons (97%) that they did not consider inconclusive. The computer program did not have the option to avoid a conclusion (or rather a likelihood ratio) in borderline cases. When examiners' conclusions on the cases they would have called inconclusive in practice were added in, the sensitivity and specificity dropped to 0.93 and 0.81, respectively.

Making Sense of These Results

The reason there are more comparisons for the examiners is that there were 77 of them and only one computer program. The 77 examiners made 77 × 60 comparisons, while the computer program made only 1 × 60 comparisons on the the 60 pairs. Those pairs, as I noted earlier, were not a representative sample. On all 79,800 possible pairings of the test fired bullets, the tireless computer program's sensitivity and specificity were both 0.99. If we can assume that the human examiners would have done at least as well as the computer program that it outperformed (on the 60 more or less "difficult" cases), their performance for all possible pairs would have been excellent.

An "Error Rate" for Court?

The experiment gives a number for the false-positive "error rate" (misclassifications across all the 22 different-source pairs) of 11%. If we conceive of the examiners' judgments as a random sample from some hypothetical population of identically conducted experiments, then the true false-positive error probability could be somewhat higher (as emphasized in the PCAST report) or lower. How should such numbers be used in admissibility rulings under Daubert v. Merrell Dow Pharmaceuticals, Inc.? At trial, to give the jury a sense of the chance of a false-positive error (as PCAST also urged)?

For admissibility, Daubert referred (indirectly) to false-positive proportions in particular studies of "voice prints," although the more apt statistic would be a likelihood ratio for a positive classification. For the Netherlands study, that would be L₊ = Pr(+|SS) / Pr(+|DS) ≈ 0.97/0.11 = 8.8. In words, it is almost nine times more probable that an examiner (like the ones in the study) will report a match when confronted with a randomly selected same-source pair of images than a randomly selected different-source pair (from the set of 60 constructed for the experiment). That validates the examiners' general ability to distinguish between same- and different-source pairs at a modest level of accuracy in that sample.

But to what extent can or should this figure (or just the false-positive probability estimate of 0.11) be presented to a factfinder as a measure of the probative value of a declared match? In this regard, arguments arise over presenting an average figure in a specific case (although that seems like a common enough practice in statistics) and the realism of the experiment. The researchers warn that

For several reasons it is not possible to directly relate the true positive and true negative rates, and the false positive and false negative rates of the examiners in this study to actual casework. One of these reasons is that the 60 comparisons we used were selected to over-represent ‘difficult’ comparisons. In addition, the use of the online questionnaire did not enable the examiners to manually compare the features of the cartridge cases as they would normally do in casework. They could not include in their considerations the features of other firearm components, and their results and conclusions were not peer reviewed. Enabling examiners to follow their standard operating procedures could result in better performance.

There Is More

Other facets of the paper also make it recommended reading. Data on the reliability of conclusions (both within and across examiners) are presented, and an analysis of the extent to which examiners' judgments of how strongly the images supported their source attributions led the authors to remark that

When we look at the actual proportion of misleading choices, the examiners judged lower relative frequencies of occurrence (and thus more extreme LRs) than expected if their judgments would have been well-calibrated. This can be seen as overconfidence, where examiners provide unwarranted support for either same-source or different-source propositions, resulting in LRs that are too high or too low, respectively. ... Simply warning examiners about overconfidence or asking them to explain their judgments does not necessarily decrease overconfidence of judgments.

Another US District Court Finds Firearms-mark Testimony Admissible in the Post-PCAST World

A new opinion on firearms-toolmark identification from the U.S. District Court for the Southern District of New York continues the clear trend of admitting such identifications notwithstanding the misgivings of three scientific study groups. The opinion of Paul G. Gardephe is more careful than most, but it has some loose ends.

In United States v. Johnson, (S5) 16 Cr. 281 (PGG), 2019 WL 1130258 (S.D.N.Y. Mar. 11, 2019), the Government brought firearms-related charges against Latique Johnson, the alleged leader of the New York and Pennsylvania street gang “Blood Hound Brims.” One indictment alleged that he shot at rival gang members in a restaurant in the Bronx. New York City Police Department Detective Jonathan Fox concluded that “toolmark identification” analysis showed that bullets collected from the restaurant were fired from an AK 47 semi-automatic assault rifle that an undercover officer purchased from a former gang member a year-and-a-half later (and that was rumored to be the one Johnson used in the restaurant).

Johnson moved before trial to prevent Detective Fox from testifying, or at least to bar an opinion that the ammunition was fired from the AK 47. During the trial, the court conducted a hearing. Detective Fox testified that the premise for “what we [microscopists] do” is “that the tools that are used to manufacture ... firearms leave marks on the inside of the firearms that are unique to that particular tool.”

The court did not question the premise that characteristics are “individual characteristics” that must be “unique” when viewed under a microscope. Why insist on empirical proof of that theoretical proposition? Surely, the marks made by guns on ammunition are unique at the molecular level. But that intuition proves too much. It means that every shell or cartridge case fired from the same gun will be unique on that scale.

So the claim of uniqueness has to pertain to features as they actually are measured or perceived, and that claim requires empirical verification. Nevertheless, the belief in universal uniqueness is not critical to the work of firearms examiners. They could be incredibly accurate in associating expended ammunition with specific guns even if the high level of similarity between ammunition components from test fires and components recovered from a crime scene is not quite unique with respect to every gun that has ever existed.

In Johnson, the court found that Detective Fox followed the Association of Firearm and Tool Mark Examiners’ “Theory of Identification as it Relates to Tool Marks.” This theory of how to interpret marks requires “sufficient agreement” for a positive association. “Sufficient agreement” comes from the examiner’s sense that the level of agreement is greater than that for different guns and is within the range for ammunition from the same gun. It “means that the agreement of individual characteristics is of a quantity and quality that the likelihood another tool could have made the mark is so remote as to be considered a practical impossibility.” 1/

In challenging the AFTE theory and its application, Johnson relied “primarily on the 2008 National Research Council Report, Ballistic Imaging ... ;  the 2009 National Research Council Report, Strengthening Forensic Science in the United States: A Path Forward ... ; and the 2016 report of the President’s Council of Advisors on Science and Technology, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods (2016) ... .” The court recognized that these reports “challenge, to varying degrees, the scientific basis for toolmark identification analysis.”

After summarizing the statements in the three reports on the limited scientific knowledge of the accuracy of firearms-mark identification 2/ and reviewing the federal case law, 3/ the court “conclude[d] that toolmark identification analysis — at least as performed by Detective Fox — is sufficiently reliable to be presented to the jury.” In reaching this result, the court considered the canonical “Daubert factors.”

"Testability"

First, it found sufficient testing of the ability of examiners to reach the correct results in experiments and proficiency tests. According to the PCAST report, the published experiments were not properly designed and hence could not establish validity. As for proficiency tests, the court later recognized that they may not be representative of casework and they were not blind. So although the AFTE theory surely is testable, the extent to which it has been adequately tested remains open to debate despite the court's conclusion.

Oddly, the court also proposed that “photographic documentation and independent review” in case work tests the theory or practice. Taking pictures is useful, but it does not validate the process, and even blind verification only tests consistency between two examiners. Such reproducibility is not the same as accuracy, and validation requires comparing the examiners’ judgments to the true state of affairs — what forensic scientists sometimes call “ground truth.”

Peer Review and Publication

Second, the court determined that there was adequate “peer review and publication” — “mostly” in a journal of the Association of Firearm and Tool Mark Examiners. This part of the opinion is perfunctory. The court accepted at face value the assurance in a 2003 AFTE Journal article that the journal uses “experts within the scientific community” for “technical review.”

Academic and other critics have complained that the publication is basically a trade journal without peer review by qualified scientists. For example, one forensic scientist wrote that

The AFTE Journal ... is not freely available and requires written testimony [sic] from existing AFTE members. It has extremely limited dissemination beyond the members of AFTE — it is found only in 18 libraries of the 72,000 libraries listed in World CAT, the largest catalog of library materials — and completely lacks integration with any of the voluminous networks for the production and exchange of scientific research information. The journal engages in peer review that is neither blind nor draws on an extensive network of researchers. This is not an attitude in keeping with the openness that is a part of any true scientific research culture. 4/

That was 2014. Today, the journal is in only nine more libraries. Only sixteen university libraries make it available.

Courts evaluating the extent of publication and peer review may wish to consider the views of the National Commission on Forensic Science. This Commission, formed by the Department of Justice and the National Institute of Standards and Technology, issued guidance on what constitutes “scientific literature” for supporting forensic science and practice. The Commision’s criteria include publication “in a journal that utilizes rigorous peer review with independent external reviewers to validate the accuracy in its publications and their overall consistency with scientific norms of practice.”

AFTE emphasizes that it has long used a formal peer review system. A position paper published in its journal in 2015 commented as follows:

While it is unclear what the NCFS considers “rigorous” and “independent”, all submissions to the AFTE Journal undergo a thorough two-stage review process conducted by subject matter experts who are members of the AFTE Editorial Committee. Occasionally, a reviewer may need to enlist the knowledge of an expert outside of the field to complete a review; however, it would be extremely difficult to use only external reviewers because most qualified potential reviewers also tend to be AFTE members. Research manuscripts submitted to the AFTE Journal are independently reviewed by at least two reviewers from the Editorial Committee ... .

Controlling Standards

Third, the court expressed more hesitation in finding “controlling standards.” It wrote that

[B]oth courts and the scientific community have voiced serious concerns about the “sufficient agreement” standard, characterizing it as “tautological,” “wholly subjective,” “circular,” “leav[ing] much to be desired,” and “not scientific.” The Court shares some of these concerns.

Nonetheless, Judge Gardephe deemed “photographic documentation and verification requirements” and “extensive AFTE training and proficiency testing” to be controlling standards.

This logic is hard to follow. Record-keeping, verification, training, and proficiency testing all are critical components of a quality assurance system. But they are not standards that control how a subjective judgment is made. Detective Fox was clear about the degree of subjectivity. He "stated that ... he employs a holistic approach incorporating his 'training as a whole' and his experience 'based on all the cartridge casings and ballistics that [he] ha[s] identified and compared.'" Still, the court managed to extract "certain principles that ground his conclusions":

For example, the CMS standard — six consecutive matching striations or two groups of three matching striations — represents a “bottom standard” or a floor for declaring a match. Detective Fox will not declare that “sufficient agreement” exists unless microscopic examination reveals a toolmark impression with one area containing six consecutive matching individual characteristics, or two areas with three consecutive matching individual characteristics. ... Detective Fox’s analysis does not end at that point, however. Instead, Detective Fox goes on to examine every impression on the ballistics evidence. “All these lines should match,” as well, and if they do not, Detective Fox will not find “sufficient agreement.”

“These criteria.” the court concluded, “provide standards for Detective Fox’s findings as to “sufficient agreement.”

"Rate of Error"

Fourth, the court regarded proficiency test data as suggestive of a modest error rate and added that “even accepting the PCAST Report’s assertion that the error rate could be as high as 1 in 46, or close to 2.2%, such an error rate is not impermissibly high.” The court did not maintain that the one unpublished and unreplicated study of cartridge shell casings that PCAST used to generate this figure (for a Ruger pistol) provided a precise estimate of a false-positive error rate. Somewhat limply, it concluded “that the absence of a definite error rate for toolmark identification does not require that such evidence be precluded.” The judge did not discuss PCAST's recommendation that the upper-bound error rate be presented to the jury.

General Acceptance

Finally, the court spent little energy on the issue of general acceptance. Assuming that the relevant scientific community was limited to “forensic scientists, and firearms examiners in particular,” it perceived “no dispute.”

* * *

In denying the motion to exclude or limit the testimony, the court ironically saw the fact that the problems with firearms identifications are apparent as favoring — or at least not inhibiting — admission. It described “the weaknesses in the methodology of toolmark identification analysis” as “readily apparent,” “as discussed at length in the scientific literature,” and as “not particularly complicated or difficult to grasp.” Thus, they “are likely to be understood by jurors if addressed on cross-examination.”

At the same time, Judge Gardephe did not approve of all types of firearms-marks testimony. He recognized that overclaiming can occur; however, he saw no risk of it happening in this case:

Having heard Detective Fox’s testimony at the Daubert hearing, it is clear that he does not intend to assert — and the Government does not intend to elicit — any particular degree of certainty as to his opinions regarding the ballistics match. ... Indeed, Detective Fox’s repeated concession at the Daubert hearing that his conclusions are “based on [his] subjective opinion” stands in stark contrast to the “tendency of [other] ballistics experts ... to make assertions that their matches are certain beyond all doubt.” ... Detective Fox also testified that he “would never” state his conclusion that ballistics evidence matches to a particular firearm “to the exclusion of all other firearms ... in a court proceeding[,] ... because I haven’t looked at all other firearms.”

Although the last concession is refreshing, it creates a puzzle -- Are "individual characteristics" truly individual? If the “individual characteristics” obviously matched — as apparently they did 5/ — why shouldn't the examiner testify that they exclude every other gun? Isn’t it “practically impossible,” to use the AFTE phrase, that another AK 47 fired the bullets?

NOTES

1. The current version of the official theory is at https://afte.org/about-us/what-is-afte/afte-theory-of-identification.
2. The opinion also quoted the trivially true statements in the reports that in describing the scientific status of the pattern-identification methods, the groups were not themselves taking a stand on the legal question of their admissibility.
3. The district court wrote that

   In assessing reliability, ‘the district court must focus on the principles and methodology employed by the expert, without regard to the conclusions the expert has reached or the district court’s belief as to the correctness of those conclusions.’ Amorgianos v. Nat’l R.R. Passenger Corp., 303 F.3d 256, 266 (2d Cir. 2002).

   In General Electric Co. v. Joiner, 522 U.S. 136 (1997), however, the Supreme Court wrote that “conclusions and methodology are not entirely distinct from one another,” and Rule 702(d) specifies that “the expert has reliably applied the principles and methods to the facts of the case.”
4. David Klatzow, Justice Denied: Role of Forensic Science in the Miscarriage of Justice (2014).
5. The court was impressed that “[t]he ‘matching’ ... is stark, even to an untrained observer.”

FURTHER READING

• David H. Kaye, Firearm-Mark Evidence: Looking Back and Looking Ahead, 68 Case W. Res. L. Rev. 723-45 (2018)

Index to comments and cases on the PCAST report

Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 1)

The FBI’s crime laboratory has had its ups and downs. Mistakes or misconduct by hair analysts, latent fingerprint examiners, explosives experts, and DNA analysts have been the subject of many a headline. 1/ In contrast, the more mundane efforts of photographic experts have not prompted extensive reports of any problems, let alone scandals. These experts testify about such matters as whether a surveillance photograph matches a photograph of a given person, 2/ the height of individuals as inferred from photographic images, 3/ and whether images in child pornography cases are those “of real, non-virtual people.” 4/

These halcyon days came to end this year when Propublica published a pair of detailed articles accusing the FBI's Forensic Audio/Video and Image Analysis Unit of relying on “flawed methods” to generate biased findings that did not deserve to be called “scientific evidence.” 5/ In an interview for the first story, the reporter told me that he discovered studies showing that analysts were unable to consistently pick out the same features from pictures and that, in one case, an FBI expert had testified that the probability of certain detailed features appearing on pictures of two different shirts would be 1 in 650 billion. He wanted to know what a statistician would say about these matters. I responded that these findings were definitely problematic, and my immediate reactions appeared in the first article:

If examiners cannot mark the same features each time they use a technique, “then you can’t rely on the result, I think that’s what any statistician would say,” said David Kaye, a Penn State University law professor and expert on DNA analysis. “It’s not a reliable measure.”

Vorder Bruegge’s statistic — 1 in 650 billion — is simply too astronomical to be true, said Kaye, the Penn State professor. There isn’t a database documenting features on plaid-shirt seams like there is for human DNA, making it impossible to determine the likelihood a different shirt would appear to match the robber’s shirt. ... “It may be an honest belief,” Kaye said, “though terribly flawed.”

At the time, I had not seen the studies and testimony in question. Having begun to read through them, I am ready to share some more informed (but still tentative) impressions.

I. Reliability

A reliable measuring instrument gives approximately the same results when used repeatedly on the same material under the same conditions. A radar gun that registers the same speed (within, say, 1 mile per hour) for all cars moving at one speed gives reliable measurements at that speed. The measurements may not be correct, but they are consistent, which is all that reliability connotes in statistics.

In pattern-matching tasks performed by human observers, the situation is slightly more complicated. If the patterns are chock full of different features that are very specific to the source of the pattern, all examiners, or even the same examiner repeating a comparison at a later time, might not use the same features in coming to a subjective judgment that the source of two patterns is the same or different. For example, one fingerprint examiner might use minutiae on one part of a print one time and minutiae from another region at other times. That kind of variation would not necessarily make the binary judgments (of same-source and different-source pairs) unreliable or invalid.

The articles cited in the Propublica exposé are not really directed at such binary judgments. The more substantial article cited as proof of unreliability proposes a more standardized and quantitative approach -- a "count-based method" -- to comparing images of hands in photographs, 6/ but it does not dismiss the current, trust-the-examiner's-global-judgment approach as invalid. Rather it states that

While there is a lack of standardization in how features are documented, [this] does not invalidate the use of such features in a photographic comparison. Future study is warranted to examine how successful examiners are when tested with comparisons leading to a conclusion in regard to individualization.

In other words, even if one or two studies indicate that image examiners are not consistent in picking out features in photographs of hands for later comparisons, it does not follow that examiners are unable to reliably and accurately classify pairs of images with respect to hands they depict. The problem -- and it is a grave problem -- is that we lack the studies to know when examiners can do what they claim (at known levels of accuracy) and when they cannot.

A reader of the Propublica articles might conclude that research has demonstrated that the FBI procedures are bunk or junk science -- the article speaks of their having been "debunked." The more accurate conclusion is that they are not science (although digital technology is part of them). Some of the underlying ideas are plausible, but their implementation has yet to be validated (or invalidated) scientifically. 7/ Looking at the write-ups of the two studies, I cannot say that they move the ball greatly..And neither of them claim to. 8/

II. Probability and Individualization

The Propublica articles singled out a leading forensic scientist — Richard Vorder Bruegge, who also chairs the Digital-Multimedia Committee of the Organization of Scientific Area Committees for Forensic Sciences (OSAC) — as a source and savior of unscientific testimony about individuals and clothing captured on film. A centerpiece of the first article is his testimony in a sixteen-year-old case about the probability of finding certain features in photos of a shirt worn by a bank robber caught on film in eight robberies.

Having spent much of my career puzzling over probability computations presented in court, I wanted to read the actual transcript. For readers who share my curiosity about how probability evidence reaches juries but who may not have extra hours on their schedules, I have greatly condensed the testimony and will present some of it together with annotations in a later posting. 9/

NOTES

1. E.g., Mark Hansen, Crime Labs Under the Microscope After a String of Shoddy, Suspect and Fraudulent Results, ABAJ, Sept. 2013; John Kelly & Phillip Wearne, Tainting Evidence: Inside The Scandals at the FBI Crime Lab (1998); Office of the Inspector General, The FBI DNA Laboratory: A Review of Protocol and Practice Vulnerabilities, May 2004.
2. United States v. Martin, No. Crim. 98–178, 2000 WL 233217 (E.D. Pa. Feb. 25, 2000), aff’d, 46 Fed.Appx. 119 (3d Cir. 2002) (on the basis of many similarities, Dr. Richard Vorder Bruegge came “very close to making a positive identification”).
3. United States v. Kyler, 2015 WL 13450483, Case No.: 4:09cr30/RH/GRJ, Case No.: 4:12cv438/RH/GRJ (N.D. Fla. Feb. 26, 2015), aff’d, 429 Fed.Appx. 828 (11th Cir. 2011) (“Dr. Vorder Bruegge testified that he used reverse projection photogrammetry to analyze video images [from three robberies to] determine[] that the ... true height of the individual in the three videos was slightly above that range [of 5'10.5" and 5'11"].
4. United States v. Rodriguez-Pacheco, 475 F.3d 434 (1st Cir. 2007).
5. Ryan Gabrielson, The FBI Says Its Photo Analysis Is Scientific Evidence. Scientists Disagree, Propublica, Jan. 17, 2019; Ryan Gabrielson, FBI Scientist’s Statements Linked Defendants to Crimes, Even When His Lab Results Didn’t, Propublica, Feb. 22, 2019..
6. Christina A. Malone, Michael J. Salyards & Meredith Hein, Inter-/Intra-observer Reliability of Hand Assessment Using Skin Detail: A Count-based Method, 60 J. Forensic Sci. 1605 (2015). In many perceptual tasks, the clearer the stimuli are, the more consistent the responses will be. Thus, this paper also suggests that some hand images produce more repeatable and reproducible observations of features than others. If image examiners are reasonably reliable under some conditions, it is not appropriate to dismiss all comparisons as unreliable. But rescuing some of them requires a metric for distinguishing the cases likely to lead to reliable measurements of individual features from the cases that are likely to be generate unreliable measurements.
7. To say, as the article quoted above does, that future study is "warranted" if "individualization" is to be introduced as scientific evidence, is an understatement. Scientific evidence requires scientific validation, which translates into serious, blind testing in this context. Of course, there is no compelling reason for "individualization." Examiners can testify to their sense of the probabilities of the observed features under alternative hypotheses about the origin of those features without assessing the probabilities of the hypotheses themselves.
8. The other study cited in the article is unpublished. It was the subject of a talk at last year's annual meeting of the American Association of Forensic Sciences. Derek A. Boyd, Aislynn MacKenzie, Briana M. Turner-Gilmore, Richard Vorder Bruegge et al., Observer Agreement in the Identification and Quantification of Dorsal Hand Traits From Digital Images (abstract). The abstract acknowledges the limited proof of the validity of comparisons of "dorsal hand traits in the identification of perpetrators and victims of these criminal activities through photographic comparison." In fact, the FBI researchers state that "[t]he qualitative nature of this method prevents it from meeting Daubert standards, as there are no known error rates associated with rates of identification of these traits from visual media." Their study of the back of one hand and six examiners checked for reliability in counting "scars, moles, freckles, and knuckle skin-creases." The results were mixed:

   Calculated coefficients of variance indicated high levels of data dispersion among scars (cv=1.206), moles (cv=1.546), and freckles (cv=1.270). Coefficients of variance calculated for counts of knuckle skin-creases on each digit suggested comparatively lower levels of dispersion (cv1=0.419; cv2=0.404; cv3=0.450; cv4=0.530; cv5=0.354). Tests of intra-observer error indicated a statistically significant difference in mean counts between first and second observations of freckles (t=-2.43, df=11, p=0.034) and knuckle skin-creases on the second digit (t=-2.80, df=11, p=0.017), but not for any other traits observed.

   Testing the hypothesis that there is no difference at all in mean counts is an odd way to discuss reliability, but it sounds like the researchers were trying to say that only a few features were not reliably measured. At least, they wrote that "[t]his exploratory study found that most traits exhibited statistically minimal intra- and inter-observe disagreement" (emphasis added). But then they wrote the sentence that Propublica highlighted: "There is considerable intra- and inter-individual variation in the specific observations made by participants, which calls into question the reliability of dorsal hand traits as suitable points of interest for photographic hand comparison." Whatever Boyd et al. meant by "specific features" and "minimal ... disagreement," they somehow concluded that "[t]hese findings are consistent with recent studies that show support for qualitative methods of identification and have implications for current efforts to develop quantitative methods based off the traits investigated here."
9. After the report was published, Propublica reporter Ryan Gabrielson kindly called my attention to the transcript of the direct examination on the fourth day of the trial and the cross-examination on the fifth day, which Propublica has made available as (low resolution) pdf files.

POSTINGS IN THIS SERIES

• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 1), Mar. 3, 2019
• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 2), Mar. 19, 2019.
• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 3), Mar. 20, 2019