Against Method: ACE-V, Reproducibility, and Now Preproducibility

Forensic-science practitioners like to describe their activities as scientific. Indeed, if the work they did were not scientific, how could one say that they were practicing forensic science?

Thus, one finds textbooks with impressive titles like "Forensic Comparative Science" devoted to “[t]he comparative science disciplines of finger prints, firearm/tool marks, shoe prints/tire prints, documents and handwriting,” and more. 1/ The practitioners of these "science disciplines" describe their work as "analogous to scientific method of critically observing details in images, determining similarities or differences in the data, performing comparative measurements to experiment whether the details in the images actually agree or disagree," and so on. 2/ They insist that they follow a multistep process that "is a scientific methodology" for "hypothesis testing"3/ — even if the process lacks any defined threshold for deciding when perceived (or even objectively measured) features are sufficiently similar or different to reach a conclusion.

An entire article in the Journal of Forensic Identification — “a scientific journal that provides more than 100 pages of articles related to forensics ... written by forensic authorities from around the world who are practitioners or academics in forensic science fields” 4/ — is devoted to demonstrating that “[a]nalysis, comparison, evaluation, and verification (ACE-V) is a scientific methodology that is part of the scientific method.” 5/ The abstract observes that

Several publications have attempted to explain ACE-V as a scientific method or its role within the scientific method, but these attempts are either not comprehensive or not explicit. This article ... outlines the scientific method as a seven-step process. The scientific method is discussed using the premises of uniqueness, persistence, and classifiability. Each step of the scientific method is addressed specifically as it applies to friction ridge impression examination in casework. It is important for examiners to understand and apply the scientific method, including ACE-V, and be able to articulate this method. 6/

The Scientific Working Group on Friction Ridge Analysis, Study, and Technology (SWGFAST) agreed, urging examiners to write in their reports that ACE-V is nothing less than “[t]he acronym for a scientific method: Analysis, Comparison, Evaluation, and Verification.” 7/

It is revealing to contrast such assertions with comments on the meaning of reproducibility in science that appear in an essay published this week in Nature. There, Philip Stark, the Associate Dean of the Division of Mathematical and Physical Sciences and Professor of Statistics at the University of California (Berkeley), noted that reproducibility means different things in different fields, but pointed to “preproducibility” as a  prerequisite to reproducibility:

An experiment or analysis is preproducible if it has been described in adequate detail for others to undertake it. ... The distinction between a preproducible scientific report and current common practice is like the difference between a partial list of ingredients and a recipe. To bake a good loaf of bread, it isn’t enough to know that it contains flour. It isn’t even enough to know that it contains flour, water, salt and yeast. The brand of flour might be omitted from the recipe with advantage, as might the day of the week on which the loaf was baked. But the ratio of ingredients, the operations, their timing and the temperature of the oven cannot.

Given preproducibility — a ‘scientific recipe’ — we can attempt to make a similar loaf of scientific bread. If we follow the recipe but do not get the same result, either the result is sensitive to small details that cannot be controlled, the result is incorrect or the recipe was not precise enough ... . 8/

Descriptions of procedures for subjective pattern-matching in traditional forensic science are much like the list of ingredients. There are no quantitative instructions for how many and which of the potentially distinguishing features to use and how long to process or cook these ingredients at each step of the process. Indeed, it is even worse than that. Even though trained examiners know what ingredients to choose from, they can pick any subset of them that they think could be effective for the case at hand. Thus, although ACE-V can be described as a series of steps within "a broadly stated framework," 9/ that does not make it a “scientific recipe.” Imprecision at every step deprives it of  "preproducibility." It might be called a "process" rather than a "method," 10/ but in the end, “ACE-V is an acronym, not a methodology." 11/

It does not follow, however, that the comparisons and conclusions are of no epistemic value or that they cannot be studied scientifically. Quite the contrary. Unlike the natural sciences, the absence of preproducibility does not preclude reproducibilty. Another laboratory expert can start with the same materials to be compared, and we can see if the outcome is the same. Moreover, we can even conduct blind tests of examiner performance to determine how accurately criminalists are able to classify traces originating from the same source and traces coming from different sources.

Such validation studies show that latent print examiners, for example, have real expertise, but these findings do not mean that expert examiners are following a particularly “scientific method” in making their judgments. Psychologists report that some individuals are phenomenally accurate in recognizing faces, 12/ but that does not mean that the “super recognizers” are using a well-defined, or indeed, any kind of scientific procedure to accomplish these feats. The training and experience that criminalists receive may include instruction in facts and principles of biology and physics, and their performance may be generally accurate and reliable, but that does not mean that they are applying a scientific method. A flow chart is not a scientific test.

Until criminalists can articulate and follow a preproducible procedure, they should not present their work as deeply scientific. In court, they can explain that scientists have studied the nature of the patterns they analyze. They can refer to any well-designed studies proving that subjective pattern-matching by trained analysts can be valid and reliable. They can vouch for the fact that criminalists have been making side-by-side comparisons for a long time. If courts are persuaded that the resulting individual opinions are helpful, then skilled witnesses can give those opinions. But such opinions should not be gussied up as a scientific method of hypothesis testing. 13/ As one critic of such rhetoric explained, "forensic science could show that it does have validation, certification, accreditation, oversight, and  basic  research  without  showing  that  it  uses  the  'scientific method.'" 14/

NOTES

1. John R. Vanderkolk, Forensic Comparative Science xii (2009).
2. Id. at 90.
3. M. Reznicek, R.M. Ruth & D.M. Schilens, ACE-V and the Scientific Method, 60 J. Forensic Identification 87, 87 (2010). See also Michele Triplett & Lauren Cooney, The Etiology of ACE-V and its Proper Use: An Exploration of the Relationship Between ACE-V and the Scientific Method of Hypothesis Testing, 56 J. Forensic Identification 345, 353 (2006), http://www.fprints.nwlean.net/JFI.pdf (“ACE-V is synonymous with hypothesis testing. A more in-depth understanding of scientific methodology can be found by reading the works of well-known scientists and philosophers such as Aristotle, Isaac Newton, Francis Bacon, Galileo Galilei, and Karl Popper, to name just a few.”).
4. Abstract of Journal of Forensic Identification (JFI), https://www.theiai.org/publications/jfi.php, accessed May 25, 2018.
5. Reznicek et al., supra note 3, at 87.
6. Id. The Department of Justice has retreated from this phrasing, preferring to call “an examiner’s belief” an “inductive inference . . . made in a logical and scientifically defensible manner.” Department of Justice, Approved Uniform Language for Testimony and Reports for the Forensic Latent Print Discipline, Feb. 22, 2018, at 2 & 2 n.2, https://www.justice.gov/file/1037171/download.
7. Scientific Working Group on Friction Ridge Analysis, Study, and Technology, Standard for Reporting Friction Ridge Examinations (Latent/Tenprint), Appendix, at 4 n.2, 2012, https://www.nist.gov/sites/default/files/documents/2016/10/26/swgfast_standard_reporting_2.0_121124.pdf (emphasis added).
8. Philip B. Stark, Before Reproducibility must Come Preproducibility, 557 Nature 613 (2018), doi: 10.1038/d41586-018-05256-0, https://www.nature.com/articles/d41586-018-05256-0.
9. Comm. on Identifying the Needs of the Forensic Sci. Cmty., Nat'l Research Council, Strengthening Forensic Science in the United States: A Path Forward 142 (2009).
10. Michele Triplett, Is ACE-V a Process or a Method?, IDentification News, June/July 2012, at 5–6, http://www.fprints.nwlean.net/ProcessOrMethod.pdf.
11. Sandy L. Zabell, Fingerprint Evidence, 13 J. L. & Pol'y 143, 178 (2005).
12. Richard Russell, Brad Duchaine, and Ken Nakayama, Super-recognizers: People with Extraordinary Face Recognition Ability, 16 Psychonomic Bull. and Rev. 252 (2009), doi: 10.3758/PBR.16.2.252.
13. David H. Kaye, How Daubert and Its Progeny Have Failed Criminalistics Evidence, and a Few Things the Judiciary Could Do About It, 86 Fordham L. Rev. 1639 (2018), https://ssrn.com/abstract=3084881.
14. Simon A. Cole, Acculturating Forensic Science: What Is ‘Scientific Culture’, and How Can Forensic Science Adopt It?, 38 Fordham Urb. L.J. 436, 451 (2010).