Fingerprinting under the Microscope: A Controlled Experiment on the Accuracy and Reliability of Latent Print Examinations (Part I)

Fingerprint analysis has been a definitive method of personal identification in criminal investigations for more than 100 years. The examination of fingerprints (as well as palm and sole prints) is known as “friction ridge skin analysis,” or latent print analysis. It consists of a series of steps involving experience-based comparisons of the impressions left by the ridges of foot or hand surfaces (the latent print) against a known, or exemplar print. The courts have accepted fingerprint evidence without challenge for most of the past century. However, several high profile cases in the United States and abroad have highlighted the fact that human errors can occur, and litigation over the evidentiary reliability of latent print examinations has increased dramatically in the last decade or so.¹

David Stout, Report Faults F.B.I.'s Fingerprint Scrutiny in Arrest of Lawyer, New York Times, Nov. 17, 2004 The Federal Bureau of Investigation wrongly implicated an Oregon lawyer in a deadly train bombing in Madrid because the F.B.I. culture discouraged fingerprint examiners from disagreeing with their superiors, a panel of forensic experts has concluded. ...

Charlene Sweeney, Lord Advocate to Appear Before Shirley McKie Fingerprint Inquiry, The Times (London), Oct. 21, 2008 Ms McKie, a former policewoman ... denied leaving a print at the scene [of a murder], even though fingerprint experts working for the Scottish Criminal Record Office maintained that it was hers. ... Ms McKie was charged with perjury. ... In February 2006, following a long battle to clear her name, Ms McKie was awarded an out-of-court settlement of £750,000 ... . ... [T]wo US fingerprint experts ... helped to clear Ms McKie of perjury by proving in court that the mark — known as “Y7” — did not belong to her.

By and large, the courtroom challenges have failed. Except for one unreported decision of a state trial judge in State v. Rose, No. K06-0545 (Md. Cir. Ct. Oct. 19, 2007), one withdrawn opinion of one federal district court judge in United States v. Llera Plaza, Cr. No. 98-362-10, 11, 12, 2002 U.S. Dist. LEXIS 344 (E.D. Pa. Jan. 7, 2002), vacated, 188 F.Supp.2d 549 (E.D. Pa. 2002), and one order of another federal district judge in United States v. Zajac, No. 2:06-cr-00811 CW (D. Utah Sept. 16, 2010), fingerprint examiners have been allowed to testify to unique matches and absolute exclusions obtained under a series of experience- and judgment-based steps known in the trade as ACE-V (for Analysis, Comparison, Evaluation, and Verification).^1,2 In 2009, however, a committee of the National Academy of Sciences characterized ACE-V as unvalidated and of doubtful reliability.³ The committee endorsed the view that “fingerprint experts should exhibit a greater degree of epistemological humility. Claims of ‘absolute’ and ‘positive’ identification should be replaced by more modest claims about the meaning and significance of a ‘match.’”^{3 (quoting 4)}

“More modest claims” would be informed by data on “the known or potential rate of error.”⁵ Such data could come from various experiments. For example, an expert panel of latent fingerprint examiners could conduct its own evaluations of a large sample of previous casework (blinded to the earlier outcomes). The results would test the conclusions reached in actual casework. However, the true source of the latent prints in these cases would not be known with certainty. The panel’s conclusions would have to be accepted as correct if they are to serve as the measure of the accuracy in the casework. Without proof of the panel's accuracy, the experiment would be subject to the criticism that it seeks to prove one unknown by means of another. One cannot validate astrology by finding that all astrologers agree on their forecasts. Despite this limitation, the expert panel experiment could be revealing. Studies of the predictive power of screening tests in medicine rely on this experimental design when they use a more precise (but still imperfect) test to measure the accuracy of the first resu.

Another design overcomes (at a cost) the lack of perfect knowledge of what is called “ground truth” in the validation of biometric systems for identification. Sacrificing the realism of casework for a perfect measure of accuracy, experimenters can present the analysts with pairs of prints—one latent print and one exemplar (the exemplar coming from known sources). Some of the pairs are “mates”—they come from the same finger. The rest are “nonmates”—they come from fingers of different individuals. After a double-blind presentation of the mated and nonmated pairs to the latent print examiners, the experimenters can measure the following rates of accuracy and error: positive associations for the mates (sensitivity) and nonmates (false positives), and exclusions for the nonmates (specificity) and mates (false negatives). And, for the particular mix of mates and nonmates in the test set, they also can determine the rate of correct judgments among the identifications (positive predictive value) and exclusions (negative predictive value).¹

In an April 25th online release the Proceedings of the National Academy of Sciences, researchers from the Noblis corporation and the FBI reported the results of a large-scale controlled experiment on the accuracy and reliability of latent fingerprint analysis—the first in the long history of criminal fingerprint identification. Noblis, a nonprofit organization, is an offshoot of MIT’s Lincoln Laboratory, which was established in 1951 to build the nation's first air defense system. The results of this research are tantalizing. Its objectives were "to determine the frequency of false positive and false negative errors, the extent of consensus among examiners, and factors contributing to variability in results." I shall summarize and comment on the major findings in later postings.

References

1. David H. Kaye, David E. Bernstein & Jennifer L. Mnookin, The New Wigmore, A Treatise on Evidence:  Expert Evidence (2011).
2. United States v. Mitchell, 365 F.3d 215 (3d Cir. 2004).
3. National Research Council Committee on Identifying the Needs of the Forensic Sciences Community, Strengthening Forensic Science in the United States: A Path Forward (2009).
4. Jennifer L. Mnookin, Fingerprint Evidence in an Age of DNA Profiling, 67 Brooklyn L. Rev. 13 (2001).
5. Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993).
6. Bradford T. Ulerya, R. Austin Hicklina, JoAnn Buscaglia & Maria Antonia Roberts, Accuracy and Reliability of Forensic Latent Fingerprint Decisions, Proc. Nat’l Acad. Sci. (2011), available at http://www.pnas.org/content/early/2011/04/18/1018707108.full.pdf+html.