On a “Ridiculous” Estimate of an “Error Rate for Fingerprint Comparisons”

On November 1, Professor Christophe Champod tweeted

“The White House report noted a study showing a 1 in 18 error rate for fingerprint comparison.” This is ridiculous.

He was referring to an interview conducted for the University of Virginia School of Law 1/ and later published on the campus news site as Fallible Fingerprints. 2/ In the interview, Professor Brandon Garrett observed that

For years, courts have grandfathered in techniques like fingerprinting that provide valuable information, but with error rates and probative value that have simply not been adequately studied. For example, the White House report noted a study showing a one-in-18 error rate for fingerprint comparison and another showing a one-in-six error rate for bite-mark comparison. A 2009 report by the National Academy of Sciences carefully detailed how much of the forensic evidence used was without “any meaningful scientific validation.” Little changed.

In a guest editorial in the Washington Post, 3/ he disseminated the same statistic on the accuracy of positive associations from latent prints:

Any human technique has an error rate, and a crucial quality control is to do testing to find out how good experts really are. It is not enough for fingerprint or bite-mark examiners to vouch for their own reliability. We must put their experience to the test. The few tests that have been done show disturbing error rates. For example, the White House report highlights a study showing a 1 in 18 error rate for fingerprint comparison and another showing a shocking 1 in 6 error rate for bite marks.

The general complaint is understandable. Courts have used a low bar in allowing criminalists to testify to unjustified — and exaggerated — claims of certainty for various forms of identification evidence. But is it true that little has changed since 2009 in the field of research into the probative value of latent fingerprint identification? The “White House report” is the study of the President’s Council of Advisers on Science and Technology (PCAST) released in September 2016 (that is the subject of other postings, pro and con, on this blog). The report lists four “early studies,” and two “black box” studies. Every study is dated 2009 or later. Despite this growing body of evidence on the accuracy of traditional latent fingerprint examinations, the courts — and the members of the bar — have not generally considered the need to use the experiments in explaining the uncertainty in latent print identifications to jurors. In other words, we have better science but not necessarily better law.

How to use the data from the studies is therefore a matter of emerging importance. To employ the research correctly, one needs to consider it as a whole — not to cherry pick results to support the positions of prosecutors or defense counsel. This makes it important to consider whether Professor Champod is right when he dismisses the 1 in 18 figure as ridiculous. There are a priori reasons to trust his judgment. As a forensic scientist at the University of Lausanne, he has been a progressive force in making latent print identification more scientific and in developing ways to incorporate a recognition of uncertainty into the reports of examiners. 4/

At the same time, Professor Garrett “is a principal investigator of UVA’s year-old Center for Statistics and Applications in Forensics Evidence, which is generating new research about forensic analysis and sharing best practices in order to facilitate justice.” 5/ Surely, the principal investigators for the government-funded Forensic Science Center of Excellence should be supplying balanced assessments as they perform their mission to “evaluate and solidify the statistical foundation for fingerprint, firearm, toolmark, and other pattern evidence analyses” so as to “allow forensic scientists to quantify the level of confidence they have in statistical computations made with these methods and the conclusions reached from those analyses.” 6/ Furthermore, Professor Garrett is a leading scholar of criminal procedure and an astute analyst of the factors that can produce wrongful convictions. 7/ 

Professor Garrett is right to insist that “[a]ny human technique has an error rate, and [it is] crucial ... to do testing ... . It is not enough for fingerprint or bite-mark examiners to vouch for their own reliability [and validity].” Still, I have to side with Professor Champod. The figure of 1 in 18 for false identifications, as a summary of the studies into the validity of latent fingerprint identification, is incredibly inflated.

To give a fair picture of the studies, one cannot just pick out one extreme statistic from one study as representative. Table 1 in the report presents the false positives for “black-box studies” as follows:

	Raw Data	Freq. (Confidence bound)	Estimated Rate	Bound on Rate
Ulery et al. 2011 (FBI)	6/3628	0.17% (0.33%)	1 in 604	1 in 306
Pacheco et al. 2014 (Miami-Dade)	42/995	4.2% (5.4%)	1 in 24	1 in 18
Pacheco et al. 2014 (Miami-Dade) (excluding clerical errors)	7/960	0.7% (1.4%)	1 in 137	1 in 73

An expert witness who presented only the 1 in 18 figure would be the target of withering cross-examination. In its summary of the studies, PCAST treated 1 in 18 as one of two equally important figures. It recommended (on page 96) that

Overall, it would be appropriate to inform jurors that (1) only two properly designed studies of the accuracy of latent fingerprint analysis have been conducted and (2) these studies found false positive rates that could be as high as 1 in 306 in one study and 1 in 18 in the other study. This would appropriately inform jurors that errors occur at detectable frequencies, allowing them to weigh the probative value of the evidence.

Although slightly more complete, this presentation also would expose an expert to major problems on cross-examination. First, the two studies are not of equal quality. The FBI-Noblis study, which produced the smaller error rate, is plainly better designed and entitled to more credence. Second, the 1 in 18 figure counts a large number of what were said to be “clerical errors.” Third, the upper bounds are not unbiased estimates. The point estimates are 1 in 604, 1 in 24 (with the alleged clerical mistakes), and 1 in 137 (without them). Fourth, verification by a second examiner (who should be blinded to the outcome of the first examination) would drastically reduce these rates.

There also are countervailing considerations. For example, the PCAST upper bound only considers sampling error. Extrapolating from the experiments to practice is a larger source of uncertainty. Nonetheless, informing jurors as PCAST proposes hardly seems calculated to provide adequate and accurate information about the probative value of positive fingerprint identifications.

In the end, and regardless of what one thinks of the PCAST wording for the statistical information to give to a jury, it seems clear that 1/18 is not an acceptable summary of what research to date suggests for the false-positive probability of latent fingerprint identifications.

References

1. Eric Williamson, Ushering in the Death of Junk Science, Oct. 31, 2016.
2. Eric Williamson, Fallible Fingerprints: Law Professor Seeks to Shore Up the Science Used in Courts, UVAToday,  Nov. 11, 2016.
3. Brandon Garrett, Calls for Limits on ‘Flawed Science’ in Court Are Well-founded: A Guest Post, Wash. Post, Sept. 20, 2016.
4. See, e.g., Christophe Champod et al., Fingerprints and Other Ridge Skin Impressions (2d ed. 2016).
5. Williamson, supra notes 1 & 2.
6. NIST, Forensic Science Center of Excellence, June 12, 2014 (updated Aug. 26, 2016).
7. See, e.g., Brandon L. Garrett, Convicting the Innocent: Where Criminal Prosecutions Go Wrong (2012).

Index to comments and cases on the PCAST report

• Previous postings on friction skin ridge validation studies can be found by clicking on the labels "error" or "fingerprint."