"Double Blind Peer Review" of Research on Hair Fibers

Yesterday I heard about some new publications on forensic hair microscopy published in, of all places, a journal on pharmaceuticals. My first thought was that the journal might be a predatory one with deceptive advertising designed to con scholars into paying for publication in what appears to be a reputable scientific journal. But that was too cynical.

The papers are

• S. Sneha Harshini, Vishnu Priya Veeraraghavan, Abirami Arthanari1, R. Gayathri, S. Kavitha, J. Selvaraj, P. K. Reshma & Y. Dinesh, Comparative Study of Male and Female Human Hair: A Microscopic Analysis, 13 J. Advanced Pharm. Tech. & Rsch. S297-S301 (2022);
• S. Nehal Safiya, Vishnu Priya Veeraraghavan, Abirami Arthanari, R. Gayathri, J. Selvaraj, S. Kavitha & Y. Dinesh, J., Comparison of Human and Animal Hair – A Microscopical Analysis, 13 Advanced Pharm. Tech. & Rsch. S112–S116 (2022); and
• S. Nehal Safiya, Vishnu Priya Veeraraghavan, Abirami Arthanari, R. Gayathri, J. Selvaraj, S. Kavitha & Y. Dinesh, A Comparative Study of Different Animal Hairs: A Microscopic Analysis, 13 Advanced Pharm. Tech. & Rsch. S117–S120 (2022)
  

The Journal of Advanced Pharmaceutical Technology & Research has a scientific society and a respectable publisher behind it. The former is the “Society of Pharmaceutical Education & Research (SPER),” which is one of the leading pharmaceutical association in the country [of India] ... with a member base of around 3,500, it spread [sic] across the country and have [sic] 13 state branches.”

The latter is Wolters Kluwer’s Medknow. Located in Mumbai, Medknow “provides publishing services for peer-reviewed, online and print-plus-online journals in medicine on behalf of learned societies and associations with a focus on emerging markets.” Wolters Kluwer insists that Medknow “journals employ a double-blind review process, in which the author identities are concealed from the reviewers, and vice versa, throughout the review process.”

Although the journal is not indexed in Medline, the research comes from the Saveetha Dental College, Chennai, Tamil Nadu, India—”one of the finest institutions in the world with a unique curriculum that is a spectacular fusion of the best practices of the east and west.”

So I read the papers. Unbelievable.

The abstract and the conclusion of "A Comparative Study of Male and Female Human Hair" announce that “[t]his study can be concluded that the structural comparison between male and female hair specimens can be used as evidence for forensic analysis at crime scenes.” How so? Well, for one thing, “[i]n this study, it is observed that the color of human male hair is completely black, while it is black on the proximal end and brown at the distal end of human female hair.”

Astonishingly, the sample of hairs is never described. How many men and women provided hair? Where did they come from? How many hairs were taken from each subject and compared? Were the examinations blind? Without this elementary information, no one can understand or assess the reported results.

The “Comparison of Human and Animal Hair – A Microscopical Analysis” is similarly devoid of any meaningful description of the research.

The “Comparative Study of Different Animal Hairs: A Microscopic Analysis” appears to be a description of four hairs – one each from a dog, a cat, a horse, and a rat. The researchers found some differences among them. This they found encouraging: "The present study might be used in forensic investigations."

So much for "double blind peer review."

Results of a Proficiency Test of Hair Examiners

Existing proficiency tests of forensic examiners who offer opinions on the origins of trace evidence are not designed to estimate the conditional probabilities for false negatives (exclusions) and false positives (inclusions). 1/ They rarely replicate the conditions of casework; indeed, proficiency exercises may be significantly easier than casework, and the examiners usually know they are being tested..

This is not to question the importance and value of basic proficiency testing. Even simple tests can inform managers of the capabilities of examiners and identify some analysts who could benefit from additional training. But it remains tempting to think that high scores on proficiency tests mean that examiners rarely make mistakes in practice. 2/ Conversely, it has been argued that poor scores on proficiency tests are danger signs for admitting or relying on expert opinions in court. 3/

To the extent that current proficiency test data are pertinent to accuracy in casework, a recent test by Forensic Testing Services may be of interest. 4/ FTS administered the test to hair examiners at an undisclosed number of laboratories, where 45 out of 52 examiners completed it. It investigated whether examiners could tell that a small number of "questioned" hairs, that FTS sampled from one individual and gave to the examiners, did not come from two other individuals (as indicated by their correspondence to reference samples that FTS produced from those individuals). The questioned sample, designated Item #3 (and said for test purposes to have been found clutched in the hand of a victim), was a set of five hairs from the scalp of a 56-year-old white woman. Item #1 was a set of ten hairs from the scalp of a deceased 87-year-old white male. Item #2 was ten hairs from the scalp of a deceased 65-year-old white male.

The examiners were asked to determine whether item #3 -- the five "questioned" hairs -- are "consistent in microscopic characteristics to the known hair sample sets" #1 and #2. They also considered macroscopic characteristics such as color. A "Statement Regarding Test Design" explained that

Different hairs from the same body region of a person exhibit variation in microscopical characteristics and features. It is difficult to prepare a microscopical examination of hair proficiency test due to this natural variation. Our approach to this test is to provide several questioned hairs (known to be from the same individual) to compare as a group to a sample of known hair. Although the test is not realistic from the standpoint that most analysts would characterize each hair individually, this approach is designed to ensure consistency between distributed tests. [5/]

We realize that most examiners would prefer larger sample sets of known hairs. The use of smaller known sample sets was also intended to ensure consistency [among] distributed tests. [6/]

In essence, the examiners responded C (consistent), X (excluded), or I (inconclusive) as shown in the following table:

	#3 from #1?	#3 from #2?
C	6	0	← false inclusions
X	37	41	← true exclusions
I	2	4	← no conclusion

Putting aside the inconclusives (which should not have any impact in court even though they are an important aspect of an examiner's proficiency at acquiring and evaluating information), the comparisons of #3 to #1 produced 6 / (6+37) = 14% false inclusions, and the comparison of #3 to #2 produced no false inclusions. Pooling these decisions, the examiner made (6+0) / (6+37 + 0+41) = 6/84 = 7.1% false inclusions.

Because the examiners had no opportunity to compare the five questioned hairs to a representative sample of the woman's hairs, the proficiency test yields no true inclusions. Consequently, it is not possible to estimate the probative value of the hair examiners' conclusions of consistency. To see this, suppose that a reference set of #3 hairs had been provided and that, disappointingly, the examiners found consistency only 7.1% of  the time in this situation. Then the examiners would be declaring consistency as often with different sources as with the same source! 7/ Findings of C would not help a judge or jury distinguish between sources and (these particular) nonsources of the questioned hairs.

Of course, it seems likely that examiners would achieve a higher proportion of true inclusions than a mere 7.1%. If the proportion were, say, 71%, judgments of C would offer ten times as much support to the hypothesis that the inclusion is correct than to the hypothesis that a nonsource is included. The highest probative value (point estimate) compatible with the reported data occurs when the examiners are perfect at responding to true sources with a judgment of C. In that case, the ratio would be 100%  / 7.1%  = 14.

But figures like 1, 10, and 14 are speculative. This proficiency test provides no data on sensitivity (to use the technical term for Pr(C | #3)), which is an essential component of probative value. 8/ Proficiency test manufacturers might want to consider adding a measure of sensitivity to their tests.

NOTES

1. See, e.g., Jonathan J. Koehler, Proficiency Tests to Estimate Error Rates in the Forensic Sciences, 12 Law, Probability & Risk 89 (2013).
2. E.g., United States v. Crisp, 324 F.3d 261, 270 (4th Cir. 2003) (handwriting expert "had passed numerous proficiency tests, consistently receiving perfect scores") United States v. Otero, 849 F.Supp.2d 425 434 (D.N.J. 2012) (because "proficiency testing is indicative of a low error rate ... for false identifications made by trained examiners ...  this Daubert factor also weighs in favor of admitting the challenged expert testimony").
3. E.g., Edward J. lmwinkelried, The Constitutionality of Introducing Evaluative Laboratory Reports Against Criminal Defendants, 30 Hastings L.J. 621,636 (1979); Randolph N. Jonakait, Forensic Science: The Need for Regulation, 4 Harv. J. L. & Tech. 109 (1991).
4. Forensic Testing Services, 2018 Hair Comparison Proficiency Test FTS‐18‐HAIR1 Summary Report. 
5. Because the examiners were told, in effect, that the questioned hairs all had the same source for a reason having nothing to do with their physical features, a report on the five questioned hairs individually or on the internal consistency of that set would not have tested their skill at distinguishing hairs on the basis of those features.
6. According to R.A. Wickenheiser & D.G. Hepworth, Further Evaluation of Probabilities in Human Scalp Hair Comparisons, 35 J. Forensic Sci. 1323, 1329 (1990), "[m]acroscopic selection of 5-13 mutually dissimilar hairs was frequently unrepresentative of the microscopic range of features present in the known samples." The FTS summary does not state how the two sets of ten reference hairs were selected. Failing to capture the full range of variation in the head hairs of an individual would increase the chance of an exclusion. For this study, that could decrease the proportion of false inclusions and inflate the proportion of true inclusions.
7. The likelihood ratio in the sample would be Pr(C | #3) / Pr(C | #1 or #2) = 7.1 / 7.1 = 1.
8. See, e.g., David H. Kaye, Review-essay, Digging into the Foundations of Evidence Law, 116 Mich. L. Rev. 915 (2017).

Comedic License

The other day, I mentioned a satire by the comedian John Oliver on developments in forensic science. A few minutes of the video was played for an international audience at Harvard Law School on October 27 in the introduction to a panel discussion entitled "Evidence, Science, and Reason in an Era of 'Post-truth' Politics." The clip included remarks from Santae Tribble and his lawyer about hair evidence in the highly (and deservedly) publicized exoneration of Mr. Tribble. 1/ I got the impression from the Oliver video that an FBI criminalist claimed that there was but a 1 in 10 million chance that the hair could have come from anyone else. In Oliver's recounting of the case,

Take Santae Tribble, who was convicted of murder, and served 26 years in large part thanks to an FBI analyst who testified that his hair matched hair left at the scene, and he will tell you the evidence was presented as being rock solid. "They said they matched my hair in all microscopical characteristics. And that's the way they presented it to the jury, and the jury took it for granted that that was my hair."

You know, I can see why they did. Who other than an FBI expert would possibly know that much about hair? ... Jurors in Tribble's case were actually told that there was one chance in ten million that it could be someone else's hair.

I have not seen the trial transcript, but a Washington Post reporter, who presumably has, provided this account:

In Tribble’s case, the FBI agent testified at trial that the hair from the stocking matched Tribble’s “in all microscopic characteristics.” In closing arguments, federal prosecutor David Stanley went further: “There is one chance, perhaps for all we know, in 10 million that it could [be] someone else’s hair.” 2/

If the Post is correct, it was not "an FBI expert" who "actually told" jurors that the probability the hair was not Tribble's was 1/10,000,000. It was an aggressive prosecutor who made up a number that he imagined "perhaps ... could" be true. Like the similar figure of 1/12,000,000 in the notorious case of People v. Collins, 438 P.2d 33 (Cal. 1968), this number plainly was objectionable.

In distinguishing between the criminalist's testimony and the lawyer's argument, I do not mean to be too critical of Oliver's condensed version and passive voice. Sometimes a little oversimplification saves a lot of pedantic explanation. The sad fact is that, whichever participants in  the trial were responsible for the 1/10,000,000 figure, Tribble's case exemplifies Oliver's earlier observation that "[t]he problem is, not all forensic science is as reliable as we have become accustomed to believing. ,,, It's not that all forensic science is bad, because it's not. But too often, its reliability is dangerously overstated ... ." 3/

NOTES

1. See The FBI's Worst Hair Days, July 31, 2014, Forensic Sci., Stat. & L., http://for-sci-law.blogspot.com/2014/07/the-fbis-worst-hair-days.html
2. Spencer S. Hsu, Santae Tribble Cleared in 1978 Murder Based on DNA Hair Test, Wash. Post, Dec. 14, 2012, https://www.washingtonpost.com/local/crime/dc-judge-exonerates-santae-tribble-of-1978-murder-based-on-dna-hair-test/2012/12/14/da71ce00-d02c-11e1-b630-190a983a2e0d_story.html?utm_term=.9dc9725b8b98 (emphasis added).
3. Cf. David H. Kaye, Ultracrepidarianism in Forensic Science: The Hair Evidence Debacle, 72 Wash. & Lee L. Rev. Online 227 (2015), ssrn.com/abstract=2647430

POSTSCRIPT

After composing these remarks, I located the 2012 motion to vacate Tribble's conviction. (Motion to Vacate Conviction and Dismiss Indictment With Prejudice on the Grounds of Actual Innocence Under the Innocence Protection Act, United States v. Tribble, Crim. No. F-4160-78 (D.C. Super. Ct. Jan. 18, 2012), available at http://www.pdsdc.org/docs/default-source/default-document-library/motion-to-vacate-conviction-on-the-grounds-of-actual-innocence-january-18-2012.pdf?sfvrsn=814394d0_0.) This document contains more extended excerpts from the trial transcript, These indicate that although the FBI agent, James Hilverda, did not generate a nonsource probability of 1/10,000,000, he did insist that it was at least "likely" that Tribble was the source, and he indicated that the probability of a coincidental match could be on the order of one in a thousand.

TESTIMONY OF JAMES HILVERDA
Q. Is it possible that two individuals could have hairs with the same characteristics?
A. It is possible, but my personal experience, I rarely have seen it. Only on very rare occasions have I seen hairs of two individuals that show the same characteristics. Usually when we find hairs that we cannot distinguish from two individuals, it is due to the fact that the hairs are either so dark that you cannot see sufficient characteristics, or they are too light that they do not have very many characteristic[ s] to examine. So you cannot make a real good analysis in that manner. But there is a potential even with sufficient characteristics to once in a while to see hairs that you cannot distinguish.
Q. And is it because of that small possibility that two hairs can be from different people that you never give absolutely positive opinion that two hairs, in fact, did come from the same person.
A. That is correct.
...
Q. If you compare them and they microscopically match in all characteristics, could you say they could have come from the same person?
A. That's correct.
Q. Have you ever seen two hairs from two. different people that microscopically match?
A. Well, it is possible in the thousands of hair examples I have done, it is very, very rare to find hairs from two different people that exhibit the same microscopic characteristics. It is possible, but as I said, very rare.
...
A. ... I found that these hairs — the hairs that I removed from the stocking matched in all microscopic characteristics with the head hair samples submitted to me from Santae Tribble. Therefore, I would say that this hair could have originated from Santae Tribble.
Q. Is there any microscopic characteristics [sic] of the known hair that did not match?
A. No. The hair that I aligned with Santae Tribble matched in all microscopic characteristics, all characteristics were the same.
Q. And were there a large number of characteristics which were the same or—
A. All the characteristics were the same and there was a sufficient number of characteristics to allow me to do my examinations.
...
A. ... I think you can identify individuals, say, as to race, you can an [sic] indication that it came from an individual because it matches in all characteristics and I would say that when I have-in my experience that I feel when I have made this type of examination, it is likely that it came from the individual which I depicted it as coming from.
...
ASSISTANT UNITED STATES ATTORNEY DAVID STANLEY ARGUED:
Mrs. McCormick said at least one of the robbers, the only one that she saw, was wearing a stocking mask. A block from where that homicide occurred, more or less, through the alley and around the corner on a fresh trail found by a trained canine dog, was found what, a stocking mask. In that stocking mask was found what, a hair.
Now whose hair was that? Well it was compared by the FBI laboratory and they can say for one thing, it wasn't Cleveland Wright's hair. They can say that that hair matched in every microscopic characteristic, the hair of Santae Tribble. Now for scientific reasons, which were explained, they cannot say positively that that is Santae Tribble's hair because on rare occasions they have seen hairs from two different people that matched. But usually that is the case where there have been few characteristics present. Now Agent Hilverda told you that on these hairs, the known hairs of Santae Tribble and the hair found in the stocking, there were plenty of characteristics present, not that situation at all. But because the FBI is cautious, he cannot positively say that that is Santae Tribble's hair.
DEFENSE COUNSEL ARGUED:
Well what did Special Agent Hilverda say, microscopic analysis. Throwing the large terms away, what did he say? "Could be." He looked at one hair, he looked at Santae Tribble's hair and said, could be.
AUSA STANLEY ARGUED:
[T]he dog found a stocking with a hair which not only could be Santae Tribble's, it exactly matches Santae Tribble's in every microscopic characteristic. And Mr. Potenza says throw away all the scientific terms and reduce it to "could be." Scientific terms are important He told you how he compares hairs with powerful microscopes. It is not just the color or wave. He could reject Cleveland Wright's hair immediately; it wasn't Cleveland Wright's hair in that stocking. But he couldn't reject Santae Tribble's because it was exactly the same. And the only reason he said could be is because there is one chance, perhaps for all we know, in ten million that it could [be] someone else's hair. But what kind of coincidence is that?
... The hair is a great deal more than "could be." And if you listened to Agent Hilverda and what he really said, the hair exactly matched Santae Tribble's hair, found in the stocking.

Science Friday and Contrived Statistics for Hair Comparisons

On May 19th, Public Radio International's Science Friday show had a segment entitled "There’s Less Science In Forensic Science Than You Think." The general theme — that some practices have not been validated by rigorous scientific testing — is a fair (and disturbing) indictment. But listeners may have come away with the impression that the FBI has determined that hair examiners make up statistics from personal experience 95% of the time to help out prosecutors.

Ira Flato, the show's host, opened with the observation that "The FBI even admitted in 2015, after decades, investigators had overstated the accuracy of hair sample matches over 95% of the time in ways that benefited the prosecution." He returned to this statistic when he asked Betty Layne DesPortes, a lawyer and the current President of the American Academy of Forensic Sciences, the following question:

Dr. DesPortes, I want to go back to that FBI admission in 2015 that for decades investigators had overstated the accuracy of their hair samples, and I mean 95% of the time in a way that benefited the prosecution. Is this a form of cognitive bias coming into the picture?

Ms. DesPortes replied that

It is, and ... you would have overstatement along the lines of, "Well, I’ve never seen in my X years of experience that two hairs would be this similar, so it must be a match," and then they would just start making statistics up based on, "Well, I’ve had a hundred cases in my practice, and there have been a thousand cases in my lab, and nobody else has ever reported similar hairs like this," so let’s just start throwing in one in a hundred thousand as a statistic — "one in a hundred thousand" — and that’s where the misstatement came in.

But neither Ms. DesPortes nor anyone else knows how often FBI examiners cited statistics like "one in a hundred thousand" based on either their recollections of their own casework or their impression of the collective experience of all hair examiners. 1/

To be sure, such testimony would have been flagged as erroneous in the FBI-DOJ Microscopy Hair Comparison Review. But so would a much more scientifically defensible statement such as

The hair removed from the towel exhibited the same microscopic characteristics as the known hair sample, and I concluded it was consistent with having originated from him. However, hair comparison is not like fingerprints, for example. It’s not a positive identification. I can’t make that statement." 2/

The Hair Comparison Review was not designed to produce a meaningful estimate of an error rate for hair comparisons. It produced no statistics on the different categories of problematic testimony. The data and the results have not been recorded (at least, not publicly) so as to allow independent researchers to ascertain the extent to which FBI examiners overstated their findings in various ways. See David H. Kaye, Ultracrepidarianism in Forensic Science: The Hair Evidence Debacle, 72 Wash. & Lee L. Rev. Online 227 (2015).

The interim results from the Hair Comparison Review prompted the Department of Justice to plan a retrospective study of FBI testimony involving other identification methods as well. In July 2016, it asked a group of statisticians how best to conduct the new "Forensic Science Disciplines Review." The informal recommendations that emerged in this "Statisticians' Roundtable" included creating a database of testimony that would permit more rigorous, social science research. But this may never happen. A new President appointed a new Attorney General, who promptly suspended the expanded study.

NOTES

1. Ms. DesPortes may not have meant to imply that all the instances of exaggerated testimony were of the type she identified.
2. That statements like these may be scientifically defensible does not render them admissible or optimal.

(For related postings, click on the label "hair.")

The Department of Justice's Plan for a Forensic Science Discipline Review

On March 21, the Department of Justice announced to the National Commission on Forensic Science that it will be

expanding its review of forensic testimony by the FBI Laboratory beyond hair matching to widely used techniques such as fingerprint examinations and bullet-tracing. Officials also said that if the initial review finds systemic problems in a forensic discipline, expert testimony could be reviewed from laboratories beyond the FBI that do analysis for DOJ. 1/

The head of the Department's Office of Legal Policy welcomed input from the Commission on the following topics:

• How to prioritize disciplines
• Scope of time period
• Sampling particular types of cases
• Consideration of inaccuracies
• Levels of review
• Legal and/or forensic reviewers
• External review processes
• Ensuring community feedback on methodology
• Duty/process to inform parties 2/

On April 8, the Department quietly posted a Notice of Public Comment Period on the Presentation of the Forensic Science Discipline Review Framework in the Federal Register. The public comment period ends on May 9.

According to an earlier statement of Deputy Attorney General Sally Yates, the review is intended to "advance the practice of forensic science by ensuring DOJ forensic examiners have testified as appropriate in legal proceedings." Obviously, the criteria for identifying what is and is not "appropriate" will be critical. For example, which of the following examples of testimony about glass fragments (or paraphrases of the testimony) would be deemed inappropriate?

• "In my opinion the refractory indices of the two glasses are consistent and they could have common origin." Varner v. State, 420 So.2d 841 (Ala. Ct. Crim. App. 1982). 
• "Test comparisons of the glass removed from the bullet and that found in the pane on the back door, through which the unaccounted-for bullet had passed, revealed that all of their physical properties matched, with no measurable discrepancies. Based upon F.B.I. statistical information, it was determined that only 3.8 out of 100 samples could have the same physical properties, based upon the refractive index test alone, which was performed." Johnson v. State, 521 So.2d 1006 (Ala. Ct. Crim. App. 1986).
• "Bradley was able to opine, to a reasonable degree of scientific certainty, that the glass standard and the third fragment had a 'good probability of common origin.'" People v. Smith, 968 N.E.2d 1271 (Ill. App. Ct. 2012).
• "Blair Schultz, an Illinois State Police forensic chemist, compared a piece of standard laminated glass from defendant's windshield to a piece of glass from Pranaitis' clothing. He found them to have the same refractive index, which means that the two pieces of glass could have originated from the same source. The likelihood of this match was one in five, meaning that one out of every five pieces of laminated glass would have the same refractive index." People v. Digirolamo, 688 N.E.2d 116 (Ill. 1997).
• "[O]ne of the glass fragments found in appellant's car was of common origin with glass from the victim's broken garage window. The prosecutor asked her if she were to break one hundred windows at random in Allen County, what would be the percentage of matching specimens she would expect. Over appellant's objection, she answered that if one hundred windows were broken, six of the windows would have the properties she mentioned." Hicks v. State, 544 N.E.2d 500, 504 (Ind. 1989).

Hopefully, the Department will learn from the FBI/DOJ Microscopic Hair Comparison Analysis Review 3/ and

(1) make public the detailed criteria is employs,
(2) use a system with measured reliability for applying these criteria, and
(3) make the transcripts or other materials under review readily available to the public.

Notes

1. Spencer S. Hsu, Justice Department Frames Expanded Review of FBI Forensic Testimony, Wash. Post, Mar. 21, 2016. 
2. Office of Legal Policy, U.S. Department of Justice, Presentation of the Forensic Science Discipline Framework to the National Commission on Forensic Science, Mar. 21, 2016
3. See Ultracrepidarianism in Forensic Science: The Hair Evidence Debacle, Washington & Lee Law Review (online), Vol. 72, No. 2, pp. 227-254, September 2015

Related Posting

"Stress Tests" by the Department of Justice and the FBI's "Approved Scientific Standards for Testimony and Reports", Feb. 25, 2016

"Stress Tests" by the Department of Justice and the FBI's "Approved Scientific Standards for Testimony and Reports"

Yesterday, Deputy Attorney General Sally Yates addressed the assembled members of the American Academy of Forensic Sciences at their annual scientific meeting in Las Vegas. 1/ Some excerpts and remarks follow:

In the near future, we expect the FBI to solicit bids for an independent review—or “root cause analysis”—to determine what went wrong and why in the hair analysis field. We hope that this review will help us identify potential blind spots in our own practices and develop effective corrective measures.

But it does not take a root cause analysis to draw some initial conclusions about errors arising in the FBI’s pre-2000 hair cases. It’s clear that, in at least some of the cases reviewed, lab examiners and attorneys either overstated the strength of the forensic evidence or failed to properly qualify the limitations of the forensic analysis. This doesn’t necessarily mean that there were problems with the underlying science—it means that the probative value of the scientific evidence wasn’t always properly communicated to juries. And as you all know, it’s crucial we put this type of evidence in its proper context, given that laypeople can misunderstand the science.

I'll resist the obvious puns about a root cause analysis for hair testimony, but I wonder whether it pays to fund a major study of the culture that produced the overstated hair testimony.  Aren't the solutions to overstated testimony — ultracrepidarianism, as I have called it 2/ — fairly obvious? They are (1) better education and training of criminalists about the limits of their knowledge; (2) clear standards specifying what testimony is permissible; (3) better education and training of prosecutors and defenders about the statements that they as well as the criminalists can make; and (4) comprehensive and reasonably frequent review, not just of expert testimony, but also of the questions and opening and closing arguments of prosecutors, to ensure compliance with testimonial standards.

To address this problem, the FBI is close to finalizing new internal standards for testimony and reporting—which they’re calling “Approved Scientific Standards for Testimony and Reports,” or ASSTR. These documents, designed for almost all forensic disciplines currently practiced by the FBI, will clearly define what statements are supported by existing science. This will guide our lab examiners when they draft reports and take the witness stand, thereby reducing the risk of testimonial overstatement.

That is welcome news about the FBI. More broadly, OSAC needs to develop similar standards, and the Department of Justice should provide better training for its prosecutors. Experts are not the only sources of misstatements at trials. Both prosecutors and defense lawyers can misstate the content or implications of expert testimony. It happens all the time with DNA random-match probabilities, and in the infamous Santae Tribble case, it was the Assistant US Attorney, and not the expert witness, who told the jury that "[t]here is one chance, perhaps for all we know, in 10 million that it could [be] someone else’s hair."3/

While the FBI is preparing an ASSTR for each discipline, it’s fair to say that the risk of overstatement can vary depending on the discipline. The risk is arguably the lowest in certain types of disciplines, such as those involving chemical analysis. In drug testing, for example, the current technology makes it possible for experts to determine the chemical composition of a controlled substance with a high degree of certainty and with very little human interpretation.

"Arguably" is a key word here. "[D]isciplines ... involving chemical analysis" have the potential for suppressing uncertainty. Take a look at the ASTM E2937-13 Standard Guide for Using Infrared Spectroscopy in Forensic Paint Examinations. This document presupposes that a major task of the chemical analyst is "to determine whether any significant differences exist between the known and questioned samples." It defines a "significant difference" as "a difference between two samples that indicates that the two samples do not have a common origin." The forensic chemist is expected to declare whether "[s]pectra are dissimilar," "indistinguishable," or "inconclusive." The standard offers no guidance on how to explain the significance of spectra that are "indistinguishable." This is precisely the problem that hair analysts faced. Moreover, just as there is subjectivity in deciding whether hairs are indistinguishable, the forensic chemistry standard offers only a self-described "rule of thumb." This rule proposes "that the positions of corresponding peaks in two or more spectra be within ± 5 cm^-1," but "[f]or sharp absorption peaks one should use tighter constraints. One should critically scrutinize the spectra being compared if corresponding peaks vary by more than 5 cm^-1. Replicate collected spectra may be necessary to determine reproducibility of absorption position."

What is the nature of the "critical scrutiny" that permits an examiner to classify peaks that differ by more than 1 / (5 cm) as "similar"? When are any replicates necessary? How many are necessary? What is the accuracy of examiners who follow this open-ended "rule of thumb"? Perusing such standards suggests that forensic chemistry may not be so radically different from the other disciplines in which the risk of ultracrepidarianism is seen as more acute.

But, as you all know, the degree of certainty may be more difficult to quantify in other forensic disciplines. For example, a relatively small number of disciplines call on forensic professionals to compare two items—such as shoe prints or tire treads—and make judgments about their similarities and differences. These so-called “pattern” or “impression” disciplines present unique challenges, especially when an examiner attempts to assess the likelihood that the two items came from the same source.

As the paint standard exemplifies, forensic professionals compare two items in many fields. The spectra used in forensic chemistry are patterns. DNA profiles are patterns. Even some of the software that is supposed to give objectively established probabilities of the components of a DNA mixture have parameters that analysts can adjust as they see fit. Perhaps the line between the quantified-degree-of-certainty fields and the difficult-to-quantify ones is not entirely congruent with a simple divide between pattern-and-impression evidence and other forensic fields.

In any business, whether it’s medicine or manufacturing, it is standard practice to regularly review your internal procedures to make sure you’re performing at the highest level possible. Our DOJ labs do this all the time, and we plan to do it here, too. The department intends to conduct a quality assurance review of other forensic science disciplines practiced at the FBI—to determine whether the same kind of “testimonial overstatement” we found during our review of microscopic hair evidence could have crept into other disciplines that rely heavily on human interpretation and where the degree of certainty can be difficult to quantify. We’re thinking of it as a forensics “stress test.”

This sounds great, but how is "quality assurance review" a "stress test"? In cardiology, a stress test "determines the amount of stress that your heart can manage before developing either an abnormal rhythm or evidence of ischemia (not enough blood flow to the heart muscle)." 4/ In the banking system, stress testing examines whether banks have "sufficient capital to continue operations throughout times of economic and financial stress and that they have robust, forward-looking capital-planning processes that account for their unique risks." 5/ Checking whether "you're performing at the highest level possible" in providing testimony in the ordinary course of affairs is not a stress test for the FBI or anybody else (although I suppose it could prove to be stressful).

Of course, whether one should call the planned review a "stress test" is purely a matter of terminology. A much more important question is which disciplines will be reviewed. It would be unfortunate if the only recipients of review are the criminalists who examine  patterns and impressions in the form of toolmarks, shoeprints, handwriting, and so on. As we have just seen, it is not so easy to specify all the "disciplines that rely heavily on human interpretation and where the degree of certainty can be difficult to quantify."

This is an important moment in forensic science. The rise of new technologies presents both tremendous opportunities and potential challenges. At the same time, we must grapple with some of the most basic questions that lie at the intersection of science and the law: how do we make complex scientific principles understandable for judges, attorneys and jurors? How do we accurately communicate to laypeople the many things that forensic science can teach us—ensuring that we neither overstate the strength of our evidence nor understate the value of this information?

There are no easy answers ...

Amen to that!

Notes

1. Office of the Deputy Attorney General, Justice News: Deputy Attorney General Sally Q. Yates Delivers Remarks During the 68th Annual Scientific Meeting Hosted by the American Academy of Forensic Science, Feb. 24, 2016, http://www.justice.gov/opa/speech/deputy-attorney-general-sally-q-yates-delivers-remarks-during-68th-annual-scientific.
2. David H. Kaye, Ultracrepidarianism in Forensic Science: The Hair Evidence Debacle, 72 Wash. & Lee L. Rev. Online 227 (2015).
3. David H. Kaye, The FBI's Worst Hair Days, Forensic Science, Statistics and the Law, July 31, 2014.
4. WebMD, Heart Disease and Stress Tests, http://www.webmd.com/heart-disease/guide/stress-test
5. Board of Governors of the Federal Reserve System, Stress Tests and Capital Planning, http://www.federalreserve.gov/bankinforeg/stress-tests-capital-planning.htm

More on Task Relevance in Forensic Tests

Yesterday, I suggested that the National Commission on Forensic Science's views on task relevance are a significant step forward, and I elaborated on the use of conditional independence in determining which information is task relevant. The NCFS position is simple -- the examiner "should rely solely on task-relevant information when performing forensic analyses."

As the Commission explained, excluding task-irrelevant information avoids subtle but possible biases. 1/ However, what if the potentially biasing information could improve the accuracy of the analyst's conclusions? Statisticians often use biased estimators because they have greater precision -- they tend to give estimates that are closer to the true value with limited data -- even though these estimates tend to lie consistently on one side of that value. Moreover, even if the bias from the task-irrelevant information would increase the risk of an incorrect conclusion, what if it would be very costly to keep it out of the examination process? One might argue that the NCFS view is too stringent.

This challenge to the simple rule of no reliance is not persuasive. First, it is rather theoretical. Reasonably cheap methods to blind analysts to biasing, task-irrelevant information are generally available. The NCFS document explains how they can work.

Second, the conclusions that are likely to be more accurate are not those that the analyst should be drawing. At least with identification evidence in the courtroom, the expert should explain the strength of the scientific evidence, leaving the conclusion as to the identity of the true source to the judge or jury to decide based on all the evidence in the case. The NCFS views document adopts this philosophy most clearly in the last sentence of the appendix, which reads "[a]ny inferences analysts might draw from the task-irrelevant information involve matters beyond their scientific expertise that are more appropriately considered by others in the justice system, such as police, prosecutors, and jurors."

But it is not just a matter of relative expertise that should limit the analyst to task-relevant information. Forensic scientists are supposed to be conveying scientific information, and if the putative scientific judgment comes from a mixture of scientific and other information, the judge or jury cannot properly evaluate its weight without knowing what is the scientific part and what is some other part. 

Information contamination also makes it difficult to discern the validity of scientific tests. Consider hair-morphology evidence. I have presented the Houck-Budowle study of the correspondence between microscopic hair examinations and mitochondrial DNA tests as evidence that the former has some modest probative value (as measured by the likelihood ratio for positive associations). 2/ But inasmuch as the examiners were not blinded to task-irrelevant information, it is hard to tell from this one study how much of the probative value comes from the features of the hair and how much comes from other information that the hair examiners might have considered.

Studies of polygraphic lie detection offer another example. The technique sounds scientific, and the graphs of physiological responses look technical. But if the examiners' conclusions used in a validation study are influenced by impressions of the subject, the study does not reveal the diagnostic value of just the information in the tracings  -- the impact of that information and the subjective impressions are confounded. (This problem can be avoided by computerized scoring of the data.)

As the NCFS appendix emphasizes, the task-irrelevant information "does not help the analyst draw conclusions from the physical evidence that has been designated for examination through correct application of an accepted analytic method." At the risk of oversimplifying a complex subject, the message is that forensic scientists should stick to the scientific information.

Even this precept is not a complete response to concerns about bias. What if the task-relevant information also poses a serious risk of bias? If the contribution to the scientific analysis is minor and the risk of distortion is great, should not the examiner be blinded to this concededly task-relevant information? NCFS expressed no view on this situation. Perhaps it never arises, but if it does, standard-setting organizations should deal with it.

Notes

1. The NCFS observes that "there are risks entailed in exposing examiners unnecessarily to task-irrelevant information." But if the information is truly task-irrelevant, why would it be necessary? And if such information exists, would not the same risk of biasing the analysis be present?
   
2. David H. Kaye, Ultracrepidarianism in Forensic Science: The Hair Evidence Debacle, 72 Wash. & Lee L. Rev. Online 227 (2015); Disentangling Two Issues in the Hair Evidence Debacle, Forensic Sci., Stat. & L., Aug. 22, 2015, http://for-sci-law.blogspot.com/2015/08/disentangling-two-issues-in-hair.html.

Hair Evidence in the “Clearly Not Exonerated” Exoneration of Mark Reid

On November 3, the fictional forensic scientist in the world’s most watched television drama was aghast that she had once performed microscopic hair comparisons. Having learned that such comparisons are entirely discredited, NCIS’s Abby Sciuto is horrified: “Can you imagine if I messed up, what that really means? It means that innocent people went to jail because of me, because of my mistakes.” As a writer for Entertainment Weekly wrote, “Abby’s spinning out of control, locked in her lab and reexamining every single case she’s ever touched.” 1/

The “16 Years” episode is fiction, but real people have gone to jail for longer than that — and some could have been executed — because of mistakes by examiners. One disturbing hair-comparison case is State v. Reid. 2/ I have cited the Connecticut Supreme Court's opinion in two publications that survey different ways to testify about the implications of similarities between trace evidence and samples from known sources (such as the defendant). 3/ In doing so, I was not expressing the slightest agreement with the supreme court’s reasoning or arguing that the court was correct to hold that the trial judge properly admitted the testimony. But the case does illustrate how a careful criminalist called upon to testify could proceed in the face of gaping scientific uncertainty about the significance of similarities in the trace material and the known samples. It also illustrates the different sorts of errors than can occur in ascertaining hair morphology and drawing inferences from it as well as the different types of exonerations that can occur with the benefit of DNA testing.

I. The Criminalist Gives “Features Only” Testimony in Reid

It seems hard to deny that various physical features of hair display at least some variation within a population.  But without extensive population data that might permit at least rough estimates of the relative frequencies of the features, and without employing methods that have demonstrated reliability in measuring the features of interest, it is not clear how, or even if, this information should be used in trials.

One possibility is to limit the testimony to a presentation of the observed features (and perhaps a characterization of the samples' features as similar or different, as the case may be). According to the Connecticut Supreme Court, the analyst in Reid pursued this “features only” approach:

[He] displayed an enlarged photograph of one of the defendant's hairs and one of the hairs recovered from the victim's clothing as they appeared side-by-side under the comparison microscope. [He] explained to the jurors how the hairs were similar and what particular features of the hairs were visible. He also drew a diagram of a hair on a courtroom blackboard for the jurors. The jurors were free to make their own determinations as to the weight they would accord the expert's testimony in the light of the photograph and their own powers of observation and comparison.

The trial court had held a pretrial hearing to decide whether this testimony satisfied the preliminary showing of scientific validity normally required of all suitably challenged scientific evidence. The court found that it did, but the supreme court did not rely on or discuss either the scientific validity or the general scientific acceptance of visual hair comparisons. It avoided the issue by holding that the testimony did not have to satisfy such standards — because it was not “scientific evidence” at all. Rather, the expert “testified about a subject that simply required the jurors to use their own powers of observation and comparison.”

Three years later, in 2003, the Superior Court granted a petition for a new trial. 4/ Its opinion casts doubt on the no-science theory. Elaborating on the supreme court’s description of the testimony, this court observed that at the trial, the expert “indicated that hair comparison analysis ... is generally accepted as reliable within the field of forensic science” and “that he could state, ‘to a reasonable degree of scientific certainty,’ that the pubic hairs found on the victim's clothing were microscopically similar to those pubic hair samples taken from Mark Reid.” On such a record, the supreme court’s conclusion that the usual standards for scientific evidence are beside the point is hard to swallow.

II. The Superior Court Orders a New Trial While Insisting that the DNA Evidence Does Not Exonerate Reid

In any event, at Reid’s trial

Mr. Settachatgul testified that the three hairs recovered from the victim's clothing were pubic hairs. These hairs were rootless, indicating that they were shed, not plucked; one was found on the victim's jeans, another on a sock, and another on her lower undergarment (panty). ... Based on the microscopic analysis, Mr. Settachatgul's conclusion was that the three rootless hairs recovered from M.'s clothing were Negroid pubic hairs which had similar characteristics to the pubic hairs supplied by petitioner.

As the State’s Attorney explained in her summation, “the hairs ... appeared to be the same color, both had an abundance of fuci, and both exhibited the shadow of twisting, indicative of pubic hair.”  However, she acknowledged that “statistics are not done in the comparison field, [and] the only conclusion that can be drawn is similar or dissimilar characteristics, not the percentage of the population which shares those hair characteristics.” She told the jury “this is not conclusive evidence. I agree with the defense to that degree. It is not conclusive. But it is supportive of the victim's I.D.”

This support collapsed when postconviction mitochondrial DNA testing established that the three public hairs came from the same individual or from individuals in the same maternal lineage. Critically, Reid was not in that maternal line, while the victim, a white woman, was. This left two major possibilities: either the unknown rapist was the source of the three hairs or the victim was. The former scenario totally exculpates Reid; the latter renders the “supportive” expert testimony inconclusive.

Reid argued that the mtDNA test proved his actual innocence — that the criminalist’s determination of race had to be true, that the victim was white, and therefore the true rapist must have been some other black man who deposited three hairs on the victim’s clothes. The court did not buy this argument. And for good reason. Because the mtDNA testing showed that the victim’s mtDNA sequences matched those of the three hairs, the most plausible conclusion is that the criminalist erred in finding “Negroid pubic hairs.” The hairs probably were the victim’s rather than any rapist’s.

Thus, nothing was left of to connect Reid to the rape except the victim’s identification of him. The Superior Court concluded:

This is a close, difficult case. The new mtDNA evidence merely excludes petitioner as the depositor of the unknown hairs; it clearly does not exonerate him. And, as stated, the victim, M., was certain and steadfast in her identification of Mark Reid, and the circumstances surrounding that identification support its reliability, at least when viewed absent the newly discovered mtDNA evidence.

There are reasons to question this rosy picture of the eyewitness testimony, but whatever one thinks of that identification, the Superior Court found that Reid was entitled to a new trial at which he could use the mtDNA evidence to devastate a major part of the state’s case — the hair testimony.

III. The End of the Story Leaves Questions Hanging

In the end, there was no retrial. According the University of Michigan Law School’s National Registry of Exonerations, the state dismissed the charges “after the victim declined to participate ... . Reid, who had other felony convictions, was deported to his native Jamaica. In 2004, Reid filed a lawsuit seeking $2 million in damages from East Hartford.” The Registry does not report the outcome of that action. 5/

Whether or not all microscopic hair testimony is scientifically invalid because the comparisons have not been shown to be scientifically reliable, there is agreement that criminalists frequently have erred by using the similarity between hairs to make strong or quantified statements about the source of the trace hairs. 6/ This type of overclaiming apparently did not occur in Reid. In granting a new trial, the Superior Court emphasized that “Mr. Settachatgul was testifying only to the very ‘narrow opinion’ that the three pubic hairs recovered from the victim's clothing were similar to the samples obtained from the defendant, and, that he, Settachatgul could not say that the questioned specimens were the pubic hairs of petitioner/defendant.”

This testimony would not be judged as scientifically invalid under the FBI’s guidelines for reviewing microscopic-hair-comparison testimony. 7/ Likewise, the prosecutor’s summation was not infected with the sort of egregious overstatements, such as “There is one chance, perhaps for all we know, in 10 million that it could [be] someone else’s hair,” heard in other cases. 8/

The hair analyst may have erred in concluding that the features were similar. But mtDNA testing cannot tell us that. The small DNA molecules in the mitochondria do not relate to hair morphology. They provide a complementary — and more specific — test for identity. Sequence differences can exclude suspects when the inherently less discriminating visible features cannot. Standing alone, this limitation does not make microscopic hair analysis scientifically invalid, and it does not mean that the analyst misjudged the visual features here. But it does underscore the need to estimate the likelihoods or conditional error rates for microscopic hair comparisons. Without this information, how can anyone know what weight to give to the criminalist’s findings of similar hairs?

Finally, given the mitochondrial results for the victim and the pubic hairs in Reid, the hair analyst probably erred in concluding that the three pubic hairs were of “Negroid origin.” As noted earlier, the woman who was attacked in Reid was white, and the simplest conclusion is that the hairs on her clothes were hers, as the mtDNA sequences suggest. Furthermore, the unqualified assurance as to the racial origin of the hairs was unjustified — even if the classification turned out to be correct. Skimming a few forensic science textbooks, I can find no reference to publications in the scientific literature to support the position that hair analysts can make firm determinations of biogeographic ancestry. Caution is usually advised. 9/

Notes

1. Sara Netzley, “16 Years,” Entertainment, http://www.ew.com/recap/ncis-season-13-episode-7/2
2. 757 A.2d 482 (Conn. 2000). The case is noted in J.M. Taupin, Forensic Hair Morphology Comparison— A Dying Art or Junk Science?, 44 Sci. & Justice 95 (2004).
3. David H. Kaye, David E. Bernstein & Jennifer L. Mnookin, The New Wigmore: A Treatise on Evidence: Expert Evidence (2d ed. 2011); David H. Kaye, Presenting Forensic Identification Findings: The Current Situation, in Communicating the Results of Forensic Science Examinations 12–30 (C. Neumann et al. eds. 2015) (Final Technical Report for NIST Award 70NANB12H014).
4. Reid v. State, No. CV020818851, 2003 WL 21235422 (Ct. Super. Ct. May 14, 2003).
5. Maurice Possley, Mark Reid, The National Registry of Exonerations.
6. David H. Kaye, Ultracrepidarianism in Forensic Science: The Hair Evidence Debacle, 72 Wash. & Lee L. Rev. Online 227 (2015)
7. Id.
8. Spencer S. Hsu, Santae Tribble Cleared in 1978 Murder Based on DNA Hair Test, Dec.14, 2012 (quoting from federal prosecutor David Stanley’s closing argument).
9. Max M. Houck & Jay A. Siegel, Fundamentals of Forensic Science 303 (2015) (“Estimating the ethnicity or ancestry of an individual from his or her hairs is just that: an estimate.”); Richard Saferstein, Forensic Science: An Introduction 419 (2d ed. 2011) (“all of these observations are general, with many possible exceptions. The criminalist must approach the determination of race from hair with caution and a good deal of experience.”).

Acknowledgment: Thanks to Chris Fabricant for thoughts on State v. Reid and for pointing me to the full history of the case.

Can Forensic Pattern Matching Be Validated?

An article in the latest issue of the International Statistical Review raises (once again) fundamental questions for forensic scientists: 1/ How can one establish the validity of human judgment in a pattern recognition task such as deciding whether two samples of fingerprints or handwriting emanate from the same source? How can one estimate error probabilities for these judgments?

The message I get reading between the lines is that convincing validation is barely possible and the subjective assessments that are today’s norm will have to be replaced by objective measurements and statistical decision rules. This conclusion may not sit well with practicing criminalists who are committed to the current mode of skill-based assessments. At the same time, the particular statistical perspective of the article (null hypothesis testing) stands in opposition to a movement in the academic segment of the forensic science world that importunes criminalists to get away from categorical judgments — whether these judgments are subjective or objective. Nevertheless, the author of the article on Statistical Issues in Assessing Forensic Evidence,  Karen Kafadar, is a leading figure in forensic statistics, 2/ and her perspective is traditional among statisticians. Thus, an examination of a few parts of the article seems appropriate.

The article focuses on “forensic evidence that involves patterns (latent fingerprints, firearms and toolmarks, handwriting, tire treads and microscopic hair),” often comparing it to DNA evidence (much as the NRC Committee on Identifying the Needs of the Forensic Science Community did in 2009). Professor Kafadar emphasizes that in the pattern-matching fields, analysts do not make quantitative measurements of a pre-specified number of well-defined, discrete features like the short tandem repeat (STR) alleles that now dominate forensic DNA testing. Instead, the analyses “depend to a large extent on the examiner whose past experience enables some qualitative assessment of the distinctiveness of the features.” In other words, human perception and judgment establish how similar the two feature sets are and how discriminating those feature sets are. Such “pattern evidence is ... subjective and in need of quantitative validation.”

I. Validating Expert Judgments

How, then, can one quantitatively validate the subjective process? The article proceeds to “define measures used in quantifying error probabilities and how they can be used for pattern evidence.”

A. Validity

The first measure is

Validity (accuracy): Given a sample piece of evidence on which a measurement is made, is the measurement accurate? That is, if the measurement is ‘angle of bifurcation’ or ‘number of matching features’, does that measurement yield the correct answer? For example, if a bifurcation appears on an image with an angle of 30°, does the measurement technology render a result of ‘30’ [degrees], at least on average if several measurements are made? As another example, if a hair diameter is 153 μm, will the measurement, or average of several measurements, indicate ‘153’?

This is only a rough definition. Suppose a measuring instrument always gives a value of 30.001 when the angle is actually 30. Are the measurements “valid”? Neither individually nor on average is the instrument entirely accurate. But the measurements always are close, so maybe they do qualify as valid. There are degrees of validity, and a common measure of validity in this example would be the root mean squared error, where an error is a difference between the true angle and the measurement of it.

But fingerprint analysts do not measure alignments in degrees. Their comparisons are more like that of a person asked to hold two objects, one in each hand, and say which one is heavier (or whether the weights are practically the same). Experiments can validate the ability of test subjects to discriminate the masses under various conditions. If the subjects rarely err, their qualitative, comparative, subjective judgments could be considered valid.

Of course, there is no specific point at which accuracy suddenly merits the accolade of “valid,” and it can take more than one statistic to measure the degree of validity. For example, two forensic scientists, Max Houck and Jay Siegel, interpret a study of the outcomes of the microscopic comparisons and mitochondrial DNA testing as establishing a 91% “accuracy,” 3/ where “accuracy” is the overall “proportion of true results.” 4/ Yet, the microscopic hair analysts associated a questioned hair with a known sample in 1/5 of the cases in which DNA testing excluded any such association (and 1/3 in cases in which there was a DNA exclusion and a definitive result from the microscopy). 5/ In dealing with binary classifications, “validity (accuracy)” may require attention to more than being correct “on average.”

Moreover, whether the validity statistic derived from one experiment applies more widely is almost always open to debate. Even if one group of “weight analysts” always successfully discriminated between the heavier and the lighter weights, the question of generalizability or “external validity,” as social scientists often call it, would remain. A rigorous, double-blind study might show that the analysts did superbly with one set of weights under particular, controlled conditions. This study would possess high internal validity. But it might not tell us much about the performance of different analysts under different conditions; its external validity might be weak. Indeed, it has been said that “[i]t is axiomatic in social science research that there is an inverse relationship between internal and external validity.” 6/

Plainly, the quick definition of “validity” in Statistical Assessments does not exhaust the subject. (Nor was it intended to.) Things get even more complicated when we think of validity as relating to the purpose of the measurement. The data from a polygraph instrument may be valid measurements of physiological characteristics but not valid measures of conscious deception. The usual idea of validity is that the instrument (human or machine) accurately measures what it is supposed to measure. This aspect of “validity” is closely related to the requirement of “fit” announced in the Supreme Court's majority opinion in Daubert v. Merrell Dow Pharmaceuticals, Inc. 7/

B. Consistency

The article indicates that it takes more than “validity” to validate a measurement or inference process. The second requirement is

Consistency (reliability): Given the same sample, how consistent (or variable) are the results? If the measurement is repeated under different conditions (e.g. different fingers, different examiners, different analysis times, different measurement systems and different levels of quality in evidence), is the measurement the same? ... Under what conditions are the measurements most variable? That is, do measurements vary most with different levels of latent print quality? Or with different fingers of the same person? Or with different times of day for the same examiner? Or with different automated fingerprint identification systems (AFIS)? Or with different examiners? If measurements are found to be most consistent when the latent print quality is high and when AFIS system type A is used, but results vary greatly among examiners when the latent print quality is low or when other AFIS systems are used, then one would be in a good position to recommend the restriction of this particular type of forensic evidence under only those conditions when consistency can be assured. ... Notice that a measurement can be highly consistent around the wrong answer (consistent but inaccurate). ...

The critical definitional point here is that “reliability” concerns consistency, but there is room for argument over whether the measuring process has to be consistent under all conditions to be considered “reliable.” If one automated system is consistent in a given domain, it is reliable in that domain. If one skilled examiner reaches consistent results, her reliability is high even if inter-examiner reliability is low. In these examples, the notion of “reliability” overlaps or blurs into the idea of external validity. Likewise, all our weight analysts might be very reliable when comparing 10 pound weights to 20 pound ones but quite unreliable in distinguishing between with 15 and 16-pound ones. This would not prove that subjective judgments are ipso facto unreliable — only that reliability is less for more difficult tasks than for easy ones.

These ruminations on terminology do not undercut the important message in Statistical Assessments that research that teases out the conditions under which reliability and validity are degraded is vital to avoiding unnecessary errors: “[M]any observational studies are needed to confirm the performance of latent print analysis under a wide array of scenarios, examiners and laboratories.”

C. Well-determined Error Probabilities

The final component of validation described in Statistical Assessments is "well-determined error probabilities." When it comes to the classification task (differentiating same-source from different-source specimens), the error probabilities indicate whether the classifications are valid. A highly specific test has relatively few false positives — when confronted with different-source specimens, examiners conclude that they do not match. A highly sensitive test has relatively few false negatives — when confronted with same-source specimens, examiners conclude that they do match.

Tests that are both sensitive and specific also can be described as generating results that have a high “likelihood ratio.” If a perceived positive association is much more probable when the specimens truly are associated, and a negative association (an exclusion) is much more probable when they are not, then the likelihood ratio LR has a large numerator (close to the maximum probability of 1) and a small denominator (close to the minimum of 0):

LR = Pr(test + | association) / Pr(test + | no association)
      = specificity / 1 – Pr(test – | no association)
      = specificity / (1 – sensitivity)
      = large (almost 1) / small (a little more than 0)
      = very large

But a high likelihood ratio does not guarantee a high probability of a true association. It signals high “probative value,” to use the legal phrase, because it justifies a substantial change in probability that the suspected source is the real source compared to that probability without the evidence. For example, if the odds of an association without knowledge of the evidence are 1 to 10,000 and the examiner’s perception is 1,000 times more probable if the specimens are from the same source (LR = 1000), then, by Bayes' rule, the odds given the evidence rise to 1000 × 1:10000 = 1:10. Odds of only 1 to 10 cannot justify a conviction or even a conclusion that the two specimens probably are associated, but the examiner’s evidence has contributed greatly to the case. With other evidence, guilt may be established; without the forensic-science evidence, the totality of the evidence may fall far short. Therefore, if the well-defined error probabilities are low (and, hence, the likelihood ratio is high), it would be a mistake to dismiss the examiner’s assessment as lacking in value.

Yet, the standard terminology of positive and negative “predictive value” used in Statistical Assessments suggests that much more than this is required for the evidence to have “value.” For example, the article states that

In the courtroom, one does not have the ‘true’ answer; one has only the results of the forensic analysis. The question for the jury to decide is as follows: Given the results of the analysis, what is the probability that the condition is present or absent? For fingerprint analysis, one might phrase this question as follows:

PPV = P{same source | analysis claims ‘same source’}.

If PPV is high, and if the test result indicates ‘same source’, then we have some reasonable confidence that the two prints really did come from the same person. But if PPV is low, then, despite the test result (‘same source’), there may be an unacceptably high chance that in fact the prints did not come from the same person—that is, we have made a serious ‘type I error’ in claiming a ‘match’ when in fact the prints came from different persons.

Yes, the fingerprint analyst who asserts that the defendant is certainly the source when the PPV is low is likely to have falsely rejected the hypothesis that the defendant is not the source. But why must fingerprint examiners make these categorical judgments? Their job is to supply probative evidence to the judge or jury so as to permit the factfinder to reach the best conclusion based on the totality of the evidence in the case. 8/ If experiments have shown that examiners like the one in question, operating under comparable conditions with comparable prints, almost always report that the prints come from the same source when they do (high sensitivity) and that they do not come from the same source when they do not (high specificity), then there is no error in reporting that the prints in question are substantially more likely to have various features in common if they came from the same finger than if they came from fingers from two different individuals. 9/ This is a correct statement about the weight of the evidence rather than the probability of the hypothesis.

Indeed, one can imagine expanding the range of evaluative conclusions that fingerprint examiners might give. Instead of thinking “it’s either an identification or an exclusion” (for simplicity, I am ignoring judgments of “insufficient” and “inconclusive”), the examiner might be trained to offer judgments on a scale for the likelihood ratio, as European forensic science institutes have proposed. 10/ A large number of clear and unusual corresponding features in the latent print and the exemplar should generate a large subjective probability for the numerator of LR and a small probability for the denominator. A smaller number of such features should generate a smaller subjective ratio.

Although this mode of reporting on the evidentiary weight of the features is more nuanced and supplies more information to the factfinder, it would increase the difficulty of validating the judgments. How could one be confident that the moderate-likelihood-ratio judgments correspond to less powerful evidence than the high-likelihood-ratio ones?

II. Validating an Objective Statistical Rule

Statistical Assessments does not seriously consider the possibility of moving from categorical decisions on source attribution to a weight-of-evidence system. Instead, it presents a schematic for validating source attributions in which quantitative measurement replaces subjective impressions of the nature and implications of the degree of similarity in the feature sets. The proposal is to devise an empirically grounded null hypothesis test for objective measurements. Development of the test would proceed as follows (using 95% as an example):

    (1) Identify a metric (or set of metrics) that describes the essential features of the data. For example, these metrics might consist of the numbers of certain types of features (minutiae) or the differences between the two prints in the (i) average distances between the features (e.g. between ridges or bifurcations), (ii) eccentricities of identified loops or (iii) other characteristics on the prints that could be measured.
    (2) Determine a range on the metric(s) that is ‘likely to occur’ (has a 95% chance of occurring) if ‘nothing interesting is happening’ (i.e. the two prints do not arise from the same source). For example, one could calculate these metrics on 10,000 randomly selected latent prints known to have come from different sources.
    (3) Identify ‘extreme range’ = range of the metric(s) outside of the ‘95%’ range. For example, one can calculate ranges in which 95% of the 10,000 values of each metric lie.
    (4) Conduct the experiment and calculate the metric(s). For example, from the ‘best match’ that is identified, one can calculate the relevant metrics.
    (5) If the metric falls in the ‘expected’ range, then data are deemed consistent with the hypothesis that ‘nothing interesting is happening’. If the metric falls in the ‘extreme’ range, the data are not consistent with this hypothesis and indicate instead an alternative hypothesis.

This type of approach keeps the risk of a false rejection of the null hypothesis (that the suspect is not the source) to no more than 5% (ignoring the complications arising from the fact that not one but many variables are being considered separately), but it is subject to well-known criticisms. First, why 5%? Why not 1%? Or 9.3%?

Second, whatever the level of the test, does it make sense to report an association when the measurements barely make it into the “extreme range” but not when they are barely shy of it?

Third, what is the risk of a false acceptance — a false exclusion of a truly matching print? To estimate that error probability, a different distribution would need to be considered — the distribution of the measured values of the features when sampling from the same finger. The 2009 NRC Report refers this issue. In a somewhat garbled passage on the postulated uniqueness of fingerprints, 11/ it observes that

Uniqueness and persistence are necessary conditions for friction ridge identification to be feasible, but [u]niqueness does not guarantee that prints from two different people are always sufficiently different that they cannot be confused, or that two impressions made by the same finger will also be sufficiently similar to be discerned as coming from the same source. The impression left by a given finger will differ every time, because of inevitable variations in pressure, which change the degree of contact between each part of the ridge structure and the impression medium. None of these variabilities — of features across a population of fingers or of repeated impressions left by the same finger — has been characterized, quantified, or compared. 12/

To rest assured that both error probabilities of the statistical test of the quantified feature set are comfortably low, the same-finger experiment also needs to be conducted. Of course, Dr. Kafadar might readily concede the need to consider the alternative hypothesis (that prints originated from the same finger) and say that replicate measurements from the same fingers should be part of the experimental validation of the more objective latent-print examination process (or that in ordinary casework, examiners should make replicate measurements for each suspect). 13/

Still, the question remains: What if a similarity score on the crime-scene latent print and the ten-print exemplar falls outside the 95% range of variability of prints from different individuals and outside the 95% range for replicate latent prints from the same individual? Which hypothesis is left standing — the null (different source) or the alternative (same source)? One could say that the fingerprint evidence is inconclusive in this case, but would it be better to report a likelihood ratio in all cases rather than worrying about the tail-end probabilities in any of them? (This LR would not depend on whether the different-source hypothesis is rejected. It would increase more smoothly with an increasing similarity score.)

III. Human Expertise and Statistical Criteria

A major appeal of objectively ascertained similarity scores and a fixed cut-off is that the system supplies consistent results with quantified error probabilities and reliability. But would the more objective process be any more accurate than subjective, human judgment in forensic pattern recognition tasks? The objective measures that might emerge are likely to be more limited than the many features that the human pattern matchers might evaluate. And, it can be argued that the statistical evaluation of them may not be as sensitive to unusual circumstances or subtleties as individual “clinical” examination would be.

Thus, the preference in Statistical Assessments for objectively ascertained similarity scores and a fixed cut-off is reminiscent of the arguments for “actuarial” or “statistical” rather than “clinical” assessments in psychology and medicine. 14/ The work in those fields of expertise raises serious doubt about claims of superior decisionmaking from expert human examiners. Nonetheless, more direct data on this issue can be gathered. Along with the research Dr. Kafadar proposes, studies of whether the statistical system outperforms the classical, clinical one in the forensic science fields should be undertaken. The burden of proof should be on the advocates of purely clinical judgments. Unless the less transparent and less easily validated human judgments clearly outperform the algorithmic approaches, they should give way to more objective measurements and interpretations.

Notes

1. Karen Kafadar, Statistical Issues in Assessing Forensic Evidence, 83 Int’l Stat. Rev. 111–34 (2015).

2. Dr. Kafadar is Commonwealth Professor and chair of the statistics department at the University of Virginia, a member of the Forensic Science Standards Board of the Organization of Scientific Area Committees of the National Institute of Standards and Technology, and a leading participant in the newly established “Forensic Science Center of Excellence focused on pattern and digital evidence” — “a partnership that includes Carnegie Mellon University (Pittsburgh, Penn.), the University of Virginia (Charlottesville, Va.) and the University of California, Irvine (Irvine, Calif.) [that] will focus on improving the statistical foundation for fingerprint, firearm, toolmark, dental and other pattern evidence analyses, and for computer, video, audio and other digital evidence analyses.” New NIST Center of Excellence  to Improve Statistical Analysis of Forensic Evidence, NIST Tech Beat, May 26, 2015.

2. As Dr. Kafadar has observed, “[s]tatistics plays multiple roles in moving forensic science forward, in characterizing forensic analyses and their underlying bases, designing experiments and analyzing relevant data that can lead to reduced error rates and increased accuracy, and communicating the results in the courtroom.” U.Va. Partners in New Effort to Improve Statistical Analysis of Forensic Evidence, UVAToday, June 2, 2015.

3. Max M. Houck & Jay A. Siegel, Fundamentals of Forensic Science 310 (2015).

4. Id.

5. David H. Kaye, Ultracrepidarianism in Forensic Science: The Hair Evidence Debacle, 72 Wash. & Lee L. Rev. Online 227 (2015).

6. E.g., Allan Steckler & Kenneth R. McLeroy, The Importance of External Validity, 98 Am. J. Public Health 9 (2008).

7. See David H. Kaye et al., The New Wigmore: A Treatise on Evidence: Expert Evidence (2d ed. 2011).

8.  As Sir Ronald Fisher reminded his follow statisticians, “We have the duty of formulating, of summarizing, and of communicating our conclusions, in intelligible form, in recognition of the right of other free minds to utilize them in making their own decisions.” Ronald A. Fisher Statistical Methods and Scientific Induction, 17 J. Roy. Statist. Soc. B 69 (1955).

9. Yet, Statistical Assessments insists that by virtue of Bayes’ rule, “low prevalence, high sensitivity and high specificity are needed for high PPV and NPV ... [there is a] need for sensible restriction of the suspect population.” This terminology is confusing. Low prevalence (guilt is rare) comes with a large suspect population rather than a restricted one. It cuts against a high PPV. Conversely, if “low prevalence” means a small suspect population (innocence is relatively rare), then it is harder to have a high NPV.

10. ENFSI Guideline for Evaluative Reporting in Forensic Science, June 9, 2015.

11. The assertion below that “[u]niqueness and persistence are necessary conditions for friction ridge identification to be feasible” ignores the value of a probabilistic identification. A declared match can be immensely probative even if a print is not unique in the population. If a particular print occurred twice in the world’s population, a match to the suspect still would be powerful evidence of identification. DNA evidence is like that — the possibility of a genetically identical twin somewhere has not greatly undermined the feasibility DNA identifications. The correspondence in the feature set still makes the source probability higher than it was prior to learning of the DNA match. The matching alleles need not make the probability equal to 1 to constitute a useful identification.

12. NRC Committee on Identifying the Needs of the Forensic Science Community, Strengthening Forensic Science in the United States: A Path Forward 144 (2009)(footnote omitted).

13. Statistical Assessments states that fingerprint analysts currently compare “latent prints found at a [crime scene] with those from a database of ‘latent exemplars’ taken under controlled conditions.” Does this mean that latent print examiners create an ad hoc databank in each case of a suspect’s latent prints to gain a sense of the variability of those prints? I had always thought that examiners merely compare a given latent print to exemplars of full prints from suspects (what used to be called “rolled prints”). In the same vein, giving DNA profiling as an example, Statistical Assessments asserts that “[a]nalysis of the evidence generally proceeds by comparing it with specimens in a database.” However, even if CODIS database trawls have become routine, the existence and use of a large database has little to do with the validity of the side-by-side comparisons that typify fingerprint, bullet, handwriting, and hair analyses.

14. See, e.g., R.M. Dawes et al., Clinical Versus Actuarial Judgment, 243 Science 1668 (1989) (“Research comparing these two approaches shows the actuarial method to be superior.”); William M. Grove, & Paul E. Meehl, Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The Clinical–statistical controversy, 2 Psych., Pub. Pol’y & L. 293 (1996) (“Empirical comparisons of the accuracy of the two methods (136 studies over a wide range of predictands) show that the mechanical method is almost invariably equal to or superior to the clinical method); Konstantinos V. Katsikopoulos et al., From Meehl to Fast and Frugal Heuristics (and Back): New Insights into How to Bridge the Clinical—Actuarial Divide, 18 Theory & Psych. 443 (2008); Steven Schwartz & Timothy Griffin, Medical Thinking: The Psychology of Medical Judgment and Decision Making (2012).

Acknowledgement

Thanks are due to Barry Scheck for calling the article discussed here to my attention.

Disentangling Two Issues in the Hair Evidence Debacle

Forensic-science practitioners commonly present findings regarding traces left at crime-scenes, on victims or suspects, or on or in their possessions. Such trace evidence can take many forms. Physical traces such as fingerprints, striations on bullets, shoe and tire prints, and handwritten documents are common examples. Biological materials, such as blood, semen, saliva, and hairs also are fodder for the crime laboratory. Comparisons of a questioned and known sample can supply valuable information on whether a specific suspect is associated in some manner with a crime. Viewers of the acronymious police procedurals—NCIS, CSI, and Law and Order SVU—know all this.

For decades, however, legal and other academics have questioned the hoary courtroom claims of absolutely certain identification of one and only one possible source of trace evidence. In the turbulent wake of the 2009 report of a National Research Council committee, these views have slowly gained traction in the forensic-science community. Indeed in the popular press and among investigative reporters, the pendulum may be swinging in favor of uncritical rejection of once unquestioned forensic sciences. Recent months have seen an episode from Frontline presenting DNA evidence as "anything but proven"; ^1/ they have included unfounded reports that as many as 15 percent of men and women imprisoned with the help of DNA evidence at trial are wrongfully convicted; ^2/ and award-winning journalists have spread the word that the FBI "faked an entire field of forensic science," ^3/ placed "pseudoscience in the witness box," ^4/ and palmed off "virtually worthless" evidence as scientific truth. ^5/

The last set of reports stem from an ongoing review of well over 20,000 cases in which the FBI laboratory issued reports on hair associations. The review spans decades of hair comparisons, and it is showing so many questionable statements that the expert evidence on hair associations stands out as "one of the country's largest forensic scandals." ^6/ Its preliminary findings provoked prominent Senators to speak of an "appalling and chilling ... indictment of our criminal justice system" ^7/ and to call for a "root cause analysis" of ubiquitous errors. ^8/ A distressed Department of Justice and FBI joined with the Innocence Project and the National Association of Defense Lawyers not only to publicize these failings, but also to call on states "to conduct their own independent reviews where ... examiners were trained by the FBI." ^9/ Projecting the outcome of cases that have yet to be reviewed, postconviction petitions refer ominously to "[t]housands of . . . cases the Justice Department now recognizes were infected by false expert hair analysis" ^10/ and "pseudoscientific nonsense." ^11/

The hair scandal illustrates two related problems with many types of forensic-science testimony. The first is the problem of foundation—What reasons are there to believe that hair or other analysts possess sufficient expertise to produce relevant evidence of associations between known and unknown samples? For years, commentators and some defense counsel have posed legitimate questions (with little impact in the courts) about the reliability and validity of physical comparisons by examiners asked to judge whether known and unknown samples are similar in enough respects—and not too dissimilar in other respects—to support a claim that they could have originated from the same individual. To paraphrase Gertrude Stein, is there enough there there to warrant any form of testimony about a positive association? This is the existential question of whether, in the words in of the Court in Daubert v. Merrell Dow Pharmaceuticals, "[t]he subject of an expert's testimony [is] 'scientific . . . knowledge.'" ^12/ Or, at the other extreme, is the entire enterprise ersatz—a "fake science" and a "worthless" endeavor?

As I see it, the harsh view that physical hair comparisons are pure pseudoscience, like astrology, graphology, homeopathy, or metoposcopy, is not supportable. The FBI's review project itself rests on the premise that hair evidence has some value. If the comparisons were worthless—like consulting the configurations of the stars or reading Tarot cards—there would be no need to review individual cases. In all cases of an association, the FBI would have exceeded the limits of science. But this only shows that the FBI thinks that there is a basis for some testimony of a positive association. What evidence supports this belief?

One reason to think that the FBI's hair analysts generally possess some expertise in making associations comes from an intriguing study (usually cited as proof of the failings of microscopic hair comparisons) done more than ten years ago. ^13/ FBI researchers took human hairs submitted to the FBI laboratory for analysis between 1996 and 2000 and mitotyped them (a form of DNA testing) whenever possible. The probability of an FBI examiner finding of a positive association when comparing hairs from the same individual (as shown by mitotyping) exceeded the probability of this finding when comparing hairs from different individuals by a factor of 2.9 (with a 95% confidence interval of 1.7 to 4.9). A test with this performance level only supplies evidence that is, on average, weakly diagnostic of an association. Still, it is not simply invalid. 

The second problem lies in the presentation of perceived associations. Even if there is a there there, are forensic-science practitioners staying within the boundaries of their demonstrated expertise? Or are they purporting to know more than they do? The FBI's revelations about hair evidence are confined to this issue of overclaiming. What the FBI has uncovered are expert assertions in one case after another that are said to outstrip the core of demonstrated knowledge. Such overclaiming is one form of scientifically invalid testimony, ^14/ but it is not the equivalent of an entire invalid science.

In considering the pervasiveness of the problem of overclaiming, the FBI's figure of 90+ percent is startling. It is so startling that one should ask whether it accurately estimates the prevalence of scientifically indefensible testimony. There are reasons to suspect that it might not. After all, the Hair Comparison Review Project was not designed to estimate the proportion of cases in which FBI examiners gave testimony that was, on balance, scientifically invalid. It is intended to spot isolated statements that claimed more than an association that, for all we know, could be quite common in the general population. The general criteria for judging whether particular testimony falls into this category have been publicized, but the protocol and specific criteria that the FBI reviewers are using have not been revealed. No representative sample of the reports or transcripts that are judged to be problematic is available, but a few transcripts in the cases in which the FBI has confessed scientific error indicate that at least some classifications are open to serious question.

It may be instructive to contrast the response to two very similar statements noted in a posting of May 23, 2015, about the court-martial of Jeffrey MacDonald: (1) "this hair ... microscopically matched the head hairs of Colette MacDonald"; and (2) "[a] forcibly removed Caucasian head hair ... exhibits the same microscopic characteristics as hairs in the K2 specimen. Accordingly, this hair is consistent with having originated from Kimberly MacDonald, the identified source of the K2 specimen." The first statement passed muster. The second did not.

If the review in this case is not aberrant, examiners can say that two hairs share the same features—they "match" and are consistent with one another—but they must not add the obvious (and scientifically undeniable) fact that this observation (if correct) means that they could have had the same origin or that they are "consistent with" this possibility.

Of course, one can criticize phrases like "consistent with" and "match" as creating an unacceptable risk that (in the absence of clarification on direct examination, cross-examination, or by judicial instruction) jurors will think the words connote a source attribution. But arguments of this sort stray from determinations that an examiner has made statements that "exceed the limits of science" (the phrase the Justice Department uses in confessing overclaiming). They represent judgments that an examiner has made statements that are scientifically acceptable but prone to being misunderstood.

To be sure, this latter danger is important to the law. It should inform rulings of admissibility under Rules of Evidence 403 and 702. It is a reason to regulate the manner in which experts testify to scientifically acceptable findings, as some courts have done. Laboratories themselves should adopt and enforce policies to ensure that reports and testimony avoid terminology that is known to convey the wrong impression. But it is misleading to include scientifically acceptable but psychologically dangerous phrasing in the counts of scientifically erroneous statements. Case-review projects ought to flag all instances in which examiners have not presented their findings as they should have, but reports ought to differentiate between statements that directly "exceed the limits of science" and those that risk being misconstrued in a way that would make them "exceed the limits of science." One size does not fit all.

NOTES

This posting is an abridged and modified version of a forthcoming essay about the FBI's Microscopic Hair Review Comparison Review. A full draft of the preliminary version that is being edited for publication is available. Comments and corrections are welcome, especially before publication while there is time to improve the essay.

1. Some inaccuracies in the documentary are noted in a June 24, 2015, posting on this blog.
2. See the posting of July 3, 2015 on the blog (debunking this initial assertion of the Rand Corporation).
3. Dahlia Lithwick, Pseudoscience in the Witness Box: The FBI Faked an Entire Field of Forensic Science, Slate (Apr. 22, 2015 5:09 PM).
4. Id. These "shameful, horrifying errors" comprised "a story so horrifying . . . that it would stop your breath." Id.
5. Erin Blakemore, FBI Admits Pseudoscientific Hair Analysis Used in Hundreds of Cases: Nearly 3,000 Cases Included Testimony About Hair Matches, a Technique that Has Been Debunked, Smartnews (Apr. 22, 2015) (quoting Ed Pilkington, Thirty Years In Jail For A Single Hair: The FBI's 'Mass Disaster' of False Conviction, Guardian, Apr. 21, 2015.
6. Spencer S. Hsu, FBI Admits Flaws in Hair Analysis over Decades, Wash. Post, Apr. 18, 2015.
7. Id. (quoting "Sen. Richard Blumenthal (D-Conn.), a former prosecutor").
8. Spencer S. Hsu, FBI Overstated Forensic Hair Matches in Nearly All Trials Before 2000, Wash. Post, Apr. 19, 2015 (quoting "Senate Judiciary Committee Chairman Charles E. Grassley (R-Iowa) and the panel's ranking Democrat, Patrick J. Leahy (Vt.)").
9. FBI, FBI Testimony on Microscopic Hair Analysis Contained Errors in at Least 90 Percent of Cases in Ongoing Review (Apr. 20, 2015).
10. Petition for a Writ of Certiorari, at 2-3, Ferguson v. Steele, 134 S.Ct. 1581 (2014) (No. 13-1069).
11. Id. at 19. Justice Breyer saw the “errors” referred to in the press release as emblematic of “flawed forensic testimony” generally and a reason to hold the death penalty unconstitutional. Glossip v. Gross, No. 14-7955 (June 29, 2015) (dissenting opinion).
12. 509 U.S. 579, 590 (1993).
13. Max M. Houck & Bruce Budowle, Correlation of Microscopic and Mitochondrial DNA Hair Comparisons, 47 J. Forensic Sci. 1 (2002).
14. For this vocabulary, see Brandon Garrett & Peter Neufeld, Invalid Forensic Science Testimony and Wrongful Convictions, 95 Va. L. Rev. 1 (2009); cf. Eric S. Lander, Fix the Flaws in Forensic Science, N.Y. Times, Apr. 21, 2015 (“The F.B.I. stunned the legal community on Monday with its acknowledgment that testimony by its forensic scientists about hair identification was scientifically indefensible in nearly every one of more than 250 cases reviewed.”).