A Two-culture Problem with Forensic Science?

Recently, U.S. Court of Appeals Judge Harry T. Edwards complained that

Forensic practitioners are the people who got us in trouble in the first place. They don't know what they don't know. ... The people who are doing this do not understand what we mean when we say to them, "what you're doing has no scientific foundation." They don't understand it because they were brought up in a different world. They don't understand science. 1/

That is a harsh and sweeping generalization. Judge Edwards, who co-chaired a committee empaneled by the National Research Council to study forensic science in the United States, knows that the forensic-science community is far from monolithic. 2/

But I fear that, in part and in some instances, the frustration expressed by practitioners with the criticism that the validity and reliability of pattern-matching practices such as fingerprint, toolmark, footwear, handwriting, and bitemark comparisons have yet to be sufficiently demonstrated does reflect a lack of appreciation for what it takes to validate a process of measurement and inferences in these fields.

An example is the reaction of the president of the International Association for Identification (IAI) to a draft recommendation of a subcommittee of the U.S. National Commission on Forensic Science. 3/ The document, which received the endorsement of 60% of the members of the Commission 4/ (including, I would guess, all of the nonforensic scientists on the panel), suggested that

Forensic science practitioners should not state that a specific individual or object is the source of the forensic science evidence and should make it clear that, even in circumstances involving extremely strong statistical evidence, it is possible that other individuals or objects could possess or have left a similar set of observed features. Forensic science practitioners should confine their evaluative statements to the support that the findings provide for the claim linked to the forensic evidence.

In other words, practitioners should evaluate the probability of the observed correspondence in the features of the specimens they examine if the specimens have a common origin and the probability of these observations if the specimens are from different sources. But they should not take it upon themselves to opine on the ultimate issue of who is the source.

Although advocacy of this approach is hardly novel in the international forensic-science community, the president of "the foremost international organization" 5/ of practitioners was incensed at the thought that latent fingerprint examiners should cease and desist from "conclusion decisions" 6/ in court. In an interview for Forensic Magazine, he said that it was just unfair. 7/ He fulminated (or elaborated):

"Even if all the minutiae all match up, you're telling me I can't say it came from the same source?" Ruslander said. "There are millions of fingerprints in AFIS, and there's never been a bad match, to my knowledge," he added. "That's a pretty good empirical study." 8/

Anyone competent in scientific methodology would have to call this kind of study fundamentally misconceived. One could design a good experiment to ascertain the validity of an automated fingerprint matcher, but it would not consist of one person’s memory of no “bad matches” — whatever that means for a system that merely produces a list of possibly matching candidates rather than single-source “conclusion decision.” Moreover, even if a perfectly accurate, fully automated system to make single-source attributions existed, what would its uncanny performance tell us about the validity and reliability of mere mortals who do not make their “conclusion decisions” the same way and who are known to err from time to time?

Now there are studies that clearly demonstrate that latent print examiners can make source attributions and exclusions at rates far better than chance — in other words, there is scientifically demonstrable expertise even if the procedure is highly subjective and not particularly “scientific” at critical junctures. But remarks like those of the leader of “the world’s oldest and largest forensic science identification association” 9/ as to what is a “good empirical study” only lend credence to Judge Edwards’ complaint. They make it appear that practitioners “were brought up in a different world” and “don't understand science.” The forensic-science community can and must do better.

NOTES

1. "Our Worst Fears Have Been Realized" -- Forensic "Evidence, Science, and Reason in an Era of 'Post-truth' Politics" (Part 1), Forensic Science, Statistics and Law, Nov. 20, 2017, http://for-sci-law.blogspot.com/2017/11/our-worst-fears-have-been-realized.html 
2. His committee’s 2009 NRC Report noted that

   the “forensic science community” ... consists of a host of practitioners, including scientists (some with advanced degrees) in the fields of chemistry, biochemistry, biology, and medicine; laboratory technicians; crime scene investigators; and law enforcement officers. There are very important differences, however, between forensic laboratory work and crime scene investigations. There are also sharp distinctions between forensic practitioners who have been trained in chemistry, biochemistry, biology, and medicine (and who bring these disciplines to bear in their work) and technicians who lend support to forensic science enterprises. (P. 7)

3. In the interest of full disclosure, I should note that I participated in drafting the "views document" that contains this recommendation and that I made suggestions to the Commission for further revisions.
4. Transcript of Meeting 13, Part 1, Apr. 10, 2017,.at 48, https://www.nist.gov/topics/forensic-science/ncfs-meeting-13-webcast
5. United States v. Herrera, 704 F.3d 480, 486 (7th Cir. 2013).
6. H.W. “Rus” Ruslander, Feb. 5,  2017, IAI Position Statement on Conclusions, Qualified Statements, and Probability Modeling, https://www.theiai.org/president/20170205_IAI_PositionStatementConclusionsQualified%20OpinionsProbability%20Modeling.pdf
7. Seth Augenstein, National Commission on Forensic Science Asks for Public Comment, Forensic Mag., Feb. 22, 2017, http://www.forensicmag.com/news/2017/02/national-commission-forensic-science-asks-public-comment
8. Augenstein, supra note 4.
9. IAI’s Mission Statement, https://www.theiai.org/about/index.php