Peering into Peer Review

After the Supreme Court, in Daubert v. Merrell Dow Pharmaceuticals, listed publication in peer reviewed scientific journals as an important factor in ascertaining whether a scientific method produces results that can be admitted in court, the term "peer review" became commonplace in opinions on scientific evidence.

But it takes more than a bland statement that publications appear in a peer reviewed journal to show that a scientific discipline treats the claims in those publications as worthy of respect. For example, some experts, like celebrity doctor Andrew Weil, tout publication in the Journal of Clinical Ecology as indicative of credible scientific findings. But courts, following the advice of the broader medical community, do not. It is not enough to have a Society for Clinical Ecology (reinvented as the American Academy of Environmental Medicine) publish a journal reviewed by peers who are true believers. Rather, "[i]t is the publication of ... basic research, using accepted research techniques, providing scientific (as opposed to anecdotal) evidence ... to which the Daubert inquiry is directed." ^1/

In this spirit, the National Commission on Forensic Science recently emphasized that “[s]cientific literature comprises manuscripts that report empirical data and have been independently peer-reviewed for quality, originality, and relevance to the discipline. To strengthen confidence in results obtained in forensic examinations, each forensic discipline must identify resources that are scientifically credible, valid and with a clear scientific foundation. Such foundational literature in forensic practice should conform to norms across all scientific disciplines.”

In this light, consider the opinion of the U.S. District Court for the Eastern District of New York in United States v. Ashburn. ^2/ In response to a motion to exclude testimony from "Detective Salvatore LaCova ... that all of the cartridge casings and deformed bullets ... were fired from [a particular] gun," the court ran through the usual checklist of Daubert factors. It quickly found "that the AFTE [Association of Firearm and Tool Mark Examiners] methodology has been published and subject to peer review, weighing in favor of admission of LaCova's testimony." This positive endorsement followed entirely from the fact that "[t]he AFTE itself publishes within the field of toolmark and firearms identification." It is not clear the court knew anything more about the journal than the remarks in earlier district court cases that "articles submitted to the AFTE Journal are subject to peer review" and that the "AFTE Journal [has a] formal process for the submission of articles." ^3/

This "formal process" apparently consists of an editorial board of firearms and tool mark examiners who screen articles and “post-publication review by the members of the Association of Firearm & Tool Mark Examiners” whose unsolicited reactions, appear in the ‘AFTE Peer Review and Letters to the Editor’ section of the Journal.” ^4/ Surely, postpublication review limited to the organization’s members, does not meet the Commission's call for publications that “conform to norms across all scientific disciplines.”

Information in journals that exist at the fringes or even outside the corpus of scientific literature can be quite valuable. But there is no excuse for courts to continue to rely on the existence such publications as proof of scientific validity. After all, the nature of the AFTE Journal is no secret to the legal profession. One prominent law review article explained that:

Another major limitation of the current forensic science culture relates to several of the publication venues for the pattern identification field. Several of the most significant journals focused on publishing pattern identification research simply do not comport with broader norms of access, dissemination, or peer review typically associated with scientific publishing. For example, the AFTE Journal, a quarterly publication of the Association of Firearm and Toolmark Examiners, has published numerous articles on firearms identification. WorldCat—the largest online catalog of library materials, which includes the holdings of 72,000 libraries worldwide, including virtually every university-based library in the United States—lists only eighteen libraries with a copy of this journal in their holdings. Furthermore, the AFTE Journal does not appear to be indexed or included in any major indexing service anywhere. The only available index to AFTE was created by an individual firearms examiner on his own initiative and was not continued past 2005. Moreover, peer review of submissions to AFTE is not blind; the author and the reviewer are both aware of each other’s identity. In addition, the peer reviewers appear to come entirely from the editorial board, which consists entirely of AFTE members, and therefore includes no members from outside the toolmark and firearms practitioner community. This journal therefore appears to have extremely limited dissemination beyond the members of AFTE itself; completely lacks integration with any of the voluminous networks for the production and exchange of scientific research information; and engages in peer review that is neither blind nor draws upon an extensive network of researchers. None of this is compatible with an accessible, rigorous, transparent culture of research. ^5/

This does not mean that no scientific literature on "the AFTE method" in which Detective LaCova was trained exists. Journals that meet the criteria listed by the National Commission have published articles on various aspects of the process for matching striations and the inferences that can be drawn from them. It is this literature that courts interested in "publications and peer review" should consult before they make up their minds.

Notes

1. Gabbard v. Linn-Benton Housing Authority, 219 F.Supp.2d 1130, 1137 (D. Or. 2002).
2. No. 11–CR–0303 (NGG), 2015 WL 739928 (E.D.N.Y. Feb. 20, 2015).
3. See also, e.g., United States v. Otero, 849 F.Supp.2d 425, 433 (D.N.J. 2012) (“AFTE theory is subject to peer review through submission to and publication by the AFTE Journal of validation studies which test the theory.”); Commonwealth v. Pytou Heang, 942 N.E.2d 927, 939 n.20 (Mass. 2011) (“The Association of Firearm and Toolmark Examiners (AFTE) is an organization of firearm and toolmark examiners that publishes the peer-reviewed AFTE Journal.”).
4. The journal's webpage, from which these quotations are taken, does not list the institutional affiliations of its editorial board members.
5. Jennifer L. Mnookin et al., The Need for a Research Culture in the Forensic Sciences, 58 UCLA L. Rev. 725, 754-56 (2011) (notes omitted).

Frontline's Expose of DNA Testing: Yes and No

The content of a recent Frontline story on “The Surprisingly Imperfect Science of DNA Testing: How a Proven Tool May Be Anything But” will come as no surprise to anyone familiar with the professional and academic literature on forensic DNA identification. The work is important, though, because the main points of the story need to be widely understood. I highlight them below. At the same time, the story relies on some putative problems that are more perceived than real. 

The story boils down to this (with annotations on the right)
Complex DNA mixtures can be tough to interpret, especially when the amounts of DNA are so small that stochastic effects in amplifying DNA sequences are important. In these situations, analysts using a variety of cues and procedures--and even following the same general procedure--can reach different conclusions. False arrests and convictions can follow.	True. See, e.g., Erin Murphy, The Art in the Science of DNA: A Layperson's Guide to the Subjectivity Inherent in Forensic DNA Typing, 58 Emory L.J. 489 (2008).
There can be a big difference between the answer to the question “What is the probability of a match between a specific DNA profile and the profile an individual picked at random?” and “What is the probability of a match between a specific DNA profile and at least one profile in a large database of profiles picked at random?”	Yes, different questions give different answers. The size of the database affects the answer to the second question, but not the first. But which question should be addressed in court? Probably neither. See, e.g., Ian Ayres & Barry Nalebuff, The Rule of Probabilities: A Practical Approach for Applying Bayes’ Rule to the Analysis of DNA Evidence, 67 Stan. L. Rev. 1447 (2015); David J. Balding, The DNA Database Search Controversy, 58 Biometrics 241 (2002); David H. Kaye, Rounding Up the Usual Suspects: A Legal and Logical Analysis of DNA Database Trawls, 87 N. Car. L. Rev. 425 (2009).
Thinking about how traces of DNA ended up where they did—and not just whose DNA ended up there—can be crucial to an investigation and prosecution. If investigators, lawyers, and jurors do not understand this, there can be mistaken arrests, prosecutions, and convictions.	A very important point. See, e.g., Peter Gill, Misleading DNA Evidence (2014); David H. Kaye, David Bernstein & Jennifer Mnookin, The New Wigmore on Evidence: Expert Evidence (2d ed. 2011).
A few things will be surprising
The 2011 Hampikian-Dror “experiment” with a complex DNA mixture was good proof that examiner bias (from knowing what detectives believed) affected two examiner's interpretations.	From the viewpoint of experimental design, a potentially important confounding variable was obviously present. See D.H. Kaye, The Design of “The First Experimental Study Exploring DNA Interpretation”, 52 Science & Justice 256 (2012). As Dror later wrote, the study was  “suggesting that the extraneous context of the criminal case may have influenced the interpretation of the DNA evidence” (emphasis added by Dror in a reply letter). This conclusion is reminiscent of a microscopic hair association reported as “suggesting that the [defendant] may have [left the hair at the crime scene].” How helpful is that?
"It’s not clear how often coincidental matches occur." Indeed, it might not be a rare event at all, considering that "a rogue Arizona state employee had run tests on the state’s database without the FBI’s permission and found" an inexplicably high number of partial matches.	The employee was not a “rogue”; she did not need the FBI’s permission to use a state database this way; the results were presented on behalf of the state laboratory at an International Symposium on Human Identification. They are not especially anomalous, but largely a consequence of trawling for partial matches among all possible pairs of profiles in a database that includes close relatives. See, e.g., David H. Kaye, Trawling DNA Databases for Partial Matches: What Is the FBI Afraid Of?, 19 Cornell J. L. & Public Pol'y 145 (2009).

48 Hours for DNA

Today's New York Times reports that

DNA matching that of two escaped killers was found in a cabin in the remote resort of Mountain View, N.Y., 15 heavily wooded miles west of the state prison in Dannemora, an official briefed on the investigation said on Monday morning.

The forensic evidence indicated that the men had been there within the last 48 hours, according to the official, who was not authorized to discuss the search and spoke on the condition of anonymity.

A pair of prison-issued underwear was also found in the cabin, the official said.

Andy Newman & William K. Rashbaum, DNA of Escaped Convicts Found in Cabin, Official Says, N.Y. Times, June 22, 2015 

It would interesting to know what "forensic evidence indicated that the men had been there within the last 48 hours." DNA itself carries no known signs of how long it has sat on some surface. If the DNA were in saliva on, say, an apple core left from a snack, would the extent of oxidation of the food allow it to be dated within a 48-hour period? Seventeen days have elapsed since the two men escaped.

Maryland v. King and Fourth Amendment Doctrine

In Maryland v. King,^1/ the Supreme Court upheld the practice of routine DNA sampling soon after arrest for certain crimes. The impact of the decision of the doctrinal framework for applying the Fourth Amendment to searches has been the subject of debate. The most extreme view is that is the opinion presages the collapse of the doctrine that criminal investigatory searches are per se unreasonable unless they fall within some well-defined exception to the requirement of a warrant based on probable cause.^2/ Another destabilizing view is that the case establishes “that any suspicionless search of an arrestee is allowed if it will be useful to solve crimes.” ^3/ Other observers were less alarmed.^4/

A case having nothing to do with DNA suggests that King is not the death knell of either the reasonable-suspicion requirement for a search incident to arrest or the warrant requirement. In Riley v. California,^5/ Chief Justice Roberts wrote that

As the text [of the Amendment] makes clear, “the ultimate touchstone of the Fourth Amendment is reasonableness." Our cases have determined that "[w]here a search is undertaken by law enforcement officials to discover evidence of criminal wrongdoing, ... reasonableness generally requires the obtaining of a judicial warrant." Such a warrant ensures that the inferences to support a search are "drawn by a neutral and detached magistrate instead of being judged by the officer engaged in the often competitive enterprise of ferreting out crime." In the absence of a warrant, a search is reasonable only if it falls within a specific exception to the warrant requirement. ^6/

Applying this framework, the Court unanimously held that police generally need a warrant to search a cellphone, The mere fact that it was acquired incident to an arrest is insufficient to justify rummaging though its contents.Thus, if Riley is any indication, Maryland v. King changed neither the basic framework of Fourth Amendment analysis nor the parameters of searches incident to arrest.

Notes

1. 133 S. Ct. 1958 (2013).
2. Erin Murphy, License, Registration, Cheek Swab: DNA Testing and the Divided Court, 127 Harv. L. Rev. 161 (2013).
3. Tracey Maclin, Maryland v King: Terry v Ohio Redux, 2013 Supreme Court Review 359, 403.
4. David H. Kaye, Why So Contrived? The Fourth Amendment and DNA Databases After Maryland v. King, 104 J. Crim. L. & Criminology 535 (2014); David H. Kaye, Maryland v. King Per Se Unreasonableness, the Golden Rule, and the Future of DNA Databases, 127 Harv. L. Rev. F. 39, 40, 42-43 (2013); Orin Kerr, A Few Thoughts on Maryland v. King, The Volokh Conspiracy, June 3, 2013 (“while King is very important from a practical standpoint, there isn’t a whole lot of academically-interesting stuff happening in the King opinions.”).
5. 134 S. Ct. 2473 (2014).
6. Id. at 248 (citations and internal quotation marks omitted)

Spitting in Syracuse: Another Disgusting DNA Case

Police have linked restaurant and grocery store employees to expectoration in food or drink. The latest case is described in Syracuse.com, which reports how a server at the Chili's Restaurant in Clay, NY, was caught after the act and convicted of disorderly conduct. Now the affected customers plan to sue him -- and the corporate owners of Chili's.

Earlier, and even more unpleasant cases, come from Seattle and Albuquerque.