Peering into Peer Review

After the Supreme Court, in Daubert v. Merrell Dow Pharmaceuticals, listed publication in peer reviewed scientific journals as an important factor in ascertaining whether a scientific method produces results that can be admitted in court, the term "peer review" became commonplace in opinions on scientific evidence.

But it takes more than a bland statement that publications appear in a peer reviewed journal to show that a scientific discipline treats the claims in those publications as worthy of respect. For example, some experts, like celebrity doctor Andrew Weil, tout publication in the Journal of Clinical Ecology as indicative of credible scientific findings. But courts, following the advice of the broader medical community, do not. It is not enough to have a Society for Clinical Ecology (reinvented as the American Academy of Environmental Medicine) publish a journal reviewed by peers who are true believers. Rather, "[i]t is the publication of ... basic research, using accepted research techniques, providing scientific (as opposed to anecdotal) evidence ... to which the Daubert inquiry is directed." ^1/

In this spirit, the National Commission on Forensic Science recently emphasized that “[s]cientific literature comprises manuscripts that report empirical data and have been independently peer-reviewed for quality, originality, and relevance to the discipline. To strengthen confidence in results obtained in forensic examinations, each forensic discipline must identify resources that are scientifically credible, valid and with a clear scientific foundation. Such foundational literature in forensic practice should conform to norms across all scientific disciplines.”

In this light, consider the opinion of the U.S. District Court for the Eastern District of New York in United States v. Ashburn. ^2/ In response to a motion to exclude testimony from "Detective Salvatore LaCova ... that all of the cartridge casings and deformed bullets ... were fired from [a particular] gun," the court ran through the usual checklist of Daubert factors. It quickly found "that the AFTE [Association of Firearm and Tool Mark Examiners] methodology has been published and subject to peer review, weighing in favor of admission of LaCova's testimony." This positive endorsement followed entirely from the fact that "[t]he AFTE itself publishes within the field of toolmark and firearms identification." It is not clear the court knew anything more about the journal than the remarks in earlier district court cases that "articles submitted to the AFTE Journal are subject to peer review" and that the "AFTE Journal [has a] formal process for the submission of articles." ^3/

This "formal process" apparently consists of an editorial board of firearms and tool mark examiners who screen articles and “post-publication review by the members of the Association of Firearm & Tool Mark Examiners” whose unsolicited reactions, appear in the ‘AFTE Peer Review and Letters to the Editor’ section of the Journal.” ^4/ Surely, postpublication review limited to the organization’s members, does not meet the Commission's call for publications that “conform to norms across all scientific disciplines.”

Information in journals that exist at the fringes or even outside the corpus of scientific literature can be quite valuable. But there is no excuse for courts to continue to rely on the existence such publications as proof of scientific validity. After all, the nature of the AFTE Journal is no secret to the legal profession. One prominent law review article explained that:

Another major limitation of the current forensic science culture relates to several of the publication venues for the pattern identification field. Several of the most significant journals focused on publishing pattern identification research simply do not comport with broader norms of access, dissemination, or peer review typically associated with scientific publishing. For example, the AFTE Journal, a quarterly publication of the Association of Firearm and Toolmark Examiners, has published numerous articles on firearms identification. WorldCat—the largest online catalog of library materials, which includes the holdings of 72,000 libraries worldwide, including virtually every university-based library in the United States—lists only eighteen libraries with a copy of this journal in their holdings. Furthermore, the AFTE Journal does not appear to be indexed or included in any major indexing service anywhere. The only available index to AFTE was created by an individual firearms examiner on his own initiative and was not continued past 2005. Moreover, peer review of submissions to AFTE is not blind; the author and the reviewer are both aware of each other’s identity. In addition, the peer reviewers appear to come entirely from the editorial board, which consists entirely of AFTE members, and therefore includes no members from outside the toolmark and firearms practitioner community. This journal therefore appears to have extremely limited dissemination beyond the members of AFTE itself; completely lacks integration with any of the voluminous networks for the production and exchange of scientific research information; and engages in peer review that is neither blind nor draws upon an extensive network of researchers. None of this is compatible with an accessible, rigorous, transparent culture of research. ^5/

This does not mean that no scientific literature on "the AFTE method" in which Detective LaCova was trained exists. Journals that meet the criteria listed by the National Commission have published articles on various aspects of the process for matching striations and the inferences that can be drawn from them. It is this literature that courts interested in "publications and peer review" should consult before they make up their minds.

Notes

1. Gabbard v. Linn-Benton Housing Authority, 219 F.Supp.2d 1130, 1137 (D. Or. 2002).
2. No. 11–CR–0303 (NGG), 2015 WL 739928 (E.D.N.Y. Feb. 20, 2015).
3. See also, e.g., United States v. Otero, 849 F.Supp.2d 425, 433 (D.N.J. 2012) (“AFTE theory is subject to peer review through submission to and publication by the AFTE Journal of validation studies which test the theory.”); Commonwealth v. Pytou Heang, 942 N.E.2d 927, 939 n.20 (Mass. 2011) (“The Association of Firearm and Toolmark Examiners (AFTE) is an organization of firearm and toolmark examiners that publishes the peer-reviewed AFTE Journal.”).
4. The journal's webpage, from which these quotations are taken, does not list the institutional affiliations of its editorial board members.
5. Jennifer L. Mnookin et al., The Need for a Research Culture in the Forensic Sciences, 58 UCLA L. Rev. 725, 754-56 (2011) (notes omitted).