In His Own Words: Justice Scalia's Poetry Slam

In discussing the exchange in Glossip v. Gross between Justices Breyer and Scalia on the deterrent effect of capital punishment, I alluded to Andrew Lang (1844-1912), Scottish poet, novelist, and literary critic. Although any further connection to science or statistics is nonexistent, I can't resist a poetic tangent. Fred Moss pointed me to an animated "poetry slam" of Justice Scalia's "snarky ... poetic gems" at the cartoonist, Mark Fiore's website. Rearranging bits of the concurrence in Glossip yields this additional free verse:

they have discovered the lost folios of Shakespeare
they ask us for clemency, as though clemency were ours to give
If only Aristotle, Aquinas, and Hume knew
that moral philosophy could be so neatly distilled
into a pocket-sized, vade mecum
full of internal contradictions and (it must be said)
gobbledy-gook Welcome to Groundhog Day.

Related posting:
Two Scientific Issues in Glossip v. Gross, July 14, 2015, Forensic Science, Statistics, and the Law

Two Scientific Issues in Glossip v. Gross

In Glossip v. Gross, the Supreme Court narrowly upheld a district court's refusal to issue a preliminary injunction against a three-drug protocol being used to kill capital offenders. The majority of five Justices, led by Justice Alito, was emphatic. Their opinion offered not one, but two "independent reasons." The second was that the district court's findings about the effect and dosage of the sedative midazolam were not clearly erroneous. Although the Court cautioned that "federal courts should not embroil themselves in ongoing scientific controversies beyond their expertise" (internal quotation marks omitted), the factual question that ultimately must be answered correctly is whether this sedative, administered to achieve deep unconsciousness, really blocks the pain caused by paralytic and heart-stopping agents. The opinion does not resolve this question, and even if the forgiving "clearly erroneous" standard was met in this case, Justice Sotomayor's opinion for the four dissenting Justices contains more than enough technical material to make one nervous about the conclusion that midazolam works as the state of Oklahoma hopes it does.

A second controversy involves social, not medical, science. Justice Breyer, joined by Justice Ginsburg), wanted the Court to ask for "full briefing on ... whether the death penalty violates the Constitution." He maintained that "the death penalty, in and of itself, now likely constitutes a legally prohibited 'cruel and unusual punishmen[t].'" One aspect of this tentative conclusion involved "the death penalty's deterrent effect" -- the subject of innumerable studies and two skeptical reports from the National Academy of Sciences.

Justice Scalia, joined by Justice Thomas, was upset at what he called the "speculat[ion] that it does not 'seem likely' that the death penalty has a 'significant' deterrent effect." His approach to this empirical question was quintessentially legal, not scientific -- pick the answer you want (or think you know) and look only for confirming evidence. Justice Scalia's argument that the existence of the death penalty (even if rarely used and long delayed) deters significantly more than life imprisonment does consists of a single sentence: "It seems very likely to me, and there are statistical studies that say so." What studies?

See, e.g., Zimmerman, State Executions, Deterrence, and the Incidence of Murder, 7 J. Applied Econ. 163, 166 (2004) (“[I]t is estimated that each state execution deters approximately fourteen murders per year on average”); Dezhbakhsh, Rubin, & Shepherd, Does Capital Punishment Have a Deterrent Effect? New Evidence from Postmoratorium Panel Data, 5 Am. L. & Econ. Rev. 344 (2003) (“[E]ach execution results, on average, in eighteen fewer murders” per year); Sunstein & Vermeule, Is Capital Punishment Morally Required? Acts, Omissions, and Life-Life Tradeoffs, 58 Stan. L. Rev. 703, 713 (2005) (“All in all, the recent evidence of a deterrent effect from capital punishment seems impressive, especially in light of its ‘apparent power and unanimity’”).

Justice Scalia would have done better to have followed the example of Andrew Lang, the novelist who tried "not to use statistics as a drunken man uses lamp-posts, for support rather than for illumination." 1/ Carelessly or cavalierly, Justice "Scalia cites a paper by Cass Sunstein for a second time, even though after the first such Scalia citation in an earlier lethal injection case, Cass Sunstein (writing with Justin Wolfers) affirmed his view that there is no credible evidence that the death penalty is a deterrent." 2/ As Professor John Donahue, who penned these words, added, "[o]ne would hope for more from a Supreme Court justice than citations to junk science and to a paper withdrawn based on more informed consideration – especially on a matter of life and death." 3/

NOTES

1. On the origin of this aphorism, see Quote Investigator, Jan. 14, 2015, http://quoteinvestigator.com/2014/01/15/stats-drunk/.
2. John J. Donohue, Glossip v. Gross: Examining Death Penalty Data for Clarity, Stanford Lawyer, June 29, 2015, https://stanfordlawyer.law.stanford.edu/2015/06/glossip-v-gross-examining-death-penalty-data-for-clarity/. Justice Breyer is both less dogmatic and more complete in his description of the body of social science research:

   Many studies have examined the death penalty's deterrent effect; some have found such an effect, whereas others have found a lack of evidence that it deters crime. [Citations omitted.] Recently, the National Research Council ... reviewed 30 years of empirical evidence and concluded that it was insufficient to establish a deterrent effect and thus should "not be used to inform" discussion about the deterrent value of the death penalty. National Research Council, Deterrence and the Death Penalty 2 (D. Nagin & J. Pepper eds. 2012).
   
   I recognize that a 'lack of evidence' for a proposition does not prove the contrary. [Citation omitted.] But suppose that we add to these studies the fact that, today, very few of those sentenced to death are actually executed, and that even those executions occur, on average, after nearly two decades on death row. ... Then, does it still seem likely that the death penalty has a significant deterrent effect?

3. Id.

DNA Evidence Causes Wrongful Convictions 15% of the Time!?

Today's Forensic Magazine reports that

Recent estimates indicate that as many as 15 of every 100 incarcerated offenders where DNA was an element in their trial may be wrongfully convicted because of misused DNA evidence matching techniques. One common reason for this error is scientifically invalid testimony on forensic evidence. [FULL STORY]

Did I read this correctly? Fifteen percent -- one out of every six or seven -- of the men and women in prison are innocent? That is almost four times the 4% rate estimated in Gross et al. (2013). But that study, the leading one in the field, was limited to defendants with death sentences. Their cases would have been scrutinized especially carefully, so the 4% figure is probably on the low side. Maybe 15% for the broader prison population is realistic.

But wait. Forensic Magazine was not referring to the entire prison population, but only to those cases that went to trial. And within that group, 15% is the figure for defendants for whom "DNA was an element in their trial[s]." Because almost no defendants introduce DNA evidence at trial, these must be cases in which the prosecution has linked the defendant to the crime by DNA testing and the state "misused DNA ... techniques." Can it be that 15% of inmates convicted because of DNA tests are falsely convicted? That is an intolerable error rate for what is supposed to be the gold standard in forensic science. Is it time to halt DNA testing until we can find out why it generates so much "scientifically invalid testimony"?

Or are the editors of Forensic Magazine seeking sensational news rather than reading what they are reporting? Let's look at the "recent estimates." To get to them requires a few steps backwards. Forensic Magazine lists the National Institute of Justice (NIJ) -- a part of the U.S. Department of Justice -- as the source of its story. NIJ funded the Rand Corporation to do a study on improving expert performance in computing a posterior probability (which is not what forensic experts routinely do). The words are Rand's rather than NIJ's. The study cautions that "[o]pinions or points of view expressed are those of the author(s) and do not necessarily reflect the official position or policies of the U.S. Department of Justice." Rand investigated two questions: "Is bias reduced when experts do not know whether the prosecution or defense is the hiring party?," and "Is bias reduced by expert consensus feedback, wherein expertise is culled from multiple sources and those sources examine the majority view to move toward a group consensus?"

The summary at the start of the Rand research report begins with the very two sentences that Forensic Magazine broadcast. The researchers gave no references to indicate where these "recent estimates" came from, but page one of the report supplies an apparent answer. It reads as follows:

As many as 15 of every 100 incarcerated offenders may be wrongfully convicted, according to DNA evidence–matching techniques (Roman et al., 2012). One reason for this is scientifically invalid testimony on forensic evidence (e.g., Gould et al., 2012; Innocence Project, 2013).

This passage reveals that Rand simply misrepresented the "recent estimates" in the first instance. Roman et al. (2012) is an NIJ-funded study from the Urban Institute that involved no cases of wrongful convictions because of misused DNA evidence. Instead, that study "analyzed the results of new DNA testing of old physical evidence from 634 sexual assault and homicide cases that took place in Virginia between 1973 and 1987." In other words, it looked at postconviction exonerations in cases in which the trials took place before DNA evidence was even available. This study of wrongful convictions for reasons other than faulty DNA evidence "found that in five percent of homicide and sexual assault cases DNA testing eliminated the convicted offender as the source of incriminating physical evidence. When sexual assault convictions were isolated, DNA testing eliminated between 8 and 15 percent of convicted offenders and supported exoneration."

Are there any cases of false convictions caused by DNA evidence? Certainly. Does the rate of such convictions approach 15%? Neither Forensic Magazine nor the Rand Corporation offers any reason to believe it.

Peering into Peer Review

After the Supreme Court, in Daubert v. Merrell Dow Pharmaceuticals, listed publication in peer reviewed scientific journals as an important factor in ascertaining whether a scientific method produces results that can be admitted in court, the term "peer review" became commonplace in opinions on scientific evidence.

But it takes more than a bland statement that publications appear in a peer reviewed journal to show that a scientific discipline treats the claims in those publications as worthy of respect. For example, some experts, like celebrity doctor Andrew Weil, tout publication in the Journal of Clinical Ecology as indicative of credible scientific findings. But courts, following the advice of the broader medical community, do not. It is not enough to have a Society for Clinical Ecology (reinvented as the American Academy of Environmental Medicine) publish a journal reviewed by peers who are true believers. Rather, "[i]t is the publication of ... basic research, using accepted research techniques, providing scientific (as opposed to anecdotal) evidence ... to which the Daubert inquiry is directed." ^1/

In this spirit, the National Commission on Forensic Science recently emphasized that “[s]cientific literature comprises manuscripts that report empirical data and have been independently peer-reviewed for quality, originality, and relevance to the discipline. To strengthen confidence in results obtained in forensic examinations, each forensic discipline must identify resources that are scientifically credible, valid and with a clear scientific foundation. Such foundational literature in forensic practice should conform to norms across all scientific disciplines.”

In this light, consider the opinion of the U.S. District Court for the Eastern District of New York in United States v. Ashburn. ^2/ In response to a motion to exclude testimony from "Detective Salvatore LaCova ... that all of the cartridge casings and deformed bullets ... were fired from [a particular] gun," the court ran through the usual checklist of Daubert factors. It quickly found "that the AFTE [Association of Firearm and Tool Mark Examiners] methodology has been published and subject to peer review, weighing in favor of admission of LaCova's testimony." This positive endorsement followed entirely from the fact that "[t]he AFTE itself publishes within the field of toolmark and firearms identification." It is not clear the court knew anything more about the journal than the remarks in earlier district court cases that "articles submitted to the AFTE Journal are subject to peer review" and that the "AFTE Journal [has a] formal process for the submission of articles." ^3/

This "formal process" apparently consists of an editorial board of firearms and tool mark examiners who screen articles and “post-publication review by the members of the Association of Firearm & Tool Mark Examiners” whose unsolicited reactions, appear in the ‘AFTE Peer Review and Letters to the Editor’ section of the Journal.” ^4/ Surely, postpublication review limited to the organization’s members, does not meet the Commission's call for publications that “conform to norms across all scientific disciplines.”

Information in journals that exist at the fringes or even outside the corpus of scientific literature can be quite valuable. But there is no excuse for courts to continue to rely on the existence such publications as proof of scientific validity. After all, the nature of the AFTE Journal is no secret to the legal profession. One prominent law review article explained that:

Another major limitation of the current forensic science culture relates to several of the publication venues for the pattern identification field. Several of the most significant journals focused on publishing pattern identification research simply do not comport with broader norms of access, dissemination, or peer review typically associated with scientific publishing. For example, the AFTE Journal, a quarterly publication of the Association of Firearm and Toolmark Examiners, has published numerous articles on firearms identification. WorldCat—the largest online catalog of library materials, which includes the holdings of 72,000 libraries worldwide, including virtually every university-based library in the United States—lists only eighteen libraries with a copy of this journal in their holdings. Furthermore, the AFTE Journal does not appear to be indexed or included in any major indexing service anywhere. The only available index to AFTE was created by an individual firearms examiner on his own initiative and was not continued past 2005. Moreover, peer review of submissions to AFTE is not blind; the author and the reviewer are both aware of each other’s identity. In addition, the peer reviewers appear to come entirely from the editorial board, which consists entirely of AFTE members, and therefore includes no members from outside the toolmark and firearms practitioner community. This journal therefore appears to have extremely limited dissemination beyond the members of AFTE itself; completely lacks integration with any of the voluminous networks for the production and exchange of scientific research information; and engages in peer review that is neither blind nor draws upon an extensive network of researchers. None of this is compatible with an accessible, rigorous, transparent culture of research. ^5/

This does not mean that no scientific literature on "the AFTE method" in which Detective LaCova was trained exists. Journals that meet the criteria listed by the National Commission have published articles on various aspects of the process for matching striations and the inferences that can be drawn from them. It is this literature that courts interested in "publications and peer review" should consult before they make up their minds.

Notes

1. Gabbard v. Linn-Benton Housing Authority, 219 F.Supp.2d 1130, 1137 (D. Or. 2002).
2. No. 11–CR–0303 (NGG), 2015 WL 739928 (E.D.N.Y. Feb. 20, 2015).
3. See also, e.g., United States v. Otero, 849 F.Supp.2d 425, 433 (D.N.J. 2012) (“AFTE theory is subject to peer review through submission to and publication by the AFTE Journal of validation studies which test the theory.”); Commonwealth v. Pytou Heang, 942 N.E.2d 927, 939 n.20 (Mass. 2011) (“The Association of Firearm and Toolmark Examiners (AFTE) is an organization of firearm and toolmark examiners that publishes the peer-reviewed AFTE Journal.”).
4. The journal's webpage, from which these quotations are taken, does not list the institutional affiliations of its editorial board members.
5. Jennifer L. Mnookin et al., The Need for a Research Culture in the Forensic Sciences, 58 UCLA L. Rev. 725, 754-56 (2011) (notes omitted).

Frontline's Expose of DNA Testing: Yes and No

The content of a recent Frontline story on “The Surprisingly Imperfect Science of DNA Testing: How a Proven Tool May Be Anything But” will come as no surprise to anyone familiar with the professional and academic literature on forensic DNA identification. The work is important, though, because the main points of the story need to be widely understood. I highlight them below. At the same time, the story relies on some putative problems that are more perceived than real. 

The story boils down to this (with annotations on the right)
Complex DNA mixtures can be tough to interpret, especially when the amounts of DNA are so small that stochastic effects in amplifying DNA sequences are important. In these situations, analysts using a variety of cues and procedures--and even following the same general procedure--can reach different conclusions. False arrests and convictions can follow.	True. See, e.g., Erin Murphy, The Art in the Science of DNA: A Layperson's Guide to the Subjectivity Inherent in Forensic DNA Typing, 58 Emory L.J. 489 (2008).
There can be a big difference between the answer to the question “What is the probability of a match between a specific DNA profile and the profile an individual picked at random?” and “What is the probability of a match between a specific DNA profile and at least one profile in a large database of profiles picked at random?”	Yes, different questions give different answers. The size of the database affects the answer to the second question, but not the first. But which question should be addressed in court? Probably neither. See, e.g., Ian Ayres & Barry Nalebuff, The Rule of Probabilities: A Practical Approach for Applying Bayes’ Rule to the Analysis of DNA Evidence, 67 Stan. L. Rev. 1447 (2015); David J. Balding, The DNA Database Search Controversy, 58 Biometrics 241 (2002); David H. Kaye, Rounding Up the Usual Suspects: A Legal and Logical Analysis of DNA Database Trawls, 87 N. Car. L. Rev. 425 (2009).
Thinking about how traces of DNA ended up where they did—and not just whose DNA ended up there—can be crucial to an investigation and prosecution. If investigators, lawyers, and jurors do not understand this, there can be mistaken arrests, prosecutions, and convictions.	A very important point. See, e.g., Peter Gill, Misleading DNA Evidence (2014); David H. Kaye, David Bernstein & Jennifer Mnookin, The New Wigmore on Evidence: Expert Evidence (2d ed. 2011).
A few things will be surprising
The 2011 Hampikian-Dror “experiment” with a complex DNA mixture was good proof that examiner bias (from knowing what detectives believed) affected two examiner's interpretations.	From the viewpoint of experimental design, a potentially important confounding variable was obviously present. See D.H. Kaye, The Design of “The First Experimental Study Exploring DNA Interpretation”, 52 Science & Justice 256 (2012). As Dror later wrote, the study was  “suggesting that the extraneous context of the criminal case may have influenced the interpretation of the DNA evidence” (emphasis added by Dror in a reply letter). This conclusion is reminiscent of a microscopic hair association reported as “suggesting that the [defendant] may have [left the hair at the crime scene].” How helpful is that?
"It’s not clear how often coincidental matches occur." Indeed, it might not be a rare event at all, considering that "a rogue Arizona state employee had run tests on the state’s database without the FBI’s permission and found" an inexplicably high number of partial matches.	The employee was not a “rogue”; she did not need the FBI’s permission to use a state database this way; the results were presented on behalf of the state laboratory at an International Symposium on Human Identification. They are not especially anomalous, but largely a consequence of trawling for partial matches among all possible pairs of profiles in a database that includes close relatives. See, e.g., David H. Kaye, Trawling DNA Databases for Partial Matches: What Is the FBI Afraid Of?, 19 Cornell J. L. & Public Pol'y 145 (2009).