What Proves that "the expert and his methods couldn’t possibly be reliable"?

I am developing an allergic reaction to the following kind of argument: "A forensic-science expert testified that a trace at the crime scene or on the victim was associated (to some degree of certainty) with the defendant. DNA later evidence exonerated the defendant. Therefore, the expert’s methods couldn’t possibly be reliable."

This reasoning is not very different from saying that a pitcher who does not strike out every batter couldn’t possibly be a reliable pitcher; that a polling firm that fails to correctly predict every election must be using methods that couldn’t possibly be reliable; or that a test for heart disease that sometimes errs couldn’t possibly be reliable. Without considering the success as well as the failure rate of the method, it is impossible to say that it is unreliable—or, by the same token, that it is reliable (in the sense of being worth relying on). 1/

Yet, the argument from cases of exonerations—lacking any comparison group—are legion in discourse on the use of trace evidence for identification. The latest example I encountered comes from a Washington Post blog site. Two days ago, Radley Balko wrote that

[A] defendant was convicted due to the testimony of a forensic expert who claimed that his “science” showed the defendant, and only the defendant, could have committed the crime. That conviction was later upheld by an appeals court in an opinion that explained in detail why the expert and his methods were legitimate and reliable. The defendant was later exonerated by DNA testing, thus demonstrating that the expert and his methods couldn’t possibly be reliable.

Mr. Balko went on to write that “not only did the courts continue to allow bite mark matching into evidence, every single time a defendant challenged its validity, that defendant lost.” 2/

Both the reasoning and the description of legal history are not quite right. To begin with, even absolute proof of innocence only shows that the test has a nonzero false-positive error rate (no surprise there) and that the witness should not have claimed to a certainty that no one else could have left the mark in question.

To be sure, some evidence that has found breathing space in the courtroom should be squeezed out entirely. But let’s face it—no scientific test meets the standard of perfection. If every case in which evidence that has produced false convictions meant “that the expert and his methods couldn’t possibly be reliable,” there could be no evidence. Overselling has occurred with every type of forensic evidence—from bitemarks to toolmarks to fingerprints to DNA. Courts, scientists, and criminalists should do their best to prevent this. Thus, whether through rules of evidence or through education and monitoring of analysts, testimony must be calibrated to the power of the scientific technique. The testimony should fairly express the known probative value of evidence from a validated method.

An example of testimony that violates this precept comes from Ege v. Yukins. 3/ In that case, a discredited dental expert testified as follows:

Q: Now, Doctor, with regard to your testimony, you indicated that it's highly consistent with the dentition of Defendant Carol Ege; is that correct?
A: Yes.
Q: Okay. With regard to—let me ask you a question. Let's say you have the Detroit Metropolitan Area, three, three and a half million people. Would anybody else within that kind of number match like she did?
A: No, in my expert opinion, nobody else would match up. 4/

The eventual outcome in the case contradicts the assertion that no challenge to the admission of bitemark evidence has succeeded. The state trial judge in Ege realized that the testimony was improper and only “denied [postconviction] relief because of the lack of a contemporaneous objection and a view that the showing of prejudice was insufficient.” 5/

A federal district court also concluded that “expert testimony identifying the petitioner as the only possible perpetrator of the alleged bite mark in the Detroit metropolitan area was improperly admitted.” 6/

The U.S. Court of Appeals for the Sixth Circuit agreed “with the district court that ‘Dr. Warnick's opinion that the petitioner was the only person in the entire Detroit metropolitan area who could have made the mark on the corpse carried an aura of mathematical precision pointing overwhelmingly to the statistical probability of guilt, when the evidence deserved no such credence.’” 7/ It affirmed the order for a new trial.

In short, the argument that a method of forensic identification that has been proved to be fallible is, for that reason alone, inadmissible proves too much. Likewise, the claim that no challenge to bitemark evidence has ever prevailed is exaggerated (although not by much). 8/

Please do not misunderstand me. The series of articles on bitemark evidence from which the remarks I have quoted were taken is impressive and useful. In offering these corrections to two small parts that seem a bit extreme, I am not arguing that bitemark analysis, which has little claim to validity, is either reliable (in the statistical sense that repeated analyses of the same marks give the same answers) or valid (in the sense that the answers are more often correct when marks from the same source are analyzed than when marks from different sources are compared). From the writing I have seen, bitemark analysis does not cut it.

I also believe that cases of false convictions should be studied and that the existence of a given type of scientific evidence in these cases should not be ignored. Finding a large number of false convictions with such evidence present is a warning signal. The evidence may come from a method that has a large false-positive rate, 9/ and that possibility must be investigated to decide whether the evidence should be excluded across the board or whether juries should receive the information -- together with an honest and clear explanation of the uncertainty in the results.

NOTES

1. See infra note 9.
2. Radley Balko, A High-ranking Obama Official Just Called for the “Eradication” of Bite Mark Evidence, The Watch, Wash. Post, July 22, 2015.
3. 485 F.3d 364 (6th Cir. 2007).
4. Ege v. Yukins, 380 F.Supp. 2d 852, 871 (E.D. Mich. 2005), affirmed in part, reversed in part, 485 F.3d 364 (6th Cir. 2007).
5. Id. at 857–58.
6. Id. at 858
7. 485 F.3d at 376.
8. The federal courts in Ege treated the answer to the 3.5 million people as "probability testimony" without questioning Michigan's general rule that bitemark identifications are admissible. A true (and wrongly decided) case of bitemark probability evidence is State v. Garrison, 585 P.2d 563 (Ariz. 1978).
9. The false-positive probability is P(+|O), where + is a positive statement ("the defendant left the mark") and O is the fact that some other person left the mark. Even if this probability is small, a disturbing number of false convictions could involve this evidence. Suppose that P(+|O) = 0.02, that 1,000 tests are performed in a set of cases with marks, and guilty defendants left the marks in 60% of these cases. The expected number of false positives is (0.02)(400) = 8. Assume that the probability of a true positive is P(+|S) = 0.96, where S means that the defendant is the source of the mark. Then the expected number of true positives is (600)(0.96) = 576. If defendants are convicted in all these cases, 8 convictions will be false (assuming that the culprit left the mark), and the many true positives will not be seen in the cases of exonerations of the innocent defendants. As indicated at the outset of these remarks, other data than exonerations are required to judge whether the test is reliable and valid.

What the FBI Hair Examiner Said About Race in State v. Manning

Two previous postings discussed the meaning and import of FBI testimony about hair in a trial of Willie Manning. I questioned Justice Breyer's suggestion that the exoneration in one of these cases came about in part because "the evidence against him, including flawed testimony from an FBI hair examiner, was severely undermined." 1/ The evidence in the case that led to the overturned conviction did not include any hair evidence at all.

The verdict in the other case -- the one with hair evidence -- is still is question. Cellmark Forensics is trying to find and analyze DNA from a rape kit and fingernail scrapings. At a later point, the laboratory may turn to "other items of evidence," including, presumably, the hair fragments from the car of one of the victims. 2/

Testimony about this hair figured prominently in the trial. In closing, the prosecutor asked "how many [people] could leave hair fragments in the car, hair fragments that came from a member of the African-American race because that's what they find when they vacuum the sweepings of the car, that's what they find in both significantly the passenger's seat and the driver's seat, just like it would be if the man rode out there as a passenger and came back as a driver . . . ." 3/

The prosecutor was able to make this argument because of testimony from an FBI examiner about the perceived racial characteristics of the hair. In my posting of July 16, I wrote that the Department of Justice's letter of May 4 confessing scientific error

is hardly a repudiation of the testimony that the hairs from the car "exhibited characteristics associated with the black race." To the contrary, it endorses this testimony as a permissible "scientific analysis." ... What "would be error," in the DOJ's view, is "any statement of probability whether the hair is from a particular racial group." But it is impossible to tell from the letter whether the FBI agent gave any such testimony. The federal district court's opinion denying Manning's habeas corpus petition makes it sound like the testimony was not of the sort later deprecated by the FBI. 4/

Today, I obtained the motion that persuaded the Mississippi Supreme Court to stay the execution and to order DNA testing. 5/ The FBI agent's testimony is appended to it. The examiner not only presented the characteristics of the hairs as associated with race ("valid" according to the DOJ), but added that "these hairs were hairs from an individual of the black race" 6/ -- not "valid."

NOTES

1. David H. Kaye, Justice Breyer in Glossip v. Gross on "flawed testimony from an FBI hair examiner", July 16, 2015, Forensic Science, Statistics & the Law
2. Letter from Cellmark Forensics to Robert L. Mink, June 5, 2015, available at http://courts.ms.gov/Images/Orders/dc00001_live.SCT.13.DR.491.35115.0.pdf.
3. Motion to Stay Execution and Set Aside Convictions, Second Motion for Leave to File Successive Petition for Post-Conviction Relief, and Motion in the Alternative for Other Forms of Relief, at 5, Manning v. Mississippi, No. 2013-DR-00491-SCT (May 6, 2013).
4. Kaye, supra note 1.
5. Motion, supra note 3.
6. Transcript at 1048.

Validity, Overclaiming, and Error: More on Willie Manning's Exoneration

Yesterday I questioned reliance on the exoneration of Willie Manning as a case in which hair evidence was severely undermined, prompting his exoneration. The problem with this understanding of the Manning exoneration is that the crimes for which his conviction was overturned did not involve any hair evidence. In addition, I tried to probe what Department of Justice reported was erroneous or invalid about the hair evidence against him in a separate case. I concluded with the observation that

It is entirely possible that DNA testing will soon exonerate Manning in the Miller-Steckler murder case (to the extent of showing that the hair in the car was not his). As of May, 2015, this had not happened. If it does, it would mean that either the original determination of the racial characteristics of the hair was wrong -- something that the FBI has not conceded -- or that the determination was correct but that Manning was not the person whose hairs were in the car -- something that the criminalist never purported to resolve.

Scott Henson, "a former journalist turned opposition researcher/political consultant, public policy researcher and blogger who is presently Executive Director of the Innocence Project of Texas" 1/ and whose Grits for Breakfast blog is a great source of information with a Texas flavor, made an important comment:

Perhaps that's "something that the criminalist never purported to resolve," but posing that implication is of course the reason the prosecutor presented his or her testimony. When they're reviewing these cases, the issue isn't just whether the forensic examiner erred but also if prosecutors overstated or misused the evidence to ask the jury to draw invalid conclusions. (In Texas' review of state-level examiners, with which I'm more familiar, they examine both the forensic testimony AND the prosecutor's closing arguments.)

If the DNA testing comes back excluding Manning, IMO it'd be fair to say that the use of hair testimony in his case has been "severely undermined," since it would have treated exculpatory evidence as inculpatory.

Overstating the scientific findings by experts or parties is a major problem. The Santae Tribble case is a glaring example of prosecutorial overclaiming. The Assistant US Attorney told the jury that "[t]here is one chance, perhaps for all we know, in 10 million that it could [be] someone else’s hair." But published estimates of the probability that hairs from different individuals will be classified as microscopically similar range from 1/4500 to 1/10 (with the latter being much more plausible). Closing arguments like this are therefore unacceptable. If courts permit hair evidence to be used, then a prosecutor who does so has a duty to use it only for what what it is worth. In Tribble, the prosecutor claimed that the hairs had matching features and then overclaimed that the probability of this match was negligible if it was someone else's hair.

The odd thing about the Manning case in which hair evidence played a role, however, is that the overclaiming did not relate to the claim. The overclaiming that the FBI spotted in its hair evidence review project pertained to the ability to identify individuals by a combination of many physical characteristics of hair. But the expert expressly stated that that could not be done in this case and only made a claim about race.

Could the prosecutor have gone beyond the expert's testimony about race and "overstated or misused the evidence to ask the jury to draw invalid conclusions"? Yes, that could be. The FBI's hair review project is limited to the reports and testimony of FBI hair examiners. It will be interesting to see whether states implement hair evidence review projects and what their scope will be.

Whether or not there was any relevant overclaiming in Manning, it is true that a DNA test showing that the hair is not Manning's would severely undermine the state's case. But it would not necessarily undermine the validity of the examiner's testimony that the hair had certain "racial characteristics." (Whether, as the DOJ letter indicated, racial classifications are valid to begin with is, of course, a different question.) 2/

To see that disproof of an inference from scientific evidence that arises from better scientific evidence does not make the original evidence invalid, let's switch from hair to serology for a moment. Sometimes hair can be analyzed for ABO blood groups. As a thought experiment, suppose that the testimony had not been about race but was only that both Willie Manning and the hair in the car were blood group B, and that Manning was among the 10% or so of the U.S. population whose ABO type is consistent with this finding. Would a later mitochondrial DNA test that excluded Manning severely undermine this ABO testimony? It certainly would contradict the conclusion that Manning left hairs in the car. It would change the hair evidence from inculpatory to exculpatory.

But it would not show that ABO testing is invalid in general or that the value of the test was overstated in this case. It would be a reason to exonerate Manning but not a reason to denominate the original evidence as scientifically invalid or as having exceeded the limits of science. In this respect, it would be no different than incontrovertible evidence that Manning was incarcerated when the murders were committed -- powerful evidence of innocence but no evidence of invalidity or overclaiming.

NOTES

1. Scott Henson, Who Is This Guy?, Grits for Breakfast, Jan. 6, 2004, http://gritsforbreakfast.blogspot.com/2005/01/about-me.html
2. For a short, sympathetic review of the theory of racial classification of hairs, see Edward J. Imwinkelried, Forensic Hair Analysis: The Case Against the Underemployment of Scientific Evidence, 39 Wash. & Lee L. Rev. 41, 49-50 (1982).

Justice Breyer in Glossip v. Gross on "flawed testimony from an FBI hair examiner"

Justice Stephen Breyer recently proposed that the Supreme Court reconsider the constitutionality of the death penalty. One of his concerns is that despite the level of scrutiny capital sentences are supposed to receive, innocent defendants may be executed. Justice Breyer's worry on this score cannot be dismissed as historically unfounded. Neither can it be written off as the predictable moaning of a bleeding-heart liberal, coming as it does from the "high court's raging pragmatist."

A related concern articulated by Justice Breyer is the "uncertainty as to whether a death sentence will in fact be carried out" and when that day will arrive. "Willie Manning," for example, "was four hours from his scheduled execution before the Mississippi Supreme Court stayed the execution. See Robertson, With Hours to Go, Execution is Postponed, N.Y. Times, Apr. 8, 2015, p. A17. Two years later, Manning was exonerated after the evidence against him, including flawed testimony from an FBI hair examiner, was severely undermined. Nave, Why Does the State Still Want to Kill Willie Jerome Manning? Jackson Free Press, Apr. 29, 2015."

Without questioning the main point about the uncertainty of confinement, I want to probe the use of the Manning case -- or rather cases -- as an exoneration in which "flawed testimony from an FBI hair examiner was severely undermined."  Justice Breyer has been called a "technocrat," and it is not surprising that he would call attention to the hair evidence against Manning and to "the more general problem of flawed forensic testimony" as illustrated by reports that "FBI Testimony on Microscopic Hair Analysis Contained Errors in at Least 90 Percent of Cases in Ongoing Review," including capital ones like Manning's. But what was the erroneous hair evidence against Manning, and in what sense was it erroneous?

The answers are complicated, and it turns out that hair evidence played no role in the case in which Manning was exonerated. Manning was convicted in not one, but two, capital cases. The hair evidence applies to a different pair of murders for which Manning has yet to be exonerated, and it is hard to see how the hair testimony there has been "severely undermined." For clarity, I discuss the separate cases separately. 

I. The Jimmerson-Jordan Murders

According to the article cited by Justice Breyer,

An Oktibbeha County jury convicted Manning for killing nonagenarian Emmoline Jimmerson and her daughter, Alberta Jordan, in the winter of 1992. The women were beaten and their throats slashed during an apparent robbery attempt at their Brookville Gardens apartment in Starkville. Manning was convicted of the crime at age 26 and sentenced to death.

The state's star witness, a man named Kevin Lucious, told police and later testified in court, that he saw Manning enter the victims' apartment from his own apartment, but police found the apartment where Lucious claimed to live was vacant at the time of the crime. The apartment manager also had no record of Lucious being a tenant.

Presiding Justice Michael K. Randolph, on behalf of the supreme court's majority, ordered the case back to circuit court for a new trial, agreeing with Manning's attorneys that "there is no question that defense counsel would have had the opportunity to meaningfully impeach Lucious' testimony that he lived in the apartment at the time of the crime and saw Manning enter the victims' apartment." ...

[Later], Luscious recanted most of his statements, saying he only testified because he feared being charged with the crime himself. ... Luscious claimed that he told Sheriff Bryan that another man, Tyrone Smith, had confessed to the murders. With the state's material witness now changing material parts of his story, the case had to be thrown out [rather than retried].

Neither this article nor the entry in the University of Michigan's National Registry of Exonerations refers to any forensic-science evidence in the case. The Registry lists the factors that contributed to the conviction as follows: "perjury or false accusation, official misconduct."

II. The Miller-Steckler Murders

In 1992, two Mississippi State University students, Tiffany Miller and Jon Steckler, who were dating, were shot and killed near the fraternity house in which one of them lived. Steckler's fraternity brother had a car that was burglarized at around the same time. The state produced evidence that Manning, who had a record of convictions for theft and other crimes, was distributing items stolen from the car. Manning conceded that he was selling stolen goods but said that he did not know who stole them. Among the other evidence introduced against Manning was hair found in Miller's car. The car was found near campus, and it had been used to run over Steckler.

An FBI criminalist testified that he could "microscopically determine if the hairs look alike and determine with some degree of certainty, although not absolutely, but with some degree of certainty if hairs, for example, found in vacuum sweepings from an automobile originated from a particularly named individual." Manning v. State, 726 So.2d 1152, 1180 (Miss. 1998). He also testified that hairs from the car "exhibited characteristics associated with the black race." Id. The examiner "went on to testify that as the hairs were only fragments, he could not compare the hairs to a known sample, and that he was limited to a determination as to the racial characteristics of the hair." Manning v. Epps, 695 F.Supp.2d 323, 380 (N.D. Miss. 2009). This determination was significant because the two victims were white, and Manning is black. Knowing that the hair had the latter "racial characteristics" made it more probable that it was Manning's and thus that he was in the car.

The DOJ issued not one, but two letters about the 1994 testimony of its agent. The first letter, dated May 2, 2013, reported that

the microscopic hair comparison analysis testimony or laboratory report presented in this case included statements that exceeded the limits of science and was, therefore, invalid. While this case did not involve a positive association of an evidentiary hair to an individual, the examiner stated or implied in a general explanation of microscopic hair comparison analysis that a questioned hair could be associated with a specific individual to the exclusion of all others -- this type of testimony exceeded the limits of the science. The examiner also assigned a statistical weight or probability or provided a likelihood that, through microscopic hair comparison analysis, the examiner could determine that a questioned hair originated from a particular source, or an opinion as to the likelihood or rareness of a positive association that could lead the jury to believe that valid statistical weight can be assigned to n microscopic hair association -- this type of testimony exceeded the limits of the science. (A copy of the documents upon which this determination was based is enclosed.)

I have not seen a copy of the testimony, but the letter does not suggest that the testimony used to link Manning (and all other African Americans) to the hair in the car was erroneous or undermined. Apparently, the criminalist overstated the power of hair features as the basis for a probabilistic or categorical statement that an individual was in fact the source of a hair, but as we saw, the expert here shied away from either of those statements. He testified only to the race of the unknown individual whose hair was in the car.

Two days later, the DOJ distributed a second letter referring to an "additional error." This letter addressed the racial identification. One might well wonder about the "racial characteristics" of hair. How definitive of race are these features? Can criminalists really zero in on African Americans this way?The letter had this to say:

We have determined that the microscopic hair comparison analysis testimony or laboratory report presented in this case included additional statements that exceeded the limits of science and was, therefore, invalid. In response to inquiries regarding whether the errors identified in the notification letter had any bearing on the examiner’s opinion regarding the racial classification of the hair, the FBI states the following: The scientific analysis of hair evidence permits an examiner to offer an opinion that a questioned hair possesses certain traits that are associated with a particular racial group. However, since a statistical probability cannot be determined for classification of hair into a particular racial group, it would be error for an examiner to testify that he can determine that the questioned hairs were from an individual of a particular racial group. Thus, an examiner cannot testify with any statement of probability whether the hair is from a particular racial group, but can testify that a hair exhibits traits associated with a particular racial group. (A copy of the FBI Microscopic Hair Analysis Report, dated May 4, 2013, is attached.)

This paragraph is hardly a repudiation of the testimony that the hairs from the car "exhibited characteristics associated with the black race." To the contrary, it endorses this testimony as a permissible "scientific analysis." (It is not clear to me that this assessment of the science is correct, but I have not researched the scientific literature.) What "would be error," in the DOJ's view, is "any statement of probability whether the hair is from a particular racial group." But it is impossible to tell from the letter whether the FBI agent gave any such testimony. The federal district court's opinion denying Manning's habeas corpus petition makes it sound like the testimony was not of the sort later deprecated by the FBI. Manning v. Epps, 695 F.Supp.2d 323, 380 (N.D. Miss. 2009).

The letter added that the FBI was prepared to perform DNA tests on the hairs or other biological material if desired. The Mississippi Supreme Court called off the execution to allow Manning "to proceed in the circuit court with his request for DNA testing ... ." Manning v. State, 119 So.3d 293, 293 (Miss. 2013). In 2015, Manning's lawyer "said several items have been sent to a lab in Houston, Texas, for analysis" and that "the timing of the testing and issuing of results is up to the lab and the FBI." 1/.

It is entirely possible that DNA testing will soon exonerate Manning in the Miller-Steckler murder case (to the extent of showing that the hair in the car was not his). As of May, 2015, this had not happened. 2/ If it does, it would mean that either the original determination of the racial characteristics of the hair was wrong -- something that the FBI has not conceded -- or that the determination was correct but that Manning was not the person whose hairs were in the car -- something that the criminalist never purported to resolve.

NOTES

1. R. I. Nave, Why Does the State Still Want to Kill Willie Jerome Manning? Jackson Free Press, Apr. 29, 2015, available at http://www.jacksonfreepress.com/news/2015/apr/29/why-does-state-still-want-kill-willie-jerome-manni/.
2. Maurice Possley, Willie Manning, Nat'l Registry of Exonerations, https://www.law.umich.edu/special/exoneration/pages/casedetail.aspx?caseid=4679 ("Manning remains on Mississippi’s Death Row for the Miller-Steckler murders as the physical evidence in that case was still undergoing DNA testing as of April 2015.").

RELATED POSTINGS

• What the FBI Hair Examiner Said About Race in State v. Manning, Forensic Science, Statistics & the Law, July 19, 2015
• Validity, Overclaiming, and Error: More on Willie Manning's Exoneration, Forensic Science, Statistics & the Law, July 17, 2015
• No Relief for Jeffrey MacDonald After FBI Declares It “Exceeded the Limits of Science” with Hair Analysis, Forensic Science, Statistics & the Law, May 23, 2015
• The FBI's Worst Hair Days, Forensic Science, Statistics & the Law, July 31, 2014

In His Own Words: Justice Scalia's Poetry Slam

In discussing the exchange in Glossip v. Gross between Justices Breyer and Scalia on the deterrent effect of capital punishment, I alluded to Andrew Lang (1844-1912), Scottish poet, novelist, and literary critic. Although any further connection to science or statistics is nonexistent, I can't resist a poetic tangent. Fred Moss pointed me to an animated "poetry slam" of Justice Scalia's "snarky ... poetic gems" at the cartoonist, Mark Fiore's website. Rearranging bits of the concurrence in Glossip yields this additional free verse:

they have discovered the lost folios of Shakespeare
they ask us for clemency, as though clemency were ours to give
If only Aristotle, Aquinas, and Hume knew
that moral philosophy could be so neatly distilled
into a pocket-sized, vade mecum
full of internal contradictions and (it must be said)
gobbledy-gook Welcome to Groundhog Day.

Related posting:
Two Scientific Issues in Glossip v. Gross, July 14, 2015, Forensic Science, Statistics, and the Law

Two Scientific Issues in Glossip v. Gross

In Glossip v. Gross, the Supreme Court narrowly upheld a district court's refusal to issue a preliminary injunction against a three-drug protocol being used to kill capital offenders. The majority of five Justices, led by Justice Alito, was emphatic. Their opinion offered not one, but two "independent reasons." The second was that the district court's findings about the effect and dosage of the sedative midazolam were not clearly erroneous. Although the Court cautioned that "federal courts should not embroil themselves in ongoing scientific controversies beyond their expertise" (internal quotation marks omitted), the factual question that ultimately must be answered correctly is whether this sedative, administered to achieve deep unconsciousness, really blocks the pain caused by paralytic and heart-stopping agents. The opinion does not resolve this question, and even if the forgiving "clearly erroneous" standard was met in this case, Justice Sotomayor's opinion for the four dissenting Justices contains more than enough technical material to make one nervous about the conclusion that midazolam works as the state of Oklahoma hopes it does.

A second controversy involves social, not medical, science. Justice Breyer, joined by Justice Ginsburg), wanted the Court to ask for "full briefing on ... whether the death penalty violates the Constitution." He maintained that "the death penalty, in and of itself, now likely constitutes a legally prohibited 'cruel and unusual punishmen[t].'" One aspect of this tentative conclusion involved "the death penalty's deterrent effect" -- the subject of innumerable studies and two skeptical reports from the National Academy of Sciences.

Justice Scalia, joined by Justice Thomas, was upset at what he called the "speculat[ion] that it does not 'seem likely' that the death penalty has a 'significant' deterrent effect." His approach to this empirical question was quintessentially legal, not scientific -- pick the answer you want (or think you know) and look only for confirming evidence. Justice Scalia's argument that the existence of the death penalty (even if rarely used and long delayed) deters significantly more than life imprisonment does consists of a single sentence: "It seems very likely to me, and there are statistical studies that say so." What studies?

See, e.g., Zimmerman, State Executions, Deterrence, and the Incidence of Murder, 7 J. Applied Econ. 163, 166 (2004) (“[I]t is estimated that each state execution deters approximately fourteen murders per year on average”); Dezhbakhsh, Rubin, & Shepherd, Does Capital Punishment Have a Deterrent Effect? New Evidence from Postmoratorium Panel Data, 5 Am. L. & Econ. Rev. 344 (2003) (“[E]ach execution results, on average, in eighteen fewer murders” per year); Sunstein & Vermeule, Is Capital Punishment Morally Required? Acts, Omissions, and Life-Life Tradeoffs, 58 Stan. L. Rev. 703, 713 (2005) (“All in all, the recent evidence of a deterrent effect from capital punishment seems impressive, especially in light of its ‘apparent power and unanimity’”).

Justice Scalia would have done better to have followed the example of Andrew Lang, the novelist who tried "not to use statistics as a drunken man uses lamp-posts, for support rather than for illumination." 1/ Carelessly or cavalierly, Justice "Scalia cites a paper by Cass Sunstein for a second time, even though after the first such Scalia citation in an earlier lethal injection case, Cass Sunstein (writing with Justin Wolfers) affirmed his view that there is no credible evidence that the death penalty is a deterrent." 2/ As Professor John Donahue, who penned these words, added, "[o]ne would hope for more from a Supreme Court justice than citations to junk science and to a paper withdrawn based on more informed consideration – especially on a matter of life and death." 3/

NOTES

1. On the origin of this aphorism, see Quote Investigator, Jan. 14, 2015, http://quoteinvestigator.com/2014/01/15/stats-drunk/.
2. John J. Donohue, Glossip v. Gross: Examining Death Penalty Data for Clarity, Stanford Lawyer, June 29, 2015, https://stanfordlawyer.law.stanford.edu/2015/06/glossip-v-gross-examining-death-penalty-data-for-clarity/. Justice Breyer is both less dogmatic and more complete in his description of the body of social science research:

   Many studies have examined the death penalty's deterrent effect; some have found such an effect, whereas others have found a lack of evidence that it deters crime. [Citations omitted.] Recently, the National Research Council ... reviewed 30 years of empirical evidence and concluded that it was insufficient to establish a deterrent effect and thus should "not be used to inform" discussion about the deterrent value of the death penalty. National Research Council, Deterrence and the Death Penalty 2 (D. Nagin & J. Pepper eds. 2012).
   
   I recognize that a 'lack of evidence' for a proposition does not prove the contrary. [Citation omitted.] But suppose that we add to these studies the fact that, today, very few of those sentenced to death are actually executed, and that even those executions occur, on average, after nearly two decades on death row. ... Then, does it still seem likely that the death penalty has a significant deterrent effect?

3. Id.

DNA Evidence Causes Wrongful Convictions 15% of the Time!?

Today's Forensic Magazine reports that

Recent estimates indicate that as many as 15 of every 100 incarcerated offenders where DNA was an element in their trial may be wrongfully convicted because of misused DNA evidence matching techniques. One common reason for this error is scientifically invalid testimony on forensic evidence. [FULL STORY]

Did I read this correctly? Fifteen percent -- one out of every six or seven -- of the men and women in prison are innocent? That is almost four times the 4% rate estimated in Gross et al. (2013). But that study, the leading one in the field, was limited to defendants with death sentences. Their cases would have been scrutinized especially carefully, so the 4% figure is probably on the low side. Maybe 15% for the broader prison population is realistic.

But wait. Forensic Magazine was not referring to the entire prison population, but only to those cases that went to trial. And within that group, 15% is the figure for defendants for whom "DNA was an element in their trial[s]." Because almost no defendants introduce DNA evidence at trial, these must be cases in which the prosecution has linked the defendant to the crime by DNA testing and the state "misused DNA ... techniques." Can it be that 15% of inmates convicted because of DNA tests are falsely convicted? That is an intolerable error rate for what is supposed to be the gold standard in forensic science. Is it time to halt DNA testing until we can find out why it generates so much "scientifically invalid testimony"?

Or are the editors of Forensic Magazine seeking sensational news rather than reading what they are reporting? Let's look at the "recent estimates." To get to them requires a few steps backwards. Forensic Magazine lists the National Institute of Justice (NIJ) -- a part of the U.S. Department of Justice -- as the source of its story. NIJ funded the Rand Corporation to do a study on improving expert performance in computing a posterior probability (which is not what forensic experts routinely do). The words are Rand's rather than NIJ's. The study cautions that "[o]pinions or points of view expressed are those of the author(s) and do not necessarily reflect the official position or policies of the U.S. Department of Justice." Rand investigated two questions: "Is bias reduced when experts do not know whether the prosecution or defense is the hiring party?," and "Is bias reduced by expert consensus feedback, wherein expertise is culled from multiple sources and those sources examine the majority view to move toward a group consensus?"

The summary at the start of the Rand research report begins with the very two sentences that Forensic Magazine broadcast. The researchers gave no references to indicate where these "recent estimates" came from, but page one of the report supplies an apparent answer. It reads as follows:

As many as 15 of every 100 incarcerated offenders may be wrongfully convicted, according to DNA evidence–matching techniques (Roman et al., 2012). One reason for this is scientifically invalid testimony on forensic evidence (e.g., Gould et al., 2012; Innocence Project, 2013).

This passage reveals that Rand simply misrepresented the "recent estimates" in the first instance. Roman et al. (2012) is an NIJ-funded study from the Urban Institute that involved no cases of wrongful convictions because of misused DNA evidence. Instead, that study "analyzed the results of new DNA testing of old physical evidence from 634 sexual assault and homicide cases that took place in Virginia between 1973 and 1987." In other words, it looked at postconviction exonerations in cases in which the trials took place before DNA evidence was even available. This study of wrongful convictions for reasons other than faulty DNA evidence "found that in five percent of homicide and sexual assault cases DNA testing eliminated the convicted offender as the source of incriminating physical evidence. When sexual assault convictions were isolated, DNA testing eliminated between 8 and 15 percent of convicted offenders and supported exoneration."

Are there any cases of false convictions caused by DNA evidence? Certainly. Does the rate of such convictions approach 15%? Neither Forensic Magazine nor the Rand Corporation offers any reason to believe it.