"Our Worst Fears Have Been Realized" -- Forensic "Evidence, Science, and Reason in an Era of 'Post-truth' Politics" (Part 1)

On October 27, a trio of panelists spoke at the Harvard Law School on "Evidence, Science, and Reason in an Era of 'Post-truth' Politics." The organizers, law professors Scott Brewer and Dan Kahan, called the panel “stellar” -- and for good reason. The speakers were

★ Judge (and now Professor) Harry T. Edwards, who co-chaired the National Academic of Science’s 17-member committee on Strengthening Forensic Science in the United States: A Path Forward. The committee’s 2009 report https://www.nap.edu/catalog/12589/strengthening-forensic-science-in-the-united-states-a-path-forward found that “In a number of forensic science disciplines, forensic science professionals have yet to establish either the validity of their approach or the accuracy of their conclusions, and the courts have been utterly ineffective in addressing this problem.”
★ Professor (and former Justice) Charles Fried, who was the U.S. Solicitor General during the Reagan administration, and who argued on behalf of Merrell Dow Pharmaceuticals in the landmark case of Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993), and
★ Professor Eric S. Lander, who is best known to the forensic-community for his early testimony and writing on DNA evidence and, of late, for his leadership role in a 2016 report of the President’s Council of Advisors on Science and Technology. This report on “Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods” noted that "[f]ederal appellate courts have not with any consistency or clarity imposed standards ensuring the application of scientifically valid reasoning and reliable methodology in criminal cases involving Daubert questions." His day job lies in directing the Broad Institute of MIT and Harvard, which “is empowering a revolution in biomedicine to accelerate the pace at which the world conquers disease.”

The program did not deal with “Evidence, Science, and Reason” writ large. Mostly, it concerned how the legal system has handled the issue of ensuring that trace evidence, which can associate individuals or objects with crimes, produces “scientific truth.”

What follows is a summary and compilation of some of the more provocative -- and sometimes ad hominem -- statements by Professor Brewer and Judge Edwards. There are also notes on a few of their remarks. I hope to touch on the remainder of the hour-and-a-half program in latter installments. The full video recording is on the web.

PROFESSOR BREWER introduced the panel and the “question of how the truth that the [best?] of science has to offer can inform and guide what is called ‘forensic science’ in such a way that when judges and jurors rely, as they clearly do, on forensic science, they are actually relying on information that legitimately and accurately claims the mantle of scientific truth ... .” He opined that “The response to both reports has been, or seems to me anyway, dispiriting.” The response to which he referred was

☁ A statement from then Attorney General Loretta Lynch that “the Department will not be adopting the recommendations [of the 2016] report related to the admissibility of forensic-science evidence”
☁ Testimony from then Senator (now Attorney General) Jeff Sessions after the 2009 report that “I don’t think we should suggest that these proven scientific scientific principles that we’ve been using for decades are somehow uncertain ... .”
☁ “More recently, the Sessions' Justice Department appointed Ted Hunt, a former prosecutor, as was Sessions, as senior forensic advisor overseeing a forensic science working group to create guidelines for forensic examiners to follow in court testimony. Hunt was ... one of two commissioners [on the National Commission on Forensic Science] to reject the recommendation that forensic experts and attorneys working on behalf of the Justice Department stop using the phrase ... ‘to a reasonable degree of scientific certainty.’” 1/
☁ The Justice Department under Attorney General Sessions “disbanded the National Commission on Forensic Science.” (For the official explanation, see The Justice Department’s Explanation for the End of the National Commission on Forensic Science, Forensic Sci., Stat. & L., April 26, 2017.)

JUDGE EDWARDS spoke of the "two-and-one-half years" that the NAS committee spent "going through all of the research that was available," the discouraging conclusions ("a community in disarray"), and the recommendation for an "independent federal agency." As for the latter,

[O]ne of our most important recommendations was the DOJ, the Department of Justice, should not be that agency. ... They had a vested interest in prosecuting. It's inconsistent with the culture of science ... and we were all, to a person, unanimous in the view that DOJ had to be kept out of it, and boy, were we prescient.
Our worst fears have been realized. ... We got no serious help from DOJ once the report issued, and that goes through ... all the administrations that have been involved. And the only time that DOJ acted in a way that has been useful is when they've been under pressure -- for example, when they were exposed on microscopic hair examinations ... and they had to 'fess up ... Other than that, we were not getting help from DOJ. ... I was appalled we could get no support from the Department to try to advance reform movement.
The other part of our worst fears realized is DOJ is now the self-annointed leader of the forensic science reform movement, which is a disaster. ... [I]t is shrewd on their part because they want to control what is and is not done -- mostly what is not done. And they are in control right now, which is really unfortunate.

On the other hand, Judge Edwards added that

Now ... just so you get the full picture, after the [2009 NAS] report issued, it was cited throughout the world ... . The settlement of the hair cases, press reports, and the National Institute of Justice [within the DOJ] has begun to sponsor some research to try to improve some of the disciplines. ... And in 2013, ... DOJ, and (in my view) under pressure, because a lot of us including the press, had been pressing for some movement, they cosponsored with the National Institutes of Standards and Technology the creation of the National Commission on Forensic Science.

But the Commission, Judge Edwards contended, suffered precisely because it was run by the Justice Department:

Now, here's the problem. DOJ held on to it. Not NIST -- DOJ. So any recommendations coming out of this group went to DOJ, and DOJ decided whether or not the recommendations would be implemented. [ 2/ ] Most of the recommendations have not been implemented. [ 3/ ] They only met twice a year. [ 4/ ] There was no real leadership, and at one point one of my colleagues, Judge Jed Rakoff, a district court judge in New York, resigned from the Commission because DOJ was going to limit the scope of their work, and he wrote an article, an op-ed piece, in the press, and they backed down and forced DOJ to open it back up again. [ 5/ ] But they had no enforcement, and the recommendations were not being accepted by DOJ, but there was a little bit of progress because at least until they were shut down, they started to come together around some recommendations that would have advanced the forensic science project. [ 6/ ]

Next, Judge Edwards discussed the PCAST report and the reaction of the Justice Department:

It's a really strong report ... essentially saying that with respect to these pattern-matching disciplines there are serious problems -- this is not science. You have people testifying about things on the assumption that it's science, and there was no scientific basis for what they were saying.

And then you have the current Department of Justice. ... They tried to block the issuance of the White House report -- DOJ did. I know about the internal battle. ... These were world-class scientists who had studied all of these disciplines [and] had come to very serious conclusions about the frailties of these disciplines, and DOJ pressed the White House not to let this report come out. It finally did come out. DOJ said "we're not interested," and when the new administration came in, DOJ said "we're still not interested, and we're not going to apply any of the recommendations here."

The current DOJ -- and I had an opportunity to hear the new leader within the Department of Justice, Ted Hunt -- who spoke to the National Academy of Sciences a week or two ago at a meeting that I was at. This is a person who was on the National Commission of Forensic Science [and] had voted against a number of proposals that would helped to reform the forensic science community. He blasted the PCAST report and said "they did not understand what they were talking about with respect to science." ... I really wish I could have videotaped the exchange. When he blasted PCAST -- and I'm sitting in a room with world-class scientists at the National Academy of Scientists -- and he did his critique on scientific research, and one of my colleagues couldn't stand it any longer. She said "what are you talking about?" She said, "I teach scientific methodology every Friday, every Friday at 1, and you haven't the faintest idea of what you are talking about." And it was exactly accurate. He had no sense of scientific methodology. ... He made a comment that was one of the most astonishing things I've ever heard. He said, "and incidentally, the jury is still out on bitemarks." The jury is not still out on bitemarks. Trust me, there is no science supporting bitemarks, and yet it is still a discipline that we use in the United States, and it's still being accepted  by the courts. And this new person who's heading the forensic science wing at DOJ has the chutzpah to say that the jury's still out. And I said, "well if you're really serious about advancing reform, wouldn't the first thing you would want to say to the world be, bitemarks is gone as far as we're concerned?" He had no interest in an independent group overseeing the reform effort, and he may kill the National Institute of Science and Technology effort.

In sum,

The National Commission on Forensic Science is dead now because Justice has killed it. So an enterprise that might have produced recommendations that could have been helpful is no longer in existence. There is this guy in Justice, Ted Hunt, who's now called head of it all, and he has his little working group, and no one knows what they're doing, and they refuse to have an independent science group oversee it.

As for NIST and its creation, the Organization of Scientific Area Committees for Forensic Science (OSAC), Judge Edwards maintained the committees are not staffed by enough "real scientists" who actually "understand science":

NIST still exists, and they oversee Scientific Area Committees. And what they are trying to do is establish standards in the disciplines for each of these groups. But I want you to understand ... while this is a noble enterprise in some respects, it's not going to get us where we need to go. The NIST enterprise with these Area Committees is pretty much dominated by forensic practitioners. Forensic practitioners are the people who got us in trouble in the first place. They don't know what they don't know. That's the problem. ... The people who are doing this do not understand what we mean when we say to them, "what you're doing has no scientific foundation." They don't understand it because they were brought up in a different world. They don't understand science. The disciplines that they are now trying to set standards for, many of them have not been validated and they have not been shown to be reliable. So how do you set standards a discipline that has not been shown to be valid and not shown to be reliable? That's one of the frailties of this whole NIST project. ...

These practitioners do not want to know sources of variability. They don't want to try and understand error rates. They don't want to believe that uncertainty exists. They object to blind studies that would help to confirm the reliability of their work. They're persuaded by very small sample sizes. And they fight the real scientists with whom they are working on these Area Committees. And they dominate the committees by 70% to 30%, and the real scientists on these committees with whom I've been in contact say it's a nightmare trying to struggle with them because they don't understand the issues.

His final remarks concerned courtroom testimony and judicial permissiveness:

The exaggerated testimony in court is horrible. We have people testifying "zero error rate, vanishingly small, essentially zero," and we have appellate court opinions in the federal courts adopting zero error rates as if it were a viable notion. ... [T]he federal rules are no help. ... Rule 702 ... was based on Daubert, which purports to talk about scientific validity, [but] Daubert has been ... a failure in ... the criminal arena ... In criminal cases, the notion of scientific validity that is very much a part of Daubert has not worked. It has failed. And it has failed because ... of judges who are wedded to precedent [and] believe that because we said it before, it must be right, and because these practitioners have been around for a long time, it must be right. In other words, history is the proof, and precedent controls. ... And when the experts come in, even when they have some science ... [in cases involving the compositional analysis of bullet lead] they did not know how to do ... a statistical analysis to look at variability and error rates -- they don't know anything about it. And the courts didn't know they didn't know anything about it. ...

NOTES

1. On the nature of Commissioner Hunt’s arguments against the Commission’s recommendation, see "Reasonable Scientific Certainty," the NCFS, the Law of the Courtroom," and that Pesky Passive Voice, Forensic Science, Stat. & L., Mar. 1, 2016; Is "Reasonable Scientific Certainty" Unreasonable?, Forensic Sci., Stat. & L., Feb. 26, 2016.
2. It is not obvious that the division of authority between NIST and DOJ was unreasonable or nefarious. I think Judge Edwards' criticism boils down to frustration with the absence of a centralized, scientific agency that regulates forensic science in America. Neither NIST nor DOJ has the power to make mandatory rules for all forensic scientists. As discussed in a posting of April 12, 2017 (Two Misconceptions About the End of the National Commission on Forensic Science), NCFS provided advice for specific actions by the Attorney General and promulgated more general views for the benefit of what DOJ and NIST call "stakeholders." NIST officials co-chaired and vice-chaired the Commission, and, with DOJ funding, NIST established a complementary structure -- the Organization of Scientific Area Committees on Forensic Science (OSAC) -- to develop science-based, voluntary standards. The stated aim of OSAC is "to identify and promote technically sound, consensus-based, fit-for-purpose documentary standards that are based on sound scientific principles." How well OSAC has met this goal is a distinct question. The Commission recommended the creation of an independent body to ensure "technical merit" of the forensic-science standards that OSAC deems meritorious. OSAC has three "resource committees," but none of them is tasked with reviewing and reporting on the technical merit of standards that the committees propose for addition to a registry of OSAC-approved standards. 
3. The Commission made recommendations for actions by the Department of Justice regarding accreditation of forensic-science service providers, proficiency testing, public release of quality-management-systems documents, a code of professional responsibility for forensic providers, AFIS interoperability, root-cause analysis of errors, certification of medicolegal death examiners, accreditation of medical examiner and coroner offices, electronic networking of those offices, a national disaster call center, a national office for medicolegal death investigation, model legislation for medicolegal death investigations, use of the term "reasonable scientific certainty," pretrial discovery, documentation and reporting, and forensic-science curriculum development. It also promulgated "views documents" that did not call on the Attorney General to take specific actions. See Work Products Adopted by the Commission, Nov. 6, 2017, https://www.justice.gov/archives/ncfs/work-products-adopted-commission.
        Computing a percentage for the adoption of the recommendation would not be trivial, as some were adopted in part and rejected in other parts. For example, the code of professional responsibility that the Department adopted omitted or altered the following provisions:

   ▹ Utilize scientifically validated methods and new technologies, while guarding against the use of unproven methods in casework and the misapplication of generally-accepted standards. [Rather than commit to using scientifically validated methods, DOJ enjoins forensic-science professionals to "[c]onduct research and forensic casework using the scientific method or agency best practices. Where validation tools are not known to exist or cannot be obtained, conduct internal or inter-laboratory validation tests in accordance with the quality management system in place."]
   ▹ Conduct independent, impartial, and objective examinations that are fair, unbiased, and fit- for-purpose. [DOJ's version omits "objective" and reads "Conduct examinations that are fair, unbiased, and fit-for-purpose."]
   ▹ Once a report is issued and the adjudicative process has commenced, communicate fully when requested with the parties through their investigators, attorneys, and experts, except when instructed that a legal privilege, protective order or law prevents disclosure. [DOJ's version reqiuires "[h]onest[] communication ... when permitted by ... agency practice."]
   ▹ Appropriately inform affected recipients (either directly or through proper management channels) of all nonconformities or breaches of law or professional standards that adversely affect a previously issued report or testimony and make reasonable efforts to inform all relevant stakeholders, including affected professional and legal parties, victim(s) and defendant(s). [DOJ preferred that its laboratory professionals have a much more limited duty to "[i]nform the prosecutors involved through proper laboratory management channels of material nonconformities or breaches of law or professional standards that adversely affect a previously issued report or testimony."]

4. The Commission did not meet only twice a year. It met four times in 2014 and 2015, three times in 2016, and twice in the first quarter of 2017, after which its charter was not renewed. The average meeting rate was thus four times a year.
5. Judge Rakoff did not write an op-ed article -- at least not one that I can find on the web. His letter of resignation appeared in the Washington Post, and newly appointed Deputy Attorney General Sally Yates flew to New York, talked with Judge Rakoff, and rescinded the decision that pretrial discovery rules were outside the Commission's mandate. The Commission made its recommendations for more complete pretrial discovery in criminal cases involving forensic-science evidence, and DOJ implemented them. See "A Bump in the Road" for the National Commission on Forensic Science, Jan. 29, 2015, Forensic Sci., Stat. & L., http://for-sci-law.blogspot.com/2015/01/justice-department-reverses-decision-on.html; Justice Department Reverses Decision on the Mandate of the National Commission on Forensic Science, Jan. 31, 2015, http://for-sci-law.blogspot.com/2015/01/justice-department-reverses-decision-on.html; Joseph Ax, After Quitting in Protest, Prominent U.S. Judge Rejoins DOJ Commission, Reuters, Jan. 30, 2015, https://www.reuters.com/article/us-rakoff-evidence/after-quitting-in-protest-prominent-u-s-judge-rejoins-doj-commission-idUSKBN0L32B020150130.
6. Sadly, the Commission was unable to attain a two-thirds majority on a subcommittee's "Final Draft Views on Report and Case Record Contents" (59% voted in favor) and "Final Draft Views on Statistical Statements in Forensic Testimony" (60% voted in favor). Whether more meetings would have produced consensus (in the sense of the required two-thirds vote) on these important matters is unclear.