Thursday, April 13, 2017

Fact Check: The National Commission on Forensic Science Vote That Wasn't

Forensic Magazine continues to report that a majority of the National Commission on Forensic Science voted in favor of its own dissolution. In a mostly recycled paragraph from an earlier article, 1/ its senior science writer, Seth Augenstein, wrote today that “the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote.” 2/

The Commission never took any vote on whether it would be a good idea to extend the Commission's life. The question put to a vote was whether to include a statement to this effect in an historical document summarizing the activities of the Commission. 3/ The subject of the vote could not have much clearer. 4/ The meeting synopsis states
[A] vote was taken to determine whether this summary report should include a statement that the Commission should continue in its current form. As a business document a simple majority of 50% “yes” votes was required to approve inclusion of this statement. A total of 42% “yes” votes were received, and therefore no statement would be included regarding the continuation of the Commission. 5/
The precise question posed and the complete vote on it were as follows: 6/
Document or Vote Question Asked Total Votes # Yes # No # Abstain
Does the NCFS Summary Report include a sentence that NCFS continues in its current form? 38 16 15 7
NOTES
  1. Seth Augenstein, Final Meeting of National Commission on Forensic Science ‘Reflects Back,’ Apr. 10, 2017, 11:59am, http://www.forensicmag.com/news/2017/04/final-meeting-national-commission-forensic-science-reflects-back. The paragraph stated that
    The NCFS produced 45 documents and recommendations in three years of work, which encompassed 600 public comments. But the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote."
  2. Seth Augenstein, Even Without Forensic Commission, Forensic Science Overhaul Proceeds at OSAC, Apr. 13, 2017, 12:12pm, http://www.forensicmag.com/news/2017/04/even-without-forensic-commission-forensic-science-overhaul-proceeds-osac. The latest paragraph states that
    The NCFS, by the end of its last meeting on Tuesday, produced 45 documents and recommendations in three years of work—many of which directed OSAC’s explorations into forensic disciplines. But the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote. Sessions announced that it would not be renewed on Monday.
    The additions are also inaccurate. Very few of the NCFS Views documents and Recommendations documents seem to have "directed OSAC's explorations."
  3. Reflecting Back—Looking Toward the Future, Dec. 16, 2016 (draft), https://www.justice.gov/ncfs/page/file/921431/download.
  4. The discussion as recorded on the meeting webcast includes the following (with intervening speaker statements omitted without ellipses):
    HON. PAM KING: This is a business record ... of this particular Commission. ... This is a document that does not take any real position as to whether something should or should not be done. ...I did get some comments from Commissioners before this meetings ... One of the ones that I really would like to get some discussion on is [the] strong feelings among some Commissioners that maybe we do want to make a statement about whether or not this Commission should continue. ...
    JULIA LEIGHTON: I would not shy away from a recommendation ... I think to scrap it altogether ... is to give up on the work we’ve done.
    GERALD LAPORTE: So I don’t agree — disagree — with anything Julia has said. ... but I don’t know if we really are in a position to make a recommendation ... .
    ARTURO CASADEVALL: I want to support what Julia said. Commissions like this develop an institutional memory. ... I strongly think we should make a recommendation that something like this continue.
    S. JAMES GATES: [A]bsent a committee like this, I don’t see a consistent driver for making progress. ...
    MATTHEW REDLE: Whether it is this form or not, ... there ought to be more work done to continue the progress that we have made ...
    JULIA LEIGHTON: [W]e need a national body [with] the gravitas of being a nonpartisan federal advisory commission. ...
    HON. JED RAKOFF: ... I do think it is important that we say some something [to] indicate that we believe the Commission should continue. ...[J]udges do pay some attention to what this Commission says and does. So I think it plays a role there that is not played by other very wonderful groups ... and some very wonderful reports. I would very much strongly encourage that we have something in there ...
    WILLIAM THOMPSON: This Commission is uniquely well situated to address those [human factors] issues ... so I hope the Commission continues to address those kinds of questions ... .
    JULES EPSTEIN: So ... for this concluding portion ... yeah, we should keep going in some shape or form. ... [M]ore needs to be done. More constituencies will look to us than to other segregated constituencies. [T]he federal advisory commission should continue.
    WILLIE MAY: Certainly, I think that the Commission’s work is not completed. [I]t would serve the country very well to continue this ... .
  5. National Commission on Forensic Science Meeting #12, Jan. 9-10, 2017, at 6, https://www.justice.gov/ncfs/page/file/953106/download
  6. Id. at 10.

Wednesday, April 12, 2017

Whither OSAC? NIST's Plans for Forensic Science Standards and Research As Told to the NCFS

The National Commission on Forensic Science held its thirteenth and final meeting on Monday. The second speaker to discuss some of the administration's plans for improving forensic science was the Acting Director of the National Institute of Standards and Technology (NIST), Dr. Kent Rochford. I edited and abridged the computer-generated transcript slightly. It includes a question from Commissioner Peter Neufeld. Finally, there is a question from Commissioner Jules Epstein to a Justice Department official about future funding for OSAC. I cannot promise riveting reading, but for anyone who wants to know what was said, here is most of it:
KENT ROCHFORD: I'd like to address the future of OSAC. OSAC was conceived under the 2013 MOU [Memorandum of Understanding] between NIST and the Department of Justice and established the Commission. The Department of Justice provides funding for the OSAC, which NIST cannot sustain on its own. The OSAC organization does not have term limits but does require funding to continue.

From the introduction of OSAC, NIST addressed the need to evolve and eventually spin off OSAC. We termed this “OSAC 2.0.” We have learned a lot from OSAC 1.0. Over the past years of operation, the organization has continued to mature as members of the group have come to a better appreciation of the standards development process. One example was seen in interested key researchers and scientists joining the FSSB [Forensic Science Standards Board]. Thank you for your assistance in supporting and strengthening the OSAC.

NIST is committed to improving OSAC, including the establishment of a clear model that will support these important goals. We are working to create a stable, sustainable operational model that provides independence from NIST. Internally a small group led by Rich Cavanaugh, who runs our special programs office, has been exploring model concepts for OSAC 2.0.

Each model is distinct yet consistent with the following goals: The new OSAC has to have a defined structure and authority. It needs to engage key stakeholders. We need to provide free access to our products. There has to be a smooth transition from the current OSAC that would create the potential for long-term sustainability. Currently, Rich's group has been looking at three models, exploring further. These involve creating federal and state partnerships that develop codes, standards, and model laws. Restructuring the OSAC so subcommittee functions are dispersed to standards development organizations, and the roles at the FSSB and SAC levels are changed to focus on quality of science and utility, respectively. And establishing a development and testing — a process we are starting, and we intend to engage the broader community to better understand the strengths and weaknesses of these possible approaches. So if you have questions about the OSAC 2.0, please reach out to Rich Cavanaugh.

I'd like to talk about NIST research efforts in forensic science. NIST remains committed to remaining its measurements and standards expertise to challenges in forensics. We played a role in strengthening forensic science since at least the 1920s. You may have seen the recent National Geographic article about William Souther, a physicist from NIST who played a role in numerous forensic cases during the 1930s, including the famous Lindbergh baby kidnapping case. The current forensic research focuses include, DNA, digital fingerprint evidence, ballistics, statistics, toxins, and trace evidence. We plan to continue working these research areas as funding is available to do so. You will see an example of how research expertise provides benefit to the forensic science community when [Dr. Elham Tabassi] talks to you about development of an ISO standard on method validation.

Let me turn to technical merit review. This past September, the President's Council of Advisors on Science and Technology recommended an expanded role for NIST in assessing the scientific foundations and maturity of various forensic disciplines. We recognize the need for and the value of such studies and are exploring ways to conduct work in this area. Without additional funding recommended by PCAST, NIST cannot make large-scale commitments to technical merit review. We are planning an exploratory study to address concerns raised by PCAST regarding DNA mixtures. This will likely involve assessing the scientific literature, developing a detailed plan for evaluating scientific validity that would include probabilistic genotyping, and assigning interlaboratory studies to measure forensic laboratory performance of DNA interpretation. These laboratory studies would build upon DNA mixture studies conducted in 2003, 2005 in 2013. NIST has a history of involving external partners in his research and standard efforts and anticipate external and internal and international collaboration.

***
In closing, I want to personally thank you for your efforts on this Commission and your commitment to strengthening forensic science through your participation in the activities of this group. Your work is made a difference, and we are grateful for your service to the nation. Thank you.
***
PETER NEUFELD: The second question for Kent, when you talked about things NIST was doing, you mentioned your current evaluation of DNA mixtures. Your predecessor stated in response to this Commission making a recommendation that NIST take on the task of making an evaluation of foundational validity and reliability of different forensic methods that they intended to do a trial. They were going to start a trial in three different areas, and the other two areas in addition to the DNA were ballistics and bitemarks. We have been told at each meeting leading up to this meeting NIST was going ahead with those trials. I noticed you only mentioned DNA. Is it still the position of NIST that they will go ahead with the trial of some ballistics and bitemarks?

KENT ROCHFORD: We still continue to do the work on ballistics and bitemarks. Given the resources we have, we're going to do the trials of the interlaboratory studies with the DNA mixtures first. Right now, the PCAST report provided a number of trials we should take on [and] it is also recommending the funding to do this. Given our current funding, we intend to start with the DNA programs. As funding may become available, we can wrap up these others areas to include trials. Currently we are doing the internal work but do not right now have the bandwidth to do the ballistics trials.
***
JULES EPSTEIN: Good morning. *** The other substantive question is, can I get clarification on OSAC? Is it now the status there is currently no further funding for OSAC?

ANSWER: No.

JULES EPSTEIN: Can we understand what is in the pipeline or the projected longevity at this moment or sustainability?

ANSWER: Right now we don't have a budget and we are in a continuing resolution. We just don't know the status so I really can't predict what it will look like.

Two Misconceptions About the End of the National Commission on Forensic Science

Several days ago, the Justice Department (DOJ) announced the end of the National Commission on Forensic Science (NCFS). Initially established to advise the Department of Justice for a two-year period, the NCFS had had its charter extended once before, in April 2015. Attorney General Sessions declined to renew it a second time.

Several explanations for this decision could be offered: (1) the current administration is unreceptive to scientific knowledge and advice — alternate facts and alternate science are more appealing; (2) DOJ does not want outside advice on producing and presenting scientific evidence; (3) DOJ is tired of spending millions of dollars for advice from this particular group of lawyers, judges, administrators, forensic scientists, and others. (4) DOJ believes that NCFS has outlived its usefulness and there are better ways to obtain advice. And of course, some combination of these things might have been at work. I have no inside information, but I thought it might be helpful to collect some of what is publicly known if only to correct misconceptions about the Commission and its role vis-a-vis the Justice Department. I'll begin by noting two such misconceptions.

Misconception 1
NCFS Was Evaluating the Validity of Forensic Science Tests and Methods

One misconception is that the Commission was conducting independent evaluations of accepted methods in forensic science. Thus, the Associated Press reported that “National Association of Criminal Defense Lawyers ... President Barry Pollack said the commission was important because it allowed ‘unbiased expert evaluation of which techniques are scientifically valid and which are not.’” 1/ But not one of the 44 documents identified as “work products” on the NCFS website examines the validity of any technique.

The two documents that directly address “technical merit” are views and then recommendations 2/ about the need to study validity and reliability of techniques. No surprise there. More importantly, the documents underscore the importance of bringing what we might call "outsider" scientific expertise to bear in these efforts, and one of them contains pointed advice to other organizations. Specifically, a recommendation calls on NIST and its creation, the Organization of Scientific Area Committees for Forensic Science (OSAC), to reform the procedure OSAC uses to review and endorse standards for test methods. It states:
The Organization of Scientific Area Committees for Forensic Science (OSAC) leadership, the Forensic Science Standards Board (FSSB), should commit to placing consensus documentary standards on the OSAC Registry of Approved Standards for only those forensic science test methods and practices where technical merit has been established by NIST, or in the interim, established by an independent scientific body. An example of an interim independent scientific body could be an OSAC-created Technical Merit Resource Committee composed of measurement scientists and statisticians appointed by NIST and tasked with the evaluation of technical merit. 3/
This recommendation, by the way, has had limited impact. Yes, NIST has announced that it will do further research in a few areas such as DNA-mixture analysis. No, OSAC has not established a Resource Committee to check the technical merit of the documents that filter up from its subject-area committees and subcommittees.  4/

Rather than performing literature reviews (or promulgating scientific standards for forensic laboratories to follow), NCFS focused on broader issues of policy, needs, and legal reforms for generating or evaluating scientific evidence. This role for the Commission relates to a second misconception.

Misconception 2
NCFS Was a Worthless “Think Tank”

According to the Washington Post,
[T]he National District Attorneys Association, which represents prosecutors, applauded the end of the commission and called for it to be replaced by an Office of Forensic Science inside the Justice Department. Disagreements between crime lab practitioners and defense community representatives on the commission had reduced it to “a think tank,” yielding few accomplishments and wasted tax dollars, the association said. 5/
A press release from the NDAA does “applaud” the DOJ’s decision not to nonrenew the Commission, but not because the NCFS was a “think tank.” The group representing “2,500 elected and appointed District Attorneys across the United States, as well as 40,000 Assistant District Attorneys” complained that
The Commission lacked adequate representation from the state and local practitioner community, was dominated by the defense community, and failed to produce work products of significance for the forensic science community. Very few of the recommendations from the Commission were adopted and signed by the previous Attorney General during its existence. Those that were signed, such as universal accreditation, had already begun to develop organically within the forensic science community as accepted best practices, thus replicating ongoing work and wasting taxpayer dollars. 6/
I have not checked the percentage of recommendations “signed” by the Attorney General, but the Commission’s views documents never were intended to be signed by anyone, and the notion that only recommendations for specific action by the Attorney General benefit “the forensic science community” is shortsighted. Among the Commission’s documents of lasting value are the following:
  • Recommendation on Transparency of Quality Management System
  • Recommendation on Model Legislation for Medicolegal Death Investigation Systems
  • Views Document on Recognizing the Autonomy and Neutrality of Forensic Pathologists
  • Recommendations on Use of the Term “Reasonable Scientific Certainty”
  • Recommendation on Pretrial Discovery
  • Recommendations on Use of the Term “Reasonable Scientific Certainty”
  • Views Document on Judicial Vouching
  • Views Document on Ensuring that Forensic Analysis is Based Upon Task-Relevant Information
  • Views Document on Facilitating Research on Laboratory Performance
  • Views Document on Identifying and Evaluating Literature that Supports the Basic Principles of a Forensic Science Method or Forensic Science Discipline
It is instructive to compare the NDAA's dismissal of the "universal accreditation" recommendation  with the assessment of it by Associate Deputy Attorney General Andrew Goldsmith, who stated in his remarks at the final NCFS meeting that
[T]here is no single commission recommendation more important for the practice of forensic science than the recommendation regarding universal accreditation. I have been told the Department's decision to publicly announce the policy on accreditation and to mandate our prosecutors to rely on accredited labs when practicable has made a difference in laboratories and in moving to accreditation. These recommendations and the Department's review and implementation are a demonstration of the measurable impact of the work of this Commission ... .
Naturally, many of the ideas or actions that the Commission endorsed were not original. The idea of accreditation was prominent in the 2009 National Research Council (NRC) report on forensic science as well as NRC reports on DNA evidence in 1992 and 1996. NCFS was not a think tank, but a mixed bag of administrators, prosecutors, defenders, judges, law professors, police officials, laboratory scientists, medical examiners and coroners, research scientists, and other individuals. It could be criticized as wasteful — 13 meetings of 41 members (including the ex officio ones) plus an unlisted number of nonmembers appointed to subcommittees at a cost of millions of dollars for taxpayers (not to mention the opportunity costs to the volunteers). Consequently, it certainly is fair to ask how much additional benefit would have come from another two years of Commission life. 7/ But the Justice Department does not plan to halt all study of in-house forensic science reform. It has announced that some of it will continue via a newly created -- and surely not costless -- task force run by the incoming Deputy Attorney General. Given that plan, is the restructuring really an effort to save taxpayer money because of a perception that NCFS had reached the point of diminishing returns? Or is it a move to control the agenda and to modify the list of people who provide input? More on that later.

NOTES
  1. Sadie Gurman, Sessions' Justice Dep't Will End Forensic Science Commission, AP News, Apr. 11, 2017, https://www.apnews.com/c076ef99c48948e3856902cfca9e7b14/Sessions'-Justice-Dep't-will-end-forensic-science-commission
  2. NCFS often prepared two “work products” per topic for its recommendations — a preliminary “views” document followed by a final, more concrete  “recommendations” document. Consequently, the total number of its "work products" is a poor quantitative measure of its accomplishments.
  3. Recommendation to the Attorney General: Technical Merit Evaluation of Forensic Science Methods and Practices, Dec. 9, 2016, at 3, https://www.justice.gov/ncfs/page/file/905541/download
  4. A more cheerful description of the response from NIST and OSAC can be found in a letter from the six research scientists (not forensic scientists) on the Commission pleading for a renewal of the charter.
  5. Spencer Hsu, Sessions Orders Justice Dept. To End Forensic Science Commission, Suspend Review Policy, Wash. Post, Apr. 10, 2017, https://www.washingtonpost.com/local/public-safety/sessions-orders-justice-dept-to-end-forensic-science-commission-suspend-review-policy/2017/04/10/2dada0ca-1c96-11e7-9887-1a5314b56a08_story.html?utm_term=.c24306818156
  6. NDAA, Press Release, National District Attorneys Association Applauds Expiration of National Commission on Forensic Science, Apr. 10, 2017, http://www.ndaa.org/pdf/NDAA%20Statement%20on%20Expiration%20of%20National%20Commission%20on%20Forensic%20Science.pdf
  7. Erin Murphy, Op-ed, Sessions Is Wrong to Take Science Out of Forensic Science, N.Y. Times, Apr. 11, 2017, https://www.nytimes.com/2017/04/11/opinion/sessions-is-wrong-to-take-science-out-of-forensic-science.html (asserting that NCFS "was even poised to issue a raft of best practices for the wild west of digital forensics, which has exploded without supervision over the years.")