Are "Exclusions" Deductive and "Identifications" Merely Probabilistic?

Lately I have heard people say that “source exclusions” are the product of deductively valid reasoning, whereas “source identifications” are less certain. But the difference between such conclusions does not arise from the fact that one is deductive and the other is not. On reflection, "exclusions" are no less probabilistic in nature than "identifications." 

Presumably, the reason people may think that exclusions are deductions is that “exclusion” can be part of a deductive argument. For example,

(1) No human being with Type O blood will leave Type A blood at a crime scene.
(2) Defendant has Type O blood.
(3) The crime-scene bloodstain is Type A.
Therefore,
(4) Defendant is excluded as the source of the stain.

This argument is a formally valid deduction. If the premises (1)–(3) are true, the conclusion (4) must be true. There is, however, no guarantee that any or all the premises are true. Perhaps something very strange (but not logically impossible) happened to convert a Type O stain into a Type A one. Or perhaps the defendant or the stain was mistyped. These are not very likely events, but deductive logic does not make them true. So although (4) is certain to be true conditional on (1)–(3), we cannot be absolutely certain that (4) is in fact true. In the symbolism of probability statements, the fact that Pr[(4) | (1)&(2)&(3)] =1 does not ensure that Pr(4) = 1 unless Pr(1) = Pr(2) = Pr(3) = 1.

One might think that I have misstated the argument. Indeed, the argument ending in exclusion might be reframed as follows:

(1) Everything we know tells us that no human being with Type O blood will leave Type A blood at a crime scene.
(2) A blood test shows that defendant has Type O blood.
(3) A blood test shows that the crime-scene bloodstain is Type A.
Therefore,
(4) The blood test excludes defendant as the source of the blood stain.

This too is a deductively valid argument. The premises entail the conclusion, and the premises are even harder to dispute than the ones in the previous example. But if this is all that a criminalist means by an exclusion, then an exclusion is not actually a statement that the defendant is not the source of the trace. The conclusion in our second argument only asserts that the test has excluded the defendant. Unless the test never errs, it does not follow (deductively) that the defendant was not the source of the bloodstain. To appreciate the force of the "deduction," we need to study how often criminalists report exclusions when examining items from different sources as opposed to items from the same source.

The situation is the same for “identification.” This conclusion also can come at the end of a deductive argument. For example,

(1) Fingerprints from the same finger always match.
(2) Fingerprints from different fingers never match.
(3) The questioned and known fingerprints being compared match.
Therefore,
(4) The fingerprints being compared are from the same finger — an “identification.”

As with an exclusion, the argument from (1)–(3) to (4) is logically impeccable. If (1)–(3) are true, then so is (4). And, once more, because propositions (1)–(3) might not all be true, the truth of the “identification” is not absolutely certain.

Again, we can rephrase the argument in a (vain) effort to make it appear that the desired conclusion is purely deductive:

(1) Fingerprints from the same finger always match.
(2) Fingerprints from different fingers never match.
(3) The questioned and known fingerprints being compared match.
Therefore,
(4) I have identified the questioned print as coming from the finger that left the known print.

But again, the deductive argument does not get us to the conclusion that the known finger is the source of the questioned print. To ascertain the probative value of a positive source classification (an “identification”), we need to study the performance of criminalists making these source attributions. We need to study how often criminalists report “identification” when examining items from the same source as opposed to items from different sources. In the end, if there is a difference in the certainty we can attach to an exclusion as opposed to an identification, it does not emanate from the difference between inductive and deductive forms of argument. It results from the fact that the premises of some inductive arguments are more probably true than the premises of other inductive arguments.

Reference

1. Brian Skyrms, Choice and Chance: An Introduction to Inductive Logic (4th ed. 2000).

A Superficial Opinion on Fingerprints in Missouri Has a Few Interesting Wrinkles

In State v. Hightower, 1/ the Missouri Court of Appeals added to the list of superficial opinions on the admissibility of latent fingerprint matches. In this case, a man pointed a gun at the driver of a car, snatched a purse and an iPad, fired into the air, and escaped. The driver did not see the robber’s face, but a detective lifted two “relatively new and undisturbed” prints from the driver’s window. A latent fingerprint examiner with the St. Louis County Police Department used a state AFIS (automated fingerprint identification system) to arrive the conclusion that the latent prints “were left by Defendant's left middle and ring fingers.” On the basis of this identification, a jury convicted David Hightower of armed robbery, and the trial court sentenced him to serve 18 years in prison.

At a pretrial hearing on general scientific acceptance, Dr. Ralph Haber, a research psychologist and “forensic scientist and expert witness” (resume, at 1) testified for the defendant that (as the state court of appeals put it) “the National Academy [of Science] and the National Institute [of Standards and Technology] have both decried the reliability and accuracy of fingerprint evidence adduced using the ACE-V method.” Apparently, he was referring to the well known 2009 report of the NAS Committee on Identifying the Needs of the Forensic Science Community and the report of the NIST Expert Working Group on Human Factors in Latent Print Analysis (D.H. Kaye ed., 2012). 2/ He also “testified he has been asked to serve on the National Commission [on Forensic Science] committee responsible for developing standards for fingerprint analysis.” 3/ The trial court was more impressed by his admission that “in every hearing he had been involved in to exclude fingerprint evidence the evidence had been deemed admissible, save one case from Maryland.” It denied the defendant’s motion to exclude the evidence.

At trial, Dr. Haber “concluded that a person could not be identified with 100% certainty based on a fingerprint,” but acknowledged on cross-examination that “he had not looked at the actual fingerprints in the present case.” When the prosecutor argued in a closing statement that “other ... experts in latent fingerprint examinations ... could have examined this but were not asked to,” the judge instructed the jury to disregard the statement.

The court of appeals assumed that the comment was improper. In a typical display of judicial unrealism, the appellate court blithely “presume[d] the court's curative instructions to the jury removed any prejudice from the prosecutor's statements.” But was the prosecutor in the wrong in the first place? Although the defense certainly has no obligation to examine fingerprints, I do not think the answer is entirely obvious — particularly when the defense produces an expert who testifies that the identification is wrong or uncertain. Here, Dr. Haber apparently choose not to examine the prints although his resume prominently advertises 120 hours of “fingerprint comparison training.” Of course, the prosecutor's comment referred not just to the testifying defense expert’s work, but to the defendant’s decision not to call on still other examiners. Moreover, the defense theory was not that other examiners would disagree with the state’s expert, but only that the meaning of an agreed-upon match is unclear.

If the meaning of a match is indeed unclear — because the entire process has not been fully validated — then it is hard to see how the evidence is generally accepted in a relevant scientific community. The opinion in Hightower does not respond to that argument. Instead, the court maintained that judges have always found subjective comparisons of fingerprints sufficient to demonstrate singular identity. But very few of these opinions have asked whether the scientific literature evinces general acceptance of the proposition that latent print examiners as a group can reliably and accurately match prints of the quality of the ones in this case. Without that showing, how can general scientific acceptance be said to exist?

Fortunately, there are empirical studies of the process that help address this question. These studies appear in reputable scientific journals. 4/ They should inform rulings on admissibility, and the scientific findings should be used to help convey the degree of certainty in any source conclusions.

Finally, the court made short work of Hightower's argument that the conviction could not rest the fingerprint identification alone, at least not without "additional evidence indicating the fingerprints could have only been impressed at the time the crime was committed." The court wrote that

This argument is without merit. “[A] fingerprint at the scene of the crime may in and of itself be sufficient to convict.” State v. Bell, 62 S.W.3d 84, 96 (Mo. App. W.D. 2001). The defendant in Bell claimed that a partial palm print found in a place accessible to the public without credible evidence establishing it was left near the time of crime was insufficient evidence to sustain a conviction. Id. The Western District denied his point, stating there “was sufficient evidence to establish that the palm print on the counter was recent and occurred near the time of the crime” because the hotel clerk testified she had cleaned the counter twenty minutes prior to the robbery and no one other than herself and the robber had touched it in between the time of the robbery and the time the police lifted the print. Id. In the present case, Ms. Gillespie testified and her mother echoed that Defendant hit the car window open-palmed and the detective who collected the fingerprints stated they appeared “fresh” and it was his belief that they had been left recently.

At least one law review article has proposed that a single item of circumstantial evidence tying a defendant to a crime should not be sufficient for a conviction. Nevertheless, when the value of the evidence is great enough, a rigid, two-pieces-of-evidence rule seems too strict.

A further problem in Hightower is that it is not clear that a lay witness can discern the age of the fingerprints or that the detective possessed the necessary expertise to do so. What skill or experience did he have in dating prints? Are there any studies to establish that anyone can discern the age of prints just by shining a flashlight on them? Ascertaining how old prints might be from their physical or chemical properties always has eluded forensic science, although a promising technique has been reported. In Hightower, though, it does not appear that the defendant objected to this part of the detective's testimony.

NOTES

1. 511 S.W.3d 454 (Mo. App. 2017).
2. It would be fairer to say that these reports called for research to establish the probabilities of false positive and negative errors in latent print examinations and for the results of comparisons to be presented in ways that recognize the degree of uncertainty in fingerprint identifications.
3. The Commission never established subcommittee on fingerprint analysis, and Dr. Haber is not listed as a member of any of the Commission’s seven subcommittees.
4. Some of them are described in this blog.

The Justice Department’s Explanation for the End of the National Commission on Forensic Science

The decision of the Department of Justice to let the NCFS expire — a decision that was as predictable as the date of the next solar eclipse — was presented to the Commission at its final meeting on Monday, April 10. Everyone present knew that the NCFS was not intended to be an indefinite fixture. New administration or not,  a decision to continue operating the Commission and its subcommittees had to come by April 23, 2017. 1/ After all, the NCFS charter of April 23, 2013, asked it “to provide recommendations and advice to the Department of Justice” for two years. Former Attorney General Eric Holder renewed the charter once, on April 23, 2015. 2/ The new Attorney General, Jeff Sessions, elected not to renew the charter a second time.

No reasons for this discretionary action were provided. An Associate Deputy Attorney General graciously thanked the Commission for its work, stated that the new Attorney General had decided to use alternative mechanisms for developing departmental policy, described some aspects of what those would be, indicated that a press release was in the works, and thanked the Commission again.

Below I describe the written documents (an Executive Order and press releases) related to the sunsetting of the Commission and reproduce an abridged version of the (slightly garbled) computer-generated transcript. I made corrections to the extent I was confident about what actually was said, and I edited out some material that was not important to seeing where the Department of Justice (DOJ) may be headed.

I. The DOJ Press Release and the Task Force on Crime Reduction and Public Safety

A press release of April 10, 2017,  announced “a series of actions the Department will take to advance forensic science and help combat the rise in violent crime.” A new “Task Force on Crime Reduction and Public Safety” within the DOJ “will spearhead the development of [a] strategic plan” to “increase the capacity of forensic science providers, improve the reliability of forensic analysis, and permit reporting of forensic results with greater specificity.”

Nothing in the creation of the Task Force specifically suggested forensic science was of any concern. It emanated from an Executive Order signed on February 9, 2017. Harking back to the rhetoric of the Nixon Administration but placing illegal immigration at the top of the list of crimes to combat, this order declared that

It shall be the policy of the executive branch to reduce crime in America. ... A focus on law and order and the safety and security of the American people requires a commitment to enforcing the law and developing policies that comprehensively address illegal immigration, drug trafficking, and violent crime.

Within three weeks, Attorney General Sessions outlined the membership of the task force. The President’s order did not specify who or what types of people should comprise the Task Force. The Attorney General designated his Deputy Attorney General as its chair and “relevant Department components” to supply its members. He named “the Director of the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF), the Administrator of the Drug Enforcement Administration (DEA), the Director of the FBI and the Director of the U.S. Marshals Service (USMS)” as key members.

This group might not possess the immediate knowledge for devising a plan to “increase the capacity of forensic science providers, improve the reliability of forensic analysis, and permit reporting of forensic results with greater specificity,” although the membership could be supplemented. Along with the membership, the stated objectives seem limited to investigative matters instead of the widely ranging interests pursued by NCFS and reflected in its recommendations.

In fact, the list of objectives itself is a little puzzling. No one would argue with the ambition of improving reliability (in the sense of trustworthy results) and laboratory capacity, but what does it mean to be able to report “forensic results with greater specificity”? Reporting that one finger is the source of a latent print, that one gun is the source of given bullet, and that one set of teeth are the source of a bitemark are already as specific as one can possibly get. One might question the scientific status of such claims, as the last President’s Council of Advisers on Science and Technology did, but the Obama DOJ rejected much of the PCAST critique. Surely, the new President and Attorney General are not expressing newfound doubts about the ability of forensic-science practitioners to make specific source attributions.

II. The Associate Deputy Attorney General’s Remarks to the Commission

Good morning, everyone. Let me say it's an honor to be here on behalf of the acting Deputy Attorney General and be able to address this group. My name is Andrew Goldsmith, I'm an Associate Deputy Attorney General and the Department’s National Criminal Discovery Coordinator. ... As some of you know, about two years ago the then Deputy Attorney General asked me to work with you on the criminal discovery recommendation, and in that role I had the pleasure of working with a number of you.
...
Before I go further I’d like to talk about the Attorney General's firm commitment to forensic science. On Friday, Attorney General Sessions and I spoke in his office, and he made clear to me in his view good forensics is not only important because it enables us to convict the guilty, but also to clear the innocent. He stressed to me we need to focus on the integrity of the process where we have prompt access to high-quality forensics technology. He found troubling the backlog in forensics analysis, and as I will discuss in more depth later, as part of the Task Force on Crime Reduction and Public Safety, he established a forensic science subcommittee to that task force. Moreover, and I also plan to address this as well later on, he is authorized me to announce here today a series of forward-looking actions that will conform to forensic science subcommittee's development of a strategic plan on forensics.
...
The Department and NIST created the Commission as a commitment to strengthening forensic science. Our justice system depends on reliable, scientifically valid evidence to solve crimes, identify wrongdoers, and ensure innocent people are not wrongly convicted. This Department. like every other Department that has come before it, remains committed to these principles. Over the past three years, the Commission has played a role in this effort, and we are grateful for your contributions. I'd like to highlight two contributions I am certain will have long-lasting effect. As you know, we announced new department-wide guidance on criminal discovery in cases with forensic evidence at the last Commission meeting. From my vantage point as the national criminal discovery coordinator, the recommendation on pretrial discovery will have long-lasting and important effects. ... 3/

... [A]s part of my training efforts including my discussion and training of forensic examiners, I have learned there is no single Commission recommendation more important for the practice of forensic science than the recommendation regarding universal accreditation. I have been told the Department's decision to publicly announce the policy on accreditation and to mandate our prosecutors to rely on accredited labs when practicable has made a difference in laboratories and moving to accreditation. These recommendations and the Department's review and implementation are a demonstration of the measurable impact of the work of this Commission over the past three years, and for that as well as many other products of this Commission, the Department thanks you.

To identify the elephant in the room, everyone knows the Commission’s charter is expiring this month, and it probably won't be a surprise to learn the charter will not be renewed. As part of any transition, it is critical to re-evaluate and realign resources to achieve a new administration’s priority. Attorney General Sessions has announced his commitment to reducing violent crime in America particularly in our cities, and he has identified the troubling rise in crime as a focus of the Department when he formed the Task Force on Crime Reduction and Public Safety and established a Forensic Science Subcommittee to the Task Force to fight against this increase in crime.

The Task Force and its various subcommittees including the subcommittees on hate crime and on forensic science advise our internal Department working groups with representation for relevant components including laboratories and prosecuting entities. Although these are internal in nature, they are each seeking relevant external stakeholder input. The forensic science subcommittee in particular has been tasked with considering how we will continue to advance the purposes of this Commission in a manner consistent with the Department’s forensics priorities and its policy to reduce crime in America and develop a strategic plan. We plan to consider all options and closely review the Commission’s summary report and secure feedback from the Commissioners and other stakeholders. We will consider all the information before we decide how to move forward.

Today I'm announcing three actions that will inform the Forensic Science Subcommittee's development of a strategic plan on forensics. First, in the coming week the Department will appoint a senior forensic advisor to interface with forensics science stakeholders, advise Department leadership and work with the Subcommittee to develop a strategic plan. The strategic plan will consider questions critical to increasing capacity and ensuring access to high-quality forensic analysis. Some of the questions that will be considered include the following: What are the biggest needs in forensic science inside the Department and outside the Department? Is there more for a body like the Commission to accomplish, or would next steps be better undertaken by some other body? What specific support do Department laboratories and prosecutors need? What does the partner community need? What is required to improve practices? What are the barriers, legal practical or otherwise, and what resources do we need to overcome those barriers? Is the structure sufficient to set standards, or is some other body needed? What is needed to improve capacity so every prosecutor can be assured he or she will receive prompt results when he or she submits evidence for testing? What resources and relationships can the Department best draw on to get thoughtful advice? What is the Department currently doing to advance this issue? Are their better ways to support state and local practitioners?

The second major part of this initiative I announce is that we are publishing an issue for comment in the Federal Register seeking broad stakeholder input on just those questions I went through and what the Department should consider after the expiration of the Commission. That notice will be open until June 9th. We invite you to submit comments and encourage you to share this notice broadly.

Third, the Department is conducting a needs assessment of forensic laboratories. As you know, in December 2016, Congress passed the Justice for All Authorization Act which has several mandates to improve and advance forensic science. The needs assessment will examine serious issues of capacity and backlog at public crime labs and in the medical-legal investigation community. It will consider other topics such as research and coordination necessary when developing a strategic plan to address the needs of the forensic science community.

At the same time, the Department is considering the previously announced projects of forensic sciences of the review and the uniform language for testimony in reports and identifying where they may fit in the subcommittees’ work. We expect this process to develop a strategic plan to be deliberate and thorough but not an endless one by any means. We have every expectation of announcing how we will continue to meet these goals in the coming months.

I know the expiration of the Commission’s charter does not impact the — and the Department supports the work — and is coordinating with NIST in whether the MOU [Memorandum of Understanding] needs to be amended. I want to emphasize at the Department we recognize our responsibility to work tirelessly, improve the work we do, and enhance the administration of justice. Part of that responsibility is to ensure we are regularly coordinating with the right people on these issues and acting in a manner that demonstrates our commitment to fair play and honest dealings in every matter we handle. We will work to understand lessons of this Commission and continue to advance our goals.

Again, the Department thanks you for your contributions and emphasizes we are not finished relying on you yet. Please expect to work with us in the coming months and review, share and respond to any public inquiries. The commitment of people in this room, the time and participation over the last three years was exemplary and represents what we are capable of doing when we work together towards a single unified goal. There is no question forensic sciences one of the most critical tools we have to reduce crime, increase public safety and it will remain a priority in the Department. In order to turn back rising crime, we need to rely on you working together. The federal government intends to use its money, research and expertise to help us figure out what your needs are and determine the best ways to ensure forensic science is accurate, reliable and available to law enforcement and prosecutors to fight crime and the Department of Justice intends to do that.

The new challenge of violent crime in our nation is real and the task in front of us is clear. We need to resist temptation to ignore this or downplay it. We need to tackle it head-on to ensure justice and safety for all Americans. The Department's pledge to identify strategic plans going forward reflects this commitment to justice and the rule of law. In maintaining the public’s confidence in the accurate and reliable forensic science analyses, we need to clear the innocent and convict the guilty. On behalf of the Attorney General, the acting Deputy Attorney General, and the men and women of the Department of Justice, I thank you once again for your efforts.

NOTES

1. The Federal Advisory Committee Act of 1972 limits such commissions to two-years of operations unless renewed.
2. The renewed charter described the duration as “indefinite” but added that "[t]he Commission's termination date is two years from the date this Charter is filed with Congress, and is subject to renewal in accordance with Section 14 of FACA [the Federal Advisory Committee Act]."
3. The reference is to the Supplemental Guidance for Prosecutors Regarding Criminal Discovery Involving Forensic Evidence and Experts, Jan. 5, 2017.  The encomium judiciously pretermits the friction with DOJ that NCFS’s work on pretrial discovery initially generated. The Commission produced nothing of much substance until 2015. That year began on a low note when DOJ tried to block NCFS from voting on a draft recommendation to have DOJ laboratories open their files in criminal cases to defense lawyers even when the Federal Rules of Criminal Procedure did not demand such access. The one federal judge on the Commission resigned in protest; newly appointed Deputy Attorney General Sally Yates rescinded the ruling that the NCFS was exceeding its mandate; Judge Jed Rakoff rejoined the group; NCFS approved the draft recommendations; and DOJ responded by issuing the Supplemental Guidance. See http://for-sci-law.blogspot.com/2015/01/justice-department-reverses-decision-on.html.

Fact Check: The National Commission on Forensic Science Vote That Wasn't

Forensic Magazine continues to report that a majority of the National Commission on Forensic Science voted in favor of its own dissolution. In a mostly recycled paragraph from an earlier article, 1/ its senior science writer, Seth Augenstein, wrote today that “the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote.” 2/

The Commission never took any vote on whether it would be a good idea to extend the Commission's life. The question put to a vote was whether to include a statement to this effect in an historical document summarizing the activities of the Commission. 3/ The subject of the vote could not have much clearer. 4/ The meeting synopsis states

[A] vote was taken to determine whether this summary report should include a statement that the Commission should continue in its current form. As a business document a simple majority of 50% “yes” votes was required to approve inclusion of this statement. A total of 42% “yes” votes were received, and therefore no statement would be included regarding the continuation of the Commission. 5/

The precise question posed and the complete vote on it were as follows: 6/

Document or Vote Question Asked	Total Votes	# Yes	# No	# Abstain
Does the NCFS Summary Report include a sentence that NCFS continues in its current form?	38	16	15	7

NOTES

1. Seth Augenstein, Final Meeting of National Commission on Forensic Science ‘Reflects Back,’ Apr. 10, 2017, 11:59am, http://www.forensicmag.com/news/2017/04/final-meeting-national-commission-forensic-science-reflects-back. The paragraph stated that

   The NCFS produced 45 documents and recommendations in three years of work, which encompassed 600 public comments. But the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote."

2. Seth Augenstein, Even Without Forensic Commission, Forensic Science Overhaul Proceeds at OSAC, Apr. 13, 2017, 12:12pm, http://www.forensicmag.com/news/2017/04/even-without-forensic-commission-forensic-science-overhaul-proceeds-osac. The latest paragraph states that

   The NCFS, by the end of its last meeting on Tuesday, produced 45 documents and recommendations in three years of work—many of which directed OSAC’s explorations into forensic disciplines. But the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote. Sessions announced that it would not be renewed on Monday.

   The additions are also inaccurate. Very few of the NCFS Views documents and Recommendations documents seem to have "directed OSAC's explorations."
3. Reflecting Back—Looking Toward the Future, Dec. 16, 2016 (draft), https://www.justice.gov/ncfs/page/file/921431/download.
4. The discussion as recorded on the meeting webcast includes the following (with intervening speaker statements omitted without ellipses):

   HON. PAM KING: This is a business record ... of this particular Commission. ... This is a document that does not take any real position as to whether something should or should not be done. ...I did get some comments from Commissioners before this meetings ... One of the ones that I really would like to get some discussion on is [the] strong feelings among some Commissioners that maybe we do want to make a statement about whether or not this Commission should continue. ...
   JULIA LEIGHTON: I would not shy away from a recommendation ... I think to scrap it altogether ... is to give up on the work we’ve done.
   GERALD LAPORTE: So I don’t agree — disagree — with anything Julia has said. ... but I don’t know if we really are in a position to make a recommendation ... .
   ARTURO CASADEVALL: I want to support what Julia said. Commissions like this develop an institutional memory. ... I strongly think we should make a recommendation that something like this continue.
   S. JAMES GATES: [A]bsent a committee like this, I don’t see a consistent driver for making progress. ...
   MATTHEW REDLE: Whether it is this form or not, ... there ought to be more work done to continue the progress that we have made ...
   JULIA LEIGHTON: [W]e need a national body [with] the gravitas of being a nonpartisan federal advisory commission. ...
   HON. JED RAKOFF: ... I do think it is important that we say some something [to] indicate that we believe the Commission should continue. ...[J]udges do pay some attention to what this Commission says and does. So I think it plays a role there that is not played by other very wonderful groups ... and some very wonderful reports. I would very much strongly encourage that we have something in there ...
   WILLIAM THOMPSON: This Commission is uniquely well situated to address those [human factors] issues ... so I hope the Commission continues to address those kinds of questions ... .
   JULES EPSTEIN: So ... for this concluding portion ... yeah, we should keep going in some shape or form. ... [M]ore needs to be done. More constituencies will look to us than to other segregated constituencies. [T]he federal advisory commission should continue.
   WILLIE MAY: Certainly, I think that the Commission’s work is not completed. [I]t would serve the country very well to continue this ... .

5. National Commission on Forensic Science Meeting #12, Jan. 9-10, 2017, at 6, https://www.justice.gov/ncfs/page/file/953106/download
6. Id. at 10.

Whither OSAC? NIST's Plans for Forensic Science Standards and Research As Told to the NCFS

The National Commission on Forensic Science held its thirteenth and final meeting on Monday. The second speaker to discuss some of the administration's plans for improving forensic science was the Acting Director of the National Institute of Standards and Technology (NIST), Dr. Kent Rochford. I edited and abridged the computer-generated transcript slightly. It includes a question from Commissioner Peter Neufeld. Finally, there is a question from Commissioner Jules Epstein to a Justice Department official about future funding for OSAC. I cannot promise riveting reading, but for anyone who wants to know what was said, here is most of it:

KENT ROCHFORD: I'd like to address the future of OSAC. OSAC was conceived under the 2013 MOU [Memorandum of Understanding] between NIST and the Department of Justice and established the Commission. The Department of Justice provides funding for the OSAC, which NIST cannot sustain on its own. The OSAC organization does not have term limits but does require funding to continue.

From the introduction of OSAC, NIST addressed the need to evolve and eventually spin off OSAC. We termed this “OSAC 2.0.” We have learned a lot from OSAC 1.0. Over the past years of operation, the organization has continued to mature as members of the group have come to a better appreciation of the standards development process. One example was seen in interested key researchers and scientists joining the FSSB [Forensic Science Standards Board]. Thank you for your assistance in supporting and strengthening the OSAC.

NIST is committed to improving OSAC, including the establishment of a clear model that will support these important goals. We are working to create a stable, sustainable operational model that provides independence from NIST. Internally a small group led by Rich Cavanaugh, who runs our special programs office, has been exploring model concepts for OSAC 2.0.

Each model is distinct yet consistent with the following goals: The new OSAC has to have a defined structure and authority. It needs to engage key stakeholders. We need to provide free access to our products. There has to be a smooth transition from the current OSAC that would create the potential for long-term sustainability. Currently, Rich's group has been looking at three models, exploring further. These involve creating federal and state partnerships that develop codes, standards, and model laws. Restructuring the OSAC so subcommittee functions are dispersed to standards development organizations, and the roles at the FSSB and SAC levels are changed to focus on quality of science and utility, respectively. And establishing a development and testing — a process we are starting, and we intend to engage the broader community to better understand the strengths and weaknesses of these possible approaches. So if you have questions about the OSAC 2.0, please reach out to Rich Cavanaugh.

I'd like to talk about NIST research efforts in forensic science. NIST remains committed to remaining its measurements and standards expertise to challenges in forensics. We played a role in strengthening forensic science since at least the 1920s. You may have seen the recent National Geographic article about William Souther, a physicist from NIST who played a role in numerous forensic cases during the 1930s, including the famous Lindbergh baby kidnapping case. The current forensic research focuses include, DNA, digital fingerprint evidence, ballistics, statistics, toxins, and trace evidence. We plan to continue working these research areas as funding is available to do so. You will see an example of how research expertise provides benefit to the forensic science community when [Dr. Elham Tabassi] talks to you about development of an ISO standard on method validation.

Let me turn to technical merit review. This past September, the President's Council of Advisors on Science and Technology recommended an expanded role for NIST in assessing the scientific foundations and maturity of various forensic disciplines. We recognize the need for and the value of such studies and are exploring ways to conduct work in this area. Without additional funding recommended by PCAST, NIST cannot make large-scale commitments to technical merit review. We are planning an exploratory study to address concerns raised by PCAST regarding DNA mixtures. This will likely involve assessing the scientific literature, developing a detailed plan for evaluating scientific validity that would include probabilistic genotyping, and assigning interlaboratory studies to measure forensic laboratory performance of DNA interpretation. These laboratory studies would build upon DNA mixture studies conducted in 2003, 2005 in 2013. NIST has a history of involving external partners in his research and standard efforts and anticipate external and internal and international collaboration.
*** In closing, I want to personally thank you for your efforts on this Commission and your commitment to strengthening forensic science through your participation in the activities of this group. Your work is made a difference, and we are grateful for your service to the nation. Thank you.
*** PETER NEUFELD: The second question for Kent, when you talked about things NIST was doing, you mentioned your current evaluation of DNA mixtures. Your predecessor stated in response to this Commission making a recommendation that NIST take on the task of making an evaluation of foundational validity and reliability of different forensic methods that they intended to do a trial. They were going to start a trial in three different areas, and the other two areas in addition to the DNA were ballistics and bitemarks. We have been told at each meeting leading up to this meeting NIST was going ahead with those trials. I noticed you only mentioned DNA. Is it still the position of NIST that they will go ahead with the trial of some ballistics and bitemarks?

KENT ROCHFORD: We still continue to do the work on ballistics and bitemarks. Given the resources we have, we're going to do the trials of the interlaboratory studies with the DNA mixtures first. Right now, the PCAST report provided a number of trials we should take on [and] it is also recommending the funding to do this. Given our current funding, we intend to start with the DNA programs. As funding may become available, we can wrap up these others areas to include trials. Currently we are doing the internal work but do not right now have the bandwidth to do the ballistics trials.
*** JULES EPSTEIN: Good morning. *** The other substantive question is, can I get clarification on OSAC? Is it now the status there is currently no further funding for OSAC?

ANSWER: No.

JULES EPSTEIN: Can we understand what is in the pipeline or the projected longevity at this moment or sustainability?

ANSWER: Right now we don't have a budget and we are in a continuing resolution. We just don't know the status so I really can't predict what it will look like.

Two Misconceptions About the End of the National Commission on Forensic Science

Several days ago, the Justice Department (DOJ) announced the end of the National Commission on Forensic Science (NCFS). Initially established to advise the Department of Justice for a two-year period, the NCFS had had its charter extended once before, in April 2015. Attorney General Sessions declined to renew it a second time.

Several explanations for this decision could be offered: (1) the current administration is unreceptive to scientific knowledge and advice — alternate facts and alternate science are more appealing; (2) DOJ does not want outside advice on producing and presenting scientific evidence; (3) DOJ is tired of spending millions of dollars for advice from this particular group of lawyers, judges, administrators, forensic scientists, and others. (4) DOJ believes that NCFS has outlived its usefulness and there are better ways to obtain advice. And of course, some combination of these things might have been at work. I have no inside information, but I thought it might be helpful to collect some of what is publicly known if only to correct misconceptions about the Commission and its role vis-a-vis the Justice Department. I'll begin by noting two such misconceptions.

Misconception 1

NCFS Was Evaluating the Validity of Forensic Science Tests and Methods

One misconception is that the Commission was conducting independent evaluations of accepted methods in forensic science. Thus, the Associated Press reported that “National Association of Criminal Defense Lawyers ... President Barry Pollack said the commission was important because it allowed ‘unbiased expert evaluation of which techniques are scientifically valid and which are not.’” 1/ But not one of the 44 documents identified as “work products” on the NCFS website examines the validity of any technique.

The two documents that directly address “technical merit” are views and then recommendations 2/ about the need to study validity and reliability of techniques. No surprise there. More importantly, the documents underscore the importance of bringing what we might call "outsider" scientific expertise to bear in these efforts, and one of them contains pointed advice to other organizations. Specifically, a recommendation calls on NIST and its creation, the Organization of Scientific Area Committees for Forensic Science (OSAC), to reform the procedure OSAC uses to review and endorse standards for test methods. It states:

The Organization of Scientific Area Committees for Forensic Science (OSAC) leadership, the Forensic Science Standards Board (FSSB), should commit to placing consensus documentary standards on the OSAC Registry of Approved Standards for only those forensic science test methods and practices where technical merit has been established by NIST, or in the interim, established by an independent scientific body. An example of an interim independent scientific body could be an OSAC-created Technical Merit Resource Committee composed of measurement scientists and statisticians appointed by NIST and tasked with the evaluation of technical merit. 3/

This recommendation, by the way, has had limited impact. Yes, NIST has announced that it will do further research in a few areas such as DNA-mixture analysis. No, OSAC has not established a Resource Committee to check the technical merit of the documents that filter up from its subject-area committees and subcommittees.  4/

Rather than performing literature reviews (or promulgating scientific standards for forensic laboratories to follow), NCFS focused on broader issues of policy, needs, and legal reforms for generating or evaluating scientific evidence. This role for the Commission relates to a second misconception.

Misconception 2

NCFS Was a Worthless “Think Tank”

According to the Washington Post,

[T]he National District Attorneys Association, which represents prosecutors, applauded the end of the commission and called for it to be replaced by an Office of Forensic Science inside the Justice Department. Disagreements between crime lab practitioners and defense community representatives on the commission had reduced it to “a think tank,” yielding few accomplishments and wasted tax dollars, the association said. 5/

A press release from the NDAA does “applaud” the DOJ’s decision not to nonrenew the Commission, but not because the NCFS was a “think tank.” The group representing “2,500 elected and appointed District Attorneys across the United States, as well as 40,000 Assistant District Attorneys” complained that

The Commission lacked adequate representation from the state and local practitioner community, was dominated by the defense community, and failed to produce work products of significance for the forensic science community. Very few of the recommendations from the Commission were adopted and signed by the previous Attorney General during its existence. Those that were signed, such as universal accreditation, had already begun to develop organically within the forensic science community as accepted best practices, thus replicating ongoing work and wasting taxpayer dollars. 6/

I have not checked the percentage of recommendations “signed” by the Attorney General, but the Commission’s views documents never were intended to be signed by anyone, and the notion that only recommendations for specific action by the Attorney General benefit “the forensic science community” is shortsighted. Among the Commission’s documents of lasting value are the following:

• Recommendation on Transparency of Quality Management System
• Recommendation on Model Legislation for Medicolegal Death Investigation Systems
• Views Document on Recognizing the Autonomy and Neutrality of Forensic Pathologists
• Recommendations on Use of the Term “Reasonable Scientific Certainty”
• Recommendation on Pretrial Discovery
• Recommendations on Use of the Term “Reasonable Scientific Certainty”
• Views Document on Judicial Vouching
• Views Document on Ensuring that Forensic Analysis is Based Upon Task-Relevant Information
• Views Document on Facilitating Research on Laboratory Performance
• Views Document on Identifying and Evaluating Literature that Supports the Basic Principles of a Forensic Science Method or Forensic Science Discipline

It is instructive to compare the NDAA's dismissal of the "universal accreditation" recommendation  with the assessment of it by Associate Deputy Attorney General Andrew Goldsmith, who stated in his remarks at the final NCFS meeting that

[T]here is no single commission recommendation more important for the practice of forensic science than the recommendation regarding universal accreditation. I have been told the Department's decision to publicly announce the policy on accreditation and to mandate our prosecutors to rely on accredited labs when practicable has made a difference in laboratories and in moving to accreditation. These recommendations and the Department's review and implementation are a demonstration of the measurable impact of the work of this Commission ... .

Naturally, many of the ideas or actions that the Commission endorsed were not original. The idea of accreditation was prominent in the 2009 National Research Council (NRC) report on forensic science as well as NRC reports on DNA evidence in 1992 and 1996. NCFS was not a think tank, but a mixed bag of administrators, prosecutors, defenders, judges, law professors, police officials, laboratory scientists, medical examiners and coroners, research scientists, and other individuals. It could be criticized as wasteful — 13 meetings of 41 members (including the ex officio ones) plus an unlisted number of nonmembers appointed to subcommittees at a cost of millions of dollars for taxpayers (not to mention the opportunity costs to the volunteers). Consequently, it certainly is fair to ask how much additional benefit would have come from another two years of Commission life. 7/ But the Justice Department does not plan to halt all study of in-house forensic science reform. It has announced that some of it will continue via a newly created -- and surely not costless -- task force run by the incoming Deputy Attorney General. Given that plan, is the restructuring really an effort to save taxpayer money because of a perception that NCFS had reached the point of diminishing returns? Or is it a move to control the agenda and to modify the list of people who provide input? More on that later.

NOTES

1. Sadie Gurman, Sessions' Justice Dep't Will End Forensic Science Commission, AP News, Apr. 11, 2017, https://www.apnews.com/c076ef99c48948e3856902cfca9e7b14/Sessions'-Justice-Dep't-will-end-forensic-science-commission
2. NCFS often prepared two “work products” per topic for its recommendations — a preliminary “views” document followed by a final, more concrete  “recommendations” document. Consequently, the total number of its "work products" is a poor quantitative measure of its accomplishments.
3. Recommendation to the Attorney General: Technical Merit Evaluation of Forensic Science Methods and Practices, Dec. 9, 2016, at 3, https://www.justice.gov/ncfs/page/file/905541/download
4. A more cheerful description of the response from NIST and OSAC can be found in a letter from the six research scientists (not forensic scientists) on the Commission pleading for a renewal of the charter.
5. Spencer Hsu, Sessions Orders Justice Dept. To End Forensic Science Commission, Suspend Review Policy, Wash. Post, Apr. 10, 2017, https://www.washingtonpost.com/local/public-safety/sessions-orders-justice-dept-to-end-forensic-science-commission-suspend-review-policy/2017/04/10/2dada0ca-1c96-11e7-9887-1a5314b56a08_story.html?utm_term=.c24306818156
6. NDAA, Press Release, National District Attorneys Association Applauds Expiration of National Commission on Forensic Science, Apr. 10, 2017, http://www.ndaa.org/pdf/NDAA%20Statement%20on%20Expiration%20of%20National%20Commission%20on%20Forensic%20Science.pdf
7. Erin Murphy, Op-ed, Sessions Is Wrong to Take Science Out of Forensic Science, N.Y. Times, Apr. 11, 2017, https://www.nytimes.com/2017/04/11/opinion/sessions-is-wrong-to-take-science-out-of-forensic-science.html (asserting that NCFS "was even poised to issue a raft of best practices for the wild west of digital forensics, which has exploded without supervision over the years.")