Saturday, May 20, 2017

Science Friday and Contrived Statistics for Hair Comparisons

On May 19th, Public Radio International's Science Friday show had a segment entitled "There’s Less Science In Forensic Science Than You Think." The general theme — that some practices have not been validated by rigorous scientific testing — is a fair (and disturbing) indictment. But listeners may have come away with the impression that the FBI has determined that hair examiners make up statistics from personal experience 95% of the time to help out prosecutors.

Ira Flato, the show's host, opened with the observation that "The FBI even admitted in 2015, after decades, investigators had overstated the accuracy of hair sample matches over 95% of the time in ways that benefited the prosecution." He returned to this statistic when he asked Betty Layne DesPortes, a lawyer and the current President of the American Academy of Forensic Sciences, the following question:
Dr. DesPortes, I want to go back to that FBI admission in 2015 that for decades investigators had overstated the accuracy of their hair samples, and I mean 95% of the time in a way that benefited the prosecution. Is this a form of cognitive bias coming into the picture?
Ms. DesPortes replied that
It is, and ... you would have overstatement along the lines of, "Well, I’ve never seen in my X years of experience that two hairs would be this similar, so it must be a match," and then they would just start making statistics up based on, "Well, I’ve had a hundred cases in my practice, and there have been a thousand cases in my lab, and nobody else has ever reported similar hairs like this," so let’s just start throwing in one in a hundred thousand as a statistic — "one in a hundred thousand" — and that’s where the misstatement came in.
But neither Ms. DesPortes nor anyone else knows how often FBI examiners cited statistics like "one in a hundred thousand" based on either their recollections of their own casework or their impression of the collective experience of all hair examiners. 1/

To be sure, such testimony would have been flagged as erroneous in the FBI-DOJ Microscopy Hair Comparison Review. But so would a much more scientifically defensible statement such as
The hair removed from the towel exhibited the same microscopic characteristics as the known hair sample, and I concluded it was consistent with having originated from him. However, hair comparison is not like fingerprints, for example. It’s not a positive identification. I can’t make that statement." 2/
The Hair Comparison Review was not designed to produce a meaningful estimate of an error rate for hair comparisons. It produced no statistics on the different categories of problematic testimony. The data and the results have not been recorded (at least, not publicly) so as to allow independent researchers to ascertain the extent to which FBI examiners overstated their findings in various ways. See David H. Kaye, Ultracrepidarianism in Forensic Science: The Hair Evidence Debacle, 72 Wash. & Lee L. Rev. Online 227 (2015).

The interim results from the Hair Comparison Review prompted the Department of Justice to plan a retrospective study of FBI testimony involving other identification methods as well. In July 2016, it asked a group of statisticians how best to conduct the new "Forensic Science Disciplines Review." The informal recommendations that emerged in this "Statisticians' Roundtable" included creating a database of testimony that would permit more rigorous, social science research. But this may never happen. A new President appointed a new Attorney General, who promptly suspended the expanded study.

NOTES
  1. Ms. DesPortes may not have meant to imply that all the instances of exaggerated testimony were of the type she identified.
  2. That statements like these may be scientifically defensible does not render them admissible or optimal.
(For related postings, click on the label "hair.")

Tuesday, May 16, 2017

The Reappearing Rapid DNA Act

With bipartisan sponsorship, the Rapid DNA Act of 2017 (H.R.510 and S. 139) is sailing through Congress. The Senate bill made it to the legislative calendar on May 11, 2017, without amendment and without a written report from the Judiciary Committee.  The Committee Chairman, Senator Grassley, wrote this about the bill:
Turning to legislation, the first bill is S.139, the Rapid DNA Act of 2017. It is sponsored by Senator Hatch. The Committee reported this bill and the Senate passed it in the last Congress. The bill would establish standards for a new category of DNA samples that can be taken more quickly and then uploaded to our national DNA index. 1/
This characterization is misleading. The bill itself contains no standards for producing profiles to upload to the national database. It orders the FBI to “issue standards.” Specifically, the part of the bill entitled “standards” adds to the DNA Identification Act of 1994, 42 U.S.C. § 14131(a), a new Section 5, which reads as follows:
(A) ... the Director of the Federal Bureau of Investigation shall issue standards and procedures for the use of Rapid DNA instruments and resulting DNA analyses.
(B) In this Act, the term ‘Rapid DNA instruments’ means instrumentation that carries out a fully automated process to derive a DNA analysis from a DNA sample. 2/
But the FBI does not need new authorization to devise standards for “Rapid DNA instruments.” The “resulting DNA analyses” are not a new category of “samples,” and some such profiles already may be in the National DNA Index System (NDIS). In fact, the FBI issued standards for “rapid” profiles years ago. One need only peek at the FBI's forthright answers to “Frequently Asked Questions on Rapid DNA Analysis.” There, the FBI explained that
Based upon recommendations from the Scientific Working Group on DNA Analysis Methods (SWGDAM), the FBI Director approved and issued The Addendum to the Quality Assurance Standards for DNA Databasing Laboratories performing Rapid DNA Analysis and Modified Rapid DNA Analysis Using a Rapid DNA Instrument (or “Rapid QAS Addendum”). The Addendum contains the quality assurance standards specific to the use of a Rapid DNA instrument by an accredited laboratory; it took effect December 1, 2014.
The FBI added that “[a]n accredited laboratory participating in NDIS may use CODIS to upload authorized known reference DNA profiles developed with a Rapid DNA instrument performing Modified Rapid DNA Analysis to NDIS if [certain] requirements are satisfied” and that “DNA records generated by an NDIS-approved Rapid DNA system performing Rapid DNA analysis in an NDIS participating laboratory are eligible for NDIS.” 3/

But if the FBI does not need the bill to develop standards or to incorporate rapid-DNA results into NDIS, what is the real purpose of the bill? The answer is simple. The bill clears the way for these results to come, not from accredited laboratories, 4/ but from police stations, jails, or prisons. The House Judiciary Committee was explicit in its brief report on the bill:
Currently, booking stations have to send their DNA samples off to state labs and wait weeks for the results. This has created a backlog that impacts all criminal investigations using forensics, not just forensics used for identification purposes. H.R. 510 would modify the current law regarding DNA testing and access to CODIS. The short turnaround time resulting from increased use of Rapid DNA technology would help to quickly eliminate potential suspects, capture those who have committed a previous crime and left DNA evidence, as well as free up current DNA profilers to do advanced forensic DNA analysis, such as crime scene analysis and rape-kits. 5/
The FBI was more succinct when it referred to “the goal of using Rapid DNA systems in the booking environment” and reported that “legislation will be needed in order for DNA records that are generated by Rapid DNA systems outside an accredited laboratory to be uploaded to NDIS.6/

Is the migration of DNA profiling from the laboratory to the police station — and potentially to the officer on the street — a good idea? The efficiency argument from the House Committee has some force. We do not demand that only accredited laboratories conduct breath alcohol testing of drivers who seem to be intoxicated. Police using properly maintained portable instruments can do the job. 7/

How is DNA different? In one respect, it is less problematic than roadside alcohol testing. Rapid DNA analysis is not for crime-scene samples. (At least, not yet.) It is for samples from arrestees or convicted offenders whose profiles can be uploaded to a database. The police have an incentive to avoid uploading inaccurate profiles. Such profiles will degrade the effectiveness of the database. Any cold hits that they might produce will be shown to be false when a later DNA test from the suspect fails to replicate the incorrect profile. In contrast, incriminating output of a faulty alcohol test usually enables a conviction and will not be shown to be in error.

But there is more to the matter than efficiently generating and uploading profiles. It could be argued that DNA information is more private that a breath alcohol measurement and that having CODIS profiles known to local police is more dangerous than having it known only to laboratory personnel. Considering the limited kind of information that is present in a CODIS profile, however, this argument does not strike me as compelling.

POSTSCRIPT

The Rapid DNA Act of 2017 met no opposition as the Senate and House passed the bills. S. 139 generated unanimous consent (and no discussion) on May 16. 8/ Its counterpart, H.R. 510, passed after receiving praise from two of its sponsors and the observation from Representative Goodlatte (R-VA) that "this is a good bill. It is a bipartisan bill. I thank Members on both sides of the aisle for their contributions to this effort." 9/

NOTES
  1. Prepared Statement by Senator Chuck Grassley of Iowa, Chairman, Senate Judiciary Committee Executive Business Meeting, May 11, 2017, https://www.judiciary.senate.gov/imo/media/doc/05-11-17%20Grassley%20Statement.pdf, viewed May 16, 2017.
  2. Rapid DNA Act of 2017, S. 139 § 2(a).
  3. The difference between “Rapid DNA Analysis” and “Modified Rapid DNA Analysis” is that the former is “a “swab in – profile out” process ... of automated extraction, amplification, separation, detection, and allele calling without human intervention,” whereas the latter uses “human interpretation and technical review” for ascertaining the alleles in a profile. FBI, Frequently Asked Questions on Rapid DNA Analysis, https://www.fbi.gov/services/laboratory/biometric-analysis/codis/rapid-dna-analysis, Nos. 1 &2, viewed May 17, 2017.
  4. The DNA Identification Act of 1994, 42 U.S.C. § 14131, which the Rapid DNA Act amends, requires the FBI to create and consider the recommendations of "an advisory board on DNA quality assurance methods." § 14131(a)(1)(A).  The members of the board must come from "nominations proposed by the head of the National Academy of Sciences and professional societies of crime laboratory officials." Id. They "shall develop, and if appropriate, periodically revise, recommended standards for quality assurance, including standards for testing the proficiency of forensic laboratories, and forensic analysts, in conducting analyses of DNA." § 14131(a)(1)(C). As the name indicates, the board is purely advisory. The Act only demands that
    The Director of the Federal Bureau of Investigation, after taking into consideration such recommended standards, shall issue (and revise from time to time) standards for quality assurance, including standards for testing the proficiency of forensic laboratories, and forensic analysts, in conducting analyses of DNA.
    § 14131(a)(2).
    The advisory board was a half-a-loaf response to the recommendation of a National  Academy of Sciences committee for "a National Committee on Forensic DNA Typing (NCFDT) under the auspices of an appropriate government agency, such as NIH or NIST, to provide expert advice primarily on scientific and technical issues concerning forensic DNA typing." NRC Committee on DNA Technology in Forensic Science, DNA Technology in Forensic Science 72-73 (1992). Now that NIST has established an Organization of Scientific Area Committees for Forensic Science to develop science-based standards for DNA testing and other forensic science methods, Congress should reconsider the need for the overlapping FBI board.
  5. On May 11, 2017, the House Committee on the Judiciary recommended adoption of H.R. 510 without holding hearings. The Judiciary Committee saw no need to consult independent scientists. It was satisfied with the fact that
    the Judiciary Committee’s Subcommittee on Crime, Terrorism, Homeland Security and Investigations held a hearing on a virtually identical bill, H.R. 320, on June 18, 2015, [at which] testimony was received from: Ms. Amy Hess, Executive Assistant Director of Science and Technology, Federal Bureau of Investigation; Ms. Jody Wolf, Assistant Crime Laboratory Administrator, Phoenix Police Department Crime Laboratory, President, American Society of Criminal Laboratory Directors; and Ms. Natasha Alexenko, Founder, Natasha’s Justice Project.
    Report to accompany H.R. 510, May 11, 2017, https://www.congress.gov/115/crpt/hrpt117/CRPT-115hrpt117.pdf
  6. FBI Answers, No. 13, https://www.fbi.gov/services/laboratory/biometric-analysis/codis/rapid-dna-analysis, viewed May 17, 2017 (emphasis added).
  7. “As of January 1, 2017, there is no Rapid DNA system that is approved for use by an accredited forensic laboratory for performing Rapid DNA Analysis.” Several systems had been approved but they do “not contain the 20 CODIS Core Loci required as of January 1, 2017.” FBI Answers, No. 6, https://www.fbi.gov/services/laboratory/biometric-analysis/codis/rapid-dna-analysis, viewed May 16, 2017. 
  8. 163 Cong. Rec. S2954-2955, 115th Cong., 1st Sess., May 16, 2017.
  9. Id. at H4205.

Saturday, May 6, 2017

Who Copy Edits ASTM Standards?

This posting is not about science or law. It is about English writing. I recently had occasion to read the “Standard Guide for Analysis of Clandestine Drug Laboratory Evidence” issued by ASTM International, a private standards development organization. The standard exemplifies a common problem with the ASTM standards for forensic science — an apparent absence of copy and line editing to achieve clear and efficient expression of the ideas of the committees that write the standards. 1/

This particular standard, known as E2882-12, opens with an observation about the “scope” of the document — namely, that
This guide does not replace knowledge, skill, ability, experience, education, or training and should be used in conjunction with professional judgment.
The word “replace” has caused a couple of readers to complain that this admonition implies that unstructured “knowledge, skill, ability, experience, education, or training” suffices for the analysis of the evidence. That is not  a fair reading of the sentence, but joining the two clauses with “and” makes it seem like they are separate points. Why not make it as easy as possible for the reader to get the intended message? I think the sentence amounts to nothing more than the following simple idea:
This standard is intended to help professionals use their knowledge and skill to analyze clandestine drug laboratory evidence.
Why not just say this? Why all the extra verbiage?

Unfortunately, this text is not an isolated example of the need for detailed editing. Another infelicity is
capacity—the amount of finished product that could be produced, either in one batch or over a defined period of time, and given a set list of variables.
The words “and given a set list of variables” are a sentence fragment. They dangle aimlessly after the comma. The copy edit is obvious:
Capacity is the amount of finished product that could be produced, for a specified set of variables, either in one batch or over a stated period of time.
It still may not be clear what a “set of variables” means here, but at least the words about unnamed variables occur where they belong.

The wording in a section on reporting is especially obscure:
Laboratories should have documented policies establishing protocols for reviewing verbal information and conclusions should be subject to technical review whenever possible. It is acknowledged that responding to queries in court or investigative needs may present an exception.
One clear statement of what the sentences seem to assert is that
Laboratories should have written protocols to ensure that oral communications from laboratory personnel are reviewed for technical correctness. However, a protocol can dispense with (1) review of some courtroom testimony and (2) review that would impede an investigation.
Whether this edited version expresses what the authors wanted to say or presents a satisfactory policy is unclear, but at least the version is more easily understood.

Other phrases that should raise red flags for editing abound. I’ll end with three examples.
  • This guide does not purport to address all of the safety concerns, if any, associated with its use. The editor would say: Make up your mind. If there are no safety concerns, then the sentence is worthless. If there are safety concerns, then the standard should address them. If there is a reason not to address all of them, then the standard can say, “There are additional safety concerns for a laboratory to consider.” If there is a desire to be very cautious, it could read, “There could be additional safety concerns for a laboratory to consider.”
  • ... calculations can be achieved from ... . Copy editor: It sounds odd to speak of "achieving" calculations. The phrase "calculations can be made by" would be more apt.
  • Quantitative measurements of clandestine laboratory samples have an accuracy which is dependent on sampling and, if a liquid, on volume calculations. This sentence is both circumlocutious ("which is dependent") and disjointed ("if a liquid" is in the wrong place to modify "samples"). It also seems to conflate measurements on subsamples of the material submitted for analysis ("clandestine laboratory samples") with inference from the subsamples to the sample of the seized items. If this reading of the dense sentence is correct, editing would expand it along the following lines: "The accuracy of quantitative measurements of a liquid sample depends on the calculated volume of the sample. When the material analyzed is not the entire sample, then the accuracy of any inferences to the entire sample also depends on the homogeneity of the sample and the procedure by which the subsample was chosen.
Good writing requires the right words in the correct order. Good editing makes the writing more readable. Many existing technical standards in forensic science still need good editing to make them fully fit for purpose.

NOTE
  1. Although some publishers distinguish between line editing and copy editing, this posting uses the phrase "copy editing" broadly, to refer to the process of reviewing and correcting written material to ensure "that whatever appears in public is accurate, easy to follow, and fit for purpose." Society for Editors and Proofreaders, FAQs: What Is Copy-editing?, https://www.sfep.org.uk/about/faqs/what-is-copy-editing/.

Friday, April 28, 2017

Are "Exclusions" Deductive and "Identifications" Merely Probabilistic?

Lately I have heard people say that “source exclusions” are the product of deductively valid reasoning, whereas “source identifications” are less certain. But the difference between such conclusions does not arise from the fact that one is deductive and the other is not. On reflection, "exclusions" are no less probabilistic in nature than "identifications." 

Presumably, the reason people may think that exclusions are deductions is that “exclusion” can be part of a deductive argument. For example,
(1) No human being with Type O blood will leave Type A blood at a crime scene.
(2) Defendant has Type O blood.
(3) The crime-scene bloodstain is Type A.
Therefore,
(4) Defendant is excluded as the source of the stain.
This argument is a formally valid deduction. If the premises (1)–(3) are true, the conclusion (4) must be true. There is, however, no guarantee that any or all the premises are true. Perhaps something very strange (but not logically impossible) happened to convert a Type O stain into a Type A one. Or perhaps the defendant or the stain was mistyped. These are not very likely events, but deductive logic does not make them true. So although (4) is certain to be true conditional on (1)–(3), we cannot be absolutely certain that (4) is in fact true. In the symbolism of probability statements, the fact that Pr[(4) | (1)&(2)&(3)] =1 does not ensure that Pr(4) = 1 unless Pr(1) = Pr(2) = Pr(3) = 1.

One might think that I have misstated the argument. Indeed, the argument ending in exclusion might be reframed as follows:
(1) Everything we know tells us that no human being with Type O blood will leave Type A blood at a crime scene.
(2) A blood test shows that defendant has Type O blood.
(3) A blood test shows that the crime-scene bloodstain is Type A.
Therefore,
(4) The blood test excludes defendant as the source of the blood stain.
This too is a deductively valid argument. The premises entail the conclusion, and the premises are even harder to dispute than the ones in the previous example. But if this is all that a criminalist means by an exclusion, then an exclusion is not actually a statement that the defendant is not the source of the trace. The conclusion in our second argument only asserts that the test has excluded the defendant. Unless the test never errs, it does not follow (deductively) that the defendant was not the source of the bloodstain. To appreciate the force of the "deduction," we need to study how often criminalists report exclusions when examining items from different sources as opposed to items from the same source.

The situation is the same for “identification.” This conclusion also can come at the end of a deductive argument. For example,
(1) Fingerprints from the same finger always match.
(2) Fingerprints from different fingers never match.
(3) The questioned and known fingerprints being compared match.
Therefore,
(4) The fingerprints being compared are from the same finger — an “identification.”
As with an exclusion, the argument from (1)–(3) to (4) is logically impeccable. If (1)–(3) are true, then so is (4). And, once more, because propositions (1)–(3) might not all be true, the truth of the “identification” is not absolutely certain.

Again, we can rephrase the argument in a (vain) effort to make it appear that the desired conclusion is purely deductive:
(1) Fingerprints from the same finger always match.
(2) Fingerprints from different fingers never match.
(3) The questioned and known fingerprints being compared match.
Therefore,
(4) I have identified the questioned print as coming from the finger that left the known print.
But again, the deductive argument does not get us to the conclusion that the known finger is the source of the questioned print. To ascertain the probative value of a positive source classification (an “identification”), we need to study the performance of criminalists making these source attributions. We need to study how often criminalists report “identification” when examining items from the same source as opposed to items from different sources. In the end, if there is a difference in the certainty we can attach to an exclusion as opposed to an identification, it does not emanate from the difference between inductive and deductive forms of argument. It results from the fact that the premises of some inductive arguments are more probably true than the premises of other inductive arguments.

Reference
  1. Brian Skyrms, Choice and Chance: An Introduction to Inductive Logic (4th ed. 2000).

Wednesday, April 26, 2017

A Superficial Opinion on Fingerprints in Missouri Has a Few Interesting Wrinkles

In State v. Hightower, 1/ the Missouri Court of Appeals added to the list of superficial opinions on the admissibility of latent fingerprint matches. In this case, a man pointed a gun at the driver of a car, snatched a purse and an iPad, fired into the air, and escaped. The driver did not see the robber’s face, but a detective lifted two “relatively new and undisturbed” prints from the driver’s window. A latent fingerprint examiner with the St. Louis County Police Department used a state AFIS (automated fingerprint identification system) to arrive the conclusion that the latent prints “were left by Defendant's left middle and ring fingers.” On the basis of this identification, a jury convicted David Hightower of armed robbery, and the trial court sentenced him to serve 18 years in prison.

At a pretrial hearing on general scientific acceptance, Dr. Ralph Haber, a research psychologist and “forensic scientist and expert witness” (resume, at 1) testified for the defendant that (as the state court of appeals put it) “the National Academy [of Science] and the National Institute [of Standards and Technology] have both decried the reliability and accuracy of fingerprint evidence adduced using the ACE-V method.” Apparently, he was referring to the well known 2009 report of the NAS Committee on Identifying the Needs of the Forensic Science Community and the report of the NIST Expert Working Group on Human Factors in Latent Print Analysis (D.H. Kaye ed., 2012). 2/ He also “testified he has been asked to serve on the National Commission [on Forensic Science] committee responsible for developing standards for fingerprint analysis.” 3/ The trial court was more impressed by his admission that “in every hearing he had been involved in to exclude fingerprint evidence the evidence had been deemed admissible, save one case from Maryland.” It denied the defendant’s motion to exclude the evidence.

At trial, Dr. Haber “concluded that a person could not be identified with 100% certainty based on a fingerprint,” but acknowledged on cross-examination that “he had not looked at the actual fingerprints in the present case.” When the prosecutor argued in a closing statement that “other ... experts in latent fingerprint examinations ... could have examined this but were not asked to,” the judge instructed the jury to disregard the statement.

The court of appeals assumed that the comment was improper. In a typical display of judicial unrealism, the appellate court blithely “presume[d] the court's curative instructions to the jury removed any prejudice from the prosecutor's statements.” But was the prosecutor in the wrong in the first place? Although the defense certainly has no obligation to examine fingerprints, I do not think the answer is entirely obvious — particularly when the defense produces an expert who testifies that the identification is wrong or uncertain. Here, Dr. Haber apparently choose not to examine the prints although his resume prominently advertises 120 hours of “fingerprint comparison training.” Of course, the prosecutor's comment referred not just to the testifying defense expert’s work, but to the defendant’s decision not to call on still other examiners. Moreover, the defense theory was not that other examiners would disagree with the state’s expert, but only that the meaning of an agreed-upon match is unclear.

If the meaning of a match is indeed unclear — because the entire process has not been fully validated — then it is hard to see how the evidence is generally accepted in a relevant scientific community. The opinion in Hightower does not respond to that argument. Instead, the court maintained that judges have always found subjective comparisons of fingerprints sufficient to demonstrate singular identity. But very few of these opinions have asked whether the scientific literature evinces general acceptance of the proposition that latent print examiners as a group can reliably and accurately match prints of the quality of the ones in this case. Without that showing, how can general scientific acceptance be said to exist?

Fortunately, there are empirical studies of the process that help address this question. These studies appear in reputable scientific journals. 4/ They should inform rulings on admissibility, and the scientific findings should be used to help convey the degree of certainty in any source conclusions.

Finally, the court made short work of Hightower's argument that the conviction could not rest the fingerprint identification alone, at least not without "additional evidence indicating the fingerprints could have only been impressed at the time the crime was committed." The court wrote that
This argument is without merit. “[A] fingerprint at the scene of the crime may in and of itself be sufficient to convict.” State v. Bell, 62 S.W.3d 84, 96 (Mo. App. W.D. 2001). The defendant in Bell claimed that a partial palm print found in a place accessible to the public without credible evidence establishing it was left near the time of crime was insufficient evidence to sustain a conviction. Id. The Western District denied his point, stating there “was sufficient evidence to establish that the palm print on the counter was recent and occurred near the time of the crime” because the hotel clerk testified she had cleaned the counter twenty minutes prior to the robbery and no one other than herself and the robber had touched it in between the time of the robbery and the time the police lifted the print. Id. In the present case, Ms. Gillespie testified and her mother echoed that Defendant hit the car window open-palmed and the detective who collected the fingerprints stated they appeared “fresh” and it was his belief that they had been left recently.
At least one law review article has proposed that a single item of circumstantial evidence tying a defendant to a crime should not be sufficient for a conviction. Nevertheless, when the value of the evidence is great enough, a rigid, two-pieces-of-evidence rule seems too strict.

A further problem in Hightower is that it is not clear that a lay witness can discern the age of the fingerprints or that the detective possessed the necessary expertise to do so. What skill or experience did he have in dating prints? Are there any studies to establish that anyone can discern the age of prints just by shining a flashlight on them? Ascertaining how old prints might be from their physical or chemical properties always has eluded forensic science, although a promising technique has been reported. In Hightower, though, it does not appear that the defendant objected to this part of the detective's testimony.

NOTES
  1. 511 S.W.3d 454 (Mo. App. 2017).
  2. It would be fairer to say that these reports called for research to establish the probabilities of false positive and negative errors in latent print examinations and for the results of comparisons to be presented in ways that recognize the degree of uncertainty in fingerprint identifications.
  3. The Commission never established subcommittee on fingerprint analysis, and Dr. Haber is not listed as a member of any of the Commission’s seven subcommittees.
  4. Some of them are described in this blog.

The Justice Department’s Explanation for the End of the National Commission on Forensic Science

The decision of the Department of Justice to let the NCFS expire — a decision that was as predictable as the date of the next solar eclipse — was presented to the Commission at its final meeting on Monday, April 10. Everyone present knew that the NCFS was not intended to be an indefinite fixture. New administration or not,  a decision to continue operating the Commission and its subcommittees had to come by April 23, 2017. 1/ After all, the NCFS charter of April 23, 2013, asked it “to provide recommendations and advice to the Department of Justice” for two years. Former Attorney General Eric Holder renewed the charter once, on April 23, 2015. 2/ The new Attorney General, Jeff Sessions, elected not to renew the charter a second time.

No reasons for this discretionary action were provided. An Associate Deputy Attorney General graciously thanked the Commission for its work, stated that the new Attorney General had decided to use alternative mechanisms for developing departmental policy, described some aspects of what those would be, indicated that a press release was in the works, and thanked the Commission again.

Below I describe the written documents (an Executive Order and press releases) related to the sunsetting of the Commission and reproduce an abridged version of the (slightly garbled) computer-generated transcript. I made corrections to the extent I was confident about what actually was said, and I edited out some material that was not important to seeing where the Department of Justice (DOJ) may be headed.

I. The DOJ Press Release and the Task Force on Crime Reduction and Public Safety

A press release of April 10, 2017,  announced “a series of actions the Department will take to advance forensic science and help combat the rise in violent crime.” A new “Task Force on Crime Reduction and Public Safety” within the DOJ “will spearhead the development of [a] strategic plan” to “increase the capacity of forensic science providers, improve the reliability of forensic analysis, and permit reporting of forensic results with greater specificity.”

Nothing in the creation of the Task Force specifically suggested forensic science was of any concern. It emanated from an Executive Order signed on February 9, 2017. Harking back to the rhetoric of the Nixon Administration but placing illegal immigration at the top of the list of crimes to combat, this order declared that
It shall be the policy of the executive branch to reduce crime in America. ... A focus on law and order and the safety and security of the American people requires a commitment to enforcing the law and developing policies that comprehensively address illegal immigration, drug trafficking, and violent crime.
Within three weeks, Attorney General Sessions outlined the membership of the task force. The President’s order did not specify who or what types of people should comprise the Task Force. The Attorney General designated his Deputy Attorney General as its chair and “relevant Department components” to supply its members. He named “the Director of the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF), the Administrator of the Drug Enforcement Administration (DEA), the Director of the FBI and the Director of the U.S. Marshals Service (USMS)” as key members.

This group might not possess the immediate knowledge for devising a plan to “increase the capacity of forensic science providers, improve the reliability of forensic analysis, and permit reporting of forensic results with greater specificity,” although the membership could be supplemented. Along with the membership, the stated objectives seem limited to investigative matters instead of the widely ranging interests pursued by NCFS and reflected in its recommendations.

In fact, the list of objectives itself is a little puzzling. No one would argue with the ambition of improving reliability (in the sense of trustworthy results) and laboratory capacity, but what does it mean to be able to report “forensic results with greater specificity”? Reporting that one finger is the source of a latent print, that one gun is the source of given bullet, and that one set of teeth are the source of a bitemark are already as specific as one can possibly get. One might question the scientific status of such claims, as the last President’s Council of Advisers on Science and Technology did, but the Obama DOJ rejected much of the PCAST critique. Surely, the new President and Attorney General are not expressing newfound doubts about the ability of forensic-science practitioners to make specific source attributions.

II. The Associate Deputy Attorney General’s Remarks to the Commission
Good morning, everyone. Let me say it's an honor to be here on behalf of the acting Deputy Attorney General and be able to address this group. My name is Andrew Goldsmith, I'm an Associate Deputy Attorney General and the Department’s National Criminal Discovery Coordinator. ... As some of you know, about two years ago the then Deputy Attorney General asked me to work with you on the criminal discovery recommendation, and in that role I had the pleasure of working with a number of you.
...
Before I go further I’d like to talk about the Attorney General's firm commitment to forensic science. On Friday, Attorney General Sessions and I spoke in his office, and he made clear to me in his view good forensics is not only important because it enables us to convict the guilty, but also to clear the innocent. He stressed to me we need to focus on the integrity of the process where we have prompt access to high-quality forensics technology. He found troubling the backlog in forensics analysis, and as I will discuss in more depth later, as part of the Task Force on Crime Reduction and Public Safety, he established a forensic science subcommittee to that task force. Moreover, and I also plan to address this as well later on, he is authorized me to announce here today a series of forward-looking actions that will conform to forensic science subcommittee's development of a strategic plan on forensics.
...
The Department and NIST created the Commission as a commitment to strengthening forensic science. Our justice system depends on reliable, scientifically valid evidence to solve crimes, identify wrongdoers, and ensure innocent people are not wrongly convicted. This Department. like every other Department that has come before it, remains committed to these principles. Over the past three years, the Commission has played a role in this effort, and we are grateful for your contributions. I'd like to highlight two contributions I am certain will have long-lasting effect. As you know, we announced new department-wide guidance on criminal discovery in cases with forensic evidence at the last Commission meeting. From my vantage point as the national criminal discovery coordinator, the recommendation on pretrial discovery will have long-lasting and important effects. ... 3/

... [A]s part of my training efforts including my discussion and training of forensic examiners, I have learned there is no single Commission recommendation more important for the practice of forensic science than the recommendation regarding universal accreditation. I have been told the Department's decision to publicly announce the policy on accreditation and to mandate our prosecutors to rely on accredited labs when practicable has made a difference in laboratories and moving to accreditation. These recommendations and the Department's review and implementation are a demonstration of the measurable impact of the work of this Commission over the past three years, and for that as well as many other products of this Commission, the Department thanks you.

To identify the elephant in the room, everyone knows the Commission’s charter is expiring this month, and it probably won't be a surprise to learn the charter will not be renewed. As part of any transition, it is critical to re-evaluate and realign resources to achieve a new administration’s priority. Attorney General Sessions has announced his commitment to reducing violent crime in America particularly in our cities, and he has identified the troubling rise in crime as a focus of the Department when he formed the Task Force on Crime Reduction and Public Safety and established a Forensic Science Subcommittee to the Task Force to fight against this increase in crime.

The Task Force and its various subcommittees including the subcommittees on hate crime and on forensic science advise our internal Department working groups with representation for relevant components including laboratories and prosecuting entities. Although these are internal in nature, they are each seeking relevant external stakeholder input. The forensic science subcommittee in particular has been tasked with considering how we will continue to advance the purposes of this Commission in a manner consistent with the Department’s forensics priorities and its policy to reduce crime in America and develop a strategic plan. We plan to consider all options and closely review the Commission’s summary report and secure feedback from the Commissioners and other stakeholders. We will consider all the information before we decide how to move forward.

Today I'm announcing three actions that will inform the Forensic Science Subcommittee's development of a strategic plan on forensics. First, in the coming week the Department will appoint a senior forensic advisor to interface with forensics science stakeholders, advise Department leadership and work with the Subcommittee to develop a strategic plan. The strategic plan will consider questions critical to increasing capacity and ensuring access to high-quality forensic analysis. Some of the questions that will be considered include the following: What are the biggest needs in forensic science inside the Department and outside the Department? Is there more for a body like the Commission to accomplish, or would next steps be better undertaken by some other body? What specific support do Department laboratories and prosecutors need? What does the partner community need? What is required to improve practices? What are the barriers, legal practical or otherwise, and what resources do we need to overcome those barriers? Is the structure sufficient to set standards, or is some other body needed? What is needed to improve capacity so every prosecutor can be assured he or she will receive prompt results when he or she submits evidence for testing? What resources and relationships can the Department best draw on to get thoughtful advice? What is the Department currently doing to advance this issue? Are their better ways to support state and local practitioners?

The second major part of this initiative I announce is that we are publishing an issue for comment in the Federal Register seeking broad stakeholder input on just those questions I went through and what the Department should consider after the expiration of the Commission. That notice will be open until June 9th. We invite you to submit comments and encourage you to share this notice broadly.

Third, the Department is conducting a needs assessment of forensic laboratories. As you know, in December 2016, Congress passed the Justice for All Authorization Act which has several mandates to improve and advance forensic science. The needs assessment will examine serious issues of capacity and backlog at public crime labs and in the medical-legal investigation community. It will consider other topics such as research and coordination necessary when developing a strategic plan to address the needs of the forensic science community.

At the same time, the Department is considering the previously announced projects of forensic sciences of the review and the uniform language for testimony in reports and identifying where they may fit in the subcommittees’ work. We expect this process to develop a strategic plan to be deliberate and thorough but not an endless one by any means. We have every expectation of announcing how we will continue to meet these goals in the coming months.

I know the expiration of the Commission’s charter does not impact the — and the Department supports the work — and is coordinating with NIST in whether the MOU [Memorandum of Understanding] needs to be amended. I want to emphasize at the Department we recognize our responsibility to work tirelessly, improve the work we do, and enhance the administration of justice. Part of that responsibility is to ensure we are regularly coordinating with the right people on these issues and acting in a manner that demonstrates our commitment to fair play and honest dealings in every matter we handle. We will work to understand lessons of this Commission and continue to advance our goals.

Again, the Department thanks you for your contributions and emphasizes we are not finished relying on you yet. Please expect to work with us in the coming months and review, share and respond to any public inquiries. The commitment of people in this room, the time and participation over the last three years was exemplary and represents what we are capable of doing when we work together towards a single unified goal. There is no question forensic sciences one of the most critical tools we have to reduce crime, increase public safety and it will remain a priority in the Department. In order to turn back rising crime, we need to rely on you working together. The federal government intends to use its money, research and expertise to help us figure out what your needs are and determine the best ways to ensure forensic science is accurate, reliable and available to law enforcement and prosecutors to fight crime and the Department of Justice intends to do that.

The new challenge of violent crime in our nation is real and the task in front of us is clear. We need to resist temptation to ignore this or downplay it. We need to tackle it head-on to ensure justice and safety for all Americans. The Department's pledge to identify strategic plans going forward reflects this commitment to justice and the rule of law. In maintaining the public’s confidence in the accurate and reliable forensic science analyses, we need to clear the innocent and convict the guilty. On behalf of the Attorney General, the acting Deputy Attorney General, and the men and women of the Department of Justice, I thank you once again for your efforts.
NOTES
  1. The Federal Advisory Committee Act of 1972 limits such commissions to two-years of operations unless renewed.
  2. The renewed charter described the duration as “indefinite” but added that "[t]he Commission's termination date is two years from the date this Charter is filed with Congress, and is subject to renewal in accordance with Section 14 of FACA [the Federal Advisory Committee Act]."
  3. The reference is to the Supplemental Guidance for Prosecutors Regarding Criminal Discovery Involving Forensic Evidence and Experts, Jan. 5, 2017.  The encomium judiciously pretermits the friction with DOJ that NCFS’s work on pretrial discovery initially generated. The Commission produced nothing of much substance until 2015. That year began on a low note when DOJ tried to block NCFS from voting on a draft recommendation to have DOJ laboratories open their files in criminal cases to defense lawyers even when the Federal Rules of Criminal Procedure did not demand such access. The one federal judge on the Commission resigned in protest; newly appointed Deputy Attorney General Sally Yates rescinded the ruling that the NCFS was exceeding its mandate; Judge Jed Rakoff rejoined the group; NCFS approved the draft recommendations; and DOJ responded by issuing the Supplemental Guidance. See http://for-sci-law.blogspot.com/2015/01/justice-department-reverses-decision-on.html.

Thursday, April 13, 2017

Fact Check: The National Commission on Forensic Science Vote That Wasn't

Forensic Magazine continues to report that a majority of the National Commission on Forensic Science voted in favor of its own dissolution. In a mostly recycled paragraph from an earlier article, 1/ its senior science writer, Seth Augenstein, wrote today that “the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote.” 2/

The Commission never took any vote on whether it would be a good idea to extend the Commission's life. The question put to a vote was whether to include a statement to this effect in an historical document summarizing the activities of the Commission. 3/ The subject of the vote could not have much clearer. 4/ The meeting synopsis states
[A] vote was taken to determine whether this summary report should include a statement that the Commission should continue in its current form. As a business document a simple majority of 50% “yes” votes was required to approve inclusion of this statement. A total of 42% “yes” votes were received, and therefore no statement would be included regarding the continuation of the Commission. 5/
The precise question posed and the complete vote on it were as follows: 6/
Document or Vote Question Asked Total Votes # Yes # No # Abstain
Does the NCFS Summary Report include a sentence that NCFS continues in its current form? 38 16 15 7
NOTES
  1. Seth Augenstein, Final Meeting of National Commission on Forensic Science ‘Reflects Back,’ Apr. 10, 2017, 11:59am, http://www.forensicmag.com/news/2017/04/final-meeting-national-commission-forensic-science-reflects-back. The paragraph stated that
    The NCFS produced 45 documents and recommendations in three years of work, which encompassed 600 public comments. But the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote."
  2. Seth Augenstein, Even Without Forensic Commission, Forensic Science Overhaul Proceeds at OSAC, Apr. 13, 2017, 12:12pm, http://www.forensicmag.com/news/2017/04/even-without-forensic-commission-forensic-science-overhaul-proceeds-osac. The latest paragraph states that
    The NCFS, by the end of its last meeting on Tuesday, produced 45 documents and recommendations in three years of work—many of which directed OSAC’s explorations into forensic disciplines. But the commission itself had voted against its own renewal at its January meeting, by a 16-15 vote. Sessions announced that it would not be renewed on Monday.
    The additions are also inaccurate. Very few of the NCFS Views documents and Recommendations documents seem to have "directed OSAC's explorations."
  3. Reflecting Back—Looking Toward the Future, Dec. 16, 2016 (draft), https://www.justice.gov/ncfs/page/file/921431/download.
  4. The discussion as recorded on the meeting webcast includes the following (with intervening speaker statements omitted without ellipses):
    HON. PAM KING: This is a business record ... of this particular Commission. ... This is a document that does not take any real position as to whether something should or should not be done. ...I did get some comments from Commissioners before this meetings ... One of the ones that I really would like to get some discussion on is [the] strong feelings among some Commissioners that maybe we do want to make a statement about whether or not this Commission should continue. ...
    JULIA LEIGHTON: I would not shy away from a recommendation ... I think to scrap it altogether ... is to give up on the work we’ve done.
    GERALD LAPORTE: So I don’t agree — disagree — with anything Julia has said. ... but I don’t know if we really are in a position to make a recommendation ... .
    ARTURO CASADEVALL: I want to support what Julia said. Commissions like this develop an institutional memory. ... I strongly think we should make a recommendation that something like this continue.
    S. JAMES GATES: [A]bsent a committee like this, I don’t see a consistent driver for making progress. ...
    MATTHEW REDLE: Whether it is this form or not, ... there ought to be more work done to continue the progress that we have made ...
    JULIA LEIGHTON: [W]e need a national body [with] the gravitas of being a nonpartisan federal advisory commission. ...
    HON. JED RAKOFF: ... I do think it is important that we say some something [to] indicate that we believe the Commission should continue. ...[J]udges do pay some attention to what this Commission says and does. So I think it plays a role there that is not played by other very wonderful groups ... and some very wonderful reports. I would very much strongly encourage that we have something in there ...
    WILLIAM THOMPSON: This Commission is uniquely well situated to address those [human factors] issues ... so I hope the Commission continues to address those kinds of questions ... .
    JULES EPSTEIN: So ... for this concluding portion ... yeah, we should keep going in some shape or form. ... [M]ore needs to be done. More constituencies will look to us than to other segregated constituencies. [T]he federal advisory commission should continue.
    WILLIE MAY: Certainly, I think that the Commission’s work is not completed. [I]t would serve the country very well to continue this ... .
  5. National Commission on Forensic Science Meeting #12, Jan. 9-10, 2017, at 6, https://www.justice.gov/ncfs/page/file/953106/download
  6. Id. at 10.

Wednesday, April 12, 2017

Whither OSAC? NIST's Plans for Forensic Science Standards and Research As Told to the NCFS

The National Commission on Forensic Science held its thirteenth and final meeting on Monday. The second speaker to discuss some of the administration's plans for improving forensic science was the Acting Director of the National Institute of Standards and Technology (NIST), Dr. Kent Rochford. I edited and abridged the computer-generated transcript slightly. It includes a question from Commissioner Peter Neufeld. Finally, there is a question from Commissioner Jules Epstein to a Justice Department official about future funding for OSAC. I cannot promise riveting reading, but for anyone who wants to know what was said, here is most of it:
KENT ROCHFORD: I'd like to address the future of OSAC. OSAC was conceived under the 2013 MOU [Memorandum of Understanding] between NIST and the Department of Justice and established the Commission. The Department of Justice provides funding for the OSAC, which NIST cannot sustain on its own. The OSAC organization does not have term limits but does require funding to continue.

From the introduction of OSAC, NIST addressed the need to evolve and eventually spin off OSAC. We termed this “OSAC 2.0.” We have learned a lot from OSAC 1.0. Over the past years of operation, the organization has continued to mature as members of the group have come to a better appreciation of the standards development process. One example was seen in interested key researchers and scientists joining the FSSB [Forensic Science Standards Board]. Thank you for your assistance in supporting and strengthening the OSAC.

NIST is committed to improving OSAC, including the establishment of a clear model that will support these important goals. We are working to create a stable, sustainable operational model that provides independence from NIST. Internally a small group led by Rich Cavanaugh, who runs our special programs office, has been exploring model concepts for OSAC 2.0.

Each model is distinct yet consistent with the following goals: The new OSAC has to have a defined structure and authority. It needs to engage key stakeholders. We need to provide free access to our products. There has to be a smooth transition from the current OSAC that would create the potential for long-term sustainability. Currently, Rich's group has been looking at three models, exploring further. These involve creating federal and state partnerships that develop codes, standards, and model laws. Restructuring the OSAC so subcommittee functions are dispersed to standards development organizations, and the roles at the FSSB and SAC levels are changed to focus on quality of science and utility, respectively. And establishing a development and testing — a process we are starting, and we intend to engage the broader community to better understand the strengths and weaknesses of these possible approaches. So if you have questions about the OSAC 2.0, please reach out to Rich Cavanaugh.

I'd like to talk about NIST research efforts in forensic science. NIST remains committed to remaining its measurements and standards expertise to challenges in forensics. We played a role in strengthening forensic science since at least the 1920s. You may have seen the recent National Geographic article about William Souther, a physicist from NIST who played a role in numerous forensic cases during the 1930s, including the famous Lindbergh baby kidnapping case. The current forensic research focuses include, DNA, digital fingerprint evidence, ballistics, statistics, toxins, and trace evidence. We plan to continue working these research areas as funding is available to do so. You will see an example of how research expertise provides benefit to the forensic science community when [Dr. Elham Tabassi] talks to you about development of an ISO standard on method validation.

Let me turn to technical merit review. This past September, the President's Council of Advisors on Science and Technology recommended an expanded role for NIST in assessing the scientific foundations and maturity of various forensic disciplines. We recognize the need for and the value of such studies and are exploring ways to conduct work in this area. Without additional funding recommended by PCAST, NIST cannot make large-scale commitments to technical merit review. We are planning an exploratory study to address concerns raised by PCAST regarding DNA mixtures. This will likely involve assessing the scientific literature, developing a detailed plan for evaluating scientific validity that would include probabilistic genotyping, and assigning interlaboratory studies to measure forensic laboratory performance of DNA interpretation. These laboratory studies would build upon DNA mixture studies conducted in 2003, 2005 in 2013. NIST has a history of involving external partners in his research and standard efforts and anticipate external and internal and international collaboration.

***
In closing, I want to personally thank you for your efforts on this Commission and your commitment to strengthening forensic science through your participation in the activities of this group. Your work is made a difference, and we are grateful for your service to the nation. Thank you.
***
PETER NEUFELD: The second question for Kent, when you talked about things NIST was doing, you mentioned your current evaluation of DNA mixtures. Your predecessor stated in response to this Commission making a recommendation that NIST take on the task of making an evaluation of foundational validity and reliability of different forensic methods that they intended to do a trial. They were going to start a trial in three different areas, and the other two areas in addition to the DNA were ballistics and bitemarks. We have been told at each meeting leading up to this meeting NIST was going ahead with those trials. I noticed you only mentioned DNA. Is it still the position of NIST that they will go ahead with the trial of some ballistics and bitemarks?

KENT ROCHFORD: We still continue to do the work on ballistics and bitemarks. Given the resources we have, we're going to do the trials of the interlaboratory studies with the DNA mixtures first. Right now, the PCAST report provided a number of trials we should take on [and] it is also recommending the funding to do this. Given our current funding, we intend to start with the DNA programs. As funding may become available, we can wrap up these others areas to include trials. Currently we are doing the internal work but do not right now have the bandwidth to do the ballistics trials.
***
JULES EPSTEIN: Good morning. *** The other substantive question is, can I get clarification on OSAC? Is it now the status there is currently no further funding for OSAC?

ANSWER: No.

JULES EPSTEIN: Can we understand what is in the pipeline or the projected longevity at this moment or sustainability?

ANSWER: Right now we don't have a budget and we are in a continuing resolution. We just don't know the status so I really can't predict what it will look like.

Two Misconceptions About the End of the National Commission on Forensic Science

Several days ago, the Justice Department (DOJ) announced the end of the National Commission on Forensic Science (NCFS). Initially established to advise the Department of Justice for a two-year period, the NCFS had had its charter extended once before, in April 2015. Attorney General Sessions declined to renew it a second time.

Several explanations for this decision could be offered: (1) the current administration is unreceptive to scientific knowledge and advice — alternate facts and alternate science are more appealing; (2) DOJ does not want outside advice on producing and presenting scientific evidence; (3) DOJ is tired of spending millions of dollars for advice from this particular group of lawyers, judges, administrators, forensic scientists, and others. (4) DOJ believes that NCFS has outlived its usefulness and there are better ways to obtain advice. And of course, some combination of these things might have been at work. I have no inside information, but I thought it might be helpful to collect some of what is publicly known if only to correct misconceptions about the Commission and its role vis-a-vis the Justice Department. I'll begin by noting two such misconceptions.

Misconception 1
NCFS Was Evaluating the Validity of Forensic Science Tests and Methods

One misconception is that the Commission was conducting independent evaluations of accepted methods in forensic science. Thus, the Associated Press reported that “National Association of Criminal Defense Lawyers ... President Barry Pollack said the commission was important because it allowed ‘unbiased expert evaluation of which techniques are scientifically valid and which are not.’” 1/ But not one of the 44 documents identified as “work products” on the NCFS website examines the validity of any technique.

The two documents that directly address “technical merit” are views and then recommendations 2/ about the need to study validity and reliability of techniques. No surprise there. More importantly, the documents underscore the importance of bringing what we might call "outsider" scientific expertise to bear in these efforts, and one of them contains pointed advice to other organizations. Specifically, a recommendation calls on NIST and its creation, the Organization of Scientific Area Committees for Forensic Science (OSAC), to reform the procedure OSAC uses to review and endorse standards for test methods. It states:
The Organization of Scientific Area Committees for Forensic Science (OSAC) leadership, the Forensic Science Standards Board (FSSB), should commit to placing consensus documentary standards on the OSAC Registry of Approved Standards for only those forensic science test methods and practices where technical merit has been established by NIST, or in the interim, established by an independent scientific body. An example of an interim independent scientific body could be an OSAC-created Technical Merit Resource Committee composed of measurement scientists and statisticians appointed by NIST and tasked with the evaluation of technical merit. 3/
This recommendation, by the way, has had limited impact. Yes, NIST has announced that it will do further research in a few areas such as DNA-mixture analysis. No, OSAC has not established a Resource Committee to check the technical merit of the documents that filter up from its subject-area committees and subcommittees.  4/

Rather than performing literature reviews (or promulgating scientific standards for forensic laboratories to follow), NCFS focused on broader issues of policy, needs, and legal reforms for generating or evaluating scientific evidence. This role for the Commission relates to a second misconception.

Misconception 2
NCFS Was a Worthless “Think Tank”

According to the Washington Post,
[T]he National District Attorneys Association, which represents prosecutors, applauded the end of the commission and called for it to be replaced by an Office of Forensic Science inside the Justice Department. Disagreements between crime lab practitioners and defense community representatives on the commission had reduced it to “a think tank,” yielding few accomplishments and wasted tax dollars, the association said. 5/
A press release from the NDAA does “applaud” the DOJ’s decision not to nonrenew the Commission, but not because the NCFS was a “think tank.” The group representing “2,500 elected and appointed District Attorneys across the United States, as well as 40,000 Assistant District Attorneys” complained that
The Commission lacked adequate representation from the state and local practitioner community, was dominated by the defense community, and failed to produce work products of significance for the forensic science community. Very few of the recommendations from the Commission were adopted and signed by the previous Attorney General during its existence. Those that were signed, such as universal accreditation, had already begun to develop organically within the forensic science community as accepted best practices, thus replicating ongoing work and wasting taxpayer dollars. 6/
I have not checked the percentage of recommendations “signed” by the Attorney General, but the Commission’s views documents never were intended to be signed by anyone, and the notion that only recommendations for specific action by the Attorney General benefit “the forensic science community” is shortsighted. Among the Commission’s documents of lasting value are the following:
  • Recommendation on Transparency of Quality Management System
  • Recommendation on Model Legislation for Medicolegal Death Investigation Systems
  • Views Document on Recognizing the Autonomy and Neutrality of Forensic Pathologists
  • Recommendations on Use of the Term “Reasonable Scientific Certainty”
  • Recommendation on Pretrial Discovery
  • Recommendations on Use of the Term “Reasonable Scientific Certainty”
  • Views Document on Judicial Vouching
  • Views Document on Ensuring that Forensic Analysis is Based Upon Task-Relevant Information
  • Views Document on Facilitating Research on Laboratory Performance
  • Views Document on Identifying and Evaluating Literature that Supports the Basic Principles of a Forensic Science Method or Forensic Science Discipline
It is instructive to compare the NDAA's dismissal of the "universal accreditation" recommendation  with the assessment of it by Associate Deputy Attorney General Andrew Goldsmith, who stated in his remarks at the final NCFS meeting that
[T]here is no single commission recommendation more important for the practice of forensic science than the recommendation regarding universal accreditation. I have been told the Department's decision to publicly announce the policy on accreditation and to mandate our prosecutors to rely on accredited labs when practicable has made a difference in laboratories and in moving to accreditation. These recommendations and the Department's review and implementation are a demonstration of the measurable impact of the work of this Commission ... .
Naturally, many of the ideas or actions that the Commission endorsed were not original. The idea of accreditation was prominent in the 2009 National Research Council (NRC) report on forensic science as well as NRC reports on DNA evidence in 1992 and 1996. NCFS was not a think tank, but a mixed bag of administrators, prosecutors, defenders, judges, law professors, police officials, laboratory scientists, medical examiners and coroners, research scientists, and other individuals. It could be criticized as wasteful — 13 meetings of 41 members (including the ex officio ones) plus an unlisted number of nonmembers appointed to subcommittees at a cost of millions of dollars for taxpayers (not to mention the opportunity costs to the volunteers). Consequently, it certainly is fair to ask how much additional benefit would have come from another two years of Commission life. 7/ But the Justice Department does not plan to halt all study of in-house forensic science reform. It has announced that some of it will continue via a newly created -- and surely not costless -- task force run by the incoming Deputy Attorney General. Given that plan, is the restructuring really an effort to save taxpayer money because of a perception that NCFS had reached the point of diminishing returns? Or is it a move to control the agenda and to modify the list of people who provide input? More on that later.

NOTES
  1. Sadie Gurman, Sessions' Justice Dep't Will End Forensic Science Commission, AP News, Apr. 11, 2017, https://www.apnews.com/c076ef99c48948e3856902cfca9e7b14/Sessions'-Justice-Dep't-will-end-forensic-science-commission
  2. NCFS often prepared two “work products” per topic for its recommendations — a preliminary “views” document followed by a final, more concrete  “recommendations” document. Consequently, the total number of its "work products" is a poor quantitative measure of its accomplishments.
  3. Recommendation to the Attorney General: Technical Merit Evaluation of Forensic Science Methods and Practices, Dec. 9, 2016, at 3, https://www.justice.gov/ncfs/page/file/905541/download
  4. A more cheerful description of the response from NIST and OSAC can be found in a letter from the six research scientists (not forensic scientists) on the Commission pleading for a renewal of the charter.
  5. Spencer Hsu, Sessions Orders Justice Dept. To End Forensic Science Commission, Suspend Review Policy, Wash. Post, Apr. 10, 2017, https://www.washingtonpost.com/local/public-safety/sessions-orders-justice-dept-to-end-forensic-science-commission-suspend-review-policy/2017/04/10/2dada0ca-1c96-11e7-9887-1a5314b56a08_story.html?utm_term=.c24306818156
  6. NDAA, Press Release, National District Attorneys Association Applauds Expiration of National Commission on Forensic Science, Apr. 10, 2017, http://www.ndaa.org/pdf/NDAA%20Statement%20on%20Expiration%20of%20National%20Commission%20on%20Forensic%20Science.pdf
  7. Erin Murphy, Op-ed, Sessions Is Wrong to Take Science Out of Forensic Science, N.Y. Times, Apr. 11, 2017, https://www.nytimes.com/2017/04/11/opinion/sessions-is-wrong-to-take-science-out-of-forensic-science.html (asserting that NCFS "was even poised to issue a raft of best practices for the wild west of digital forensics, which has exploded without supervision over the years.")

Wednesday, March 29, 2017

After Moore v. Texas Is a Single IQ Score Really Determinative?

Bobby J. Moore has been on death row for the last 37 years. On Monday, the Supreme Court ruled that the Texas Court of Criminal Appeals (the state’s highest court for criminal cases) erred in finding that Moore is not intellectually disabled. Justice Ginsburg wrote for the five-member majority. The Chief Justice wrote a strong dissent for the other three justices. Neither opinion (on my quick reading at least) comes to grips with an obvious statistical principle—that combining information reduces uncertainty.

Moore v. Texas is the third case to try to clarify the rule in Atkins v. Virginia, 536 U.S. 304 (2002). There, the Supreme Court held that the Eighth Amendment’s Cruel and Unusual Punishment Clause prevents a state from executing an intellectually disabled offender, but it left the states with latitude in defining the disability. In Moore, the Court held that the Texas tribunal applied a medically outdated—and (hence?) constitutionally impermissible—standard in rejecting Moore’s claim of disability. Most of the majority opinion concerns “adaptive functioning,” which must be substantially impaired for a diagnosis of intellectual disability to be made.

However, the Court in Hall v. Florida, 572 U.S. __ (2014), allowed a state to refuse to inquire into adaptive functioning if an offender’s true IQ score is at least 70. Hall explicitly stated that the following statutory definition of intellectual disability was constitutionally acceptable:
“significantly subaverage general intellectual functioning existing concurrently with deficits in adaptive behavior and manifested during the period from conception to age 18,” where “significantly subaverage general intellectual functioning” is “performance that is two or more standard deviations from the mean score on a standardized intelligence test.”
Because IQ scores for the whole population are roughly normally distributed with a mean of approximately 100 and a standard deviation of about 15, Hall allows the state to execute offenders whose "true scores" are above 70.

In deciding whether a true score is above 70, Hall demanded that the state attend to the error of measurement. As the Moore Court, quoting from Hall, explained, "'[f]or purposes of most IQ tests,' [the] imprecision in the testing instrument 'means that an individual’s score is best understood as a range of scores on either side of the recorded score . . . within which one may say an individual’s true IQ score lies.'" For a single test with a standard error of 2.5 IQ points, it follows (for normally distributed errors) that the measured score must be greater than or equal to 75 (= 70 + two standard errors) to avoid "an unacceptable risk that persons with intellectual disability will be executed." 1/

But what about multiple scores? In that common situation, the Hall Court seemed conflicted. Justice Kennedy opaquely opined that “[e]ven when a person has taken multiple tests, each separate score must be assessed using the SEM, and the analysis of multiple IQ scores jointly is a complicated endeavor.” Does this mean that no matter how many IQ tests have been administered and no matter how many of them lie above 70, a single score of 75 or less makes a conclusive case for “significantly subaverage general intellectual functioning”?

From a statistical perspective, a lowest-single-score seems very strange indeed. If I want to know whether I have a fever and I take ten measurements of my temperature (with ten thermometers), I would not say that I have a fever just because one thermometer gives a high reading. I would use an average, and the mean temperature would have greater precision (smaller standard error) than the single highest reading of the ten.

Justice Ginsburg’s opinion in Moore seems to fly in the face of this common-sense statistical point. The Texas court focused on two test scores — "a 78 in 1973 and 74 in 1989." It pointed to factors that might have biased the latter score toward the low end, leaving the higher one as entitled to more weight. Specifically, it wrote that there was expert testimony that Moore might not have been putting much effort into answering the questions in the lower-scoring test, which was given to him in prison, and that he "also took the WAIS–R under adverse circumstances; he was on death row and facing the prospect of execution, and he had exhibited withdrawn and depressive behavior." Ex Parte Moore, 470 S.W.3d 481, 519 (Tex. Ct. Crim. App. 2015). Thus, the court concluded,
These considerations might tend to place his actual IQ in a somewhat higher portion of that 69 to 79 range. ... Considering these factors together, we find no reason to doubt that applicant's [higher] WAIS–R score accurately and fairly represented his intellectual functioning as being above the intellectually disabled range.
The Supreme Court assumed that it was necessary to consider each test in isolation and without making a clinical adjustment to the statistically determined plus-or-minus-five-point margin of error. Justice Ginsburg called the statistical range of error "clinically established." She described and condemned the Texas court's evaluation of the clinical testimony as follows:
Based on the two scores, but not on the lower portion of their ranges, the court concluded that Moore’s scores ranked “above the intellectually disabled range” (i.e., above 70). ... But the presence of other sources of imprecision in administering the test to a particular individual, cannot narrow the test-specific standard-error range. [W]e require that courts continue the inquiry and consider other evidence of intellectual disability where an individual’s IQ score, adjusted for the test’s standard error, falls within the clinically established range for intellectual-functioning deficits.
Thus, she insisted that just because "Moore’s score of 74, adjusted for the standard error of measurement, yields a range of 69 to 79" so that "the lower end ... falls at or below 70, the [Court of Criminal Appeals] had to move on to consider Moore’s adaptive functioning." (Emphasis added.)

In sum, Moore seems to say that a clinician cannot tinker with the statistical margin of error (two standard errors as constitutionalized in Hall). The dissent vigorously disagreed with this rule and maintained that the constitution permits states to make adjustments for individual circumstances that experts agree affect performance. A statistical argument for the dissent's position would be this: Computationally, the standard error reflects the variation in performance of a population of test-takers. This population-based figure is then applied to all individuals regardless of how strongly the sources of error apply to them. IQ tests administered in prison to inmates exhibiting signs of depression may not be part of that population. Those scores might have a larger or a smaller standard error, and they are generally lower than the true score for a person taking the test in normal circumstances. In other words, the clinician is not modifying the margin of error as much as adjusting the entire estimate upward.

This analysis does not necessarily render Moore's rule legally faulty. It might be undesirable to give clinicians this latitude to adjust scores. Under the majority's approach to the Eighth Amendment, the issue becomes whether the clinical guidelines for diagnosing disability allow individualized modifications of the statistical rule. The guidelines discussed in Hall are not completely clear. The dissent reads them as requiring an expert or a court to take the usual standard error seriously in interpreting an IQ score, but permitting reasoned and reasonable departures from them.

Even if Moore forbids individual adjustments to the statistical rule of plus-or-minus two standards errors (for the general population), why allow the confidence interval for a single test score to be dispositive when multiple tests scores are present? The clinical guidelines do not mandate this rule, and it is not so obvious that Moore does. Texas apparently made no effort to combine the two scores into a single point estimate with a margin of error applicable to the combined statistic. Hall claimed that combining scores from different IQ test forms was "complicated," although the literature it cited gave a simple procedure for doing so. So neither Hall nor Moore can be said to firmly establish that an appropriately averaged score is impermissible. After all, neither case presented the Court with an interval estimate for the true IQ score derived from multiple scores by an accepted statistical procedure, and the many-thermometer example given above illustrates the statistical deficiency in a rule that looks to every measured IQ score in isolation.

The single-score-too-low rule bends over backward to avoid misclassifying a disabled offender as normal. The rule might be defended on exactly that ground. But that is not the logic of Moore, which only asks what clinical guidelines for interpreting IQ scores allow. Moreover, if the real objective is make determinations of intellectual disability as fully informed as possible, it would seem more direct just to demand the inquiry into adaptive functioning along with IQ scores in all cases. On the other hand, if true IQ scores matter as a threshold to a richer inquiry into both intellectual and adaptive functioning, then statistically sound procedures for integrating all the IQ test results ought to be followed.

Further reading: David H. Kaye, Deadly Statistics: Quantifying an "Unacceptable Risk" in Capital Punishment, 15 Law, Probability & Risk __ (2017).

Note
  1. If the standard error were substantially less than 2.5, then the measured score would not have to be all five points above 70. The use of two standard errors also is on the high side; 1.96 standard errors provides 95% coverage. Justice Kennedy's opinion in Hall was not as clear as it should have been on these points, but this is the only interpretation consistent with the concept of confidence intervals and standard errors used in the opinion. In Bromfield v. Cain, 135 S.Ct. 2269 (2015), however, the Court wrote that after "[a]ccounting for this margin of error, Brumfield's reported IQ test result of 75 was squarely in the range of potential intellectual disability." Id. at 2278. The Court did not disclose the standard error of measurement for the test.

Friday, February 3, 2017

Connecticut Trial Court Deems PCAST Report on Footwear Mark Evidence Inapplicable and Unpersuasive

In an unpublished (but rather elaborate) opinion, a trial court in Connecticut found no merit in a motion “to preclude admission of footwear comparison evidence relative to footwear found on Wolfe Road in Warren, Connecticut and footprints found at the residence where the victim was killed.” State v. Patel, No. LLICR130143598S (Conn. Super. Ct., Dec. 28, 2016). The court did not describe the case or the footwear evidence, but its opinion responded to the claim of defendant Hiral Patel that “the scientific community has rejected the validity of the footwear comparison proposed by the state.” Judge John A. Danaher III was unimpressed by Patel's reliance on
a September 2016 report by the President's Council of Advisors on Science and Technology [stating] that ‘there are no appropriate empirical studies to support the foundational validity of footwear analysis to associate shoeprints with particular shoes based on specific identifying marks (sometimes called randomly 'randomly [sic] acquired characteristics'). Such conclusions are unsupported by any meaningful evidence or estimates of their accuracy and thus are not scientifically valid.’
The court reasoned that the state had no need to prove that the “expert testimony ... albeit scientific in nature” was based on a scientifically validated procedure because the physical comparison was “neither scientifically obscure nor instilled with 'aura of mystic infallibility' ... which merely places a jury ... in in [sic] a position to weigh the probative value of the testimony without abandoning common sense and sacrificing independent judgment to the expert's assertions.” Patel (quoting Maher v. Quest Diagnostics, Inc., 269 Conn. 154, 170-71 n.22, 847 A.2d 978 (2004)).

But the Superior Court did not stop here. Judge Danaher wrote that the President’s Council (PCAST) lacked relevant scientific expertise, and their skepticism did not alter the fact that courts previously had approved of “the ACE-V method under Daubert for footwear and fingerprint impressions.” He declared that "[t]here is no basis on which this court can conclude, as the defendant would have it, that the PCAST report constitutes 'the scientific community.'" These words might mean that the relevant scientific community disagrees with the Council that footwear-mark comparisons purporting to associate a particular shoe with a questioned impression lack adequate scientific validation. Other scientists might disagree either because they do not demand the same type or level of validation, or because they find the existing research satisfies PCAST's more demanding standards. The former is more plausible than the latter, but it is not clear which possibility the court accepted as true.

To reject the PCAST Report's negative finding, Judge Danaher relied exclusively on the testimony of “Lisa Ragaza, MSFS, CFWE, a ‘forensic science examiner 1’ ... who holds a B.S. degree from Tufts University and an M.S. degree from the University of New Haven.” What did the forensic-science examiner say to support the conclusion that PCAST erred in its determination that no adequate body of scientific research supports the accuracy of examiner judgments? To begin with,
Ms. Ragaza testified that, in her opinion, footwear comparison analysis is generally accepted in the relevant scientific community. She testified that such evidence has been admitted in 48 or 49 of the 50 states in the United States, in many European countries, and also in India and China. In fact, she testified, such analyses have been admitted in United States courts since the 1930s, although she is also aware that one such analysis was carried out in Scotland as early as 1786.
It seems odd to have forensic examiners instruct the court in the law. That the courts in these jurisdictions (not all of which even require a showing of scientific validity) admit the testimony of footwear analysts that a given shoe is the source of a mark says little about the extent to which these judgments have been subjected to scientific testing. As a committee of the National Academy of Sciences reported in 2009, “Daubert has done little to improve the use of forensic science evidence in criminal cases.” NRC Committee on Strengthening Forensic Science in the United States, Strengthening Forensic Science in the United States: A Path Forward 106 (2009). Instead, “courts often ‘affirm admissibility citing earlier decisions rather than facts established at a hearing.’” Id. at 107.

Second, 
Ms. Ragazza testified that there are numerous treatises and journals, published in different parts of the world, on the topic of footwear comparison analysis. She testified that there have been studies relative to the statistical likelihood of randomly acquired characteristics appearing in various footwear.
But the existence of “treatises and journals” — including what the NAS Committee called “trade journals,” id. at 150 — does not begin to contradict PCAST’s conclusion about the dearth of studies of the accuracy of examiner judgments. PCAST commented (pp. 116-17) on one of the “studies relative to the statistical likelihood”:
a mathematical model by Stone that claims that the chance is 1 in 16,000 that two shoes would share one identifying characteristics and 1 in 683 billion that they would share three characteristics. Such claims for “identification” based on footwear analysis are breathtaking—but lack scientific foundation. ... The model by Stone is entirely theoretical: it makes many unsupported assumptions (about the frequency and statistical independence of marks) that it does not test in any way.
Third,
Ms. Ragazza testified that her work is subject to peer review, including having a second trained examiner carry out a blind review of each analysis that she does. In response to the defendant's question as to whether such reviews have ever resulted in the second reviewer concluding that Ms. Ragazza had carried out an erroneous analysis, she responded that there were no such instances. Most of her work is not done in preparation for litigation. It is frequently done for investigative purposes and may be used to inculpate, but also exculpate, an individual. She indicated that the forensic laboratory carries out its analyses for both prosecutors and defense counsel.
Verification of an examiner’s conclusion by another examiner is a good thing, but it does almost nothing to establish the validity of the examination process. Making sure that two readers of tea leaves agree in their predictions does not validate tea reading (although it could offer data on measurement reliability, which is necessary for validity).

Fourth,
Ms. Ragazza explained how footwear comparison analysis is carried out, using a protocol known as ACE-V, and employing magnifiers and/or microscopes.
Plainly, this misses the point. If tea reading were expanded to include magnifiers and microscopes, that would not make it more valid. (Actually, I believe that footwear-mark comparisons based on “randomly acquired characteristics” are a lot better than tea reading, but I still am searching for the scientific studies that let us know how much better.)

Sixth,
Ms. Ragazza does not agree with the PCAST report because, in her view, that report did not take into account all of the available research on the issue of footwear comparison evidence.
Maybe there is something to this complaint, but what validity studies does the PCAST report overlook? The Supporting Documentation for Department of Justice Proposed Uniform Language for Testimony and Reports for the Forensic Footwear and Tire Impression Discipline (2016) begins “The origin of the principles used in the forensic analysis of footwear and tire impression evidence dates back to when man began hunting animals.” But the issue the PCAST Report addresses is not whether a primitive hunter can distinguish between the tracks of an elephant and a tiger. It is the accuracy with which modern forensic fact hunters can identify the specific origin of a shoeprint or a tire tread impression. If Ms. Ragazza provided the court with studies of this particular issue that would produce a different conclusion about the extent of the validation research reported on in both the NRC and PCAST reports, the court did not see fit to list them in the opinion.

A footnote to the claim that "an examiner can identify a specific item of footwear/tire as the source of the footwear/tire impression" can be found in the Justice Department document mentioned above. This note (#12) lists the following publications:
  1. Cassidy, M.J. Footwear Identification. Canadian Government Publishing Centre: Ottawa, Canada, 1980, pp. 98-108; 
  2. Adair, T., et al. (2007). The Mount Bierstadt Study: An Experiment in Unique Damage Formation in Footwear. Journal of Forensic Identification 57(2): 199-205; 
  3. Banks, R., et al. Evaluation of the Random Nature of Acquired Marks on Footwear Outsoles. Research presented at Impression & Pattern Evidence Symposium, August 4, 2010, Clearwater, FL;
  4. Stone, R. (2006). Footwear Examinations: Mathematical Probabilities of Theoretical Individual Characteristics. Journal of Forensic Identification 56(4): 577-599;
  5. Wilson, H. (2012). Comparison of the Individual Characteristics in the Outsoles of Thirty-Nine Pairs of Adidas Supernova Classic Shoes. Journal of Forensic Identification 62(3): 194-203.
I wish I could say that I have read these books and papers. At the moment, I can only surmise their contents from the titles and places of publication, but I would be surprised if any of them contains an empirical study of the accuracy of footwear-mark examiners’ source attributions. (If my guess is wrong, I hope to hear about it.)

Finally,
She testified that, to her knowledge, the PCAST members did not include among their membership any forensic footwear examiners.
It's true. The President's Council of Advisors on Science and Technology does not include footwear examiners. But would we say that only tea-leaf readers are able to judge whether there have been scientific studies of the validity of tea-leaf reading? That only polygraphers are capable of determining whether the polygraph is a valid lie detector? That only pathologists can ascertain whether an established histological test for cancer is accurate?

PCAST's conclusion was that no direct experiments currently establish the sensitivity and specificity of footwear-mark identification. In the absence of a single counter-example from the opinion, that conclusion seems sound. But the legal problem is whether to accept the PCAST report's premise that this information is essential to admissibility of footwear evidence under the standard for scientific expert testimony codified in Federal Rule of Evidence 702. Is it true, as a matter of law (or science), that only a large number of so-called black box studies with large samples can demonstrate the scientific validity of subjective identification methods or that the absence of precisely known error probabilities as derived from these experiments dictates exclusion? I fear that the PCAST report is too limited in its criteria for establishing the requisite scientific validity for forensic identification techniques, for there are other ways to test examiner performance and to estimate error rates.  But however one comes out on such details, the need for courts to demand substantial empirical as well as theoretical studies that demonstrate the validity and quantify the risks of errors in using these methods remains paramount.

Although Patel is merely one unpublished pretrial ruling with no precedential value, the case indicates that defense counsel cannot just cite the conclusions of the PCAST report and expect judges to exclude familiar types of evidence. They need to convince courts that "the reliability requirements" for scientific evidence include empirical proof that a technique actually works as advertised. Then the parties can focus on whether PCAST's assessments of the literature omit or give too little weight to studies that would warrant different conclusions. Broadbrush references to "treatises and journals" and a history of judicial acceptance should not be enough to counter PCAST's findings of important gaps in the research base of a forensic identification method.