Evidence-centric and Conclusion-centric: What's the Difference?

In recent months, various forensic practitioners have asked for a short and simple explanation of the difference between the "evidence-centric" and "conclusion-centric" forms of reporting and testifying. I figure that's what I get for introducing the phrases in a panel asked to talk about “Bringing Statistics into the Courtroom” at the Conference on Forensics, Statistics, and Law, at the University of Virginia in March 2018.^\1/ The particular phrases may have been original (at least I think they were), but the underlying ideas, which are what matter, have been around for a long time.

For concreteness, I will focus on a specific example--something that is the target of criticism du jour in US courts. I am thinking of firearms-toolmark examinations culminating in conclusions about the source of a bullet or shell casing. However, the dichotomy between statements of the evidence and its probative value, on the one hand, and statements of the inferences or conclusions that follow from the evidentiatary statements, on the other hand, is far more general. It applies to techniques for shedding light on the possible source of fingerprints, treadmarks, handwriting, images, glass fragments, paint chips, fibers, bones, bitemarks, or any other traces acquired at or around crime scenes or victims. Indeed, the distinction applies to all manner of empirical inquiry, as suggested by the subtitle of the late Frederick Schauer's book, The Proof: Uses of Evidence in Law, Politics, and Everything Else.^\2/ Moreover, nothing in the terminology reveals whether the evidence in question comes from a scientifically validated process, from (typically) less reliable modes of acquiring and assessing data, or from superstitions masquerading as knowledge.

A Recent Case

With these broad obervations in mind, let's turn to a recent appellate opinion comparing the state's arguments for introducing the conclusions of a firearms examiner to arguments that could be marshalled for believing in palmistry. In People v. Tidd, No. A167548, 2024 WL 3982134 (Cal. Ct. App., Aug. 29, 2024), someone in a white sport utility vehicle shot an enebriated pedestrian in the leg. Investigators located a nine-millimeter Luger cartridge case, manufactured by Speer, in the vicinity. Six days later, police arrrested Raymond Tidd as he was about to enter a white SUV carrying a loaded nine-millimeter Sig Sauer pistol. He pled guilty to some firearms possession and carrying charges but went to trial for attempted murder and other felonies for the shooting. Before trial, he objected to testimony from a criminalist with the San Francisco Police Department named Jacobus Swanepoel. The trial court held a hearing and ruled that the expert could testify that the cartridge case came from the defendant's gun as long as he refrained from asserting his opinion “to a scientific certainty.”

At trial, Swanepoel "flatly asserted the recovered cartridge and test cartridges fired from Tidd's gun 'were fired in the same firearm.'" The jury found Tibbs guilty of two firearms felonies for the shooting (but not of attempted murder). However, the court of appeals reversed because the prosecution never showed that the highly subjective judgmental procedure the expert described was "reasonably reliable."

The court's analysis of what it would take to achieve such reliability is open to debate, but that is a subject for another day. My target today is the statements that were made or could have been made about the cartridge cases and the gun or guns that fired them. What statements are "conclusion-centric"? Which are "evidence-centric"?

A Preliminary Point

"Evidence" exists in the context of some hypothesis about a matter of fact (a state of nature). Nothing can sensibly be deemed "evidence" without asking, "evidence of what?" That the sky is dark is evidence that it is nighttime. It is not conclusive evidence (nothing is conclusive evidence of an empirical fact) because darkness in the daytime can come from a total solar eclipse (or even less likely events). But darkness is pretty good evidence (E) for the hypothesis (H) that night has arrived. The hypotheses of most interest in cases like Tidd are source hypotheses like Swanepoel's assertion that the cartridges were "fired in the same gun" or "the cartridges were fired in different guns." These assertions are conclusions that the analyst reaches (or does not reach) on the basis of the evidence about the marks on the objects examined.

Conclusion-centric Statements

With respect to the source hypothesis, a scan of the opinion in Tibbs reveals the following conclusion-centered statements from the firearms examiner:

• The cartridges "were fired in the same gun." In other words, the same-gun hypothesis H is true (given the markings seen in the examination). It may not be "scientifically certain," but it is true nonetheless. In symbols, P(H|E) is practically (or exactly?) 1, where the expression "P(H|E)" stands for "the probability of the hypothesis H given the evidence E." Obviously, this is conclusion-centric, as it tells the jury what they should believe about H (or at least what the examiner believes about this conclusion).
• "[I]t was 'more likely than not that this [Sig Sauer pistol] fired the cartridge casing' he was analyzing." This sounds like a statement that, given the marks (E), H is probably true: P(H|E) > 1/2. This statement is also conclusion-centered. It is a statement about the (probable) truth of H. However, in this case,
• "He explained that by agreeing to 'more likely' he was 'saying it is this firearm,' and by 'not' he was expressing that the chance of finding another firearm 'with the same signature or the same fingerprint is remote or really small.'" Without getting hung up on the dissonance in this compound sentence, this is a statement that given the markings (E), the probability of not-H is "really small." Mathematically, P(not-H|E) implies that H is not merely probable--it is really probable: P(H|E) is almost 1. Another conclusion-centric statement.
• "[T]he Sig Sauer pistol could not be excluded from the class of firearms that could have fired the cartridge case submitted for analysis." Also conclusion-centered. "Could not be excluded" means "is included as" (or "is consistent with") a possible source, it asserts that P(H|E) > 0. That focuses on the (possible) truth of the conclusion. But classifying this phrasing as conclusion-centric is a close case. If "could not be excluded" were a more neutral statement of the pair of observations themselves (such as "has the same characteristics" or "matches") then it is a statement about the evidence E rather than what E proves.

Evidence-centric Statements

• "Investigators also located a nine-millimeter Luger cartridge case ... on the street near where the shooting took place." With respect to the source conclusion, the statement of the caliber is evidence-centric in the sense that it specifies a feature of ammunition that might have used in the shooting without drawing a conclusion (be it categorical or probabilistic) about the gun taken from Tidd as having had that ammunition fired in it. That Tidd was found with a gun of the same caliber makes the characteristic incriminating evidence--it raises the probability of H. In symbols, P(H|caliber match) > P(H), where P(H) is the probability of H prior to considering the caliber data.
• “[I]t would be rare to find another firearm that exhibits that same ... pattern ... .” Evidence-centric. We have an assertion that the frequency of the incriminating marks is unusual in the population of guns that might have been used in the shooting. Finding a rare pattern supports H because it would be surprising to see the same (or more persuasive) evidence E if a different gun had been used. This is an assertion about the value of the evidence as proof of H rather than assertion about the truth of H itself.^\3/

So far we have encountered (1) statements of conclusions (or their probabilities) about a source hypothesis and (2) statements of the evidence ("it's a 9-mm caliber bullet," "the bullets have matching patterns," and so on) with regard to the source hypothesis. But there is a further category of evidence-centric statements that are more interesting. These evidence-centric statements go beyond descriptions of the data and comparisons using the data, but they do not go so far as to express conclusions. Rather, they are statements of the degree to which the evidence supports one hypothesis relative to alternative hypotheses.

I cannot find these in Tidd, for the examiner there was following the conclusion-centric paradigm espoused by the Association of Firearm and Tool Mark Examiners (AFTE). But they not hard to envision. Their defining feature is that they concern the probability of the evidence under the different hypotheses rather than the conclusion-centric probability of the hypothesis given the evidence. For example:

• The patterns (considering all the similarities and differences) are much more probable for bullets fired from defendant's gun than for bullets fired from different guns.
• The patterns are much more compatible with the hypothesis that bullets were fired from the defendant's gun than the hypothesis that they were fired from different guns.
• The patterns provide strong support for the hypothesis that bullets were fired from the defendant's gun rather than the hypothesis that they were fired from different guns.

These statements all flow from assessing how probable the patterns are for pairs of hypotheses. They look to P(E|H) rather than to the examiner's judgments of P(H|E). The underlying principle is that patterns that are more probable under one hypothesis than another are stronger evidence for the former hypothesis and thus support it instead of supporting the competing hypothesis. The hope is that experts will better assist the judge or jury by telling them about the strength of the evidence in this way instead of telling them which hypothesis to believe (to some degree of certainty).

Notes

1. David H. Kaye, The Nikumaroro Bones: How Can Forensic Scientists Assist Factfinders?, 6 Va. J. Crim. L. 101 (2018), https://www.ssrn.com/abstract=3177752. See also David H. Kaye, Forensic Statistics in the Courtroom, in Handbook of Forensic Statistics 225 (David Banks et al. eds., 2021), https://www.ssrn.com/abstract=3561914.
2. Frederick Schauer, The Proof: Uses of Evidence in Law, Politics, and Everything Else (2022).
3. Thus, "really small" is a qualitative expression of p-value. In statistics, a p-value is the the probability of such extreme evidence arising when the "alternative hypothesis" (such as "different gun") is true. Everything else being equal, the smaller the p-value, the harder it is to believe the alternative hypothesis is true.

Who Published Bayes' Theorem?

For the legal profession, "American Law Reports or ALR is a longstanding, highly trusted series of in-depth articles, called annotations, on specific legal issues."^\1/ But the report on "Use of Bayes' Theorem in Criminal and Civil Cases"^\2/ may be less worthy of trust than the average annotation. Historians and statisticians, at least, would be surprised at its opening sentences:

Bayes' Theorem, named for the English clergyman and scientist who published it in 1763, is a scientific principle of the likelihood ratio used to calculate conditional probabilities leading to a variety of statistical "Bayesian" methodologies. It speculates on the probability that a particular fact is true or that a particular event will occur, given our knowledge of one or more related facts or events.

There is no doubt (asymptotically speaking) that in 1763, Thomas Bayes did not publish the theorem that bears his name. He was dead. A close friend, the Reverend Richard Price, read the paper to the Royal Society of London after retrieving it from Bayes's papers and adding to it.^\3/ This expanded and modified version appeared in the Society's Philosophical Transactions.

The ALR's rendering of 18th Century history won't cause any problems for lawyers employing or confronting Bayesian analyses in court. Posthumous or otherwise, discovered first by Bayes or by someone else,^\4/ the theorem is what it is. But what will lawyers make of the idea that the theorem is "a scientific principle of the likelihood ratio" that "speculates" about conditional probability? Is the theorem mere speculation? Is it only about a likelihood ratio? No, and no.

The usual understanding is that a likelihood ratio is the data-based, objective part of the theorem and that speculation becomes an issue if and when the prior probability is not objective as well. As Judea Pearl and Dana MacKenzie explain,

Bayesian statistics gives us an objective way of combining the observed evidence with our prior knowledge (or subjective belief) to obtain a revised belief ... . Still, what frequentists could not abide was that Bayesians were allowing opinion, in the form of subjective probabilities, to intrude into the pristine kingdom of statistics. Mainstream statisticians were won over only grudgingly, when Bayesian analysis proved a superior tool for a variety of applications ... .^\5/

Fitting expert prior probabilities and likelihoods into the legal process poses some special problems for mainstream statisticians. That accommodation is the subject of continuing, if often repetitive, multidisciplinary dialog that does not fit neatly and quickly into a blog.

Notes

1. University of South Carolina School of Law, Legal Research, Analysis & Writing, https://guides.law.sc.edu/LRAWSpring/LRAW/alr.
2. Deborah F. Buckman, Annot., Use of Bayes' Theorem in Criminal and Civil Cases, 47 A.L.R.7th Art. 4 (2019).
3. Stephen M. Stigler, Richard Price, The First Bayesian, 33 Statistical Science 117 (2018), https://doi.org/10.1214/17-STS635; Martyn Hooper, Richard Price, Bayes’ Theorem, and God, Significance 36-39 (Feb. 2013), https://www.york.ac.uk/depts/maths/histstat/price.pdf.
4. Stephen M. Stigler, Who Discovered Bayes’s Theorem?, 37 Am. Stat. 290-296 (1983).
5. Judea Pearl & Dana MacKenzie, The Book of Why: The New Science of Cause and Effect 90 (2018).

NIST's Forensic Science Environmental Scan 2023

Yesterday, I received an email that included the following notice:

NIST released a new report today from an intensive 18-month, multi-pronged effort by the NIST Forensic Science team to assess the strategic opportunities for forensic science research and standards that are needed to drive significant advancements in the practice of forensic science over the coming decade.

The report is based on an extensive literature review, input from external stakeholders, and feedback from NIST subject matter experts. Throughout 2023, NIST conducted an assessment of the forensic science environment to inform strategic planning efforts. The resulting Forensic Science Environmental Scan 2023 report captured salient issues and trends across five different landscapes: governance, economic, societal, scientific and technological, and legal and regulatory. In addition, NIST held a roundtable in September 2023 with forensic science thought leaders from across forensic disciplines to discuss the long-term vision and strategic priorities for forensic science in the United States. NIST used these inputs to identify the grand challenges facing forensic science today, the strategies for addressing them through advances in research and standards, and the subsequent implementation of the advances into forensic science practice.

I can't say I have read the report yet, but I am posting the notice for anyone who wants to scan the "environmental scan."

Smith v. Arizona: The Facts and the Outcome

In A.E. Van Vogt’s murky but classic science fiction novel, The World of Null-A, the main character repeatedly dies, only to reappear in a new body with no memory of his former life. With help from extra brain matter, he moves on anyway. That, very roughly, is what the state of Arizona tried to accomplish in Smith v. Arizona, 144 S.Ct. 1785 (2024). It wrote off one expert witness but allowed her to live on through another expert. This plot twist left the defendant complaining that his Sixth Amendment “right … to be confronted with the witnesses against him” had been violated. This right, the Supreme Court repeatedly has held, prevents the prosecution from introducing “testimonial hearsay” without producing the author of the hearsay for cross-examination.

The Court’s struggle to define “testimonial” is becoming epic, and the definition of hearsay is one of those terrors of law school that leaves many students wondering whether they should have gone to some other professional or graduate school. One essential feature of hearsay is that the assertion is offered to prove “the truth of the matter asserted.” This phrase was at the center of the dispute in Smith, and the Court usefully clears up some of the confusion created by the opinions in Williams v. Illinois, 567 U.S. 50 (2012). Unfortunately, the Court did not stop there but also expressed some hasty thoughts about what might make a forensic-science expert’s hearsay statements nontestimonial.

This post describes the facts of Smith. They are drawn primarily from Justice Kagan's opinion for the Court. Ruminations on both parts of the majority opinion—and the three concurring opinions—may come later.

Jason Smith was arrested in a shed in Yuma county that contained “a large quantity of what appeared to be drugs and drug-related items.” He pleaded not guilty to charges of possession. The State asked “a crime lab run by the Arizona Department of Public Safety (DPS) for a ‘full scientific analysis.’” The State informed the lab of who the defendant was, what he was charged with, and the fact that a trial was pending. An analyst named Elizabeth Rast “ran the requested tests.” She typed out notes and submitted a signed report. The notes described each item, its weight, the tests performed, and a conclusion about its identity. The report of “results/interpretations” stated that four items “[c]ontained a usable quantity of methamphetamine,” three “[c]ontained a usable quantity of marijuana,” and one “[c]ontained a usable quantity of cannabis.”

After Rast “stopped working at the lab, for unexplained reasons,” prosecutors replaced her name on the “final pre-trial conference statement” with thast of a current employee. They promised that “Greggory Longoni, [a] forensic scientist (substitute expert),” who had no previous connection to the case, would “provide an independent opinion on the drug testing performed by Elizabeth Rast.” At trial, Longoni purported to give an independent opinion on the nature of the eight items. Yet, he arrived at his opinion by consulting nothing more than Rast's report and notes. He did no re-examination or retesting of his own. He merely

referred to those materials and related what was in them, item by item by item. As to each, he described the specific ‘scientific method[s]’ Rast had used … (e.g., a microscopic examination, a chemical color test, a gas chromatograph/mass spectrometer test). … [H]e stated that the testing had adhered to ‘general principles of chemistry,’ as well as to the lab's ‘policies and practices …. [H]e noted, for example, that Rast had run a “blank” to confirm that testing equipment was not contaminated. \1/

Readers of the opinion might think that "those materials" to which Longoni referred "[w]hen [he] took the stand" included Rast's report, but this is not quite correct. The prosecution was careful not to ask Longoni to recite the conclusions in the report. For example, the prosecutor posed these questions:

Q Let me be clear. You’re not testifying as to her report, you’re testifying as to review of lab notes?
A Correct.
Q In reviewing what was done, your knowledge and training as a forensic scientist, your knowledge and experience with DPS’s policies, practices, procedures, your knowledge of chemistry, the lab notes, the intake records, the chemicals used, the tests done, can you form an independent opinion on the identity of Item 26?
A Yes.

In this manner, Longoni told the jury that his “independent opinion” was that “Item 26 was 'a usable quantity of marijuana,' … Items 20A and 20B were 'usable quantit[ies] of methamphetamine,' and … Item 28 was '[a] usable quantity of cannabis.'”

The jury convicted, and Smith appealed on the ground that he was convicted via Rast’s written statements with no opportunity to cross-examine her. The State insisted that “Longoni testified about ‘his own independent opinions,’ even though making use of Rast's records.” The Arizona Court of Appeals agreed that Longoni was “present[ing] his independent expert opinions” as “based on his review of Rast's work.” It affirmed, relying on a 2014 case in which it had stated that an expert may testify to “the substance of a non-testifying expert's analysis, if such evidence forms the basis of the [testifying] expert's opinion.” State ex rel. Montgomery v. Karp, 236 Ariz. 120 (Ct. App. 2014). According to the Court of Appeals in Karp, the “underlying facts” are then “used only to show the basis of [the in-court witness's] opinion and not to prove their truth.” All this seemed so obvious to the court that it did not think its opinion was even worth publishing. The Arizona Supreme Court declined review without comment.

The U.S. Supreme Court was much more interested. In Williams v. Illinois, 567 U.S. 50 (2012), five Justices—a majority of the Court—had rejected this very reasoning. These Justices had dismissed the basis-only rationale as “legal fiction” (Thomas, J., concurring), “very weak,” “factually implausible,” “nonsense,” and “sheer fiction.” (Kagan, Scalia, Ginsburg & Sotomayor, JJ., dissenting and quoting D. Kaye, D. Bernstein, & J. Mnookin, The New Wigmore: Expert Evidence §4.10.1, pp. 196-197 (2d ed. 2011); id., §4.11.6, at 24 (Supp. 2012)). However, the remaining four Justices, in a plurality opinion written by Justice Alito, had vigorously advanced the not-for-its-truth theory for basis evidence.

The absence of a unifying rationale from a majority of the Court in Williams enabled the Arizona court to rely on one part of the Williams plurality opinion in one breath and to say that the “plurality decision … has limited if any precedential value” in the next. Meanwhile, other states had found the only-to-show-the-basis argument entirely unpersuasive.

And, so, for the fourth time, the Supreme Court granted a writ of certiorari to review either the introduction of forensic-science test findings made by analysts who were not presented for cross-examination or references to another analyst's findings by the expert presenting the scientific evidence. The petition propounded the question:

Whether the Confrontation Clause of the Sixth Amendment permits the prosecution in a criminal trial to present testimony by a substitute expert conveying the testimonial statements of a nontestifying forensic analyst, on the grounds … that the testifying expert offers some independent opinion and the analyst's statements are offered not for their truth but to explain the expert's opinion … .

On this narrow issue, the Court was unanimous: At least when the surrogate expert endorses the basis for his opinion as reliable, introducing the missing witness’s out-of-court but “testimonial” statements of and about her findings violates the Confrontation Clause. Justice Kagan wrote an opinion, joined in its discussion of this issue by every other Justice (except for Justice Alito and the Chief Justice). Justices Thomas and Gorsuch also filed concurring opinions that no one else joined. Justice Alito, joined only by the Chief Justice, filed an concurring opinion accusing the rest of the Court of “inflict[ing] a needless, unwarranted, and crippling wound on modern evidence law” and of “blow[ing] up the Federal Rules” with “a radical change” to the Rule 703 framework for admitting expert testimony. This is an obvious exaggeration, but explaining why merits a later post to unpack the logical foundation of Rule 703.

Beyond the hearsay issue, based on little more than sketchy suggestions from the Deputy Solicitor General at oral argument, Justice Kagan proposed that laboratory documentation could be introduced with no opportunity for confrontation if its “primary purpose” was for accreditation, quality control, or unofficial “notes to self.” Two of the seven Justices signing the majority opinion defected from this part of it. Justice Gorsuch expressed broader skepticism of the primary-purpose tests the Court had introduced in previous Confrontation Clause cases. Already, crime laboratories are talking about how to restructure or rewrite their documentation to fit into these new, nontestimonial categories. \2/ As I see it (so far), the nature of forensic science makes these alleged purposes too intertwined with the knowing production of evidence for a “primary purpose” test to disentangle them, but that conclusion also begs for later explanation.

In any event, the Supreme Court unanimously vacated the state court of appeals order affirming Smith's conviction and remanding for the Arizona Court of Appeals “[t]o address the additional issue of whether Rast's records were testimonial (including whether that issue was forfeited).” The state then conceded that it had forfeited the opportunity to argue that parts of the testimony were not “testimonial” hearsay under Crawford because it had not raised the argument in its appeal. The Arizona Court of Appeals remanded to the Yuma County Superior Court to vacate the judgment of conviction.

But this did not guarantee Jason Smith his freedom. The state could have tried him a second time. Perhaps Rast could be located to testify. Or, the lab could test the drugs again. (At trial, Longoni had testified that retesting would have taken only two to three hours.) After that, an analyst who did such testing could have testified to truly independent findings. On August 20, however, Jason Smith entered into a plea bargain, sparing the state the burden of retrying him. What concession he received in return, I do not know.

Note

1. 144 S.Ct. at 1799. But see id. at 1802 (declining to resolve a dispute between the parties as to whether Longoni's references to the report indicated that he partially relied on it as opposed to only the notes).
   
2. Consortium of Forensic Science Organizations, Untitled Memorandum, Aug. 20, 2024, https://thecfso.org/wp-content/uploads/2024/08/Smith-v-Arizona-Final.pdf

A Draft Standard on "Terminology for a Suspected Pattern of Dental Origin"

The Academy Standards Board (ASB) of the American Academy of Forensic Sciences is seeking comments by September 9 on a Technical Report 194, First Edition, 2024, titled "Terminology for a Suspected Pattern of Dental Origin." Although only a "template" for organizing comments is listed on the ASB website, the draft standard can be found at https://www.aafs.org/sites/default/files/media/documents/194_TR_Ballot01.pdf.

The OSAC page on “Standards Open for Comment” (which you won't see by perusing the OSAC website unless you click on “How To Work With Us” in the navigation pane) says “NOTE: This is OSAC 2021-N-0030, Terminology for a Suspected Pattern of Dental Origin, currently on the OSAC Registry.” Does that mean the ASB committee believed that nothing in the OSAC product, which emerged with no review from an advisory scientific and technical panel of experts, needed improvement? It is good to go public as an SDO-approved standard for terminology unless someone objects and proposes something better? 

Alas, it is not that good. Although a discussion of the choice of various terms to define and the definitions themselves could occupy pages, it is too tedious an undertaking for me to write or for many readers to plough through. Suffice it to say that some of the standard has a stream-of-consciousness feeling to it. Like "spurious observation anomaly not intrinsically present feature not related to the source."

The larger question is why try to promulgate a free-floating standard terminology rather than articulate standard procedures with appropriate terms? What might these standard practices be? Presumably, the proposed terminology is a precursor to performing "suspected pattern of dental origin analysis," which is defined as "forensic examination, analysis, and determination of the pattern for potential links to dental origins." This "potential links" study seems to be subdivided into (1) "bitemark assessment analysis," (2) "bitemark analysis," (3) "bitemark comparison analysis," and (4) "bitemark individualization analysis." Can any of these analyses produce results of "evidentiary value" (defined as "information of sufficient usefulness to serve as the basis for making an empirically significant scientific determination")?

The dentists are not prepared to say so. But neither are they willing to list in their bibliography any of the well-known articles and reports concluding that demonstrations of the scientific validity of these analyses are little more than wishful thinking. Rather than claim that these terms refer to procedures that have "evidentiary value" or opine that they lack such value, they merely note that their definitions are "not an endorsement of [the] scientific validity" of the processes they are supposed to describe. Apparently, the drafters from OSAC are agnostics rather than atheists. Or maybe they have their doubts about "bitemark individualization analysis." That phrase is marked "deprecated." Yet, the process of "visual comparison" is not deprecated as unvalidated or invalid, and a note suggests that other "individualization method[s]" for bitemarks could be just dandy.

So the best comment might be a recommendation to jettison this standard. Researchers can use their own clearly defined terms in devising and validating procedures that can be used in criminal investigations that involve what might be toothmarks, bitemarks, or wounds from other sources and mechanisms. When demonstrably valid procedures become available, the time will be ripe for a standard with uniform terminology.

Meanwhile, promulgating these terms and definitions, even with the agnostic disclaimers, risks encouraging the acceptance of dubious forensic "science." The very existence of an expert standard with these terms and definitions might suggest that the words describe something meaningful and encourage testimony that the field has a standardized system of some kind. I can imagine testimony that

The terms I am using in the analysis of what I have determined to be bitemarks are generally accepted in forensic odontology, medicine, and forensic science. I am following the ASB technical report on the subject. The report was produced with funding from the National Institute of Standards and Technology and is included and recommended for adoption by the government-supported Organization of Scientific Area Committees for Forensic Science.

Is this what forensic science and the law needs?

“Predictive or Profiling Evidence” and Diaz v. United States

Today the Association of American Law School’s (AALS) Section on Evidence distributed the following announcement to its members:

In a divided decision, the Supreme Court recently concluded that expert testimony about the likely mental state of individuals arrested with drugs in their position is admissible in criminal trials under the Federal Rules of Evidence in Diaz v. United States, 602 U. S. ____ (2024). This decision was sharply contested in its own right, but also drew attention to an area of broader controversy in the law of evidence: the increasing use of "predictive" or "profiling" evidence, by which expert witnesses present testimony suggesting that an individual is more or less likely to have had a particular mental state or behaved in a particular way based on their personal circumstances or characteristics. Scholars who have written about this phenomenon have expressed disquiet about the use of such evidence (and some courts have limited the use of such evidence in edge cases involving particular prejudice or overbroad characterizations), but no consensus has emerged as to the reasons for objecting to predictive evidence or as to how to systematically distinguish between such evidence and other forms of indirect and circumstantial evidence that is routinely admitted. This panel brings together scholars to discuss predictive and profiling evidence from a variety of perspectives. Was the Supreme Court right to find such evidence consistent with the federal rules governing the admissibility of expert evidence? Is such evidence generally consistent with due process and equal protection concerns? Is it generally empirically sound? Do we need new federal rules or common law doctrines to limit the admissibility of some forms of predictive or profiling evidence?

Who can object to a scholarly program on the subject, even if it is hardly new, having been the subject of a multitude of opinions, a number of statutes, and even a previous AALS program decades ago? Hopefully, the courts and the professoriate have made some progress in understanding what has come to be called "framework evidence." Neither would it be fair to criticize a necessarily brief announcement for not defining "predictive or profiling evidence." Presumably, the evidence teachers do not need a definition.

But I was surprised to read my fellow law professors’ sweeping characterization of Diaz v. United States, 144 S.Ct. 1727 (2024), as establishing that "such evidence [is] consistent with the federal rules governing the admissibility of expert evidence" and "that expert testimony about the likely mental state of individuals arrested with drugs in their position [sic?] is admissible in criminal trials under the Federal Rules of Evidence." Although I doubt that the announcement will cause its narrow range of readers to believe that the case stands for more than it does, I do worry that the same kind of language will crop up in undiscerning judicial opinions and commentary on Diaz. Therefore, it may be be worth listing some problematic aspects of the statements.

First, the case plainly held that certain "expert testimony about the likely mental state of individuals arrested with drugs in their position [possession?] is" not "admissible in criminal trials under the Federal Rules of Evidence." Every Justice agreed that no expert can testify that a defendant charged with importing proscribed drugs knew that they were transporting drugs. That would be explicit ultimate-opinion testimony on a criminal defendant's state of mind. (There are cases limiting the Rule 704(b) ban to mental health professionals, but the Court did not consider that possible way to interpret the rule. It stuck with a more literal reading of the text.)

Second, no “predictive or profiling evidence” was introduced in the case, and only one Justice thought it worth discussing. Certainly, Diaz is not a case of a criminal profiler predicting (“inferring” would be more precise) the characteristics of a criminal from the type or manner of the crimes under investigation (or any other such “predictive or profiling evidence”). As the excerpts from the trial transcript reproduced below show, the witness did not claim such expertise; furthermore, the trial judge barred him from stating a belief about the defendant’s knowledge (although one could well think that his testimony made it plain enough what his belief was). 

Third, the issue before the Court was not admissibility under the rules of evidence writ large. It was the scope of a single part of a solitary rule. Federal Rule 704(b), which has no counterpart in the rules of most states, declares that no expert witness may "state an opinion about whether the defendant did or did not have a mental state or condition that constitutes an element of the crime charged or of a defense" because "[t]hose matters are for the trier of fact alone." All that the Diaz Court held was that when such an opinion is not stated and is an inference that does not necessarily follow (as a matter of deductive logic) from the witness’s statements, then Rule 704(b) does not preclude its admission. It did not—and could not—have held that the Rule makes it admissible. Cf. David H. Kaye, The Ultimate Opinion Rule and Forensic Science Identification, 60 Jurimetrics J. 75 (2020).

Thus, Diaz should not be read as supporting—or opposing—the use of “predictive or profiling evidence” generally, or even in the subcategory of testimony offered to prove a defendant’s state-of-mind. 

More thoughts on the three opinions in the case and how the result fits into the range of possible interpretations of Rule 704(b) will appear in the upcoming supplement to The New Wigmore on Evidence: Expert Evidence § 2.2.3(b) (available online in VitalLaw) and, time permitting, in a further positing about the case here.

---

Excerpts from Trial Transcript
Mar. 18, 2021

The most pertinent portion of the testimony at issue in Diaz is as follows (with italics added):

BY MR. OLAH [Assistant US Attorney]:
Q. Where do you work?
A. I’m a special agent with Homeland Security Investigations.
Q. And how long have you been with HSI?
A. I’ve been a special agent since 1996. So going on 20 — I believe 28 years.
Q. Were you in law enforcement before joining HSI?
A. Prior to becoming a special agent, I was a U.S. Border Patrol agent. And prior to that, I was a sheriff’s corrections deputy.
***
Q. Have you been involved in drug trafficking investigations as a special agent with HSI?
A. Yes, I have.
Q. Approximately how many such investigations?
A. I’ve been involved in over 500 investigations dealing with distribution of drugs and also the – which would include the importation of drugs.
Q. And can you summarize for the jury the various investigation techniques you’ve used?
A. The techniques I’ve used, I’ve utilized wiretaps, where you actually listen to a drug trafficker talk on the telephone and how they conduct business. I’ve done controlled purchases where I utilized an undercover agent or a cooperating source. And we actually go out on the street and buy the drugs. I’ve spoken with cooperating defendants that have been arrested for drug trafficking related offenses. I’ve talked to cooperating sources that have information related to the distribution of drugs and drug trafficking organizations. I have spoken with other agents that work drug trafficking organizations and have worked on task forces with other agencies such as the Federal Bureau of Investigation, the Drug Enforcement Administration, and local police departments dealing with drug trafficking related crimes.
***
Q. Agent Flood, why are drugs imported into the United States?
MS. IREDALE: Objection, 401.
THE COURT: Overruled.
THE WITNESS: Based upon drugs that – some drugs are manufactured in Mexico and outside the United States. Therefore, they’re brought across the border, into the United States to be sold.
BY MR. OLAH:
***
Q. With respect to vehicles, can you describe the general process of movement from Mexico to wherever it goes?
A. From Mexico, they are packaged. They are put into *** a vehicle. *** I have seen drugs hidden in every area of a vehicle. *** And then they are transported from point A to point B across the border.
Q. And based on your training and experience, are the transporters compensated for their efforts?
A. Yes. It’s a job. It’s to take it from point A to point B.
***
Q. Agent Flood, based on your training and experience, are large quantities of drugs entrusted to drivers that are unaware of those drugs?
MS. IREDALE: Objection. 401, 403.
THE COURT: Overruled.
THE WITNESS: No. In extreme circumstances – actually, in most circumstances, the driver knows they are hired. It’s a business. They are hired to take the drugs from point A to point B.
BY MR. OLAH:
Q. And why aren’t – why don’t they use unknowing couriers, generally?
MS. IREDALE: Objection. 401, 403.
THE COURT: Overruled. You may answer.
THE WITNESS: Generally, it’s a risk of your – your cargo not making it to the new market; not knowing where it’s going; not being able to retrieve it at the ending point, at your point B. So there’s a risk of not delivering your product and, therefore, you’re not going to make any money.
***
Cross Examination
***
Q. So you said that unknowing couriers are very rare.
A. Yes.
***
Q. You work for HSI. Right?
A. Correct.
Q. And you’re aware that your own agency has identified many schemes where drug trafficking organizations use unknowing couriers. Right?
A. I – I know of three schemes that were primarily identified as being possible for an unknowing courier. It doesn’t necessarily mean that they are unknowing couriers. ***

Volunteer Bias in Interlaboratory Studies

The National Institute for Standards and Technology (NIST) is soliciting laboratories to join an "interlaboratory study" of CG-MS (gas chromatography mass spectrometry) for seized drug analysis. The announcement that I received, in abbreviated form, reads:

Forensic Science Quality Assurance Program
Seized Drugs General Method for GC-MS Reporting Limits Study

. . .
Study Design, Purpose, and Rationale
The goals of the study are 1) to capture the range of methods, instrumentation, and analytical approaches used in the community, 2) investigate mass spectral variability across methods, and 3) investigate how different reporting practices effect the limit of seized drug reporting.
Timeline and Commitment
Registration is currently open and will close on July 5, 2024. To participate in this study, laboratories must be accredited forensic laboratories based in the United States and have a valid Schedule I & II DEA license, a validated seized drug screening method using GC-MS, and a documented reporting practice. To be considered for the study, participants will be required to complete a pre-study questionnaire pertaining to the method that will be used for sample analysis. After acceptance into the study, participants will be provided a kit of 10 solutions containing mixtures of controlled substances and asked to analyze the solutions and report whether the analytes are present above their established reporting thresholds. Participants will also be required to report chromatographic peak height/area and retention time of each peak and provide the raw datafile from each run. Standards used for comparison will also be reported. [D]ue to a limited number of available kits, completion of the pre-study questionnaire does not guarantee acceptance into the study. . . .
Publication of Results
Upon closure of data entry, laboratories will receive a preliminary report containing a summary of reported data, consensus results, and a summary of analytes present in each mixture. [A] final report . . . will be made publicly available by Spring 2025. . . . NIST will not knowingly reveal laboratory identities associated with study results.
For questions, contact andrea.yarberry@nist.gov
To signup, go to: https://forms.gle/gPDU9aENHguPkw1D7

The effort is laudable, but one might ask why NIST is not beginning with a sampling frame of laboratories created to represent "the community" and then drawing a probability sample from this list. Will the laboratories that notice the announcement and ask to participate present the full "range of methods, instrumentation, and analytical approaches used in the community"? Will the volunteer sample be skewed toward higher quality labs? Will it include all the "different reporting practices [that] effect [sic] the limit of seized drug reporting [whatever this "limit" denotes in the population of laboratories]"?

Rigor in sampling may not be required to answer certain questions, but it seems relevant to determining what the sign-up form refers to as "the current landscape of GC-MS methods and associated reporting practices and how those factors effect [sic] the concentration of drug that is/is not ultimately reported." Certainly, it should be a consideration for the legal community if and when the results of the study are presented as an indication of "the known or potential rate of error" for GC-MS analysis as practiced in forensic-science laboratories (Daubert v. Merrell Dow Pharm., 509 U.S. 579, 594 (1993)).

ISO Standards on Forensic Science: Pay to Play?

"ISO, the International Organization for Standardization, brings global experts together to agree on the best way of doing things – for anything from making a product to managing a process." ^1/ For the last few years, it has been devising the following overarching set of standards for all of forensic science:

• Forensic sciences (TC 272) ISO/DIS 21043-1, Forensic sciences - Part 1: Terms and definitions - 5/27/2024, $58.00
• ISO/DIS 21043-3, Forensic Sciences - Part 3: Analysis - 5/26/2024, $62.00
• ISO/DIS 21043-4, Forensic Sciences - Part 4: Interpretation - 5/26/2024, $67.00
• ISO/DIS 21043-5, Forensic Sciences - Part 5: Reporting - 5/26/2024, $53.00 ^2/

These are

part of a series which, when completed, will include the different components of the forensic process from scene to courtroom ... . The series describes primarily “what” is standardized, not the “how” or “who”. Best practice manuals and standard operating procedures should describe “how” the requirements of this document would be met. ^3/

It sounds like the standards in progress will not specify "the best way of doing things." Will they merely list the things that are in need of "standardization"? Will they be too open-ended to constitute what the U.S. Supreme Court refers to as "standards controlling the technique's operation"^4/?

I cannot answer these questions because I have not seen the drafts that were open for public comment. Members of the public cannot read the drafts without paying IS0 the $240 listed above. If anyone who has paid to play has thoughts on these documents that they would like to share beyond the TC (Technical Committee) that drafted the standards, I'll post them--at no charge.

Notes

1. Int'l Org. for Standardization, About ISO.
2. ANSI Standards Action, Mar. 15, 2024, at 48.
3. ISO 21043-1:2018(en) Forensic sciences — Part 1: Terms and definitions.
4. Daubert v. Merrell Dow Pharm., 509 U.S. 579, 594 (1993).

What's Uniqueness Got to Do with It?

Columbia University has announced that "AI Discovers That Not Every Fingerprint Is Unique"! The subtitle of the press release of January 10, 2024, boldly claims that

Columbia engineers have built a new AI that shatters a long-held belief in forensics–that fingerprints from different fingers of the same person are unique. It turns out they are similar, only we’ve been comparing fingerprints the wrong way!

Forensic Magazine immediately and uncritically rebroadcast (quoting verbatim without acknowledgment from the press release) the confused statements about uniqueness. According to the Columbia release and Forensic Magazine, "It’s a well-accepted fact in the forensics community that fingerprints of different fingers of the same person—or intra-person fingerprints—are unique and therefore unmatchable." Forensics Magazine adds that "Now, a new study shows an AI-based system has learned to correlate a person’s unique fingerprints with a high degree of accuracy."

Does this mean that the "well-accepted fact" and "long-held belief" in uniqueness been shattered or not? Clearly, not. The study is about similarity, not uniqueness. In fact, uniqueness has essentially nothing to do with it. I can classify equilateral triangles drawn on a flat surface as triangles rather than as other regular polygons whether or not the triangles are each different enough from one another (uniqueness within the set of triangles) that I notice these differences. To say that objects "are unique and therefore unmatchable" is a nonsequitur. A human genome is probably unique to that individual, but forensic geneticists know that six-locus STR profiles are "matchable" to those of other individuals in the population. A cold hit to a person who could not have been the source of the six-locus profile in the U.K. database occurred long ago (as was to be expected for the random-match probabilities of the genotypes).

Perhaps the myth that the study shatters is that it is impossible to distinguish fingerprints left by different fingers of the same individual X from fingerprints left by fingers of different individuals (not-X). But there is no obvious reason why this would be impossible even if every print is distinguishable from every other print (uniqueness).

The Columbia press release describes the study design this way:

[U]ndergraduate senior Gabe Guo ... who had no prior knowledge of forensics, found a public U.S. government database of some 60,000 fingerprints and fed them in pairs into an artificial intelligence-based system known as a deep contrastive network. Sometimes the pairs belonged to the same person (but different fingers), and sometimes they belonged to different people.

Over time, the AI system, which the team designed by modifying a state-of-the-art framework, got better at telling when seemingly unique fingerprints belonged to the same person and when they didn’t. The accuracy for a single pair reached 77%. When multiple pairs were presented, the accuracy shot significantly higher, potentially increasing current forensic efficiency by more than tenfold.

The press release reported the following odd facts about the authors' attempts to publish their study in a scientific journal:

Once the team verified their results, they quickly sent the findings to a well-established forensics journal, only to receive a rejection a few months later. The anonymous expert reviewer and editor concluded that “It is well known that every fingerprint is unique,” and therefore it would not be possible to detect similarities even if the fingerprints came from the same person.

The team ... fed their AI system even more data, and the system kept improving. Aware of the forensics community's skepticism, the team opted to submit their manuscript to a more general audience. The paper was rejected again, but [Professor Hod] Lipson ... appealed. “I don’t normally argue editorial decisions, but this finding was too important to ignore,” he said. “If this information tips the balance, then I imagine that cold cases could be revived, and even that innocent people could be acquitted.” ...

After more back and forth, the paper was finally accepted for publication by Science Advances. ... One of the sticking points was the following question: What alternative information was the AI actually using that has evaded decades of forensic analysis? ... “The AI was not using ... the patterns used in traditional fingerprint comparison,” said Guo ... . “Instead, it was using something else, related to the angles and curvatures of the swirls and loops in the center of the fingerprint.”

Proprietary fingerprint matching algorithms also do not arrive at matches the way human examiners do. They "see" different features in the patterns and tend to rank the top candidates for true matches in a database trawl differently than the human experts. Again, however, these facts about automated systems neither prove nor disprove claims of uniqueness. And, theoretical uniqueness has little or nothing to do with the actual probative value of assertions of matches by humans, automated systems, or both.

Although not directly applicable, the day after the publicity on the Guo et al. paper, I came across the following report on "Limitations of AI-based predictive models" in a weekly survey of papers in Science:

A central promise of artificial intelligence (AI) in health care is that large datasets can be mined to predict and identify the best course of care for future patients. Unfortunately, we do not know how these models would perform on new patients because they are rarely tested prospectively on truly independent patient samples. Chekroud et al. showed that machine learning models routinely achieve perfect performance in one dataset even when that dataset is a large international multisite clinical trial (see the Perspective by Petzschner). However, when that exact model was tested in truly independent clinical trials, performance fell to chance levels. Even when building what should be a more robust model by aggregating across a group of similar multisite trials, subsequent predictive performance remained poor. -- Science p. 164, 10.1126/science.adg8538; see also p. 149, 10.1126/science.adm9218

Note: This posting was last modified on 1/12/24 2:45 PM

SWGDE's Best Practices for Remote Collection of Digital Evidence from a Networked Computing Environment

SWGDE 22-F-003-1.0, Best Practices for Remote Collection of Digital Evidence from a Networked Computing Environment, is a forensic-science standard proposed for inclusion on the Organization of Scientific Area Committees for Forensic Science (OSAC) Registry—"a repository of selected published and proposed standards … to promote valid, reliable, and reproducible forensic results.”

The best practices “may not be applicable in all circumstances.” In fact, “[w]hen warranted, an examiner may deviate from these best practices and still obtain reliable, defensible results.” I guess that is why they are called best practices rather than required practices. But what circumstances would justify using anything but the best practices? On this question, the standard is silent. It merely says that “[i]f examiners encounter situations warranting deviation from best practices, they should thoroughly document the specifics of the situation and actions taken.” 

Likewise, the best practices for “preparation” seem rather rudimentary. “Examiners should ascertain the appropriate means of acquiring data from identified networked sources.” No doubt, but how could they ever prepare to collect digital information without ascertaining how to acquire data? What makes a means “appropriate”? All that a digital evidence expert can glean from this document is that he or she “should be aware of the limitations of each acquisition method and consider actions to mitigate these limitations if appropriate” and should consider “methods and limitation variables as they relate to various operating systems.” How does such advice regularize or improve anything?

Same thing with a recommendation that “[p]rior to the acquisition process, examiners should prepare their destination media”? What steps for preparing the destination media are best? Well, [s]terilization of destination media [whatever the process of “sterilization” is in this context] is not generally required.” But it is required “when needed to satisfy administrative or organizational requirements or when a specific analysis process makes it a prudent practice.” When would sterilization be prudent? The drafters do not seem to be very sure. “[E]xaminers may need to sanitize destination media provided to an external recipient to ensure extraneous data is not disclosed.” Or maybe they don’t? “Examiners may also be required to destroy copies of existing data to comply with legal or regulatory requirements.” Few people would dispute that the best practice is to follow the law, but examiners hardly need best practices documents from standards developing organizations to know that.

The standard is indeterminate when it comes to what it calls “triage”—“preview[ing] the contents of potential data sources prior to acquisition.” We learn that “[e]xaminers may need to preview the contents of potential data sources prior to acquisition” to “reduce the amount of data acquired, avoid acquiring irrelevant information, or comply with restrictions on search authority.” What amount of data makes "triage" a best practice? How does the examiner know that irrelevant information may be present? Why can "triage" sometimes be skipped? When it is desirable and how should it be done? The standard merely observes that “[t]here may be multiple iterations of triage … .” When are multiple iterations advisable? Well, it “depend[s] on the complexity of the investigation.” Equally vague is the truism that “[e]xaminers should use forensically sound processes to conduct triage to the extent possible.” 

Finally, designating steps like “perform acquisition” and “validate collected data” as “best practices” does little to inform examiners of how to collect digital evidence from a network. To be fair, a few parts of the standard are more concrete, and, possibly, other SWGDE standards fill in the blanks. But, on its face, much of this remote acquisition standard simply gestures toward possible best practices. It does not expound them. In this respect, it resembles other forensic-science standards that emerge from forensic-science standards developing organizations only to be criticized as vague at critical points.

"Conditions Regarding the Use of SWGDE Documents"

SWGDE is the Scientific Working Group on Digital Evidence. Its website describes it as a meta-organization—a group that “brings together organizations actively engaged in the field of digital and multimedia evidence to foster communication and cooperation as well as to ensure quality and consistency within the forensic community.” Structured as a non-profit corporation, it solicits "your donations or sponsorship." \1/ Its 70 “member organizations” consist of (by a quick and possibly error-prone categorization and count):

• 16 local, state, and federal police agencies; \2/
• 4 digital forensics software companies; \3/
• 18 training and consulting organizations; \4/
• 6 prosecutors' offices; \5/
• 8 crime laboratories and coroners' or medical examiners' offices; \6/
• 3 major corporations; \7/
• 3 universities; \8/
• A swath of federal executive agencies (or parts of them), including NASA, NIST, and the Departments of Defense, Homeland Security, Interior, Justice, Labor, and Treasury. \9/

SWGDE has produced “countless academic papers,” although none are listed on its website. SWGDE "encourages the use and redistribution of our documents," but it regards them as private property. It states that "The Disclaimer and Redistribution policies (also included in the cover pages to each document) also establish what is considered SWGDE's Intellectual Property."

These policies are unusual, if not unique, among among standards developing organizations. An IP lawyer would find it odd, I think, to read that admonitions such as the following are part of an author's copyright:

Individuals may not misstate and/or over represent [sic] duties and responsibilities of SWGDE work. This includes claiming oneself as a contributing member without actively participating in SWGDE meetings; claiming oneself as an officer of SWGDE without serving as such ... .

With respect to actual IP rights, SWGDE purports to control not only the specific expression of ideas—as allowed by copyright law—but all "information" contained in its documents—a claim that far exceeds the scope of copyright. It imposes the following "condition to the use of this document (and the information contained herein) in any judicial, administrative, legislative, or other adjudicatory proceeding in the United States or elsewhere":

notification by e-mail before or contemporaneous to the introduction of this document, or any portion thereof, as a marked exhibit offered for or moved into evidence in such proceeding. The notification should include: 1) The formal name of the proceeding, including docket number or similar identifier; 2) the name and location of the body conducting the hearing or proceeding; and 3) the name, mailing address (if available) and contact information of the party offering or moving the document into evidence. Subsequent to the use of this document in the proceeding please notify SWGDE as to the outcome of the matter.

As author (or otherwise), an SDO certainly can ask readers to do anything it would like them to do with its publications—and the SWGDE "conditions regarding use" do contain the phrase "the SWGDE requests." Even reformulating the paragraph as a polite request rather than a demand supposedly supported by copyright law, however, one might ask what legislative proceeding with a "formal name" would have a forensic-science standard "offered or moved into evidence." Impeachment and subsequent trial, I guess.

Notes

1. Neither its full name nor its acronym turned up in a search of the IRS list of tax-exempt 501(c)(3) organizations, so donors seeking a charitable deduction on their taxes might need to inquire further.
2. As listed on the website, they are the Columbus, Ohio Police Department; Eugene Police Department; Florida Department of Law Enforcement (FDLE); Lawrence, KS Police Department; Johnson County, KS Sheriff's Office; Los Angeles County, CA Sheriff's Department; Louisville, KY Metro Police Department; Massachusetts State Police; Oklahoma State Bureau of Investigation; New York State Police; New York City Police Department (NYPD); Plano, TX Police Department; Seattle Police Department; Weld County, CO Sheriff's Office; US Department of Justice - Federal Bureau of Investigation (FBI); US Department of Homeland Security - US Secret Service (USSS); and the US Postal Inspection Service (USPIS).
3. Amped Software USA Inc.; AVPreserve; BlackRainbow; SecurCube.
4. National White Collar Crime Center (NW3C); Digital Forensics.US LLC / Veritek Cyber Solutions; MetrTech Consultancy; Midwest Forensic Consultants LLC; Hexordia; Forensic Data Corp; Forensic Video & Audio Associates, Inc; Laggui And Associates, Inc.; Loehrs Forensics; N1 Discovery; Precision Digital Forensics, Inc. (PDFI); Premier Cellular Mapping & Analytics; Primeau Forensics, Recorded Evidence Solutions, LLC; AVPreserve; LTD; BEK TEK; TransPerfect Legal Solutions; VTO Labs; Unique Wire, Inc
5. Adams County, CO District Attorney's Office; Burlington County, NJ Prosecutor's Office; Dallas County, TX District Attorneys Office; Middlesex County, NJ Prosecutor's Office; State of Wisconsin Department of Justice; US Department of Justice - Executive Office United States Attorney Generals Office.
6. City of Phoenix, AZ Crime Lab; Houston Forensic Science Center; Boulder County Coroner's Office; Miami-Dade County, FL; Medical Examiner Department; Virginia Department of Forensic Science; Westchester County, NY Forensic Lab; North Carolina State Crime Laboratory; and the US Department of Defense - Army Criminal Investigation Laboratory (Army CID).
7. Carrier Corporation; Target Corporation; and Walmart Stores Inc.
8. San Jose State University; University of Colorado Denver - National Center for Media Forensics (NCMF); University of Wisconsin Stevens Point.
9. NASA Office of Inspector General - Computer Crimes Division; National Institute of Standards and Technology; Treasury Inspector General for Tax Administration; US Department of Defense - Defense Cyber Crimes Center (DC3); US Department of Homeland Security - Homeland Security Investigations (HSI); US Department of Justice - Office of the Inspector General (DOJ OIG); US Department of Labor - Office of Inspector General (DOL OIG); US Department of the Interior - Office of the Inspector General (DOI OIG); US Department of Treasury - Internal Revenue Service (IRS); US Postal Service - Office of Inspector General (Postal OIG). Yet another organizational member is the Puerto Rico Office of the Comptroller, Division of Database Analysis, Digital Forensic and Technological Development.

How Accurate Is Mass Spectrometry in Forensic Toxicology?

Mass spectrometry (MS) is the "[s]tudy of matter through the formation of gas-phase ions that are characterized using mass spectrometers by their mass, charge, structure, and/or physicochemical properties." ANSI-ASB Standard 098 for Mass Spectral Analysis in Forensic Toxicology § 3.11 (2023). MS has become "the preferred technique for the confirmation of drugs, drug metabolites, relevant xenobiotics, and endogenous analytes in forensic toxicology." Id. at Foreword.

But no "criteria for the acceptance of mass spectrometry data have been ... universally applied by practicing forensic toxicologists." Id. Therefore, the American Academy of Forensic Sciences' Academy Standards Board (ASB) promulgated a "consensus based forensic standard[] within a framework accredited by the American National Standards Institute (ANSI)," id., that provides "minimum requirements." Id. § 1.

To a nonexpert reader (like me), the minimum criteria for the accuracy of MS "confirmation" are not apparent. Consider Section 4.2.1 on "Full-Scan Acquisition using a Single-Stage Low-Resolution Mass Analyzer." It begins with the formal requirement that

[T]he following shall be met when using a single-stage low-resolution mass analyzer in full-scan mode.
a) A minimum of a single diagnostic ion shall be monitored.

It is hard to imagine an MS test method that would not meet the single-ion minimum. Perhaps what makes this requirement meaningful is that the one or more ions must be "diagnostic." However, this adjective begs the question of what the minimum requirement for diagnositicity should be. A "diagnostic ion" is a "molecular ion or fragment ion whose presence and relative abundance are characteristic of the targeted analyte." Id. § 3.4. So what makes an ion "characteristic"? Must it always be present (in some relative abundance) when the "targeted analyte" is in the specimen (at or above some limit of detection)? That would make the ion a marker for the analyte with perfect sensitivity: Pr(ion|analyte) = 1. Even so, it would not be characteristic of the analyte unless its presence is highly specific, that is, unless Pr(no-such-ion|something-else) ≅ 1. But the standard contains no minimum values for sensitivity, specificity, or the likelihood ratio Pr(ion|analyte) / Pr(ion|something-else), which quantifies the positive diagnostic value of a binary test. \1/

This is not to say that there are no minimum requirements in the standard. There certainly are. For example, Section 4.2.1 continues:

b) When monitoring more than one diagnostic ion:
1. ratios of diagnostic ions shall agree with those calculated from a concurrently analyzed reference material given the tolerances shown in Table 1; OR
2. the spectrum shall be compared using an appropriate library search and be above a pre-defined match factor as demonstrated through method validation.

But the standard does not explain how the tolerances in Table 1 were determined. What are the conditional error probabilities that they produce?

Likewise, establishing a critical value for the "match factor" \2/ before using it is essential to a frequentist decision rule, but what are the operating characteristics of the rule? "Method validation" is governed (to the extent that voluntary standards govern anything) by ANSI-ASB 036, Standard Practices for Method Validation in Forensic Toxicology (2019). This standard requires testing to establish that a method is "fit for purpose," but it gives no accuracy rates that would fulfill this vague directive.

Firms that sell antibody test kits for detecting Covid-19 infections no longer can sell whatever they deem is fit for purpose. In May 2020, the FDA stopped issuing emergency use permits for these diagnostic tests without validation showing that they "are 90% 'sensitive,' or able to detect coronavirus antibodies, and 95% 'specific,' or able to avoid false positive results." \3/ Forensic toxicologists do not seem to have proposed such minimum requirements for MS tests.

NOTES

1. Other toxicology standards refer to ASB 098 as if it indicates what it required to apply the label "diagnostic." ANSI/ASB 113, Standard for Identification Criteria in Forensic Toxicology, § 4.5.2 (2023) ("All precursor and product ions are required to be diagnostic per ASB Standard 098, Standard for Mass Spectral Data Acceptance in Forensic Toxicology (2022).").
2. Section 3.13 defines "match factor" as a "mathematical value [a scalar?] that indicates the degree of similarity between an unknown spectrum and a reference spectrum."
3. See How Do Forensic-science Tests Compare to Emergency COVID-19 Tests?, Forensic Sci., Stat. & L., May 5, 2020 (quoting Thomas M. Burton, FDA Sets Standards for Coronavirus Antibody Tests in Crackdown on Fraud, Wall Street J., Updated May 4, 2020 8:24 pm ET, https://www.wsj.com/articles/fda-sets-standards-for-coronavirus-antibody-tests-in-crackdown-on-fraud-11588605373).

Use with Caution: NIJ's Training Course in Population Genetics and Statistics for Forensic Analysts

The National Institute of Justice (NIJ) "is the research, development and evaluation agency of the U.S. Department of Justice . . . dedicated to improving knowledge and understanding of crime and justice issues through science." It offers a series of webpages and video recordings (a "training course") on Population Genetics and Statistics for Forensic Analysts. The course should be approached with caution. I have not worked through all the pages and videos, but here are a few things that rang alarm bells:

NIJ's Training	Comment
Many statisticians have employed what is known as Bayesian probability ... which is based on probability as a measure of one's degree of belief. This type of probability is conditional in that the outcome is based on knowing information about other circumstances and is derived from Bayes Theorem.	Bayes' rule applies to both objective and subjective probabilities. Both types of probability include conditional probabilities. The "type of probability" is not derived from Bayes' Theorem.
Conditional probability, by definition, is the probability P of an event A given that an event B has occurred. ... Take the example of a die with six sides. If one was to throw the die, the probability of it landing on any one side would be 1/6. This probability, however, assumes that the die is not weighted or rigged in any way, and that all of the sides contain a different number. If this were not true, then the probability would be conditional and dependent on these other factors.	The "other factors" are nothing more than part of the description of the experiment whose outcomes are the events that are observed. They are not conditioning events in a sample space.
The following equation can be used to determine the probability of the evidence given that a presumed individual is the contributor rather than a random individual in the population: LR = P(E/H1) / P(E/H0) ... . In the case of a single source sample, the hypothesis for the numerator (the suspect is the source of the DNA) is a given, and thus reduces to 1. This reduces to: LR = 1/ P(E/H0) which is simply 1/P, where P is the genotype frequency.	The hypothesis for the numerator of a likelihood ratio is always "a given"--that is, it goes on the right-hand-side of the expression for a conditional probability. So is the hypothesis in the denominator. Neither probability "reduces to 1" for that reason. Only if the "evidence" is the true genotype in both the recovered sample and the sample from the defendant can it be said that P(E|H1) = 1. In other words, to say that the probability of a reported match is 1 if the defendant is the source treats the probability of laboratory error as zero. That may be acceptable as a simplifying assumption, but the assumption should be made visible in a training course.
Although likelihood ratios can be used for determining the significance of single source crime stains, they are more commonly used in mixture interpretation. ... The use of any formula for mixture interpretation should only be applied to cases in which the analyst can reasonably assume "that all contributors to the mixed profile are unrelated to each other, and that allelic dropout has no practical impact."	This limitation does not apply to modern probabilistic genotyping software!

Is "Match Form" Testimony Poor Form?

The likelihood ratio (LR) is essentially a number that expresses how many times more probable the data from an experiment are if one hypothesis is true than if another hypothesis is true. For example, suppose we make a single measurement of the height of a known individual. Then we do the same for an individual who is covered from head to foot by a sheet. We want know if we have measured the same individual twice or two different individuals once. The closer the two measured heights are to one another, the more the measurements support the same-source hypothesis as opposed to the different-source hypothesis.

Why? Because closer measurements are more probable for same-source pairs than for different-source pairs. This implies that in repeated experiments with some proportion of same-source and different-source pairs, the closer measurements will tend to filter out the different-source pairs (which tend to have more distance between the two measurements) and to include more same-source pairs (which tend to be marked by the more similar measurements).

By quantifying the relative probability for the data given each hypothesis, the LR indicates how well a given degree of similarity discriminates between the hypotheses. Its value is

LR = Probability(data | H₁) / Probability(data | H₂),

where H₁ is the same-source hypothesis and H₂ is the different-source hypothesis.

Likelihood ratios are routinely reported in cases with samples from crime scenes or victims that contain DNA from several individuals. A DNA analyst might testify that the electropherograms are ten thousand times more probable if the defendant's DNA is present than if an unrelated person's DNA is there. \1/ We may call such statements "relative-probability-of-the-data" testimony.

But some DNA experts prefer what they call a "match form" for the presentation. \2/ An example of a "match form" statement is that “[a] match between the shoes … and [the defendant] is 9.67 thousand times more probable than a coincidental match to an unrelated African-American person.” \3/ More generally, a match-form presentation states that “a match between the evidence and reference [samples] is (some number) times more probable than coincidence.” \4/

This formulation has been criticized as highly misleading. According to William Thompson, it is

likely to mislead lay people and foster misunderstandings that are detrimental to people accused of a crime. I recommend that Cybergenetics immediately cease using this misleading language and find a better way to explain its findings. Standards development organizations such as OSAC should consider developing standards that address the appropriateness, or inappropriateness, of such presentations. Courts should refuse to admit PG [probabilistic genotyping] evidence when it is mischaracterized in this manner. Lawyers involved in cases in which defendants were convicted based on this misleading language should consider the appropriateness of appellate remedies. \5/

The main concern is that juxtaposing “match” and “coincidence” will lead judges and jurors to think that the "match statistic" pertains to the probabilities of hypotheses (H₁ and H₂) about the source of the DNA rather than probabilities about the laboratory’s data. In simpler terms, the concern is that most people will understand "coincidence" and "coincidental match" as an assertion that the observed match is the result of coincidence; moreover, they will think that "match" is an assertion that the defendant is the matcher. If that happens, then the assertion that a match is 10,000 times more likely than coincidence would be (mis)understood as a statement that the odds against a coincidence having occurred are 10,000 to 1.

Instead, LR = 10,000 should be understood (according to Bayes' rule) as a statement about the change in the odds that defendant, as opposed to some unknown, unrelated person, is the matcher. For example, if defendant has a strong alibi—strong enough, in conjunction with other evidence, to establish that the prior odds of H₁ as opposed to H₂ are only 1 to 5,000—then this LR raises the odds to 10,000 x 1:5,000 = 2:1. Such final odds are far from overwhelming.

Cybergenetics does not seems disposed to abandon "match form" testimony. Dr. Thompson claims that for fingerprint comparisons, "'[m]atch' is shorthand for source identification, [s]o, it is predictable that many lay people will interpret the term 'match,' when used to describe DNA evidence, to mean that the person of interest has been identified either definitively or with a high degree of certainty as a contributor." Pointing to a dictionary, Cybergenetics angrily responds that this is just "Thompson’s private language." \6/ But a tradition in forensic science is to equate a "match" with an identification, as shown by the title of articles such as "Is a Match Really a Match? A Primer on the Procedures and Validity of Firearm and Toolmark Identification." \7/ In popular culture, the term may have a similar connotation. Perhaps Youtube trumps Merriam-Webster. \8/

As far as I know, no studies compare the comprehensibility of relative-probability-of-the-data testimony to match-form testimony. Therefore, the law and the practice has to be guided by intuition. My sense is that avoiding the transposition of the probabilities in a likelihood ratio requires special care if the match-versus-coincidence approach is used. The witness must explain not only that a "DNA match" is merely a degree of similarity between the electropherograms being compared, but also that "coincidence" or "coincidental match" is shorthand for the proposition that the "match" is a match to an unrelated person (or other specified source)—and that it is not a conclusion that a coincidence has occurred. The phrase "coincidental match" is too ambiguous to be left undefined.

In short, I am not sure that an absolute rule against match-form testimony is necessary, but I see no clear benefit to the phraseology. Relative-probability-of-the-data testimony seems to be a more straightforward description of a DNA likelihood ratio. However, it too needs explanation to reduce the risk of blindly transposing the conditional probabilities for the data into conditional probabilities for the hypotheses. Cases announcing that a likelihood ratio is a ratio of source-hypothesis probabilities are legion. \9/

Notes

1. Cf. Commonwealth v. McClellan, 178 A.3d 874 (Pa. Super. Ct. 2018) ("[I]t was determined that the DNA sample taken from the gun's grip was at least 384 times more probable if the sample originated from Appellant and two unknown, unrelated individuals than if it originated from a relative to Appellant and two unknown, unrelated individuals").
2. Mark Perlin, Explaining the Likelihood Ratio in DNA Mixture Interpretation, in Proceedings of Promega's Twenty First International Symposium on Human Identification at 7 (Dec. 29, 2010); cf. Mark W. Perlin, Joseph B. Kadane & Robin W. Cotton, Match Likelihood Ratio for Uncertain Genotypes, 8 Law, Probability & Risk 289 (2009), https://doi.org/10.1093.
3. United States v. Anderson, No. 4:21-CR-00204, 2023 WL 3510823, at *3 (M.D. Pa. Apr. 26, 2023). For additional instances of “match form” testimony or reporting, see Howell v. Schweitzer, No. 1:20-cv-2853, 2023 WL 1785530 (N.D. Ohio Jan. 11, 2023); Sanford v. Russell, No. 17-13062, 2021 WL 1186495 (E.D. Mich. Mar. 30, 2021); State v. Anthony, 266 So.3d 415 (La. Ct. App. 2019).
4. Mark W. Perlin et al., TrueAllele Casework on Virginia DNA Mixture Evidence: Computer and Manual Interpretation in 72 Reported Criminal Cases, 9 PLOS ONE e92837, at 8 (2014).
5. William C. Thompson, Uncertainty in Probabilistic Genotyping of Low Template DNA: A Case Study Comparing STRMix™ and TrueAllele™, 68 J. Forensic Sci. 1049, 1059 (2023), doi:10.1111/1556-4029.15225.
6. Mark W. Perlin et al., Reporting Exclusionary Results on Complex DNA Evidence, A Case Report Response to 'Uncertainty in Probabilistic Genotyping of Low Template DNA: A Case Study Comparing Strmix™ and Trueallele®' Software 31 (May 18, 2023), available at SSRN: https://ssrn.com/abstract=4449313 or http://dx.doi.org/10.2139/ssrn.4449313.
7. Stephen G. Bunch et al., Is a Match Really a Match? A Primer on the Procedures and Validity of Firearm and Toolmark Identification, 11 Forensic Science Communications, No. 3 (2009), https://archives.fbi.gov/archives/about-us/lab/forensic-science-communications/fsc/july2009/review/2009_07_review01.htm.
8. In addition, a dictionary definition of "match" (https://www.merriam-webster.com/dictionary/match) is "a pair suitably associated." Suitable association suggests that a hypothesis about the nature of the association is true.
9. E.g., State v. Pickett, 246 A.3d 279 (N.J. App. 2021) (The "likelihood ratio [is] a statistic measuring the probability that a given individual was a contributor to the sample against the probability that another, unrelated individual was the contributor.") (citing Justice Ming W. Chin et al., Forensic DNA Evidence § 5.5 (2020)).

No "Daubert Hearing" on Latent Fingerprint Matching in US v. Ware

Last month, in United States v. Ware, 69 F.4th 830 (11th Cir. 2023), the U.S. Court of Appeals for the Eleventh Circuit  "carefully review[ed]" the convictions of Dravion Sanchez Ware arising out of a month-long crime spree near Atlanta, in 2017. He was found to have participated "in robbing ... three spas, four massage parlors, a nail salon, and a restaurant." The opinion  recounts the nine brutal robberies in luxuriant detail. It also discusses Mr. Ware's argument that the district court erred "by not holding a formal Daubert hearing before admitting expert fingerprint evidence."

In a word, the Eleventh Circuit rejected the argument as "unpersuasive." No surprise there. More surprising is the opinion's incoherent discussion of the 2009 NRC report on forensic science and the 2016 PCAST follow-up report. \1/ On the one hand, we are told that "[t]he science could not possibly have been so unreliable as to be inadmissible." On the other hand, "[t]he District Court here could have held a Daubert hearing to assess the relatively new reports Ware presented." So which is it? If a type of evidence cannot possibly be excluded as scientifically invalid under Daubert, how can it be proper to hold a pretrial testimonial hearing on admissibility under Daubert? And, was the court of appeals correct in concluding that the two reports do not impeach, to the point of requiring a hearing, the traditional practice of admitting latent fingerprint comparisons?

During Ware's trial, an unnamed "crime lab scientist with the Georgia Bureau of Investigation Division of Forensic Sciences" "outlined the science behind fingerprints themselves, including their uniqueness" and explained the four-step process the lab follows ... : 'Analysis, Comparison, Evaluation, and Verification,' or ACEV.” The last step "involves another examiner completing the whole process a second time." The opinion does not indicate whether the verifying analyst is blinded to the knowledge of the main examiner's finding. Interestingly as well (think Confrontation Clause), the opinion implies that the testifying expert in Ware was not the main examiner. "[S]he was the verifying examiner," and "she testified that the lab concluded the latent print ... led to an identification conclusion matched to Ware's left middle finger." After that,

Defense counsel specifically asked about the PCAST report [and] vigorously cross-examined ... discussing the possibility of a latent fingerprint not being usable ... , the subjectiveness of every step ... , and the bias that may creep into the verification process ... . The expert and defense counsel discussed ... the potential for false positives and negatives. On cross, the defense also attacked the expert's claim that she did not know of the Georgia Bureau of Investigation ever misidentifying someone with a fingerprint comparison, and that she did not know the rate at which a verifier disagrees with the original assessment.

To preclude such testimony about his unique fingerprint on an item stolen in one of the robberies, Ware had moved before the trial for an order excluding fingerprint-comparison evidence. Of course, such a ruling would have been extraordinary, but the defense contended that the 2009 and the 2016 reports required nothing less \2/ and asked for a full-fledged pretrial hearing on the matter. In response, "[t]he District Court conditionally denied the motion ... unless Ware's counsel could produce before trial a case from this Court or a district court in this Circuit that favors excluding fingerprint expert evidence under Daubert." \3/

The court of appeals correctly observed that "[f]ingerprint comparison has long been accepted as a field worthy of expert opinions in this Circuit, as well as in almost every one of our sister circuits." The only problem is that all the opinions cited to show this solid wall of precedent predate the NRC or the PCAST reports. A more complete analysis has to establish that the scientists' reviews of friction-ridge pattern matching do not raise enough of a doubt to expect that a hearing would let the defense breach the wall. 

Along these lines, the court of appeals wrote that

The [District] Court considered the reports and arguments presented and found that fingerprint evidence was reliable enough as a general matter to be presented to the jury. Many of the critiques of fingerprint evidence found in the PCAST report go to the weight that ought to be given fingerprint analysis, not to the legitimacy of the practice as a whole. Appellant Br. at 25 (“The studies collectively demonstrate that many examiners can, under some circumstances, produce correct answers at some level of accuracy.” (emphasis in original)).

This quotation from the PCAST report is faint praise. Although the court of appeals was sure that "Ware's contrary authority even says that fingerprint evidence can be reliable," the depth of its knowledge about the PCAST (and the earlier NRC committee) reports is open to question. The circuit court had trouble keeping track of the names of the groups. It transformed the National Research Council (the operating arm of the National Academies of Science, Engineering, and Medicine) into a "United States National Resource Council" (69 F.4th at 840) and then imagined an "NCAST report[]" (id. at 848). \4/ 

Deeper inspection of "Ware's contrary authority" is in order. The 2009 NRC committee report quoted with approval the searing conclusion of Haber & Haber that “[w]e have reviewed available scientific evidence of the validity of the ACE-V method and found none.” It reiterated the Habers' extreme claim that because "the standards upon which the method’s conclusions rest have not been specified quantitatively ... the validity of the ACE-V method cannot be tested." To be sure, the committee agreed that fingerprint examiners had something going for them. It wrote that "more research is needed regarding the discriminating value of the various ridge formations [to] provide examiners with a solid basis for the intuitive knowledge they have gained through experience." But does "intuitive knowledge" qualify as "scientific knowledge" under Daubert? Is a suggestion that friction-ridge comparisons need a more solid basis equal to a statement that the comparisons are "reliable" within the meaning of that opinion? The response to "NCAST" was underwhelming.

But research has progressed since  2009. The second "contrary authority," the PCAST report, reviewed this research. At first glance, this report supports the court's conclusion that no hearing was necessary. It assures courts that "latent fingerprint analysis is a foundationally valid subjective methodology." In doing so, it rejects the NRC committee's notion that the absence of quantitative match rules precludes testing whether examiners can reach valid conclusions. It discusses two so-called black-box studies of the work of examiners operating in the "intuitive" mode. Yet, the Ware court does not cite or quote the boxed and highlighted finding (Number 5).

Perhaps the omission reflects the fact that the PCAST finding is so guarded. PCAST added that "additional black-box studies are needed to clarify the reliability of the method," undercutting the initial assurance, which was "[b]ased largely on two ... studies." Furthermore, according to PCAST, to be "scientifically valid," latent-print identifications must be accompanied by admissions that "false positive rates" could be very high (greater than 1 in 18). \5/

The Ware court transforms all of this into a blanket and bland assertion that the report establishes reliability even though it "may cast doubt on the error rate of fingerprint analysis and comparison." The latter concern, it says, goes not to admissibility, but only to "weight" or "credibility." 

Can it really be this simple? Are not "error rates" an explicit factor affecting admissibility (as well as weight) under Daubert? Certainly, the Eleventh Cicuit's view that the problems with fingerprint comparisons articulated in the two scientific reports are not profound enough to force a wave of pretrial hearings is defensible, but the court's explanation of its position in Ware is sketchy.

At bottom, the problem with the fingerprint evidence introduced against Ware (as best as one can tell from the opinion) is not that it is speculative or valueless. The difficulty is that the judgments are presented as if they were scientific truths. The Ware court is satisfied because "Defense counsel put the Government's expert through his paces during cross-examination, and counsel specifically asked the expert about the findings in the PCAST report." But would it be better to moderate the presentations to avoid overclaiming in the first place? 

The impending amendment to Rule 702 of the Federal Rules of Evidence is supposed to encourage this kind of "gatekeeping." Defense counsel might be more successful in constraining overreaching experts than in excluding them altogether. That too should be part of the "considerable leeway" granted to district courts seeking to reconcile expert testimony.with modern scientific knowledge.

Notes

1. President's Council of Advisors on Sci. & Tech., Exec. Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods (2016),  [https://perma.cc/R76Y-7VU]
2. In Ware

   The pretrial motion to exclude the fingerprint identification "relied on the 2009 United States National Resource Counsel (“NRC”) report and subsequent 2016 President's Counsel of Advisors on Science and Technology (“PCAST”) report, which supposedly revealed a dearth of "proper scientific studies of fingerprint comparison evidence" and claimed that "there is no scientific basis for concluding a fingerprint was left by a specific person," positing that "because fingerprint analysis involves individual human judgement, the resulting [fingerprint comparison] conclusion can be influenced by cognitive bias."

3. Why insist on a pre-existing determination in one particular geographic region that scientific validity is lacking in order to grant a hearing on whether scientific validity is present? Is the "science" underlying fingerprint comparisons different in Georgia and the other southeastern states comprising the 11th Circuit different from that in the rest of the country?
4. OK, these peccadillos are not substantive, but one would have thought that three circuit court judges, after "carefully reviewing the record," could have gotten the names and acronyms straight. Senior Judge Gerald Tjoflat wrote the panel opinion. At one point, he was a serious contender for the Supreme Court seat filled by Justice Anthony Kennedy. After Judge Tjoflat announced that he would retire to senior status on the bench in 2019, President Donald Trump nomined Robert J. Luck to the court. In addition to Judge Luck, Judge Kevin C. Newsom, a 2017 appointee of President Trump was on the panel. Judicial politics being what it is, over 30 senators voted against the confirmation of Judges Newsom and Luck.
5. PCAST suggested that if a court agreed that what it called "foundational validity" were present, then to achieve "validity as applied" some very specific statements about "error rates" would be required:

   Overall, it would be appropriate to inform jurors that (1) only two properly designed studies of the accuracy of latent fingerprint analysis have been conducted and (2) these studies found false positive rates that could be as high as 1 in 306 in one study and 1 in 18 in the other study. This would appropriately inform jurors that errors occur at detectable frequencies, allowing them to weigh the probative value of the evidence.

   The studies actually found conditional false-positive proportions of 6/3628 (0.17%) and 42/995 (4.2%, or 7/960 = 1.4% if one discards "clerical errors.") (P. 98, tbl. 1). Earlier postings discuss these FBI-Noblis and Miami Dade police department numbers.