Evidence-centric and Conclusion-centric: What's the Difference?

In recent months, various forensic practitioners have asked for a short and simple explanation of the difference between the "evidence-centric" and "conclusion-centric" forms of reporting and testifying. I figure that's what I get for introducing the phrases in a panel asked to talk about “Bringing Statistics into the Courtroom” at the Conference on Forensics, Statistics, and Law, at the University of Virginia in March 2018.^\1/ The particular phrases may have been original (at least I think they were), but the underlying ideas, which are what matter, have been around for a long time.

For concreteness, I will focus on a specific example--something that is the target of criticism du jour in US courts. I am thinking of firearms-toolmark examinations culminating in conclusions about the source of a bullet or shell casing. However, the dichotomy between statements of the evidence and its probative value, on the one hand, and statements of the inferences or conclusions that follow from the evidentiatary statements, on the other hand, is far more general. It applies to techniques for shedding light on the possible source of fingerprints, treadmarks, handwriting, images, glass fragments, paint chips, fibers, bones, bitemarks, or any other traces acquired at or around crime scenes or victims. Indeed, the distinction applies to all manner of empirical inquiry, as suggested by the subtitle of the late Frederick Schauer's book, The Proof: Uses of Evidence in Law, Politics, and Everything Else.^\2/ Moreover, nothing in the terminology reveals whether the evidence in question comes from a scientifically validated process, from (typically) less reliable modes of acquiring and assessing data, or from superstitions masquerading as knowledge.

A Recent Case

With these broad obervations in mind, let's turn to a recent appellate opinion comparing the state's arguments for introducing the conclusions of a firearms examiner to arguments that could be marshalled for believing in palmistry. In People v. Tidd, No. A167548, 2024 WL 3982134 (Cal. Ct. App., Aug. 29, 2024), someone in a white sport utility vehicle shot an enebriated pedestrian in the leg. Investigators located a nine-millimeter Luger cartridge case, manufactured by Speer, in the vicinity. Six days later, police arrrested Raymond Tidd as he was about to enter a white SUV carrying a loaded nine-millimeter Sig Sauer pistol. He pled guilty to some firearms possession and carrying charges but went to trial for attempted murder and other felonies for the shooting. Before trial, he objected to testimony from a criminalist with the San Francisco Police Department named Jacobus Swanepoel. The trial court held a hearing and ruled that the expert could testify that the cartridge case came from the defendant's gun as long as he refrained from asserting his opinion “to a scientific certainty.”

At trial, Swanepoel "flatly asserted the recovered cartridge and test cartridges fired from Tidd's gun 'were fired in the same firearm.'" The jury found Tibbs guilty of two firearms felonies for the shooting (but not of attempted murder). However, the court of appeals reversed because the prosecution never showed that the highly subjective judgmental procedure the expert described was "reasonably reliable."

The court's analysis of what it would take to achieve such reliability is open to debate, but that is a subject for another day. My target today is the statements that were made or could have been made about the cartridge cases and the gun or guns that fired them. What statements are "conclusion-centric"? Which are "evidence-centric"?

A Preliminary Point

"Evidence" exists in the context of some hypothesis about a matter of fact (a state of nature). Nothing can sensibly be deemed "evidence" without asking, "evidence of what?" That the sky is dark is evidence that it is nighttime. It is not conclusive evidence (nothing is conclusive evidence of an empirical fact) because darkness in the daytime can come from a total solar eclipse (or even less likely events). But darkness is pretty good evidence (E) for the hypothesis (H) that night has arrived. The hypotheses of most interest in cases like Tidd are source hypotheses like Swanepoel's assertion that the cartridges were "fired in the same gun" or "the cartridges were fired in different guns." These assertions are conclusions that the analyst reaches (or does not reach) on the basis of the evidence about the marks on the objects examined.

Conclusion-centric Statements

With respect to the source hypothesis, a scan of the opinion in Tibbs reveals the following conclusion-centered statements from the firearms examiner:

• The cartridges "were fired in the same gun." In other words, the same-gun hypothesis H is true (given the markings seen in the examination). It may not be "scientifically certain," but it is true nonetheless. In symbols, P(H|E) is practically (or exactly?) 1, where the expression "P(H|E)" stands for "the probability of the hypothesis H given the evidence E." Obviously, this is conclusion-centric, as it tells the jury what they should believe about H (or at least what the examiner believes about this conclusion).
• "[I]t was 'more likely than not that this [Sig Sauer pistol] fired the cartridge casing' he was analyzing." This sounds like a statement that, given the marks (E), H is probably true: P(H|E) > 1/2. This statement is also conclusion-centered. It is a statement about the (probable) truth of H. However, in this case,
• "He explained that by agreeing to 'more likely' he was 'saying it is this firearm,' and by 'not' he was expressing that the chance of finding another firearm 'with the same signature or the same fingerprint is remote or really small.'" Without getting hung up on the dissonance in this compound sentence, this is a statement that given the markings (E), the probability of not-H is "really small." Mathematically, P(not-H|E) implies that H is not merely probable--it is really probable: P(H|E) is almost 1. Another conclusion-centric statement.
• "[T]he Sig Sauer pistol could not be excluded from the class of firearms that could have fired the cartridge case submitted for analysis." Also conclusion-centered. "Could not be excluded" means "is included as" (or "is consistent with") a possible source, it asserts that P(H|E) > 0. That focuses on the (possible) truth of the conclusion. But classifying this phrasing as conclusion-centric is a close case. If "could not be excluded" were a more neutral statement of the pair of observations themselves (such as "has the same characteristics" or "matches") then it is a statement about the evidence E rather than what E proves.

Evidence-centric Statements

• "Investigators also located a nine-millimeter Luger cartridge case ... on the street near where the shooting took place." With respect to the source conclusion, the statement of the caliber is evidence-centric in the sense that it specifies a feature of ammunition that might have used in the shooting without drawing a conclusion (be it categorical or probabilistic) about the gun taken from Tidd as having had that ammunition fired in it. That Tidd was found with a gun of the same caliber makes the characteristic incriminating evidence--it raises the probability of H. In symbols, P(H|caliber match) > P(H), where P(H) is the probability of H prior to considering the caliber data.
• “[I]t would be rare to find another firearm that exhibits that same ... pattern ... .” Evidence-centric. We have an assertion that the frequency of the incriminating marks is unusual in the population of guns that might have been used in the shooting. Finding a rare pattern supports H because it would be surprising to see the same (or more persuasive) evidence E if a different gun had been used. This is an assertion about the value of the evidence as proof of H rather than assertion about the truth of H itself.^\3/

So far we have encountered (1) statements of conclusions (or their probabilities) about a source hypothesis and (2) statements of the evidence ("it's a 9-mm caliber bullet," "the bullets have matching patterns," and so on) with regard to the source hypothesis. But there is a further category of evidence-centric statements that are more interesting. These evidence-centric statements go beyond descriptions of the data and comparisons using the data, but they do not go so far as to express conclusions. Rather, they are statements of the degree to which the evidence supports one hypothesis relative to alternative hypotheses.

I cannot find these in Tidd, for the examiner there was following the conclusion-centric paradigm espoused by the Association of Firearm and Tool Mark Examiners (AFTE). But they not hard to envision. Their defining feature is that they concern the probability of the evidence under the different hypotheses rather than the conclusion-centric probability of the hypothesis given the evidence. For example:

• The patterns (considering all the similarities and differences) are much more probable for bullets fired from defendant's gun than for bullets fired from different guns.
• The patterns are much more compatible with the hypothesis that bullets were fired from the defendant's gun than the hypothesis that they were fired from different guns.
• The patterns provide strong support for the hypothesis that bullets were fired from the defendant's gun rather than the hypothesis that they were fired from different guns.

These statements all flow from assessing how probable the patterns are for pairs of hypotheses. They look to P(E|H) rather than to the examiner's judgments of P(H|E). The underlying principle is that patterns that are more probable under one hypothesis than another are stronger evidence for the former hypothesis and thus support it instead of supporting the competing hypothesis. The hope is that experts will better assist the judge or jury by telling them about the strength of the evidence in this way instead of telling them which hypothesis to believe (to some degree of certainty).

Notes

1. David H. Kaye, The Nikumaroro Bones: How Can Forensic Scientists Assist Factfinders?, 6 Va. J. Crim. L. 101 (2018), https://www.ssrn.com/abstract=3177752. See also David H. Kaye, Forensic Statistics in the Courtroom, in Handbook of Forensic Statistics 225 (David Banks et al. eds., 2021), https://www.ssrn.com/abstract=3561914.
2. Frederick Schauer, The Proof: Uses of Evidence in Law, Politics, and Everything Else (2022).
3. Thus, "really small" is a qualitative expression of p-value. In statistics, a p-value is the the probability of such extreme evidence arising when the "alternative hypothesis" (such as "different gun") is true. Everything else being equal, the smaller the p-value, the harder it is to believe the alternative hypothesis is true.

Who Published Bayes' Theorem?

For the legal profession, "American Law Reports or ALR is a longstanding, highly trusted series of in-depth articles, called annotations, on specific legal issues."^\1/ But the report on "Use of Bayes' Theorem in Criminal and Civil Cases"^\2/ may be less worthy of trust than the average annotation. Historians and statisticians, at least, would be surprised at its opening sentences:

Bayes' Theorem, named for the English clergyman and scientist who published it in 1763, is a scientific principle of the likelihood ratio used to calculate conditional probabilities leading to a variety of statistical "Bayesian" methodologies. It speculates on the probability that a particular fact is true or that a particular event will occur, given our knowledge of one or more related facts or events.

There is no doubt (asymptotically speaking) that in 1763, Thomas Bayes did not publish the theorem that bears his name. He was dead. A close friend, the Reverend Richard Price, read the paper to the Royal Society of London after retrieving it from Bayes's papers and adding to it.^\3/ This expanded and modified version appeared in the Society's Philosophical Transactions.

The ALR's rendering of 18th Century history won't cause any problems for lawyers employing or confronting Bayesian analyses in court. Posthumous or otherwise, discovered first by Bayes or by someone else,^\4/ the theorem is what it is. But what will lawyers make of the idea that the theorem is "a scientific principle of the likelihood ratio" that "speculates" about conditional probability? Is the theorem mere speculation? Is it only about a likelihood ratio? No, and no.

The usual understanding is that a likelihood ratio is the data-based, objective part of the theorem and that speculation becomes an issue if and when the prior probability is not objective as well. As Judea Pearl and Dana MacKenzie explain,

Bayesian statistics gives us an objective way of combining the observed evidence with our prior knowledge (or subjective belief) to obtain a revised belief ... . Still, what frequentists could not abide was that Bayesians were allowing opinion, in the form of subjective probabilities, to intrude into the pristine kingdom of statistics. Mainstream statisticians were won over only grudgingly, when Bayesian analysis proved a superior tool for a variety of applications ... .^\5/

Fitting expert prior probabilities and likelihoods into the legal process poses some special problems for mainstream statisticians. That accommodation is the subject of continuing, if often repetitive, multidisciplinary dialog that does not fit neatly and quickly into a blog.

Notes

1. University of South Carolina School of Law, Legal Research, Analysis & Writing, https://guides.law.sc.edu/LRAWSpring/LRAW/alr.
2. Deborah F. Buckman, Annot., Use of Bayes' Theorem in Criminal and Civil Cases, 47 A.L.R.7th Art. 4 (2019).
3. Stephen M. Stigler, Richard Price, The First Bayesian, 33 Statistical Science 117 (2018), https://doi.org/10.1214/17-STS635; Martyn Hooper, Richard Price, Bayes’ Theorem, and God, Significance 36-39 (Feb. 2013), https://www.york.ac.uk/depts/maths/histstat/price.pdf.
4. Stephen M. Stigler, Who Discovered Bayes’s Theorem?, 37 Am. Stat. 290-296 (1983).
5. Judea Pearl & Dana MacKenzie, The Book of Why: The New Science of Cause and Effect 90 (2018).

NIST's Forensic Science Environmental Scan 2023

Yesterday, I received an email that included the following notice:

NIST released a new report today from an intensive 18-month, multi-pronged effort by the NIST Forensic Science team to assess the strategic opportunities for forensic science research and standards that are needed to drive significant advancements in the practice of forensic science over the coming decade.

The report is based on an extensive literature review, input from external stakeholders, and feedback from NIST subject matter experts. Throughout 2023, NIST conducted an assessment of the forensic science environment to inform strategic planning efforts. The resulting Forensic Science Environmental Scan 2023 report captured salient issues and trends across five different landscapes: governance, economic, societal, scientific and technological, and legal and regulatory. In addition, NIST held a roundtable in September 2023 with forensic science thought leaders from across forensic disciplines to discuss the long-term vision and strategic priorities for forensic science in the United States. NIST used these inputs to identify the grand challenges facing forensic science today, the strategies for addressing them through advances in research and standards, and the subsequent implementation of the advances into forensic science practice.

I can't say I have read the report yet, but I am posting the notice for anyone who wants to scan the "environmental scan."

Smith v. Arizona: The Facts and the Outcome

In A.E. Van Vogt’s murky but classic science fiction novel, The World of Null-A, the main character repeatedly dies, only to reappear in a new body with no memory of his former life. With help from extra brain matter, he moves on anyway. That, very roughly, is what the state of Arizona tried to accomplish in Smith v. Arizona, 144 S.Ct. 1785 (2024). It wrote off one expert witness but allowed her to live on through another expert. This plot twist left the defendant complaining that his Sixth Amendment “right … to be confronted with the witnesses against him” had been violated. This right, the Supreme Court repeatedly has held, prevents the prosecution from introducing “testimonial hearsay” without producing the author of the hearsay for cross-examination.

The Court’s struggle to define “testimonial” is becoming epic, and the definition of hearsay is one of those terrors of law school that leaves many students wondering whether they should have gone to some other professional or graduate school. One essential feature of hearsay is that the assertion is offered to prove “the truth of the matter asserted.” This phrase was at the center of the dispute in Smith, and the Court usefully clears up some of the confusion created by the opinions in Williams v. Illinois, 567 U.S. 50 (2012). Unfortunately, the Court did not stop there but also expressed some hasty thoughts about what might make a forensic-science expert’s hearsay statements nontestimonial.

This post describes the facts of Smith. They are drawn primarily from Justice Kagan's opinion for the Court. Ruminations on both parts of the majority opinion—and the three concurring opinions—may come later.

Jason Smith was arrested in a shed in Yuma county that contained “a large quantity of what appeared to be drugs and drug-related items.” He pleaded not guilty to charges of possession. The State asked “a crime lab run by the Arizona Department of Public Safety (DPS) for a ‘full scientific analysis.’” The State informed the lab of who the defendant was, what he was charged with, and the fact that a trial was pending. An analyst named Elizabeth Rast “ran the requested tests.” She typed out notes and submitted a signed report. The notes described each item, its weight, the tests performed, and a conclusion about its identity. The report of “results/interpretations” stated that four items “[c]ontained a usable quantity of methamphetamine,” three “[c]ontained a usable quantity of marijuana,” and one “[c]ontained a usable quantity of cannabis.”

After Rast “stopped working at the lab, for unexplained reasons,” prosecutors replaced her name on the “final pre-trial conference statement” with thast of a current employee. They promised that “Greggory Longoni, [a] forensic scientist (substitute expert),” who had no previous connection to the case, would “provide an independent opinion on the drug testing performed by Elizabeth Rast.” At trial, Longoni purported to give an independent opinion on the nature of the eight items. Yet, he arrived at his opinion by consulting nothing more than Rast's report and notes. He did no re-examination or retesting of his own. He merely

referred to those materials and related what was in them, item by item by item. As to each, he described the specific ‘scientific method[s]’ Rast had used … (e.g., a microscopic examination, a chemical color test, a gas chromatograph/mass spectrometer test). … [H]e stated that the testing had adhered to ‘general principles of chemistry,’ as well as to the lab's ‘policies and practices …. [H]e noted, for example, that Rast had run a “blank” to confirm that testing equipment was not contaminated. \1/

Readers of the opinion might think that "those materials" to which Longoni referred "[w]hen [he] took the stand" included Rast's report, but this is not quite correct. The prosecution was careful not to ask Longoni to recite the conclusions in the report. For example, the prosecutor posed these questions:

Q Let me be clear. You’re not testifying as to her report, you’re testifying as to review of lab notes?
A Correct.
Q In reviewing what was done, your knowledge and training as a forensic scientist, your knowledge and experience with DPS’s policies, practices, procedures, your knowledge of chemistry, the lab notes, the intake records, the chemicals used, the tests done, can you form an independent opinion on the identity of Item 26?
A Yes.

In this manner, Longoni told the jury that his “independent opinion” was that “Item 26 was 'a usable quantity of marijuana,' … Items 20A and 20B were 'usable quantit[ies] of methamphetamine,' and … Item 28 was '[a] usable quantity of cannabis.'”

The jury convicted, and Smith appealed on the ground that he was convicted via Rast’s written statements with no opportunity to cross-examine her. The State insisted that “Longoni testified about ‘his own independent opinions,’ even though making use of Rast's records.” The Arizona Court of Appeals agreed that Longoni was “present[ing] his independent expert opinions” as “based on his review of Rast's work.” It affirmed, relying on a 2014 case in which it had stated that an expert may testify to “the substance of a non-testifying expert's analysis, if such evidence forms the basis of the [testifying] expert's opinion.” State ex rel. Montgomery v. Karp, 236 Ariz. 120 (Ct. App. 2014). According to the Court of Appeals in Karp, the “underlying facts” are then “used only to show the basis of [the in-court witness's] opinion and not to prove their truth.” All this seemed so obvious to the court that it did not think its opinion was even worth publishing. The Arizona Supreme Court declined review without comment.

The U.S. Supreme Court was much more interested. In Williams v. Illinois, 567 U.S. 50 (2012), five Justices—a majority of the Court—had rejected this very reasoning. These Justices had dismissed the basis-only rationale as “legal fiction” (Thomas, J., concurring), “very weak,” “factually implausible,” “nonsense,” and “sheer fiction.” (Kagan, Scalia, Ginsburg & Sotomayor, JJ., dissenting and quoting D. Kaye, D. Bernstein, & J. Mnookin, The New Wigmore: Expert Evidence §4.10.1, pp. 196-197 (2d ed. 2011); id., §4.11.6, at 24 (Supp. 2012)). However, the remaining four Justices, in a plurality opinion written by Justice Alito, had vigorously advanced the not-for-its-truth theory for basis evidence.

The absence of a unifying rationale from a majority of the Court in Williams enabled the Arizona court to rely on one part of the Williams plurality opinion in one breath and to say that the “plurality decision … has limited if any precedential value” in the next. Meanwhile, other states had found the only-to-show-the-basis argument entirely unpersuasive.

And, so, for the fourth time, the Supreme Court granted a writ of certiorari to review either the introduction of forensic-science test findings made by analysts who were not presented for cross-examination or references to another analyst's findings by the expert presenting the scientific evidence. The petition propounded the question:

Whether the Confrontation Clause of the Sixth Amendment permits the prosecution in a criminal trial to present testimony by a substitute expert conveying the testimonial statements of a nontestifying forensic analyst, on the grounds … that the testifying expert offers some independent opinion and the analyst's statements are offered not for their truth but to explain the expert's opinion … .

On this narrow issue, the Court was unanimous: At least when the surrogate expert endorses the basis for his opinion as reliable, introducing the missing witness’s out-of-court but “testimonial” statements of and about her findings violates the Confrontation Clause. Justice Kagan wrote an opinion, joined in its discussion of this issue by every other Justice (except for Justice Alito and the Chief Justice). Justices Thomas and Gorsuch also filed concurring opinions that no one else joined. Justice Alito, joined only by the Chief Justice, filed an concurring opinion accusing the rest of the Court of “inflict[ing] a needless, unwarranted, and crippling wound on modern evidence law” and of “blow[ing] up the Federal Rules” with “a radical change” to the Rule 703 framework for admitting expert testimony. This is an obvious exaggeration, but explaining why merits a later post to unpack the logical foundation of Rule 703.

Beyond the hearsay issue, based on little more than sketchy suggestions from the Deputy Solicitor General at oral argument, Justice Kagan proposed that laboratory documentation could be introduced with no opportunity for confrontation if its “primary purpose” was for accreditation, quality control, or unofficial “notes to self.” Two of the seven Justices signing the majority opinion defected from this part of it. Justice Gorsuch expressed broader skepticism of the primary-purpose tests the Court had introduced in previous Confrontation Clause cases. Already, crime laboratories are talking about how to restructure or rewrite their documentation to fit into these new, nontestimonial categories. \2/ As I see it (so far), the nature of forensic science makes these alleged purposes too intertwined with the knowing production of evidence for a “primary purpose” test to disentangle them, but that conclusion also begs for later explanation.

In any event, the Supreme Court unanimously vacated the state court of appeals order affirming Smith's conviction and remanding for the Arizona Court of Appeals “[t]o address the additional issue of whether Rast's records were testimonial (including whether that issue was forfeited).” The state then conceded that it had forfeited the opportunity to argue that parts of the testimony were not “testimonial” hearsay under Crawford because it had not raised the argument in its appeal. The Arizona Court of Appeals remanded to the Yuma County Superior Court to vacate the judgment of conviction.

But this did not guarantee Jason Smith his freedom. The state could have tried him a second time. Perhaps Rast could be located to testify. Or, the lab could test the drugs again. (At trial, Longoni had testified that retesting would have taken only two to three hours.) After that, an analyst who did such testing could have testified to truly independent findings. On August 20, however, Jason Smith entered into a plea bargain, sparing the state the burden of retrying him. What concession he received in return, I do not know.

Note

1. 144 S.Ct. at 1799. But see id. at 1802 (declining to resolve a dispute between the parties as to whether Longoni's references to the report indicated that he partially relied on it as opposed to only the notes).
   
2. Consortium of Forensic Science Organizations, Untitled Memorandum, Aug. 20, 2024, https://thecfso.org/wp-content/uploads/2024/08/Smith-v-Arizona-Final.pdf

A Draft Standard on "Terminology for a Suspected Pattern of Dental Origin"

The Academy Standards Board (ASB) of the American Academy of Forensic Sciences is seeking comments by September 9 on a Technical Report 194, First Edition, 2024, titled "Terminology for a Suspected Pattern of Dental Origin." Although only a "template" for organizing comments is listed on the ASB website, the draft standard can be found at https://www.aafs.org/sites/default/files/media/documents/194_TR_Ballot01.pdf.

The OSAC page on “Standards Open for Comment” (which you won't see by perusing the OSAC website unless you click on “How To Work With Us” in the navigation pane) says “NOTE: This is OSAC 2021-N-0030, Terminology for a Suspected Pattern of Dental Origin, currently on the OSAC Registry.” Does that mean the ASB committee believed that nothing in the OSAC product, which emerged with no review from an advisory scientific and technical panel of experts, needed improvement? It is good to go public as an SDO-approved standard for terminology unless someone objects and proposes something better? 

Alas, it is not that good. Although a discussion of the choice of various terms to define and the definitions themselves could occupy pages, it is too tedious an undertaking for me to write or for many readers to plough through. Suffice it to say that some of the standard has a stream-of-consciousness feeling to it. Like "spurious observation anomaly not intrinsically present feature not related to the source."

The larger question is why try to promulgate a free-floating standard terminology rather than articulate standard procedures with appropriate terms? What might these standard practices be? Presumably, the proposed terminology is a precursor to performing "suspected pattern of dental origin analysis," which is defined as "forensic examination, analysis, and determination of the pattern for potential links to dental origins." This "potential links" study seems to be subdivided into (1) "bitemark assessment analysis," (2) "bitemark analysis," (3) "bitemark comparison analysis," and (4) "bitemark individualization analysis." Can any of these analyses produce results of "evidentiary value" (defined as "information of sufficient usefulness to serve as the basis for making an empirically significant scientific determination")?

The dentists are not prepared to say so. But neither are they willing to list in their bibliography any of the well-known articles and reports concluding that demonstrations of the scientific validity of these analyses are little more than wishful thinking. Rather than claim that these terms refer to procedures that have "evidentiary value" or opine that they lack such value, they merely note that their definitions are "not an endorsement of [the] scientific validity" of the processes they are supposed to describe. Apparently, the drafters from OSAC are agnostics rather than atheists. Or maybe they have their doubts about "bitemark individualization analysis." That phrase is marked "deprecated." Yet, the process of "visual comparison" is not deprecated as unvalidated or invalid, and a note suggests that other "individualization method[s]" for bitemarks could be just dandy.

So the best comment might be a recommendation to jettison this standard. Researchers can use their own clearly defined terms in devising and validating procedures that can be used in criminal investigations that involve what might be toothmarks, bitemarks, or wounds from other sources and mechanisms. When demonstrably valid procedures become available, the time will be ripe for a standard with uniform terminology.

Meanwhile, promulgating these terms and definitions, even with the agnostic disclaimers, risks encouraging the acceptance of dubious forensic "science." The very existence of an expert standard with these terms and definitions might suggest that the words describe something meaningful and encourage testimony that the field has a standardized system of some kind. I can imagine testimony that

The terms I am using in the analysis of what I have determined to be bitemarks are generally accepted in forensic odontology, medicine, and forensic science. I am following the ASB technical report on the subject. The report was produced with funding from the National Institute of Standards and Technology and is included and recommended for adoption by the government-supported Organization of Scientific Area Committees for Forensic Science.

Is this what forensic science and the law needs?

“Predictive or Profiling Evidence” and Diaz v. United States

Today the Association of American Law School’s (AALS) Section on Evidence distributed the following announcement to its members:

In a divided decision, the Supreme Court recently concluded that expert testimony about the likely mental state of individuals arrested with drugs in their position is admissible in criminal trials under the Federal Rules of Evidence in Diaz v. United States, 602 U. S. ____ (2024). This decision was sharply contested in its own right, but also drew attention to an area of broader controversy in the law of evidence: the increasing use of "predictive" or "profiling" evidence, by which expert witnesses present testimony suggesting that an individual is more or less likely to have had a particular mental state or behaved in a particular way based on their personal circumstances or characteristics. Scholars who have written about this phenomenon have expressed disquiet about the use of such evidence (and some courts have limited the use of such evidence in edge cases involving particular prejudice or overbroad characterizations), but no consensus has emerged as to the reasons for objecting to predictive evidence or as to how to systematically distinguish between such evidence and other forms of indirect and circumstantial evidence that is routinely admitted. This panel brings together scholars to discuss predictive and profiling evidence from a variety of perspectives. Was the Supreme Court right to find such evidence consistent with the federal rules governing the admissibility of expert evidence? Is such evidence generally consistent with due process and equal protection concerns? Is it generally empirically sound? Do we need new federal rules or common law doctrines to limit the admissibility of some forms of predictive or profiling evidence?

Who can object to a scholarly program on the subject, even if it is hardly new, having been the subject of a multitude of opinions, a number of statutes, and even a previous AALS program decades ago? Hopefully, the courts and the professoriate have made some progress in understanding what has come to be called "framework evidence." Neither would it be fair to criticize a necessarily brief announcement for not defining "predictive or profiling evidence." Presumably, the evidence teachers do not need a definition.

But I was surprised to read my fellow law professors’ sweeping characterization of Diaz v. United States, 144 S.Ct. 1727 (2024), as establishing that "such evidence [is] consistent with the federal rules governing the admissibility of expert evidence" and "that expert testimony about the likely mental state of individuals arrested with drugs in their position [sic?] is admissible in criminal trials under the Federal Rules of Evidence." Although I doubt that the announcement will cause its narrow range of readers to believe that the case stands for more than it does, I do worry that the same kind of language will crop up in undiscerning judicial opinions and commentary on Diaz. Therefore, it may be be worth listing some problematic aspects of the statements.

First, the case plainly held that certain "expert testimony about the likely mental state of individuals arrested with drugs in their position [possession?] is" not "admissible in criminal trials under the Federal Rules of Evidence." Every Justice agreed that no expert can testify that a defendant charged with importing proscribed drugs knew that they were transporting drugs. That would be explicit ultimate-opinion testimony on a criminal defendant's state of mind. (There are cases limiting the Rule 704(b) ban to mental health professionals, but the Court did not consider that possible way to interpret the rule. It stuck with a more literal reading of the text.)

Second, no “predictive or profiling evidence” was introduced in the case, and only one Justice thought it worth discussing. Certainly, Diaz is not a case of a criminal profiler predicting (“inferring” would be more precise) the characteristics of a criminal from the type or manner of the crimes under investigation (or any other such “predictive or profiling evidence”). As the excerpts from the trial transcript reproduced below show, the witness did not claim such expertise; furthermore, the trial judge barred him from stating a belief about the defendant’s knowledge (although one could well think that his testimony made it plain enough what his belief was). 

Third, the issue before the Court was not admissibility under the rules of evidence writ large. It was the scope of a single part of a solitary rule. Federal Rule 704(b), which has no counterpart in the rules of most states, declares that no expert witness may "state an opinion about whether the defendant did or did not have a mental state or condition that constitutes an element of the crime charged or of a defense" because "[t]hose matters are for the trier of fact alone." All that the Diaz Court held was that when such an opinion is not stated and is an inference that does not necessarily follow (as a matter of deductive logic) from the witness’s statements, then Rule 704(b) does not preclude its admission. It did not—and could not—have held that the Rule makes it admissible. Cf. David H. Kaye, The Ultimate Opinion Rule and Forensic Science Identification, 60 Jurimetrics J. 75 (2020).

Thus, Diaz should not be read as supporting—or opposing—the use of “predictive or profiling evidence” generally, or even in the subcategory of testimony offered to prove a defendant’s state-of-mind. 

More thoughts on the three opinions in the case and how the result fits into the range of possible interpretations of Rule 704(b) will appear in the upcoming supplement to The New Wigmore on Evidence: Expert Evidence § 2.2.3(b) (available online in VitalLaw) and, time permitting, in a further positing about the case here.

---

Excerpts from Trial Transcript
Mar. 18, 2021

The most pertinent portion of the testimony at issue in Diaz is as follows (with italics added):

BY MR. OLAH [Assistant US Attorney]:
Q. Where do you work?
A. I’m a special agent with Homeland Security Investigations.
Q. And how long have you been with HSI?
A. I’ve been a special agent since 1996. So going on 20 — I believe 28 years.
Q. Were you in law enforcement before joining HSI?
A. Prior to becoming a special agent, I was a U.S. Border Patrol agent. And prior to that, I was a sheriff’s corrections deputy.
***
Q. Have you been involved in drug trafficking investigations as a special agent with HSI?
A. Yes, I have.
Q. Approximately how many such investigations?
A. I’ve been involved in over 500 investigations dealing with distribution of drugs and also the – which would include the importation of drugs.
Q. And can you summarize for the jury the various investigation techniques you’ve used?
A. The techniques I’ve used, I’ve utilized wiretaps, where you actually listen to a drug trafficker talk on the telephone and how they conduct business. I’ve done controlled purchases where I utilized an undercover agent or a cooperating source. And we actually go out on the street and buy the drugs. I’ve spoken with cooperating defendants that have been arrested for drug trafficking related offenses. I’ve talked to cooperating sources that have information related to the distribution of drugs and drug trafficking organizations. I have spoken with other agents that work drug trafficking organizations and have worked on task forces with other agencies such as the Federal Bureau of Investigation, the Drug Enforcement Administration, and local police departments dealing with drug trafficking related crimes.
***
Q. Agent Flood, why are drugs imported into the United States?
MS. IREDALE: Objection, 401.
THE COURT: Overruled.
THE WITNESS: Based upon drugs that – some drugs are manufactured in Mexico and outside the United States. Therefore, they’re brought across the border, into the United States to be sold.
BY MR. OLAH:
***
Q. With respect to vehicles, can you describe the general process of movement from Mexico to wherever it goes?
A. From Mexico, they are packaged. They are put into *** a vehicle. *** I have seen drugs hidden in every area of a vehicle. *** And then they are transported from point A to point B across the border.
Q. And based on your training and experience, are the transporters compensated for their efforts?
A. Yes. It’s a job. It’s to take it from point A to point B.
***
Q. Agent Flood, based on your training and experience, are large quantities of drugs entrusted to drivers that are unaware of those drugs?
MS. IREDALE: Objection. 401, 403.
THE COURT: Overruled.
THE WITNESS: No. In extreme circumstances – actually, in most circumstances, the driver knows they are hired. It’s a business. They are hired to take the drugs from point A to point B.
BY MR. OLAH:
Q. And why aren’t – why don’t they use unknowing couriers, generally?
MS. IREDALE: Objection. 401, 403.
THE COURT: Overruled. You may answer.
THE WITNESS: Generally, it’s a risk of your – your cargo not making it to the new market; not knowing where it’s going; not being able to retrieve it at the ending point, at your point B. So there’s a risk of not delivering your product and, therefore, you’re not going to make any money.
***
Cross Examination
***
Q. So you said that unknowing couriers are very rare.
A. Yes.
***
Q. You work for HSI. Right?
A. Correct.
Q. And you’re aware that your own agency has identified many schemes where drug trafficking organizations use unknowing couriers. Right?
A. I – I know of three schemes that were primarily identified as being possible for an unknowing courier. It doesn’t necessarily mean that they are unknowing couriers. ***

Volunteer Bias in Interlaboratory Studies

The National Institute for Standards and Technology (NIST) is soliciting laboratories to join an "interlaboratory study" of CG-MS (gas chromatography mass spectrometry) for seized drug analysis. The announcement that I received, in abbreviated form, reads:

Forensic Science Quality Assurance Program
Seized Drugs General Method for GC-MS Reporting Limits Study

. . .
Study Design, Purpose, and Rationale
The goals of the study are 1) to capture the range of methods, instrumentation, and analytical approaches used in the community, 2) investigate mass spectral variability across methods, and 3) investigate how different reporting practices effect the limit of seized drug reporting.
Timeline and Commitment
Registration is currently open and will close on July 5, 2024. To participate in this study, laboratories must be accredited forensic laboratories based in the United States and have a valid Schedule I & II DEA license, a validated seized drug screening method using GC-MS, and a documented reporting practice. To be considered for the study, participants will be required to complete a pre-study questionnaire pertaining to the method that will be used for sample analysis. After acceptance into the study, participants will be provided a kit of 10 solutions containing mixtures of controlled substances and asked to analyze the solutions and report whether the analytes are present above their established reporting thresholds. Participants will also be required to report chromatographic peak height/area and retention time of each peak and provide the raw datafile from each run. Standards used for comparison will also be reported. [D]ue to a limited number of available kits, completion of the pre-study questionnaire does not guarantee acceptance into the study. . . .
Publication of Results
Upon closure of data entry, laboratories will receive a preliminary report containing a summary of reported data, consensus results, and a summary of analytes present in each mixture. [A] final report . . . will be made publicly available by Spring 2025. . . . NIST will not knowingly reveal laboratory identities associated with study results.
For questions, contact andrea.yarberry@nist.gov
To signup, go to: https://forms.gle/gPDU9aENHguPkw1D7

The effort is laudable, but one might ask why NIST is not beginning with a sampling frame of laboratories created to represent "the community" and then drawing a probability sample from this list. Will the laboratories that notice the announcement and ask to participate present the full "range of methods, instrumentation, and analytical approaches used in the community"? Will the volunteer sample be skewed toward higher quality labs? Will it include all the "different reporting practices [that] effect [sic] the limit of seized drug reporting [whatever this "limit" denotes in the population of laboratories]"?

Rigor in sampling may not be required to answer certain questions, but it seems relevant to determining what the sign-up form refers to as "the current landscape of GC-MS methods and associated reporting practices and how those factors effect [sic] the concentration of drug that is/is not ultimately reported." Certainly, it should be a consideration for the legal community if and when the results of the study are presented as an indication of "the known or potential rate of error" for GC-MS analysis as practiced in forensic-science laboratories (Daubert v. Merrell Dow Pharm., 509 U.S. 579, 594 (1993)).

ISO Standards on Forensic Science: Pay to Play?

"ISO, the International Organization for Standardization, brings global experts together to agree on the best way of doing things – for anything from making a product to managing a process." ^1/ For the last few years, it has been devising the following overarching set of standards for all of forensic science:

• Forensic sciences (TC 272) ISO/DIS 21043-1, Forensic sciences - Part 1: Terms and definitions - 5/27/2024, $58.00
• ISO/DIS 21043-3, Forensic Sciences - Part 3: Analysis - 5/26/2024, $62.00
• ISO/DIS 21043-4, Forensic Sciences - Part 4: Interpretation - 5/26/2024, $67.00
• ISO/DIS 21043-5, Forensic Sciences - Part 5: Reporting - 5/26/2024, $53.00 ^2/

These are

part of a series which, when completed, will include the different components of the forensic process from scene to courtroom ... . The series describes primarily “what” is standardized, not the “how” or “who”. Best practice manuals and standard operating procedures should describe “how” the requirements of this document would be met. ^3/

It sounds like the standards in progress will not specify "the best way of doing things." Will they merely list the things that are in need of "standardization"? Will they be too open-ended to constitute what the U.S. Supreme Court refers to as "standards controlling the technique's operation"^4/?

I cannot answer these questions because I have not seen the drafts that were open for public comment. Members of the public cannot read the drafts without paying IS0 the $240 listed above. If anyone who has paid to play has thoughts on these documents that they would like to share beyond the TC (Technical Committee) that drafted the standards, I'll post them--at no charge.

Notes

1. Int'l Org. for Standardization, About ISO.
2. ANSI Standards Action, Mar. 15, 2024, at 48.
3. ISO 21043-1:2018(en) Forensic sciences — Part 1: Terms and definitions.
4. Daubert v. Merrell Dow Pharm., 509 U.S. 579, 594 (1993).

What's Uniqueness Got to Do with It?

Columbia University has announced that "AI Discovers That Not Every Fingerprint Is Unique"! The subtitle of the press release of January 10, 2024, boldly claims that

Columbia engineers have built a new AI that shatters a long-held belief in forensics–that fingerprints from different fingers of the same person are unique. It turns out they are similar, only we’ve been comparing fingerprints the wrong way!

Forensic Magazine immediately and uncritically rebroadcast (quoting verbatim without acknowledgment from the press release) the confused statements about uniqueness. According to the Columbia release and Forensic Magazine, "It’s a well-accepted fact in the forensics community that fingerprints of different fingers of the same person—or intra-person fingerprints—are unique and therefore unmatchable." Forensics Magazine adds that "Now, a new study shows an AI-based system has learned to correlate a person’s unique fingerprints with a high degree of accuracy."

Does this mean that the "well-accepted fact" and "long-held belief" in uniqueness been shattered or not? Clearly, not. The study is about similarity, not uniqueness. In fact, uniqueness has essentially nothing to do with it. I can classify equilateral triangles drawn on a flat surface as triangles rather than as other regular polygons whether or not the triangles are each different enough from one another (uniqueness within the set of triangles) that I notice these differences. To say that objects "are unique and therefore unmatchable" is a nonsequitur. A human genome is probably unique to that individual, but forensic geneticists know that six-locus STR profiles are "matchable" to those of other individuals in the population. A cold hit to a person who could not have been the source of the six-locus profile in the U.K. database occurred long ago (as was to be expected for the random-match probabilities of the genotypes).

Perhaps the myth that the study shatters is that it is impossible to distinguish fingerprints left by different fingers of the same individual X from fingerprints left by fingers of different individuals (not-X). But there is no obvious reason why this would be impossible even if every print is distinguishable from every other print (uniqueness).

The Columbia press release describes the study design this way:

[U]ndergraduate senior Gabe Guo ... who had no prior knowledge of forensics, found a public U.S. government database of some 60,000 fingerprints and fed them in pairs into an artificial intelligence-based system known as a deep contrastive network. Sometimes the pairs belonged to the same person (but different fingers), and sometimes they belonged to different people.

Over time, the AI system, which the team designed by modifying a state-of-the-art framework, got better at telling when seemingly unique fingerprints belonged to the same person and when they didn’t. The accuracy for a single pair reached 77%. When multiple pairs were presented, the accuracy shot significantly higher, potentially increasing current forensic efficiency by more than tenfold.

The press release reported the following odd facts about the authors' attempts to publish their study in a scientific journal:

Once the team verified their results, they quickly sent the findings to a well-established forensics journal, only to receive a rejection a few months later. The anonymous expert reviewer and editor concluded that “It is well known that every fingerprint is unique,” and therefore it would not be possible to detect similarities even if the fingerprints came from the same person.

The team ... fed their AI system even more data, and the system kept improving. Aware of the forensics community's skepticism, the team opted to submit their manuscript to a more general audience. The paper was rejected again, but [Professor Hod] Lipson ... appealed. “I don’t normally argue editorial decisions, but this finding was too important to ignore,” he said. “If this information tips the balance, then I imagine that cold cases could be revived, and even that innocent people could be acquitted.” ...

After more back and forth, the paper was finally accepted for publication by Science Advances. ... One of the sticking points was the following question: What alternative information was the AI actually using that has evaded decades of forensic analysis? ... “The AI was not using ... the patterns used in traditional fingerprint comparison,” said Guo ... . “Instead, it was using something else, related to the angles and curvatures of the swirls and loops in the center of the fingerprint.”

Proprietary fingerprint matching algorithms also do not arrive at matches the way human examiners do. They "see" different features in the patterns and tend to rank the top candidates for true matches in a database trawl differently than the human experts. Again, however, these facts about automated systems neither prove nor disprove claims of uniqueness. And, theoretical uniqueness has little or nothing to do with the actual probative value of assertions of matches by humans, automated systems, or both.

Although not directly applicable, the day after the publicity on the Guo et al. paper, I came across the following report on "Limitations of AI-based predictive models" in a weekly survey of papers in Science:

A central promise of artificial intelligence (AI) in health care is that large datasets can be mined to predict and identify the best course of care for future patients. Unfortunately, we do not know how these models would perform on new patients because they are rarely tested prospectively on truly independent patient samples. Chekroud et al. showed that machine learning models routinely achieve perfect performance in one dataset even when that dataset is a large international multisite clinical trial (see the Perspective by Petzschner). However, when that exact model was tested in truly independent clinical trials, performance fell to chance levels. Even when building what should be a more robust model by aggregating across a group of similar multisite trials, subsequent predictive performance remained poor. -- Science p. 164, 10.1126/science.adg8538; see also p. 149, 10.1126/science.adm9218

Note: This posting was last modified on 1/12/24 2:45 PM