Evidence-centric and Conclusion-centric: What's the Difference?

In recent months, various forensic practitioners have asked for a short and simple explanation of the difference between the "evidence-centric" and "conclusion-centric" forms of reporting and testifying. I figure that's what I get for introducing the phrases in a panel asked to talk about “Bringing Statistics into the Courtroom” at the Conference on Forensics, Statistics, and Law, at the University of Virginia in March 2018.^\1/ The particular phrases may have been original (at least I think they were), but the underlying ideas, which are what matter, have been around for a long time.

For concreteness, I will focus on a specific example--something that is the target of criticism du jour in US courts. I am thinking of firearms-toolmark examinations culminating in conclusions about the source of a bullet or shell casing. However, the dichotomy between statements of the evidence and its probative value, on the one hand, and statements of the inferences or conclusions that follow from the evidentiatary statements, on the other hand, is far more general. It applies to techniques for shedding light on the possible source of fingerprints, treadmarks, handwriting, images, glass fragments, paint chips, fibers, bones, bitemarks, or any other traces acquired at or around crime scenes or victims. Indeed, the distinction applies to all manner of empirical inquiry, as suggested by the subtitle of the late Frederick Schauer's book, The Proof: Uses of Evidence in Law, Politics, and Everything Else.^\2/ Moreover, nothing in the terminology reveals whether the evidence in question comes from a scientifically validated process, from (typically) less reliable modes of acquiring and assessing data, or from superstitions masquerading as knowledge.

A Recent Case

With these broad obervations in mind, let's turn to a recent appellate opinion comparing the state's arguments for introducing the conclusions of a firearms examiner to arguments that could be marshalled for believing in palmistry. In People v. Tidd, No. A167548, 2024 WL 3982134 (Cal. Ct. App., Aug. 29, 2024), someone in a white sport utility vehicle shot an enebriated pedestrian in the leg. Investigators located a nine-millimeter Luger cartridge case, manufactured by Speer, in the vicinity. Six days later, police arrrested Raymond Tidd as he was about to enter a white SUV carrying a loaded nine-millimeter Sig Sauer pistol. He pled guilty to some firearms possession and carrying charges but went to trial for attempted murder and other felonies for the shooting. Before trial, he objected to testimony from a criminalist with the San Francisco Police Department named Jacobus Swanepoel. The trial court held a hearing and ruled that the expert could testify that the cartridge case came from the defendant's gun as long as he refrained from asserting his opinion “to a scientific certainty.”

At trial, Swanepoel "flatly asserted the recovered cartridge and test cartridges fired from Tidd's gun 'were fired in the same firearm.'" The jury found Tibbs guilty of two firearms felonies for the shooting (but not of attempted murder). However, the court of appeals reversed because the prosecution never showed that the highly subjective judgmental procedure the expert described was "reasonably reliable."

The court's analysis of what it would take to achieve such reliability is open to debate, but that is a subject for another day. My target today is the statements that were made or could have been made about the cartridge cases and the gun or guns that fired them. What statements are "conclusion-centric"? Which are "evidence-centric"?

A Preliminary Point

"Evidence" exists in the context of some hypothesis about a matter of fact (a state of nature). Nothing can sensibly be deemed "evidence" without asking, "evidence of what?" That the sky is dark is evidence that it is nighttime. It is not conclusive evidence (nothing is conclusive evidence of an empirical fact) because darkness in the daytime can come from a total solar eclipse (or even less likely events). But darkness is pretty good evidence (E) for the hypothesis (H) that night has arrived. The hypotheses of most interest in cases like Tidd are source hypotheses like Swanepoel's assertion that the cartridges were "fired in the same gun" or "the cartridges were fired in different guns." These assertions are conclusions that the analyst reaches (or does not reach) on the basis of the evidence about the marks on the objects examined.

Conclusion-centric Statements

With respect to the source hypothesis, a scan of the opinion in Tibbs reveals the following conclusion-centered statements from the firearms examiner:

• The cartridges "were fired in the same gun." In other words, the same-gun hypothesis H is true (given the markings seen in the examination). It may not be "scientifically certain," but it is true nonetheless. In symbols, P(H|E) is practically (or exactly?) 1, where the expression "P(H|E)" stands for "the probability of the hypothesis H given the evidence E." Obviously, this is conclusion-centric, as it tells the jury what they should believe about H (or at least what the examiner believes about this conclusion).
• "[I]t was 'more likely than not that this [Sig Sauer pistol] fired the cartridge casing' he was analyzing." This sounds like a statement that, given the marks (E), H is probably true: P(H|E) > 1/2. This statement is also conclusion-centered. It is a statement about the (probable) truth of H. However, in this case,
• "He explained that by agreeing to 'more likely' he was 'saying it is this firearm,' and by 'not' he was expressing that the chance of finding another firearm 'with the same signature or the same fingerprint is remote or really small.'" Without getting hung up on the dissonance in this compound sentence, this is a statement that given the markings (E), the probability of not-H is "really small." Mathematically, P(not-H|E) implies that H is not merely probable--it is really probable: P(H|E) is almost 1. Another conclusion-centric statement.
• "[T]he Sig Sauer pistol could not be excluded from the class of firearms that could have fired the cartridge case submitted for analysis." Also conclusion-centered. "Could not be excluded" means "is included as" (or "is consistent with") a possible source, it asserts that P(H|E) > 0. That focuses on the (possible) truth of the conclusion. But classifying this phrasing as conclusion-centric is a close case. If "could not be excluded" were a more neutral statement of the pair of observations themselves (such as "has the same characteristics" or "matches") then it is a statement about the evidence E rather than what E proves.

Evidence-centric Statements

• "Investigators also located a nine-millimeter Luger cartridge case ... on the street near where the shooting took place." With respect to the source conclusion, the statement of the caliber is evidence-centric in the sense that it specifies a feature of ammunition that might have used in the shooting without drawing a conclusion (be it categorical or probabilistic) about the gun taken from Tidd as having had that ammunition fired in it. That Tidd was found with a gun of the same caliber makes the characteristic incriminating evidence--it raises the probability of H. In symbols, P(H|caliber match) > P(H), where P(H) is the probability of H prior to considering the caliber data.
• “[I]t would be rare to find another firearm that exhibits that same ... pattern ... .” Evidence-centric. We have an assertion that the frequency of the incriminating marks is unusual in the population of guns that might have been used in the shooting. Finding a rare pattern supports H because it would be surprising to see the same (or more persuasive) evidence E if a different gun had been used. This is an assertion about the value of the evidence as proof of H rather than assertion about the truth of H itself.^\3/

So far we have encountered (1) statements of conclusions (or their probabilities) about a source hypothesis and (2) statements of the evidence ("it's a 9-mm caliber bullet," "the bullets have matching patterns," and so on) with regard to the source hypothesis. But there is a further category of evidence-centric statements that are more interesting. These evidence-centric statements go beyond descriptions of the data and comparisons using the data, but they do not go so far as to express conclusions. Rather, they are statements of the degree to which the evidence supports one hypothesis relative to alternative hypotheses.

I cannot find these in Tidd, for the examiner there was following the conclusion-centric paradigm espoused by the Association of Firearm and Tool Mark Examiners (AFTE). But they not hard to envision. Their defining feature is that they concern the probability of the evidence under the different hypotheses rather than the conclusion-centric probability of the hypothesis given the evidence. For example:

• The patterns (considering all the similarities and differences) are much more probable for bullets fired from defendant's gun than for bullets fired from different guns.
• The patterns are much more compatible with the hypothesis that bullets were fired from the defendant's gun than the hypothesis that they were fired from different guns.
• The patterns provide strong support for the hypothesis that bullets were fired from the defendant's gun rather than the hypothesis that they were fired from different guns.

These statements all flow from assessing how probable the patterns are for pairs of hypotheses. They look to P(E|H) rather than to the examiner's judgments of P(H|E). The underlying principle is that patterns that are more probable under one hypothesis than another are stronger evidence for the former hypothesis and thus support it instead of supporting the competing hypothesis. The hope is that experts will better assist the judge or jury by telling them about the strength of the evidence in this way instead of telling them which hypothesis to believe (to some degree of certainty).

Notes

1. David H. Kaye, The Nikumaroro Bones: How Can Forensic Scientists Assist Factfinders?, 6 Va. J. Crim. L. 101 (2018), https://www.ssrn.com/abstract=3177752. See also David H. Kaye, Forensic Statistics in the Courtroom, in Handbook of Forensic Statistics 225 (David Banks et al. eds., 2021), https://www.ssrn.com/abstract=3561914.
2. Frederick Schauer, The Proof: Uses of Evidence in Law, Politics, and Everything Else (2022).
3. Thus, "really small" is a qualitative expression of p-value. In statistics, a p-value is the the probability of such extreme evidence arising when the "alternative hypothesis" (such as "different gun") is true. Everything else being equal, the smaller the p-value, the harder it is to believe the alternative hypothesis is true.

Who Published Bayes' Theorem?

For the legal profession, "American Law Reports or ALR is a longstanding, highly trusted series of in-depth articles, called annotations, on specific legal issues."^\1/ But the report on "Use of Bayes' Theorem in Criminal and Civil Cases"^\2/ may be less worthy of trust than the average annotation. Historians and statisticians, at least, would be surprised at its opening sentences:

Bayes' Theorem, named for the English clergyman and scientist who published it in 1763, is a scientific principle of the likelihood ratio used to calculate conditional probabilities leading to a variety of statistical "Bayesian" methodologies. It speculates on the probability that a particular fact is true or that a particular event will occur, given our knowledge of one or more related facts or events.

There is no doubt (asymptotically speaking) that in 1763, Thomas Bayes did not publish the theorem that bears his name. He was dead. A close friend, the Reverend Richard Price, read the paper to the Royal Society of London after retrieving it from Bayes's papers and adding to it.^\3/ This expanded and modified version appeared in the Society's Philosophical Transactions.

The ALR's rendering of 18th Century history won't cause any problems for lawyers employing or confronting Bayesian analyses in court. Posthumous or otherwise, discovered first by Bayes or by someone else,^\4/ the theorem is what it is. But what will lawyers make of the idea that the theorem is "a scientific principle of the likelihood ratio" that "speculates" about conditional probability? Is the theorem mere speculation? Is it only about a likelihood ratio? No, and no.

The usual understanding is that a likelihood ratio is the data-based, objective part of the theorem and that speculation becomes an issue if and when the prior probability is not objective as well. As Judea Pearl and Dana MacKenzie explain,

Bayesian statistics gives us an objective way of combining the observed evidence with our prior knowledge (or subjective belief) to obtain a revised belief ... . Still, what frequentists could not abide was that Bayesians were allowing opinion, in the form of subjective probabilities, to intrude into the pristine kingdom of statistics. Mainstream statisticians were won over only grudgingly, when Bayesian analysis proved a superior tool for a variety of applications ... .^\5/

Fitting expert prior probabilities and likelihoods into the legal process poses some special problems for mainstream statisticians. That accommodation is the subject of continuing, if often repetitive, multidisciplinary dialog that does not fit neatly and quickly into a blog.

Notes

1. University of South Carolina School of Law, Legal Research, Analysis & Writing, https://guides.law.sc.edu/LRAWSpring/LRAW/alr.
2. Deborah F. Buckman, Annot., Use of Bayes' Theorem in Criminal and Civil Cases, 47 A.L.R.7th Art. 4 (2019).
3. Stephen M. Stigler, Richard Price, The First Bayesian, 33 Statistical Science 117 (2018), https://doi.org/10.1214/17-STS635; Martyn Hooper, Richard Price, Bayes’ Theorem, and God, Significance 36-39 (Feb. 2013), https://www.york.ac.uk/depts/maths/histstat/price.pdf.
4. Stephen M. Stigler, Who Discovered Bayes’s Theorem?, 37 Am. Stat. 290-296 (1983).
5. Judea Pearl & Dana MacKenzie, The Book of Why: The New Science of Cause and Effect 90 (2018).

NIST's Forensic Science Environmental Scan 2023

Yesterday, I received an email that included the following notice:

NIST released a new report today from an intensive 18-month, multi-pronged effort by the NIST Forensic Science team to assess the strategic opportunities for forensic science research and standards that are needed to drive significant advancements in the practice of forensic science over the coming decade.

The report is based on an extensive literature review, input from external stakeholders, and feedback from NIST subject matter experts. Throughout 2023, NIST conducted an assessment of the forensic science environment to inform strategic planning efforts. The resulting Forensic Science Environmental Scan 2023 report captured salient issues and trends across five different landscapes: governance, economic, societal, scientific and technological, and legal and regulatory. In addition, NIST held a roundtable in September 2023 with forensic science thought leaders from across forensic disciplines to discuss the long-term vision and strategic priorities for forensic science in the United States. NIST used these inputs to identify the grand challenges facing forensic science today, the strategies for addressing them through advances in research and standards, and the subsequent implementation of the advances into forensic science practice.

I can't say I have read the report yet, but I am posting the notice for anyone who wants to scan the "environmental scan."