Use with Caution: NIJ's Training Course in Population Genetics and Statistics for Forensic Analysts

The National Institute of Justice (NIJ) "is the research, development and evaluation agency of the U.S. Department of Justice . . . dedicated to improving knowledge and understanding of crime and justice issues through science." It offers a series of webpages and video recordings (a "training course") on Population Genetics and Statistics for Forensic Analysts. The course should be approached with caution. I have not worked through all the pages and videos, but here are a few things that rang alarm bells:

NIJ's Training	Comment
Many statisticians have employed what is known as Bayesian probability ... which is based on probability as a measure of one's degree of belief. This type of probability is conditional in that the outcome is based on knowing information about other circumstances and is derived from Bayes Theorem.	Bayes' rule applies to both objective and subjective probabilities. Both types of probability include conditional probabilities. The "type of probability" is not derived from Bayes' Theorem.
Conditional probability, by definition, is the probability P of an event A given that an event B has occurred. ... Take the example of a die with six sides. If one was to throw the die, the probability of it landing on any one side would be 1/6. This probability, however, assumes that the die is not weighted or rigged in any way, and that all of the sides contain a different number. If this were not true, then the probability would be conditional and dependent on these other factors.	The "other factors" are nothing more than part of the description of the experiment whose outcomes are the events that are observed. They are not conditioning events in a sample space.
The following equation can be used to determine the probability of the evidence given that a presumed individual is the contributor rather than a random individual in the population: LR = P(E/H1) / P(E/H0) ... . In the case of a single source sample, the hypothesis for the numerator (the suspect is the source of the DNA) is a given, and thus reduces to 1. This reduces to: LR = 1/ P(E/H0) which is simply 1/P, where P is the genotype frequency.	The hypothesis for the numerator of a likelihood ratio is always "a given"--that is, it goes on the right-hand-side of the expression for a conditional probability. So is the hypothesis in the denominator. Neither probability "reduces to 1" for that reason. Only if the "evidence" is the true genotype in both the recovered sample and the sample from the defendant can it be said that P(E|H1) = 1. In other words, to say that the probability of a reported match is 1 if the defendant is the source treats the probability of laboratory error as zero. That may be acceptable as a simplifying assumption, but the assumption should be made visible in a training course.
Although likelihood ratios can be used for determining the significance of single source crime stains, they are more commonly used in mixture interpretation. ... The use of any formula for mixture interpretation should only be applied to cases in which the analyst can reasonably assume "that all contributors to the mixed profile are unrelated to each other, and that allelic dropout has no practical impact."	This limitation does not apply to modern probabilistic genotyping software!

Aleatory and Epistemic Uncertainty

An article in the Royal Society's Open Science journal on "communicating uncertainty about facts, numbers and science" is noteworthy for the sheer breadth of the fields it surveys and its effort to devise a taxonomy of uncertainty for the purpose of communicating its nature or degree. The article distinguishes between "aleatory" and "epistemic" uncertainty:

[A] large literature has focused on what is frequently termed 'aleatory uncertainty' due to the fundamental indeterminacy or randomness in the world, often couched in terms of luck or chance. This generally relates to future events, which we can't know for certain. This form of uncertainty is an essential part of the assessment, communication and management of both quantifiable and unquantifiable future risks, and prominent examples include uncertain economic forecasts, climate change models and actuarial survival curves.

By contrast, our focus in this paper is uncertainties about facts, numbers and science due to limited knowledge or ignorance—so-called epistemic uncertainty. Epistemic uncertainty generally, but not always, concerns past or present phenomena that we currently don't know but could, at least in theory, know or establish.

The distinction is of interest to philosophers, psychologists, economists, and statisticians. But it is a little hard to pin down with the definition in the article. Aleatory uncertainty applies on the quantum mechanical level, but is it true that "in theory" predictions like weather and life span cannot be certain? Chaos theory shows that the lack of perfect knowledge about initial conditions of nonlinear systems makes long-term predictions very uncertain, but is it theoretically impossible to have perfect knowledge? The card drawn from a well-shuffled deck is a matter of luck, but if we knew enough about the shuffle, couldn't we know the card that is drawn? Thus, I am not so sure that the distinction is between (1) "fundamental ... randomness in the world" and (2) ignorance that could be remedied "in theory."

Could the distinction be between (1) instances of a phenomenon that has variable outcomes at the level of our existing knowledge of the world and (2) a single instance of a phenomenon that we do not regard as the outcome of a random process or that already has occurred, so that the randomness is gone? The next outcome of rolling a die (an alea in Latin) is always uncertain (unless I change the experimental setup to precisely fix the conditions of the roll), 1/ but whether the last roll produced a 1 is only uncertain to the extent that I cannot trust my vision or memory. I could reduce the latter, epistemic uncertainty by improving my system of making observations. For example, I could have several keen and truthful observers watch the toss, or I could film it and study the recording thoroughly. From this perspective, the frequency and propensity conceptions of probability concern aleatory uncertainty, and the subjective and logical conceptions traffic in both aleatory and epistemic uncertainty.

When it comes to the courtroom, epistemic uncertainty is usually in the forefront, and I may get to that example at a later date. For now, I'll just note that, regardless of whether the distinction offered above between aleatory and epistemic uncertainty is philosophically rigorous, people's attitudes toward aleatory and epistemic risk defined in this way do seem to be somewhat different. 2/

NOTES

1. Cf. P. Diaconis, S. Holmes & R. Montgomery, Dynamical Bias in the Coin Toss, 49(2) SIAM Rev. 211-235 (2007), http://epubs.siam.org/doi/abs/10.1137/S0036144504446436?journalCode=siread
2. Gülden Ülkümen, Craig R.  Fox & B. F. Malle, Two Dimensions of Subjective Uncertainty: Clues from Natural Language, 145(10) Journal of Experimental Psychology: General, 1280-1297. http://dx.doi.org/10.1037/xge0000202; Craig R. Fox & Gülden Ülkümen, Distinguishing Two Dimensions of Uncertainty, in Perspectives on Thinking, Judging, and Decision Making (Brun, W., Keren, G., Kirkebøen, G., & Montgomery, H.  eds. 2011).

Likelihoods, Paternity Probabilities, and the Presumption of Innocence in People v. Gonis

In People v. Gonis, 1/ Illinois prosecuted Kenneth Gonis for sexual penetration with his daughter, T.G., when she was 16 years old. T.G. had two children. The first, J.G., was born when she was 17 years old; A.G. was born two years later. To investigate the sexual assault charge, the Illinois State Police Joliet laboratory conducted DNA tests of Gonis, T.G, and the two children. The lab sent the results to the Northeastern Illinois Regional Crime Laboratory for interpretation. That laboratory’s DNA technical leader, Kenneth Pfoser, “entered the DNA profiles into a computer containing a statistical calculator.” He learned that

• “at least 99.9999% of the North American Caucasian/White men would be excluded as being the biological father of [J.G. and A.G.]”;
• the “paternity index” with respect to J.G. was about 195,000,000 and with respect to A.G., it was 26,000,000; and
• “the probability that defendant was the biological father of J.G. and A.G. was 99.9999%.”

In a bench trial before Judge Lance Peterson, the court admitted these findings and convicted Gonis. On appeal, Gonis argued that the trial court erred in denying a pretrial motion to exclude the DNA test results. In an opinion written by Justice Daniel Schmidt, Illinois’ intermediate court of appeals described the motion as asserting only that

[T]he tests were inconsistent with the presumption of innocence because a statistical formula used in the testing assumed a prior probability of paternity. Specifically, the motion alleged:

Assuming that the Northeastern Illinois Regional Crime Laboratory tested the DNA sample using widely accepted practices in the scientific community, said testing was conducted using a statistical mathematical formula. These formulae, as their basis, include a component to determine paternity which by its nature ‘assumes’ that sexual intercourse has in fact taken place.

In other words,

The motion alleged that to allow such paternity test results would violate the presumption of innocence because “the state would be allowed to introduce statistical evidence presuming sexual intercourse, in order to prove an act of sexual intercourse.”

The argument is fallacious for three reasons. First, the probability pertains to the chance that the child was conceived by the mother and the accused man. Conception—the fertilization of an ovum—can occur without penetration.  In the hearing on the motion to exclude, the technical leader referred to artificial insemination, but as insemination and hence pregnancy can occur without penetration by natural mechanisms as well.

Second, even if conception were not merely improbable, but impossible without penetration, it would not follow that a probability of paternity presumes penetration. After all, a probability is not a certainty. To say that an electron has a probability Ψ*ΨdV of being located in a small volume dV is not to presume that the electron is actually located there. To say that the probability of an extended trade war between the U.S. and China is 0.5 (or some other number less than 1) does not presume that this event will occur. That the paternity probability for the defendant is 0.5 (or 0.99999, or any other other number less than 1) also does not presume that the defendant truly is the source of the fertilizing spermatazoon.

Finally, the evidentiary aspect of the presumption of innocence merely directs the judge or jury not to use the fact of the indictment as evidence of guilt. The probabilities in question do not change depending on whether or not a man is indicted.

The opinion in Gonis seems to rely on the activity-level possibility of artificial insemination to reject the defendant's presumption-of-innocence objection. It also comes close to recognizing the second rejoinder, for it states that "Logically, since Bayes's Theorem allowed for the possibility that defendant may not be the father of T.G.'s children, it did not assume that defendant necessarily had sexual intercourse with T.G."

But the court thought that the details of Bayes' Theorem rather than the very definition of probability made the computation compatible with the presumption of innocence. The opinion states that

Pfoser testified that Bayes's Theorem was a likelihood ratio based on two competing hypotheses: (1) defendant was the father, or (2) a random, unrelated individual was the father. Pfoser stated that Bayes's Theorem took “the assumed probability that the person in question is the father of the child” and divided it “by the probability that some unrelated person within the same race group in the general population is the father of the child.” Thus, Pfoser's testimony indicated that Bayes's Theorem posited that either defendant or an individual other than defendant could have been the father of T.G.'s children. Logically, since Bayes's Theorem allowed for the possibility that defendant may not be the father of T.G.'s children, it did not assume that defendant necessarily had sexual intercourse with T.G.

Although a likelihood ratio appears in Bayes' rule, that is not all there is to it, and the description of how the rule works is garbled. The probability that the defendant is the father is not obtained by dividing a probability that he is the father by the probability that an unrelated man is the father. If the expert knew the probability that an unrelated man is the father (and no other alternatives to the defendant's paternity were worthy of consideration), Bayes' rule would be surperfluous. The probability not assigned to the random man is the defendant's probability, so if we have the random-man probability, all we need to do is to subtract it from 1. What remains is the defendant's probability.

The technical leader used Bayes' rule because he did not know the probability that a random man was the father. Let’s look at his explanation of the computation, as presented by the appellate court. The court starts by recounting that

Pfoser testified that DNA paternity testing had three components. The first component of the test involved an exclusion analysis where Pfoser entered the DNA profiles into a computer containing a statistical calculator. If there were any inconsistencies between the alleged father and the child, the computer would give a result of “0” for paternity index.

Apparently, each child shared at least one allele per locus with the defendant, so the computer program did not report an approximate probability of zero, 2/ and the opinion continued:

The next stage involved the calculation of the paternity index, which was a formula used to determine “the likelihood that the assumed alleged father in question is in fact a father as opposed to a random individual that's unrelated in the general population.”

If "likelihood" has the technical meaning of statistical "support" for a hypothesis, this statement could be literally true. But if "likelihood" means probability, as the court evidently and understandably thought, then the explanation is either meaningless or misleading. There is no probability that the defendant is the father "as opposed to a random individual that's unrelated." There is a probability that the defendant is the father (as opposed to everyone else in the population, given all the evidence in the case). And, the paternity index is not even a probability, let alone that one. It is a ratio of two different probabilities. As the court wrote, "the paternity index is the ratio of 'the probability of the alleged father transmitting the alleles and the probability of selecting these alleles at random from the gene pool.' ... (quoting Ivey v. Commonwealth, 486 S.W.3d 846, 851 (Ky. 2016) (quoting D.H. Kaye, The Probability of an Ultimate Issue: The Strange Cases of Paternity Testing, 75 Iowa L. Rev 75, 89 (1989))."

The important aspect of the paternity index is that it is a likelihood ratio that expresses the support that the reported DNA profiles of the mother-child-defendant trio provide (if correctly determined) for the hypothesis that the defendant is the biological father relative to the hypothesis that an unrelated man is the father. The idea is that if the profiles are some number L times more probable under one hypothesis than the other, then they support that hypothesis L times more than they support the alternative. This ratio does not assume that one hypothesis is true and the other false. Rather, it treats both hypotheses as equally worthy of consideration and addresses the probability of the evidence when each one is considered. Thus, the use of the ratio to describe the strength of the evidence for the better supported hypothesis does not conflict with the presumption of innocence. Had the expert simply given the paternity index and spoken of relative support, the defendant's objection would have had even less traction that it did.

But the technical leader did not describe the paternity index in this “likelihoodist” way. Instead, to quote from the opinion,

Pfoser testified that the third component of DNA paternity testing converted the paternity index into a probability of paternity percentage using a statistical, mathematical formula called “Bayes' Theorem.” Pfoser explained:

“Bayes' Theorem is essentially a basis for a likelihood ratio. Like I kind of described before, you're basing it on two conflicting hypotheses or two conflicting assumptions. One is that the individual in question is in fact the father as opposed to a completely random unrelated individual could be the father.”

Pfoser further explained:

“[S]o you're taking two, essentially two, calculations, one calculation is * * * the prior probability or the assumed probability that the person in question is the father of the child and that is divided by the probability that some unrelated person within the same race group in the general population is the father of the child.”

And     "Pfonis testified that the prior probability of paternity was set at 50%.”

As the earlier remarks on Bayes' rule indicate, this explanation of Bayes' rule cries out for corrections at every turn, but now I will just focus on the last sentence because the concept of a prior probability is what triggers worries about the presumption of innocence. The idea of a prior probability is intuitive but not easily mapped onto the legal setting. If I want to infer whether a furry animal that I glimpsed running outside my window is a squirrel (as opposed to a groundhog, a rabbit, a chipmunk, a possum, a cat, a skunk, a bear, or any other furry critter in this neck of the woods), I can start by asking how often various creatures go by. Based on my past observations, I would order the possibilities as squirrel, chipmunk, rabbit, groundhog, cat, and so on. If squirrels account for half of the past sightings, I might select 50% for the probability of a squirrel. This is my prior probability.

Now I think about the details of what I saw in the periphery of my vision. How small was it? What color? Did it seem to have short legs? Was it scurrying or hopping? To the extent that the set of characteristics I was able to discern are more probable for squirrels than for other creatures, I should adjust my probability upwards to arrive at my posterior probability.

Bayes’ rule is a prescription for making the adjustment. It instructs us to multiply the prior odds by the likelihood ratio. Then, voilà, the posterior odds emerge. Suppose my likelihood ratio is 3. I think the characteristics I perceived are 3 times as probable when a squirrel zips by than when the average non-squirrel does. 3/ If the prior probability is ½, the prior odds are 1 to 1, and the posterior odds are 3 × 1:1 = 3:1. Odds of 3:1 correspond to a posterior probability of 3/(3+1) = 3/4. Following Bayes’ rule, I moved from a prior probability of ½ to a posterior of 3/4.

The expert in Gonis arrived at his posterior probability of paternity by making up a set of prior odds — he chose 1:1 — for defendant’s paternity and multiplying them by the paternity index. This looks like a Bayesian calculation. 4/ But in the squirrel-sighting case, there was an empirical basis for the prior odds. I know something about the animals in my neighborhood. The DNA technical leader apparently offered no such justification for his choice of the same number. And how could he? His expertise does not extend to the sexual and criminal conduct of the defendant and everyone else in the male population. The judge or jury, not the DNA profiling expert, is supposed to consider the nongenetic evidence in the case and to rely on its general background information in processing the totality of the evidence in the case to reach its best verdict.

In Gonis, trial judge, who was the factfinder in the case, was explicit about why he found the prior probability of ½ to be acceptable:

The court noted that the cases cited by the State explained why “the .5 number presumption that they start off with is actually just a truly neutral number. It assumes the same likelihood that the defendant was not the father of the child as it does that he would be the father of the child.

This rationale is specious. For a Bayesian, starting with a probability of ½ amounts to believing, before learning about the DNA profiles, that the defendant owns half the probability and that the other half is distributed across every else in the population. Maybe the other evidence in the case would justify that belief, but it hardly seems “neutral” toward the defendant. It treats him very differently from every other man in the population. The more “neutral” position might be to assign the same per capita probability to everyone, including the defendant, and then make adjustments according to the specifics of the case.

The appellate court took no stand on whether the trial court’s conception of neutrality was scientifically or legally tenable. Construing the defendant’s objection narrowly, the court did "not reach the issue of whether a 50% prior probability is a neutral number."

A bona fide Bayesian procedure would be to display the posterior probability for many values of the prior probability. This “variable prior odds approach” avoids the need for the expert to tell the judge or jury which prior probability is correct. 5/

That said, the uncontested likelihood ratios in Gonis, as Justice Schmidt observed, would swamp most prior probabilities. Even if we regarded all the men in the Chicago metropolitan area as equally likely, a priori, to have fathered the two children, the posterior odds of paternity still would be substantial. There are fewer than five million men (of all ages) living in the metropolitan area. So the per capita prior odds are 1:5 million. For the likelihood ratios of 195 million and 26 million, the posterior odds would be more than 39:1 for the paternity of J.G. and 5:1 for the paternity of A.G.

NOTES

1. 2018 IL App (3d) 160166, No. 3-16-0166, 2018 WL 6582850 (Ill. App. Ct. Dec. 13, 2018).
2. Particularly at a single locus, an exclusion does not mean that the probability of paternity is strictly zero. Mutations at some of the STR loci are known to occur at nonzero rates.
3. The phrasing about an "average non-squirrel" is imprecise. There are n+1 mutually exclusive hypotheses H₀, H₁, H₂, ..., H_n, about the animal. Each H_j has a prior probability Pr(H_j) and a likelihood Pr(E|H_j). Let H₀ be the squirrel hypothesis. The appropriate factor for the multiplication of the prior odds is the squirrel likelihood Pr(E|H₀) divided by a weighted average of the other likelihoods. The weight for each non-squirrel hypothesis H_j (j = 1, .., n) is my prior probability on that hypothesis renormalized to reflect that it is conditional on ~H₀. In other words, the Bayes factor is Pr(E|H₀) × [1−Pr(H₀)] divided by Pr(H₁) × Pr(E|H₁) + ... + Pr(H_n) × Pr(E|H_n).
4. By limiting attention to an unrelated man as the only possible alternative, the technical leader was ignoring the terms in the denominator of the Bayes factor for possible related men. See supra note 3. As a result, the Bayesian interpretation he provided was not strictly correct.
5. For discussions of such proposals and their reception in court and in the scholarly literature, see David H. Kaye, David E. Bernstein & Jennifer Mnookin, The New Wigmore on Evidence: Expert Evidence ch. 15 (2d ed. 2011) (updated annually).

Last updated: 16 May 2019, 1:20 PM

Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 3)

Yesterday, I presented some of the testimony behind the one-in-650-biliion probability quoted in ProPublica's reporting on FBI testimony in United States v. McKreith. The figure came from the uniform probability model as applied to the placement of lines on plaid shirts. The selection of that probability model was motivated by observations of the manufacturing process.

Physical scientists are trained to make rough approximations of quantities — both large an small — and they should not be pilloried for trying to use a simple probability model to get a sense of how rare an event might be. But experts should not testify to rough calculations of extremely persuasive probabilities without any error bounds and without trying to verify the assumptions of the modeling effort by collecting and analyzing a reasonable amount of data.

The prosecution's enthusiasm for the largely theoretical probability model in McKreith extended not merely to the shirt mentioned in the ProPublica article, but also to stripes on a handbag. Dr. Vorder Bruegge's testimony on direct examination was relatively mild. He testified, without computing any probabilities, that a photograph and a handbag were "indistinguishable" with respect to "class" and "individual" features that the jurors could see for themselves:

Q. I believe we were at the Mary Kay bag, Government Exhibit 14. [W]ere you able to identify this exhibit as something that was presented to you for ... photographic comparison purposes?
A. Yes, this is the bag that was submitted to me at the FBI laboratory for comparison in this case.
Q. [W]hat observations were you able to make in terms of the class characteristics of the bag?
A. Basically, this is a handbag that has ... two dark straps. It's got a pocket on [one] side ... . It's made primarily of ... multiple panels — two side panels, two end panels, a bottom and a secondary panel that is overlapping this side panel. It's a striped bag that has these bright snaps on the end. ... [B]asically, it's ... dark silver and black stripes on the bag. ... The manufacturer of the bag is indicated by the name Mary Kay on the side.
Q. [C]an you identify ... what distinguishing features there are about the stripes?
A. Well, the stripes are evenly sized stripes. Each one is about a quarter of an inch wide, and they are alternating black and silver stripes.
Q. At the spacing, the same consistent —
A. The spacing is consistent throughout, across the bag, yes.
Q. And you indicated that it has hand straps?
A. Yes. ...
Q. [S]howing you what is marked as Government’s Exhibit 7-EE, regarding the bank robbery at SouthTrust, is this a chart that you prepared for comparison analysis? ...
A. Yes. Government exhibit V7B-EE is a chart that I prepared. ...
Q. ... [W]ere you able to identify, from looking at the bag itself, any individual characteristics, identifying characteristics, which would make this bag unique from all the other Mary Kay bags that may have come off the assembly line at sometime, using the same fabric, being the same sized bag?
A. [T]here are some small white markings on the bag ... . It's not clear to me whether it would be marker or some kind of staining on the bag that occurs at various places around the bag. [O]n the back here [are] little white marks that could be used to differentiate this bag from all of the bags.
      As far as the manufacturing characteristics, we've got another repeating pattern, much like the one in the shirt. in which we've got dark bright stripes. [I]f we look at this end panel, for example, you'll see that the very top stripe on this end of the bag is a ... totally black stripe. And if we look at this end ... where the top is, ... there's actually a little silver there. Likewise, if you look at the edges ... [at] the very top of this [side panel] is about half of one of those silver lines. On the other side, it's not quite a half of one of those silver lines. [Also, it’s] basically silver on the top of the sides — silver on the top of this side panel, end panel, but black [with] maybe a little bit of silver there, [but] when it's folded over, it's black on the top. Also, if you look at where the end panel meets the side panel, you've basically got ... the silver effectively lining up with the black, going from the end panel to the side panel. At the other end, it's a slight offset, slightly different, where it's kind of half and half. It doesn't match up exactly.
Q. ... Are the results of the randomly identifying features in terms of the way the bag was manufactured when these parts were sewn together?
A. Now, I have never been to a bag manufacturing plant, but assuming that the same sewing practices were used —
MR. HOWES: Judge, I'm going to object.
COURT: Sustained.
BY MR, STEFIN:
Q. [S]o you don't know the manufacturing process with respect to that particular bag?
A. That's correct.
Q. Okay. Were you able to do a comparison in any regard to determine whether or not there were points of identification which are similar to the government's Exhibit 14 with the bag depicted in the robbery photos from the SouthTrust bank robbery?
A. Yes I was. ...
Q. And what ... did your comparison yield?
A. Basically, I found a similarity in class characteristics between the bag, the Mary Kay bag — Government’s Exhibit 14 — and the bag carried by the Robert in the SouthTrust bank robbery as depicted on the left hand side of Government’s Exhibit VB7-EE. ...
Q. Are you able to offer an opinion as to whether Government Exhibit 14 is indistinguishable from the bag that's depicted in the bank surveillance photographs of the SouthTrust Bank robbery?
A. Yes.
Q. And what is your opinion?
A. This Government’s Exhibit 14 is indistinguishable from the bag in Government Exhibit 7-EE.

The court sustained the objection to testimony about "randomly identifying features in terms of the way the bag was manufactured when these parts were sewn together" because Dr. Vorde Bruegge forthrightly acknowledged that he was assuming certain facts not in evidence (and outside his expertise as an image analyst). Evidently, the court wanted more than a mere assumption "that the same sewing practices were used." But the next day, on re-direct examination, the prosecutor had Dr. Vorde Bruegge present the same probability model for the placement of the bag's stripes that he had used for the shirt:

Q. ... And with respect to the Mary Kay bag, Government Exhibit 14, didn't you identify individual characteristics of that bag which makes it different than other Mary Kay bags that may have come off the same assembly line?
A. Yes, I did.
Q. And in fact, how many different characteristics were you able to identify looking at that exhibit, in comparison with the bank surveillance photographs of a bag being carried by the robber?
A. There were four specific characteristics that I noted.
Q. And would you remind us of what those four individual characteristics were?
A. The first one was the alignment of the black and silver stripes from the back side of the bag with the end of the bag, the fact that the silver lines on the inside line up with the black lines on the back side. The second characteristic was the location of the snaps at the top on a silver line. The third characteristic was the very small silver line at the top of the back piece. And the last characteristic was the silver line at the very top of the back piece.
Q. And did you come up with any ... odds or probabilities that these items would appear exactly as they are on that bag in a random fashion?
A. Yes, I did. ...
Q. [H]ow were you able to arrive at a probability as far as the individual characteristic that would exist?
A. Basically I'm dealing with a black or white situation. In this case, black or silver. Either you're going to get the black line in one place or you're going to get the silver line in that place. I'm not breaking down by 50% of the black line or 50% of the silver line. I'm just saying it's either a black line or a silver line, which is a 50/50. You got like one chance in two of a specific feature being black or silver. In particular, these silver snaps on the end can either be on a silver line or a black line. They're on a silver line. That eliminates all of the other bags that would have the snaps on a black line.
      Likewise at the top, there is either a silver line at the top or there's a black line at the top. One chance in two, 50/50. So with this, the snaps and the top of the side of the back, it’s one in four. With the addition of the back of the bag silver at the top, it's 1 in 8 — 2 times 2 times 2. And then with the sides here having silver aligning with black, the silver’s either going to align with black, or the silver’s going to align with silver. That's another one in two chance. So 2 times 2 times 2 is 1 in 16.
Q. 2 times 2 times 2 times 2?
A. Yes, correct. 2 to the 4th power.
Q. 2 to the 4th power. So it is possible then to eliminate 15 out of 16 silver bags that would be coming the manufacturing process from whatever company made those bags?
A. That would be the hypothesis, correct.
Q. And did you, fact, find those four same characteristics in the photographs depicting the robber carrying the same bag?
A. Yes. Yes, I did.

This computation of 1/16 for the probability of a four-feature match sounds suspiciously like an application of the principle of insufficient reason discussed yesterday. Every feature is "black or white," present or absent. Not knowing which is more probable, we can presume that each state (present or absent) is equally probable. Four independent features then create 16 equally likely states of nature.

Over a century ago, the New York Court of Appeals soundly rejected this Laplacean reasoning. In People v. Risley, 108 N.E. 200 (N.Y. 1915), a mathematician "was permitted to testify that, by the application of the law of mathematical probabilities, the chance of such defects [in letters typed on an allegedly altered affidavit] being produced by another typewriting machine was so small as to be practically a negative quantity." The mathematics professor "defined the law of probabilities as 'a proper fraction expressing the ratio of the number of ways an event may happen, divided by the total number of ways in which it can happen.'" The court wrote that the extended multiplication of one-half for each peculiarity "was not based upon actual observed data, but was simply speculative ... ."

If the choice of 1/2 for the probability of each binary feature on the handbag was based on nothing more than the fact that that there are two possibilities -- present or absent -- then it too is "simply speculative." If it was based on the more plausible assumption that the bag is sewn together in a way that would be expected to produce a uniform distribution, then the objection -- that the expert had not even visited a Mary Kay plant to learn how the bags were made -- applies. But McKreith's lawyer did not renew the objection. Neither did he argue that the applicability of the model was not verified by data on a sample of Mary Kay bags. If anything, the 1/16 figure for the four-feature handbag match is more speculative than the one in 650 billion probability of the eight-seam shirt match.

Risley and McKreith are not the only cases in which experts have multiplied a lot of small fractions together to get a smaller number. In the 1968 California case of People v. Collins, a prosecutor had a local mathematics professor testify to the product (one in 12 million) of a series of probabilities for characteristics supplied by eyewitnesses who desctibed an interracial couple that drove a yellow automobile away from the scene of a robbery. The probabilities were data-free estimates that the prosecutor fed to the mathematician. The California Supreme Court reversed the conviction, famously remarking that "[m]athematics, a veritable sorcerer in our computerized society, while assisting the trier of fact in the search for truth, must not cast a spell over him," and sparking much distrust in the legal community of probability calculations.

However, the probability model in McKreith, for handbags as well as shirts, is more plausible than the one in Collins. For one thing, the assumption of uncorrelated features is more plausible, and there is at least a modicum of knowledge of the process generating the features. Nevertheless, as I observed with regard to another case in which appellate lawyers for a defendant put an explicitly probabilistic assessment of evidence into their brief, "[t]he attempt to use probability theory ... was heroic. Like many acts of heroism, it also was hasty. Although there were some measurements and estimates of quantities bearing on guilt or innocence, the empirical data were so sketchy that the computations inevitably were more creative than convincing." 1/

NOTES

1. D. H. Kaye, Book Review, Statistics for Lawyers and Law for Statistics, 89 Mich. L. Rev. 1520, 1543 (1991).
   

POSTINGS IN THIS SERIES

• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 1), Mar. 3, 2019
• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 2), Mar. 19, 2019.
• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 3), Mar. 20, 2019

Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 2)

As noted earlier this month, one of the more striking parts of the ProPublica articles on the FBI laboratory's image image analysis unit 1/ is its discovery of testimony, given more than sixteen-years ago, about the probability of finding certain features in photos of a shirt worn by a bank robber caught on film in a string of robberies. The government accused Wilbert McKreith of eight robberies and of and possession of firearms having been previously convicted of a felony. 2/

According to ProPublica, a ruling that Dr. Richard Vorder Bruegge’s “testimony met the Daubert standard” “enshrined the FBI unit’s techniques and testimony as reliable scientific evidence.” This characterization seems to overstate the importance of a single, unpublished pretrial ruling that followed a short, spur-of-the-moment hearing. 2/ Still, the testimony was central to McKreith’s conviction on the bank robbery counts. Dr. Vorder Bruegge’s conclusions fortified the testimony from ordinary witnesses that, among other things, McKreith and the bank robber wore two wrist watches at the same time; wore or had ski masks in south Florida; wore plaid shirts; drove a maroon, burgundy or red-colored car; and had the same general appearance.

Dr. Vorder Bruegge’s signal contribution came from comparing video images from bank cameras to items seized from the defendant’s home and to McKreith himself. Some of this testimony was not definitive. For example, Vorder Bruegge testified that enlargements of the photographs from several bank robberies lacked sufficient resolution for him to say “with a scientific certainty” what that they showed wristwatches. Instead, he stated that they were “consistent with these features being two watches on the left wrist of the bank robber.” Likewise, he testified that a bank photograph that showed a profile of the robber’s face lacked the resolution to reveal allegedly “individual identifying characteristics [such as moles, stars, chipped teeth, ear patterns or other facial minutiae],” but “the overall characteristics of the profile, which include the shape of the nose, mouse, and chin” displayed “similarities” “consistent with” McKreith’s being the bank robber.

Dr. Vorde Bruegge’s analysis of the pattern of a plaid Van Heusen shirt in photos from seven of the robberies produced more conclusive evidence. A condensed version of the testimony on direct examination on December 16, 2002, is in the shaded boxes that follow. Discussion is interspersed between the boxes. ProPublica posted the full transcript.of the testimony from Dec. 16 and 17.

"Individual Identifying Characteristics"

MR. [Roger] STEFIN [Assistant US Attorney]: Our next witness would be Richard Vorder Bruegge.
THE COURT: Okay, members of the jury, [o]ur next witness will be an expert witness. ....
[Defense counsel objected that the government first needed “to lay a sufficient predicate” and “request[ed] a Daubert ... hearing. The court held an impromptu hearing, with no written briefing, at which Dr. Vorder Bruegge affirmed that “the techniques [are] well recognized ... in forensics [a]nd in the scientific community at large.” The court overruled the defendant’s objection.]
Q. [by MR. STEPHIN] [H]ave you had the opportunity to view the eight by ten photographs that have been introduced in evidence in this particular case with respect to the bank robbery images in each of the eight bank robberies?
A. Yes, I have. ...
Q. ... Let’s talk about the shirt now. What is it about the manufacturing process of shirts that might enable you to identify features of that shirt that could be then compared with image analyses?
A. Well, any, any comparison analysis involves a comparison of first of all, the class characteristics. ... In a shirt like this, the class characteristics ... include such things as, does it have a pattern, which this shirt does — it has a plaid pattern or a checked pattern, if you will. ... It also is a button-down shirt — it's not a pull-over, and it's a long-sleeved shirt. ...
Once one has ... found ... class characteristics that match, one can move on to look at the individual identifying characteristics. ... [I]f I were trying to differentiate one person from another and reach a positive identification, then I would need individual identifying characteristics, such as moles, scars, freckle patterns, chipped teeth. These, these features enable you to really differentiate people down to saying this person is unique from all other people. [B]ecause this shirt is a patterned shirt, there are individual identifying characteristics ... based upon ... the way ... the pieces on the shirt are cut out and then sewn together.

The class-individual distinction is entrenched in forensic science, and there is a grain of truth in it, But it assumes what is to be proved -- that every "individual characteristic" (or some unspecified combination of them) makes an individual distinguishable from every other individual. "Class characteristics" are known to be generic, whereas "individual" ones might not be. Logically, both work the same way -- the presence or absence of a characteristic changes the probability of a common source for the specimens or images being compared. A less tendentious pair of terms would be "generic" and "randomly acquired or incorporated," keeping in mind that random characteristics are not necessarily specific to an individual.

"One in-35 Through Random Processes"

Q. So what can you conclude with respect to the pattern itself on this shirt?
A. [B]ecause we don't see the patterns matching across the seams, we can use that as a way to individualize this shirt relative to the other shirts that would be manufactured at the same time. ...
Q. [W]hat do you call this when you find an individual feature?
A. Individual identify characteristic. Basically, each seam ... can be considered on its own as an entire set of individual identifying characteristics. It's almost one individual identifying characteristic. ... [I]f we look at where the right yoke meets the right front panel, there's this ... thick dark that I've been talking about this comes down from the neck, and it terminates here about 2/3 of the way across this panel ... . Likewise, if we look at the left sleeve, I point out that ... very thin dark line here, which is coming up the right sleeve. It almost exactly meets the point where the yoke joins the front panel. ... Likewise, ... look at the collar itself. You'll see that the collar has a very dark line [that] just cuts the corner of the bottom hole on this side. That is paralleled on the other side because it's one piece. So that's another individuating characteristic for this shirt. ...
Q. All right. And did you take precise measurements of these types of lining up of the different — the thick lines or the thin lines in order to further identify, you know, the individual characteristics of this particular shirt?
A. I measured the width of these features on this shirt so that I could figure out [the] relationship ... between places meeting on one side with those meeting on the other side.
Q. And I think you said that the pattern repeats basically every three-and-one-half inches?
A. Yes. Basically, ... if we just repeat going from this dark line to the next dark line, it's three-and-a-half inches. ...
Q. The thick line or the thin line?
A. The thin line. Each ... of these features repeats every three-and-a-half inches. It's just that the thin line is the smallest feature that we can see very easily.
Q. So you were using the thin line as a point of reference in measuring out the three-and-one-half inches? A. Exactly. Recall that I said there's a curved surfaces here where the sleeve meets the other seams. That would complicate any type of repeat analysis because since it's not a straight line, that's going perpendicular to that feature, so it's actually going to be longer.
So basically, if we did have a single seam that we were trying to match this pattern to run across it, the chances of that very thin dark line matching up, matching up or at least touching the black line on the other side, would be one in 35 through random processes.
Q. Could you explain that, explain that a little bit, how you came up to the one in 35? ...
A. ... Let's use, let’s use the yoke and the sleeve. ...
Q. OK ... And the yoke, again, is this back panel ... ?
A. Right. I'm using the yoke in the back because they're almost aligned. ... You can see here that on the yoke the dark line is ... slightly below the seam here. And you have to go about half an inch down before you hit the black line on the other side.
Q. Okay. In other words, you can measure the distance between the black line on the yoke to the black line on the sleeve, and you measured, say ... a half an inch?
A. Yeah.
Q. All right. And so just taking that one point of reference, what would be the odds that two shirts coming out the same manufacturing plant being manufactured in this, in this method would end up having the same alignment of yoke and the sleeve, whereby you could have this ... half an inch offset between them just for this one point of identification?
A. Just for that one point, it would be a 1 in 35 chance.
Q. And how did you come up with the number that it's a 1 in 35 of probability that these two items would line up the same in more than one shirt?
A. Because the feature itself that I’m looking to align is 1/35th of the overall repeat length. And to see ... that one feature come up randomly happens to be one in 35 times.
Q. So one in every 35 shirts manufactured by the company using this patterned cloth, the exact same cloth, you could say that one in 35 probability-wise would come up with the exact same alignment between the yoke and the left sleeve?
A. That is correct.

Several things are going on here to give rise to a uniform probability distribution. First, the plaid design of the shirt comes from a repeated 3.5"-wide distinctive pattern that contains lines of at least two different thicknesses. Second, these lines can be offset from one another across the seams of the shirt. Third, Dr. Vorder Bruegge measures how large the offset is in 1/10th" strips on the 8x10" photographs. Finally, the exact offset -- and hence, which strip a corresponding line on the other side of a seam falls, is determined at random.Thus, the number (call it X) of 1/10th" strips that separate the starting point of every plaid block on the two different cuts of fabric that are sewn together along a seam can take on the value x = 0, 1, 2, ..., 34, with probability f(x) = 1/35 for every x. The following sketch of a repeating block showing only one line in the pattern may clarify what x stands for:

1/10" strip ---------| |
1/10" strip          | |
1/10" strip          | |---------} x=2 (2/10" offset)
1/10" strip          | |
...
1/10" strip ---------| |
1/10" strip          | |
1/10" strip          | |---------
1/10" strip          | |
...                   s
1/10" strip ---------|e|
1/10" strip          |a|
1/10" strip          |m|--------

The idea of assigning an equal probability to elementary events dates back to Bernouilli (1713) and Laplace (1814). The underlying principle of insufficient reason, indifference, or symmetry holds that if there is no reason to believe that any event is more likely to occur than any other event in a set of possible, mutually exhaustive events, then one should assume that all the events are equally probable. This principle offers one way to motivate or understand the axioms of probability theory. Although it has fallen out of favor, it is not without defenders. 4/

But Vorder Bruegge did not base his probability model on a metaphysical or philosophical principle. He gave an empirical justification. During the short Daubert inquiry, he testified that he visited "manufacturing plants, factories where articles of clothing [are] made" and that "in this particular instance, I visited ... cutting plants in Alabama where the patterned material is cut out, as well as manufacturing plants where the cut out pieces are sewn together so that I can see for myself how the process takes place." Furthermore, "I've also been to another plant, Guess plant in Southern California, where shirts were also manufactured, and found that they use the same manufacturing process at the Guess Factory as they do at the Arrow shirt factories in Alabama and ... Georgia." He learned that "[m]anufacturers do not make an effort, in general, to make these features align, because to do so would be prohibitively expensive" and that "there are also places where it is not possible ... to make them align because of the curvature such as along the arms and the sleeves."

In essence, he proposed that busy workers stitched the pre-cut pieces of fabric together without regard to how well their patterns aligned with one another. He elaborated:

Q. All right. Could you ... explain the manufacturing process ... as to how shirts of this nature would be made in a factory?
A. [Y]ou start off with a huge bolt of cloth that can be hundreds and hundreds of yards long. The cloth is then laid down at a cutting plant on a table that [is] maybe about a hundred yards long. Now, these huge bolts will be rolled out on the table and then rolled back onto itself until the entire roll is done. The another role is attached, ... and it continues until you have on the order of 500 plies of material. Now, because of the way that the material is rolled back in, this pattern is not going to line up from one ply to the next. ...
Once the fabric is laid out and you've got 500 plies on top, the manufacturers lay out, basically, tracing paper that has cutting patterns. Just like if you were sewing at home and making your own clothes, you would have a pattern to cut out. Only this is a hundred yards long [and] has been designed by engineers whose only job is to figure out how best to place each piece of this shirt in its closest proximity so they are wasting as little of that fabric as possible. ...
[T]hey slap it down and they will actually have people get on there with jigsaws and cut them out by hand. Or in some of the plants they will have computers that can manage the cutting out. Once all of those pieces are cut out, they [are] transferred over to the sewers ... [Y]ou've got people who do nothing all day but sew on sleeves onto shoulders or yolks on to the back.
Q. ... So there will be, like, thousands of pieces for the collar and thousands ... for the sleeves and so forth?
A. ... If you have 500 plies thick, you have 500 pieces in one particular area. ... [O]n this shirt we've got two pieces on the collar. If you look closely at this shirt, you'll see there's actually a piece here it appears on the outside and a piece on the inside. There's another piece that goes around the collar; so that's three pieces. ... [All together, there are] 16 pieces of this same pattern cloth on this shirt [that must be sewn together].
Q. What significance does that have in terms of determining whether or not, you know, identifying the uniqueness of a particular shirt as opposed to any of the other shirts that are manufactured by the plant using the same cloth coloring?
A. Well, as I mentioned before, every one of these pieces is cut out. The plies line up on every row. If, in the manufacturing process, they happened to take a piece from the top layer and try to stitch it to a piece from the next layer down, you're going to get a lot of randomness in this process. Because there is this offset, you're not going to get this pattern lining up across the seam in any consistent way from one shirt to the next. ... [I]n fact, you can't get a curved seam like this to have an alignment across it, because it isn't geometrically possible.
You look at the pockets on this shirt and you'll see that they line up. [T]he fact that one of these pockets lines up makes this shirt twice as expensive to manufacture as it would be if this pocket were on the bias. [T]hey have to make sure that this pocket is cut out and exactly the same orientation as the front panel. Furthermore, they have to actually have a human being take the time to physically line this pocket up when they're sewing ... .
Q. Is there any effort to line up any of the other pieces of cloth? In other words, lining up the lines from the back to the sleeves, or the yolk to the sleeves, or the yolk to the front panels —
A. No. No. And you can see that for yourself just by comparing the way the left sleeve doesn't align with the yoke in the same way that the right sleeve aligns with the yoke. ...

The uniform-probability model of the placement of the lines is not as "preposterous" and "outrageous" as the ProPublica article suggests. But no one can see for themselves that there is no "effort to line up any of the other pieces of cloth" from the mere fact that the plaid patterns are displaced where a "sleeve aligns with the yoke." That is like saying that a marksman is shooting entirely at random because he missed the bullseye. There is a random component, but if the the marksman is skilled, bullets are more likely to arrive near the center of the target than to be dispersed equally across it.

Our theory about the marksman firing at random would be more credible if we saw that he was wearing a blindfold and firing rapidly. Likewise, the theory in McKreith is that the workers won't "take the time to physically line [the plaid patterns] up when they are sewing." It could be a good theory, but how has it been validated? Might not some workers try to get the unit patterns to line up just a little bit as they join the pre-cut pieces of fabric? If that happened, smaller cross-seam offsets would be more probable than larger ones. We could test the theory empirically by inspecting shirts. If the uniform probability model is correct, we would expect to find pretty much the same number of 1/10" offsets (x = 0, 1, 2, and so on) at a given seam in a large sample of Van Heusen shirts with the same design. The FBI apparently had no such data to support the probability model.

To be sure, the probability model was empirically motivated — Dr. Vorde Bruegge informed himself about the manufacturing process by visiting several manufacturing plants. His sense of the process might be entirely correct. When interviewed, I was not told what model he had adopted and what the basis for it was. `But even if I had been asked to study the testimony before reacting, I still might have said that even a plausible model could be "terribly flawed" (or at least not well validated).

"650 Billion, Give or Take a Few Billion"

A characteristic that results from the manufacturing process that occurs with a probability of 1/35 does not "individualize" an item in the only-one-in-the-universe sense that criminalists use the term. It is a generic feature, and Dr. Vorder Bruegge did not claim otherwise. To arrive at the ultimate opinion took a few more steps. The first was to assume that the offset at each seam is statistically independent of the offset at every other seam (and combination of them).

Q. Now, would the same randomness apply to all the other features in pieces of cloth that go into the shirt?
A. Yes, they would.
Q. So if you were able to, for example, reach a measurement as far as the left sleeve is concerned, with the yoke, would the line up — would the line up be the same or would it be, again, random with respect to the right sleeve?
A. There's going to be a 1 in 35 chance that it's going to be the same on the right sleeve as it is on the left sleeve.
Q. All right. So if you were able to find, for example, from the photographs, two points of identification, whereby the photograph matches the shirt in one area, maybe the yoke to the sleeve on the left side, and then you're able to find a second point of comparison or identification, say on the right sleeve and the yoke, what would be the odds of two shirts randomly being manufactured coming from the factory that would match this particular shirt?
A. It would be ... 1 in 35. But to simplify things and to be conservative, I prefer to use one in 30. By saying one in 30, that's — each giving it a better chance of being the same, but it makes the math easier. Thirty times 30 is 900. So one in 900 chance that you're going to find another shirt that has the left sleeve aligned to the yoke the same way and the right sleeve aligned to the yoke in the same way.
Q. All right. Now let's say you ... have a good enough picture that you can make three points identification. ...
A. Well, ... if we had sleeve to yoke, yoke to back, and yoke to sleeve, then that’s 30 times 30 times 30, which is one in 27,000.
Q. So if you were able to make three items of — points of identification the odds would be one in 27,000 that there would be two shirts randomly made from the factory that would have those three points of identification exactly the same?
A. Correct. ...
Q. And were you able to actually make those types of identification with respect to the photographs depicted in the bank robbery surveillance photos with this shirt ... .?
A. Yes. I was.
Q. And what was the highest number of points of identification that you were able to match up with respect to any particular bank robbery photo — in the surveillance photographs with respect to this shirt?
A. Eight. ...
Q. So 30 to the eighth power [would] be the odds in which two shirts would be randomly manufactured by the company [with] all those eight points of identification lining up exactly the same?
A. That's correct.
Q. And does your pocket calculator actually print out all the numbers that would come out if you were to insert 30 to the 8th power?
A. No. It came out to be 6.5 x 10 to the 11th, which is basically 650 billion, give or take a few billion.

Some of ProPublica's criticism of this part of the testimony misses the mark. The one example of the "[m]any problems in the examiner’s testimony [that] went unnoticed, or were simply unknown, during trial" is that "Vorder Bruegge undercut the precision of his calculations when he admitted having rounded down the shirt measurements used in his calculations because 'it makes the math easier.'" But how could anyone not notice that he used the figure of 1/30 instead of 1/35? And why is that reduction in "the precision of the calculations" a problem for the defendant? It means that the joint probability is even smaller than the pocket calculator's output.

When ProPublica contacted me and quoted the 1/650,000,000,000 figure, my reaction was "How could you get that number"? Not knowing anything about the case, I assumed it was the product of frequency estimates for different kinds of characteristics, such as shirt size, color, style, pattern, imperfections, discolorations, and so on. I doubted that such frequencies were known with sufficient accuracy to justify giving a single  astronomically large denominator to a jury. 5/

Apparently, "Karen Kafadar, chair of the statistics department at the University of Virginia," had a similar reaction and was among the "seven statisticians and independent forensic scientists [who] told ProPublica that "[t]he statistics were also preposterous" because "[t]he features Vorder Bruegge matched might be common in plaid shirts, making them of little value for identifying the garments." Indeed, Dr. Kafadar inveighed "that the 1-in-650-billion claim 'makes about as much sense as the statement two plus two equals five.'"

However, while the proposition that "two plus two equals five" seems to violate mathematical logic, it is not illogical to argue for a uniform probability distribution of offsets and to multiply probabilities as Dr. Vorder Bruegge did. If the 1/10" measurements are all correct, if the offsets are uniformly distributed over such strips, and if the independence assumption for different seams holds, then the probability of an equal offset at every one of n corresponding seams is (1/35)ⁿ, just as he testified. The assumptions are part of a perfectly logical argument, and they are not inherently "preposterous." But they have not been the subject of any systematic study (that I know of).

The Probability of Uniqueness

With all that said, the testimony had an unusual virtue over the typical "individualization" thinking of its day. Dr. Vorde Bruegge followed Lord Kelvin's dictum that "when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be." Dr. Vorde Bruegge was explicit and numerical about the grounds for concluding that only one shirt was involved in the pictures:

Q. [D]id you, during your research, contact the ... Van Heusen Company or the factory to determine ... approximately how many shirts using this cloth and this design were made by them?
A. They do not keep records of every single shirt that they ever make, but they have people who recognize patterns and they also know what their typical runs are. [A]t most, they said they would have made about no more than 18,000 of these shirts. ...
Q. So seven points of identification with respect to the bank robbery surveillance photos and the shirt with the Commerce Union bank robbery?
A. Yes.
Q. Let's go to the next robbery. ...
A. That would be seven, I believe.
Q. So the odds of this being random with two shirts, 30 to the seventh power?
A. If we use 30.
Q. Being conservative?
A. Yes. ...
A. That's seven points of identification with respect to the Bank of America shirt?
A. Yes.
Q. [F]rom the ... Bank United robbery? ... Five positive points of comparison?
A. Right.
Q. Okay. South Trust ... Well it's either 30 to the 7th power or 30 to the 8th power or 30 to the 6th power? A. That's it for South Trust. ...
Q. So that would be five points of identification for the Union Bank photographs?
A. Yes.
Q. [I]s there a point that you were able to conclude that all the photographs in all the different bank robberies are depicting the same shirt?
A. Well, for me in this case, it only took three points. Because with one to the 30th chance each time ... having three points ... I've got 30 times 30 times 30, which is 27,000, which is half again as many as 18,000. [S]o for my opinion it's enough to go over to say that three of those are enough. The fact that I've got four, five, six, seven and eight, just makes me all the more certain.
Q. And what is your opinion with respect to this comparison analysis?
A. They're all the same shirt.
Q. All right. And all the shirts. In the bank robbery surveillance photographs, do you have an opinion as to whether or not they are the same shirt as Exhibit 11, which is the questioned shirt?
A. Government Exhibit 11, in my opinion, is the shirt worn by the bank robber in each of these seven bank robberies. ...

Right or wrong, the testimony is transparent about the threshold for concluding that a picture has the defendant's shirt in it. For 18,000 shirts, Dr. Vorder Bruegge asserted, it takes only three seams to conclude that there is but one shirt in existence. But is this a reasonable criterion for an inference of uniqueness?

It might seem that way. One might be tempted to reason as follows: The random-match probability for two seams is 1/900. For 18,000 shirts manufactured as described in the uniform probability model, we would expect to find about 18,000 shirts x 1/900 matches per shirt = 200 shirts with two matching seams. That is way too many to claim individuality. But for 3 matching seams, the expected number is 18,000 x 1/27,000 = 2/3. In other words, we would expect to find lots of two-seam matches, but not even one three-seam match. So it does not look like a second shirt is very likely for three or more matching seams.

But there is a problem here that did go unrecognized in the case and the article about it. I have called it the expected value fallacy. 6/ It consists of thinking that as long as the expected number of items in a population is less than 1, the probability that there really is less than one is very high. Life, or at least mathematics, is not this simple. Even when the expected number is a little less than 1, as it is here, the probability of at least one additional matching item is appreciable. In this case, it is about 3/10, as shown in Box 1.

BOX 1. THE PROBABILITY OF ANOTHER MATCHING SHIRT
The number y of items with a given characteristic that has a constant, small probability p of appearing in each item in a large population of size n is a Poisson random variable with the parameter λ = np. The probability of any y is f(y; λ) = λ^ye^-λ/y! For the three-seam match, λ = 2/3.The conditional probability that at least one more item in the population has the characteristic, given the known fact that one such item is present is [1 – f(0, λ) – f(1; λ)] / [1 – f(0; λ)] = 0.30.

In short, even ignoring any modeling and measurement uncertainty in McKreith, there is a 30% probability that another Van Heusen shirt that matches at three seams has been manufactured. This large a risk of an erroneous conclusion of individuality would not be acceptable under a 2000 FBI policy that uses similar reasoning in explaining when an analyst may testify that a DNA profile comes from a named individual.

Thus, it appears that Dr. Vorde Bruegge choose a relatively undemanding quantitative threshold for source attribution. We can say this only because he was unusually clear as to the probabilistic basis for his conclusion that only one shirt —  the defendant's —  appeared in all the pictures. It also should be noted that the final opinion would have been the same had he selected a threshold as high as five, for which the conditional probability of matching Van Heusen shirts of the same type would be only 0.0004, or 0.04%. Still, the prosecution introduced the three-seam testimony to make the matches at five and more seams appear more impressive than they were. Even if this abuse of statistical reasoning did not affect the outcome, it was unfortunate.

NOTES

1. Ryan Gabrielson, The FBI Says Its Photo Analysis Is Scientific Evidence. Scientists Disagree, Propublica, Jan. 17, 2019; Ryan Gabrielson, FBI Scientist’s Statements Linked Defendants to Crimes, Even When His Lab Results Didn’t, Propublica, Feb. 22, 2019.
2. United States v. McKreith, 140 Fed.Appx. 112, 2005 WL 1600471 (11th Cir. 2005) (per curiam).
3. Hans-Werner Sinn, A Rehabilitation of the Principle of Insufficient Reason, 94 Q. J. Econ. 493 (1980); Jon Williamson, Justifying the Principle of Indifference, 8 European J. Phil. Sci. 559 (2018).
4. An oral ruling in the midst of a trial rarely creates or enshrines a rule of law. The U.S. Court of Appeals for the Eleventh Circuit affirmed the conviction, but it did not consider its opinion important enough to release for publication, and McKreith did not raise the Daubert claim on appeal. Although the January article states that McKreith “exhausted his appeals, most of which attempted to dispute the FBI Lab findings,” the Westlaw database’s history of the case displays only one direct appeal and one petition for postconviction relief for ineffective assistance of counsel. In neither of these attacks did McKreith challenge the scientific basis of the FBI lab’s work, and no court seems to have cited the case to support admitting similar testimony.
5. The article states that "The statisticians who reviewed Vorder Bruegge’s materials for ProPublica said the examiner’s calculations cannot be correct. Vorder Bruegge’s statistic — 1 in 650 billion — is simply too astronomical to be true, said Kaye, the Penn State professor. There isn’t a database documenting features on plaid-shirt seams like there is for human DNA, making it impossible to determine the likelihood a different shirt would appear to match the robber’s shirt." I was not one of "[t]he statisticians who reviewed Vorder Bruegge’s materials for ProPublica," but I was (and am) of the view that estimating such small numbers on the basis of strong modeling assumptions alone is fraught with danger. The same thing can be said about DNA evidence. Decades ago, England's leading forensic statistician, Ian Evett, rhetorically asked me why American experts give such small probabilities in DNA matches instead of stopping at a number like one in a million.
6. David H. Kaye, The Expected Value Fallacy in State v. Wright, 51 Jurimetrics J. 1 (2011).

POSTINGS IN THIS SERIES

• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 1), Mar. 3, 2019
• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 2), Mar. 19, 2019.
• Propublica's Picture of Photographic Analysis at the FBI Laboratory (pt. 3), Mar. 20, 2019

Likelihood Ratios for Amelia Earhart

Early in the morning of July 2, 1937, Amelia Earhart took off in her twin-engine Lockheed 10E Electra from Papua New Guinea. She was in the midst of a sensational attempt to circle the globe. Her immediate destination was Howland Island. She planned to refuel at this tiny mound. She never made it. The wreckage of her plane and the remains of either Earhart or the navigator who accompanied her have never been located. Or have they?

Earhart probably ran out of fuel while searching for Howland Island. But where? Nikumaroro Island lies some 350 nautical miles south of Howland. Bones found on this uninhabited coral ring in 1939 and sent to Fiji for examination (and now lost) were measured in 1941 by a physician who concluded that they probably belonged to a 45- to 55-year-old male who was around five-and-a-half feet tall.

However, an early release of an article by University of Tennessee anthropologist Richard Jantz, in the University of Florida's new journal, Forensic Anthropology, rejects this conclusion. Arguing that that the skeletal features for sex determination at the time were only weakly probative, Professor Jantz maintains that in 1941, the examiner “could easily have been presented with morphology that he considered male, even though it may have been female.” 1/

As to the lengths of the bones, Dr. Jantz compares estimates for Earhart (inferred from photographs and clothing) with the reported lengths of the Nikumaroro bones using a statistic known as the Mahalanobis distance. By this distance measure for multivariate data, Earhart is more similar to the Nikumaroro bones than 99% of individuals in a sample of bones from 2,776 other people. The sample is not clearly described in the article, but it is well known to forensic anthropologists, and “Jantz told Fox News that 2,776 individuals used in the reference group were all Americans of European ancestry [who] lived during the last half of the 19th century and most of the 20th century ... .” 2/

With respect to this sample, Jantz wrote that:

Earhart’s rank is 19, meaning that 2,758 (99.28%) individuals have a greater distance from the Nikumaroro bones than Earhart, but only 18 (0.65%) have a smaller distance. The rank is subject to sampling variation, so I conducted 1,000 bootstraps of the 2,776 distances, omitting Earhart, then replacing her to determine her rank. Her rank ranged from 9 to 34, the 95% confidence intervals ranging from 12 to 29. If we take the maximum rank resulting from 1,000 bootstraps, 98.77% of the distances are greater and only 1.19% are smaller. If these numbers are converted to likelihood ratios as described by Gardner and Greiner (2006), one obtains 154 using her rank as 19, or 84 using the maximum bootstrap rank of 34. The likelihood ratios mean that the Nikumaroro bones are at least 84 times more likely to belong to Amelia Earhart than to a random individual who ended up on the island.

Let's not get bogged down in the details of the distance measure, bootstrapping, and the conversion to a likelihood ratio. 3/ There is a broader point to note. A likelihood ratio of at least 84 hardly means that “the Nikumaroro bones are at least 84 times more likely to belong to Amelia Earhart than to a random individual” — or even to a random Caucasian-American of the relevant time period. Likelihoods are measures of statistical support for a claim like the one that the particular bones are those of Amelia Earhart. The ratio indicates how much the bone-length data support the Earhart source hypothesis as contrasted to the support those data provide for random-Caucasian-American source hypothesis.

Relative support contributes to the odds in favor of a hypothesis, but it does not express those odds directly. Suppose I pick a coin at random from a box that contains one trick coin (it has heads on both sides) and 128 fair coins. I flip the coin seven times and observe seven heads. The likelihood ratio for the hypotheses of the trick coin as opposed to a fair coin is the probability of the data (seven out of seven heads) for the trick coin divided by the probability for a fair coin. The value of ratio therefore is 1 / (1/2)⁷ = 1 / (1/128) = 128. But the odds that it is a trick coin are nowhere near 128 to 1. They are 1 to 1. Because it is no more probable that the coin is a trick coin than a fair one, I cannot even say that the preponderance of the statistical evidence favors the trick-coin hypothesis. See Box 1.

Box 1. The Odds on the Trick Coin

---

Consider what would happen if we repeated the coin picking and tossing experiment 129 times (replacing the picked coin each time). We expect to pick the trick coin from the box only once and hence to see seven heads for that reason one time in 129. We expect to pick fair coins the other 128 times. We also expect that one of the 128 seven-flip tests of these fair coins will produce seven heads. Observing seven heads does not prove that the coin is more likely to be a trick coin than a fair one. We should post the same odds on each hypothesis about the coin. (This heuristic argument easily could be replaced with a more rigorous proof using Bayes' rule.)

Unfortunately, the statement that “the Nikumaroro bones are at least 84 times more likely to belong to Amelia Earhart than to a random individual” sounds like an assertion that the odds on Earhart as opposed to “a random individual” are at least 84 to 1. Mathematically, if there were no other possibilities to consider, these posterior odds would mean that the probability that the bones are Earhart’s is at least 84/(84+1) = 0.988, or just about 99%. If the probability that the bones are from a member of some other ancestral group (such as Micronesians) is, say, 1/10, then the probability for Earhart would decline to 0.889 (about 89%). See Box 2.

Box 2. Going from the Odds of Two Non-exhaustive Hypotheses to a Probability

---

Let E denote the event that the bones are Earhart’s, let R be the event that they are “a random individual” (among all Caucasian Americans of the time) and let X be the event that they come from someone else. Also, let p, r, and x be the probabilities of each of these events, respectively. Then 84 = p/r, and p+r+x = 1. Substituting and rearranging terms, tt follows that p = 84(1–x) / 85. If x = 0, the probability of E is p = 84/85 = 0.988. If x = 1/10, p = (9/10)×(84/85) = 0.889. 4/

Journalists have picked up on the 99% figure — in some strange ways. The BBC announced that the Forensic Anthropology article “claims they have a 99% match, contradicting an earlier conclusion.” 5/ But being in the upper percentile on a list of distances is not the same as being 99% similar. Fox News crowed  that “Amelia Earhart Disappearance '99 Percent' Solved,” 6/ whatever that may mean.

Other than seeming to conflate the likelihood ratio of 84 with the odds in favor of Earhart, the article does not contend that the probability that the bones are Earhart's is at least 99%. Hewing to the proper interpretation of likelihood as a measure of support for a hypothesis, Professor Jantz wrote that his “analysis ... strongly supports the conclusion that the Nikumaroro bones belonged to Amelia Earhart.” He did go on to mention Bayes' rule and to illustrate what the posterior probability of this hypothesis might be, but following the recommended forensic practice, he did not settle on a specific prior probability. 7/ That probability turns on the non-anthropological information in the case — things like the fact that the Coast Guard cutter off of Howard Island received radio transmissions from Earhart (suggesting that she was not near Nikumaroro Island). Nevertheless, the article seems to propose that the bone lengths in and of themselves prove that the remains probably are Earhart's. It states that:

If [the] sex estimate, can be set aside, it becomes possible to focus attention on the central question of whether the Nikumaroro bones may have been the remains of Amelia Earhart. There is no credible evidence that would support excluding them. On the contrary, there are good reasons for including them. The bones are consistent with Earhart in all respects we know or can reasonably infer. Her height is entirely consistent with the bones. The skull measurements are at least suggestive of female. But most convincing is the similarity of the bone lengths to the reconstructed lengths of Earhart’s bones. Likelihood ratios of 84–154 would not qualify as a positive identification by the criteria of modern forensic practice, where likelihood ratios are often millions or more. They do qualify as what is often called the preponderance of the evidence, that is, it is more likely than not the Nikumaroro bones were (or are, if they still exist) those of Amelia Earhart. If the bones do not belong to Amelia Earhart, then they are from someone very similar to her. And, as we have seen, a random individual has a very low probability of possessing that degree of similarity.

Contrary to the highlighted text, likelihood ratios of 84–154 do not necessarily mean that evidence satisfies the preponderance-of-the-evidence or more-probable-than-not standard of most civil litigation. The legal standard applies to a posterior probability, not to a likelihood ratio standing alone. 8/ Even a large likelihood ratio may not suffice to overcome a small prior probability. (That is what happened in the coin-flipping example of Box 1.) Conversely, even a small likelihood ratio may be enough to boost a prior probability into the more-probable-than-not range.

Professor Jantz recognizes that no one can resolve the historical mystery on the basis of his statistical analysis alone. He writes:

Ideally in forensic practice a posterior probability that remains belong to a victim can be obtained. Likelihood ratios can be converted to posterior odds by multiplying by the prior odds. For example, if we think the prior odds of Amelia Earhart having been on Nikumaroro Island are 10:1, then the likelihood ratios given above become 840–1,540, and the posterior probability is 0.999 in both cases. The prior odds or prior probability pertain to information available before skeletal evidence is considered. It is often impossible to assign specific numbers to the prior probability, because it depends on how the non-osteological evidence is evaluated, and different people will usually evaluate it differently. In jury trials, experts are often advised to testify only to the likelihood ratio developed from the biological evidence. The jury then supplies its own prior odds based on the entire context (e.g., Steadman et al. 2006).

Judging the entire historical record, Jantz adopts a high prior probability (perhaps higher than the 10:1 figure for the prior odds quoted above) to conclude that “[u]ntil definitive evidence is presented that the remains are not those of Amelia Earhart, the most convincing argument is that they are hers.” In other words, the product of the moderately large likelihood ratio and the prior odds (already sufficient to establish a preponderance) is so large that only definitive evidence for an alternative hypothesis could possibly overcome it.

* * *

In the end, do “modern methods produce results that suggest a 99 percent certainty that the bones belonged to Earhart,” as a respected fact-checking website concluded? 8/ Well, the “modern methods” try to exploit the anthropological data more fully than the earlier analyses, but any conclusion about “the certainty that the bones belonged to Earhart” necessarily rests on a judgment of other information as well — the radio transmissions received by the Coast Guard cutter, the failure to spot any signs of Earhart’s presence in a contemporaneous search of the island, other artifacts found on the island in later investigations, and much more.

Professor Jantz is widely reported to have a personal probability of 99% for Earhart as the source of the remains. ABC News, for example, quoted him as stating that "I am 99 percent sure that these bones belong to Amelia Earhart." 10/ This level of belief may be appropriate, based on his review of all the historical information and his latest statistical analysis of the bone lengths. But the article, at least, does not assign a posterior probability to what it presents as “the most convincing argument,” and a forensic anthropologist who did so in court would be relying on knowledge from outside the realm of forensic osteology.

NOTES

1. Richard L. Jantz, Amelia Earhart and the Nikumaroro Bones: A 1941 Analysis versus Modern Quantitative Techniques, 1(2) Forensic Anthropology 1-16 (2018), available at http://journals.upress.ufl.edu/fa/article/view/525/518.
2. James Rogers, Amelia Earhart Mystery Solved? Scientist '99 Percent' Sure Bones Found Belong to Aviator, Fox News, Mar. 7, 2018, http://www.foxnews.com/science/2018/03/07/amelia-earhart-mystery-solved-scientist-99-percent-sure-bones-found-belong-to-aviator.html.
3. The length estimates for Earhart's bones are not exact, bootstrapping is not the same as drawing repeated probability samples from the desired population, and (as discussed in the article) deriving a likelihood ratio involves categorizing continuous measurements into discrete intervals whose size is somewhat arbitrary. Accounting for these sources of uncertainty would produce a broader range of plausible likelihood ratios.
4. Jantz argues that the sizes of the recovered bones are less typical of Pacific Islanders than of “Euro-Americans,” but the article does not maintain that the probability of a different ancestry is zero or that it should be ignored.
5. Amelia Earhart: Island Bones 'Likely' Belonged to Famed Pilot, BBC News, Mar. 8, 2018, http://www.bbc.com/news/world-us-canada-43323944.
6. Rogers, supra note 2.
7. Cf. Ira M. Ellman & David H. Kaye, Probabilities and Proof: Can HLA and Blood Test Evidence Prove Paternity?, 55 NYU L. Rev. 1131 (1979), available at http://ssrn.com/abstract=1466964.
8. See, e.g., John Kaplan, Decision Theory and the Factfinding Process, 20 Stan. L. Rev. 1065 (1968); David H. Kaye, Clarifying the Burden of Persuasion: What Bayesian Decision Rules Do and Do Not Do, 3 Int'l J. Evid. & Proof 1 (1999), available at http:/ssrn.com/abstract=2702990
9. Alex Kasprak, Have Amelia Earhart’s Remains Been Located?, Mar. 15, 2018, https://www.snopes.com/fact-check/amelia-earharts-remains-located/.
10. E.g., Professor Believes Bones Found on Pacific Island Belong to Amelia Earhart, ABC7 Eyewitness News, http://abc7chicago.com/science/professor-believes-bones-found-on-pacific-island-belong-to-amelia-earhart/3190174/.

Four Cases and Two Meanings of a Likelihood Ratio

In Transposing Likelihoods, I quoted descriptions of likelihood ratios in four recent cases. The opinion in Commonwealth v. McClellan, No. 2014 EDA 2016, 2018 WL 560762 (Pa. Super. Jan. 26, 2018), is the only one that presents a likelihood as a probability of the evidence given a hypothesis. The opinion refers to “the conclusion that the DNA sample was at least 29 times more probable if the sample originated from Appellant ... than if it originated from a relative to Appellant and two unknown, unrelated individuals.” Evidence that is 29 times more probable under one hypothesis than under some rival hypothesis favors the former hypothesis over the latter.

The other three opinions mischaracterize the likelihood ratio by treating it as a statement about the relative probabilities of the hypotheses themselves. ^1/

To give a simple example of the conceptual difference between a likelihood and a probability, suppose a card is drawn from a well shuffled deck, and ♦ is the hypothesis that it is a diamond. A witness who never makes a mistake informs us that the card is red. The witness’s report certainly is relevant evidence. It is more probable to occur when the card is a diamond than when it is not. The likelihood ratio simply tells us how many times more probable the evidence is for the hypothesis ♦ than for the rival, composite hypothesis of ♥ or ♠ or ♣.

The value of the likelihood ratio in this case is 3. To see this, first consider the probability of this evidence E if ♦ is true. Because all diamond cards are red, this conditional probability is P(E|♦) = 1.

The probability of E if ♥ or ♠ or ♣ is true is the proportion of red cards among the hearts, spades, and clubs. This proportion is 1/3.

The likelihood ratio for diamonds therefore is P(E | ♦) / P(E | ♥ or ♠ or ♣) = 1 / (1/3) = 3. It is three times more probable to learn that the randomly drawn card is red if it is a diamond than if it is not.

It does not follow, however, that the probability that the card is a diamond given that it is red is three times the probability that it is not a diamond given that it is red. Because half of the red cards are diamonds, the conditional probability of a diamond is P(♦ | E) = ½. Thus, the ratio of the probability of the hypotheses of ♦ to non-♦ is only ½ / ½ = 1.  ^2/

Notes

1. The magnitude of the ratio in these cases makes the error resulting from the transposition somewhat academic. See False, But Highly Persuasive: How Wrong Were the Probability Estimates in McDaniel v. Brown?, 108 Mich. L. Rev. First Impressions 1 (2009).
2. Before learning the card's color, the probability it was a diamond was 1/4 (odds of 1:3). After the witness's report, the probability became 1/2 (odds of 1:1). In the terminology of Bayesian inference, the posterior odds of 1:1 are the Bayes factor of 3 times the prior odds of 1:3. That is, 3 x 1:3 = 3:3 = 1:1.