| The story boils down to this (with annotations on the right) | |||
| Complex DNA mixtures can be tough to interpret, especially when the amounts of DNA are so small that stochastic effects in amplifying DNA sequences are important. In these situations, analysts using a variety of cues and procedures--and even following the same general procedure--can reach different conclusions. False arrests and convictions can follow. | True. See, e.g., Erin Murphy, The Art in the Science of DNA: A Layperson's Guide to the Subjectivity Inherent in Forensic DNA Typing, 58 Emory L.J. 489 (2008). | ||
| There can be a big difference between the answer to the question “What is the probability of a match between a specific DNA profile and the profile an individual picked at random?” and “What is the probability of a match between a specific DNA profile and at least one profile in a large database of profiles picked at random?” | Yes, different questions give different answers. The size of the database affects the answer to the second question, but not the first. But which question should be addressed in court? Probably neither. See, e.g., Ian Ayres & Barry Nalebuff, The Rule of Probabilities: A Practical Approach for Applying Bayes’ Rule to the Analysis of DNA Evidence, 67 Stan. L. Rev. 1447 (2015); David J. Balding, The DNA Database Search Controversy, 58 Biometrics 241 (2002); David H. Kaye, Rounding Up the Usual Suspects: A Legal and Logical Analysis of DNA Database Trawls, 87 N. Car. L. Rev. 425 (2009). | ||
| Thinking about how traces of DNA ended up where they did—and not just whose DNA ended up there—can be crucial to an investigation and prosecution. If investigators, lawyers, and jurors do not understand this, there can be mistaken arrests, prosecutions, and convictions. | A very important point. See, e.g., Peter Gill, Misleading DNA Evidence (2014); David H. Kaye, David Bernstein & Jennifer Mnookin, The New Wigmore on Evidence: Expert Evidence (2d ed. 2011). | ||
| A few things will be surprising | |||
| The 2011 Hampikian-Dror “experiment” with a complex DNA mixture was good proof that examiner bias (from knowing what detectives believed) affected two examiner's interpretations. | From the viewpoint of experimental design, a potentially important confounding variable was obviously present. See D.H. Kaye, The Design of “The First Experimental Study Exploring DNA Interpretation”, 52 Science & Justice 256 (2012). As Dror later wrote, the study was “suggesting that the extraneous context of the criminal case may have influenced the interpretation of the DNA evidence” (emphasis added by Dror in a reply letter). This conclusion is reminiscent of a microscopic hair association reported as “suggesting that the [defendant] may have [left the hair at the crime scene].” How helpful is that? | ||
| "It’s not clear how often coincidental matches occur." Indeed, it might not be a rare event at all, considering that "a rogue Arizona state employee had run tests on the state’s database without the FBI’s permission and found" an inexplicably high number of partial matches. | The employee was not a “rogue”; she did not need the FBI’s permission to use a state database this way; the results were presented on behalf of the state laboratory at an International Symposium on Human Identification. They are not especially anomalous, but largely a consequence of trawling for partial matches among all possible pairs of profiles in a database that includes close relatives. See, e.g., David H. Kaye, Trawling DNA Databases for Partial Matches: What Is the FBI Afraid Of?, 19 Cornell J. L. & Public Pol'y 145 (2009). | ||
Showing posts with label Birthday Problem. Show all posts
Showing posts with label Birthday Problem. Show all posts
Wednesday, June 24, 2015
Frontline's Expose of DNA Testing: Yes and No
The content of a recent Frontline story on “The Surprisingly Imperfect Science of DNA Testing: How a Proven Tool May Be Anything But” will come as no surprise to anyone familiar with the professional and academic literature on forensic DNA identification. The work is important, though, because the main points of the story need to be widely understood. I highlight them below. At the same time, the story relies on some putative problems that are more perceived than real.
Tuesday, July 30, 2013
More on the Population-wide Database
Chris Halikes, posted a comment on yesterday's blog posting about expanding the DNA database to the entire population. He expressed the following concern:
In evaluating this risk, the birthday problem is not on point. In the birthday problem, the number of comparisons grows exponentially with the number of people of in the room because there is no single birthday of interest. For a group of size N, there are approximately N2/2 comparisons. For a fixed birthday, however, there are only N comparisons.
In case work, no one examines all possible pairs of profiles. Like a fixed birthday, a single crime-scene profile is compared to the profiles in the database. Because the number of comparisons is N, the risk grows only linearly with the size of the database. The exponential growth in the Birthday Paradox does not occur.
CODIS is expanding the number of loci in the profile, so the chance that two people (other than monozygotic twins) share the same profile will be even smaller than it is now. When the crime-scene sample yields a good profile (say, 16 or more loci), I cannot see why increasing the database size to that of the entire population would produce many matches to people whose DNA was not at the crime-scene.
The record of an innocent monozygotic twin would pop up in every database trawl for a crime-scene DNA profile that actually belongs to the guilty twin. Strictly speaking, these are not these "coincidental," because there is a deterministic explanation. For such matches -- and for truly coincidental ones -- the population-wide database flags the problem for the police. They get matches to more than one individual! They will know that they must investigate further. Thus, universality alleviates the problem of a "coincidental match" by making every such match apparent.
The problem in the Kumra case, according to prosecutors, was secondary transfer to fingernails that (I assume) did not have DNA from the actual killers. Likewise, if the police or anyone else plants DNA from a target in an incriminating place where the perpetrator's DNA might be found -- and the perpetrator's DNA is not there -- the database trawl will identify the target instead of the perpetrator.
If a crime-scene sample becomes contaminated with extraneous DNA from individuals who were not present at the crime-scene in an amount sufficient to yield a clear and complete profile, the result could be a false accusation and ensuing conviction for people living in the vicinity, being in the correct age range and physical capacity, and lacking a convincing alibi.
Another limitation of DNA databases is that, at best, they can only show that an individual was at a crime-scene at some point. If police and prosecutors unreflectively equate presence with guilt and a suspect has no persuasive explanation for his presence, injustice could follow. However, this is a problem today. Arguably, a universal database might diminish the problem: cases of innocent presence would arise more often, leading to greater sensitivity to this limitation.
Less worrisome are errors such as mislabeling or mistyping the samples in the database. If the profile recorded for me is not my profile but was present at the crime-scene, this mistake will become apparent when I am retested after the cold hit, as is standard procedure. The confirmatory test will exclude me.
A population-wide has other advantages than those I mentioned today and yesterday -- but it also has its share of disadvantages.
References
The mathematics of coincidental matches, particularly with respect to the size of the database, is a thorny subject. However, the odds of a coincidental match seem to rise (the so-called birthday problem). Given the strength of the tunnel vision that sets in when DNA is involved (the Lukis Anderson/Raveesh Kumra case is a recent example), I would think long and hard about the wisdom of having an all-inclusive database.The prospect of more false accusations definitely deserves thought.
In evaluating this risk, the birthday problem is not on point. In the birthday problem, the number of comparisons grows exponentially with the number of people of in the room because there is no single birthday of interest. For a group of size N, there are approximately N2/2 comparisons. For a fixed birthday, however, there are only N comparisons.
In case work, no one examines all possible pairs of profiles. Like a fixed birthday, a single crime-scene profile is compared to the profiles in the database. Because the number of comparisons is N, the risk grows only linearly with the size of the database. The exponential growth in the Birthday Paradox does not occur.
CODIS is expanding the number of loci in the profile, so the chance that two people (other than monozygotic twins) share the same profile will be even smaller than it is now. When the crime-scene sample yields a good profile (say, 16 or more loci), I cannot see why increasing the database size to that of the entire population would produce many matches to people whose DNA was not at the crime-scene.
The record of an innocent monozygotic twin would pop up in every database trawl for a crime-scene DNA profile that actually belongs to the guilty twin. Strictly speaking, these are not these "coincidental," because there is a deterministic explanation. For such matches -- and for truly coincidental ones -- the population-wide database flags the problem for the police. They get matches to more than one individual! They will know that they must investigate further. Thus, universality alleviates the problem of a "coincidental match" by making every such match apparent.
The problem in the Kumra case, according to prosecutors, was secondary transfer to fingernails that (I assume) did not have DNA from the actual killers. Likewise, if the police or anyone else plants DNA from a target in an incriminating place where the perpetrator's DNA might be found -- and the perpetrator's DNA is not there -- the database trawl will identify the target instead of the perpetrator.
If a crime-scene sample becomes contaminated with extraneous DNA from individuals who were not present at the crime-scene in an amount sufficient to yield a clear and complete profile, the result could be a false accusation and ensuing conviction for people living in the vicinity, being in the correct age range and physical capacity, and lacking a convincing alibi.
Another limitation of DNA databases is that, at best, they can only show that an individual was at a crime-scene at some point. If police and prosecutors unreflectively equate presence with guilt and a suspect has no persuasive explanation for his presence, injustice could follow. However, this is a problem today. Arguably, a universal database might diminish the problem: cases of innocent presence would arise more often, leading to greater sensitivity to this limitation.
Less worrisome are errors such as mislabeling or mistyping the samples in the database. If the profile recorded for me is not my profile but was present at the crime-scene, this mistake will become apparent when I am retested after the cold hit, as is standard procedure. The confirmatory test will exclude me.
A population-wide has other advantages than those I mentioned today and yesterday -- but it also has its share of disadvantages.
References
- Persi Diaconis & Frederick Mosteller, Methods for Studying Coincidences, 84 J. Am. Stat. Ass’n 853 (1989) (discussed in Gina Kolata, 1-in-a-Trillion Coincidence, You Say? Not Really, Experts Find, N.Y. Times, Feb. 27, 1990, http://www.nytimes.com/1990/02/27/science/1-in-a-trillion-coincidence-you-say-not-really-experts-find.html?pagewanted=all&src=pm)
- David H. Kaye, Beyond Uniqueness: The Birthday Paradox, Source Attribution, and Individualization in Forensic Science Testimony, 12 Law, Probability & Risk 3 (2013)
- David H. Kaye, On the Hypothetical Population-wide Database, Forensic Science, Statistics & the Law, July 29, 2013, http://for-sci-law-now.blogspot.com/2013/07/on-hypothetical-population-wide-dna.html
- -----, Good Point, Bad Math: DNA Database Statistics Misunderstood (Again), July 26, 2013, http://for-sci-law-now.blogspot.com/2013/07/good-point-bad-math-dna-database.html
Friday, July 26, 2013
Good Point, Bad Math: DNA Database Statistics Misunderstood (Again)
In a New York Times op-ed article, High-Tech, High-Risk Forensics, Hastings Law School Professor Osagie K. Obasogie mentions a false arrest in the investigation of the murder near San Jose of millionaire investor Raveesh Kumra. A database trawl seemed to implicate Lukis Anderson, a homeless man who then “spent more than five months in jail with a possible death sentence hanging over his head” before prosecutors received records showing that Mr. Anderson was in a hospital for alcohol intoxication on the night of the murder. Prosecutors have suggested that paramedics who treated Anderson for intoxication at a liquor store in San Jose a few hours before the murder inadvertently transferred his DNA to Mr. Kumra’s fingernails.
This is by no means the first instance of secondary transfer of DNA to a crime scene (if that is the correct explanation), and forensic scientists have been conducting studies to see how often such transfer occurs. The case shows that police and prosecutors, as well as defense counsel and jurors, should not be overawed by matches from DNA database trawls. This is the good point that Professor Obasogie makes.
The rest of the op-ed article perpetuates mathematical fallacies about DNA databases. First, Professor Obosagie questions “the frequent claim that it is highly unlikely, if not impossible, for two DNA profiles to match by coincidence.” Nothing is impossible, but not many people share the same DNA identification profiles. The op-ed tries to deny this with the observation that “[a] 2005 audit of Arizona’s DNA database showed that, out of some 65,000 profiles, nearly 150 pairs matched at a level typically considered high enough to identify and prosecute suspects. Yet these profiles were clearly from different people.”
The 150 or so matches were, in fact, mismatches. That is, they were partial matches that actually excluded every “matching” pair. Only if an analyst improperly ignored the nonmatching parts of the profiles or if these did not appear in a crime-scene sample could they be reported to match.
Moreover, even we treat all 150 partial matches as tantamount to false full matches in casework, a proper analysis must account for artificially pairing every profile with every other profile and for the many ways to find some kind of partial match. The scientific and legal literature clearly shows that many partial matches are to be expected under these conditions. The 65,000-some samples then in the database gave rise to over a trillion possible partial matches. (The same combinatorial explosion explains the well-known “Birthday paradox” in probability theory.) A mere 150 partial matches out of a trillion opportunities to make such matches represents a quasi-false match rate on the order of about 100 per trillion (0.0000000001). This number is not quite zero, but neither is it “high risk.”
Second, Professor Obosagie notes that “There are also problems with the way DNA evidence is interpreted and presented to juries.” True enough. A common problem is the confusion of a random match probability with a source probability, and the op-ed makes this very mistake when it claims that jurors in the San Francisco prosecution of John Puckett five years ago were “told that there was only a one-in-1.1 million chance that this DNA match was pure coincidence.”
According to Mr. Puckett’s brief on appeal, the DNA analyst testified not to this source probability, but only that “the profile would occur at random among unrelated individuals in about 1 in 1.1 million U.S. Caucasians ... .” (The op-ed also mistakenly asserts that the defendant, who died before his appeal was heard, “is now serving a life sentence.”)
How prosecutors should present statistics in the cases in which the match came about from a database trawl is an important question, but it would have been misleading to tell jurors that the “chance that this DNA match [to John Puckett] was pure coincidence” was “one in three,” as Puckett wanted to do. Putting the possibility of laboratory error to the side (as the op-ed does), the probability of a database trawl suggesting that John Puckett was the killer of 22-year-old Diana Sylvester is 1 if Puckett was in fact the killer. It is about 1 in 1.1 million if he was not.
A figure like 1 in 3 therefore is not suitable for presenting to a jury as a measure of how revealing a database match is, but perhaps it is useful for another purpose. Puckett produced the probability by multiplying the tiny random-match probability of 1 in 1.1 million by the size of the database. This multiplication yields an estimate of how often trawling a database populated entirely by people innocent of every crime for which the database is used would produce matches to anyone. The larger the number, the greater the risk that “innocent databases”—those that fail to contain profiles of the individuals leaving their DNA at crime-scenes—will lead to false accusations.
Of course, we know that this is not a meaningful estimate of the false positive rate of the real databases, for most matches are corroborated with other evidence. If all databases were innocent, and the 1-in-3 number applied, we would be seeing lots of matches to people too young or too old to have committed the crime being investigated, in prison at the time, or having other solid alibis like Mr. Anderson’s.
Thus, it is improbable that strictly coincidental database matches are common—particularly in the run-of-the-mill cases in which the crime-scene DNA is not a mixture or too small or too degraded to be fully typed. Nevertheless, no one can say exactly how often DNA database searches turn up plausible suspects who are, in fact, innocent. This takes me back to Professor Obasogie’s one good point—let’s not get carried away with DNA database matches. There can be innocent explanations for reported matches. But let’s not become overly skeptical of the value of DNA databases to generate investigative leads—and let’s not use bad math to prop up our conjectures.
References
This is by no means the first instance of secondary transfer of DNA to a crime scene (if that is the correct explanation), and forensic scientists have been conducting studies to see how often such transfer occurs. The case shows that police and prosecutors, as well as defense counsel and jurors, should not be overawed by matches from DNA database trawls. This is the good point that Professor Obasogie makes.
The rest of the op-ed article perpetuates mathematical fallacies about DNA databases. First, Professor Obosagie questions “the frequent claim that it is highly unlikely, if not impossible, for two DNA profiles to match by coincidence.” Nothing is impossible, but not many people share the same DNA identification profiles. The op-ed tries to deny this with the observation that “[a] 2005 audit of Arizona’s DNA database showed that, out of some 65,000 profiles, nearly 150 pairs matched at a level typically considered high enough to identify and prosecute suspects. Yet these profiles were clearly from different people.”
The 150 or so matches were, in fact, mismatches. That is, they were partial matches that actually excluded every “matching” pair. Only if an analyst improperly ignored the nonmatching parts of the profiles or if these did not appear in a crime-scene sample could they be reported to match.
Moreover, even we treat all 150 partial matches as tantamount to false full matches in casework, a proper analysis must account for artificially pairing every profile with every other profile and for the many ways to find some kind of partial match. The scientific and legal literature clearly shows that many partial matches are to be expected under these conditions. The 65,000-some samples then in the database gave rise to over a trillion possible partial matches. (The same combinatorial explosion explains the well-known “Birthday paradox” in probability theory.) A mere 150 partial matches out of a trillion opportunities to make such matches represents a quasi-false match rate on the order of about 100 per trillion (0.0000000001). This number is not quite zero, but neither is it “high risk.”
Second, Professor Obosagie notes that “There are also problems with the way DNA evidence is interpreted and presented to juries.” True enough. A common problem is the confusion of a random match probability with a source probability, and the op-ed makes this very mistake when it claims that jurors in the San Francisco prosecution of John Puckett five years ago were “told that there was only a one-in-1.1 million chance that this DNA match was pure coincidence.”
According to Mr. Puckett’s brief on appeal, the DNA analyst testified not to this source probability, but only that “the profile would occur at random among unrelated individuals in about 1 in 1.1 million U.S. Caucasians ... .” (The op-ed also mistakenly asserts that the defendant, who died before his appeal was heard, “is now serving a life sentence.”)
How prosecutors should present statistics in the cases in which the match came about from a database trawl is an important question, but it would have been misleading to tell jurors that the “chance that this DNA match [to John Puckett] was pure coincidence” was “one in three,” as Puckett wanted to do. Putting the possibility of laboratory error to the side (as the op-ed does), the probability of a database trawl suggesting that John Puckett was the killer of 22-year-old Diana Sylvester is 1 if Puckett was in fact the killer. It is about 1 in 1.1 million if he was not.
A figure like 1 in 3 therefore is not suitable for presenting to a jury as a measure of how revealing a database match is, but perhaps it is useful for another purpose. Puckett produced the probability by multiplying the tiny random-match probability of 1 in 1.1 million by the size of the database. This multiplication yields an estimate of how often trawling a database populated entirely by people innocent of every crime for which the database is used would produce matches to anyone. The larger the number, the greater the risk that “innocent databases”—those that fail to contain profiles of the individuals leaving their DNA at crime-scenes—will lead to false accusations.
Of course, we know that this is not a meaningful estimate of the false positive rate of the real databases, for most matches are corroborated with other evidence. If all databases were innocent, and the 1-in-3 number applied, we would be seeing lots of matches to people too young or too old to have committed the crime being investigated, in prison at the time, or having other solid alibis like Mr. Anderson’s.
Thus, it is improbable that strictly coincidental database matches are common—particularly in the run-of-the-mill cases in which the crime-scene DNA is not a mixture or too small or too degraded to be fully typed. Nevertheless, no one can say exactly how often DNA database searches turn up plausible suspects who are, in fact, innocent. This takes me back to Professor Obasogie’s one good point—let’s not get carried away with DNA database matches. There can be innocent explanations for reported matches. But let’s not become overly skeptical of the value of DNA databases to generate investigative leads—and let’s not use bad math to prop up our conjectures.
References
- Osagie K. Obasogie, High-Tech, High-Risk Forensics, N.Y. Times, July 25, 2013, at A27
- D. J. Daly et al., The Transfer of Touch DNA from Hands to Glass, Fabric and Wood, 6 Forensic Sci. Int’l Genetics 41 (2012)
- David H. Kaye, Trawling DNA Databases for Partial Matches: What Is the FBI Afraid Of?, Cornell Journal of Law and Public Policy, Vol. 19, No. 1, Fall 2009, pp. 145-171
- ----- , Rounding Up the Usual Suspects: A Legal and Logical Analysis of DNA Database Trawls, North Carolina Law Review, Vol. 87, No. 2, January 2009, pp. 425-503
- -----, Rehash and Mishmash in the Washington Monthly, Double Helix Law, Feb. 27, 201
- Carll Ladd et al., A Systematic Analysis of Secondary DNA Transfer, 44 J. Forensic Sci. 1270 (1999)
- Alex Lowe et al., The Propensity of Individuals to Deposit DNA and Secondary Transfer of Low Level DNA from Individuals to Inert Surfaces, 129 Forensic Sci. Int’l 25 (2002)
- S. Malsom et al., The Prevalence of Mixed DNA Profiles in Fingernail Samples Taken from Couples Who Co-habit Using Autosomal and Y-STRs, 3 Forensic Sci. Int’l Genetics 57 (2009)