Comparing Countries by Cases of COVID-19: America First?

The following map appeared in the Los Angeles Times newsletter, Coronavirus Today, on April 20, with the caption, Where Is the Coronavirus Spreading?


It does not take a Ph.D. in statistics to see that the total number of cases since the outbreak of the disease is not a measure of where the virus known as SARS-CoV-2 is currently spreading. It is an indirect measure of where it has been up to a given time. To know where the virus is spreading, we should look at new cases of the disease (incidence rates).

Even as a measure of cumulative cases, shading entire countries is misleading. If we want to see where cases have clustered since the recordkeeping started,we could look at the heights of bars placed on top of small, similarly sized geographic regions, where the heights are proportional to the number of cases in each region. Alaska would not stand out in such a graph. Indeed, the Times newsletter has a link to far better infographics from Johns Hopkins University, one of which clearly shows this fact.

Media reporting on only the total number of cases by country promotes false impressions of the incidence and prevalence of the disease across countries. For example, the table below gives approximate numbers for the US and Spain, which have the greatest cumulative numbers of cases as reported today on the Johns Hopkins website. It also includes China, which is ranked 9th in reported cases, and Switzerland (15th).

Country	Cumulative cases	Population	Relative frequency (cases per 100,000)
United States	788,000	328,000,000	240
Spain	204,000	47,000,000	430
China	84,000	1,400,000,000	6
Switzerland	28,000	8,600,000	330

Spain has only about a quarter of the number of cases reported in the US, but it has almost twice the prevalence of the disease. On the basis of reported cases and population size, China's population has emerged relatively unscathed, and on this scale, the US is by no means the most ravaged -- so far.

Likelihood Ratios for Amelia Earhart

Early in the morning of July 2, 1937, Amelia Earhart took off in her twin-engine Lockheed 10E Electra from Papua New Guinea. She was in the midst of a sensational attempt to circle the globe. Her immediate destination was Howland Island. She planned to refuel at this tiny mound. She never made it. The wreckage of her plane and the remains of either Earhart or the navigator who accompanied her have never been located. Or have they?

Earhart probably ran out of fuel while searching for Howland Island. But where? Nikumaroro Island lies some 350 nautical miles south of Howland. Bones found on this uninhabited coral ring in 1939 and sent to Fiji for examination (and now lost) were measured in 1941 by a physician who concluded that they probably belonged to a 45- to 55-year-old male who was around five-and-a-half feet tall.

However, an early release of an article by University of Tennessee anthropologist Richard Jantz, in the University of Florida's new journal, Forensic Anthropology, rejects this conclusion. Arguing that that the skeletal features for sex determination at the time were only weakly probative, Professor Jantz maintains that in 1941, the examiner “could easily have been presented with morphology that he considered male, even though it may have been female.” 1/

As to the lengths of the bones, Dr. Jantz compares estimates for Earhart (inferred from photographs and clothing) with the reported lengths of the Nikumaroro bones using a statistic known as the Mahalanobis distance. By this distance measure for multivariate data, Earhart is more similar to the Nikumaroro bones than 99% of individuals in a sample of bones from 2,776 other people. The sample is not clearly described in the article, but it is well known to forensic anthropologists, and “Jantz told Fox News that 2,776 individuals used in the reference group were all Americans of European ancestry [who] lived during the last half of the 19th century and most of the 20th century ... .” 2/

With respect to this sample, Jantz wrote that:

Earhart’s rank is 19, meaning that 2,758 (99.28%) individuals have a greater distance from the Nikumaroro bones than Earhart, but only 18 (0.65%) have a smaller distance. The rank is subject to sampling variation, so I conducted 1,000 bootstraps of the 2,776 distances, omitting Earhart, then replacing her to determine her rank. Her rank ranged from 9 to 34, the 95% confidence intervals ranging from 12 to 29. If we take the maximum rank resulting from 1,000 bootstraps, 98.77% of the distances are greater and only 1.19% are smaller. If these numbers are converted to likelihood ratios as described by Gardner and Greiner (2006), one obtains 154 using her rank as 19, or 84 using the maximum bootstrap rank of 34. The likelihood ratios mean that the Nikumaroro bones are at least 84 times more likely to belong to Amelia Earhart than to a random individual who ended up on the island.

Let's not get bogged down in the details of the distance measure, bootstrapping, and the conversion to a likelihood ratio. 3/ There is a broader point to note. A likelihood ratio of at least 84 hardly means that “the Nikumaroro bones are at least 84 times more likely to belong to Amelia Earhart than to a random individual” — or even to a random Caucasian-American of the relevant time period. Likelihoods are measures of statistical support for a claim like the one that the particular bones are those of Amelia Earhart. The ratio indicates how much the bone-length data support the Earhart source hypothesis as contrasted to the support those data provide for random-Caucasian-American source hypothesis.

Relative support contributes to the odds in favor of a hypothesis, but it does not express those odds directly. Suppose I pick a coin at random from a box that contains one trick coin (it has heads on both sides) and 128 fair coins. I flip the coin seven times and observe seven heads. The likelihood ratio for the hypotheses of the trick coin as opposed to a fair coin is the probability of the data (seven out of seven heads) for the trick coin divided by the probability for a fair coin. The value of ratio therefore is 1 / (1/2)⁷ = 1 / (1/128) = 128. But the odds that it is a trick coin are nowhere near 128 to 1. They are 1 to 1. Because it is no more probable that the coin is a trick coin than a fair one, I cannot even say that the preponderance of the statistical evidence favors the trick-coin hypothesis. See Box 1.

Box 1. The Odds on the Trick Coin

---

Consider what would happen if we repeated the coin picking and tossing experiment 129 times (replacing the picked coin each time). We expect to pick the trick coin from the box only once and hence to see seven heads for that reason one time in 129. We expect to pick fair coins the other 128 times. We also expect that one of the 128 seven-flip tests of these fair coins will produce seven heads. Observing seven heads does not prove that the coin is more likely to be a trick coin than a fair one. We should post the same odds on each hypothesis about the coin. (This heuristic argument easily could be replaced with a more rigorous proof using Bayes' rule.)

Unfortunately, the statement that “the Nikumaroro bones are at least 84 times more likely to belong to Amelia Earhart than to a random individual” sounds like an assertion that the odds on Earhart as opposed to “a random individual” are at least 84 to 1. Mathematically, if there were no other possibilities to consider, these posterior odds would mean that the probability that the bones are Earhart’s is at least 84/(84+1) = 0.988, or just about 99%. If the probability that the bones are from a member of some other ancestral group (such as Micronesians) is, say, 1/10, then the probability for Earhart would decline to 0.889 (about 89%). See Box 2.

Box 2. Going from the Odds of Two Non-exhaustive Hypotheses to a Probability

---

Let E denote the event that the bones are Earhart’s, let R be the event that they are “a random individual” (among all Caucasian Americans of the time) and let X be the event that they come from someone else. Also, let p, r, and x be the probabilities of each of these events, respectively. Then 84 = p/r, and p+r+x = 1. Substituting and rearranging terms, tt follows that p = 84(1–x) / 85. If x = 0, the probability of E is p = 84/85 = 0.988. If x = 1/10, p = (9/10)×(84/85) = 0.889. 4/

Journalists have picked up on the 99% figure — in some strange ways. The BBC announced that the Forensic Anthropology article “claims they have a 99% match, contradicting an earlier conclusion.” 5/ But being in the upper percentile on a list of distances is not the same as being 99% similar. Fox News crowed  that “Amelia Earhart Disappearance '99 Percent' Solved,” 6/ whatever that may mean.

Other than seeming to conflate the likelihood ratio of 84 with the odds in favor of Earhart, the article does not contend that the probability that the bones are Earhart's is at least 99%. Hewing to the proper interpretation of likelihood as a measure of support for a hypothesis, Professor Jantz wrote that his “analysis ... strongly supports the conclusion that the Nikumaroro bones belonged to Amelia Earhart.” He did go on to mention Bayes' rule and to illustrate what the posterior probability of this hypothesis might be, but following the recommended forensic practice, he did not settle on a specific prior probability. 7/ That probability turns on the non-anthropological information in the case — things like the fact that the Coast Guard cutter off of Howard Island received radio transmissions from Earhart (suggesting that she was not near Nikumaroro Island). Nevertheless, the article seems to propose that the bone lengths in and of themselves prove that the remains probably are Earhart's. It states that:

If [the] sex estimate, can be set aside, it becomes possible to focus attention on the central question of whether the Nikumaroro bones may have been the remains of Amelia Earhart. There is no credible evidence that would support excluding them. On the contrary, there are good reasons for including them. The bones are consistent with Earhart in all respects we know or can reasonably infer. Her height is entirely consistent with the bones. The skull measurements are at least suggestive of female. But most convincing is the similarity of the bone lengths to the reconstructed lengths of Earhart’s bones. Likelihood ratios of 84–154 would not qualify as a positive identification by the criteria of modern forensic practice, where likelihood ratios are often millions or more. They do qualify as what is often called the preponderance of the evidence, that is, it is more likely than not the Nikumaroro bones were (or are, if they still exist) those of Amelia Earhart. If the bones do not belong to Amelia Earhart, then they are from someone very similar to her. And, as we have seen, a random individual has a very low probability of possessing that degree of similarity.

Contrary to the highlighted text, likelihood ratios of 84–154 do not necessarily mean that evidence satisfies the preponderance-of-the-evidence or more-probable-than-not standard of most civil litigation. The legal standard applies to a posterior probability, not to a likelihood ratio standing alone. 8/ Even a large likelihood ratio may not suffice to overcome a small prior probability. (That is what happened in the coin-flipping example of Box 1.) Conversely, even a small likelihood ratio may be enough to boost a prior probability into the more-probable-than-not range.

Professor Jantz recognizes that no one can resolve the historical mystery on the basis of his statistical analysis alone. He writes:

Ideally in forensic practice a posterior probability that remains belong to a victim can be obtained. Likelihood ratios can be converted to posterior odds by multiplying by the prior odds. For example, if we think the prior odds of Amelia Earhart having been on Nikumaroro Island are 10:1, then the likelihood ratios given above become 840–1,540, and the posterior probability is 0.999 in both cases. The prior odds or prior probability pertain to information available before skeletal evidence is considered. It is often impossible to assign specific numbers to the prior probability, because it depends on how the non-osteological evidence is evaluated, and different people will usually evaluate it differently. In jury trials, experts are often advised to testify only to the likelihood ratio developed from the biological evidence. The jury then supplies its own prior odds based on the entire context (e.g., Steadman et al. 2006).

Judging the entire historical record, Jantz adopts a high prior probability (perhaps higher than the 10:1 figure for the prior odds quoted above) to conclude that “[u]ntil definitive evidence is presented that the remains are not those of Amelia Earhart, the most convincing argument is that they are hers.” In other words, the product of the moderately large likelihood ratio and the prior odds (already sufficient to establish a preponderance) is so large that only definitive evidence for an alternative hypothesis could possibly overcome it.

* * *

In the end, do “modern methods produce results that suggest a 99 percent certainty that the bones belonged to Earhart,” as a respected fact-checking website concluded? 8/ Well, the “modern methods” try to exploit the anthropological data more fully than the earlier analyses, but any conclusion about “the certainty that the bones belonged to Earhart” necessarily rests on a judgment of other information as well — the radio transmissions received by the Coast Guard cutter, the failure to spot any signs of Earhart’s presence in a contemporaneous search of the island, other artifacts found on the island in later investigations, and much more.

Professor Jantz is widely reported to have a personal probability of 99% for Earhart as the source of the remains. ABC News, for example, quoted him as stating that "I am 99 percent sure that these bones belong to Amelia Earhart." 10/ This level of belief may be appropriate, based on his review of all the historical information and his latest statistical analysis of the bone lengths. But the article, at least, does not assign a posterior probability to what it presents as “the most convincing argument,” and a forensic anthropologist who did so in court would be relying on knowledge from outside the realm of forensic osteology.

NOTES

1. Richard L. Jantz, Amelia Earhart and the Nikumaroro Bones: A 1941 Analysis versus Modern Quantitative Techniques, 1(2) Forensic Anthropology 1-16 (2018), available at http://journals.upress.ufl.edu/fa/article/view/525/518.
2. James Rogers, Amelia Earhart Mystery Solved? Scientist '99 Percent' Sure Bones Found Belong to Aviator, Fox News, Mar. 7, 2018, http://www.foxnews.com/science/2018/03/07/amelia-earhart-mystery-solved-scientist-99-percent-sure-bones-found-belong-to-aviator.html.
3. The length estimates for Earhart's bones are not exact, bootstrapping is not the same as drawing repeated probability samples from the desired population, and (as discussed in the article) deriving a likelihood ratio involves categorizing continuous measurements into discrete intervals whose size is somewhat arbitrary. Accounting for these sources of uncertainty would produce a broader range of plausible likelihood ratios.
4. Jantz argues that the sizes of the recovered bones are less typical of Pacific Islanders than of “Euro-Americans,” but the article does not maintain that the probability of a different ancestry is zero or that it should be ignored.
5. Amelia Earhart: Island Bones 'Likely' Belonged to Famed Pilot, BBC News, Mar. 8, 2018, http://www.bbc.com/news/world-us-canada-43323944.
6. Rogers, supra note 2.
7. Cf. Ira M. Ellman & David H. Kaye, Probabilities and Proof: Can HLA and Blood Test Evidence Prove Paternity?, 55 NYU L. Rev. 1131 (1979), available at http://ssrn.com/abstract=1466964.
8. See, e.g., John Kaplan, Decision Theory and the Factfinding Process, 20 Stan. L. Rev. 1065 (1968); David H. Kaye, Clarifying the Burden of Persuasion: What Bayesian Decision Rules Do and Do Not Do, 3 Int'l J. Evid. & Proof 1 (1999), available at http:/ssrn.com/abstract=2702990
9. Alex Kasprak, Have Amelia Earhart’s Remains Been Located?, Mar. 15, 2018, https://www.snopes.com/fact-check/amelia-earharts-remains-located/.
10. E.g., Professor Believes Bones Found on Pacific Island Belong to Amelia Earhart, ABC7 Eyewitness News, http://abc7chicago.com/science/professor-believes-bones-found-on-pacific-island-belong-to-amelia-earhart/3190174/.

A DNA Dog Fight

Want to track the neighborhood dogs depositing potentially pathogen-laden poop in public places? Two or three scrappy firms will do it with DNA profiling. The big dog in the market seems to be BioPet Vet Lab of Knoxville, Tennessee, with its PooPrints service. It claims to have "eliminated the dog waste problem in 3,000 properties across the U.S." thanks to its "patented DNA World Pet Registry database." [1]

BioPet sued a Dallas, Texas, competitor who had been a local distributor for PooPrints and then established his own brand.  The new company, PoopCSI, boasted that "We are the only firm used by Federal Prosecutors to link Pet DNA from dog feces to convict a man in the home-invasion and rap [sic] of a woman in Texas. The Canine CODIS database in which [sic] we invented is the first multi-agency forensic DNA database of dogs." [2]

Actually, the "Canine CODIS" is not a Dallas creation, but a joint effort involving the more reputable Veterinary Genetics Laboratory of the University of California at Davis to cope with the serious business of dog fighting competitions. [3] VGL Forensics also does DNA testing for law enforcement agencies in cases of animal attacks and when animal DNA "from hair, saliva, blood, urine, or feces [occurs] during the commission of a crime—from the victim's pet to the suspect or crime scene, and from the suspect's pet to the victim or crime scene." [4]

PoopCSI is now called PET CSI. It accuses PooPrints ("dog poop franchise competitors") of "spread[ing] false rumors and produc[ing] fake press releases," having a "fraudulent business model," using "deceptive trade practices," and "theft of pet DNA intellectual property." [5] PET CSI has the worst possible rating (F) from the Better Business Bureau.

There is some irony in PET CSI's stream-of-consciousness complaint that "They are now attempting to lay claim they invented the DNA pet waste matching service when our lab has been doing this it since 1955 and even on their own website they mention they only got started 2008 and only location as early as last year got started." [5] Neither PET CSI nor DNA profiling methods existed in 1955 -- a scant two years after Watson and Crick elucidated the structure of DNA. Human forensic DNA profiling did not begin until at least 30 years later. [6] 1955 might have been the year that the UC-Davis veterinary lab -- which must be "our lab," as indicated by PET CSI's unacknowledged copying of material from VGL's website -- started, under the name of "the Serology Laboratory, ... established ... for the purpose of verifying parentage for cattle registries" using blood typing. [7]

Also nipping at PooPrints' heels is Mr Dog Poop's CRIME LAB. [8] This Tampa, Florida, upstart contrasts its computer database technology as "star trek" compared to PooPrints' "stone-age" methods. Not only does it offer "Dog poop DNA testing [for] $35/dog," but it conducts "Poop And Run DNA Investigations [for] Only $50/incident." After all, "If the FBI can use DNA technology to enforce the law, why can't HOAs, COAs and Property managers?" [9]

This development dismayed one Washington Post writer, who lamented

Yes, it has come to this: We live in a society where, rather than speaking to one another and gingerly asking neighbors to clean up their dogs’ messes, we mail a portion of said messes to Tennessee in a small bottle so that, using genetic sequencing and mathematical logarithms [sic], the canine hooligan can be identified. [10]

Notes

1. The DNA Solution for Pet Waste Management, https://www.pooprints.com/. For varying comments from property managers, see Rick Montgomery, Growing Pet DNA Industry Identifies Poop Offenders, Kansas City Star, May 1, 2014, http://www.kansascity.com/news/local/article348234/Growing-pet-DNA-industry-identifies-poop-offenders.html; PooPrints is [sic] Fabricated its Pet Waste Business Nationwide, Ripoff Report, Apr. 30, 2015, https://www.ripoffreport.com/reports/pooprints/nationwide/pooprints-is-fabricated-its-pet-waste-business-nationwide-1225902.
2. Eric Nicholson,Two Companies, PooPrints and PoopCSI, Are Battling for the Right to DNA Test Dallas' Dog Crap, Dallas Observer, Aug. 20, 2013, http://www.dallasobserver.com/news/two-companies-pooprints-and-poopcsi-are-battling-for-the-right-to-dna-test-dallas-dog-crap-7102482.
3. University of California at Davis Veterinary Genetics Laboratory, Canine CODIS: Using a CODIS (Combined DNA Index System) to Fight Dog Fighting, https://www.vgl.ucdavis.edu/forensics/CANINECODIS.php.
4. University of California at Davis Veterinary Genetics Laboratory, VGL Forensics, 2018, https://www.vgl.ucdavis.edu/forensics/index.php.
5. PET CSI® Difference, 2018, http://www.petcsi.com/pet-csi-difference.
6. David H. Kaye, The Double Helix and the Law of Evidence (2010).
7. University of California at Davis Veterinary Genetics Laboratory, About the Veterinary Genetics Laboratory, 2018, https://www.vgl.ucdavis.edu/vgl/about.php.
8. Compare DNA Dog Poop Services, http://mrdogpoop.com/splash/compare.html.
9. What Is Mr Dog Poop's® CRIME LAB® Dog Poop DNA Service?, http://mrdogpoop.com.
10. Karen Heller, Using DNA to Catch Canine Culprits — and Their Owners, Wash. Post, Dec. 26, 2014, available at https://www.washingtonpost.com/lifestyle/style/using-dna-to-catch-canine-culprits--and-their-owners/2014/12/26/8d833fc8-8247-11e4-8882-03cf08410beb_story.html?utm_term=.7c49e9dc734a.

Related Postings

• Canine DNA Databases to the Rescue, Apr. 17, 2011
• Adopted Family Members Placed Under Genetic Surveillance, July 3, 2011
• A Probability for Dog DNA, Jan. 14, 2015

Related News Stories

• Debra Cassens Weiss, Lawyer Says Condo’s Proposed PooPrints DNA Program Is ‘Absolutely Ridiculous,' ABAJ, May 20, 2010, http://www.abajournal.com/news/article/lawyer_says_condos_suggested_pooprints_dna_program_is_absolutely_ridiculous/
• Stanley Coren, CSI Meets Dog Poop, Psychology Today,  June 30, 2011, https://www.psychologytoday.com/blog/canine-corner/201106/csi-meets-dog-poop.
• Danny Lewis, Dog Owners Beware, DNA in Dog Poop Could Be Used to Track You Down, Smithsonian Mag., Mar. 30, 2016, https://www.smithsonianmag.com/smart-news/dog-owners-beware-dna-dog-poop-could-used-track-you-down-180958596/
• Maria Arcega-Dunn, City Uses DNA to Track Dog Owners Who Don’t Pick Up, Fox 5 (San Diego), Apr. 29, 2015, http://fox5sandiego.com/2015/04/29/san-diego-turns-to-dna-to-trace-dog-owners-who-dont-pick-up/
• 7NEWS, Colorado Dog Owners Fined Hundreds After Not Picking Up Pet Poop, Denver Post, Nov. 10, 2015, https://www.denverpost.com/2015/11/10/colorado-dog-owners-fined-hundreds-after-not-picking-up-pet-poop/
• Eli Pace, Breckenridge Weighs DNA Testing Dog Poop After Complaints Pile Up, Summit Daily, Mar. 5, 2018, https://www.summitdaily.com/news/breckenridge-weighs-dna-testing-dog-poop-after-complaints-pile-up/

Why the News Stories on the Louisiana Lawyer Dog Are Misleading

The news media and the bloggers are abuzz with stories of how Louisiana judges think that a suspect's statement to "get me a lawyer dog" (or maybe "give me a lawyer, dog") is not an invocation of the right to counsel, which, under Miranda v. Arizona, requires the police to terminate a custodial interrogation. Although the case has nothing to do with forensic science or statistics, this blog often points out journalists' misrepresentations, and I'll digress from the main theme of this blog to explain how the media has misrepresented the case.

Five days ago, a reason.com blog called "Hit and Run" observed that Justice Scott Chricton of the Louisiana Supreme Court wrote a concurring opinion to explain why he agreed that the court need not review the case of Warren Demesme. It seems that Demesme said to his interlocutors, "If y'all, this is how I feel, if y'all think I did it, I know that I didn't do it so why don't you just give me a lawyer dog cause this is not what's up." The police continued the interrogation. Demesme made some admissions. Now he is in jail on charges of aggravated rape and indecent behavior with a juvenile.

Reason.com writer Ed Krayewski (formerly with Fox Business and NBC) read the Justice's opinion and thought

Chricton's argument relies specifically on the ambiguity of what a "lawyer dog" might mean. And this alleged ambiguity is attributable entirely to the lack of a comma between "lawyer" and "dog" in the transcript. As such, the ambiguity is not the suspect's but the court's. And it requires willful ignorance to maintain it.

Credulous writers at Slate, the Washington Post, and other news outlets promptly amplified and embellished Krayewski's report. Slate writer Mark Joseph Stern announced that

Justice Scott Crichton ... wrote, apparently in absolute seriousness, that “the defendant’s ambiguous and equivocal reference to a ‘lawyer dog’ does not constitute an invocation of counsel that warrants termination of the interview.”
Reason’s Ed Krayewski explains that, of course, this assertion is utterly absurd. Demesme was not referring to a dog with a license to practice law, since no such dog exists outside of memes. Rather, as Krayewski writes, Demesme was plainly speaking in vernacular; his statement would be more accurately transcribed as “why don’t you just give me a lawyer, dawg.” The ambiguity rests in the court transcript, not the suspect’s actual words. Yet Crichton chose to construe Demesme’s statement as requesting Lawyer Dog, Esq., rather than interpreting his words by their plain meaning, transcript ambiguity notwithstanding.

This Slate article also urged the U.S. Supreme Court to review the case (if it were to receive a petition from the as-yet-untried defendant). The Post's Tom Jackman joined the bandwagon, arguing that

When a friend says, “I’ll hit you up later dog,” he is stating that he will call again sometime. He is not calling the person a “later dog.”
But that’s not how the courts in Louisiana see it. .... It’s not clear how many lawyer dogs there are in Louisiana, and whether any would have been available to represent the human suspect in this case ... .

Yet, the case clearly does not turn on "the lack of a comma between 'lawyer' and 'dog,'" and Justice Chricton did not maintain that Mr. Demesme's request was too ambiguous because "lawyer" was followed by "dog." Public defender Derwyn D. Bunton contended that when “Mr. Demesme said "with emotion and frustration, 'Just give me a lawyer,'" he "unequivocally and unambiguously asserted his right to counsel." (At least, this is what the Washington Post reported.) If this were all there was to the request, there would no doubt that the police violated Miranda.

The problem for Mr. Demesme is that the "unambiguous" assertion "just give me a lawyer" did not stand alone. It was conditional. What he said was "if y'all think I did it, I know that I didn't do it so why don't you just give me a lawyer ...[?]" For Justice Chricton, the "if" was the source of the ambiguity. That ambiguity did not arise from the phrase "lawyer dog." It would have made no difference if defendant had said "lawyer" without the "dog." Contrary to the media howling, Justice Chricton was not taking the phrase "lawyer dog" literally. He was taking the phrase "if y'all think" literally. Here is what the judge actually wrote:

I agree with the Court’s decision to deny the defendant’s writ application and write separately to spotlight the very important constitutional issue regarding the invocation of counsel during a law enforcement interview. The defendant voluntarily agreed to be interviewed twice regarding his alleged sexual misconduct with minors. At both interviews detectives advised the defendant of his Miranda rights and the defendant stated he understood and waived those rights. ... I believe the defendant ambiguously referenced a lawyer—prefacing that statement with “if y’all, this is how I feel, if y’all think I did it, I know that I didn’t do it so why don’t you just give me a lawyer dog cause this is not what’s up.”... In my view, the defendant’s ambiguous and equivocal reference to a “lawyer dog” does not constitute an invocation of counsel that warrants termination of the interview ... .

The Justice cited a Louisiana Supreme Court case and the U.S. Supreme Court case, Davis v. United States, 512 U.S. 452 (1994). In Davis, Naval Investigative Service agents questioned a homicide suspect after reciting Miranda warnings and securing his consent to be questioned. An hour and a half into the questioning, the suspect said "[m]aybe I should talk to a lawyer." At that point, "[a]ccording to the uncontradicted testimony of one of the interviewing agents, the interview then proceeded as follows:"

[We m]ade it very clear that we're not here to violate his rights, that if he wants a lawyer, then we will stop any kind of questioning with him, that we weren't going to pursue the matter unless we have it clarified is he asking for a lawyer or is he just making a comment about a lawyer, and he said, [']No, I'm not asking for a lawyer,' and then he continued on, and said, 'No, I don't want a lawyer.'

They took a short break, after which "the agents reminded petitioner of his rights to remain silent and to counsel. The interview then continued for another hour, until petitioner said, 'I think I want a lawyer before I say anything else.' At that point, questioning ceased."

The Supreme Court held that the initial statement “[m]aybe I should talk to a lawyer,” coming after a previous waiver of the right to consult counsel and followed by the clarification that "I'm not asking for a lawyer," could be deemed too equivocal and ambiguous to have forced the police to have terminated the interrogation immediately.

The Louisiana case obviously is different. Police did not seek any clarification of the remark about a lawyer "if y'all think I did it." From what has been reported, they continued without missing a beat. However, in the majority opinion for the Court, Justice Sandra Day O'Connor went well beyond the facts of the Davis case to write that

Of course, when a suspect makes an ambiguous or equivocal statement it will often be good police practice for the interviewing officers to clarify whether or not he actually wants an attorney. That was the procedure followed by the NIS agents in this case. Clarifying questions help protect the rights of the suspect by ensuring that he gets an attorney if he wants one, and will minimize the chance of a confession being suppressed due to subsequent judicial second-guessing as to the meaning of the suspect's statement regarding counsel. But we decline to adopt a rule requiring officers to ask clarifying questions. If the suspect's statement is not an unambiguous or unequivocal request for counsel, the officers have no obligation to stop questioning him.

The Louisiana courts -- and many others -- have taken this dictum -- repudiated by four concurring Justices -- to heart. Whether it should ever apply and whether Justice Chricton's application of it to the "if ..." statement is correct are debatable. But no responsible and knowledgeable journalist could say that the case turned on an untranscribed comma or on the difference between "lawyer" and "lawyer dog." The opinion may be wrong, but it is clearly unfair to portray it as "willful ignorance" and "utterly absurd." The majority opinion in Davis and the cases it has spawned are fair game (and the Post article pursues that quarry), but the writing about the dispositive role of the lawyer dog meme in the Louisiana case is barking up the wrong tree.

PCAST's Review of Firearms Identification as Reported in the Press

According to the Washington Post,

The President’s Council of Advisors on Science and Technology [PCAST] said that only one valid study, funded by the Pentagon in 2014, established a likely error rate in firearms testing at no more than 1 in 46. Two less rigorous recent studies found a 1 in 20 error rate, the White House panel said. 1/

The impression that one might receive from such reporting is that errors (false positives? false negatives?) occur in about one case in every 20, or omaybe one in 40.

Previous postings have discussed the fact that a false-positive probability is not generally the probability that an examiner who reports an association is wrong. Here, I will indicate how well the numbers in the Washington Post correspond to statements from PCAST. Not all of them can be found in the section on "Firearms Analysis" (§ 5.5) in the September 2016 PCAST report, and there are other numbers provided in that section.

But First, Some Background

By way of background, the 2016 report observes that

AFTE’s “Theory of Identification as it Relates to Toolmarks”—which defines the criteria for making an identification—is circular. The “theory” states that an examiner may conclude that two items have a common origin if their marks are in “sufficient agreement,” where “sufficient agreement” is defined as the examiner being convinced that the items are extremely unlikely to have a different origin. In addition, the “theory” explicitly states that conclusions are subjective. 2/

A number of thoughtful forensic scientists agree that such criteria are opaque or circular. 3/ Despite its skepticism of the Association of Firearm and Tool Mark Examiners' criteria for deciding that components of ammunition come from a particular, known gun, PCAST acknowledged that

relatively recently ... its validity [has] been subjected to meaningful empirical testing. Over the past 15 years, the field has undertaken a number of studies that have sought to estimate the accuracy of examiners’ conclusions.

Unfortunately, PCAST finds almost all these studies inadequate. "While the results demonstrate that examiners can under some circumstances identify the source of fired ammunition, many of the studies were not appropriate for assessing scientific validity and estimating the reliability because they employed artificial designs that differ in important ways from the problems faced in casework." 4/ "Specially, many of the studies employ 'set-based' analyses, in which examiners are asked to perform all pairwise comparisons within or between small samples sets." Some of these studies -- namely, "closed-set" designs "may substantially underestimate the false positive rate." The only valid way to study validity and reliability, the report insists, is with experiments that require examiners to examine pairs of items in which the existence of a true association is independent of an association in each and every other pair.

The False-positive Error Rate in the One Valid Study

According to the Post, the "one valid study ... established a likely error rate in firearms testing at no more than 1 in 46." This sentence is correct. PCAST reported a "bound on rate" of "1 in 46." 5/ This figure is the upper bound of a one-sided 95% confidence interval. Of course, the "true" error rate -- the one that would exist if there were no random sampling error in the selection of examiners -- could be much larger than this upper bound. Or, it could be much smaller. 6/ The Post omits the statistically unbiased "estimated rate" of "1 in 66" given in the PCAST report.

The 1 in 20 False-positive Error Rate for "Less Rigorous Recent Studies"

The statement that "[t]wo less rigorous recent studies found a 1 in 20 error rate" seems even less complete. The report mentioned five other studies. Four "set-to-set/closed" studies suggested error rates of 1 in 5103 (1 in 1612 for the 95% upper bound). Presumably, the Post did not see fit to mention all the "less rigorous" studies because these closed-set studies were methodologically hopeless -- at least, that is the view of them expressed in .the PCAST report.

The Post's "1 in 20 figure" apparently came from PCAST's follow-up report of 2017. 7/ The addendum refers to a re-analysis of a 14-year-old study of eight FBI examiners co-authored by Stephen Bunch, who "offered an estimate of the number of truly independent comparisons in the study and concluded that the 95% upper confidence bound on the false-positive rate in his study was 4.3%." 8/ This must be one of the Post's "two less rigorous recent studies."  In the 2016 report, PCAST identified it as a "set-to-set/partly open" study with an "estimated rate" of 1 in 49 (1 in 21 for the 95% upper bound). 9/

The second "less rigorous" study is indeed more recent (2014). The 2016 report summarizes its findings as follows:

The study found 42 false positives among 995 conclusive examinations. The false positive rate was 4.2 percent (upper 95 percent confidence bound of 5.4 percent). The estimated rate corresponds to 1 error in 24 cases, with the upper bound indicating that the rate could be as high as 1 error in 18 cases. (Note: The paper observes that “in 35 of the erroneous identifications the participants appeared to have made a clerical error, but the authors could not determine this with certainty.” In validation studies, it is inappropriate to exclude errors in a post hoc manner (see Box 4). However, if these 35 errors were to be excluded, the false positive rate would be 0.7 percent (confidence interval 1.4 percent), with the upper bound corresponding to 1 error in 73 cases.) 10/

Another Summary

Questions of which studies count, how much they count, and what to make of their limitations are intrinsic to scientific literature reviews. Journalists limited to a few sentences hardly can be expected to capture all the nuances. Even so, a slightly more complete summary of the PCAST review might read as follows:

The President’s Council of Advisors on Science and Technology said that an adequate body of scientific studies does not yet exist.to show that toolmark examiners can associate discharged ammunition to a specific firearm with very high accuracy. Only one rigorous study with one type of gun, funded by the Defense Department, has been conducted. It found that examiners who reached firm conclusions made positive associations about 1 time in 66 when examining cartridge cases from different guns. Less rigorous studies have found both higher and lower false-positive error rates for conclusions of individual examiners, the White House panel said.

NOTES

1. Spencer S. Hsu & Keith L. Alexander, Forensic Errors Trigger Reviews of D.C. Crime Lab Ballistics Unit Prosecutors Say, Wash. Post, Mar. 24, 2017.
2. PCAST, at 104 (footnote omitted).
3. See, e.g., Christophe Champod, Chris Lennard, Pierre Margot & Milutin Stoilovic, Fingerprints and Other Ridge Skin Impressions 71 (2016) (quoted in David H. Kaye, "The Mask Is Down": Fingerprints and Other Ridge Skin Impressions, Forensic Sci., Stat. & L., Aug. 11, 2017, http://for-sci-law.blogspot.com/2017/08/the-mask-is-down-fingerprints-and-other.html)
4. PCAST, at 105.
5. Id. at 111, tbl. 2.
6. The authors of the study had this to say about the false-positive errors:

   [F]or the pool of participants used in this study the fraction of false positives was approximately 1%. The study was specifically designed to allow us to measure not simply a single number from a large number of comparisons, but also to provide statistical insight into the distribution and variability in false-positive error rates. The result is that we can tell that the overall fraction is not necessarily representative of a rate for each examiner in the pool. Instead, examination of the data shows that the rate is a highly heterogeneous mixture of a few examiners with higher rates and most examiners with much lower error rates. This finding does not mean that 1% of the time each examiner will make a false-positive error. Nor does it mean that 1% of the time laboratories or agencies would report false positives, since this study did not include standard or existing quality assurance procedures, such as peer review or blind reanalysis. What this result does suggest is that quality assurance is extremely important in firearms analysis and that an effective QA system must include the means to identify and correct issues with sufficient monitoring, proficiency testing, and checking in order to find false-positive errors that may be occurring at or below the rates observed in this study.

   David P. Baldwin, Stanley J. Bajic, Max Morris, and Daniel Zamzow, A Study of False-Positive and False-Negative Error Rates in Cartridge Case Comparisons, May 2016, at 18, available at https://www.ncjrs.gov/pdffiles1/nij/249874.pdf.
7. PCAST, An Addendum to the PCAST Report on Forensic Science in Criminal Courts, Jan. 6, 201.
8. Id. at 7.
9. PCAST, at 111, tbl. 2
10. Id. at 95 (footnote omitted).

Index to comments and cases on the PCAST report

Beyond the Higgs Boson: The “Prosecutor’s Fallacy” Does Not Explain Why Experimenters Are Cautious About Announcing New Discoveries

A posting of July 6, 2012, "The Probability that the Higgs Boson Has Been Discovered,"  mentioned the transposition of a p-value in stories in the popular press about the discovery of the Higgs Boson. Reporting about the excitement last month over a highly tentative announcement of a far more massive relative of the Higgs particle, the New York Times did not learn from its earlier error (or else physicist Kyle Cranmer misspoke). The Times stated that

When all the statistical effects are taken into consideration, Dr. Cranmer said, the bump in the Atlas data had about a 1-in-93 chance of being a fluke — far stronger than the 1-in-3.5-million odds of mere chance, known as five-sigma, considered the gold standard for a discovery. That might not be enough to bother presenting in a talk except for the fact that the competing CERN team, named C.M.S., found a bump in the same place.

One perceptive science writer, Faye Flam, spotted the transposition error in the Times article and promptly called attention to it in her science column for Bloomberg Views. However, she then argued that 1/93 is too high a significance level to use in particle physics because of “a problem of statistics called the prosecutor’s fallacy” (referring to  UCLA physicist Robert Cousins as the source of this argument).

That argument is wrong, or at least incomplete. The “prosecutor’s fallacy” is just a special case of the ubiquitous practice of transposing the terms in a conditional probability and thinking that the value stays the same. The usual name for this in statistics is the transposition fallacy. The transposition fallacy is a fallacy because Pr(A | B) <> P(B | A) unless P(A) = P(B).  In other words, P(A | B) actually could be higher — or lower — than P(B | A).

This inequality does not reveal why particle physicists normally choose a much more demanding significance level than 1/93.  Flam uses two examples with low prior probabilities P(A) to give the impression that P(A | B) must be greater than P(B | A), so that a higher value for P(B | A) is necessary to achieve some desired value for P(A | B). The argument works only in situations in which the prior probability of a new particle is small. That is a fair claim here, because the Standard Theory, which is well entrenched, does not predicts the super-massive particle). However, it is not clear why that always would be the case. Thus, the question of why particle physicists used the stringent 5-sigma rule for the discovery of the Higgs Boson remains.

One standard argument for setting an especially demanding significance level is that the cost of a false positive is far greater than the cost of a false negative (which can be tolerated while one waits for more data). Flam and Cousins mention error costs in this way, but this consideration provides a very different motivation than does correcting the errors that might arise from computing a probability by naive transposition.

Another common argument is that even if the costs of errors are not so disparate, and a moderate significance level such as 1/93 is acceptable, multiple opportunities to find significance make it too easy to find a “significant” result. Such data mining goes on a bit in hunting for new particles. See More on Statistical Reasoning and the Higgs Boson, July 11, 2012. Again, however, this valid reason for being more demanding with regard to a significance level does not flow from the transposition fallacy, the prosecutor’s fallacy, or whatever else it might be called. It applies regardless of the significance level that the experiment is seeking. It is a correction designed to make the declared significance level applicable to the mined data. The corrected level is still subject to the transposition fallacy.

References

• Faye Flam, Lies, Damned Lies and Physics, Dec. 16, 2015,http://www.bloombergview.com/articles/2015-12-30/lies-damned-lies-and-physics
• Dennis Overbye, Physicists in Europe Find Tantalizing Hints of a Mysterious New Particle, N.Y. Times, Dec. 16, 2015, at A18, http://www.nytimes.com/2015/12/16/science/physicists-in-europe-find-tantalizing-hints-of-a-mysterious-new-particle.html?_r=0

What Proves that "the expert and his methods couldn’t possibly be reliable"?

I am developing an allergic reaction to the following kind of argument: "A forensic-science expert testified that a trace at the crime scene or on the victim was associated (to some degree of certainty) with the defendant. DNA later evidence exonerated the defendant. Therefore, the expert’s methods couldn’t possibly be reliable."

This reasoning is not very different from saying that a pitcher who does not strike out every batter couldn’t possibly be a reliable pitcher; that a polling firm that fails to correctly predict every election must be using methods that couldn’t possibly be reliable; or that a test for heart disease that sometimes errs couldn’t possibly be reliable. Without considering the success as well as the failure rate of the method, it is impossible to say that it is unreliable—or, by the same token, that it is reliable (in the sense of being worth relying on). 1/

Yet, the argument from cases of exonerations—lacking any comparison group—are legion in discourse on the use of trace evidence for identification. The latest example I encountered comes from a Washington Post blog site. Two days ago, Radley Balko wrote that

[A] defendant was convicted due to the testimony of a forensic expert who claimed that his “science” showed the defendant, and only the defendant, could have committed the crime. That conviction was later upheld by an appeals court in an opinion that explained in detail why the expert and his methods were legitimate and reliable. The defendant was later exonerated by DNA testing, thus demonstrating that the expert and his methods couldn’t possibly be reliable.

Mr. Balko went on to write that “not only did the courts continue to allow bite mark matching into evidence, every single time a defendant challenged its validity, that defendant lost.” 2/

Both the reasoning and the description of legal history are not quite right. To begin with, even absolute proof of innocence only shows that the test has a nonzero false-positive error rate (no surprise there) and that the witness should not have claimed to a certainty that no one else could have left the mark in question.

To be sure, some evidence that has found breathing space in the courtroom should be squeezed out entirely. But let’s face it—no scientific test meets the standard of perfection. If every case in which evidence that has produced false convictions meant “that the expert and his methods couldn’t possibly be reliable,” there could be no evidence. Overselling has occurred with every type of forensic evidence—from bitemarks to toolmarks to fingerprints to DNA. Courts, scientists, and criminalists should do their best to prevent this. Thus, whether through rules of evidence or through education and monitoring of analysts, testimony must be calibrated to the power of the scientific technique. The testimony should fairly express the known probative value of evidence from a validated method.

An example of testimony that violates this precept comes from Ege v. Yukins. 3/ In that case, a discredited dental expert testified as follows:

Q: Now, Doctor, with regard to your testimony, you indicated that it's highly consistent with the dentition of Defendant Carol Ege; is that correct?
A: Yes.
Q: Okay. With regard to—let me ask you a question. Let's say you have the Detroit Metropolitan Area, three, three and a half million people. Would anybody else within that kind of number match like she did?
A: No, in my expert opinion, nobody else would match up. 4/

The eventual outcome in the case contradicts the assertion that no challenge to the admission of bitemark evidence has succeeded. The state trial judge in Ege realized that the testimony was improper and only “denied [postconviction] relief because of the lack of a contemporaneous objection and a view that the showing of prejudice was insufficient.” 5/

A federal district court also concluded that “expert testimony identifying the petitioner as the only possible perpetrator of the alleged bite mark in the Detroit metropolitan area was improperly admitted.” 6/

The U.S. Court of Appeals for the Sixth Circuit agreed “with the district court that ‘Dr. Warnick's opinion that the petitioner was the only person in the entire Detroit metropolitan area who could have made the mark on the corpse carried an aura of mathematical precision pointing overwhelmingly to the statistical probability of guilt, when the evidence deserved no such credence.’” 7/ It affirmed the order for a new trial.

In short, the argument that a method of forensic identification that has been proved to be fallible is, for that reason alone, inadmissible proves too much. Likewise, the claim that no challenge to bitemark evidence has ever prevailed is exaggerated (although not by much). 8/

Please do not misunderstand me. The series of articles on bitemark evidence from which the remarks I have quoted were taken is impressive and useful. In offering these corrections to two small parts that seem a bit extreme, I am not arguing that bitemark analysis, which has little claim to validity, is either reliable (in the statistical sense that repeated analyses of the same marks give the same answers) or valid (in the sense that the answers are more often correct when marks from the same source are analyzed than when marks from different sources are compared). From the writing I have seen, bitemark analysis does not cut it.

I also believe that cases of false convictions should be studied and that the existence of a given type of scientific evidence in these cases should not be ignored. Finding a large number of false convictions with such evidence present is a warning signal. The evidence may come from a method that has a large false-positive rate, 9/ and that possibility must be investigated to decide whether the evidence should be excluded across the board or whether juries should receive the information -- together with an honest and clear explanation of the uncertainty in the results.

NOTES

1. See infra note 9.
2. Radley Balko, A High-ranking Obama Official Just Called for the “Eradication” of Bite Mark Evidence, The Watch, Wash. Post, July 22, 2015.
3. 485 F.3d 364 (6th Cir. 2007).
4. Ege v. Yukins, 380 F.Supp. 2d 852, 871 (E.D. Mich. 2005), affirmed in part, reversed in part, 485 F.3d 364 (6th Cir. 2007).
5. Id. at 857–58.
6. Id. at 858
7. 485 F.3d at 376.
8. The federal courts in Ege treated the answer to the 3.5 million people as "probability testimony" without questioning Michigan's general rule that bitemark identifications are admissible. A true (and wrongly decided) case of bitemark probability evidence is State v. Garrison, 585 P.2d 563 (Ariz. 1978).
9. The false-positive probability is P(+|O), where + is a positive statement ("the defendant left the mark") and O is the fact that some other person left the mark. Even if this probability is small, a disturbing number of false convictions could involve this evidence. Suppose that P(+|O) = 0.02, that 1,000 tests are performed in a set of cases with marks, and guilty defendants left the marks in 60% of these cases. The expected number of false positives is (0.02)(400) = 8. Assume that the probability of a true positive is P(+|S) = 0.96, where S means that the defendant is the source of the mark. Then the expected number of true positives is (600)(0.96) = 576. If defendants are convicted in all these cases, 8 convictions will be false (assuming that the culprit left the mark), and the many true positives will not be seen in the cases of exonerations of the innocent defendants. As indicated at the outset of these remarks, other data than exonerations are required to judge whether the test is reliable and valid.

DNA Evidence Causes Wrongful Convictions 15% of the Time!?

Today's Forensic Magazine reports that

Recent estimates indicate that as many as 15 of every 100 incarcerated offenders where DNA was an element in their trial may be wrongfully convicted because of misused DNA evidence matching techniques. One common reason for this error is scientifically invalid testimony on forensic evidence. [FULL STORY]

Did I read this correctly? Fifteen percent -- one out of every six or seven -- of the men and women in prison are innocent? That is almost four times the 4% rate estimated in Gross et al. (2013). But that study, the leading one in the field, was limited to defendants with death sentences. Their cases would have been scrutinized especially carefully, so the 4% figure is probably on the low side. Maybe 15% for the broader prison population is realistic.

But wait. Forensic Magazine was not referring to the entire prison population, but only to those cases that went to trial. And within that group, 15% is the figure for defendants for whom "DNA was an element in their trial[s]." Because almost no defendants introduce DNA evidence at trial, these must be cases in which the prosecution has linked the defendant to the crime by DNA testing and the state "misused DNA ... techniques." Can it be that 15% of inmates convicted because of DNA tests are falsely convicted? That is an intolerable error rate for what is supposed to be the gold standard in forensic science. Is it time to halt DNA testing until we can find out why it generates so much "scientifically invalid testimony"?

Or are the editors of Forensic Magazine seeking sensational news rather than reading what they are reporting? Let's look at the "recent estimates." To get to them requires a few steps backwards. Forensic Magazine lists the National Institute of Justice (NIJ) -- a part of the U.S. Department of Justice -- as the source of its story. NIJ funded the Rand Corporation to do a study on improving expert performance in computing a posterior probability (which is not what forensic experts routinely do). The words are Rand's rather than NIJ's. The study cautions that "[o]pinions or points of view expressed are those of the author(s) and do not necessarily reflect the official position or policies of the U.S. Department of Justice." Rand investigated two questions: "Is bias reduced when experts do not know whether the prosecution or defense is the hiring party?," and "Is bias reduced by expert consensus feedback, wherein expertise is culled from multiple sources and those sources examine the majority view to move toward a group consensus?"

The summary at the start of the Rand research report begins with the very two sentences that Forensic Magazine broadcast. The researchers gave no references to indicate where these "recent estimates" came from, but page one of the report supplies an apparent answer. It reads as follows:

As many as 15 of every 100 incarcerated offenders may be wrongfully convicted, according to DNA evidence–matching techniques (Roman et al., 2012). One reason for this is scientifically invalid testimony on forensic evidence (e.g., Gould et al., 2012; Innocence Project, 2013).

This passage reveals that Rand simply misrepresented the "recent estimates" in the first instance. Roman et al. (2012) is an NIJ-funded study from the Urban Institute that involved no cases of wrongful convictions because of misused DNA evidence. Instead, that study "analyzed the results of new DNA testing of old physical evidence from 634 sexual assault and homicide cases that took place in Virginia between 1973 and 1987." In other words, it looked at postconviction exonerations in cases in which the trials took place before DNA evidence was even available. This study of wrongful convictions for reasons other than faulty DNA evidence "found that in five percent of homicide and sexual assault cases DNA testing eliminated the convicted offender as the source of incriminating physical evidence. When sexual assault convictions were isolated, DNA testing eliminated between 8 and 15 percent of convicted offenders and supported exoneration."

Are there any cases of false convictions caused by DNA evidence? Certainly. Does the rate of such convictions approach 15%? Neither Forensic Magazine nor the Rand Corporation offers any reason to believe it.

Frontline's Expose of DNA Testing: Yes and No

The content of a recent Frontline story on “The Surprisingly Imperfect Science of DNA Testing: How a Proven Tool May Be Anything But” will come as no surprise to anyone familiar with the professional and academic literature on forensic DNA identification. The work is important, though, because the main points of the story need to be widely understood. I highlight them below. At the same time, the story relies on some putative problems that are more perceived than real. 

The story boils down to this (with annotations on the right)
Complex DNA mixtures can be tough to interpret, especially when the amounts of DNA are so small that stochastic effects in amplifying DNA sequences are important. In these situations, analysts using a variety of cues and procedures--and even following the same general procedure--can reach different conclusions. False arrests and convictions can follow.	True. See, e.g., Erin Murphy, The Art in the Science of DNA: A Layperson's Guide to the Subjectivity Inherent in Forensic DNA Typing, 58 Emory L.J. 489 (2008).
There can be a big difference between the answer to the question “What is the probability of a match between a specific DNA profile and the profile an individual picked at random?” and “What is the probability of a match between a specific DNA profile and at least one profile in a large database of profiles picked at random?”	Yes, different questions give different answers. The size of the database affects the answer to the second question, but not the first. But which question should be addressed in court? Probably neither. See, e.g., Ian Ayres & Barry Nalebuff, The Rule of Probabilities: A Practical Approach for Applying Bayes’ Rule to the Analysis of DNA Evidence, 67 Stan. L. Rev. 1447 (2015); David J. Balding, The DNA Database Search Controversy, 58 Biometrics 241 (2002); David H. Kaye, Rounding Up the Usual Suspects: A Legal and Logical Analysis of DNA Database Trawls, 87 N. Car. L. Rev. 425 (2009).
Thinking about how traces of DNA ended up where they did—and not just whose DNA ended up there—can be crucial to an investigation and prosecution. If investigators, lawyers, and jurors do not understand this, there can be mistaken arrests, prosecutions, and convictions.	A very important point. See, e.g., Peter Gill, Misleading DNA Evidence (2014); David H. Kaye, David Bernstein & Jennifer Mnookin, The New Wigmore on Evidence: Expert Evidence (2d ed. 2011).
A few things will be surprising
The 2011 Hampikian-Dror “experiment” with a complex DNA mixture was good proof that examiner bias (from knowing what detectives believed) affected two examiner's interpretations.	From the viewpoint of experimental design, a potentially important confounding variable was obviously present. See D.H. Kaye, The Design of “The First Experimental Study Exploring DNA Interpretation”, 52 Science & Justice 256 (2012). As Dror later wrote, the study was  “suggesting that the extraneous context of the criminal case may have influenced the interpretation of the DNA evidence” (emphasis added by Dror in a reply letter). This conclusion is reminiscent of a microscopic hair association reported as “suggesting that the [defendant] may have [left the hair at the crime scene].” How helpful is that?
"It’s not clear how often coincidental matches occur." Indeed, it might not be a rare event at all, considering that "a rogue Arizona state employee had run tests on the state’s database without the FBI’s permission and found" an inexplicably high number of partial matches.	The employee was not a “rogue”; she did not need the FBI’s permission to use a state database this way; the results were presented on behalf of the state laboratory at an International Symposium on Human Identification. They are not especially anomalous, but largely a consequence of trawling for partial matches among all possible pairs of profiles in a database that includes close relatives. See, e.g., David H. Kaye, Trawling DNA Databases for Partial Matches: What Is the FBI Afraid Of?, 19 Cornell J. L. & Public Pol'y 145 (2009).

The Junk DNA Wars

This month, the New York Times' published a report on “the junk DNA wars” asking “Is Most of Our DNA Garbage”? 1/ Readers of the article (and an anonymous follow-up piece on the reactions appearing in science blogs) 2/ would come away thinking that there is a serious debate in the scientific community over the proposition that “junk DNA” is “mostly functional.”

Without defining terms like “functional” and “junk,” however, it is impossible to know what is in dispute and what is not.The follow-up piece is particularly frustrating. It observes that

Some scientists, like T. Ryan Gregory, a evolutionary biologist ... argue that if DNA is mostly functional, then it’s hard to explain why rather humble species, like the onion, have far more DNA than we do. ...

Those who disputed Gregory’s findings [sic — Gregory did not discover the long-standing C-value paradox 3/ ], including supporters of intelligent design, cited the Encode Project, an N.I.H.-sponsored attempt to catalog the functional elements of the genome. Encode scientists found that 80 percent of the genome had “biochemical functions,” suggesting that there was a lot less junk DNA than scientists had thought. But did “biochemical function” really mean anything?

For many scientists, it didn’t. A University of Toronto biochemist, Larry Moran, wrote that “the general public has been snowed by the Encode publicity campaign and by naïve journalists who have enthusiastically reported that junk DNA is dead.”

But the Times' writers did not explain why “many scientists” are not snowed by the 80% statistic. After reading some of the ENCODE papers and the surrounding (typically hyperbolic) publicity, I concluded that:

The ENCODE papers show that 80% of the genome displays signs of certain types of biochemical activity—even though the activity may be insignificant, pointless, or unnecessary. This 80% includes all of the introns, for they are active in the production of pre-mRNA transcripts. But this hardly means that they are regulatory or otherwise functional. Indeed, if one carries the ENCODE definition to its logical extreme, 100% of the genome is functional—for all of it participates in at least one biochemical process—DNA replication.

That the ENCODE project would not adopt the most extreme biochemical definition is understandable—that definition would be useless. But the ENCODE definition is still grossly overinclusive from the standpoint of evolutionary biology. From that perspective, most estimates of the proportion of “functional” DNA are well under 80%. 4/

In short, evolutionary biologists reject "biochemical function" as a criterion for recognizing "junk" because not every bit of biochemical activity affects the reproductive fitness of organisms. (Neither does chemical activity per se show any influence on phenotypes that are related to the healthy functioning of those organisms.) To the evolutionary biologists, the term “junk DNA” means parts of the genome in which the particular DNA sequences (the order of the base pairs) do not have evolutionary significance. The Times article defines “junk DNA” differently, and vaguely, as “pieces of DNA that do nothing for us.” This is not the scientific definition. In fact, the earliest papers on “junk DNA” proposed that much of it might “do something” for us.

The “junk DNA war” (or rather the confusion about the meaning of the term “junk”) has spilled over into the legal realm. A brief that leading genetics and genomics researchers submitted to the U.S. Supreme Court to clarify the privacy implications of forensic DNA typing tried to address it. 5/ These researchers observed that

• In genetics, “junk DNA” denotes sequences that lie outside of genes and that are not under detectable selective pressure: that such DNA exists is not in doubt.
• “Junk” DNA sequences could be biologically useful or interesting yet not be useful for disease diagnosis or prediction.
• ENCODE data do not reveal that anywhere near 80% of the genome contains medically relevant information.
• The ENCODE findings indicate that the system that regulates gene expression is exquisitely complex, but they do little to change the status of “junk DNA” in general.

As far as I know, these conclusions have not been contradicted by new studies, but I have not conducted a recent literature review and would be grateful to hear of relevant papers that undermine these observations.

Notes

1. Carl Zimmer, Is Most of Our DNA Garbage?, N.Y. Times Mag., Mar. 5, 2015 
2. Re: Is Most of Our DNA Garbage?, N.Y. Times Sunday Mag., Mar. 20, 2015
3. See Sean R. Eddy, The C-value Paradox, Junk DNA and ENCODE, 22 Current Biology R898 (2012)
4. David H. Kaye, ENCODE’S “Functional Elements” and the CODIS Loci (Part II. Alice in Genomeland), Forensic Science, Statistics, and the Law, Sept. 18, 2012 (note omitted)
5. Brief of Genetics, Genomics, and Forensic Science Researchers as Amici Curiae in Support of Neither Party, Maryland v. King, No. 12-204, Dec, 28, 2012, reprinted in part in Henry T. Greely & David H. Kaye, A Brief of Genetics, Genomics and Forensic Science Researchers in Maryland v. King, 53 Jurimetrics J. 43 (2013), available at http://ssrn.com/abstract=2403063http://ssrn.com/abstract=2403063. Disclosure statement: I prepared an initial draft of the brief and coordinated the revisions to it.

Up in Smoke: 5 Million Neonatal Blood Samples Incinerated

Originally posted: Double Helix Law Blog, Mar. 15, 2010

An effort to avoid the pointless destruction of the millions of Guthrie cards maintained by the Texas Department of Health Services has come to naught. Plaintiffs who sued the department as well as privacy advocates initially were open to the idea of preserving all the cards with neonatal bloodspots for future research while seeking consent for their storage from millions of parents. [1]

However, this was not to be. After the state quickly settled the dubious lawsuit, an enterprising but inadequately informed journalist published Internet stories alleging that the department had turned “over hundreds of dried blood samples to the federal government to help build a vast DNA database–a forensics tool designed to identify missing persons and crack cold cases” and that the samples “were forwarded along to the federal government to create a vast DNA database, one that could help crack cold cases and identify missing persons.” [2] In her latest installment of this tall tale she continues to write that the samples will “help identify missing persons and crack cold cases.” [1]

The suggestion that the U.S. military is using the samples to build a database to “crack cold cases” or to identify “missing persons” in this country is preposterous. To summarize a previous posting [2]:

First, the research project is limited to mitochondrial DNA, which rarely is used in forensic investigations because it is not capable of providing specific identification. Second, AFDIL does not maintain any databases of DNA profiles to crack cold cases. Third, even if AFDIL were authorized to maintain a database of civilian DNA profiles for criminal investigations, a collection of nameless mtDNA sequences from de-identified samples would be pretty useless. Finally, the true purpose of the research is clear from “Federal MtDNA Paper” posted on the Tribune's website. The AFDIL paper explains that the research database, which cannot be used to identify individuals, simply allows geneticists to put estimates of random-match probabilities for mtDNA on a sounder footing. These estimates are necessary to understand the probative value of an mtDNA match in any criminal investigation or trial. They have nothing in particular to do with cold hits or missing persons.

In sum, the research database has virtually no meaningful privacy implications. Some parents might not want their children’s blood samples used to improve the criminal justice system, but that alone is not much of a reason to destroy what the article calls a medical “treasure trove.” The children’s DNA is not going into any military or law-enforcement database for tracking down missing persons or cracking cold cases.

Yet, this fear apparently was the monkey wrench that jammed the effort to preserve the samples while seeking consent. Here are some excerpts from the latest news as described by the same journalist:

[T]he Department of State Health Services . . . agreed in December to destroy the blood spots, after a civil rights attorney and several Texas parents sued the state for storing them for research purposes without permission. But after the court settlement was signed, privacy advocates lobbied the agency for an alternate solution: a research database that would keep the blood spots intact while seeking electronic consent from parents. They got the go-ahead from some key lawmakers and from the lawsuit’s plaintiffs, who pledged to void the settlement, but not from DSHS.

When The Texas Tribune discovered last month that state health officials had turned hundreds of baby blood spots over to a federal Armed Forces lab between 2003 and 2007 to build a mitochondrial DNA database . . . any chance for saving the blood spots fizzled out. All 5 million blood spots were sent to a Houston-area incinerator last week.

“If there was any way the blood spots were going to be saved, the whole thing fell apart at that point,” said state Sen. Bob Deuell, R-Greenville, . . . “When this came out about these specimens going to the military, I said, ‘We’ve lost this one.’”

. . . State health officials say Austin-based national patient privacy advocate Deborah Peel and Deuell, a physician, approached DSHS Commissioner David Lakey early this year about using electronic consents to save the 5 million existing blood spots from destruction. The agency reviewed the idea but never pursued it. . . .

Critics say . . . DSHS conveniently settled the lawsuit before the trial went to the discovery phase, meaning the documents on the federal DNA study were never disclosed to the plaintiffs. (The Tribune obtained the documents on the federal project — designed to build a forensics tool to help identify missing persons and crack cold cases — through Texas open-records laws.) “Unfortunately, that of course confirmed the plaintiffs’ worst fears,” said Peel, founder of the nonprofit advocacy group Patient Privacy Rights.

Peel said the state’s decision not to seek a non-destructive solution is a shame. . . . “We were going to … reach out to those 5 million families and let them know they had an alternative to having their blood spots destroyed,” Peel said. . . .

Deuell said the impression he got from state health officials was that they feared they would be subject to litigation from other parents if they negotiated with the plaintiffs not to destroy the blood spots. . . . “They said, ‘The plaintiffs are just three people out of 5 million. Who’s to say somebody else wouldn’t come back and file a new suit?’”

Harrington [plaintiffs' attorney] said that worry is “utter nonsense.” He said both sides could have gone back to the judge to have a new settlement drafted — one that would’ve protected the agency. “What’s the harm in that?” Harrington asked. “We would have supplemented or amended the settlement. It would have been totally possible.”

But once news broke that some of the blood spots had been turned over to the federal lab — and that the state had no intention of destroying those samples — the plaintiffs’ offer was off the table. Instead, they have demanded that the state get the blood spots back from the federal government, or they’ll file another lawsuit. . . . [1]

Maybe another lawsuit would be a good thing. With competent lawyering and journalism, the people of Texas finally might realize that none of their children’s DNA has found its way into any DNA database for identifying anyone.

References

1. Emily Ramshaw, DNA Destruction, Tex. Tribune, http://www.texastribune.org/stories/2010/mar/09/blood-drive/
2. David H. Kaye, A Texas Tall Tale of “DNA Deception,” Double Helix Law, Mar. 4, 2010

A Texas Tall Tale of “DNA Deception”

Originally posted: Double Helix Law Blog, Mar. 4, 2010

A “non-profit, nonpartisan public media organization,” the Texas Tribune broke a remarkable story. The story goes like this. Texas, like every other state, pricks the heels of new born children for a blood sample. It screens these samples for rare, metabolic genetic diseases and stores spots of blood on a card for each child. As the March of Dimes explains, “[w]hen test results show that the baby has a birth defect, early diagnosis and treatment can make the difference between lifelong disabilities and healthy development.” [1]

As these “Guthrie cards” began to accumulate, it became clear that they might be useful for medical research. In 1994, law professor Jean McEwen and doctor-lawyer Phil Reilly called them “inchoate databases” and found that many laboratories were open to the idea of sharing them — in anonymized form — for research that would benefit the public. [2]

The Texas State Department of Health Services did exactly this. It provided medical researchers with de-identified Guthrie cards to study “the gene involved in club foot, to inspect the DNA of infants who develop childhood cancer, [and] to examine prenatal lead exposure.” [3] For its efforts, the department was sued. It had treated the cards as free for the taking, without going back to parents to obtain explicit permission to release their child’s blood spots (with no name attached). Although it is a huge jump from any case law, and even though the legally cognizable damages suffered by any parent whose child’s nameless blood spot made its way to a laboratory are obscure, five plaintiffs alleged violations of the protection of the Fourth Amendment, the Texas Constitution, and the common law. On their behalf and seeking to represent a much larger class of plaintiffs, the Texas Civil Rights Project sought declaratory and injunctive relief. [4]

The case promptly settled. The state agreed to destroy millions of cards, to give parents clearer procedures to opt out of the storage of the cards, and to pay $26,000 in attorneys fees and costs.

There things might have stayed — but for a journalist’s “review of nine years’ worth of e-mails and internal documents on the Department of State Health Services’ newborn blood screening program.” [3] She found that the state had concealed its involvement in a nefarious and far-reaching military or law-enforcement project. The Texas doctors had turned “over hundreds of dried blood samples to the federal government to help build a vast DNA database — a forensics tool designed to identify missing persons and crack cold cases.” [3] The samples, she repeated, “were forwarded along to the federal government to create a vast DNA database, one that could help crack cold cases and identify missing persons.” [5] The database would be shared worldwide, “for international law enforcement and investigation in the context of homeland security and anti-terrorism efforts.” [3]

Incensed, the lawyer for the five plaintiffs fired off a letter to the governor and the attorney-general. He accused the “TDSHS [of] supplying those blood samples taken from newborn babies to the military, not just for research, but so that the military can build a mitochondria DNA data base, which can be used in part for law enforcement purposes.” [5] He complained that “[t]his … alarming development … raises the specter of the federal government building an international DNA data base,” and he demanded that “within ten (10) days of this letter, you retrieve from the federal government all the blood samples that Texas has sent to the U.S. military and retrieve and destroy all information taken from those samples … .” [5] Indeed, he suddenly realized that this military project was why the state was so willing to settle the case: “‘Sometimes there are slam-dunk cases, but I’d never seen this kind of case settle without discovery,’ says [Jim] Harrington, director of the Texas Civil Rights Project. ‘This explains the mystery of why they gave up so fast.’” [3]

The trouble is that it’s all smoke and no fire. The reporter and the lawyer apparently misread the report of the Armed Forces DNA Identification Laboratory (AFDIL) detailing its efforts to collect and study mitochondrial DNA (mtDNA) from varied people and places. As explained in Chapters 11 of The Double Helix and the Law of Evidence, AFDIL is a world leader in mitochondrial DNA sequencing because the technique is exceedingly valuable in identifying the remains of soldiers missing in action. [6] But mtDNA is not used to “crack cold cases,” at least not by generating cold hits in any law-enforcement database of DNA profiles from possible offenders. The national database (NDIS) maintained by the FBI — the one that actually helps in cracking cold cases — is limited to checking crime-scene samples against STR profiles in the DNA from the cell nucleus. These DNA sequences are wonderful for discriminating among individuals. When a 13-locus match from a crime-scene to one of the more than seven million profiles in NDIS pops up, it can constitute a practically conclusive identification to a known individual. [6] And, the bigger NDIS is, the more likely it is that the culprit will be in it. This kind of database is “only as valuable as its … size.” [3]

Not so with mtDNA. Everyone in the same maternal line shares the same sequence, and other essentially unrelated maternal lineages might have the same sequences. [6] Moreover, it would be inane to put anonymous sequences — nuclear or mitochondrial — into the database used in searching for cold hits. A hit from a crime-scene sample to a profile from a Guthrie card with no name linked to it would have little or no investigative value. The (nameless) Texas children need not fear being swept up in criminal or terrorist investigations because AFDIL sequenced their anonymous DNA.

But if the federal government does not want the samples for a database that will be used to catch criminals or terrorists, what nefarious international database are these profiles going into? Prosaically, they are part of a scientific, population-genetics database that will be helpful in understanding the significance of a match in an ordinary criminal case. Consider State v. Ware, the very first case with mtDNA evidence. Hairs were found in the bed where a young girl was attacked. [6, chap. 12] The hairs looked similar to the defendant’s under a microscope, but there have been false convictions with hairs that happen to look similar. (Just check with the Innocence Project.) Nuclear DNA, which could yield well-nigh conclusive results, were absent in the hair shafts, but there were enough mitochondria to get a useful sequence, and this sequence matched the defendant’s. [6]

Because mtDNA just does not have the power of nuclear DNA to differentiate among individuals, however, defense counsel in such cases can object (appropriately) that the evidence is confusing or misleading without statistics on how rare the mitotype in question would be in the general population. How many people would be falsely incriminated by the mtDNA sequence in the case?

By understanding the variations in the mtDNA sequences in different places and populations, scientists can estimate how rare or how common a mitotype that incriminates a suspect might be. Such estimates require reference databases, but the existing forensic-statistical-reference databases, defense counsel and a number of scientists have argued, are too small and full of gaps in the population groups represented. [6] Indeed, the federal government has received considerable flak from the media and a vocal group of scientists, lawyers, and sundry others for its refusal to supply de-identified nuclear-DNA profiles from law-enforcement databases for new studies to supplement the existing statistical-reference databases long used to estimate the probability of random STR-profile matches in criminal cases. [8, 9]

In sum, the AFDIL study is a response to a legitimate scientific and legal concern. The federal government (as it should) wants to improve the infrastructure for using mtDNA evidence in court by enlarging the statistical-reference databases. Thus, the AFDIL report — the supposed smoking gun posted on the Tribune's website — is entitled “Development and Expansion of High-quality Control Region Databases to Improve Forensic mtDNA Evidence Interpretation.” As the title indicates, these scientific databases do not generate DNA evidence. They “improve” the “interpretation” of mtDNA evidence from other sources. The very first sentence of the report makes it plain that the databases are for statistical purposes only:

Mitochondrial DNA testing in the forensic context requires appropriate, high-quality population databases for estimating the rarity of questioned haplotypes. However, large forensic mtDNA databases, which adhere to strict guidelines in terms of their generation and maintenance, are not widely available for many regional populations of the United States or most global populations outside of the United States and Western Europe. [7]

After elaborating, the report continues:

In order to address this issue, the Armed Forces DNA Identification Lab (AFDIL) has undertaken a high-throughput control region databasing effort. … Global populations that are currently underrepresented in available forensic mtDNA databases will comprise approximately 25% of the total number of samples. The remaining individuals will represent regional samples of various U.S. populations and global populations that contribute to the overall mtDNA diversity of the U.S. The high-quality mtDNA data generated from these efforts will be publicly available to permit examination of regional mtDNA substructure and admixture, and ultimately to improve our ability to interpret mtDNA evidence. [7]

This population-genetics study is entirely different from building a huge database of mitotypes to generate cold hits. MtDNA does not work well for this purpose, and even if the FBI wanted to do it, anonymous data from AFDIL would be useless. All that those data can do is help investigators, judges and juries better assess the results of a match to a known suspect or defendant. Suggestions that neonatal samples are being put into databases that could result in the unknowing “donors” being swept up in future investigations of crime or terrorism are troubling — but not because they are true.

References

[1] March of Dimes, Newborn Screening Tests, Mar. 2008, http://www.marchofdimes.com/pnhec/298_834.asp
[2] J. E. McEwen & P. R. Reilly, Stored Guthrie Cards as DNA “Banks,” 55 Am. J. Human Genetics 196-200 (1994), available at http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1918213/
[3] Emily Ramshaw, DNA Deception, Texas Tribune, Feb. 22, 2010, available at http://www.texastribune.org/stories/2010/feb/22/dna-deception/, last viewed, March 2, 2010
[4] Beleno v. State Dep’t of Health Serv., Civ. No. SA09CA1088 (W.D. Tex. Mar. 12, 2009) (complaint)
[5] Emily Ramshaw, TribBlog: AG’s Office Fires Back at Blood Spot Attorney, Feb. 22, 2010, available at http://www.texastribune.org/blogs/post/2010/feb/22/tribblog-attorney-asks-perry-get-dna-back-feds/, last viewed, March 2, 2010
[6] David H. Kaye, The Double Helix and the Law of Evidence (2010)
[7] Jodi A. Irwin et al., Development and Expansion of High-quality Control Region Databases to Improve Forensic mtDNA Evidence Interpretation, 1 Forensic Sci. Int’l: Genetics 154-157 (2007)
[8] David H. Kaye, Trawling DNA Databases for Partial Matches: What Is the FBI Afraid Of?, 19 Cornell J. L. & Public Pol’y 145-171 (2009)
[9] D. E. Krane et al., Time for DNA Disclosure, 326 Science 1631-1632 (2009), DOI: 10.1126/science.326.5960.1631

Related posting:  A Texas Tall Tale of "DNA Deception", July 26, 2014