The Transposition Fallacy in Matrixx Initiatives, Inc. v. Siracusano: Part II

The previous posting promised a simple example that would demonstrate the fallacy in claims such this one:

For a p-value of .09, the odds of observing the AER [adverse event report] is 91 percent divided by 9 percent. Put differently, there are 10-to-1 odds that the adverse effect is “real” (or about a 1 in 10 chance that it is not).

Brief of Amici Curiae Statistics Experts Professors Deirdre N. McCloskey and Stephen T. Ziliak in Support of Respondents, Matrixx Initiatives, Inc. v. Siracusano, 131 S.Ct. 1309 (2011) (No. 09-1156).

Here is one such example. A bag contains 100 coins. One of them is a trick coin with tails on both sides; the other 99 are biased coins that have a 0.3 chance of coming up tails and a 0.7 chance of coming up heads. I pick one of these coins at random and flip it twice, obtaining two tails. On the basis of only this sample data (the two tails), you must decide which type of coin I picked. The p-value with respect to the “null hypothesis” (N) that the coin is a normal (albeit biased) heads-tails one is the probability of seeing two tails in the two tosses: p = 0.3 x 0.3 = 0.09. Should you reject the null hypothesis N and conclude that I flipped the unique tails-tails coin? Are the odds for this alternative hypothesis (A) 10:1, as the brief of the statistical experts asserts?

Of course not. Just consider repeating this game over and over. Ninety-nine percent of the time, you would expect me to pick a heads-tails coin. In 9% of those cases, you expect me to get tails-tails on the two tosses (9% x 99% = 8.91%). The other way to get tails-tails on the tosses is to pick the tails-tails coin. You expect this to happen about 1% of the time. Thus, the odds of the tails-tails coin given the data on the outcome of the tosses are 1% to 8.91% = 1:8.91, which is about 1:9. Despite the allegedly significant (in “practical, human, or economic” terms) p-value of 0.09, the alternative hypothesis remains improbable.

A more formal derivation uses Bayes' rule for computing posterior odds. Let tt be the event that the coin I picked produced the two tails when tossed (the data), and let "|" stand for "given that" or "conditioned on." Then Bayes' rule reveals that

Odds(A|tt) = L x Odds(A),
where L is the "likelihood ratio" given by P(tt|A) / P(tt|N) and Odds(A) are the odds prior to flipping the coin. The value of L is 1/.09 = 100/9. Hence,

Odds(A|tt) = (100/9) Odds(A).
Because there is only 1 trick coin and 99 normal coins in the bag, the prior odds of A are Odds(A) = 1:99. Hence, the posterior odds are Odds(A|tt) = (100/9)(1/99) = 100/891 = 1:8.91. In other words, the odds for the alternative hypothesis are only about 1:9 -- practically the opposite of the 10:1 odds quoted in the statistics experts' brief.

The lesson of this example is not that a statistic with a p-value of 0.9 always can be safely ignored. It is that the p-value, by itself, cannot be converted into a probability that the alternative hypothesis is true (“that the adverse effect is ‘real’”). Knowing that the two tails arise only 9% of the time when the head-tails coin is the cause does not imply that 9% is the probability that a heads-tails coin is the cause or that 91% is the probability that the tails-tails coin is the “real” cause. Statisticians have warned against this confusion of a p-value with a posterior probability time and again. The brief of "Amici Curiae Statistics Experts" thus brings to mind the old remark, "With friends like these, who needs enemies?" A more complete review of the brief is available at Nathan Schachtman's website (see Further Readings).

Further Reading

David H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence (2d ed. 2011).

Nathan A. Schachtman, The Matrixx Oversold, Apr. 4, 2011, http://schachtmanlaw.com/the-matrixx-oversold/

The Transposition Fallacy in Matrixx Initiatives, Inc. v. Siracusano: Part I

One might expect to hear phrases like “Not statistical significance there!” and “There is no way that anybody would tell you that these ten cases are statistically significant” hurled by a disgruntled professor at an underperforming statistics student. Yet, in January 2011, they came from the Supreme Court bench during the argument in Matrixx Initiatives, Inc. v. Siracusano.[1]

The issue before the Court was “[w]hether a plaintiff can state a claim under § 10(b) of the Securities Exchange Act and SEC Rule 10b-5 based on a pharmaceutical company's nondisclosure of adverse event reports even though the reports are not alleged to be statistically significant.” [2] In the case, the manufacturer of the Zicam nasal spray for colds issued reassuring press releases at a time when it was receiving case reports from physicians of loss of smell (anosmia) in Zicam users. The pharmaceutical company, Matrixx Initiatives, succeeded in getting a security fraud class action dismissed on the ground that the plaintiffs failed to plead “statistical significance.”

Because case reports are just a series of anecdotes, it is not immediately obvious how they could be statistically significant, but a determined statistician could compare the number of reports in the relevant time period to the number that would be expected under some model of the world in which Zicam is neither a cause nor a correlate of anosmia. If the observed number departed from the expected number by a large enough amount—one that would occur no more than about 5% of the time when the assumption of no association is true (along with all the other features of the model)—then the observed number would be statistically significant at the 0.05 level.

The Court rejected any rule that would require securities-fraud plaintiffs to engage in such statistical modeling or computation before filing a complaint. This result makes sense because a reasonable investor might want to know about case reports that do not cross the line for significance. Such anecdotal evidence could be an impetus for further research, FDA action, or product liability claims—any of which could affect the value of the stock. In rejecting a bright-line rule of p < 0.05, the Court made several peculiar statements about statistical significance and the design of studies, but these are not my subject for today. (An older posting, on March 25, has some comments on this issue.)

Instead, I want to look at a small part of an amicus brief from “statistics experts” filed on behalf of the plaintiffs. There is much in this brief, which really comes from two economists (or perhaps these eclectic scholars should be designated historians or philosophers of economics and statistics), with which I would agree (for whatever my agreement is worth). But I was shocked to find the following text in the “Brief of Amici Curiae Statistics Experts Professors Deirdre N. McCloskey and Stephen T. Ziliak in Support of Respondents”:

The 5 percent significance rule insists on 19 to 1 odds that the measured effect is real.²⁶ There is, however, a practical need to keep wide latitude in the odds of uncovering a real effect, which would therefore eschew any bright-line standard of significance. Suppose that a p-value for a particular test comes in at 9 percent. Should this p-value be considered “insignificant” in practical, human, or economic terms? We respectfully answer, “No.” For a p-value of .09, the odds of observing the AER [adverse event report] is 91 percent divided by 9 percent. Put differently, there are 10-to-1 odds that the adverse effect is “real” (or about a 1 in 10 chance that it is not). Odds of 10-to-1 certainly deserve the attention of responsible parties if the effect in question is a terrible event. Sometimes odds as low as, say, 1.5-to-1 might be relevant.²⁷ For example, in the case of the Space Shuttle Challenger disaster, the odds were thought to be extremely low that its O-rings would fail. Moreover, the Vioxx matter discussed above provides an additional example. There, the p-value in question was roughly 0.2,²⁸ which equates to odds of 4 to 1 that the measured effect — that is, that Vioxx resulted in increased risk of heart-related adverse events — was real. The study in question rejected these odds as insignificant, a decision that was proven to be incorrect.

26. At a 5 percent p-value, the probability that the measured effect is “real” is 95 percent, whereas the probability that it is false is 5 percent. Therefore, 95 / 5 equals 19, meaning that the odds of finding a “real” effect are 19 to 1.

27. Odds of 1.5 to 1 correspond to a p-value of 0.4. That is, the odds of the measured effect being real would be 0.6 / 0.4, or 1.5 to 1.

28. Lisse et al., supra note 14, at 543-44.

Why is this explanation out of whack? The fundamental problem is that, within the framework of classical (Neyman-Pearson) hypothesis testing, hypotheses like “the adverse effect is real” or “a measured effect being real” do not have odds or probabilities attached to them. In Bayesian inference, statements like “the probability that the measured effect is ‘real’ is 95 percent, whereas the probability that it is false is 5 percent” are meaningful, but frequentist p-values play no role in that framework. Equating the p-value with the probability that a null hypothesis is true and regarding the complement of a p-value as the probability that the alternative hypothesis is true (that something is “real”) is known as the transposition fallacy. [2] That two “statistics experts” would rely on this crude reasoning to make an otherwise reasonable point is depressing.

The preceding paragraph is a little technical. Soon, I shall post a simple example that should make the point more concretely and with less jargon.

References

1. Transcript of Oral Argument, Matrixx Initiatives, Inc. v. Siracusano, 131 S.Ct. 1309 (2011) (No. 09-1156), 2011 WL 65028, at *12 & *16 (Kagan, J.).

2. Petition for Writ of Certiorari at i, Matrixx Initiatives, Inc. v. Siracusano, 131 S.Ct. 1309 (2011) (No. 09-1156), 2010 WL 1063936.

3. David H. Kaye, David E. Bernstein & Jennifer L. Mnookin, The New Wigmore: A Treatise on Evidence: Expert Evidence (2d ed. 2011).

Third Circuit Upholds Federal Arrestee DNA Database Law

In 2009, the United States District Court for the Western District of Pennsylvania made legal history. For the first time, a federal court held that the government lacked the constitutional power to compel individuals who had been arrested and charged with a crime to provide a DNA sample. Today, the Court of Appeals for the Third Circuit (one of the 12 appellate courts that sit one rung below the Supreme Court in the federal judicial system) reversed this ruling. Yet, both courts applied the "totality of the circumstances" standard for ascertaining the reasonableness of searches and seizures. What, then, accounts for the anticlinal outcomes?

Basically, the two courts took very different views of the individual's Fourth Amendment interests and their role in evaluating the legislation. The following passages from the district court opinion and the opinion of the majority of the court of appeals are illustrative.

• District Court: [T]he search in this instance is one that reveals the most intimate details of an individual's genetic condition, implicating compelling and fundamental “interests in human dignity and privacy. See Schmerber v. California, 384 U.S. 757, 770 (1966).
• Court of Appeals: Schmerber recognized society‘s judgment that
  blood tests "do not constitute an unduly extensive imposition on an individual‘s personal privacy and bodily integrity."
• District Court: [T]o compare the fingerprinting process and the resulting identification information obtained therefrom with DNA profiling is pure folly. Such oversimplification ignores the complex, comprehensive, inherently private information contained in a DNA sample.
• Court of Appeals: While we acknowledge the seriousness of Mitchell‘s concerns about the possible misuse and future use of DNA samples, we conclude that these hypothetical possibilities are unsupported by the record before us and thus do not have any substantial weight in our totality of the circumstances analysis.

The district court was not reassured by the fact that DNA identification profiling currently is little more than a token of personal identity. On the basis of a student law review article, it feared that "DNA samples may reveal private information regarding familial lineage and predisposition to over four thousand types of genetic conditions and diseases; they may also identify genetic markers for traits including aggression, sexual orientation, substance addiction, and criminal tendencies."

The majority of the en banc court of appeals was less fearful that the government would change its use of the samples to go beyond the current production and trawling of identification profiles. It observed that "[t]he judiciary risks error by elaborating too fully on the Fourth Amendment implications of emerging technology before its role in society has become clear. ... At this juncture, ... we consider the amount and type of personal information to be contained in the DNA profile to be nominal."

Thus, the district court saw the retained samples as a potentially rich source of private information about the arrestee that the government might exploit some day (although it did not explain why the government would be interested in performing genetic tests for 4,000 or more medical conditions). The court of appeals was content to uphold the status quo: "As currently structured and implemented . . . the DNA Act‘s compulsory profiling of qualified federal offenders can only be described as minimally invasive--both in terms of the bodily intrusion it occasions, and the information it lawfully produces."

Similar cases are pending before the Second and Ninth Circuits. I predict that both will uphold the federal law -- over some vigorous dissents. When will the Supreme Court step in?

References

United States v. Mitchell, 681 F.Supp.2d 597 (W.D. Pa. 2009)

-United States v. Mitchell, No. 02-2859 (3d Cir. Aug. 25, 2011) (en banc)

Cross-posted from the Double Helix Law Blog

Junk Science in United States v. Pool

Having granted en banc review in United States v. Pool, 621 F.3d 1213 (9th Cir. 2010), the U.S. Court of Appeals for the Ninth Judicial Circuit is likely to produce as wild a set of conflicting opinions on DNA databases as it did in United States v. Kincade, 379 F.3d 813 (9th Cir. 2004).

The panel that heard the Pool case divided 2-1 and generated 3 opinions. Judge Callahan wrote an opinion upholding the federal law on taking DNA after an arrest. Visiting Judge Lucero (of the Tenth Circuit) joined in this opinion, but he also wrote a separate opinion. Judge Schroeder dissented.

The briefs that informed these three opinions left something to be desired. Here, I'll focus on one of my pet peeves--disingenuous or inane claims about the CODIS STR loci as a threat to privacy.

Appellant's Opening brief (available from a link on EPIC's website, along with a one-sided list of vaguely related articles) is rather shameless in this regard. It starts with the claim that "DNA profiles derived by STR may yield probabilistic evidence of the contributor’s race or sex." [1] Probabilistic evidence of sex from autosomal STRs? The arresting officers or jailers need a genetic test for that?

Then the brief cites Simon Cole's writing to support its sweeping statement that "scientific studies have debunked the notion that these regions of the genetic code are devoid of any biological function." Yet, the brief cites no study that "debunks" the notion that the length polymorphisms of the CODIS tetranucleotide STRs lack "biological function." The concurring opinion of Judge Lucero recognizes that Cole rejects the claim of functionality (for the moment). [2] However, a group in France has a theory and some data for a mechanism through which one such STR could regulate the expression of an enzyme. [3]

Finally, the brief proposes that the "specter of discrimination and stigma could arise where one or more STRs is found to correlate with another genetic marker whose function is known, so that the presence of the seemingly innocuous STR serves as a 'flag' for that genetic predisposition or trait." [4] An accompanying footnote gives this example: "A study in England from 2000 found that one of the markers used in DNA identification is closely related to the gene that codes for insulin, which itself relates to diabetes." [5]

The accused STR is TH01. It has been used in many studies investigating the association between (a) SNPs, VNTRs, and this STR in a complex of genes and (b) a large number of diseases. Unsurprisingly, associations have been observed. Some of the reported associations were spurious and were not replicated. Other associations probably are real. This does not mean that TH01, by itself, is a useful predictor of any of these diseases in a given population. In fact, one forensic biologist used the 2000 paper cited in Pool's brief to show that "such associations [between forensic STRs and disease-causing alleles in genes] are so ridiculously weak that serious protest could never form." [6] His explanation follows:

This is illustrated well by the possible association between certain alleles of an STR named TH01 and diabetes type 1 (Bennett and Todd, 1996; Stead et al., 2000). TH01 alleles are used routinely in DNA typing, and for a minute, the manufacturers of genetic fingerprint kits started to feel the heat over the possible association between an exonic illness and an intronic allele. Fortunately, it takes just a pen and a piece of paper to brush off possible concerns: four out of 1000 Europeans will eventually get diabetes type 1. If you carry one of the ‘risk’ alleles in the intronic TH01 region, your chances of getting diabetes type 1 is 0.13 out of 1000. If I find out that you are carrying the alleged risk allele in my laboratory during DNA typing, I could—but I am not allowed to—calculate your total risk for diabetes as 0.4 × 1.3 = 0.52%. In plain language: in the worst case scenario, one allele of your possible genetic fingerprint might tell me that your general risk of getting diabetes type 1 is increased from 0.4 to 0.52%. All other alleles will not tell me anything about you, or your potential risk for illnesses. Abuse of such information is impossible because it simply has no practical predictive value.

I do not want to "brush off possible concerns," and I understand the pressures and temptations of advocacy. Still, I wonder whether the Sacramento Federal Defender consulted the scientific literature on TH01 before citing an old article. Or whether he knew that the claims in the law review essay cited in the brief were the subject of an extensive rejoinder in the same journal. [7] If he did, he choose not to share this fact with the court. To my mind, that is not good advocacy.

Notes

1. Brief at 12 (quoting from a plurality opinion in Kincade).
2. 621 F.3d at 1230.
3. See Rolando Meloni, Post-genomic Era and Gene Discovery for Psychiatric Diseases: There Is a New Art of the Trade? The Example of the HUMTH01 Microsatellite in the Tyrosine Hydroxylase Gene, 26 Molecular Neurobiology 389 (2001).
4. Brief at 12.
5. Id. at 12 n.8.
6. Mark Benecke, Coding orNon-coding?, That Is the Question, 3 European Molecular Biology Organization Reports 498 (2002).
7. David H. Kaye, Please, Let's Bury the Junk: The CODIS Loci and the Revelation of Private Information, 102 Nw. U. L. Rev. Colloquy 70 (2007).

Crossposted from the Double Helix Law Blog

Confusing DNA Samples with DNA Profiles?

It is tough for lawyers to get science right. I say this not to denigrate lawyers—I am one myself—but to stress the importance of taking the time and effort to communicate the scientific facts clearly so that the value judgments are persuasive. An article attacking the constitutionality of an Arkansas law on DNA sampling from arrestees illustrates this point. In “Step Out of the Car: License, Registration, and DNA Please,” Associate Professor Brian Gallini of the University of Arkansas School of Law, gives an account of DNA profiling that makes it appear that the process of forensic DNA profiling reveals “the totality of a person’s genetic makeup” to arrive at an identification profile. At least, that is how the following exposition of DNA profiling for identification could be read:

[E]ven the layperson knows that taking a DNA sample requires an intrusion into the body, which thereafter reveals the totality of a person’s genetic makeup. ... Although courts have characterized DNA swabs as only “minimally intrusive,” they do so without recognizing ... the intrusion upon the arrestee’s interest in keeping the information revealed by a DNA sample private. From a buccal swab, the state obtains an analyzable sample of an arrestee’s DNA. That, in turn, allows the state to perform a polymerase chain reaction procedure (PCR), which involves replicating the DNA sample. This replication then allows the tester to look at “short tandem repeats” (STR). At this stage, the STRs reveal specific areas of DNA known as “loci.” In total, the tester is looking to isolate thirteen different loci in order to identify an individual’s exact genetic makeup. Once complete, the sample potentially “provides the instructions for all human characteristics, from eye color to height to blood type.”

What is wrong with this picture? Let me count the ways:

1. PCR does not replicate the DNA sample. Human cells can replicate the full nuclear genome, but PCR can only replicate short stretches of DNA from targeted locations—the loci.
   
2. Replication itself does not allow the tester to look at STRs. Visualization or ascertainment comes later.
   
3. STRs do not “reveal specific areas of DNA known as ‘loci.’” An STR is a certain type of DNA sequence that occurs at, well, an STR locus. PCR primers used in forensic identification amplify only the sequences at these loci. The rest of the genome remains terra incognito.
   
4. The tester is not seeking “to identify an individual’s exact genetic makeup.” Rather, the laboratory is seeking to ascertain a small number of variations that are not in genes (or not in the exons of genes).
   
5. The physical sample was complete before it was typed. “Once complete,” the tiny profile cannot possibly “provide[] the instructions for all human characteristics, from eye color to height to blood type.” The STR typing never gives any instructions for phenotypes.
   

Do these corrections mean that samples could not be used to gain information about human phenotypes such as eye color? Of course not. Eye color is a phenotype that can be deduced (in some instances) from genotyping. But such genotyping is not STR profiling.

And how much would it invade your privacy if a laboratory technician were to figure out your eye color in this roundabout way—instead of looking you in the eye? But that’s another story, and I have argued elsewhere against indefinite sample retention.

Reference

Brian Gallini, Step Out of the Car: License, Registration, and DNA Please, 62 Ark. L. Rev. 475 (2009)

Crossposted from the Double Helix Law blog.