For a p-value of .09, the odds of observing the AER [adverse event report] is 91 percent divided by 9 percent. Put differently, there are 10-to-1 odds that the adverse effect is “real” (or about a 1 in 10 chance that it is not).Brief of Amici Curiae Statistics Experts Professors Deirdre N. McCloskey and Stephen T. Ziliak in Support of Respondents, Matrixx Initiatives, Inc. v. Siracusano, 131 S.Ct. 1309 (2011) (No. 09-1156).
Here is one such example. A bag contains 100 coins. One of them is a trick coin with tails on both sides; the other 99 are biased coins that have a 0.3 chance of coming up tails and a 0.7 chance of coming up heads. I pick one of these coins at random and flip it twice, obtaining two tails. On the basis of only this sample data (the two tails), you must decide which type of coin I picked. The p-value with respect to the “null hypothesis” (N) that the coin is a normal (albeit biased) heads-tails one is the probability of seeing two tails in the two tosses: p = 0.3 x 0.3 = 0.09. Should you reject the null hypothesis N and conclude that I flipped the unique tails-tails coin? Are the odds for this alternative hypothesis (A) 10:1, as the brief of the statistical experts asserts?
Of course not. Just consider repeating this game over and over. Ninety-nine percent of the time, you would expect me to pick a heads-tails coin. In 9% of those cases, you expect me to get tails-tails on the two tosses (9% x 99% = 8.91%). The other way to get tails-tails on the tosses is to pick the tails-tails coin. You expect this to happen about 1% of the time. Thus, the odds of the tails-tails coin given the data on the outcome of the tosses are 1% to 8.91% = 1:8.91, which is about 1:9. Despite the allegedly significant (in “practical, human, or economic” terms) p-value of 0.09, the alternative hypothesis remains improbable.
A more formal derivation uses Bayes' rule for computing posterior odds. Let tt be the event that the coin I picked produced the two tails when tossed (the data), and let "|" stand for "given that" or "conditioned on." Then Bayes' rule reveals that
where L is the "likelihood ratio" given by P(tt|A) / P(tt|N) and Odds(A) are the odds prior to flipping the coin. The value of L is 1/.09 = 100/9. Hence,
Because there is only 1 trick coin and 99 normal coins in the bag, the prior odds of A are Odds(A) = 1:99. Hence, the posterior odds are Odds(A|tt) = (100/9)(1/99) = 100/891 = 1:8.91. In other words, the odds for the alternative hypothesis are only about 1:9 -- practically the opposite of the 10:1 odds quoted in the statistics experts' brief.
The lesson of this example is not that a statistic with a p-value of 0.9 always can be safely ignored. It is that the p-value, by itself, cannot be converted into a probability that the alternative hypothesis is true (“that the adverse effect is ‘real’”). Knowing that the two tails arise only 9% of the time when the head-tails coin is the cause does not imply that 9% is the probability that a heads-tails coin is the cause or that 91% is the probability that the tails-tails coin is the “real” cause. Statisticians have warned against this confusion of a p-value with a posterior probability time and again. The brief of "Amici Curiae Statistics Experts" thus brings to mind the old remark, "With friends like these, who needs enemies?" A more complete review of the brief is available at Nathan Schachtman's website (see Further Readings).
Further Reading
David H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence (2d ed. 2011).
Nathan A. Schachtman, The Matrixx Oversold, Apr. 4, 2011, http://schachtmanlaw.com/the-matrixx-oversold/
No comments:
Post a Comment