The Mattrix: Missing the Difference Between Statistical Significance and Study Design

This week, the Supreme Court released a unanimous opinion on the need for "a statistically significant number" in a securities fraud complaint. In a rare display of solidarity, there were no separate opinions to accompany Justice Sotomayer's opinion for the Court in Matrixx Initiatives, Inc. v. Siracusano, No. 09-1156, 2011 WL 977060 (U.S. Mar. 22, 2011).

The issue was whether a complaint for securities fraud under federal law could be based on the failure of a manufacturer of a homeopathic remedy for colds to disclose "adverse events associated with a product if the reports do not disclose a statistically significant number of adverse events." Adverse event reports (AERs) are reports from consumers or physicians sent to the FDA. The adverse events here were loss of smell (anosmia) following the use of the nasal spray or gel marketed as the Zicam Cold Remedy.

I have no quarrel with the result -- a reasonable investor might want to know of such reports if they are sufficiently extensive and disturbing that they could prompt the FDA to take some action or might lead to costly lawsuits. Of course, the fact that investors might feel this way does not mean that AERs provide valid statistical proof of causation, and the Court cautioned that "we do not attempt to define here what constitutes reliable evidence of causation." Nevertheless, the Court offered some remarks on this latter topic that are not entirely lucid.

The problem with AERs is that they are "anecdotal" -- a series of little stories of event X (e.g., use of the remedy) followed by event Y (e.g., anosmia). At best, such anecdotes can establish an association between X and Y, but even this requires a comparison group. After all, some people who do not use the Mattrix product also suffer a loss of smell. If the condition occurs no more often among users of ZiCam (or similar nostrums) than among people who do not use it, then the number of AERs does not prove that ZiCam is associated with -- let alone causes -- asnomia.

This is where statistical significance comes in. One has to consider, not the sheer number of AERs, but a statistic such as the difference between the proportion of ZiCam users who develop asnomia and the proportion of nonusers who come down with it. If one posits that each group is a random sample of some larger population, then the probability of observing a difference of this size or larger can be computed. If this probability is small, then the AERs are statistically significant evidence that something other than chance is at work.

That "something" might not be ZiCam at all, but some other factor also associated with ZiCam use. That is why clinical trials (or, at a minimum, further analysis of potentially confounding variables in observational studies) are important. They can help eliminate other factors as explanations for an observed difference. Thus, "a statistically significant number of adverse events" does very little to establish causation, but it can serve as a trigger for further study. In the absence of a significant association of any kind, however, there is little reason to undertake further studies that might clarify the speculative causal links. In simple terms, there is no reason to consider rival explanations for an association if there is no association to explain.

Justice Sotomayer's opinion is a little fuzzy about all this. On the one hand, the opinion states that “the mere existence of reports of adverse events . . . says nothing in and of itself about whether the drug is causing the adverse events.” Fair enough. Mere temporal association is not necessarily causation. That is a matter of study design, not of statistical significance.

On the other hand, the Justice lists "a temporal relationship" in a single patient as one indication of "a reliable causal link." Furthermore, the opinion suggests that even when the number of AERs cannot reasonably be attributed to anything but chance, medical researchers or regulators could treat them as proving causation, at least when combined with other information. As support for this view, Justice Sotomayer writes that "ethical considerations may prohibit researchers from conducting randomized clinical trials to confirm a suspected causal link for the purpose of obtaining statistically significant data."

To be sure, other types of studies and knowledge of biological mechanisms are relevant in causal analysis. But the fact that "medical professionals and researchers do not limit the data they consider to the results of randomized clinical trials or to statistically significant evidence" obviously does not mean that one can infer causation in the absence of good experimental or observational data. AERs do not constitute such data. Shareholders may care about them, and they can stimulate further research that occasionally pans out, but courts should not accept them as proof of "a reliable causal link."

References

Matrixx Initiatives, Inc. v. Siracusano, No. 09-1156, 2011 WL 977060 (U.S. Mar. 22, 2011)

David H. Kaye, David E. Bernstein & Jennifer L. Mnookin, The New Wigmore: A Treatise on Evidence: Expert Evidence, New York: Aspen Pub. Co. (2d ed. 2011)

David H. Kaye & David A. Freedman, Statistics, in Reference Manual on Scientific Evidence (Federal Judicial Center ed., 3d ed. 2011) (preprint)

The Good, the Bad, and the Ugly

BioMedLib offers a free"search engine that takes your query and finds the best responses among millions of biomedical articles in the National Library of Medicine's MEDLINE® database." I tested it with the query, "forensic 'low template'" and obtained a list of 11 articles -- every one of which was relevant. That's a hit rate (true positives) of 100%. I know the search missed some relevant articles, but I do not know how many.

This 100% specificity was stunning because of the inane email that BioMedLib sent me advertising "Who is Publishing in My Domain?", a service that "keeps you updated on research published on the very same topic as your own article, so that you can stay afresh and ahead in your field. This literature-monitoring service takes your article, and periodically sends you articles in the same domain as your article."

It's free, and the generous folks at BioMedLib tried to impress me with the results they obtained for an article of mine entitled "The good, the bad, the ugly: the NAS report on strengthening forensic science in America." They came up with the following "top 10 articles published on the same topic since you published yours.":

Chuva de Sousa Lopes SM, Roelen BA: On the formation of germ cells: The good, the bad and the ugly. Differentiation; 2010 Mar;79(3):131-40

Regan SL, Maggs JL, Hammond TG, Lambert C, Williams DP, Park BK: Acyl glucuronides: the good, the bad and the ugly. Biopharm Drug Dispos; 2010 Oct;31(7):367-95

Miao V, Davies J: Actinobacteria: the good, the bad, and the ugly. Antonie Van Leeuwenhoek; 2010 Aug;98(2):143-50

Berwick M: The good, the bad, and the ugly of sunscreens. Clin Pharmacol Ther; 2011 Jan;89(1):31-3

Baudouin C, Labbé A, Liang H, Pauly A, Brignole-Baudouin F: Preservatives in eyedrops: the good, the bad and the ugly. Prog Retin Eye Res; 2010 Jul;29(4):312-34

Wilson K: Evidence-based medicine. The good the bad and the ugly. A clinician's perspective. J Eval Clin Pract; 2010 Apr;16(2):398-400

Clint SA, Eastwood DM, Chasseaud M, Calder PR, Marsh DR: The "Good, Bad and Ugly" pin site grading system: A reliable and memorable method for documenting and monitoring ring fixator pin sites. Injury; 2010 Feb;41(2):147-50

van der Gun BT, Melchers LJ, Ruiters MH, de Leij LF, McLaughlin PM, Rots MG: EpCAM in carcinogenesis: the good, the bad or the ugly. Carcinogenesis; 2010 Nov;31(11):1913-21

Thomas MG, Loschi M, Desbats MA, Boccaccio GL: RNA granules: the good, the bad and the ugly. Cell Signal; 2011 Feb;23(2):324-34

Kaye DH: The good, the bad, the ugly: the NAS report on strengthening forensic science in America. Sci Justice; 2010 Mar;50(1):8-11

From Double Helix Law to Here

This blog expands into topics outside the scope of the Double Helix Law blog, published at Penn State. That blog, based loosely on the book, The Double Helix and the Law of Evidence, tracks news and publications about forensic genetics. This one branches out into other areas of forensic science, scientific evidence, and statistical evidence. However, Double Helix Law also includes the following tidbits on forensic statistics or forensic science in general:

• Taking Liberties with the Numbers: Following the lead of the Los Angeles Times, a prize-winning writer misinforms the California Bar in "Guilt by the Numbers: How Fuzzy is the Math that Makes DNA Evidence Look So Compelling to Jurors?"
• Rehash and Mismash in the Washington Monthly: Another misadventure in DNA statistics in which a reporter manufactures "DNA's Dirty Little Secret."
• McDaniel v. Brown: Prosecutorial and Expert Misstatements of Probabilities Do Not Justify Postconviction Relief -- At Least Not Here and Not Now: The Supreme Court's encounter with the transposition fallacy in McDaniel v. Brown.
• Another Expert Succumbs to the Transposition Fallacy: Dubious advice for lawyers in Dealing with DNA Evidence: A Legal Guide (London: Routledge-Cavendish 2007).
• An Open Letter to the Arizona Forensic Services Advisory Committee: A state crime laboratory director defends business as usual after the NAS report on Strengthening Forensic Science in the United States.
• The Spin Doctors: Parts of the forensic science establishment respond to the NAS report with various forms of denial.
• Just the Facts, Ma'am: Reflections on the American Association of Forensic Science's disdain for "philosophy, psychology, probability theory, statistics, differential equations."
• The Dalian Letters: A BIT Much: Sleazy correspondence with BIT Life Sciences, the intrepid organizers of World Congresses in Dalian, China.

In addition, the defunct Science & Law blog on the Law Professors' Blog Network has some good information. The archived material is in the right-hand column.