P-values versus Statistical Significance in the Selective Enforcement Case of United States v. Lopez

For several reasons, United States v. Lopez, 415 F.Supp.3d 422 (S.D.N.Y. 2019), is another stimulating opinion from U.S. District Court Judge Jed S. Rakoff. It sets forth a new rule (or a new refinement of a rule) for handling discovery requests when a criminal defendant argues that he or she is the victim of unconstitutional selective investigation. In addition, the opinion reproduces the declaration of the defendants' statistical expert that the judge found "compelling." Too often, the work of statistical consultants is not readily available for public inspection.

In Lopez, Judge Rakoff granted limited discovery into a claim of selective prosecutions stemming from one type of DEA investigation -- "reverse stings." In these cases, law enforcement agents use informants to identify individuals who might want to steal drugs:

An undercover agent or informant then poses as a drug courier and offers the target an opportunity to steal drugs that do not actually exist. Targets in turn help plan and recruit other individuals to participate in a robbery of the fictitious drugs. Just before the targets are about to carry out their plan, they are arrested for conspiracy to commit the robbery and associated crimes.

Id. at 424. Defendant Johansi Lopez and other targets "who are all men of color, allege that ... the DEA limits such operations in the Southern District of New York to persons of color ... in violation of the Fifth Amendment's Equal Protection Clause." Id. at 425.

Seeking to sharpen the usual "broad discretion" approach to discovery, the court adopted the following standard for ordering discovery from the investigating agency:

where a defendant who is a member of a protected group can show that that group has been singled out for reverse sting operations to a statistically significant extent in comparison with other groups, this is sufficient to warrant further inquiry and discovery.

Id. at 427. The court was persuaded that the "of color" group was singled out on the basis of a "combination of raw data and statistical analysis." Id. This conclusion is entirely reasonable, but was the "singled out ... to a statistically significant extent" rule meant to make classical hypothesis testing at a fixed significance level the way to decide whether to grant discovery, or would an inquiry into p-values as measures of the strength of the statistical evidence against the hypothesis of "not singled out" be more appropriate? The discussion that follows suggests that little is gained by invoking the language and apparatus of hypothesis testing.

I. The "Singled Out" Standard

The "singled out" standard was offered as a departure from the one adopted by the Supreme Court in United States v. Armstrong, 517 U.S. 456 (1996). In Armstrong, Chief Justice Rehnquist wrote for the Court that to obtain discovery for a selective prosecution defense, "the required threshold [is] a credible showing of different treatment of similarly situated persons." Id. at 470 (emphasis added). The Court deemed insufficient a "study [that] failed to identify individuals who were not black and could have been prosecuted for the offenses for which respondents were charged, but were not so prosecuted." Id. (emphasis added).

In contrast, Judge Rakoff focused on "the racially disparate impact of the DEA's reverse sting operations," id. (emphasis added), and looked to other groups in the general population in ascertaining the magnitude and significance of the disparity. He maintained that "as now recognized by at least three federal circuits, selective enforcement claims should be open to discovery on a lesser showing than the very strict one required by Armstrong." Lopez, 415 F.Supp.3d at 425. In finding that this "lesser showing" had been made, he also considered three more restricted populations (of arrested individuals) that might better approximate groups in which all people are similarly plausible targets for sting operations.

Applying the "singled out ... to a statistically significant extent " standard requires not just the selection of an appropriate population in which to make comparisons among protected and unprotected classes but also a judicial determination of statistical significance. The opinion was not clear about the significance level required, but the court was impressed with an expert declaration that, unsurprisingly, applied the 0.05 level commonly used in many academic fields and most litigation involving statistical evidence.

Unfortunately, transforming a statistical convention into a rule of law does not necessarily achieve the ease of administration and degree of guidance that one would hope for. A p < 0.05 rule still leaves open considerable room for discretion in deciding whether p < 0.05, invites the ubiquitous search for significance, and requires an understanding of classical hypothesis testing that most judges and many experts lack if it is to be applied sensitively. Lopez itself shows some of the difficulties.

II. "Raw Data" Does Not Speak for Itself

The "raw data" (as presented in the body of the opinion) was that "not a single one of the 179 individuals targeted in DEA reverse sting operations in SDNY in the past ten years was white, and that all but two were African-American or Hispanic," whereas

• "New York and Bronx Counties ... are 20.5% African-American, 39.7% Hispanic, and 29.5% White"; and
• The breakdown of "NYPD arrests" is
  • felony drug: "42.7% African-American, 40.8% Hispanic, and 12.7% White";
  • firearms: "65.1% African-American, 24.3% Hispanic, 9.7% White"; and
  • robbery: "60.6% African-American, 31.1% Hispanic, 5.1% White."

Id. In other words, the "raw data" are summarized by the statistics p = 177/179 = 98.88% African-American and Hispanic (AH) targets among all the DEA-targets and π = 60.2%, 83.5%, 89.4%, or 91.7%, where π is the proportion of AH targets in the four surrogate populations.

Presumably, Judge Rakoff uses "statistically significant" to denote any difference p − π that would arise less than one time out of 20 when repeatedly picking 179 targets without regard to race (or a variable correlated with race) in the (unknown) population from which the DEA picked its "reverse sting" targets in the Southern District. (The 0.05 significance level is conventionally used in academic writing in many fields, and, as discussed below, it was the used by the defendant's expert in the case; however, in recent years, it has been said to be too weak to produce scientific findings that are likely to be reproducible.)

Notice that all the variability in the difference between p and π arises from p. The population proportion π is fixed (for each surrogate population and, of course, in the unknown population from which targets are drawn). It also is worth asking why the data and population are limited to the Southern District. Does the DEA have a different method for picking sting targets in, say, the Eastern District? If not, and if the issue is discriminatory intent, might the "significant" difference in the Southern District be an outlier from a process that does not look to the race of the individuals who are targeted?

In itself, "raw data" tell us nothing about statistical significance. "Significance" is just a word that characterizes a set of values for a statistic. It takes some sort of analysis to determine the zone in which "significance" exists. The evaluation may be intuitive and impressionistic, or it may be formal and quantitative. Intuitively, it seems that p = 177/179 is far enough from what would be expected from a race-independent system that it should be called "significant." But how do we know this intuition is correct? This is where probability theory comes into play.

III. One Simple Statistical Analysis

Even if the probability of targeting an African-American or a Hispanic in each investigation were the largest of the surrogate population proportions (0.917), the expected number of AH targets is still only 164, and the probability of picking 177 or more (an excess of 13 or more) is 0.0000271. The probability of picking as few or fewer than 151 AH targets (a deficit of 13 or more is 0.000881. Hence, the probability of the observed departure from the expected value (or an even greater departure) is 0.000918, or about 1/1,100. \1/

This two-sided p-value of 1/1,100 is less than 1/20, so we can reject the idea that the DEA selected targets without regard to AH status or some variable that is correlated with that status. That the p-value is smaller than the 1/20 cutoff makes the excess count statistically significant evidence against the hypothesis of a fixed selection probability of 0.917 in each case. The data suggest that the selection probability is at least somewhat larger than that.

Or do they? The binomial probability model that gives rise to these numbers could itself be wrong. What if there are fewer stings than targets? If an informant proposes multiple targets for one sting (think several members of the same gang, for example), won't knowing that the first one is AH increase the probability that the second one is AH?

IV. An "Exact" Analysis

The court did not discuss the formal analysis of statistical significance in the case. However, it relied on such an analysis. After describing the "raw data," Judge Rakoff added:

Furthermore, defendants have provided compelling expert analysis demonstrating that these numbers are statistically significant. According to a rigorous analysis conducted by Dr. Crystal S. Yang, a Harvard law and economics professor, it is highly unlikely, to the point of statistical significance, that the racially disparate impact of the DEA's reverse sting operations is simply random.

Id. at 427. The phrase "highly unlikely to the point of statistical significance" sounds simple, but the significance level of α = 0.05 is not all that stringent, and neither 0.05 nor p-values for the observed proportion p represent the probability that targets of the sting operations are "simply random" as far as racial status goes. Instead, α is the probability of an observed disparity that triggers the conclusion "not simply random" assuming that the model with the parameter π is correct. If we use the court's rule for rejecting random variation as an explanation for the disparity between p and π, then the chance of ordering discovery when the disparity is nothing more than statistical noise is just one in 20. But the probability that the disparity is merely noise when that disparity is large enough to reject that explanation need not be one in 20. If I flip a fair coin five times and observe five heads, I can reject the hypothesis that the coin is fair (i.e., the string of heads is merely noise) at the α = 0.05 level. The probability of the extreme statistic p = 5/5 is (1/2)⁵ = 1/32 < 1/20 if the coin is fair. But if I tell you that I blindly picked the coin from a box of 1,000 coins in which 999 were fair and one was biased, would you think that 1/20 is the probability that the coin I picked was biased? The probability for that hypothesis would be much closer to 1/1,000. (How much closer is left as an exercise to the reader who wishes to consult Bayes' theorem.)

Many opinions gloss over this distinction, transposing the arguments in the conditional probability that is the significance level to report things such as "[s]ocial scientists consider a finding of two standard deviations significant, meaning there is about 1 chance in 20 that the explanation for a deviation could be random." Waisome v. Port Authority, 948 F.2d 1370, 1376 (2d Cir. 1991). The significance level of 1/20 relates to the probability of data given the hypothesis, not the probability of the hypothesis given the data.

The expert report in this case did not equate the improbability of a range of data to the improbability of a hypothesis about the process that generated the data. The report is reproduced below. Dr. Yang used not just the four populations listed in the body of the opinion, but a total of eight surrogate populations as "benchmarks" to conclude that "it is extremely unlikely that random sampling from any of the hypothetical populations could yield a sample of 179 targeted individuals where 177 or more individuals are Latino or Black." Id. at 431. That is a fair statement. If we postulate simple random sampling in infinite populations with the specified values for the proportion π who are AH, the sample data for which p = 177/179 lie in a region outside of the central 95% of all possible samples. Standing alone, that critical region is not "extremely unlikely," but the largest p-value reported for all the populations in the declaration is 1/10,000.

The declaration is refreshingly free from exaggeration, but its explanation of the statistical analysis is slightly garbled. Dr. Yang swore that

Using the exact hypothesis test, I test whether the observed proportion of Latinos or Blacks observed in the sample (x = 177, n = 179) is equal to the hypothesized population probability/proportion p [π in the notation I have used]. Under this test, the null hypothesis is that the observed proportion is not statistically different from the hypothesized population proportion. The alternative hypothesis is that the observed proportion is statistically different from the hypothesized population proportion, a two-sided test. Each exact hypothesis test produces a corresponding p-value, which is the probability of observing a proportion as extreme or more extreme than the observed proportion assuming that the null hypothesis is true. A small p-value implies that the observed proportion is highly unlikely under the null hypothesis, favoring the rejection of the null hypothesis.

Id. at 430. The bottom line is fine, but the exposition confuses the notion of a p-value as a measure of the strength of evidence with the idea of a significance level as protection against false rejection of null hypotheses. A statistical distribution (or a non-parametric procedure) produces a p-value that can be compared to a pre-established value such as α = 0.5 to reach a test result of thumbs up (significant) or down (not significant). The hypothesis test does not "produce a ... p-value." At most, it uses a p-value to reach a conclusion. Once α is fixed, the p-value produces the test result.

Moreover, the description of the hypotheses that are being tested makes little sense. Hypothesis are statements about unknown parameter values. They are not statements about the data or about the data combined with the parameter. What does it mean to say that "the null hypothesis is that the observed proportion is not statistically different from the hypothesized population proportion" or that "[t]he alternative hypothesis is that the observed proportion is statistically different from the hypothesized population proportion, a two-sided test"? These are not hypotheses about the true value of an unknown parameter of the binomial distribution used in the exact test. The observed proportion (p in my notation) is known to be different from the hypothetical population proportion (π in my notation). There is no uncertainty about that. The hypotheses to be tested have to be statements about what produced the observed "statistical difference" p − π rather than statements of whether there is a "statistical difference."

Thus, the hypothesis that was tested was not "whether the observed proportion of Latinos or Blacks observed in the sample (x = 177, n = 179) is equal to the hypothesized population probability/proportion." We already know that these two known quantities are not equal. The hypothesis that Dr. Yang tested was whether (within the context of the model) the (unknown) selection probability is the known value π for a given surrogate population. If we call this unknown binomial probability Θ, she was testing claims about the true value of Θ. The null hypothesis is that the unknown Θ equals a known population proportion π; the alternative hypothesis is that Θ is not precisely equal to π. The tables in the expert declaration present strange expressions for these hypotheses, such as π = 0.917 rather than Θ = 0.917.

Nonetheless, these complaints about wording and notation might be dismissed as pedantic. The p-values for properly stated null hypotheses do signal a clear thumbs down for the null hypothesis Θ = 0.917 at the prescribed α = 0.05 level. \3/

Of more interest than these somewhat arcane points is the response to the question of dependent outcomes in selecting multiple targets for a particular investigation. Dr. Yang reasoned as follows:

[S]uppose that we took the most conservative approach and assumed that there is perfect homophily (i.e. a perfect correlation of 1) such that if one individual targeted in the operation is Latino or Black, all other individuals in that same operation are also Latino or Black. Under this conservative assumption, we can then treat the observed sample as if there were only 46 independent draws (rather than 179 draws). To observe a racial composition of 98.9% Latino or Black would thus require that at least 45 out of 46 draws resulted in a Latino or Black individual being targeted.

Id. at 431 (notes omitted). Rerunning the exact computation for p = 45/46 changes the p-value to 0.18, or roughly 1 in 5. Id. at 438 (Table 2, row h). This much larger p-value fails to meet the α = 0.05 rule that the court set.

The expert declaration reports this result along with those from seven other  hypothesis tests (one for each surrogate population). Five of the tests produce "significance," and three do not. Does this mixed picture establish that p < 0.05? It seems to me that the usual procedures for assessing multiple-comparisons do not apply here because the data are identical for all eight tests. The value of π in the surrogate population is being varied, and π is not a random variable. The use of multiple populations is a way to examine the robustness of the results of the statistical test. For p = 177/179, which is the only figure the court mentioned, the finding of significance is stable. For p = 45/46, however, it becomes necessary to ask which surrogate populations best approximate the unknown population of potential targets for the reverse stings. Numerical analysis cannot answer that question. \2/ As Macbeth soliloquized, "we still have judgment here."

NOTES

1. An exact binomial computation performed in R (using binom.test(177,179,0.917)) gives a two-sided p-value of 0.0000578, which is about 1/17,300. Table 1, row h of the expert declaration reports a value 0.0001, or 1/10,000.
2. Dr. Yang intimates that discovery might shed light on how much of a correction is needed. Id. at 431 ("It is impossible to know the true degree of homophily or correlation among individuals targeted in reverse-sting operations, particularly when the DEA's selection criteria is unknown."); id. at 432 ("Again, because I have no knowledge of the DEA's selection criteria of potential targets, it is impossible to know which of the hypothesized populations captures the relevant pool of similarly situated individuals. A more definitive statistical analysis may be possible if the government provides the requested selection criteria.").
3. As the declaration points out, the p-values are small enough to yield "significance" at even smaller levels, but it would not be acceptable to set up a test at the α = 0.05 level and then interpret it as demonstrating significance at a different level. Changing α after analyzing the data vitiates the interpretation of α as the probability of a false alarm, which can be written as Pr("significant difference" | Θ = π).

APPENDIX: DECLARATION OF PROFESSOR CRYSTAL S. YANG

Pursuant to 28 U.S.C. § 1746, I, CRYSTAL S. YANG, J.D., Ph.D., declare under penalty of perjury that the following is true and correct:

1. I am a Professor of Law at Harvard Law School and Faculty Research Fellow at the National Bureau of Economic Research. My primary research areas are in criminal law and criminal procedure, with a focus on testing for and quantifying racial disparities in the criminal justice system. Before joining the Harvard Law School faculty, I was an Olin Fellow and Instructor in Law at The University of Chicago Law School. I am admitted to the New York State Bar and previously worked as a Special Assistant United States Attorney in the U.S. Attorney's Office for the District of Massachusetts.

2. I received a B.A. in economics summa cum laude from Harvard University in 2008, an A.M. in statistics from Harvard University in 2008, a J.D. magna cum laude from Harvard Law School in 2013, and a Ph.D. in economics from Harvard University in 2013. My undergraduate and graduate training involved substantial coursework in quantitative methods.

3. I have published in peer-reviewed journals such as the American Economic Review, The Quarterly Journal of Economics, the American Economic Journal: Economic Policy, and have work represented in many other peer-reviewed journals and outlets.

4. I make this Declaration in support of a motion being submitted by the defendants to compel the government to provide discovery in this case.

Statistical Analyses Pertinent to Motion

5. I was retained by the Federal Defenders of New York to provide various statistical analyses relevant to the defendants' motion in this case. Specifically, I was asked to evaluate whether the observed racial composition of targets in reverse-sting operations in the Southern District of New York could be due to random chance.

6. To undertake this statistical analysis, I first had to obtain the racial composition of targeted individuals in DEA reverse-sting stash house cases brought in the Southern District of New York for the ten-year period beginning on August 5, 2009 and ending on August 5, 2019. Based on the materials in Lamar and Garcia-Pena, as well as additional searches conducted by the Federal Defenders of New York, I understand that there have been 46 fake stash house reverse-sting operations conducted by the DEA during this time period. These 46 operations targeted 179 individuals of whom zero are White, two are Asian, and 177 are Latino or Black – the “sample.” \1/ Given these counts, this means that of the targeted individuals, 98.9% are Latino or Black (and 100% are non-White). Thus, the relevant question at hand is whether the observed racial composition of the sample could be due to random chance alone if the DEA sampled from a population of similarly situated individuals.

7. Second, I had to define what the underlying population of similarly situated individuals is. In other words, what is the possible pool of all similarly situated individuals who could have been targeted by the DEA in a reverse-sting operation? Because the DEA's criteria for being a target in these reverse-sting cases is unknown, my statistical analysis will assume a variety of hypothetical benchmark populations. If the government provides its selection criteria for being a target in these reverse-sting cases, a more definitive statistical analysis may be possible. Based on materials from Garcia-Pena and Lamar, I have identified eight hypothetical benchmark populations. Below, I present the hypothesized populations and the racial composition (% Latino or Black) in each population in order of least conservative (i.e. smallest share of Latino or Black) to most conservative (i.e. highest share of Latino or Black):

a. 2016 American Community Survey 5-year estimates on counties in the SDNY (from Garcia-Pena): 48.1% Latino or Black
b. 2016 American Community Survey 5-year estimates on Bronx and New York Counties (from Garcia-Pena): 60.2% Latino or Black
c. New York Police Department (NYPD) data from January 1 – December 31, 2017, on felony drug arrests in New York City (from Garcia-Pena): 83.5% Latino or Black
d. Estimates by Prof. Kohler-Hausmann on men aged 16-49 living in New York City who have prior New York State (NYS) violent felony convictions (from Lamar): 87.1% Latino or Black
e. Estimates by Prof. Kohler-Hausmann on men aged 16-49 living in New York City who have prior NYS felony convictions (from Lamar): 87.5% Latino or Black
f. NYPD data from January 1 – December 31, 2017 on firearms seizures arrests in New York City (from Garcia-Pena): 89.4% Latino or Black
g. Reverse-sting operation defendants in the Northern District of Illinois (from Garcia-Pena): 87.7-90.7% Latino or Black \2/
h. NYPD data from January 1 – December 31, 2017 on robbery arrests in New York City (from Garcia-Pena): 91.7% Latino or Black

8. For each of these eight hypothesized populations, I then conduct an exact hypothesis test for binomial random variables. This is the standard statistical test used for calculating the exact probability of observing x “successes” out of n “draws” when the underlying probability of success is p and the underlying probability of failure is 1-p. Here, each defendant represents an independent draw and a success occurs when the defendant is Latino or Black. Using the exact hypothesis test, I test whether the observed proportion of Latinos or Blacks observed in the sample (x = 177, n = 179) is equal to the hypothesized population probability/proportion p. Under this test, the null hypothesis is that the observed proportion is not statistically different from the hypothesized population proportion. The alternative hypothesis is that the observed proportion is statistically different from the hypothesized population proportion, a two-sided test. \3/ Each exact hypothesis test produces a corresponding p-value, which is the probability of observing a proportion as extreme or more extreme than the observed proportion assuming that the null hypothesis is true. A small p-value implies that the observed proportion is highly unlikely under the null hypothesis, favoring the rejection of the null hypothesis.

9. The following Table 1 presents each of the eight hypothesized population proportions, the null hypothesis under each population, the alternative hypothesis under each population, and the corresponding p-value using the observed proportion of Latinos or Blacks in the sample assuming 179 independent draws:

Table 1 (x = 177, n = 179)

Hypothesized Population Proportion	Null Hypothesis H0	Alternative Hypothesis Ha	p-value
a. 48.1% Latino or Black	H0: p = 0.481	Ha: p ≠ 0.481	0.0000
b. 60.2% Latino or Black	H0: p = 0.602	Ha: p ≠ 0.602	0.0000
c. 83.5% Latino or Black	H0: p = 0.835	Ha: p ≠ 0.835	0.0000
d. 87.1% Latino or Black	H0: p = 0.871	Ha: p ≠ 0.871	0.0000
e. 87.5% Latino or Black	H0: p = 0.875	Ha: p ≠ 0.875	0.0000
f. 89.4% Latino or Black	H0: p = 0.894	Ha: p ≠ 0.894	0.0000
g. 90.7% Latino or Black	H0: p = 0.907	Ha: p ≠ 0.907	0.0000
h. 91.7% Latino or Black	H0: p = 0.917	Ha: p ≠ 0.917	0.0001

10. The above statistical calculations in Table 1 show that regardless of which of the eight hypothesized population proportions is chosen, one could reject the null hypothesis at conventional levels of statistical significance. For example, one could reject the null hypothesis at the standard 5% significance level which requires that the p-value be less than 0.05. All eight p-values are substantially smaller than 0.05 and would lead to a rejection of the null hypothesis even using more conservative 1%, 0.5%, or 0.1% significance levels. In other words, it is extremely unlikely that random sampling from any of the hypothetical populations could yield a sample of 179 targeted individuals where 177 or more individuals are Latino or Black.

11. Alternatively, one may be interested in the reverse question of what the underlying population proportion would have to be such that the observed proportion could be due to random chance alone assuming there are 179 independent draws. Using the standard 5% significance level, I have calculated that the hypothesized population would have to be composed of at least 96.0% Latinos or Blacks in order for one to not be able to reject the null hypothesis. In other words, unless the pool of similarly situated individuals is comprised of at least 96.0% Latinos or Blacks, it is highly unlikely that one could get a sample of 179 targeted individuals where 177 or more individuals are Latino or Black.

12. One potential question with the statistical analyses in Table 1 is whether the assumption that each of the 179 targeted individuals is an independent draw is reasonable. For example, what if the race/ethnicity of individuals in each reverse-sting operation is correlated, such that if one individual targeted in an operation is Latino or Black, the other individuals are also more likely to be Latino or Black? This correlation within operations could result if there is homophily, or “the principle that a contact between similar people occurs at a higher rate than among dissimilar people.” Miller McPherson et al., Birds of a Feather: Homophily in Social Networks, 27 Ann. Rev. Soc. 315, at 416 (2001). It is impossible to know the true degree of homophily or correlation among individuals targeted in reverse-sting operations, particularly when the DEA's selection criteria is unknown. But suppose that we took the most conservative approach and assumed that there is perfect homophily (i.e. a perfect correlation of 1) such that if one individual targeted in the operation is Latino or Black, all other individuals in that same operation are also Latino or Black. Under this conservative assumption, we can then treat the observed sample as if there were only 46 independent draws (rather than 179 draws). \4/ To observe a racial composition of 98.9% Latino or Black would thus require that at least 45 out of 46 draws resulted in a Latino or Black individual being targeted. \5/

13. For each of the eight hypothesized benchmark populations, I then test whether the observed proportion of Latinos or Blacks observed in this alternative sample (x = 45, n = 46) is equal to the hypothesized proportion from an underlying population assuming that there are only 46 independent draws. The following Table 2 presents each of the eight hypothesized population proportions, the null hypothesis under each population, the alternative hypothesis under each population, and the corresponding p-value using the observed proportion of Latinos or Blacks in the sample assuming 46 independent draws:

Table 2 (x = 45, n = 46)

Hypothesized Population Proportion	Null Hypothesis H0	Alternative Hypothesis Ha	p-value
a. 48.1% Latino or Black	H0: p = 0.481	Ha: p ≠ 0.481	0.0000
b. 60.2% Latino or Black	H0: p = 0.602	Ha: p ≠ 0.602	0.0000
c. 83.5% Latino or Black	H0: p = 0.835	Ha: p ≠ 0.835	0.0045
d. 87.1% Latino or Black	H0: p = 0.871	Ha: p ≠ 0.871	0.0255
e. 87.5% Latino or Black	H0: p = 0.875	Ha: p ≠ 0.875	0.0257
f. 89.4% Latino or Black	H0: p = 0.894	Ha: p ≠ 0.894	0.0871
g. 90.7% Latino or Black	H0: p = 0.907	Ha: p ≠ 0.907	0.1240
h. 91.7% Latino or Black	H0: p = 0.917	Ha: p ≠ 0.917	0.1795

14. Under this conservative assumption of perfect homophily, the above statistical calculations in Table 2 show that under the first five hypothesized population proportions (a-e), one could reject the null hypothesis at the standard 5% significance level. In other words, even if the hypothesized population proportion of Latinos or Blacks is as high as 87.5%, it is highly unlikely that random sampling could yield a sample of 46 individuals where 45 or more individuals are Latino or Black. One, however, cannot reject the null hypothesis for the next three hypothesized population proportions (f-h). Again, because I have no knowledge of the DEA's selection criteria of potential targets, it is impossible to know which of the hypothesized populations captures the relevant pool of similarly situated individuals. A more definitive statistical analysis may be possible if the government provides the requested selection criteria.

15. As before, I also ask the reverse question of what the underlying population proportion would have to be such that the observed proportion could be due to random chance alone assuming that there are only 46 independent draws. Using the standard 5% significance level, I have calculated that the hypothesized population would have to be composed of at least 88.5% Latinos or Blacks in order for one to not be able to reject the null hypothesis. In other words, unless the pool of similarly situated individuals is comprised of at least 88.5% Latinos or Blacks, it is highly unlikely that one could get a sample of 46 targeted individuals where 45 or more individuals are Latino or Black.

Dated: Cambridge, Massachusetts
September 13, 2019
/s/ Crystal S. Yang
Crystal S. Yang

Footnotes

   1. In consultation with the Federal Defenders of New York, this sample is obtained by taking the 33 cases and 144 defendants identified in Garcia-Pena or Lamar, excluding two cases and five defendants that are either not DEA cases or reverse-sting cases, and including an additional 15 cases and 40 defendants that were not covered by the time frames included in the Lamar or Garcia-Pena analysis.
   2. I choose 90.7% (the upper end of the range) as the relevant proportion given that it yields the most conservative estimates.
   3. This two-sided test takes the most conservative approach (in contrast to a one-sided test) because it allows for the possibility of both an over-representation and under-representation of Latinos or Blacks relative to the hypothesized population proportion.
   4. I make the simplifying assumption that each of the 46 operations targeted the average number of codefendants, 3.89= 179/46.
   5. Technically, 45.494 draws would need to be of Latino or Black individuals but I conservatively round down to the nearest integer.

 UPDATED: 1/24/2021 3:13 ET

Can P-values or Confidence Intervals Prove Non-association?

Last week, the scientific community heard two prominent calls to end to the labeling of results of statistical studies as statistically "significant" or "not significant." One of the manifestos (Amrhein et al.) is particularly irate about negative claims -- conclusions of "no association" based on sample data that are not extreme enough to reject the null hypothesis. The authors "are frankly sick of seeing such nonsensical 'proofs of the null' and claims of non-association in presentations, research articles, reviews and instructional materials." The "pervasive problem," as they articulate it, is in the box on the right. It is a recurring issue in toxic tort an other litigation.

PERVASIVE PROBLEM
Let’s be clear about what must stop: we should never conclude there is ‘no difference’ or ‘no association’ just because a P value is larger than a threshold such as 0.05 or, equivalently, because a confidence interval includes zero. Neither should we conclude that two studies conflict because one had a statistically significant result and the other did not. These errors waste research efforts and misinform policy decisions. 1/

It is hard to disagree with the first observation in the box. Logically, we cannot conclude that the null hypothesis is true "just because a P value is larger than a threshold." For one thing, if the study lacks power, random error plausibly could be the source of the observed difference. However, this is not a sufficient reason to abjure hypothesis tests. It is a reason to attend to power. If a study has ample power and is well designed, then it is not so obvious that it wastes research efforts or misinforms policy decisions to conclude that the failure to achieve statistical significance at a rather undemanding level demonstrates the lack of a true association.

The second observation also is literally correct. Different outcomes of a null hypothesis significance test are not logically sufficient to establish a conflict between the studies. In 2013, a paper in the International Journal of Cardiology reported "that the use of selective COX-2 inhibitors [such as Vioxx] was not associated with atrial fibrillation risk" because the researchers did not find a statistically significant association. 2/ But a 2011 study in the British Medical Journal had found just such an association. 3/ It might seem that the second study is in conflict with the first, but as Amrhein et al. explain:

The researchers describing their statistically non-significant results found a risk ratio of 1.2 (that is, a 20% greater risk in exposed patients relative to unexposed ones). They also found a 95% confidence interval that spanned everything from a trifling risk decrease of 3% to a considerable risk increase of 48% (P = 0.091; our calculation). The researchers from the earlier, statistically significant, study found the exact same risk ratio of 1.2. That study was simply more precise, with an interval spanning from 9% to 33% greater risk (P = 0.0003; our calculation).

It is ludicrous to conclude that the statistically non-significant results showed “no association”, when the interval estimate included serious risk increases; it is equally absurd to claim these results were in contrast with the earlier results showing an identical observed effect. Yet these common practices show how reliance on thresholds of statistical significance can mislead us ... . 4/

So the results of the second study do not undermine those of the first. The studies provide consistent point estimates. The second one merely has a smaller standard error. As such, it does not merit as much weight, but it is fully consistent with the first result. 5/

What happens, however, when the point estimates of the relative risk are different? Is "no association" a reasonable statement when the confidence interval is tightly centered around RR = 1? Suppose, for example, that the second study produced a 95% confidence interval of RR = 1.0 ± 0.2. From a maximum likelihood standpoint, "no association" is the best estimate. Would it still be ludicrous to describe the observed RR of 1 (no difference in the risk) as proof of "no association"?

Amrhein et al. do not answer this specific question, but they grudgingly accept the idea of proof of "no association" in the form of no practical importance. As stated at the outset, they are "frankly sick of seeing such nonsensical 'proofs of the null' ... ." They argue that all values inside a confidence interval are "compatible" with the data (as are some outside of the interval), implying that you cannot single out any one value to the exclusion of the others. But they also propose that some values are more compatible than others. Is compatibility a likelihood? A probability? Something else? In the end, their answer to the question of whether the data can be said to prove a negative -- that there is no true association -- is this: "if you deem all of the values inside the interval to be practically unimportant, you might then be able to say something like ‘our results are most compatible with no important effect’." 6/

NOTES

1. Valentin Amrhein, Sander Greenland, Blake McShane et al., Comment, Retire Statistical Significance, 567 Nature 305, 305-06 (2019).
2. T.-F. Chao, C.-J. Liu, S.-J. Chen, K.-L. Wang, Y.-J. Lin, S.-L. Chang, et al., The Association Between the Use of Non-steroidal Anti-inflammatory Drugs and Atrial Fibrillation: a Nationwide Case–control Study, 168 Int’l J. Cardiology 312 (2013).
3. M. Schmidt, C.F. Christiansen, F. Mehnert, K.J. Rothman, & H.T. Sørensen, Non-steroidal Anti-inflammatory Drug Use and Risk of Atrial Fibrillation or Flutter: Population Based Case-control Study, 343 Brit. Med. J. d3450 (2011).
4. Amrhein, supra note 1, at 306..
5. In fact, it lends strength to the conclusion that the true relative risk exceeds 1. Morten Schmidt & Kenneth J. Rothman, Mistaken Inference Caused by Reliance on and Misinterpretation of a Significance Test, 177(3) Int'l J. Cardiology 1089, 1090 (2014) (meta-analysis give a CI of 1.1 to 1.3).
6. Amrhein, supra note 1, at 307.

Legal Implications of the Statisticians' Manifestos on Statistical Significance

Manifestos in The American Statistician 1/ and Nature 2/ last week urge an end to "statistical significance." As these communications make clear, statisticians have long denounced presenting results that are barely significant (for example, p = 0.048) as radically different from ones that are almost significant (for example, p = 0.052). They have cautioned that "no significant difference" with a study that lacks power is not proof of no real difference; that a "statistically significant difference" can, for all practical purposes, be of no importance; that a p-value is not the probability that the null hypothesis is true; and so on (and on).

Indeed, after reading such books as Statistics: Concepts and Controversies (1979) by David S. Moore, and The Significance Test Controversy: A Reader, edited by Denton Morrison and Ramon Henkel (1970), I once proposed that courts exclude testimony using phrases like “significant difference” and “95% confidence interval” as unduly confusing and potentially misleading under Federal Rule of Evidence 403 and the common law from which the rule is derived. 3/ That idea fell on barren soil.

Now a similar rule has been proposed for scientific discourse generally. The American Statistician’s editorial advises all scientists as follows: “‘statistically significant’—don’t say it and don’t use it.” It observes that

The ASA Statement on P-Values and Statistical Significance [Feb. 5, 2016] stopped just short of recommending that declarations of “statistical significance” be abandoned. We take that step here. We conclude, based on our review of the articles in this special issue and the broader literature, that it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “p < 0.05,” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way.

Regardless of whether it was ever useful, a declaration of “statistical significance” has today become meaningless. Made broadly known by Fisher’s use of the phrase (1925), Edgeworth’s (1885) original intention for statistical significance was simply as a tool to indicate when a result warrants further scrutiny. But that idea has been irretrievably lost. Statistical significance was never meant to imply scientific importance, and the confusion of the two was decried soon after its widespread use (Boring 1919). Yet a full century later the confusion persists.

And so the tool has become the tyrant. The problem is not simply use of the word “significant,” although the statistical and ordinary language meanings of the word are indeed now hopelessly confused (Ghose 2013); the term should be avoided for that reason alone. The problem is a larger one, however: using bright-line rules for justifying scientific claims or conclusions can lead to erroneous beliefs and poor decision making (ASA statement, Principle 3). A label of statistical significance adds nothing to what is already conveyed by the value of p; in fact, this dichotomization of p-values makes matters worse.

Also eschewing what it calls "dichotomania," the comment in Nature, which garnered "more than 800 signatories" when a pre-publication draft was circulated, reads

[I]n line with many others over the decades, we are calling for a stop to the use of P values in the conventional, dichotomous way — to decide whether a result refutes or supports a scientific hypothesis. .... The misuse of statistical significance has done much harm to the scientific community and those who rely on scientific advice. P values, intervals and other statistical measures all have their place, but it’s time for statistical significance to go.

This is powerful stuff, but I do not expect testifying experts to change their ways or courts to stop prizing declarations of “statistically significant” findings. Neither do I expect the scientific establishment to change overnight. I doubt that phrases like "trending toward significance" or "highly significant" will disappear (or, more radically, that p-values will be abandoned). There are lots of things to consider. Do we need words of some kind to demarcate studies whose estimates reasonably can be considered to be close to a true value as opposed to those that could not be consistently replicated because of randomness in the data? In what contexts should an adjective be used for this purpose? It is one thing to say that journal editors should not use an arbitrary line to reject articles under all circumstances or even that scientists should just state a number rather than using a less precise adjective, but what about more popular writing? If a term like "significant" is used for demarcation in any context, what is the tipping point? More fundamentally, why use p-values to grade the strength of evidence? Should something else play this role? The 43 articles in The American Statistician and the correspondence in Nature on the Amrhein et al. article indicate that no simple fix is imminent in the scientific community. 4/

Nonetheless, the recent manifestos should have some influence in the legal world. They should cause lawyers or testifying experts to hesitate to puff up evidence with modest p-values (in the vicinity of 0.05, for example) as "significant!", if only because the American Statistician's editorial, the earlier ASA Statement, and the Nature comment and the large number of statisticians who publicly endorsed it, all supply fodder for cross-examination (depending on the jurisdiction's hearsay rule for "learned treatises"). Relatedly, they could support motions to limit testimony with such verbiage as misleading or unfairly prejudicial, as I once suggested.

In addition, they should reinforce the tendency of most courts to reject a mechanical rule for admissibility based on a false alarm probability of 0.05. After seeing the manifestos, a law school colleague wrote me that he had "never understood why an empirical study that was 1% less significant than what a social science journal would accept is therefore ‘junk’ that should not even be considered by a jury." However, the argument for a categorical rule is more subtle. The relevant standard for admissibility is not, "Is this study junk science?" Under Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993), and Federal Rule of Evidence 702, it is whether a finding presented as scientific qualifies as "scientific knowledge" and, as such, possess "evidentiary reliability." The argument that all statistical findings that fail the p < 0.05 test are not reliable enough to consider (along with other evidence) has always been dubious. At most, the mechanical rule would be that p ≥ 0.05 shows that the finding is insufficient for scientists -- and hence legal factfinders -- to conclude that the alternative hypothesis is true.

But even though Rule 702 does not lead to the categorical exclusion of findings for which p ≥ 0.05, such evidence is subject to the balancing test of Rule 403. One could argue for a bright-line rule to simplify that inquiry. Juries and judges often do not understand what a p-value is and might misconstrue a study with p exceeding 0.05 as good proof of a real association or effect (especially if the result is not characterized as "not significant"). For example, jurors might think that if p is "only" 0.052, then they should be almost 95% "confident" that there is a real association or effect. Such naive transposition is often incorrect. Of course, 0.05 is (or any other number) is somewhat arbitrary, but it may not be a ridiculous dividing point if the benefits of a simple rule exceed the costs of the misclassifications it produces relative to purely ad hoc case-by-case balancing. 5/

My own preference is for a more flexible approach -- with judicial recognition that the probative value of studies for which p is near (and, a fortiori, much larger than) 0.05 is quite limited. The findings on either side of this dividing line are not necessarily "junk science," but in the vicinity of p = 0.05, they are not all that surprising even if there is no real association or effect. As such, it may not be worth the time it takes to educate the jury about the weak evidence. In ruling on motions to exclude on this Rule 403 basis, courts should consider not only the issue of randomness in the data (which is all the p-value addresses), but also the design and quality of the statistical study as well as the availability (or not) of other, more probative evidence on the same point.

NOTES

1. Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Lazar, Moving to a World Beyond “p < 0.05”, 73:sup1 Am. Statistician 1–19 (2019) DOI: 10.1080/00031305.2019.1583913
2. Valentin Amrhein, Sander Greenland, Blake McShane et al., Comment, Retire Statistical Significance, 567 Nature 305-307 (2019). This commentary is discussed further in Can P-values or Confidence Intervals Prove Non-association?, Forensic Sci., Stat. & L., Mar. 30, 2019.
3. D.H. Kaye, Is Proof of Statistical Significance Relevant?, 61 Wash. L. Rev. 1333 (1986).
4. John P. A. Ioannidis, Letter, Retiring Statistical Significance Would Give Bias a Free Pass, 567 Nature 461 (2019); Valen E. Johnson, Raise the Bar Rather than Retire Significance, 567 Nature 461 (2019); Julia M. Haaf, Alexander Ly & Eric-Jan Wagenmakers, Letter, Retire Significance, But Still Test Hypotheses, 567 Nature 461 (2019).
5. Even with a categorical rule, courts would have discretion to exclude those studies with p-values that are small enough to pass through the filter. Consider a study that produces a p-value barely below 0.05. As with p = 0.052, the number may be transposed into a false sense of confidence (especially if it is praised as "statistically significant").

FURTHER READING

• David H. Kaye, David E. Bernstein & Jennifer L. Mnookin, The New Wigmore: A Treatise on Evidence -- Expert Evidence ch. 12 (2d ed. 2011, updated annually).

Last updated: Mar. 30, 2019, 8:55 AM

P-values in "the home for big data in biology"

A p-value indicates the significance of the difference in frequency of the allele tested between cases and controls i.e. the probability that the allele is likely to be associated with the trait. 1/

Such is the explanation of "p-value" provided by "the EMBL-EBI Training Programme [which] is celebrating 10 amazing years of providing onsite, offsite and online training in bioinformatics" for "scientists at all levels." 2/ The explanation appears in the answer to the question "What are genome wide association studies (GWAS)?" 3/ It comes from people in the know -- "The home for big data in biology" 4/ at "Europe's flagship laboratory for the life sciences." 5/

Above the description is a Manhattan plot of the p-values for the differences between the frequencies of the single nucleotide alleles comprising a genome-wide SNP array in samples of "cases" (individuals with a trait) and "controls" (individuals without the trait). Sadly, the p-values in the plot cannot be equated to "the probability that the allele is likely to to be associated with the trait."

I. Transposition

To begin with, a p-value is the probability of a difference at least as large as the one observed in the sample of "cases" and the sample of "controls" -- when the probability that a random person has the allele is the same for both cases and controls in the entire population. For a single allele, the expected value of the difference in the sample proportions is zero, but the observed value will vary from one pair of samples to the next. Because most pairs will have small differences, a big difference in a single study is evidence against the hypothesis that there is no difference at the population level. Differences that rarely would occur by chance alone are indicative of a true association. They are not usually false positives. At least, that is the theory behind the p-value.

For example, a p-value of 1/100,000, is a rare occurrence when the two samples come from a population in which the probability of the trait is the same with or without the allele. Consequently, the difference that corresponds to this p-value is thought to be strong evidence that the allele really is more (or less) common among cases than controls.

But even if the reasoning that "we would not expect it if H is true, therefore H is likely to be false" is correct, the p-value of 1/100,000 is not "the probability that the allele is likely to to be associated with the trait." It is the probability of that sort of a discrepancy if the allele has absolutely no association with the trait. In contrast, the probability of "no association" is not even defined in the statistical framework that gives us p-values.

Another way to say it: The p-value is a statement about the evidence (the observed difference) given the hypothesis of no association. It does not represent the probability of the hypothesis of zero association given an observed association. Equating the probability associated with the evidence with the probability associated with the hypothesis is so common that it has a name -- the transposition fallacy.

II. Multiple Comparisons

A second defect of the EMBL-EBI's definition is that a p-value of, say, 1/100,000 is not a good measure of surprise for GWAS data. Suppose there are 500,000 SNPs in the array and none of their alleles has any true association with the trait. If all apparent associations are independent, the expected number with the p-value is 500,000 × (1/100,000) = 5. Because of the many opportunities for individually impressive differences to appear, it is no surprise that some alleles have this small p-value. The p-value would have to be much smaller than 1/100,000 for the apparent association to be as surprising as the reported p-value would suggest. P-values that would produce a reasonable false discovery rate could be very small indeed.

An oversimplified analogy 6/ is this: Flipping a fair coin 18 times and getting 18 heads or tails has a probability on the order of 1/100,000. 7/ Flipping 500,000 coins 18 times each and finding that some of these experiments yielded 18 heads or tails is not strong evidence against the proposition that all the coins are fair. It is just what we would expect to see if the coins are all fair.

Of course, the bioinformaticists at EMBL-EBI are acutely aware of the effect of multiple comparisons. Their catalog of findings only includes "variant-trait associations ... if they have a p-value <1.0 × 10^-5 in the overall (initial GWAS + replication) population." 8/ But why insist on so small a number if this p-value is "the probability that the allele is likely to to be associated with the trait"? For multiple comparisons, the p-value of 1/100,000 is not the measure of surprise that it is supposed to be.

NOTES

1. EMBL-EBI, What Are Genome Wide Association Studies (GWAS)?, https://www.ebi.ac.uk/training/online/course/gwas-catalog-exploring-snp-trait-associations/why-do-we-need-gwas-catalog/what-are-genom, last visited Sept. 16, 2018.
2. EMBL-EBI, EMBLI-EBI Training, https://www.ebi.ac.uk/training/online/course/gwas-catalog-exploring-snp-trait-associations/why-do-we-need-gwas-catalog/what-are-genome, last visited Sept. 16, 2018.
3. EMBL-EBI, supra note 1.
4. EMBL-EBI, https://www.ebi.ac.uk/, last visited Sept. 16, 2017.
5. Id. 
6. It is oversimplified because not all associations are independent. Nearby SNPs tend to be inherited together, but methods that take account of dependencies and enable researchers to pick out the associations that are real have been studied. 
7. The more exact probability is 1/131,072.
8. EMBL-EBI, Where Does the Data Come From?, https://www.ebi.ac.uk/training/online/course/gwas-catalog-exploring-snp-trait-associations/what-gwas-catalog/where-does-data-come, last visited Sept. 17, 2018. The phrase "the overall (initial GWAS + replication) population" is puzzling. It sounds like the data from an exploratory study are combined with those from the replication study to give a p-value for a larger sample (not a population). If so, the p-values for each study could be more than 1/100,000.

Overvaluing P-values

Seventy-two "big names in statistics want to shake up [the] much maligned P-value." 1/ These academic researchers give the following one-sentence summary of their proposal to change the way scientific articles are written:

We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005. 2/

The President’s Council of Advisors on Science and Technology (PCAST) effectively invoked the current "default P-value" of 0.05 as a rule for admitting scientific evidence in court. 3/ In light of this new (but not novel) call for reducing the conventional p-value, one might think that PCAST was being too generous toward forensic-science identifications. After all, if 5% is ten times the value that should be used to declare differences “statistically significant” in scientific research, then it seems way too large as a limit for what PCAST called “scientific reliability” for courtroom use. But that conclusion would rest on a misunderstanding of the objective and nature of the proposal to change the nomenclature for p-values.

The motivation for moving from 0.05 to 0.005 is “growing concern over the credibility of claims of new discoveries based on ‘statistically significant’ findings.” The authors argue that regarding 0.05 as proof of a real difference (a true positive) is “a leading cause of non-reproducibility” of published discoveries in scientific fields that could easily be corrected by referring to findings with p-values between 0.005 and 0.05 as “suggestive” rather than “significant.” The problem with the verbal tag of “statistically significant” (p < 0.05) is that, in comparison to the state of scientific research ninety years ago when Sir Ronald Fisher floated the 0.05 level, “[a] much larger pool of scientists are now asking a much larger number of questions, possibly with much lower prior odds of success,” resulting in too many apparent discoveries that cannot be replicated in later experiments.

Not only is the group of 72 addressing the perils of the p-value in a different context, but their proposal is not intended as a bright-line rule for deciding what to publish. They explain:

We emphasize that this proposal is about standards of evidence, not standards for policy action nor standards for publication. Results that do not reach the threshold for statistical significance (whatever it is) can still be important and merit publication in leading journals if they address important research questions with rigorous methods. This proposal should not be used to reject publications of novel findings with 0.005 < P < 0.05 properly labeled as suggestive evidence.

So too, “[r]esults that do not reach the threshold for statistical significance (whatever it is) can still be important” in litigation, and the desire to shake things up in the research community does not reveal much about appropriate standards for admissibility in court.

However, PCAST is on firm ground in emphasizing the need to present forensic-science findings without overstating their probative value. The 72 researchers focus on probative value when they discuss a “more direct measure of the strength of evidence.” They suggest that a “two-sided P-value of 0.05 [often] corresponds to Bayes factors ... that range from about 2.5 to 3.4.” Such evidence, they note, is weak. In contrast, they defend the "two-sided P-value of 0.005" in part on the ground that it "corresponds to Bayes factors between approximately 14 and 26." As such, it "represents ‘substantial’ to ‘strong’ evidence according to conventional Bayes factor classifications."

Forensic scientists who advocate describing the strength of evidence rather than only false-positive rates are more demanding. They usually consider Bayes factors between 10 and 100 to constitute "moderate" rather than “strong” evidence. 4/

Notes

1. Dalmeet Singh Chawla, Big Names in Statistics Want To Shake Up Much-maligned P value, Nature News, July 27, 2017, http://www.nature.com/news/big-names-in-statistics-want-to-shake-up-much-maligned-p-value-1.22375
2. Daniel J. Benjamin et al., Redefine Statistical Significance (2017), PsyArXiv. July 22. https://osf.io/preprints/psyarxiv/mky9j (forthcoming in Nature Human Behavior).
3. For discussion, see The Source and Soundness of PCAST's 5% Rule, Forensic Sci., Stat. & L., July 23, 2017, http://for-sci-law.blogspot.com/2017/07/the-source-and-soundness-of-pcasts-5.html.
4. E.g., R. Marquis et al., Discussion on How to Implement a Verbal Scale in a Forensic Laboratory: Benefits, Pitfalls and Suggestions to Avoid Misunderstandings, 56(5) Sci. & Just. 364-70 (2016), doi: 10.1016/j.scijus.2016.05.009, preprint available at http://wp.unil.ch/forensicdecision/files/2015/01/1-s2.0-S1355030616300338-main.pdf (Appendix A).

Index to comments and cases on the PCAST report

The Source and Soundness of PCAST's 5% Rule

The President’s Council of Advisors on Science and Technology (PCAST) Report on comparative pattern matching in forensic science has a deceptively simple rule for the admissibility of evidence of a match between a questioned and a known sample: if examiners would declare that the two samples have the same source as often as one time in 20 when analyzing pairs of samples actually that come from different samples, then the comparisons are “scientifically unreliable.” The report gives no explanation of how it arrived at this rule beyond the following enigmatic paragraph: 1/

False positive rate (abbreviated FPR) is defined as the probability that the method declares a match between two samples that are from different sources (again in an appropriate population), that is, FPR = P(M|H0). For example, a value FPR = 0.01 would indicate that two samples from different sources will be (mistakenly) called as a match 1 percent of the time. Methods with a high FPR are scientifically unreliable for making important judgments in court about the source of a sample. To be considered reliable, the FPR should certainly be less than 5 percent and it may be appropriate that it be considerably lower, depending on the intended application. 2/

Five percent has a crisp, authoritative ring to it, but why is 5% “certainly” the maximum tolerable FPR for courtroom use of the test? And what “intended applications” would demand a lower FPR? Is the underlying thought that greater “scientific reliability” is required as the gravity of the case increases—from a civil case, to a misdemeanor, to a major crime, on up to a capital case?

Statistical Practice as the Basis for the 5% Rule

Inasmuch as the paragraph is found in an appendix entitled "statistical issues," we should expect statistical concepts and practice to help answer such questions. And in fact, 5% is a common number in statistics. In many applications, statistical hypothesis tests try to keep the risk of a false rejection of the “null hypothesis” H0—a false-positive conclusion—below 5%. Researchers and journal editors in many fields prize results that can be said to be “statistically significant,” usually at the 0.05 level or better. The expression p < 0.05 is therefore a common accoutrement of experimental or observational results indicating an association between variables. Likewise, the Food and Drug Administration demands clinical trials to show that a new drug is effective for its intended use (“validity,” if you will), with “the typical ‘cap’ on the type I [false positive] error rate ... set at 5% .”3/ In the forensic pattern-matching context, the null hypothesis H0 in the PCAST paragraph would be that a questioned and a known sample are not associated with the same source.

Thus, to the extent PCAST was thinking of the 5% FPR as the significance level required to reject H0, its emphasis on 5% is well grounded in statistical practice. Using certain standard levels of significance, particularly 5%, can be traced to the 1920s. The eminent British statistician Sir R. A. Fisher wrote:

It is convenient to draw the line at about the level at which we can say: ‘Either there is somethng in the treatment, or a coincidence has occurred such as does not occur more than once in twenty trials.’ ... If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach that level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance. 4/

For FPRs larger than 5%, the reports of criminalists do not meet (Fisher’s) criterion for establishing a “scientific fact.” Their conclusions of positive association for such error-prone procedures are not, in PCAST’s words, “scientifically reliable.”

Having equated PCAST’s unexplained choice of 5% with a common implementation of statistical hypothesis testing, we also can see why the report suggested that a “considerably lower” number might be required for scientific “reliability.” A 5% FPR lets in examiner conclusions that might be wrong about one time in twenty when defendants are innocent and there is no true association between the questioned item and the known one. False positives tend to increase the rate of false convictions, whereas false negatives tend to would increase the rate of false acquittals. The norm that false convictions are worse than false acquittals counsels caution in relying on an examiner’s conclusion to convict a defendant. And if false convictions in the most serious of cases are worse still, we can see why the PCAST report stated that “the FPR should certainly be less than 5 percent and it may be appropriate that it be considerably lower, depending on the intended application.” Five percent may be good enough for an editor to publish an interesting paper purporting to have discovered something new in social psychology, but this scientific convention does not mean that 5% is good enough for a criminal conviction, let alone one that would lead to an execution.

So we can see that PCAST’s 5% figure did not come from thin air. Indeed, some statisticians and psychologists think that it is too weak a standard—that the general rule in science ought to be p < 0.005. 5/ Nevertheless, the general use of the arguably lenient 5% significance level does not establish that the 5% rule is legally compelled. The law incorporates the intensified concern for false positives into the burden of persuasion for the evidence as a whole. The jury is instructed to acquit unless it has no reasonable doubt that a defendant in a criminal case is guilty; in contrast, in a civil case, the plaintiff can prevail on a mere preponderance of the evidence. But these burdens do not apply to individual items of evidence. The standard for admitting scientific—and other—evidence does not change with how much is at stake in the particular case. After all, the probative value of scientific evidence is no different in a criminal case than in a civil one. Although the PCAST report insists that its statements about science are merely designed to inform courts about scientific standards, if “scientific reliability” depends on the “importance” of the “judgments in court” and varies according to “the intended application,” then PCAST's "scientific reliability" turns out to be based on what is considered socially or legally “appropriate.”

Beyond the FPR

In sum, it is (or would have been) fair for PCAST to point out that it is uncommon for results at higher significance levels than 0.05 to be credited in the scientific literature. But a more deeply analytical report would have noted that there is uneasiness in the statistical community with the hypothesis testing framework and particularly with over-reliance on the p < 0.05 rule. (Today's mail includes an invitation to attend a "Symposium on Statistical Inference: A World Beyond p < 0.05" sponsored by the American Statistical Association.)

Only part of the world beyond p < 0.05 comes from the fact that the FPR is not the only quantity that determines “scientific reliability.” Superficially, the false-positive error probability might look like the appropriate statistic for considering the probative value of a positive finding, but that cannot be right. Scientific evidence, like all circumstantial evidence, has probative value to the extent it changes the probability of a material fact. That there is much more to probative value than the FPR therefore is easily seen through the lens of Bayes’ rule. As the PCAST report notes, in this context, Bayes' theorem prescribes how probability or odds change with the introduction of evidence. The odds after learning of the examiner’s finding are the odds without that information multiplied by the Bayes factor: posterior odds = prior odds × BF.

The Bayes factor thus indicates the strength of the evidence. Stronger evidence has a larger BF and hence a greater impact on the prior odds than weaker evidence. The Bayes factor is a simple ratio. The FPR appears as the denominator, and the sensitivity—or true positive rate—forms the numerator. In symbols, BF = sensitivity / FPR.

The report acknowledges that sensitivity matters (for some purposes at least). Earlier, the report states that “[i]t is necessary to have appropriate empirical measurements of a method’s false positive rate and the method’s sensitivity. [I]t is necessary to know these two measures to assess the probative value of a method.” 6/  Because it takes both operating characteristics to express the probative value of the test, PCAST cannot sensibly dismiss a test as having so little probative value as to be considered “scientifically reliable” on the basis of only one number. Realizing this prompts the next question for devising a rule in the spirit of PCAST's—namely, what is the sensitivity that, together with an FPR of 5%, would define the threshold for “scientific reliability”?

One might imagine that PCAST would consider any false-negative rate in excess of 5% as too high. 7/ If so, it follows that the scientists are saying that, in their view of what is important or what is the dominant convention in various domains, subjective pattern matching must shift the prior odds by a factor of at least .95/.05 = 19 to be considered “scientifically reliable.” On the other hand, if the scientists on PCAST think it is appropriate for a false-negative probability to be ten times the maximum acceptable false-positive probability, then their minimum for “reliability” would become a FNR of 50% and a FPR of 5%, for a Bayes’ factor of only ten.

What Does the Law Require?

Whether the cutoff comes from the FPR alone or the more complete Bayes factor, the very notion of a sharp cutoff is questionable. The purpose of a forensic-science test for identity is to provide evidence that will assist judges or jurors. Forensic scientists who present results and reasonable estimates of the likelihoods or conditional error probabilities associated with their conclusions are staying within the bounds of what is scientifically known.

Consider a hypothetical pattern-matching test for identity for which FPR = 10% and sensitivity = 70% as shown by extensive experiments, each of which demonstrates an ability to distinguish sources from nonsources with accuracy above what would be expected by chance (p < 0.05). According to the PCAST report, this test would be inadmissible for want of “scientific reliability” or “foundational validity” because the FPR of 10% is too high. But if this were a test for a disease, would we really want a diagnosing physician to ignore the positive test result just because the FPR is greater than 5%? The positive finding from the lab would raise the prior odds from, say, 1 to 2, to 7 to 2 (corresponding to an increase in probability from 33% to 78%). Like the physician trying to reach the best possible diagnosis, the judge or jury trying to reach the best possible reconstruction of the events could benefit from knowing that an examiner, who can perform at the empirically established level of accuracy, has found a positive association.

The logic behind a high hurdle for scientific evidence is that “it is likely to be shrouded with an aura of near infallibility, akin to the ancient oracle of Delphi.” 8/ As one federal judge (an advisor to PCAST) wrote in excluding the testimony of a handwriting expert:

[I]t is the Court's role to ensure that a given discipline does not falsely lay claim to the mantle of science, cloaking itself with the aura of unassailability that the imprimatur of ‘science’ confers and thereby distorting the truth-finding process. There have been too many pseudo-scientific disciplines that have since been exposed as profoundly flawed, unreliable, or baseless for any Court to take this role lightly. 9/

Under this rationale, a court should be able to admit the positive test result if the jury is informed of and can appreciate the limitations of the finding. A result that is ten time more probable when the samples have the reported source than when they have different sources is not unreliable “junk science.” Of course, it may not be the product of a particularly scientific (or even a very standardized) procedure, and that must be made clear to the factfinder. When the criminalists employing the highly subjective procedure truly have specialized knowledge—as evidenced by rigorous and repeated tests of their ability to arrive at correct answers—their findings can be presented along with their known error rates without creating “an aura of near infallibility.” If this view of what judges and juries can understand is correct, then a blanket rule against all expert evidence that has known error rates in excess of 5% is unsound.

This criticism of PCAST's 5% rule does not reject the main theme of the report—that when a forensic identification procedure relies on a vaguely defined judgmental process (such as "sufficient similarities and explicable dissimilarities in the light of the examiner's training and experience"), well-founded estimates of the ability of examiners to make the correct judgments are vital to admitting source attributions in court. Of course, Daubert v. Merrell Pharmaceuticals 9/ did not make any single factor, including a "known or potential rate of error," absolutely necessary for admitting all types of scientific evidence. But the Daubert Court painted with an amazingly broad brush. The considerations that will be most important can vary from one type of evidence to another.  When it comes to source attributions from entirely subjective assessments of the similarities and differences in feature sets, there is a cogent argument that the only acceptable way to validate the psychological process is to study how often examiners reach the right conclusions when confronted with same-source and different-source samples.

Notes

1. Thanks to Ken Melson for calling to my attention to this paragraph.
2. PCAST Report at 161-52.
3. Russell Katz, FDA: Evidentiary Standards for Drug Development and Approval, 1(3) NeuroRx 307–316, (2004), doi: 10.1602/neurorx.1.3.307.
4. R.A. Fisher, The Arrangement of Field Experiments, 33 J. Ministry Agric. Gr. Brit. 504 (1926), as quoted in L. Savage, On Rereading R.A. Fisher, 4 Annals of Statistics 471 (1976).
5. Kelly Servick, It Will Be Much Harder To Call New Findings ‘Significant’ If This Team Gets Its Way, Jul. 25, 2017, 2:30 PM, Science, DOI: 10.1126/science.aan7154.
6. PCAST Report at 50 (emphasis added).
7. However, the report made no mention of the fact that the false-negative rate was higher than that in at least one of the two experiments on latent print identification of which it approved.
8. United States v. Alexander, 526 F.2d 161, 168 (8th Cir. 1975).
9. Almeciga v. Center for Investigative Reporting, Inc., 185 F. Supp. 3d 401, 415 (S.D.N.Y. 2016) (Rakoff, J.).
10. 509 U.S. 579 (1993).

Index to comments and cases on the PCAST report

Multiple Hypothesis Testing in Karlo v. Pittsburgh Glass Works

The following posting is adapted from a draft of an annual update to the legal treatise The New Wigmore on Evidence: Expert Evidence. I am not sure of the implications of the calculations in note 23 and the fact that the age-based groups are overlapping. Advice is welcome.

The Age Discrimination in Employment Act of 1967 (ADEA) 1/ covers individuals who are at least forty years old. The federal circuit courts are split as to whether a disparate-impact claim is viable when it is limited to a subgroup of employees such as those aged fifty and older. In Karlo v. Pittsburgh Glass Works, 2/ the Third Circuit held that statistical proof of disparate impact on such a subgroup can support a claim for recovery. The court countered the employer’s argument that “plaintiffs will be able to ‘gerrymander’ arbitrary age groups in order to manufacture a statistically significant effect” 3/ by promising that “the Federal Rules of Evidence and Daubert jurisprudence [are] a sufficient safeguard against the menace of unscientific methods and manipulative statistics.” 4/ In Daubert v. Merrell Dow Pharmaceuticals, the Supreme Court famously reminded trial judges applying the Federal Rules of Evidence that they are gatekeepers responsible for ensuring that scientific evidence presented at trials is based on sound science. By the end of the Karlo opinion, however, the court appeals held that the Senior District Judge Terrence F. McVerry had been too vigorous a gatekeeper when he found inadmissible a statistical analysis of reductions in force offered by laid-off older workers.

The basic problem was that plaintiffs claimed to have observed statistically significant disparities in various overlapping age groups without correcting for the fact that by performing a series of hypothesis tests, they had more than one opportunity to discover something "significant." By way of analogy, if you flip a coin five times and observe five heads, you might begin to suspect that the coin is not fair. The probability of five heads in a row with a fair coin is p = (1/2)⁵ = 1/32 = 0.03. We can say that the five heads in the sample are "statistically significant" proof (at the conventional 0.05 level) that the coin is unfair.

But suppose you get to repeat the experiment five times. Now the probability of at least one sample of 5 flips with 5 heads is about five times larger. It is 1 - (1 - 1/32)⁵ = 0.146785, to be exact. This outcome is not so far out line with what is expected of a fair coin. It would be seen about 15% of the time for a fair coin. This is weak evidence that the coin is unfair; certainly, it is not as compelling as the 3% p-value. So the extra testing, with the opportunity to select any one or more of the five samples as proof of unfairness, has reduced the weight of the statistical evidence of unfairness. The effect of the opportunity to search for significance is sometimes known as "selection bias" or, of late, "p-hacking."

In Karlo, Dr. Michael Campion—a distinguished professor of management at Purdue University with degrees in industrial and organizational psychology—compared proportions of Pittsburgh Glass workers older than 40, 45, 50, 55, and 60 who were laid off to the proportion of younger workers who were laid off. He found that the disparities in three of the five categories were statistically significant at the 0.05 level. 5/ The disparity for the 40-and-older range, he said, fell “just short,” being “ significant at the 13% level.” Dr. Campion maintained that “[t]hese results suggest that there is evidence of disparate impact.” 6/ He also misconstrued the 0.05 level as “a 95% probability that the difference in termination rates of the subgroups is [] due to chance alone.” 7/ The district court expressed doubt as to whether Dr. Campion was a qualified statistical expert 8/ and excluded the testimony under Daubert as inadequate “data snooping.” 9/

Apparently, Judge McVerry was more impressed with the report of Defendant’s expert, James Rosenberger — a statistics professor at Pennsylvania State University and a fellow of the American Statistical Association and the American Association for the Advancement of Science. The report advocated adjusting the significance level to account for the five groupings of over-40 workers. The Chief Judge of the Third Circuit, D. Brooks Smith (also an adjunct professor at Penn State), described the recommended correction as follows:

The Bonferroni procedure adjusts for that risk [of a false positive] by dividing the “critical” significance level by the number of comparisons tested. In this case, PGW's rebuttal expert, Dr. James L. Rosenberger, argues that the critical significance level should be p < 0.01, rather than the typical p < 0.05, because Dr. Campion tested five age groups (0.05 / 5 = 0.01). Once the Bonferroni adjustment is applied, Dr. Campion's results are not statistically significant. Thus, Dr. Rosenberger argues that Dr. Campion cannot reject the null hypothesis and report evidence of disparate impact. 10/

Another way to apply the Bonferroni correction is to change the p-value. That is, when M independent comparisons have been conducted, the Bonferroni correction is either to set “the critical significance level . . . at 0.05/M” (as Professor Rosenberger recommended) or “to inflate all the calculated P values by a factor of M before considering against the conventional critical P value (for example, 0.05).” 11/

The Court of Appeals was not so sure that this conservative adjustment was essential to the admissibility of the p-values or assertions of statistical significance. It held that the district court erred in excluding the subgroup analysis and granting summary judgment. It remanded “for further Daubert proceedings regarding plaintiffs' statistical evidence.” 12/ Further proceedings were said to be necessary partly because the district court had applied “an incorrectly rigorous standard for reliability.” 13/ The lower court had set “a higher bar than what Rule 702 demands” 14/ because “it applied a bright-line exclusionary rule” for all studies with multiple comparisons that have no Bonferroni correction. 15/

But the district court did not clearly articulate such a rule. It wrote that “Dr. Campion does not apply any of the generally accepted statistical procedures (i.e., the Bonferroni procedure) to correct his results for the likelihood of a false indication of significance.” 16/ The sentence is grammatically defective (and hence confusing). On the one hand, it refers to "generally accepted statistical procedures." On the other hand, the parenthetical phrase suggests that only one "procedure" exists. Had the district court written “e.g.” instead of “i.e.,” it would have been clear that it was not promulgating a dubious rule that only the Bonferroni adjustment to p-values or significance levels would satisfy Daubert. To borrow from Mark Twain, "the difference between the almost right word and the right word is really a large matter—'tis the difference between the lightning-bug and the lightning." 17/

Understanding the district court to be demanding a Bonferroni correction in all cases of multiple testing, the court of appeals essentially directed it to reconsider its exclusionary ruling in light of the fact that other procedures could be superior. Indeed, there are many adjustment methods in common use, of which Bonferroni’s is merely the simplest. 18/ However, plaintiff’s expert apparently had no other method to offer, which makes it hard to see why the possibility of some alternative adjustment, suggested by neither expert in the case, made the district court's decision to exclude Dr. Campion's proposed testimony an abuse of discretion.

A rule insisting on a suitable response to the multiple-comparison problem does not seem “incorrectly rigorous.” To the contrary, statisticians usually agree that “the proper use of P values requires that they be ... appropriately adjusted for multiple testing when present.” 19/ It is widely understood that when multiple comparisons are made, reported p-values will exaggerate the significance of the test statistic. 20/ The court of appeal’s statement that “[i]n certain cases, failure to perform a statistical adjustment may simply diminish the weight of an expert's finding.” 21/ is therefore slightly misleading. In virtually all cases, multiple comparisons degrade the meaning of a p-value. Unless the statistical tests are all perfectly correlated, multiple comparisons always make the true probability of the disparity (or a larger one) under the model of pure chance greater than the nominal value. 22/

Even so, whether the fact that an unadjusted p-value exaggerates the weight of evidence invariably makes unadjusted p-values or reports of significance inadmissible under Daubert is a more delicate question. If no reasonable adjustment can be devised for the type of analysis used and no better analysis can be done, then the nominal p-values might be presented along with a cautionary statement about selection bias. In addition, in extreme cases, the adjustment will be small and the degree of exaggeration will not be so formidable as to render the unadjusted p-value inadmissible. For instance, if the nominal p-value were 0.001, the fact that the corrected figure is 0.005 would not be a fatal flaw. The disparity would be highly statistically significant even with the correction. But that was not the situation in Karlo. In this case, statistical significance was not apparent. It was undisputed that as soon as one considered the number of tests performed, not a single subgroup difference was significant at the 0.05 level. 23/

Consequently, the rejection of the district court’s conclusion that the particular statistical analysis in the expert’s report was unsound seems harsh. It should be within the trial court’s discretion to prevent an expert from testifying to the statistical significance of disparities (or their p-values) unless the expert avoids multiple comparisons that would seriously degrade the claims of significance or modifies those claims to reflect the negative impact of the repeated tests on the strength of the statistical evidence. 24/ The logic of Daubert does not allow an expert to dismiss the problem of selection bias on the theory -- advanced by plaintiffs in Karlo -- that “adjusting the required significance level [is only] required [when the analyst performs] ‘a huge number of analyses of all possibilities to try to find something significant.'’’ 25/ The threat to the correct interpretation of a significance probability does not necessarily disappear when the number of comparisons is moderate rather than “huge.” Given the lack of highly significant results here (even nominally), it is not statistically acceptable to ignore the threat. 26/ Although the Third Circuit was correct to observe that not all statistical imperfections render studies invalid within the meaning of Daubert, the reasoning offered in support of the claim of significant disparities in Karlo was not statistically acceptable. 27/

Notes

l. 29 U.S.C. §§ 621–634.
2. 849 F.3d 61 (3d Cir. 2017).
3. Id. at 76.
4. Id.
5. He testified that he did not compute a z-score (a way to analyze the difference between two proportions when the sample sizes are large) for the 60-and-over group “because ‘[t]here are only 14 terminations, which means the statistical power to detect a significant effect is very low.’” Karlo, 849 F.2d at 82 n.15.
6. Karlo v. Pittsburgh Glass Works, LLC, 2015 WL 4232600, at *11, No. 2:10–cv–1283 (W.D. Penn. July 13, 2015), vacated, 849 F.3d 61 (3d Cir. 2017).
7. Id. at *11 n.13. "A P value measures a sample's compatibility with a hypothesis, not the truth of the hypothesis." Naomi Altman & Martin Krzywinski, Points of Significance: Interpreting P values, 14 Nature Methods 213, 213 (2017).
8. Id. at *12.
9. Id. at *13.
10. 849 F.3d at 82 (notes omitted).
11. Pak C. Sham & Shaun M. Purcell, Statistical Power and Significance Testing in Large-scale Genetic Studies, 15 Nature Reviews Genetics 335 (2014) (Box 3).
12. Id. at 80 (note omitted).
13. Id. at 82.
14. Id at 83.
15. Id. (internal quotation marks and ellipsis deleted).
16. Karlo, 2015 WL 4232600, at *1.
17. George Bainton, The Art of Authorship 87–88 (1890.
18. Martin Krzywinski & Naomi Altman, Points of Significance: Comparing Samples — Part II, 11 Nature Methods 355, 355 (2014)
19. Naomi Altman & Martin Krzywinski, Points of Significance: Interpreting P values, 14 Nature Methods 213, 214 (2017)
20. Krzywinski & Altman, supra note 18
21. Id. at 83 (emphasis added).
22. Because each age group included some of the same older workers, the tests here were not completely independent. But neither were they completely dependent.
23. However, that three out of five groups exhibited significant associations between age and terminations is surprising under the null hypothesis that those variables are uncorrelated. If each test were independent, then the probability of a significant result in each group would be 0.05. The probability of one or more significant results in five tests would be 0.226; that of two or more would be 0.0226; of three or more, 0.00116.
24. Joseph Gastwirth, Case Comment: An Expert's Report Criticizing Plaintiff's Failure to Account for Multiple Comparisons Is Deemed Admissible in EEOC v. Autozone, 7 Law, Probability & Risk 61, 62 (2008).
25. Karlo, 849 F.3d at 82.
26. Dr. Campion also believed that “his method [was] analogous to ‘cross-validating the relationship between age and termination at different cut-offs,’ or ‘replication with different samples.’” Id. at 83. Although the court of appeals seemed to take these assertions at face value, cross-validation involves applying the same statistical model to different data sets (or distinct subsets of one larger data set). For instance, a equation that predicts law school grades as a function of such variables as undergraduate grades and LSAT test scores might be derived from one data set, then checked to ensure that it performs well in an independent data set. Findings in one large data set of statistically significant associations between particular genetic loci and a disease could be checked to see if the associations were present in an independent data set. No such validation or replication was performed in this case.
27. The Karlo opinion suggested that the state of statistical knowledge or practice might be different in social science than in the broader statistical community. The court pointed to a statement (in a footnote on regression coefficients) in a treatise on statistical evidence in discrimination cases that “the Bonferroni adjustment [is] ‘good statistical practice,’ but ‘not widely or consistently adopted’ in the behavioral and social sciences.” Id. (quoting Ramona L. Paetzold & Steve L. Willborn, The Statistics of Discrimination: Using Statistical Evidence in Discrimination Cases § 6:7, at 308 n.2 (2016 Update)). The treatise writers were referring to an unreported case in which the district court found itself unable to resolve the apparent conflict between the generally recognized problem of multiple comparisons and an EEOC expert’s insistence that labor economists do not make such corrections and courts do not require them. E.E.O.C. v. Autozone, Inc., No. 00-2923, 2006 WL 2524093, at *4 (W.D. Tenn. Aug. 29, 2006). In the face of these divergent perceptions, the district judge decided not to grant summary judgment just because of this problem. Id. (“[T]he Court does not have a sufficient basis to find that ... the non-utilization [of the Bonferroni adjustment] makes [the expert's] results unreliable.”). The notion that multiple comparisons generally can be ignored in labor economics or employment discrimination cases is false, Gastwirth, supra note 23, at 62 (“In fact, combination methods and other procedures that reduce the number of individual tests used to analyse data in equal employment cases are basic statistical procedures that have been used to analyse data in discrimination cases.”), and any tendency to overlook multiple comparisons in “behavioral and social science” more generally is statistically indefensible.
That said, the outcome on appeal in Karlo might be defended as a pragmatic response to the lower court's misunderstanding of the meaning of the ADEA. The court excluded the unadjusted findings of significance for several reasons. In addition to criticizing Professor Campion's refusal to make any adjustment for his series of hypothesis tests across age groups, Judge McVerry noted that "the subgrouping analysis would only be helpful to the factfinder if this Court held that Plaintiffs could maintain an over-fifty disparate impact claim." Karlo, 2015 WL 4232600, at *13 n.16. He sided with "the majority view amongst the circuits that have considered this issue ... that a disparate impact analysis must compare employees aged 40 and over with those 39 and younger ... ." Id. (Petruska v. Reckitt Benckiser, LLC, No. CIV.A. 14–03663 CCC, 2015 WL 1421908, at *6 (D.N.J. Mar.26, 2015)). The Third Circuit decisively rejected this construction of the ADEA, pulling this rug out from under the district court. Having held that the district court erred in interpreting the ADEA, requiring the district court to re-examine the statistical showing under the ADEA, correctly understood, might seem appropriate.
Of course, ordinarily an evidentiary ruling that can be supported on several independent grounds will be upheld on appeal as long as at least one of the independent grounds is valid. Here, the ADEA argument was literally a footnote to the independent ground that the failure to adjust for multiple comparisons invalidated the expert's claim of significant disparities. Nevertheless, the independent-grounds rule normally applies after a trial. It avoids retrials when the trial judge would or could rule the same way on retrial. Because Karlo is a summary judgment case, there is less reason to sustain the evidentiary ruling. But even so, the court of appeals did not have to vacate the judgment. Instead, it could have followed the usual independent-grounds rule to affirm the summary judgment while noting that district court could reconsider its Daubert ruling in light of the court of appeals' explanation of the proper reach of the ADEA and the range of statistically valid responses to the problem of multiple hypothesis tests. As a practical matter, however, there may be little difference between having counsel address the issue in the context of a motion to reconsider and a renewed motion for summary judgment.