How Do Forensic-science Tests Compare to Emergency COVID-19 Tests?

The Wall Street Journal recently reported that

At least 160 antibody tests for Covid-19 entered the U.S. market without previous FDA scrutiny on March 16, because the agency felt then that it was most important to get them to the public quickly. Accurate antibody testing is a potentially important tool for public-health officials assessing how extensively the coronavirus has swept through a region or state.
Now, the FDA will require test companies to submit an application for emergency-use authorization and require them to meet standards for accuracy. Tests will need to be found 90% “sensitive,” or able to detect coronavirus antibodies, and 95% “specific,” or able to avoid false positive results. \1/

How many test methods in forensic science have been shown to perform at or above these emergency levels? It is hard to say. For FDA-authorized tests, one can find the manufacturers' figures on the FDA's website, but for forensic-science tests, there is no such repository of information on the standards adopted by voluntary standards development organizations. The forensic-science test-method standards approved by consensus bodies such as the Academy Standards Board and ASTM Inc. rarely state the performance characteristics of these tests.

For the FDA's minimum operating characteristics of a yes-no test, the likelihood ratio for a positive result is Pr(+ | antibodies) / Pr(+ | no-antibodies) = 0.90/(1 − .95) = 18. The likelihood ratio for a negative result is Pr(− | no-antibodies) / Pr(− | antibodies) = .95/(1 − .90) = 9.5. In other words, a clean bill of health on a serological test with minimally acceptable performance would occur less than ten times as frequently for people with less than the detectable level of the virus as compared to people with detectable levels.

According to an Ad Hoc Working Group of the forensic Scientific Working Group on DNA Analysis Methods (SWGDAM), such a likelihood ratio may be described as providing "limited support." This description is near the lower end of a scale for likelihood ratios. These "verbal qualifiers" go from "uninformative" (L=1), to "limited" (2 to 99), "moderate" (100 to 999), "strong" (1,000 to 999,999), and, finally, "very strong" (1,000,000 or more). \2/

A more finely graded table appears "for illustration purposes" in an ENFSI [European Network of Forensic Science Institutes] Guideline for Evaluative Reporting in Forensic Science. The table classifies L = 9.5 as "weak support." \3/

NOTES

1. Thomas M. Burton, FDA Sets Standards for Coronavirus Antibody Tests in Crackdown on Fraud, Wall Street J., Updated May 4, 2020 8:24 pm ET, https://www.wsj.com/articles/fda-sets-standards-for-coronavirus-antibody-tests-in-crackdown-on-fraud-11588605373; see also Mark McCarty, FDA’s Stenzel Highlights Sensitivity, Specificity for COVID-19 Antibody Testing Antibodies Fighting Coronavirus, BioWorld, May 6, 2020, https://www.bioworld.com/articles/434912-fdas-stenzel-highlights-sensitivity-specificity-for-covid-19-antibody-testing ("the agency’s performance expectations for serological tests are that overall sensitivity is 90%, and overall specificity is at least 95%. The specificity level of 95% is applicable to each antibody isotype, assuming results for each isotype are broken out, and sensitivity should be 90% for IgG if reported. If IgM is reported separately, sensitivity should be at least 70%, but the list of tests suggests that not all the tests currently operating under the EAU meet those benchmarks."),
2. Recommendations of the SWGDAM Ad Hoc Working Group on Genotyping Results Reported as Likelihood Ratios, 2018, available via https://www.swgdam.org/publications.
3. ENFSI Guideline for Evaluative Reporting in Forensic Science, 2016, p. 17, http://enfsi.eu/wp-content/uploads/2016/09/m1_guideline.pdf.

Hazard Ratios and Heart Failure

Today’s big news in medicine is a new drug, designated LCZ696 by its manufacturer, Novartis. According to the New York Times, LCZ696 “has shown a striking efficacy in prolonging the lives of people with heart failure and could replace what has been the bedrock treatment for more than 20 years.” [1] Specifically, more than 8,400 patients in 47 countries enrolled in a randomized, double-blind experiment in which they received either LCZ696 or an ACE inhibitor called enalapril (in addition to whatever else their doctors prescribed).

The trial was halted after a median follow-up time of 27 months “ because the boundary for an overwhelming benefit with LCZ696 had been crossed.” [2] “By that point, 21.8 percent of those who received LCZ696 had died from a cardiovascular cause or had been hospitalized for worsening heart failure. That figure was 26.5 percent for those receiving enalapril. That represents a 20 percent relative reduction in risk using a statistical measure called the hazard ratio.” [1]

This is good news for patients (if the drug receives regulatory approval and performs as expected in practice). But the account in the Times poses a small statistical puzzle. How does the difference between 21.8 and 26.5 percentage points translate into “a 20 percent relative reduction in risk”? The average risk across patients dropped by 26.5 – 21.8 = 4.7 percentage points. This absolute reduction is appreciable, but 4.7 percentage points is not 20% of the original 26.5 percent risk of hospitalizations or deaths in the control group (4.7 / 26.5 = 17.7%). What accounts for the discrepancy?

The answer lies in the details of a technique known in biostatistics as survival analysis. The statistical technique is not limited to the analysis of death rates. It can be applied to all sorts of situations involving different times to some outcome. The outcome can be the overruling of a Supreme Court case, the firing of a worker, or the exoneration of a prison inmate sentenced to die, to pick a few examples from forensic statistics.

So what does the 20% “relative reduction in risk” cited in the Times article mean? Well, a hazard function is the probability that if you survive to a given time t (the event in question has not already occurred), you will survive in the next instant. A hazard ratio is the ratio of the hazard in the treatment group to the hazard in the control group at t. The heart failure study used an estimation procedure known as proportional hazards regression, which assumes that the hazard in one group is a constant proportion of the hazard in the other group. Under this assumption, in a clinical trial where death is the endpoint, the hazard ratio indicates the relative likelihood of death in treated versus control subjects at any given point in time.

Thus, unlike the ordinary relative risk discussed in many court opinions, the “hazard ratio” is not simply the proportion with a disease in an exposed group divided by the proportion in an unexposed group. In the LCZ696 study, the hazard ratio was 0.80, meaning that the probability that a randomly selected patient taking LCZ696 would die from or be hospitalized for heart failure the next day is 80% of the probability for a randomly selected patient taking enalapril. To put it another way, the probability of hospitalization or death tomorrow from heart failure drops by 20% when LCZ696 is substituted for enalapril.

Yet a third formulation is that the odds that a randomly selected patient treated with LCZ696 will be hospitalized or die sooner than a randomly selected control patient are 0.8 (to 1) — that's 4 to 5, corresponding to a probability of 4/9 = 44%. [3]

How long either patient can expect to live and avoid hospitalization from heart failure is another story. As one article on hazard ratios explains, “[t]he difference between hazard-based and time-based measures is analogous to the odds of winning a race and the margin of victory.” [3] By itself, the hazard ratio picks the winning horse (probably), but it does not give the number of lengths for its expected success.

References

1. Andrew Pollack, New Novartis Drug Effective in Treating Heart Failure, N.Y. Times, Aug.31, 2014, at A4
2. John J.V. McMurray et al., Angiotensin–Neprilysin Inhibition versus Enalapril in Heart Failure, New Engl. J. Med., Aug. 30, 2014
3. Spotswood L. Spruance et al., Hazard Ratio in Clinical Trials, 48 Antimicrobial Agents and Chemotherapy 2787 (2004)