Using Serological Tests for COVID-19

The next wave of tests developed in response to the COVID-19 pandemic are designed to detect antibodies in the blood of infected individuals. Some of the antibodies (known as IgM) are short-term. Others (IgG) hang around longer. Just how long -- and just how much immunity the antibodies confer -- remains to be determined, \1/ but one hope is that these immunological tests will enable public health authorities to identify individuals with immunity who could be freed from distancing or isolation rules.

Will there be a database of the names of the immune among us? "If I were emperor and had unlimited resources, I’d do serological testing on every person in this country as fast as I could,” said one ER physician interviewed by the Los Angeles Times. \2/ The article reports that "Germany may issue 'immunity certificates' allowing release from quarantine for those who show they have already been exposed and fought off the coronavirus."

According to Times reporters Anita Chabria and Emily Baumgaertner, the FDA has been "lax" in allowing "the sale of more than 40 serological tests, not requiring them to undergo a formal emergency-use approval process," creating "potential problems because there is no centralized collection of results, no uniformity of methods and slim evidence that the tests work with acceptable accuracy."

Of course, what accuracy is acceptable depends on the use to which the tests are put, and a formal but abbreviated EUA process is required for tests to be marketed to diagnose the disease. The FDA granted its first EUA for a serological test on April 1, 2020, to the North Carolina phamaceutcial company Cellex. \3/ Other countries have been using other tests (that probably are as well validated) for some time. \4/

Cellex's fact sheet for physicians, \5/ like the ones that accompany molecular diagnostics tests for the virus itself, contains no estimates of the sensitivity and specificity of the test -- either for the particular antibodies it is intended to identify, for the virus (SAR-CoV-2), or for the disease (COVID-19). With respect to disease diagnosis, the fact sheet states that a "positive test result with the qSARS-CoV-2 IgG/IgM Rapid Test indicates that antibodies to SARS-CoV-2 were detected, and the patient has potentially been exposed to COVID-19." \6/ Well, yes, a positive test result for the antibodies is a positive test result for the antibodies. But what is known about the sensitivity for antibody detection--how probable is it to have a positive test result when the antibodies actually are circulating in the bloodstream? As for false positives (which arise from imperfect specificity), the fact sheet merely announces that the "qSARS-CoV-2 lgG/lgM Rapid Test has been designed to minimize the likelihood of false positive test results" -- whatever that means. \7/

Moreover, Cellex stays away from saying much about the implications of correctly detected antibodies. To be sure, "[w]hen IgM antibodies are present, they can indicate that a patient has an active or recent infection with SARS-CoV-2," but once again,  the fact sheet does not state how strong this possible indication is. Instead, it only warns that

A positive result for IgM or IgG may not mean that a patient’s current symptoms are due to COVID-19 infection. Laboratory test results should always be considered in the context of clinical observations and epidemiological data in making a final diagnosis and patient management decisions.

The fact sheet also steers clear of quantitative statements for negative test results. It gives the definition of a negative test result -- "A negative test result with this test means that SARS-CoV-2 specific antibodies were not present in the specimen above the limit of detection." Then it adds:

However, a negative result does not rule out COVID-19 and should not be used as the sole basis for treatment, patient management decisions, or to rule out active infection. Patients tested early after infection may not have detectable IgM antibody despite active infection; in addition, not all patients will develop a detectable IgM and/or IgG response to SARS-CoV-2 infection. The absolute sensitivity of the qSARS-CoV-2 IgG/IgM Rapid test is unknown.

When diagnostic testing is negative, the possibility of a false negative result should be considered in the context of a patient's recent exposures and the presence of clinical signs and symptoms consistent with COVID-19. ... Direct testing for virus (e.g., PCR testing) should always be performed in any patient suspected of COVID-19, regardless of the qSARS-CoV-2 IgG/IgM Rapid test.

Despite the absence of quantitative information in the fact sheet, Cellex did perform a validity study, which it summarized in the package insert. \8/ In one experiment, fifty normal whole blood samples were spiked with positive serum (diluted to 1:100), and fifty others were spiked with negative serum at the same dilution. The result was that "[a]ll spiked samples were correctly identified by the test except for one of the negative samples, which was tested positive with the test. Thus, there was a 99% concordance rate with expected results when venous whole blood specimens are used." The two-by-two table for these results is

	No antibodies in serum	Antibodies in serum
Test –	49	0
Test +	1	50

The estimated sensitivity is therefore 50/50 = 1.00 (95% CI of 0.93 to 1), and the estimated specificity is 49/50 = 0.98 (95% CI of 0.89 to 1). \9/

A test of "clinical performance" compared (1) antibody-test outcomes on 98 serum or plasma samples from individuals who tested positive with a RT-PCR method for SARS-CoV-2 infection and who had mild or no symptoms with (2) antibody-test outcomes on 180 negative serum or plasma samples that had been collected before September 2019. The classifications were not as accurate.

	No Virus	Virus
Test –	174	7
Test +	6	91
Sensitivity	91/98 = 0.93 [0.86, 0.97]
Specificity	174/180 = 0.97 [0.93, 0.99]

An additional 30 samples were collected from hospitalized individuals who were clinically confirmed positive for SARS-CoV-2 infection and exhibited severe symptoms. These samples, along with 70 negative serum or plasma samples collected prior to September 2019 generated the following results: \10/

	NoVirus	Severely Ill
Test –	65	1
Test +	5	29
Sensitivity	29/30 = 0.97 [0.83, 0.999]
Specificity	65/70 = 0.93 [0.84, 0.98]

Although the population from which the samples came is not stated, let's assume that they are fully applicable to the U.S. population. How well would the serological test work if extended, as suggested earlier, to "every person in this country," or perhaps, to every person in the labor force? The Census Bureau estimates the July 2019 population at a bit under 300 million, and it estimates the labor force percentage from 2014-2018 at almost 64%, for a total of about 211 million workers. Let's round this off to 200 million. In widespread testing, the error probabilities would be higher than those obtained under laboratory conditions, but I'll use the ballpark sensitivity and specificity estimates of 0.95 from the tables above. \11/

Lets assume that 10% of the 200 million workers have been infected and are immune. This percentage is far higher than the percentage of infections based on the number of reported cases to date in the country (about 300,000), but the ratio of unreported to reported cases could well be on the order of 10 to 1. \12/ For this guess as to the prevalence of past infection, and under the optimistic assumption that every infection confers immunity, we have some 20 million immune workers and 180 million non-immune workers. Now we apply the sensitivity and specificity estimates of 0.95 = 19/20:

• Of the 20 million immune workers, 19/20, will test positive (antibodies detected). That is 19 million true positives.
• Of the 20 million immune workers, 1/20 will test negative. That is 1 million false negatives.
• Of the 180 million non-immune workers, 19/20 will test negative. That is 171 million true negatives.
• Of the 180 million non-immune workers, 1/20 will test positive. That is 9 million false positives.

The negatives (apparently non-immune workers) will continue to be excluded from the workplace. There will only be 1/(1 + 171) false exclusions -- about half a percent. That is pretty good.

The positives (apparently immune workers) will return to work. There will be 9/(9 + 19) = 9/28 -- about a third -- false inclusions. If these workers mistakenly believe that the blood test indicates a 95% probability of immunity (as they might if they were told that the false positive rate is only 5%), they will be sorely mistaken about how likely it is that they are immune. \13/

This arithmetic is a "mind-bending" instance of Bayes' rule in operation. \14/ Before testing, the probability of a randomly selected worker being immune was (assumed to be) 1/10, or 10%. A positive test result raises the probability of immunity to 19/28, or 68%. The odds have improved from 1:9 to about 2:1. More precisely, the posterior-to-prior-odds ratio is [19/(28 – 19)] / (1/9) = 19. This Bayes factor is just the positive likelihood ratio LR+ = (sensitivity) /  (1 – specificity) = 0.95 / 0.05 = 19. For any set of prior odds, the posterior odds are this same likelihood ratio times the stipulated prior odds, and any such test:

Posterior Odds = [(sensitivity) /  (1 – specificity)] × Prior Odds.

Proposals from economists and public policy analysts to relax employment limitations according to serological test results are emerging. \15/ To think through their merits, we would also need to consider the cost of the testing and the consequences of the false positives and negatives for individuals and the rest of society. However, in the Washington world of volatile decisionmaking, policies are likely to change before comprehensive and convincing analyses of their probable costs and benefits are available.

UPDATED: Note 14 added 6/20/20 9:15 AM ET

NOTES

1. The period of immunity could be short. An article in Scientific American reported that

   [I]mmunity functions on a continuum. ... Immunity to seasonal coronaviruses (such as those that cause common colds), for example, starts declining a couple of weeks after infection. And within a year, some people are vulnerable to reinfection. ... But studies of SARS-CoV—the virus that causes severe acute respiratory syndrome, or SARS, which shares a considerable amount of its genetic material with SARS-CoV-2—are more promising. Antibody testing shows SARS-CoV immunity peaks at around four months and offers protection for roughly two to three years. ...
   
   Even if the antibodies stick around in the body, however, it is not yet certain that they will prevent future infection. What we want, [Dawn Bowdish, a professor of pathology and molecular medicine and Canada Research Chair in Aging and Immunity at McMaster University in Ontario] says, are neutralizing antibodies. These are the proteins that reduce and prevent infection by binding to the part of a virus that connects to and “unlocks” host cells. ... In contrast, nonneutralizing antibodies still recognize parts of the pathogen, but they do not bind effectively and so do not prevent it from invading cells. ...
   
   Nevertheless, a few small studies of cells in laboratory dishes suggest that SARS-CoV-2 infection triggers the production of neutralizing antibodies. And animal studies indicate such antibodies do prevent reinfection, at least for a couple of weeks. ...

   Stacey McKenna, What Immunity to COVID-19 Really Means, Sci. Am. Apr. 10, 2020, https://www.scientificamerican.com/article/what-immunity-to-covid-19-really-means/. See also Marc Lipsitch, Who Is Immune to the Coronavirus?, N.Y. Times, Apr. 13, 2020, https://www.nytimes.com/2020/04/13/opinion/coronavirus-immunity.html (sketching existing information and the major gaps in what is known).
2. Anita Chabria & Emily Baumgaertner, A Coronavirus Immunity Test Is Essential for the U.S. But Will It Work?, L.A. Times, Apr. 2, 2020, 12:06 PM, https://www.latimes.com/science/story/2020-04-02/coronavirus-test-immunity-detection-accuracy
3. Letter of Apr. 1, 2020, from RADM Denise M. Hinton, Chief Scientist, Food and Drug Administration, to James X. Li, Chief Executive Officer, Cellex Inc., available at https://www.fda.gov/media/136622/download.
4. Apoorva Mandavilli, F.D.A. Approves First Coronavirus Antibody Test in U.S., N.Y. Times, Apr. 2, 2020, https://www.nytimes.com/2020/04/02/health/coronavirus-antibody-test.html.
5. Cellex, Fact Sheet for Healthcare Providers, qSARS-CoV-2 lgG/lgM Rapid Test– Cellex Inc., Apr. 1, 2020, available at https://www.fda.gov/media/136623/download
6. Presumably, "exposed to COVID-19" really means "exposed to the virus (SARS-CoV-2)."
7. For an effort to unpack the meaning of minimize, see  https://for-sci-law.blogspot.com/2020/04/he-or-she-tested-negative-or-positive.html. 
8. Cellex qSARS-CoV-2 IgG/IgM Rapid Test, Package Insert, available at https://www.fda.gov/media/136625/download
9. In an unstated number of tests and specimens, no reactivity to a number of other virus was observed, and spiking specimens with a variety of substances that might be found in blood plasma did not generate false negatives or positives. Id.
   UPDATE (June 11, 2020): The FDA has posted a summary of sensitivity, specificity, and predictive values for serologic tests that it has authorized for emergency use. It reports a sensitivity of 93.8% (120/128) and a specificity of 96.0% (240/250). U.S. Food & Drug Administration, EUA Authorized Serology Test Performance, June 9, 2020, https://www.fda.gov/medical-devices/emergency-situations-medical-devices/eua-authorized-serology-test-performance
10. The day of collection relative to the onset of illness was unknown. The package insert reports "a Positive Percent Agreement and Negative Percent Agreement of 93.75% (95% CI: 88.06-97.26%) and 96.40% (95% CI: 92.26-97.78%), respectively." Id.
11. To the extent that many business and government activities are considered essential and continue to employ their workers, I am also overstating the number of workers who have been idled by precautionary measures, but we'll be looking at ratios that do not depend on the absolute numbers.
12. See Kathleen M. Jagodnik, Forest Ray, Federico M. Giorgi & Alexander Lachmann, Correcting Under-reported COVID-19 Case Numbers: Estimating the True Scale of the Pandemic, doi: https://doi.org/10.1101/2020.03.14.20036178.
13. This overstates the impact of false negative test results because workers who already recovered from a clear case of COVID-19 will know that their probability of immunity is high notwithstanding the more general probability of a false inclusion.
14. A Nature weekly briefing of 6/19/20 noted that "A mathematical pitfall plagues antibody tests that are less than 100% accurate: the lower the infection rate, the more likely it is that a positive result is wrong. Scientific American explains, with a very handy graphic, how this mind-bending fact arises. (Scientific American | 3 min read)."
15. E.g., Aaron Edlin & Bryce Nesbitt, The ‘Certified Recovered’ from Covid-19 Could Lead the Economic Recovery, STAT, Apr. 6, 2020, https://www.statnews.com/2020/04/06/the-certified-recovered-from-covid-19-could-lead-the-economic-recovery/; David M. Studdert and Mark A. Hall, Disease Control, Civil Liberties, and Mass Testing — Calibrating Restrictions during the Covid-19 Pandemic, New Eng. J. Med., Apr. 9, 2020, DOI: 10.1056/NEJMp2007637.