Bayes Theorem, COVID-19, and False Positives

During the last week, there has been an upswing in discussions of Bayes Theorem regarding serotype testing for COVID-19. We’ll get to Bayes in a bit, but first, serological testing. Serological testing has the potential to identify who has been infected with SARS-CoV2 by examining their immune response, even if they haven’t shown any symptoms (asymptomatic), and, hopefully, should be unable to contract COVID-19 (though the extent and length of protection is unknown). These serotype positive people hopefully will be ‘safe.’

The problem is that none of these tests are perfect. The sensitivity–the probability that a serotype positive person correctly tests positive–and the specificity–the probability that a serotype negative person (i.e., hasn’t been infected) correctly tests negative–usually aren’t one hundred percent. Let’s walk through an example. Suppose I test 2,000 people, and being the All-Knowing Mad Biologist, I know that twenty of those people really are seropositive. Now, to keep things simple, suppose the sensitivity of the test is 100% (everyone who is seropositive is correctly identified) and the specificity of the test is 95% (five out of 100 people who are seronegative incorrectly test as seropositive). The specificity is especially important because we don’t want to accidentally claim someone is seropositive (and thus could be safe from COVID-19) when they’re not.

In my example, there are twenty real positives. But there are also five percent of the remaining 1,980 seronegative people who falsely test positive: 99 false positives. That means only 20 out 119 (~17% or roughly one out of six) are actually protected from COVID-19. That’s not good.

To be clear, this isn’t some stunning HOT TAKE the Mad Biologist has had–Bayes developed this argument in 1763. But I think it might be overapplied given what we know–and, importantly, don’t know about the validation of these tests. There are a bunch of serology tests, but, for most, there has been no release of data or the testing regime. For some, the sample sizes are ludicrously small, such as a claimed specificity of 100% based on a sample size of fourteen.

The one test with the most publicly available data and a description of the experimental methods that I have been able to find, by Cellex, has a sensitivity of 93.8% and a specificity of 96%. The problem is that each sample was tested only once (I imagine this is an issue for the studies with n = 14 as well). What we don’t know is if those errors are reproducible. That is, we can imagine two failure modes. One is that false positive people are always false positive. If I test them three days in a row, I’ll get three negatives*. The other scenario is that the day-to-day performance of the test is variable.**

In the first scenario, we’re stuck. A serotype test, once COVID-19 becomes common, will be pretty good, but not near-certain proof for any one person. The second scenario provides more hope, because we can test people multiple times, and be much more likely to get the right answer. Suppose we say that, to be cleared as seropositive, you have to have at least two out of three tests be positive. If we think of that as an ‘effective specificity’, then the probability that a seronegative person tests positive at least two out of three times is only 0.47%.*** That’s much better! And the rare seronegative person who ‘sneaks through’ will be surrounded by seropositive people, and could be protected by herd immunity.

So, if we’re going to use serotyping effectively, we need to understand how these tests fail, and if they fail ‘in the right way’, then test people multiple times.

*There might be a very, very low false positive rate due to some kind of random error, but the point is that the underlying biology makes the test very stereotypical.

**Of course, both options could be operating, where some people are always negative, and others are affected by randomness. But that’s really hard, so we’ll ignore that for now…

***There is a one percent chance that someone who is actually seropositive would test negative two out of three times.

This entry was posted in COVID-19, Statistics. Bookmark the permalink.