Vaccine confidence and statistical evidence

It’s not every day that some maths outreach can conceivably help improve public health. In the wake(field?) of this ridiculous and dangerous change to the CDC’s vaccine messaging, here’s my attempt to contribute.

TLDR: In statistics we can never really prove that a real-world effect doesn’t exist (that there is exactly zero correlation between things). The US administration is knowingly asking for evidence that science can’t supply for any question – not just vaccines vs autism. But worldwide systematic surveys have (a) shown no statistical support for a link; (b) shown that any link (if one does exist) must be a small effect; and (c) there is no more reason to think that vaccines cause autism than there is to think that vaccines prevent it.

Vaccines are obviously not a guaranteed cause of autism. Millions of people have been vaccinated and don’t have autism. So we are in the position of asking “how much do vaccines increase the chance of autism?”

The way we ask this is often by looking, across lots of people, at:
(1) What’s the chance of having autism if someone is vaccinated?
(2) What’s the chance of having autism if someone isn’t vaccinated?

Then we divide (1) by (2), in other words, the chance of autism with vaccination divided by the chance of autism without vaccination. This gives us what is called the “odds ratio” (other stats like “hazard ratio” and “relative risk” are also used, but the differences aren’t important for this argument).

If we get exactly 1, there is no difference in the chance of autism with and without vaccination.

But here’s the problem – we will never get exactly 1 from any real experiment, even if vaccines are completely unrelated to autism. Even if autism and vaccination are unrelated, it’s overwhelmingly likely that we’ll happen to choose a set of vaccinated people and a set of unvaccinated that — by chance — have slightly different frequencies of autism.

Think of the alternative. Say we choose 100000 vaccinated and 100000 unvaccinated people. To get a odds ratio of exactly 1 we would need, by chance, exactly the same number of people in each group to have autism (for example, 1000 vaccinated and 1000 unvaccinated people). As soon as we have 1001 and 1000, for example, our odds ratio is no longer exactly 1.

Or imagine the following. We ask two people to toss the same coin ten times. One throws four heads, the other throws six heads. We shouldn’t use this as evidence that the coin’s behaviour depends on the person. Even when a simple explanation is true (the coin doesn’t depend on the person; autism doesn’t depend on vaccination), we can still see random differences in the two outcomes.

So what can we do? We can place bounds on the likely range of the real odds ratio, based on our observations. We’ll often see results like “The cohort data revealed no relationship between vaccination and autism (OR: 0.99; 95% CI: 0.92 to 1.06)” [1]. The “OR” is our best guess – given our observations – at the odds ratio. Here it’s 0.99, very close to 1 and in fact slightly below (so the chance of autism is slightly lower in the vaccinated group). But the “95% CI” is what we call the confidence interval on this number – the range that it could reasonably take given our observations. This is saying that it’s highly likely that the true odds ratio is between 0.92 and 1.06.

If a study found a range of highly likely values for the odds ratio that was always greater than 1 (for example, 95% CI: 1.25 to 1.50), this would be statistical support for the idea that vaccines are positively linked to autism (proving they are the true cause is much harder; correlation is not causation).

What the US administration is asking for, effectively, is evidence that the hazard ratio is exactly 1 “(OR: 1; 95% CI: 1 to 1)”. This is impossible, and they know it. Most systematic studies (often “meta-analyses”, which pull together results from lots of different studies) report no evidence against an odds ratio of 1 – like the above. But we can’t provide evidence for an odds ratio of exactly 1 – that’s not how statistics works. We can only provide evidence against a particular value, and so far – despite immense, worldwide effort — we have never seen reliable evidence against the value of 1. And the confidence intervals often go either side of 1 – so any unobserved effect could be a small decrease as well as a small increase.

At the same time, vaccines clearly, unambiguously save many many lives. They are of course not without risk! Adverse vaccine reactions certainly happen — thankfully rarely — and can be serious. There is no evidence that autism is among them.

[1] https://pubmed.ncbi.nlm.nih.gov/24814559/