How did those vaccine trials work, anyway?

A short dive into how we know they work

When the three vaccines approved for use in the US submitted their applications, they each came with a headline statistic: Pfizer was 91% effective. Moderna 94%. J&J 66%. Apparently, the threshold for the FDA was 50%. What do these percentages mean? How were they calculated? 

Drugs go through several steps to gain approval. This will only focus on (at the level of a rough sketch) the third stage, which focuses on proving the efficacy of the vaccine. The first two stages are focused on safety and figuring out the right dosage. 

So how does this third stage work? 

The first step is to recruit volunteers. For Moderna, the number of volunteers in the third stage was 30,000. Half of the volunteers get the vaccine and half get a placebo. Critically, neither the volunteer nor the clinician knows who is getting what. This is called a “double blinded” trial because both the person giving the shot and the person receiving it are blind to what they’re getting. They only find out what they got later, once the results are “unblinded”. This is done to ensure that the treatment group (the group that gets the real vaccine) and the control group (the group that gets the placebo) are as equivalent as possible. 

Then the folks running the trial wait for a certain number of people to get covid. The study (or at least the part that determines efficacy) will take as long or as short as it takes for a certain number of people to test positive. For the Moderna phase 3 trial, that was 151 people. Why 151 people? 

To answer this question, we first need to define what efficacy is, and to answer the question I posted at the beginning: what did it mean for Moderna to be 94% effective? What this means is that, of the people that would have gotten covid, 94% of those cases will be prevented. It does not mean that you have a 6% chance of getting Covid if you have the Moderna vaccine. It means that if you would have gotten Covid without the vaccine, then you only have a 6% chance of getting Covid with the vaccine. So your chances of actually getting Covid are much lower than 6%, because you have to multiply 6% by your pre-vaccine probability of getting the vaccine. 

They determined that with 151 cases they could achieve the following: 

The target VE against COVID-19 is 60% (with 94% confidence interval lower bound ruling out 30%, rejecting the null hypothesis H0: VE ≤ 30%)

This means that if the Moderna vaccine were actually 60% effective, then once 151 people had gotten the disease and they learned who was who, they would be able to say with 95% confidence that the vaccine was at least 30% effective. 

So let’s say the vaccine was 60% effective. That would mean that, on average, of the 151 people that got sick during the trial, that 43 had gotten the Moderna vaccine, and that 108 had gotten the placebo. Why those numbers? 

Well, we know that we estimated the vaccine to be 60% effective. This means that the vaccine prevents 60% of the cases that would have happened without the vaccine. An estimate of the number of cases that would have happened without the vaccine is the number of cases that happened … without the vaccine; that is, in the control group. And if the vaccine prevented 60% of the cases, this means that it allows 40% of the cases. So we know that

Cases allowed by vaccine / Cases in placebo = 40%

Cases allowed by vaccine + Cases in placebo = 151

Two equations, two unknowns; solve this out and you get 43 and 108. 43/108 ~= 40% and 43+108=151. 

But the vaccine actually came in at 94% effective. So how many people got sick with the Moderna vaccine during the trial? We can use the same approach:

Cases allowed by vaccine / Cases in placebo = 6%

Cases allowed by vaccine + Cases in placebo = 151

This would imply that 9 people got sick after getting the Moderna vaccine, and 146 got sick with just the placebo. 

Now, I guess things changed slightly when they actually ran the study, because the total results are slightly different, but broadly the same: 

Symptomatic Covid-19 illness was confirmed in 185 participants in the placebo group (56.5 per 1000 person-years; 94% confidence interval [CI], 48.7 to 65.3) and in 11 participants in the mRNA-1273 group (3.3 per 1000 person-years; 94% CI, 1.7 to 6.0); vaccine efficacy was 94.1% (94% CI, 89.3 to 96.8%; P<0.001).

So if you divide the cases in the vaccine group (11) by those in the placebo group (185), you get 5.9%. So the vaccine prevented all but 5.9% of the cases; equivalently it was 100%-5.9%, or 94.1%, effective. 

So there you go! That’s how the trials worked. 

---

Finally, I want to add a personal note. I am just getting this newsletter off the ground and want it to reach as wide an audience as possible; to do so, I need your help! First and foremost, if you like what I’m doing here, please share it with other folks that you think might like it. 

✨My goal right now is to get to my first 100 subscribers! You can help me do that! ✨

Shouting into the void has its time and place, but not here! 

Also, if you reply to this email with thoughts or feedback, that would be incredible. I’ll get the email, and we can chat about whatever you found interesting (or hated). 

Thank you for reading,

Tom