In the case of vaccine effectiveness vs. severe disease, it is the fact that both vaccination status and risk of severe disease are systematically higher in the older age group that makes overall effectiveness numbers if estimated without stratifying by age misleading, producing a paradoxical result that the overall effectiveness (67.5%) is much lower than the effectiveness for either of the age groups (91.8% and 85.2%). Since the <50yr and >50yr groups are quite heterogeneous in terms of vaccination rates and risk of severe disease, it is instructive to stratify by even finer age groups:
We see quite high effectiveness in all age groups, with the 80-89 group having the lowest effectiveness (81.1%) and all others between 88.7% and 100%. We see that the current Israeli data provide strong evidence that the Pfizer vaccine is still strongly protecting vs. severe disease, even for the Delta variant, when analyzed properly to stratify by age.
Conclusion
In conclusion, as long as there is a major age disparity in vaccination rates, with older individuals being more highly vaccinated, then the fact that older people have an inherently higher risk of hospitalization when infected with a respiratory virus means that it is always important to stratify results by age; if not the overall effectiveness will be biased downwards and a poor representation of how well the vaccine is working in preventing serious disease (the same holds for effectiveness vs. death). Even more fundamentally, it is important to use infection and disease rates (per 100k, e.g.) and not raw counts to compare unvaccinated and vaccinated groups to adjust for the proportion vaccinated. Use of raw counts exaggerates the vaccine effectiveness when vaccinated proportion is low and attenuates the vaccine effectiveness when, like in Israel, vaccines proportions are high. To do this is to fall for the base rate fallacy. This is not just an issue of making vaccines look worse than they are ... any summary computing "proportion of hospitalized that are unvaccinated" that covers a period of time in which the proportion vaccinated was low can be similarly misleading, especially if there was a massive Covid-19 surge during that time periods. For example, computing total proportion of hospitalized covid infections in the USA from unvaccinated individuals while aggregating over the entire 2021 (January to present), a time periods that includes the early months in which virtually all USA residents were unvaccinated and there was a massive winter surge, will be similarly misleading. Thus, these artifacts can be used by some to make the vaccines look better than they in fact are, e.g. any report suggesting things like 99.9% of hospitalizations are from unvaccinated when covering a long period of time like this.
The bottom line is there is very strong evidence that the vaccines have high effectiveness protecting against severe disease, even for Delta, and even in these Israeli data that on the surface appear to suggest the Pfizer vaccine might have waning effectiveness. This is clearly evident if the data are analyzed carefully, and agrees with all other published results to date from other countries.
While this is just a snapshot of currently active infections on August 15, 2021, the principles apply to other analyses done on Israeli data, as well as others.
One caveat with any effectiveness analyses with the Israeli dashboard data is that the previously infected are not separated out. Note that:
-
Israel did not allow previously infected to be vaccinated until 3 months into the vaccination campaign (in March)
-
Then made only optional (given they awarded immunity passports to previously infected even if unvaccinated) and only limited them to one shot.
Given the high vaccination rate, it is plausible that a substantial proportion of unvaccinated were previously infected. Given the overwhelming evidence that previous infection confers strong and lasting immune protection from dozens of published papers, this means those unvaccinated have strong immune protection (possible comparable to vaccinated). This would serve to attenuate the effectiveness estimates, and may be one reason why the effectiveness vs. severe disease is not higher than 85-92%. Also, this might make their single-dose effectiveness appear much higher than other places since it also includes those previously infected who were eventually vaccinated. More caveats to keep in mind ... By the way, earlier reports on vaccinated cases at Israeli hospitals when there were 152 hospitalized breakthrough infections showed that a full 40% of these cases were immunocompromised, and 96% had co-morbidities including hypertension (71%), diabetes (48%), congestive heart failure (27%), chronic kidney and lung diseases (24% each), dementia (19%) and cancer (24%). At that time point, virtually none of the active serious breakthrough infections in Israel were in individuals without significant pre-existing conditions.
Similar effects could be lurking in other variables and settings, e.g. if people who have particular jobs like health care workers both have (1) higher vaccination rates and (2) higher probability of exposure to SARS-CoV-2, then this phenomenon could similarly bias the overall effectiveness vs. infection numbers if results not stratified by these factors that might differentially affect the probability of exposure. This comes into play especially when assessing whether vaccine effectiveness vs. infection wanes over time, given that in most countries the subset of young people who were vaccinated early are nearly all HCW who also have disproportionally high exposures to SARS-CoV-2 and thus higher probabilities of infection than the younger people vaccinated later who are not HCW or other "essential personnel" prioritized for early vaccination. Similarly, we can expect that immunocompromised people were in the earliest priority vaccination group, and thus it is possible that the reduced effectiveness in people vaccinated earlier could be in part due to these factors if they are not taken into account in the analysis.
With real-world observational data, we always need to think carefully about factors like these when trying to assess vaccine effectiveness against infection, severe disease, or death.
As a result, we should be wary of any claims that simply report raw counts or overall effectiveness figures without stratification, and we need to look to careful data analyses from published papers that take these factors into account using available statistical methods for causal inference, transparently described in detail, if we want an accurate sense of the potential causal effect of vaccines. Many of the papers I have seen published from Israel, the UK, Canada, the USA and elsewhere have used rigorous methodology to adjust for these factors, which can include stratification, re-weighting, matching by confounding factors or propensity scores, or covariate adjustment, but the details of how they adjust for such factors always must be carefully evaluated when trying to interpret the implications of results from any observational study.
A few details to point out about the data and analysis:
The data used in this blog post were downloaded from the Israeli Ministry of Health Dashboard. The box on the far left, second from the top has a down-arrow that can be clicked to obtain the data of currently active serious covid-19 cases by age and vaccination status. This data includes only Israeli residents age 12 and older. This is the data I downloaded on August 15, 2021, for this illustration and analysis. Here is the data set just as I downloaded it (the only change is I used google translate to get English headers since I don't read Hebrew)
Given they had both raw counts of cases for unvaccinated, partially vaccinated, and fully vaccinated as well as counts per 100k, I back calculated the number of fully/partially/unvaccinated in each age group. I focused here on severe infections, but the table also has numbers for total infections in each age group. For simplicity of presentation, I focused on fully vaccinated vs. unvaccinated, although the data is there for partially vaccinated as well. I also aggregated data into "young (<50)" and "old (>50)" groups to simplify the presentation, but present effectiveness estimates for each age group at the end. Here is the data set after these columns and rows were added that I used for the analyses presented:
For brevity, I focused the tables on fully and non-vaccinated only, and didn't include partially vaccinated (1 dose Pfizer). This is why the % don't add up to 100%, but if you take 100% - %unvax - %fullvax you get % partialvax. For example, overall it is 100%-18.2%-78.7%=3.5% partially vaccinated
BTW, My original table had two typos -- the 91.9 was 90.9 and 2,133,516 was 2,170,563. These were powerpoint cut and paste typos, and did not affect the %, cases per 100k, or effectiveness numbers. These are all correct.
