That Cochrane review

[liv]

So there was an update to the Cochrane review on masks to reduce the spread of respiratory viruses. It concluded that Wearing masks in the community probably makes little or no difference to the outcome of influenza‐like illness (ILI)/COVID‐19 like illness compared to not wearing masks.

The reception of this review has been fascinating, though extremely depressing. The lead author, Tom Jefferson, gave an interview informally summarizing the conclusion as, There is just no evidence that [masks] make any difference.. That's not a totally ridiculous single sentence summary of the conclusions of the meta-analysis, but it's in the context of a very inflammatory article. Some mainstream media outlets ran the story as if it were, new study proves masks make ZERO difference. (Not linking to any of those directly, I'm sure you'll have seen them around. But deeply misleading by confusing a study with a meta-analysis of previous studies, and assuming that a finding of no evidence counted as "proof" of no effect.)

Very quickly, all of public-facing social media was absolutely full of highly networked, bot-like accounts posting a link to the study on every thread that referenced Covid precautions, saying, LOL, masks don't work! (I don't think all the comments to this effect were posted by literal automated bots, but by the kind of mix of human and automated postings that swarm Twitter repeating the same message when it's convenient to someone influential.) There were even a few anonymous posts on my quiet backwater little DW with a similar message, the Cochrane review shows masks don't work, you can stop bothering. Some people I actually know also mentioned that the review had convinced them to give up on masks.

There were also a bunch of debunkings which seemed very knee-jerk, again, particularly on Twitter but all over the place. A much retweeted thread which actively lied about the studies included in the meta-analysis and all but libelled the authors. I read a lot of stuff. I dismissed very extreme refutations which amounted to little more than, well, everybody knows masks work so if this meta-analysis showed they don't it must be biased and wrong. That's not very scientific, that's just confirming your pre-existing views. I skimmed the review itself but didn't read all the studies cited in it, but gave myself enough of a sense of when people referring to it were reporting broadly true facts about its findings, rather than making up nonsense to support either the pro- or anti-mask cause.

[Sidebar: This, by the way, is an example of why I disagree with siderea that I think you should trust me (within limits) because I cite my sources.. Lots of not at all trustworthy people cite sources all the time. They post lists and lists of links to highly technical academic articles, and make completely untrue claims about what those articles show, and hope that you won't have time or the right skills to go back to the cited source and notice for yourself that the summary is a distortion, or even the opposite, of what the purported citation shows. Like, there's literally a playbook circulating in the nasty bits of the internet with a bibliography of peer reviewed journal articles that purport to show that intelligence is genetic and correlated with race. Transphobes regularly cite "studies" which apparently show that trans people are a danger to normal children and cis women, when the studies either have been debunked or show no such thing. Even Wikipedia, with its citation needed culture, can easily be used for disinformation because a pile of little blue numbers make false articles look authoritative, and people believe the air of academic-ness without noticing that the citations don't support what is claimed on the page. So anyway, lots of people "cite" the Cochrane review or the studies within it, but are wrong, intentionally or accidentally, about what such sources say.]

The conclusion I tentatively reached was that there are, essentially, two well-powered, high quality RCTs of mask wearing for prevention of actual Covid. The Bangladesh study which found a modest but consistent positive effect, and the DANMASK study which found little to no effect (and like the review that relies on it, was widely cited as definite proof for the anti-mask case when it came out). You can't do a meta-analysis by combining only two studies, so the Cochrane team had a choice: they could include studies that aren't RCTs, or they could include studies that are not on Covid. They chose the latter, which is not a completely unreasonable decision and I don't think I'm better able to judge than actual experts in medical evidence which way they should have gone. Also, it was an update of a pre-Covid review about respiratory illness in general, so it makes sense to include studies from before the emergence of Covid.

But to me this meant that the meta-analysis added little to what I already knew, that the Bangladesh study found masks to be somewhat effective in a real-world setting, and the Danish study found that in a situation where case levels are already extremely low, they weren't able to measure much difference between almost no cases in the control group and almost no cases in the intervention group. I did a lot of reflecting about my own attitudes; I certainly trust Cochrane reviews on the whole, and I certainly want to be the sort of person who updates my opinions based on new evidence. I was pretty sure I had a bias towards looking for flaws in the Cochrane review, and I tried to compensate for that.

For some people, "masks don't work" is good news, because it means they can push back against anyone asking them to wear a mask when they don't want to. But I don't feel like that at all; I would be very unhappy to learn that masks don't work, because it wouldn't mean time-travelling back to 2019 as I'm sure many of us would love to be able to do, rather it would mean that the main precaution I'm taking to be able to do any in-person activities at all is essentially worthless. Obviously my unhappiness is completely the opposite of evidence about what is actually the reality out there. So I wanted to be very careful not to over-weight evidence that aligns with what I want to be true.

I had some amount of trust in this rebuttal by Macintyre et al. It contains a great deal of non-RCT, but compelling, evidence in favour of masks, and some convincing arguments why RCTs based purely on outcomes for the mask-wearer may not be the only good quality evidence for this question. It also has what looks to me like a valid, not hysterical, critique of the Cochrane review, namely that they are comparing a lot of different things that aren't really comparable, and also that many of the studies which came out not showing much effect were studies of whether telling people to wear masks is effective and didn't take into account compliance or lack thereof. I generally trust the last author, Trish Greenhalgh, but she has been passionately pro-mask throughout and indeed it's reading her research that convinced me wearing masks was worthwhile even when the official advice was against it. Macintyre herself declares a conflict of interest in that she works for mask manufacturers, which is a pretty glaring source of potential bias!

My conclusion until today was: telling people to wear masks in the community, regardless of Covid levels, and regardless of mask quality, doesn't strongly predict better outcomes (less Covid, in this case). But there is pretty good evidence that well-fitted masks worn by most people do actually provide some protection when Covid levels are high, even if there isn't much in the way of well-powered RCTs showing benefit. I held that conclusion pretty lightly, because I knew I was biased and because I hadn't actually read all the original studies in detail, mostly summaries.

The latest news is that the editor-in-chief of the Cochrane library has semi-retracted the notorious review. As reported by Zeynep Tufekci in the New York Times, Soares-Weiser has issued a statement that Many commentators have claimed that a recently updated Cochrane Review shows that ‘masks don’t work’, which is an inaccurate and misleading interpretation ... [our] wording was open to misinterpretation, for which we apologize. (I am aware that some people are boycotting the NYT due to its horrible transphobia; right now they are the outlet breaking this news, so feel free to wait until other press wires pick it up if you don't want to give them clicks. Also, I bypassed the paywall by turning off JavaScript; up to you whether you're comfortable with doing that.)

The remainder of the article summarizes some of the (non RCT) evidence in favour of masks, and notes that the Cochrane review lead author has a history of anti-mask views and even pandemic minimizing. I have a lot of time for Tufekci, but some people don't like her because she doesn't come from a scientific or medical background, and she tends to take into account evidence from a range of sources and in particular doesn't accept some of the most extreme Covid catastrophizing. IMO she's been pretty consistently right, supporting masks and opposing restrictions on outdoor socializing from very early on, campaigning for Long Covid sufferers but rejecting absurdly high projections of levels of severe long-term organ and immune damage. In my view the important thing is the statement by Soares-Weiser where she is very clear that the Cochrane review has been misinterpreted as being definitively anti-mask.

So there you go, that takes us more or less back to the status quo ante, before the January update of the Cochrane Review. We have some evidence in favour of some types of masks in some situations, but we can't definitively say 'masks work' or 'masks don't work'. I think that may be some of the reason why some of the mask debate has become so painful and polarized; a lot of people don't feel comfortable with the idea of a mitigation that has some effect but isn't perfect, they want interventions to either "work" or "not work". Same as the vaccination debate. Vaccines, like masks, give some protection, but they're not, as Tufecki puts, it, a talisman or a magic wand.

Comments

[seekingferret]

I saw a few of the twitter debunkings which were like "See, Cochrane is totally wrong and bullshit, the high quality studies actually show a tiny positive effect from masks!" and I was thinking... we're doing all this shouting over a tiny positive effect?

I still mask indoors because I can tolerate it well enough and I figure better to do it than not, even if it only has a marginal impact, but if I get COVID anyway from a masked exposure I'm not going to be surprised or upset.

[liv]

I care quite a lot about small effects, because small effects (and thus layered protection) are what we've got. Vaccination has a big effect on protection from death and severe disease, but everything else, including vaccination on other outcomes, has small effects. Also viral infections are exponential so in the medium term small protective effects can snowball (equally small increases in absolute risk can lead to major population-wide disasters, so.)

I don't think shouting at people about the exact numbers of how much protection you get from masks in exactly what circumstances is at all productive; people who have made anti-mask part of their identity aren't going to accept any evidence anyway, much less emotional pressure from people they perceive as angry and threatening. The group worth persuading, I think, are people who basically accept the consensus that masks are protective, but believe that the downsides of masking are worse than repeatedly catching Covid.

Like you I wear a mask when it's practical; I don't find them particularly unpleasant, and also: masks are a physical barrier. The level of evidence I need in order to believe that a physical barrier which prevents viruses from entering your lungs is effective, is much lower than the level of evidence I need to believe in a pharmaceutical intervention. I might get Covid while wearing a mask, but I probably won't ever know whether it was because the mask wasn't sealed well enough, or because I was unlucky and got exposed when I took the mask off briefly, or when I intentionally removed the mask because outdoors is IMO 'safe enough'. But I am mostly relying on trying to minimize my exposures as much as I can.

[aldabra]

Swiss cheese model: https://en.wikipedia.org/wiki/Swiss_cheese_model If none of your prophylaxes work reliably you layer them. Once you're layering them, the questions for each one are: is it safe; what does it cost; is it likely to help?

Also, RCTs are expensive and cumbersome, and inappropriate in some cases. They're especially expensive and cumbersome if you're looking for unreliable effects. You need them in medicine, because we don't understand how organisms work, but you don't need them in physics (and some of how masks are thought to work qualifies as physics).

I think masks work well enough to qualify as a layer and not well enough to rely on.

I don't see how the Cochrane authors didn't foresee that they were walking into a shitshow.

[liv]

Yeah, I 100% believe in the Swiss cheese model here. There was never going to be one magic bullet, partly because the virus is transmitted by human behaviour and it's never been possible to have a public health programme that relies on everybody behaving in an ideal manner.

I do agree with some of the criticism of the Cochrane review that RCTs are not the best way to measure mask effectiveness. I do think part of the problem that has led to genuine disagreement among the scientific community about masks (aside from the political / culture war nonsense which has been thoroughly poisoned by bad actors) is confusion over whether masks are basically a medical prophylactic, or basically a physical barrier. I tend to the second view and I am therefore not that worried about RCTs. However neither side of the debate is really listening to social and behavioural scientists, and wearing masks is a behaviour first and foremost, so a lot of the conclusions are hopeless because people are missing half the data.

IMO the Cochrane authors knew full well they were walking into a shitshow. They may well have lit the fuse semi-deliberately, because academia these days has really broken incentives and sometimes publishing something controversial gets you more career recognition than publishing something correct. If it's true that Jefferson is personally anti-mask / pro-Covid, even more so. That Substack exclusive really doesn't sound like an interview with someone who carried out a neutral, objective review of the evidence and dutifully published what he found purely for the sake of advancing human knowledge.

[nameandnature]

See also Zeynep's Twitter thread and paywall defeater for the NYT. She's certainly a glutton for punishment, what with this and defending Harry & Megan and pointing out that "COVID damages your immune system" is false. I admire her ability not to just pick a side.

I've reduced my credence in the idea mask mandates work since Cochrane came out. I suspect masks themselves work a bit, especially the FFP2 ones.

[liv]

The thing is that the two "sides" here are pro-disease and pro-human. There are a few fringe people who are convinced that all mitigations are an absolute good regardless of whether they're effective and cost-effective, or who are convinced that any attempt to control viral spread is an unacceptable attack on freedom. But most of those aren't serious analysts, they're grifters or they have a political, not a public health agenda.

One of the ways I find Tufekci particularly clear-sighted is that she updates her ideas as the situation changes; she often points out that some of the groups just shouting across each other are in fact trying to "relitigate 2020" as she puts it. I have my opinions about what the authorities did right and wrong pre-vaccine and during the vaccination programme, but those shouldn't really colour what I think is the correct approach now.

The Cochrane review doesn't really have good evidence about mask mandates because it lumps together all types of mask promotion: mandates, partial mandates, advice / education / advertising, distribution of masks, and so on. We don't really know even whether mandates lead to better compliance than advice without enforcement, let alone whether mandates improve Covid outcomes. And there's a kind of philosophical debate too, namely the belief that if masks have only very small effects on health, it's an unfair burden to impinge on people's freedom by making them wear them. Versus the belief that if masks have only a small cost it's worthwhile to require them even for a small benefit. So people can sometimes seem to be arguing about mask "effectiveness" but implicitly they're arguing about the trade-offs between liberty and social benefit.

[adrian_turtle]

Here is the best study I've seen about the usefulness of requiring masks.
https://www.nejm.org/doi/full/10.1056/NEJMoa2211029

I appreciate that they were looking at 2022, rather than trying to extrapolate from how the pre-variant virus spread among unvaccinated people wearing cloth masks. And I appreciate how they looked at whole communities, counting it as a win if schoolchildren didn't get sick and bring the virus home to infect their parents. (Rather than trying to compare the infection rates of masked children and their unmasked parents.)

[liv]

Yes, I like that study a lot, it is well designed and I agree with you that it's measuring the right outcomes. The clear divergence between districts that were previously experiencing similar rates goes a long way to rule out confounders around contexts with mask mandates also being more careful in other ways. I can completely understand why the study wasn't included in the meta-analysis, partly because of the timing, and partly because it doesn't meet the criteria being applied to define a high quality study. But I find it more convincing than most RCTs.

[starlady]

It seems to me the standard being "Wearing masks in the community probably makes little or no difference to the outcome of influenza‐like illness (ILI)/COVID‐19 like illness compared to not wearing masks." is almost a prejudgment on the part of the review -- for most people, the point of wearing masks is/has been to reduce transmission, whether to or from themselves, not to have a better outcome from your course of disease. If you get it while masking, by definition masks have failed. And it matters what kind of mask people wear, too.

It feels sensible that masks would not have much noticeable effect when circulation is low, but I'm still wearing mine on transit including planes for the foreseeable. I usually wear one in indoor settings like the grocery store as well, though I've also eaten inside at restaurants unmasked. It's a historically proven NPI that won't harm anyone and definitely can reduce transmission, so it makes sense to me to continue wearing mine in at least some circumstances.

[liv]

Some have at least hypothesized that people who get infected anyway while wearing a mask may have a lower viral dose and thus less severe illness than those who have no protection. But I agree, it's not quite the right outcome. We've known for a long time that low quality masks do more for source control than to protect the wearer; you might expect that to be somewhat visible in the meta-analysis, but you also might expect it not to be depending exactly what the included studies were measuring.

One conclusion I might tentatively draw from the meta-analysis is that it is worthwhile to have flexible mask programmes, and actively encourage people to do more activities unmasked while cases are low. I think that's sort of what the US CDC was going for in 2021-2, but they really fluffed it. Also, it really does depend on good monitoring of cases, and most highly vaccinated / high resource countries have given up on that. In the era of omicron and no real public health measures, cases are never what I would call "low", certainly nowhere near the kind of numbers in the Danish study. But I don't know what counts as low cases with a vaccinated population. Another possible interpretation is that it's a statistical artefact, too, because, well, it wasn't that masks didn't work with low cases, it's that the study wasn't powered to detect a difference against a very low background level.

I agree with masking on transit, to me it seems common sense, because all I want to be doing while I'm sitting on transit is disappear into my phone, nothing that a mask impedes even slightly. And it's the situation where I'm most sharing air with strangers. I wear a mask in grocery stores because, well, it's easy enough and it might at worst make other customers and staff feel more comfortable, so why not? I have mostly avoided restaurants since 2020, but I do take my mask off for other situations, particularly religious gatherings, where the mask actually causes problems with eating and communication. The problem with that reasoning though is that it doesn't give me an answer regarding what the policy should be for public life, it's a sensible individual decision, but how much effort should go into encouraging, or even enforcing, mask wearing in public areas?

[davidgillon]

Even Wikipedia, with its citation needed culture, can easily be used for disinformation because a pile of little blue numbers make false articles look authoritative

There doesn't even need to be intent involved. One problem for wikipedia is a lot of mass-market books contain errors that have made their way into popular knowledge and thence to wiki articles, so when an actual expert comes along and corrects something, they're instantly 'corrected' back by people who don't know anything about the subject, but know a book that doesn't say that.

[liv]

Yeah, Wiki is a kind of weird case in several ways, partly because of their attempt to systematize what sources are considered "reliable" to be cited, including the idea that printed books trump expert opinion. I've even heard of people getting into trouble trying to update their Wikipedia articles to say that 'X has two three children' because 'I had another baby' isn't considered a reliable source, even aside from the fact that you're not supposed to edit an article about yourself.

[davidgillon]

'I had another baby' isn't considered a reliable source

*facepalm*

[ewt]

I'm afraid I haven't followed much of this particular controversy, because most of the way I have observed most people wearing masks in the real world would probably help reduce droplet transmission but have only a small effect on aerosol transmission, and I'm pretty solidly convinced that aerosol transmission is a substantial concern. I pretty much assumed that the RCTs didn't use the right sort of mask, or didn't ensure good fit, or didn't measure actual use rather than mandates. Perhaps that's too cynical.

I don't think I know anyone who has managed to get covid while wearing an elastomeric respirator -- but that is a very small sample, as I only know a handful of people who use them. That's not a RCT, we're a self-selected group of cautious people.

[liv]

I could off the top of my head name half a dozen elastomeric respirator users I know personally who have had Covid. Certainly the never-Covid portion of my acquaintance over-represents people taking this precaution, but it's still not a magic talisman. And maybe they didn't wear the elastomeric properly, or took it off in an inappropriate / unlucky situation; most people don't know the exact moment when they caught Covid.

Lab evidence indicates that an elastomeric mask isn't that much better than an FFP3 / N95. I would predict that the main difference in practice is that elastomerics are more comfortable to wear so people are more likely to actually wear them all the time they're needed. I would find it plausible that it's somewhat easier to wear a rigid half-face mask ideally, with a good fit, than a disposable mask, so the elastomerics may be better for that reason. But we're unlikely to get clear real-world evidence on that because as you say, it's tiny numbers of people who wear elastomerics consistently, and they're generally very cautious in other ways.

I do think that the kind of evidence looked at in the Cochrane review is important, because if it turns out that telling people to 'wear a mask' means most people wear the wrong kind, or wear it inconsistently, or intentionally refuse to follow the advice / mandate, that itself is useful data. But it's not relevant to whether I personally should wear the best mask available to me.

[ewt]

The only elastomeric respirator users I personally know who have had covid either stopped wearing the elastomeric respirator in some contexts and then got covid, or got covid and then started using elastomerics. But I also don't really know anyone outside my own household who is as consistent as we are in wearing them in indoor non-home spaces, either. Some of those less-careful-than-us people have had covid, some haven't. I'm sure that no mask is magic; I'm also sure that, used consistently, they reduce my odds of getting COVID enough to be worthwhile. At the end of the day it is a numbers game and a filter that excludes 99.99% of particles larger than 0.3 microns in a mask that is sealed against my face is going to change the numbers substantially.

One of my main issues with non-elastomerics is that I personally find it difficult to take them off and then put them on again and still get a decent seal: there's a lot of fussing about with the nose wire etc every time. It might be the shape of my face, though I don't think it's particularly unusual. It might just be that I am a person who is hard on things -- shoes, backpacks, bikes, clothing all wear out fast. Either way, the non-elastomerics do become impossible to seal well pretty fast for me, even if I modify them by adding decent sewing elastic. I know that isn't everyone's experience, but it is one part of why I am so keen on elastomerics.

[liv]

Oh, I should have been clearer, I'm not at all trying to talk you out of your elastomeric respirator masks, they're clearly working well for you, and you're in a good position to be extremely careful and get the maximum possible protection from them, which is a lot of protection. My problem is that masks, even good masks, are a single method of protection used by individuals, and as such likely to be inadequate if we carry on forever with 1 in 50 of the population or worse having Covid.

I want everyone to be able to avoid Covid, including young children, including people with customer-facing jobs and bosses who aren't willing to accommodate them, including people who can't stand having something pressing on their face. You might in fact be better than 99% of the population at taking the right precautions – that's a really good thing, and I admire you for your ability to assess the evidence and implement an effective solution, but protecting the best 1% of the population isn't public health, it doesn't do much to stop Covid.

[ewt]

Oh, absolutely. I would like all the people I know who wear masks sometimes, and the people I know who never wear masks (regardless of the reason), to have the benefit of better ventilation standards (starting with schools and hospitals), better sick pay policies, better access to working remotely where that's feasible, free testing, proper contact tracing (insofar as that can be done without infringing civil liberties -- I think it likely can be, but I don't think it's trivial). All of those things substantially impact the numbers game too. Hopefully, in time, better vaccines would also change the numbers, and better treatments would make those numbers matter less because the risks would be lower (and these are... less scary than they were in, say, March 2020, but still Not Great).

In the meantime... I know not everyone can take the precautions I do, but that makes it feel more important, not less, that I do it, because then it's less likely that I'll pass covid (or influenza or whatever) on to someone who can't take my precautions. And then I really get annoyed by the "well, not everyone can avoid catching covid, and most people recover, so nobody should bother to try" narrative that seems to be assumed in a lot of the "this single intervention isn't perfect" media coverage. I don't think that's where you were going at all, though, and I think we largely agree that better public health interventions would be desirable and effective, and that individual mitigations in the meantime are imperfect but still often worthwhile (subject to individual context), and that ideally individuals wouldn't be put in the position of having to decide whether to wear a mask for activities where it's clearly easier not to but the infection risk is high or unknown, or having to research which masks work well for them, or having to learn how to wear them properly. It's pretty absurd that any of us are in this position at all.

I'm also still really angry at being told for most of my life that handwashing and not touching my face and not getting sneezed or coughed on directly were The Important Things To Focus On To Not Get Sick, only to find that I haven't had so much as a cold since February 2020 because I stopped exposing myself to aerosol transmission of viruses. Don't get me wrong, I do still wash my hands, but... not all that much more than I did pre-pandemic, I think. And I'm angry that as a society we appear to be turning around and ignoring that; I'm angry that I asked a choir I would love to sing in about CO2 monitoring and didn't even get a response; I'm angry that we can't afford to do the kind of air filtration we would need to at church to keep the space safer when the soup kitchen is operating, there are no grants available for improving it, and we don't want to just open the doors because we're also advertising it as a "warm space" for people who don't have any heating at home (...and yeah, if the energy bills were lower, maybe we could afford a consultation and the equipment to do the filtration, but... that's not where we are, and I don't exactly think artificially lowering energy bills is a sustainable approach either given the climate situation. During the times when there were restrictions on gathering we offered an outdoor-only service, but it's not necessarily better to make people sit in the cold). So I get a bit spiky in my replies sometimes, because all of that feels intractable and impossible, even though I do see signs of improvement (like this news about air quality regulations in Belgium and elsewhere: https://www.nature.com/articles/d41586-023-00642-9 and something similar I read a month or two ago regarding air in schools in France -- though if the government has actually rolled out CO2 monitors to every UK classroom, as that article claims it "started to" in August 2021, it's news to me). I'm sorry if I am coming across as unreasonably cantankerous as a result.

[kaberett]

Thank you very much for writing this up.