Indistinguishable from magic
May. 25th, 2011 09:39 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
livThis entry is going to contain a lot of stuff that's been swirling around in my mind for several weeks, and I'm not sure it all quite fits together, but I want to put some thoughts out there.
The first trigger was that I tangentially got involved in one of those discussions about whether science is better than religion. I normally don't bother with that argument because it's boring and frequently stupid, and also because I don't think it's a meaningful comparison. Science is not only no good, but completely irrelevant, for organizing a regular rota of visitors to check up on an old lady with Alzheimer's who is estranged from her daughter. Religion is not only no good, but completely irrelevant, for understanding how prions in the old lady's brain aggregated to cause her to lose her memory and functionality. (I have no intention of asserting that atheists never visit lonely senile people, just that they don't use science to do so, because they are not idiots.)
But anyway, I joined in with this discussion because![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
pw201 is intelligent and interesting, and there was an issue of terminology I was curious about. The discussion led to Paul asserting (relevantly):
The thing is, "believing in" science in this way doesn't just offend me as a scientist; it kills people. Let me talk about a lecture I attended recently. The talk was given by the DUETs people, who are working to put conventional, evidence-based medicine on an even more scientific basis. But they are not doing this by claiming that good science should still be true decades and centuries after findings are reported. Quite the opposite! They are claiming that good science, and good evidence-based medicine, should be flexible in how it responds to new evidence, and established views should be constantly challenged. This isn't just to make people feel better intellectually, it's a really critical aspect of patient safety.
Example 1: for many decades in the second half of the 20th century, medical wisdom was that babies should be encouraged to sleep on their fronts. This advice was pretty universal, and even made it to Dr Spock's famous book about childcare. It was based on the best evidence available at the time, but by the 70s there was an increasing body of evidence that sleeping prone is a significant risk factor for cot death. However, this evidence took a very long time (decades) to percolate into mainstream medical advice, because doctors and even the medical research community were reluctant to challenge the established scientific fact. They were especially reluctant to rely on data from soft sciences and observation of large human populations, in order to overturn data based on the "more reliable" physical experiments that led to the earlier bad advice. Dr Spock was wrong, not because he was a bad scientist (neither morally bad nor incompetent), but because cot death wasn't really on the radar at the time he was writing. The data he relied on measured physical parameters of how well individual babies did, and was very likely correct that prone sleeping reproducibly improved those parameters in the short term. It was still wrong, and following Spock just because he had the authority accorded to a successful scientist still led to preventable deaths.
Example 2: some decades ago, there was some robust, reproducible, statistically valid scientific research showing that giving caffeine to premature babies helped to reduce the frequency of a condition called apnoea where the infant briefly ceases breathing. However, this research was often not applied clinically because there wasn't any real evidence to show that reducing apnoea occurrence was particularly important. Nobody was being a bad scientist, nobody was following superstition or religious beliefs at the expense of evidence, there wasn't even a big problem with doctors being unaware of the state of the art of research. It's a perfectly medically valid decision that you don't want to give a powerful drug with unknown long-term effects to premature babies who are extremely vulnerable anyway. It's a perfectly valid ethical decision that you don't want to do double blind randomized controlled trials on premature babies, with the very real possibility of harming them. Again, it took population studies and extrapolations from soft science observations to demonstrate that the frequency of apnoea is correlated with long-term risk of cerebral palsy and reduced life-expectancy. That's a lot of avoidable disability and death because only one sort of clinical trial counts as properly scientific.
Example 3: some decades ago, there was some robust, statistically valid, properly designed and controlled research showing that steroids can be helpful in patients with severe brain injury. So doctors very sensibly started treating brain-injured patients with steroids. And scientists very sensibly did what scientists do, and repeated and extended the original experiments over the course of the intervening decades. They didn't just assume that the original research must be "true" because it was "scientific". They didn't prefer to work on more glamorous, more prestigious new stuff at the expense of low-status confirmatory work. The effect size and statistical significance tended to decline with subsequent studies. This doesn't mean that the original research was wrong, or that the original scientists were biased, incompetent or lying, it's just an artefact of the way that scientific culture works. If you're going to publish something novel, you have to have a pretty watertight case, with strong statistical significance and a relatively big effect, and that's as it should be. But if you're just confirming something that is already known, then rather less dramatic and conclusive results are acceptable because they support the established fact. And of course, we all know but can easily forget that 1 experiment in 100 will show that something is true at the 99% significance level purely by chance and sampling error!
After many decades, a consensus started to emerge that the effect of steroids in brain injured patients was small and not terribly reproducible. Not false, just marginal. Meanwhile, treating people with high doses of powerful steroids has known side-effects. The medical community started to suspect that the definite, quite serious harm caused by steroids was greater than the small, poorly reproducible benefits. But there wasn't enough evidence to stop treating brain injured patients with drugs that might save at least some people's lives, until there was a huge, expensive publicly funded trial involving 10,000 brain injury patients across the EU which definitively proved that steroids do more long-term harm than good in this situation. So, ok, you might well say that this is a happy ending, this is medical research and evidence-based medicine working exactly as they should. But you have to take into account that even an optimal scenario means several decades of people receiving treatments which are actually harmful on balance, and which undoubtedly caused unnecessary deaths and suffering during this time period.
What are the implications for "rationalist" rhetoric? I think the most important is that scientific research, and particularly opinions couched in scientific-sounding language which include numbers, technical jargon and statistics, should be treated with at least a comparable level of skepticism to "woo" and alternative medicine. Lay people can't expect to directly evaluate every individual piece of research they read about; indeed scientists can't do that either, because most of it is outside their field and they have to spend at least some of their time studying new questions rather than confirming, validating and challenging old conclusions. But just accepting something as fact because it's "scientific" is not the way to deal with this!
Just accepting the authority of someone because they have scientific qualifications leads to things like believing Wakefield about MMR because he did experiments and used statistics and medical terms. It leads to believing a popular book based on extremely dubious research because the authors have some academic credentials. And because neuroscience is a "real" science, they have more authority to talk about anthropology and sexual psychology than, you know, actual anthropologists and sexuality researchers because human sciences don't count. It leads to giving racist propaganda the benefit of the doubt, because it uses statistics and hard sciencey jargon. Yes, it is a basic principle of science that one should accept unpalatable results if they are supported by data from well-designed and well-executed experiments. But all those people who piously recite this principle in response to badly-designed, biased and thoroughly debunked "experiments" "proving" that white people are inherently superior to other ethnic groups are strangely unwilling to give the same benefit of the doubt to the vast body of good research indicating that, you know, racism actually harms people. True, you can't weigh and measure those harms, you can't do double-blind experiments, but that doesn't mean that social science is just a matter of what's politically fashionable just now.
And that brings me on to my second point: if you believe that science is the best way of looking at the world, you should also accept that social science is the best way of studying human societies! That's especially the case if you (or the journalists you rely on for your information) can't tell the difference between actual physical / natural science and people using vaguely sciencey technobabble, but even good physics is relatively unhelpful for looking at social and cultural phenomena.
And yes, that goes for medicine too; there is lots of really vital medical information that just isn't going to be found by doing randomized controlled trials and measuring the physical outcomes and applying statistics. Partly because a lot of randomized controlled trials that would be informative are also unethical. And partly because the information that can be measured physically isn't always the most important; "how fast do babies put on weight?" can be measured easily, but a more important research question is "how likely are babies to die for no discernible reason?"
Drug trials are (relatively) easy to carry out in the time-honoured "hard" science way; you give the drug to half the patients and a placebo to the other half, and you measure objective parameters about how well the two groups do. I'm in no way arguing against doing this kind of experiment – hell, I spend most of my working life doing that myself – but it doesn't mean that drugs are the best possible treatment for all possible conditions! For example most patients with joint pain would prefer physiotherapy and exercise rather than strong painkillers (and by the way, the reason I know this is because social scientists did serious research into the issue, not because some arrogant biologist assumed that his credentials totally qualified him to throw together an internet survey.) There is some evidence that the former has more benefits and fewer side-effects for a greater proportion of patients than the latter. But it's rather harder to do a double-blind trial of physiotherapy, and you can't use pure bioscience to answer questions like "how well do patients on this regime integrate into their communities and lead normal lives?" which may be as important as "what is the level of pain-related chemicals in the bloodstream of patients taking this drug versus a placebo?"
And thirdly, I suppose, don't put too much faith in the scientific process. In the best possible circumstances it is slow and inefficient and people get harmed while science is sorting out the answer to difficult questions. When we're talking about medicine, individual variation within the population is inevitable, and however good the evidence is for a particular treatment, that best treatment will do nothing for or actively harm a proportion of patients. And to be honest, the best possible circumstances don't always apply; it's hopelessly naive to believe that all science is pure and unbiased and free of the influence of culture and political and financial considerations! Criticize superstition and woo and political bias, of course, but don't couch your criticisms in terms of assuming that the scientific mainstream is always right. That's bad rhetoric and it's atrociously bad science.
The first trigger was that I tangentially got involved in one of those discussions about whether science is better than religion. I normally don't bother with that argument because it's boring and frequently stupid, and also because I don't think it's a meaningful comparison. Science is not only no good, but completely irrelevant, for organizing a regular rota of visitors to check up on an old lady with Alzheimer's who is estranged from her daughter. Religion is not only no good, but completely irrelevant, for understanding how prions in the old lady's brain aggregated to cause her to lose her memory and functionality. (I have no intention of asserting that atheists never visit lonely senile people, just that they don't use science to do so, because they are not idiots.)
But anyway, I joined in with this discussion because
pw201 is intelligent and interesting, and there was an issue of terminology I was curious about. The discussion led to Paul asserting (relevantly): I think it is fair to say that the established results of the physical and biological sciences are less likely to be overturned than those of the social sciences. Evolution is a fact, current theories of anthropology will be outdated in a few decades.Woah! That really, really brought me up short. I mean, it's trivially not true, but even if it were it wouldn't be a good thing! The whole point of why science is "better" than religion as a way of understanding how the world works is that scientific theories and models get changed when someone finds new data that contradicts the old view. This is a really good example of the way that selling science as an alternative to religion does a massive disservice to science (I care surprisingly little about vocal atheists misrepresenting religion): it leads to people, intelligent people I respect, trying to treat science as a source of eternal verities. I also absolutely disagree that physical science is inherently better than social science; it just isn't, but trying to cram science into the niche where religion or Humanism or other philosophical systems belong can really easily lead to that sort of misguided hierarchy between branches of science.
The thing is, "believing in" science in this way doesn't just offend me as a scientist; it kills people. Let me talk about a lecture I attended recently. The talk was given by the DUETs people, who are working to put conventional, evidence-based medicine on an even more scientific basis. But they are not doing this by claiming that good science should still be true decades and centuries after findings are reported. Quite the opposite! They are claiming that good science, and good evidence-based medicine, should be flexible in how it responds to new evidence, and established views should be constantly challenged. This isn't just to make people feel better intellectually, it's a really critical aspect of patient safety.
Example 1: for many decades in the second half of the 20th century, medical wisdom was that babies should be encouraged to sleep on their fronts. This advice was pretty universal, and even made it to Dr Spock's famous book about childcare. It was based on the best evidence available at the time, but by the 70s there was an increasing body of evidence that sleeping prone is a significant risk factor for cot death. However, this evidence took a very long time (decades) to percolate into mainstream medical advice, because doctors and even the medical research community were reluctant to challenge the established scientific fact. They were especially reluctant to rely on data from soft sciences and observation of large human populations, in order to overturn data based on the "more reliable" physical experiments that led to the earlier bad advice. Dr Spock was wrong, not because he was a bad scientist (neither morally bad nor incompetent), but because cot death wasn't really on the radar at the time he was writing. The data he relied on measured physical parameters of how well individual babies did, and was very likely correct that prone sleeping reproducibly improved those parameters in the short term. It was still wrong, and following Spock just because he had the authority accorded to a successful scientist still led to preventable deaths.
Example 2: some decades ago, there was some robust, reproducible, statistically valid scientific research showing that giving caffeine to premature babies helped to reduce the frequency of a condition called apnoea where the infant briefly ceases breathing. However, this research was often not applied clinically because there wasn't any real evidence to show that reducing apnoea occurrence was particularly important. Nobody was being a bad scientist, nobody was following superstition or religious beliefs at the expense of evidence, there wasn't even a big problem with doctors being unaware of the state of the art of research. It's a perfectly medically valid decision that you don't want to give a powerful drug with unknown long-term effects to premature babies who are extremely vulnerable anyway. It's a perfectly valid ethical decision that you don't want to do double blind randomized controlled trials on premature babies, with the very real possibility of harming them. Again, it took population studies and extrapolations from soft science observations to demonstrate that the frequency of apnoea is correlated with long-term risk of cerebral palsy and reduced life-expectancy. That's a lot of avoidable disability and death because only one sort of clinical trial counts as properly scientific.
Example 3: some decades ago, there was some robust, statistically valid, properly designed and controlled research showing that steroids can be helpful in patients with severe brain injury. So doctors very sensibly started treating brain-injured patients with steroids. And scientists very sensibly did what scientists do, and repeated and extended the original experiments over the course of the intervening decades. They didn't just assume that the original research must be "true" because it was "scientific". They didn't prefer to work on more glamorous, more prestigious new stuff at the expense of low-status confirmatory work. The effect size and statistical significance tended to decline with subsequent studies. This doesn't mean that the original research was wrong, or that the original scientists were biased, incompetent or lying, it's just an artefact of the way that scientific culture works. If you're going to publish something novel, you have to have a pretty watertight case, with strong statistical significance and a relatively big effect, and that's as it should be. But if you're just confirming something that is already known, then rather less dramatic and conclusive results are acceptable because they support the established fact. And of course, we all know but can easily forget that 1 experiment in 100 will show that something is true at the 99% significance level purely by chance and sampling error!
After many decades, a consensus started to emerge that the effect of steroids in brain injured patients was small and not terribly reproducible. Not false, just marginal. Meanwhile, treating people with high doses of powerful steroids has known side-effects. The medical community started to suspect that the definite, quite serious harm caused by steroids was greater than the small, poorly reproducible benefits. But there wasn't enough evidence to stop treating brain injured patients with drugs that might save at least some people's lives, until there was a huge, expensive publicly funded trial involving 10,000 brain injury patients across the EU which definitively proved that steroids do more long-term harm than good in this situation. So, ok, you might well say that this is a happy ending, this is medical research and evidence-based medicine working exactly as they should. But you have to take into account that even an optimal scenario means several decades of people receiving treatments which are actually harmful on balance, and which undoubtedly caused unnecessary deaths and suffering during this time period.
What are the implications for "rationalist" rhetoric? I think the most important is that scientific research, and particularly opinions couched in scientific-sounding language which include numbers, technical jargon and statistics, should be treated with at least a comparable level of skepticism to "woo" and alternative medicine. Lay people can't expect to directly evaluate every individual piece of research they read about; indeed scientists can't do that either, because most of it is outside their field and they have to spend at least some of their time studying new questions rather than confirming, validating and challenging old conclusions. But just accepting something as fact because it's "scientific" is not the way to deal with this!
Just accepting the authority of someone because they have scientific qualifications leads to things like believing Wakefield about MMR because he did experiments and used statistics and medical terms. It leads to believing a popular book based on extremely dubious research because the authors have some academic credentials. And because neuroscience is a "real" science, they have more authority to talk about anthropology and sexual psychology than, you know, actual anthropologists and sexuality researchers because human sciences don't count. It leads to giving racist propaganda the benefit of the doubt, because it uses statistics and hard sciencey jargon. Yes, it is a basic principle of science that one should accept unpalatable results if they are supported by data from well-designed and well-executed experiments. But all those people who piously recite this principle in response to badly-designed, biased and thoroughly debunked "experiments" "proving" that white people are inherently superior to other ethnic groups are strangely unwilling to give the same benefit of the doubt to the vast body of good research indicating that, you know, racism actually harms people. True, you can't weigh and measure those harms, you can't do double-blind experiments, but that doesn't mean that social science is just a matter of what's politically fashionable just now.
And that brings me on to my second point: if you believe that science is the best way of looking at the world, you should also accept that social science is the best way of studying human societies! That's especially the case if you (or the journalists you rely on for your information) can't tell the difference between actual physical / natural science and people using vaguely sciencey technobabble, but even good physics is relatively unhelpful for looking at social and cultural phenomena.
And yes, that goes for medicine too; there is lots of really vital medical information that just isn't going to be found by doing randomized controlled trials and measuring the physical outcomes and applying statistics. Partly because a lot of randomized controlled trials that would be informative are also unethical. And partly because the information that can be measured physically isn't always the most important; "how fast do babies put on weight?" can be measured easily, but a more important research question is "how likely are babies to die for no discernible reason?"
Drug trials are (relatively) easy to carry out in the time-honoured "hard" science way; you give the drug to half the patients and a placebo to the other half, and you measure objective parameters about how well the two groups do. I'm in no way arguing against doing this kind of experiment – hell, I spend most of my working life doing that myself – but it doesn't mean that drugs are the best possible treatment for all possible conditions! For example most patients with joint pain would prefer physiotherapy and exercise rather than strong painkillers (and by the way, the reason I know this is because social scientists did serious research into the issue, not because some arrogant biologist assumed that his credentials totally qualified him to throw together an internet survey.) There is some evidence that the former has more benefits and fewer side-effects for a greater proportion of patients than the latter. But it's rather harder to do a double-blind trial of physiotherapy, and you can't use pure bioscience to answer questions like "how well do patients on this regime integrate into their communities and lead normal lives?" which may be as important as "what is the level of pain-related chemicals in the bloodstream of patients taking this drug versus a placebo?"
And thirdly, I suppose, don't put too much faith in the scientific process. In the best possible circumstances it is slow and inefficient and people get harmed while science is sorting out the answer to difficult questions. When we're talking about medicine, individual variation within the population is inevitable, and however good the evidence is for a particular treatment, that best treatment will do nothing for or actively harm a proportion of patients. And to be honest, the best possible circumstances don't always apply; it's hopelessly naive to believe that all science is pure and unbiased and free of the influence of culture and political and financial considerations! Criticize superstition and woo and political bias, of course, but don't couch your criticisms in terms of assuming that the scientific mainstream is always right. That's bad rhetoric and it's atrociously bad science.
(no subject)
Date: 2011-05-27 09:56 am (UTC)Stockholm: could do afterwork and / or dinner on Tuesday 14th; does that work for you?
(no subject)
Date: 2011-05-27 08:58 pm (UTC)Afterwork followed by dinner sounds good! Hmm, did you ever go to the hummous cafe on Söder? It's right by me. Though doesn't have alcohol, so that would have to come before or after if desired.
But open to suggestions as to where to meet. As to when, I am pretty flexible, but don't usually leave work before about 6, though can do if more convenient.