A controversial study that claims to have found proof that acupuncture is effective in treating colic in infants has been widely rebuked. In fact, even the associate editor of the journal where the study was published said “in technical terms, the study is negative … the primary outcome did not turn out to be statistically significant”.
The paper was authored by two academics from Lund University in Sweden, and involved blind testing of 147 infants over a two-week period. The infants were divided into three groups, variously treated with traditional acupuncture (up to five needle points), minimal acupuncture (one needle point) and no acupuncture.
The authors claimed evidence that those infants treated with either form of acupuncture performed better in alleviating colic, fussing and crying than those in the no-acupuncture group.
The paper appeared in the Acupuncture in Medicine journal, itself published by the British Medical Journal. The BMJ’s involvement with the AIM has been an aspect of the debate that has also raised the eyebrows (if not the ire) of many commentators.
Much criticism has been laid against the small sample size for the research. Michael Cummings, the AIM associate editor mentioned above, agreed it was small but put that down to lack of funding.
But most criticism by qualified commentators is in reference to the interpretation of the data and statistical significance (if any) of the results.
While Cummings’ comment about the results being “not statistically significant” may have been given begrudgingly, others have been much more overt and harsher.
Prof Edzard Ernst, Emeritus Professor of Complementary Medicine at the University of Exeter, and noted campaigner against poor evidence in alternative medicine, said that the study “confirms previous research showing that acupuncture is a ‘theatrical placebo’. The study was designed without an adequate placebo group; therefore, conclusions about specific therapeutic effects of acupuncture are not permissible.
“We know that colicky babies respond even to minimal attention, and this trial confirms that a little additional TLC will generate an effect. The observed outcome is therefore not necessarily related to acupuncture.”
Prof David Colquhoun, Emeritus Professor of Pharmacology at University College London, was even blunter, describing it as “one of the worst papers on acupuncture that I’ve come across”.
Colquhoun, who told Australian Skeptics that “I’ve been dealing with little else but this ever since it came out”, said in a UK Science Media Centre Roundup that “The statistical analysis in the paper is incompetent. This should have been detected by the referees, but wasn’t.
“For a start, the opening statement, ‘A two-sided P value ≤0.05 was considered statistically significant’ is simply unacceptable in the light of all recent work about reproducibility.”
(More detail of Colquhoun’s views on the statistics procedure involved in the research can be found in the Roundup.)
The paper received much publicity, with hundreds of media outlets covering the research, often basing their reports on a press release issued by the BMJ.
Calling the release “hyperbole” and “shockingly misleading”, Colquhoun said in his blog DC’s Improbable Science that “The BMJ group should not be mixing up press releases about real medicine with press releases about quackery. There seems to be something about quackery that’s clickbait for the mainstream media.”
Nonetheless, many media outlets also included comments from Colquhoun and others decrying what they said were sloppy statistical analysis and over-reaching conclusions.
The one person quoted in the Science Media Centre Roundup with a positive response was Prof George Lewith, Professor of Health Research at the University of Southampton, who said: “This looks to me to be a good sized fastidious well conducted study (Cochrane A for an acupuncture trail in which the therapist cannot be blinded). The outcome is clear and the power of the study seems reasonable which suggests that [medical acupuncture] is a reasonable and as far as we know safe intervention for infantile colic.”
However, Colquhoun described Lewith as “the well-known apologist for all things quackish” adding that Lewith showed what he called “an ignorance of statistics that would shame an undergraduate”.
Colquhoun summed up: “The design of the trial is reasonable (apart from the control group) but the statistical analysis is appalling. It’s very likely that there aren’t any real effects of acupuncture at all. This paper serves more to muddy the waters than to add useful information. It’s a model for the sort of mistakes that have led to the crisis in reproducibility. The BMJ should not be publishing this sort of stuff, and the referees seem to have no understanding of statistics.”