So
the good ol' Dana Ullman was arguing about homeopathy on twitter
again. (I know tell you something new right?) and I asked for what he
calls “good science” published in a peer review journal. He
provided two links so for:
Which
is a “Review
of 4 high quality studies on respiratory allergies” according
to him and “To
test the hypothesis that homoeopathy is a placebo by examining its
effect in patients with allergic rhinitis and so contest the evidence
from three previous trials in this series.”
according to the study objective. Sure why not, totally a review of 4
studies. Lets look at the paper though:
The
intervention was “Random
assignment to an oral 30c homoeopathic preparation of principal
inhalant allergen or to placebo.”
Alright, makes sense. What next? My favorite part:
“This trial, the fourth in a series, was designed in response to a challenge from an independent clinical team to contest the evidence from the three preceding trials that homoeopathic dilutions seem to differ from placebo”
Designed
to contest evidence from THREE preceding trials. So why link to this
one? No idea.
“These were not trials of treatments; they were designed to address the placebo hypothesis, using allergy as a model. In this study, as before, patients with atopic inhalant allergies received, randomly and double blind, either an oral 30c homoeopathic preparation of their principal allergen or a placebo. The previous trials studied effects in atopic patients with hay fever1,2 and asthma,3whereas this study focused on perennial allergic rhinitis. We report the results of this fourth trial and an overview of the series.”
Ohhhh
so it also
has an overview of the other studies. Good to know. Still not exactly
a review of 4 studies but still. Now we know that the study was
double blind and randomized right? Welll..... it's more complicated
than that.
“At the start of the qualification period the doctor assessed each patient's history, allergy status, and nasal obstruction. The principal allergen determining the prescription was then chosen on the basis of the largest skin test weal concordant with the allergy history. In seven cases in which the prescriber had difficulty in determining the principal allergen, telephone advice was given by a doctor experienced in prescribing homoeopathic immunotherapy. The first of three phials of placebo corresponding to the principal allergen was then administered by the doctor on to the patient's tongue. Patients were unaware that the phials contained placebo, although the researchers were not blinded.
The following two weeks served jointly as the qualification and baseline period. At the second visit, qualifying patients were randomised by a restricted technique of permuted blocks of two,7 generated from random number tables and stratified for the indicated allergen. A double blinded prescription was immediately dispensed. The trial ended with a follow up visit four weeks later.”
(Bolding
mine) So...
not entirely double blinded then right? Why would you, in a double
blinded study single blind the first round? Again, no idea. Also
while the qualifications make sense why did they keep in the 7 cases
that were difficult to determine? How valid was this telephone
advice? I am wary of that personally but moving on.
“At the same time each morning and evening patients recorded three successive nasal inspiratory peak flow measurements. Before making these measurements patients oriented themselves by noting each morning (on a 0 to 4 integer scale) how their symptoms had interfered with their sleep and, each night, rating blocked, runny, or itchy nose symptoms, sneezing, and any eye and chest symptoms. Patients then recorded their daily overall visual analogue scale score. To allow comparison with our previous trials1–3 the identical wording was used: “Overall today I felt . . .” on a scale of 0-100 mm, where 0 is fine and 100 is terrible. Visual analogue scale scores are a recommended measure of the severity of rhinitis.16 Adverse events, including initial aggravations of symptoms as observed in our previous rhinitis trial,2 were documented by the patients, clarified by the doctor, and recorded on standard adverse experience report forms. Any use of conventional drugs was also noted.”
Any
use of “conventional drugs” being noted? Why is that? Oh yeah
because patients
were allowed to use conventional medicine during the trial.
Maybe that made some sort of difference? I would think so but then
again, I am not a doctor, I am a psychologist.
Moving
on to the analysis I loved this part:
“With
a choice of 5% significance and 80% power, we estimated that 60
patients would be required in each group to avoid false negative
results.”
60
patients
would be required. But wait... lets go back up to the top of this
very study to find... “51
patients with perennial allergic rhinitis.”
Well. No avoiding false negative results then. Moving on “To
summarise our results, the original data19 from
all four trials were pooled and analysed by an independent worker
using Cochrane Collaboration meta-analysis software”
Wait. So this wasn't an independent study? It was more of an, we
didn't like the results of the first trials so we're going to jimmy
in some more patients rather than use the data like we said we would?
“Statistical
heterogeneity was assumed to be present when the P value for
heterogeneity was less than 0.10, and therefore the random effects
model was used
“ Ohhhh
well. Lets use a P=.10 because that's totally acceptable in most
trials where at least P=.05 is the norm. Alright, whatever you say.
Into
the results section. Maybe they explain why they didn't even get
their own stated significance levels:
“Fifty one patients successfully completed qualification screening and were randomised (fig (fig1).1). Because of the exacting screening, strict qualification criteria, and the prospectively defined requirement to stop enrolment before the pollen season, we did not recruit the number of patients that the power calculation had estimated we required.”
We
did not recruit the number of patients that we required. So why are
you continuing the study? Why didn't you screen enough people to get
at
least
the 60 people you
knew you required.
Nonsense.
“Initial aggravations of rhinitis symptoms were provoked more by homoeopathy than by placebo. By 48 hours after randomisation seven (29%) patients in the homoeopathy group reported a worsening of rhinitis, two with wheeze, compared with two (7%) patients in the placebo group, neither of whom had wheezing (P=0.04, Fisher's exact test). By 14 days, 11 (46%) patients in the homoeopathy group had reported adverse events, 10 of whom had rhinitis related aggravations, compared with seven (26%) in the placebo group, five of whom had rhinitis related aggravations (χ2=3.28, P=0.07). In general, most aggravations were short lived, averaging four days, and all had resolved by day 16. Aggravations of rhinitis in patients who received homoeopathy seemed to point to a good outcome. Initial deterioration was followed by subjective improvement and a corresponding improvement in nasal inspiratory peak flow. Only one patient in each group resorted to conventional rhinitis drugs, and both took them for less than four days.”
Believe
it or not I can see a reason this pattern may have occurred. Why?
Because the first randomization set was not
double blinded.
It makes sense that the homeopathy group would report a worsening of
symptoms since that is what patients are told to expect with
homeopathy. The standard line is “you'll get worse before you get
better” I should know. I used that line many times while I was in
school for this stuff. Also the whole “aggravations of rhinitis in
patients who received homeopathy seemed to point to a good outcome”
thing makes perfect sense because it's likely that short lived events
like rhinitis aggravations would regress to the mean quickly and
because the initial sweep of treatments was not double blinded so
over time the self induced symptoms would disappear just like the way
homeopaths say they should since the
person likely believes in it. Why
do I say that? Because way back up at the top of the study there was
this little line that mentioned that they were recruiting people that
were familiar with homeopathy. Hmmmm that may
have an impact when the blinding isn't exactly as is claimed.
“On average, over the last two weeks after randomisation, patients who received homoeopathy had a 28% improvement compared with 3% among those in the placebo group”
Well,
given that in the first two weeks, as we have already seen, the
homeopathy group had an increase
in symptoms they had further to go to improve than the placebo group
did. This change makes sense and DOESN'T make homeopathy different
than placebo.
I
am going to quote out much of the discussion here because it's
fantastic:
“Although the objective measure consistently improved in all five centres, the subjective results were better than placebo in only four of the five centres and overall there was no difference between the groups. If the objective results are valid the discrepancy in the subjective measurements may be partly due to under recruitment compounded by aggravations and possible initial placebo responses during the run-in period in both groups, perhaps reflecting the positive expectations of the participants.23 Patients with rhinitis are keen to enter studies in their quest for better symptom control.24Subjective improvement began before the end of the placebo run-in phase in both groups, and this lessened the chance of distinguishing between the groups.25,26 A larger sample size may have shown a subjective difference between the groups. More initial aggravations occurred in patients who received homoeopathy, and this may have further complicated the subjective results. The pattern of temporary worsening followed by improvement, seen in this trial and observed in clinical homoeopathy for over 200 years,27,28 is not typical of placebo.”
So...
objective measures are the same while subjective measures show no
difference. Alright, I can buy that ignoring my previous issues with
this. Did you catch the fantastic part? “ If
the objective results are valid the discrepancy in the subjective
measurements may
be partly due to under recruitment
compounded by aggravations and possible
initial placebo responses during the run-in period
in both groups, perhaps reflecting the positive expectations of the
participants.23 Patients
with rhinitis are keen to enter studies in their quest for better
symptom control”
That's right. The results may be partially due to under recruitment.
Which the study had despite that last line there. Patients are keen
to enter studies. So why did this one have under recruitment I
wonder? Hmm.
There
are no declared conflicts of interest for this study and I am curious
to see if there is any link to homeopathy that I can find with a
quick google search of the authors. Here I go:
Morag
A Taylor, research associate:
I
found a Morag Taylor that is a Neuroscience research assistant that
seems to do falls and balance research. Doesn't seem right, but then
neither does the only other Morag Taylor I can find involved in
research: the one from the University of Leeds who, according tho
their linked-in profile “I work as part of a large team within the
colorectal cancer (CRC) research team. I am involved in several
different projects looking at biomarkers of CRC both pre and post
patient diagnosis, to better understand both disease progression and
response to treatment respectively.” Again doesn't exactly seem
like they would do a homeopathy study like this. Curious.
David
Reilly, honorary
senior lecturer in medicine:
Ohhh
Score here:
“Dr
David Reilly, Scottish
Government's Lead Clinician for Integrative Care & Director of
TheWEL and The Healing Shift Programmes.
Honorary Senior Lecturer in Medicine, The University of Glasgow.
Consultant Physician, The NHS Centre for Integrative Care. - See more at: http://www.gcph.co.uk/events/125#sthash.yNnc2GiH.dpuf”
Honorary Senior Lecturer in Medicine, The University of Glasgow.
Consultant Physician, The NHS Centre for Integrative Care. - See more at: http://www.gcph.co.uk/events/125#sthash.yNnc2GiH.dpuf”
I
am leaving that link in because frankly it's too fantastic to pass
up. He works in integrative care, I poked at his webpage and low and
behold... he's a homeopath!
Robert
H Llewellyn-Jones,lecturer:
He's
a psychologist focusing on dementia, depression and other mood
disorders What this has to do with allergies I have no idea. Apart
from the two homeopathy trials he participated in I don't see
anything else in his work that points to any reason to be
participating in this research.
Charles
McSharry, principal
immunologist :
Oh
hey someone relevant! An immunologist that deals with asthma!
Tom
C Aitchison, senior
lecturer in statistics :
I
can't find much on this guy except links to stuff he's published
which seem to be on a range of mobility issues. Maybe he works with
the Morag Taylor that works in falls and balance research. Or Robert
Llewellyn-jones's work with aging people with dementia. No idea to be
honest.
Overall
I am not impressed with good ol' Dana Ullman's “research” here.
He sent me a second one but I feel no need to take that
piece of work apart as it was done for me by the lovely guys over at
Skeptics with a K way back in about the second episode of their
podcast. Bah Dana. Do better.
Oh, here is the second link. Just for fun. Review of 3 high quality studies on childhood diarrhea: http://www.ncbi.nlm.nih.gov/pubmed/12634583