In a recent post I said that scientists are often much too dismissive. They are “evidence snobs,” Alex Tabarrok might say. A letter in the current issue of the American Journal of Clinical Nutrition criticizes a important example of just such dismissiveness:
In conclusion, whereas we agree that policy decisions should be evidence-based and not hasty, we do not agree that the evidence base [used to make those decisions] should be constrained to one type of study [long-term randomized controlled trials]—in particular, not to a study design that is inherently limited. Do we really want to wait perhaps decades for results of long-term RCTs, which almost certainly will not provide definitive evidence, while ignoring other relevant evidence involving shorter-term endpoints? An example is provided in the panel’s own summary statement (2). In lauding RCTs as the “gold standard for evidence-based decision making,” the panel proudly points to the fact that, even though folate was well known to decrease the risk of neural tube defects in animal studies, policy recommendations for folate supplementation to prevent neural tube defects were delayed while authorities waited some years for confirmation from RCTs. One can only wonder how many infants were born with neural tube defects while authorities waited.
“Proudly,” huh? Inclusion of that word shows how pissed the authors of the letter are — and rightly so. One author is Bruce Ames, a neighbor of mine, for whom I have great respect; another is Walter Willett, the Harvard epidemiologist. In 1998, Willett wrote a smart article challenging the popular belief that a low-fat diet is a good way to lose weight.
Here is part of the reply from the authors of the report that Ames et al. criticized:
It is important to note that our panel was not charged with asking whether vitamins and minerals play a role in human disease –a topic that occupies much of the letter by Ames et al, and for which observational evidence is indeed central — but, as a State-of-the Science Panel, was charged to reflect on the state of the available evidence for a treatment recommendation on the use of vitamins and minerals in the general population. For treatment decisions, the RCT is the established standard. No better proof of this principle can be found than in the RCTs reviewed in our report, which showed serious harm from vitamin ingestion in certain circumstances.
A less-than-reassuring answer. A commentator on my earlier post thought I should address the strongest arguments on the other side. I had trouble thinking of any. It’s hard to argue that less evidence is better. You can see that those who wrote this paragraph — some of the most prominent nutrition scientists in the country — were equally baffled.
I will revise my “common mistakes” article to mention the Ames et al. letter.
This is a hot debate regarding lung cancer screening. Researchers at Cornell led by Dr. Claudia Henschke have for years published data from their observational studies that found low-dose CT screening in high risk subjects saves lives. Henschke et al screen all subjects with CT and then randomize them to either treatment or not treatment.
On the other side, are investigators for the National Lung Screening Trail, an 8-year, $200 million NIH RCT study whose results won’t be available until 2010. In this trial, subjects are randomized to either CT or chest x-ray.
The observational studies have looked at thousands of cases, amassed impressive data not only regarding the efficacy of screening but also protocols for reducing false positives and downstream invasive procedures, and protocols for observing differences in lesion makeup and minute lesion growth. A model they constructed claims to reduce lung cancer mortality by 80% when caught early in stage I, which is what screening does.
The folks at NLST always point out that Henschke et al have not proven that screening actually saves lives. The observational researchers claim to have an effective “cure” rate, which they extrapolate to a reduction in mortality. But the NLST investigators claim that many of the cancers found by Henschke et al are indolent and not biologically dangerous. Subjects would have died by other causes with those lesions present. NLST advocates claim that Henschke et al muddy the waters talking about cure rates rather than a reduction in deaths. They claim that only the RCT study will prove whether CT screening reduces mortality or not. In the meantime, their best advice is to quit smoking.
And so it goes. At medical conferences, these two opposing camps sometimes snipe and spar with each other, obfuscating any real news that might be presented.
In the meantime, a British panel has deemed lung cancer screening safe and effective based on the Cornell research.
David Yankelevitz at Cornell wrote a good article dissecting the two approaches.