Why the evidence pyramid is wrong, anecdote is sometimes right, and expert opinion is always the final word

The Kevin Bass Show

In this episode, I discuss why the evidence pyramid is a misleading and sometimes an unscientific way to interpret evidence; why anecdote is sometimes right; and why expert opinion is always the final word.

Specifically, sometimes systematic reviews and meta-analyses of randomized controlled trials are misleading, either designed improperly or using bad data. An excellent example of this is the Ivermectin systematic review and meta-analysis of randomized controlled trials that purported to show that Ivermectin was a useful treatment for COVID-19 in late 2021. This meta-analysis, published in a reputable journal (PMID 34145166) took the Internet by storm when it was used by Ivermectin promoters to show that Ivermectin was effective for COVID-19. It was later shown to have used fraudulent data (no fault of the paper authors), which biased the results and resulted in false conclusions. This gold standard study design spit out a false result which would have misled countless clinicians and resulted in harm if it had been implemented in clinical practice.

This demonstrates that such gold standard study designs cannot be taken at face value. Systematic reviews and meta-analyses are not "gold standard" but rather exist along a continuum of quality.

Similarly, anecdotes, such as the one used by Barry Marshall to establish that H. pylori caused gastric ulcers, are sometimes admissible when the rules of causal inference are followed cautiously. Other times they are not admissible, as with many nutrition anecdotes. It depends on the details of the anecdote. At the high-quality end of "anecdotes", we have N-of-1 trials, which are serious scientific tools.

What all of this establishes is that "high-quality" methods can sometimes be wrong, and "low-quality" methods can sometimes be preferred. All of these require expert methods to properly evaluate and arrive at appropriate conclusions. There is, in a word, no cookbook approach to science, and all good science ultimately depends on competent expert opinion.