Statistics in Small Doses 18 – What about systematic reviews and meta-analyses?

In the last Statistics in small doses, it was concluded that the triangle of study designs can be misleading if viewed as a hierarchy. The clinical trial, sitting as it does above the observational studies, is considered by many as the ‘gold standard’ of study designs even though it is not suitable, feasible or ethical to use that design to study all research questions. So let us proceed to the pinnacle of the triangle; surely systematic reviews and meta-analyses will give us the right answer – after all, aren’t hours of reading and careful examination involved? Yes, but the summer 2016 edition of Columbia Magazine includes an article that gives us cause for pause – “The great salt debate that wasn’t.”

Are we or are we not eating too much salt? Researchers from Columbia’s Mailman School of Public Health did a meta-analysis of hundreds of studies published between 1979 and 2014 that investigated the health effects of salt consumption. They found “two almost distinct bodies of scholarship.” One camp claimed that reduction of salt intake lowers risk for heart disease, stroke, and death while the other camp claimed that the evidence was too scant. Not only did experts on both sides of the question tend to cite those reports that drew conclusions similar to their own, but they did a poor job evaluating reports of the opposing view. What’s the lesson learned from such academic logrolling? Rather than rushing to judgment on those reports whose ideas run counter to our own, we should instead focus fresh eyes on the quality of a study’s design, execution, and statistical approach.

To that end, next month’s Statistics in small doses journeys into the realm of parametric statistics. We begin with the question, “Why do the tests that inhabit this realm consider their membership ‘exclusive’?”