Les Jones

How “Breakthrough: Medical Findings Disappear:

A Relative Risk (RR) of 1.0 would mean there was no benefit, and a number less than 1.0 indicates a health benefit, and a number greater than 1.0 indicates that the intervention is actually harmful. When there were only a little data available, fish oil looked like manna from heaven. But with new studies and more data, the beneficial effect has shrunk to almost nothing. The current best estimate of relative risk (bottom row of table) is 0.96, barely below 1.0. And the “confidence interval” (the range of numbers in parentheses), which is an indicator of how reliable the current estimate is, actually runs to a value slightly greater than 1.0.

Why does this happen? Small studies do a poor job of reliably estimating the effects of medical interventions. For a small study (such as Sacks’ and Leng’s early work in the top two rows of the table) to get published, it needs to show a big effect — no one is interested in a small study that found nothing. It is likely that many other small studies of fish oil pills were conducted at the same time of Sacks’ and Leng’s, found no benefit and were therefore not published. But by the play of chance, it was only a matter of time before a small study found what looked like a big enough effect to warrant publication in a journal editor’s eyes.

The BBC reports on a study of the health consequences of eating processed meat:

One in every 17 people followed in the study died. However, those eating more than 160g of processed meat a day – roughly two sausages and a slice of bacon – were 44% more likely to die over a typical follow-up time of 12.7 years than those eating about 20g.

The experiment involved 1000 people who were put in cages and fed a controlled diet, with 500 eating bacon and sausage every day and a control group of 500 who ate no bacon or sausage. Researchers kept the humans in cages for 40 years and tracked the incidence of heart disease, cancer, and other conditions.

Except, you know, that isn’t what happened, because even in mad scientist circles  experimenting on humans in cages is a no-no. So instead the researchers did the next best thing. And I use “next best thing” in the sense that a kick in the teeth is the next best thing to eating ice cream. The researchers gave surveys to experimental subjects, who self-reported their diet and other habits. Researchers then tracked their health over the years.

The study “showed people who ate a lot of processed meat were also more likely to smoke, be obese and have other behaviours known to damage health. However, the researchers said even after those risk factors were accounted for, processed meat still damaged health.”

For the sake of argument I’ll assume the researchers were 100% successful in controlling for smoking, obesity and other listed factors. I’ll even grant that they were so darned good they could control for the difference between a pack a day smoker and a two pack a day smoker.

Here’s the thing. People lie. They don’t want to judged. When it comes to their behavior they know to give the right answer and not the true answer.

The two pack a day smoker self-reports as a pack a day smoker and tells himself he’s at least being honest enought to own up to his tobacco monkey.  A woman drinks a bottle of wine every night, has margaritas on Tuesday night with her fajitas, downs a few martinis with the girls on Friday night, and has beers and Jaegermeister at the football party. When asked how much she drinks she checks the box for “one glass of wine per day.”

So the researchers might think they’re controlling for those factors, but their coefficients for them is too low. The man’s health problems are from under-reported smoking, not bacon. Likewise for the woman’s drinking.

Too, even if they’re controlling for those factors, who says they’ve controlled for all possible factors? They accounted for smoking, drinking, and obesity. How about other health behaviors? If they asked about illegal drug use people are even less likely to self-report honestly. Likewise, some jobs bring risk factors. Coal miners shouldn’t have the same statistical treatment as accountants.

So sure, intuitively I think most people would be better off eating less bacon because of the fat, salt, and preservatives and I’m pretty sure it does have an effect on health and lifespan. Just as intuitively, I just don’t find myself entirely convinced by the study’s claim that eating six ounces of processed meat per day will double your risk of death over 13 years, and the statistical nature of the study makes it easier for me to doubt the results.