Science 101 series. You can’t “disprove” trial research using anecdotes: an example and explanation in the field of type 2 diabetes

Reading Time: 3 minutes

I don’t much care for Neal Barnard as a nutrition authority–I frequently complain about him to my vegan friends–but the scientific illiteracy demonstrated in this tweet is frustrating and sad. And tweets like this are representative of much of nutrition Twitter. “Study shows X. Yet I saw Y. Conspiracy!!!” So this post is really about the use of anecdote in science. EIAsjaSWsAAupMuThe results in the Barnard study represent the intervention group average under certain trial conditions; the anecdote on the other hand represents a single person’s experience under what could be entirely different conditions. 

Barnard’s study is not incompatible with many individuals in that study also experiencing a similar outcome as the anecdote. It is even possible that Bernard’s intervention is more effective on average than the intervention advocated by the person criticizing Bernard. 

This is because the person criticizing Bernard might in fact be an outlier compared to many other people pursuing the same strategy as him. Without more information, we don’t know.

As an aside, I do think a low-carbohydrate diet can play an important role in treating type 2 diabetes. I also think that a high-carbohydrate diet can also effectively treat type 2 diabetes.

The current evidence suggest that weight loss is the main factor in type 2 diabetes treatment, and that carbohydrate restriction can produce immediate gains in glycemic control that are not obtainable on a high-carbohydrate diet.

This would make carbohydrate restriction a desirable treatment modality in patients who cannot permanently lose adequate weight to reverse the underlying metabolic dysfunction. However, because a low- and high-carbohydrate diet do not produce different levels of weight loss, inasmuch as the underlying metabolic dysfunction is being reversed (which is driven by organ fat), a low- and high-carbohydrate diet are in this respect basically equivalent.

Back to the scientific problem.

Such extreme anecdotes lend toward their being overreported. That’s why we hear about them instead of about failures, etc. This makes the testimonials of people who follow diet gurus look so much more impressive than most trial results. This is also why case studies are not regarded as acceptable scientific evidence for an intervention: because they could be highly selective and unusual.

In fact, this is why small trials are regarded with the same skepticism.

A well-known pattern in the Alzheimer’s literature is in fact a highly successful case study or small trial followed by a failed larger clinical trial.

What seemed extraordinarily promising turns out to be totally ineffective when tested rigorously and in large numbers. This is in part because, out of many small trials being conducted, a number of them will turn out to show great promise by sheer chance. Large trials showing such benefit by sheer chance are far, far less likely.

In any case, this person’s anecdote amounts to a case study with less documentation. It is not even as trustworthy as a small trial.

What is frustrating is how popular this sort of anecdote is on Twitter. These anecdotes are only compelling if one believes that their representativeness of the efficacy of the treatment itself has been shown. But this is not the case. To my awareness, high- vs. low-carbohydrate diabetes reversal trials (using highly intensive interventions in each group, such as with Virta) have never been conducted.

I don’t think that a vegan diet in itself is a good idea for treating diabetes, and I do think that Bernard’s results are rather unimpressive and show the inadequacy of his specific intervention for treating the disease. However, the use of an anecdote to criticize the study is just plain silly. An anecdote cannot be used to criticize a large trial. That people seem to think that it can shows that many people simply do not understand what or why scientists do what they do.

It is my hope that some people who are just now learning about science can benefit from this discussion. If you would like to learn more, please ask questions in replies or over DM.

Producing high-quality scientific work like this takes many hours of work. This is made possible through the support of readers and viewers like you. Please consider donating below.

In just a few clicks, sign up to donate $5 monthly here:
Or $20 monthly here:

For one-time donations, or to become a patron on Patreon, click here.

Advertisements

Posted In:

2 Comments

  1. It took me quite a while (75% through) to figure out your thrust of this post. Why? Your post is about an individual using his own experience (anecdote) to disprove Dr. Neal Barnard’s study. Yet you started out describing your dislike of Dr Barnard, then “ but the scientific illiteracy demonstrated in this tweet” which presumes the reader knows the background of both.

    It took me time to figure out both issues, that you disliked Dr Barnard (though I could not figure out why in this post, so why include this view?) and there was someone on Twitter who criticized the study. This theme was eventually fleshed out, and was worthwhile to figure out your point.

    1. Your point being: people reading large studies “debunk” these by using their individual experiences, or by other few outliers they know of or have read about. Your view is that while this is common, it is scientific illiteracy and needs to be restrained from use for those whom wish to use nutritional research to best advantage.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.