1 Part 1: The Value Of Anecdotal Evidence

A few people have asked me in email about why anecdotal evidence is not something that the scientific community tends to accept, and one person even went so far as to say it was stupid to ignore the stories that Lyme disease patients share about their experiences and how certain patients respond to different treatments.

After all, anecdotes are based on real life experiences - and it is possible that a treatment does in fact work and someone's personal experience may be the first indication that there is something meaningful happening - either an improvement in symptoms or a step closer to a cure.

Is ancedotal evidence ever acceptable in scientific research and in professional healthcare? Why or why not?

I'd have to say that in general, it's acceptable on two counts:

1) Cases. When anecdotes are documented very carefully, they are called case reports or a case series. These case reports or case series are about individual patients and doctors' experience in treating them, and their treatment is not part of a controlled experiment. But careful documentation over time of patients' treatments, symptom changes, test results, and other findings at least ensures that information is recorded as events and changes unfold and also can demonstrate other variables which may have led to the patients' conditions (preexisting conditions, addition of new medications, insomnia, etc.).

2) As a means for a starting point for future research. If enough case reports or case series are completed that are pointing to a useful treatment or cure, then this can be used to develop and design controlled studies that can ensure that treatment is specifically helping improve certain symptoms or cure patients. Controlled studies help reduce or eliminate variables which may cloud or muddy the issue of what is or isn't changing patients' conditions, which is why they get more weight as a valid scientific approach than anecdote does.

Under both of these conditions, the information would have to be scrupulously maintained and recorded by someone who is reporting honestly about patients' conditions and their treatments.

Now that these two conditions are defined, what other conditions make anecdotal evidence less acceptable to a scientist who is trying to learn the truth about whether or not a treatment is effective in reducing symptoms or curing a condition?

In a way, we can think back to what I wrote about tomatoes in a comment a little while ago.

Two centuries ago, people in North America and parts of Europe did not make tomato sauce. Pizza as we know it was yet to be invented. The reason was because no one would eat tomatoes, as they were members of the nightshade family and it was believed they were poisonous.

Somewhere along the line, someone must have gotten up the nerve to be the first person to eat a tomato and discover after eating it that they weren't dead. From there, perhaps they went on to discover the joy of fresh salads with tomato, tomatoes roasted over an open fire, and other tomato-based dishes before others began slowly adopting the tomato as an edible item.

That first person could be thought of as brave, stupid, or both. In retrospect, though, most people just eat their spaghetti today without much thought about how people got around to eating it. If that person had been wrong, we would all be sitting around here with more fettucini alfredo than we knew what to do with.

But here's the thing:

What if the first person had gotten sick after eating that first tomato, and lived to tell about it? Maybe people would look at their experience and still regard the tomato as something less than fatal if consumed - but still something toxic and bad to eat. And then they would have been wrong, without knowing what the whole story was.

One of the first flaws in using anecdotal evidence is not being able to consider all the variables that affect one's condition and the outcome. In order for anecdotal evidence to have more value, other variables must be eliminated to make sure of the truth.

Looking at the problem of the tomato and the first person who got sick after eating it, one can come up with the following basic question:

How can you be so sure it was a toxin in the tomato that caused their symptoms?

They could have done any of the following:

• Eaten a bad piece of fish an hour before they ate the tomato, and the fish really made them sick.
• Contracted a stomach flu or virus from their next door neighbor, and that made them sick.
• Eaten an insect that was within the tomato that didn't agree with them.
• Had a small amount of mold or fungus on the tomato that didn't agree with them.
• Been unlucky enough to eat a tomato that was a bad tomato - not ripe enough, or somewhat overripe; perhaps bordering on rotten.
• Had an allergic reaction to the tomato itself.
• Had a medical condition that was completely unrelated to anything they ate or a passing viral infection.

So, one can think of a number of reasons why someone could have eaten a tomato and got ill afterwards - but each of these reasons has nothing to do with the tomato itself being a toxic food.

Without knowing any of these potential reasons, from external observation, one could be led to follow simple cause and effect, where the person observed to have eaten the tomato got sick shortly thereafter and everyone assumed during that time that toxins in the tomato was the reason why.

Well, you know what they say: Never assume, because then you make an...

Anyway, eventually someone else probably heard that others enjoyed eating tomatoes in other countries and lived quite long and fruitful lives despite this, and tried eating a tomato and didn't have any problems. Once enough people began eating tomatoes and not puking their guts out or dropping like flies, then more people felt willing to try eating them - even if they might later turned out to have had a serious allergy to them, at worst - or decided they just didn't like them, at best.

In the end,  most people on the planet have had the experience of eating a tomato by now and most of them don't get horribly ill from consuming them.

It might have taken a while to build up enough anecdotal evidence to convince people that tomatoes were safe to eat. After all, people were convinced for a long time that they were poisonous, and any of the above issues listed may have led to people getting ill around the eating of a tomato. It was probably clearer tomatoes were okay if the people eating them weren't also coming down with the stomach 'flu', washed their tomato before eating it, and they hadn't eaten anything else all day.

So, tomatoes are, by and large, safe to eat - and many would say they're quite tasty, too. (I do.)

Let's talk about ancedotal evidence using the tomato in another way, though - by making a claim.


What if a person were to say, "I feel so much better after eating this tomato," or even "Symptom x has improved ever since I ate a tomato."

Well, I admit that many times after I've eaten a tomato, I feel better, too - but usually it's because I've been starving and finally had something to eat. And it was tasty. And filling. Which brings me a certain amount of emotional and physical satisfaction.

But once I begin saying something as specific as, "Symptom x has improved ever since I ate a tomato", well, then that provokes a question in response to that claim:

How can you be so sure it was the tomato that caused symptom x to improve?

The person who ate the tomato could have done any of the following or had them happen to them:

• Had a temporary improvement in that symptom based on the usual ups and downs of their condition.
• Had a lasting improvement that occurred for reasons unknown to anyone.
• Experienced an improvement due to some medication they'd already been taking over time, and just noticed a distinct improvement after eating the tomato.
• Experienced an improvement due to some medication they'd started recently.
• Experienced an improvement due to some other supplement or food they'd been consuming over time, and just noticed a distinct improvement after eating the tomato.
• Experienced an improvement due to some other supplement or food they'd started consuming recently.
• Experienced improvement because of their frame of mind - belief in the tomato making them feel better actually led to them feeling better. (Also known as the placebo effect.)

And of course, it is possible that symptom x really did improve because they ate that tomato, but then: How would you know?

The only way you could know for certain would be to eliminate all the other variables as much as one possibly could, and test it. Find some way to measure the improvement in symptom x in that person, and eliminate the other variables - then go on to repeat that experiment with a larger group of people.

To be sure that it is the tomato that is leading to symptom improvement, the highest test of the tomato-causes-symptom-x-improvement-hypothesis is to give a fairly homogenous group of people with symptom x (where the above variables are eliminated) a tomato and measure improvement in symptom x post-consumption, and to give another fairly homogeous group of people with symptom x (where the above variables are eliminated) a non-tomato placebo that looks and tastes like a tomato and measure improvement in symptom x post-consumption of that placebo.

Next, don't tell which group has the real tomatoes and which group has the fake or placebo tomatoes. Better yet: Don't even tell the scientists running the experiment who has the real tomatoes and the fake ones, either. That way, when scientists come around to record data from the patients, they can't accidentally let slip in conversation or body language whether or not the patients in front of them are consuming the real tomatoes or the fake ones. They don't know anything about it. So they can't influence the outcome much.

This is what double blind studies are all about - and with them, it's supposed to be easier to obtain more reliable evidence that a particular treatment works (or not).

One of the problems with discussion of improvement of subjective symptoms - such as reporting "I feel better", or "symptom x is better", is that they do not make good outcome measures.

How do you measure them? Where do you draw a line in the sand between, say, pain that is a 6 out of 10 in one person compared to pain that is a 6 out of 10 in the next? What does '5' mean? (Ask Allie - she'll tell you what she thinks it means.) Does one person count having a runny nose as having a cold? If that person takes some Vitamin C, thinking it will help them avoid colds, they might dismiss their runny nose and report (and even remember) that they did not get any colds while taking the treatment.

One thing scientists have said all along is this: The plural of anecdote is anecdotes - not data. 

One has to come up with a well-defined set of criteria that can be consistently measured across a larger population in studies, and subjective reporting is often a problem given the examples above.

So, knowing all of this, what is the person who has already tried all of the tried-and-true, scientifically-studied treatments going to do?

What if one is a Lyme disease (and other tickborne infections) patient and has already blown through a number of antibiotics and is still symptomatic?

These questions and more will be addressed in part 2.

This work by Camp Other is licensed under a Creative Commons
Attribution-NonCommercial-ShareAlike 3.0 Unported License.