Types and Strength of Scientific Evidence

We try to report science and base the information on this website on the best science we can find. We recognize – as should everyone – that all scientific reports are not equal. Some experiments and studies are done with more care than others and some are better designed than others. This recognition is not an insult to well-meaning investigators. Large studies are expensive and often impractical even if money were available. Experiments or studies with many patients provide stronger claims than smaller ones do, purely because of statistical considerations. Further, observations of patients over a long time provide stronger evidence than short-term looks.

We must also distinguish between experiments and observations. Experiments – often done to test treatments – are under the control of the investigators. A well-designed experiment with a large patient pool can provide powerful evidence. Observational studies can also provide good evidence. The environment and setup are not under investigator control, and often these studies follow patients for years or even decades.

It is also worth distinguishing between randomized clinical trials (RCTs) and controlled clinical trials (CCTs).

A CCT is a more general term for a clinical study between two groups. Patients in one group get the treatment or therapy of interest. The other group (the control group) gets no treatment or a placebo, or another treatment.

RCTs imply that patients are assigned to groups randomly. The patients agree to participate in the study but they do not get to choose whether they are in the treatment group or the control group, and in most cases do not know which group they are in.

You sometimes run into the term "double blind". That means neither the patients nor the investigators know which patients are getting the treatment under investigation. It is more difficult to design a double-blind RCT and set protocols and follow them, but if you can do it, RCTs are the best you can do, and all other things being equal, provide the strongest evidence.

Formal observation studies include cohort studies and case-control studies.

A cohort study is an observational study but one in which the data is approached in a way so as to simulate an experiment to some extent. These studies often take historical data about patients or follow them over a period time. The word cohort indicates a group of people with something in common such as a shared experience in their past, same age, etc. Case-control studies look at two populations with different attributes in some area of interests compared. The analyst then attempts to figure out (or make an argument for) what in the past produced the difference. For instance, if we had two groups of people in their 80s – one with dementia and one without – and we knew their sleep habits when there were younger, we could try to see if there was a difference between the sleep habits of the two groups and develop evidence that sleep affects the risk for dementia.

In the scientific/academic/regulatory world, there is a hierarchy of sources of evidence that goes roughly like this:

Expert opinion in last place? Well, yes, but responsible experts who keep up with the news in their fields will have incorporated the findings of the experiments and observational studies into their judgment. Expert opinion should be the primary source of evidence only when nothing else is available such as when unusual situations are encountered. Case reports are anecdotal but that doesn’t mean they are useless. We just have to remember that one-time anecdotes have limits – the experiences of one patient do not necessarily mean others will have the same results, problems, and benefits.

Critical Thinking

In considering a question, problem, issue - you have to look at available evidence – which may be contradictory – and make an assessment of the evidence. It is not always easy, but this is where experience in weighing evidence and drawing conclusions – and in knowing how firm your conclusion is – plays a part.

The Sleepdex book is now available on Amazon.com.

Click here