Understanding scientific evidence

The question behind every claim

When a product says it is “backed by science”, four questions decide whether that means anything. What exactly was tested? In whom? Against what comparison? And what was measured? Most of the distance between a promising study and a useful product is hidden in the answers.

What each kind of study can and cannot show

In silico (computer models)

Predicts whether a molecule might bind a target. Generates hypotheses; shows nothing about effects in a body.

In vitro (cells or proteins in a dish)

Shows that a compound can act on a target at the concentration tested. It cannot show that the concentration is reachable in people. Some compounds also interfere with the tests themselves: curcumin is classed as a pan-assay interference compound for that reason ⁴.

Animal studies

Show effects in a whole living system. Valuable, but species differ in metabolism, dose and lifespan. Rapamycin extended lifespan in mice even when started late in life ⁵; no trial has shown the same in people.

Observational human studies

Compare people who happen to take or eat something with people who do not. Good for spotting patterns and harms, weak at proving cause, because the groups differ in many other ways.

Randomized controlled trials

Assign people at random to the intervention or a control, which balances other differences. The strongest single design for cause and effect, but only for the outcome, dose, form, duration and population studied.

Systematic reviews and meta-analyses

Gather and pool all trials on a question. As good as the trials they include, and they must show how much those trials disagree.

This is not a strict ladder. A large, careful observational study can answer some safety questions better than a small, sloppy trial. What matters is how well a study was designed and run for the question being asked.

When patterns and trials disagree

Before the AREDS2 trial, observational data suggested that people who ate more lutein and zeaxanthin, or more omega-3 fats, might have a lower risk of advanced macular degeneration. When the idea was tested at random in people already at risk, adding these nutrients to the existing formula did not further reduce progression in the main analysis ⁶. Patterns in the population are where questions start; trials are where they get answered.

Reading the numbers

Statistical significance versus importance

A result is “statistically significant” when chance is an unlikely explanation. It says nothing about size. A meta-analysis of 19 trials found melatonin shortened time to fall asleep by 7.06 minutes, with a 95% confidence interval of 4.37 to 9.75 minutes ¹. That is real and consistent, and also modest. Whether 7 minutes matters is a judgment, not a statistic.

Confidence intervals

A confidence interval is the range of effects compatible with the data. In the VITAL trial, the hazard ratio for invasive cancer with vitamin D was 0.96, with a 95% confidence interval of 0.88 to 1.06 ⁷. Because the range runs from a 12% lower to a 6% higher rate, the trial could not distinguish a small benefit from a small harm, and the best estimate was close to no effect.

Relative versus absolute risk

“Cuts risk by half” sounds the same whether risk falls from 2 in 100 to 1 in 100, or from 2 in 100,000 to 1 in 100,000. The relative change is identical; the absolute change differs a thousandfold. Always look for the absolute numbers, or the number of people who would need to take something for one to benefit. (These figures are illustrative.)

Heterogeneity

A pooled result can hide trials that disagree. In a meta-analysis of turmeric for knee pain, the I-squared statistic, which measures disagreement between trials, was 86%: the trials pointed the same way but differed hugely in size of effect ⁸. High heterogeneity means the average may not describe any real product or patient.

Markers versus outcomes that matter

Many studies measure a biomarker instead of how people feel or function, because markers change faster. Markers can mislead. In a small randomized trial, nicotinamide riboside raised NAD+ metabolism within weeks, but effects on blood pressure and arterial stiffness were not established and were left for larger trials ⁹. A marker moving is a reason to run an outcome trial, not a substitute for one.

Bias, funding and the missing studies

Findings are less likely to be true when studies are small, effects are small, designs and outcomes are flexible, and there is financial interest or intense competition in a field ³. Negative results also tend to go unpublished, which tilts the visible literature toward positive findings. Good reviews search trial registries for unpublished studies and test for this bias ².

Certainty is outcome-specific

The GRADE approach rates certainty separately for each outcome, considering risk of bias, inconsistency between studies, indirectness, imprecision and publication bias ². We follow the same principle. Turmeric can have limited evidence for knee pain and no support for joint structure at the same time ⁸. “Does turmeric work?” has no answer; “does this extract, at this dose, reduce knee pain over 12 weeks in adults with osteoarthritis?” does.

Interesting, promising, supported, actionable

• Interesting: a plausible mechanism or a striking lab result.
• Promising: consistent preclinical results and early human signals.
• Clinically supported: replicated randomized trials for a specific outcome, form and dose.
• Actionable: supported, with a benefit large enough to matter, acceptable safety, and a product that matches what was studied.

Most supplement marketing presents the first level in the language of the fourth. The Research to Reality tool walks through real cases stage by stage.

Common misconceptions

• “Peer-reviewed means proven.” Peer review checks that a study is reasonable to publish, not that its conclusion is right.
• “A mechanism explains why it works.” A mechanism explains why it might. Many plausible mechanisms have failed in trials, including antioxidants for mortality ¹⁰.
• “No significant effect means no effect.” It can also mean a trial was too small or short. Look at the confidence interval.

Practical takeaways

1. Find the actual study. Check who was studied, what was given, for how long, and what was measured.
2. Look for the size of the effect and its confidence interval, not just “significant”.
3. Prefer outcomes people feel or experience over markers.
4. Ask whether the product you are considering matches what was tested.

Technical layer: worked reading of a meta-analysis

The turmeric knee meta-analysis reports a standardized mean difference (SMD) for pain of -0.82 (95% CI -1.17 to -0.47) versus placebo ⁸. An SMD expresses the difference in units of standard deviation so trials using different pain scales can be pooled; by a common convention, 0.8 is a large effect. Read alongside I-squared of 86% and moderate risk of bias, the honest summary is: a consistent direction, an uncertain size, and a high chance that some of the larger trial effects reflect bias or product differences.

Meta-regression found body mass index explained 38% of heterogeneity in pain and 67% in function, with less improvement at higher BMI ⁸. Subgroup and meta-regression findings are hypothesis-generating: they identify where to look next, not reliable rules for individuals.