Most “it works” claims quietly depend on an unstated comparison. If you’ve ever watched two headlines about the same symptom clash—one saying a treatment “works,” another saying it “doesn’t”—this is often the missing piece. In clinical research, an effect is a difference between groups, defined by both the intervention and the comparator. That is why trial reporting standards ask authors to report the primary result as a between-group effect size with precision (CONSORT 2010, Item 17a), and to describe what each group actually received (CONSORT 2010, Item 5). Cochrane methods and ICH E9(R1) make the same point more formally: a “treatment effect” only exists once the conditions being compared are specified (Cochrane Handbook; ICH E9(R1) estimands). If a write-up emphasizes within-group improvement (“symptoms got better”) and jumps to “it works,” it may be mixing natural history, regression to the mean, and context effects into one convenient story.
This piece is here to make that hidden comparator visible, because comparator literacy is one of the fastest ways to reduce confusion when evidence seems contradictory, especially in symptom-heavy areas like women’s health. This matters even more when trials under-represent key groups (for example different ages, ethnicities, or postpartum status), because baseline risk and “usual care” can differ—so the same comparator label can hide very different realities. Comparator choice changes the question a study can answer (placebo/sham vs usual care vs active comparator) (ICH E9(R1)). And with symptom scores, the control group can quietly do a lot of the work. Control conditions can act like part of the “active ingredient” when outcomes are subjective (for example pain, fatigue, hot flashes), with placebo vs no-treatment gaps often larger for subjective than objective outcomes (Hróbjartsson & Gøtzsche, 2001; later meta-analyses showing the same subjective-vs-objective pattern). A “failed replication” is sometimes just a different contrast (for example waitlist control vs attention-matched control), not a true contradiction (Cunningham et al., 2013; Boutron et al., 2008).
By the end, you’ll have a practical, repeatable way to read results as “improved more than ___,” extract the comparator details that matter (CONSORT Item 5; TIDieR; CONSORT-NPT), and do a quick bias-signal check, especially for the high-risk combination of unblinded designs plus subjective endpoints, which tends to inflate estimated benefits (Wood et al., 2008; Savović et al., 2012; Moustgaard et al., 2020). The goal isn’t cynicism. It’s precision. If you’ve been told your symptoms are “just stress” while the evidence feels all over the place, this is a way to separate real signal from study design noise. When the comparator is explicit, evidence becomes easier to apply to real decisions: compared to what you’re currently doing, in the setting you’re actually in, with the outcomes you care about.
Effects Are Differences: The Comparator Behind Every “It Works” Claim
“It worked”… compared to what, exactly?
Trial reporting standards treat the primary result as the difference between an intervention group and a comparator group, with an effect size and precision (CONSORT 2010 Item 17a) and enough detail to understand what each group received (CONSORT Item 5). Cochrane reviews and ICH E9(R1) formalize the same idea: a “treatment effect” only exists once the treatment conditions being compared are specified (Cochrane Handbook; ICH E9(R1) estimands). If a write-up says symptoms improved and implies “it works,” it may be bundling natural history, regression to the mean, and context effects into one story. A more accurate headline is often: “Improved more than ___.”
Comparator choice changes the question the result can answer
Different comparators answer different decision questions, even when the intervention is identical (ICH E9(R1)).
- Placebo/sham: “Is there benefit beyond expectancy and the treatment ritual?”
- Usual care: “Does adding this to what people typically receive help in practice?”
- Active comparator: “Is this better (or at least not worse) than an existing option?”
If you’re trying to decide what to do, placebo-controlled evidence can show a signal (promising), but active-comparator evidence usually answers the real “which option is better?” question (higher decision-relevance).
None is automatically “more honest.” They are different questions, and CONSORT’s interpretation guidance emphasizes reading results as between-group effects, not within-group improvement (Moher et al., CONSORT Explanation & Elaboration, 2010).
When outcomes are subjective, the control condition becomes part of the “active ingredient”
For patient-reported outcomes like pain, fatigue, hot flashes, and sexual function, attention, reassurance, and expectancy built into the comparator can change the apparent benefit. Placebo vs no-treatment differences are often larger for subjective outcomes than for objective outcomes (Hróbjartsson & Gøtzsche, 2001; later meta-analyses showing the same subjective-vs-objective pattern), and supportive interaction can produce graded improvement even under sham treatment (Kaptchuk et al., 2008). Many women’s health trials in symptom-heavy subfields rely heavily on symptom scales, which are especially sensitive to expectancy and care context. That is an endpoint and context issue, not a claim that women are “more placebo-responsive.”
When “Replication Failed” Is Really a Different Comparison
A common pattern: Trial A uses a waitlist or no-treatment control and reports a large benefit; Trial B uses an attention control (matched contact) or an active comparator and finds a smaller benefit. That can be two valid estimates of different contrasts, not a contradiction. Meta-research in behavioral and psychotherapy-style interventions shows larger effects against waitlist controls (Cunningham et al., 2013). For complex nonpharmacologic interventions, CONSORT-NPT flags unmatched attention and contact as a major interpretability problem because extra time with a clinician can function like an ingredient (Boutron et al., 2008).
Comparator choice also tends to travel with bias risk: unblinded designs plus subjective outcomes can inflate estimated benefits via expectancy effects, differential reporting, and differential co-interventions (Wood et al., 2008; Savović et al., 2012; Moustgaard et al., 2020). Before calling results incompatible, check:
1) Did both studies use the same kind of control, with similar time and attention?
2) Were outcomes similarly vulnerable (subjective, unblinded) to bias-related exaggeration?
3) Was “usual care” described well enough to know baseline support (Cochrane Handbook; CONSORT pragmatic extension, Zwarenstein et al., 2008)?
A Practical Map of Control Groups: Why “Nothing” Isn’t One Thing
Three very different “nothings”
- Placebo/sham: can isolate benefit beyond expectancy if the sham is credible and matched. Ethics constrain when placebo is acceptable (Declaration of Helsinki, Article 33).
- Attention control: equalizes “soft ingredients” (time, check-ins, reassurance, accountability) that can move symptom scores. In IBS, outcomes improved as supportive interaction increased, even under sham (Kaptchuk et al., 2008). That’s exactly why an attention control often shrinks effect sizes: it removes “extra care” as a confounder and leaves you closer to the intervention’s specific contribution.
- Waitlist/no-treatment: simple, sometimes necessary, and tends to inflate differences in behavioral-style interventions (Cunningham et al., 2013) via expectation asymmetry, disappointment/nocebo, and regression to the mean, especially in unblinded, subjective-outcome trials (Wood et al., 2008; Savović et al., 2012; Moustgaard et al., 2020).
Real-world comparators: more decision-relevant, more context-dependent
“Usual care” varies by country, clinic, insurance rules, and calendar year. Without detail, it’s unclear whether the intervention beat nothing or a robust care package. Reporting guidance exists because this is common: CONSORT Item 5, the pragmatic extension, and TIDieR push specificity (visits, medications allowed, referrals, self-management advice) rather than labels (CONSORT 2010; Zwarenstein et al., 2008; Hoffmann et al., 2014).
Active comparator trials ask the question most people care about: “better than what’s already used?” They can mislead if the comparator is underdosed, inconsistently delivered, or not actually the current standard; ICH E10 emphasizes implementing the control with integrity (ICH E10). Per ICH E9(R1), the estimand depends on the conditions actually compared, not the abstract’s label (ICH E9(R1)). A fast reality check is date plus location plus what counted as standard care—for example, whether “usual care” meant a single handout and reassurance, or scheduled follow-ups with medication adjustment pathways and referral options.
To interpret an effect size, the comparator needs the same replicable detail as the intervention (CONSORT 2010 Item 5; TIDieR; CONSORT-NPT). Extract for both arms: number/length of visits; setting/provider; scripts/talking points; materials/handouts; devices/procedures (rituals); monitoring/check-ins; allowed/prohibited co-interventions; contamination control; fidelity/adherence (Boutron et al., 2008; Hoffmann et al., 2014).
If a paper says “usual care,” look for whether it included follow-up calls, medication adjustments, or referrals—those details belong in your “comparator” notes, not in the margin as “misc.”
Pair endpoint subjectivity with blinding, every time
Unblinded trials with subjective primary outcomes tend to overestimate benefits (Wood et al., 2008; Savović et al., 2012). Check whether participants, clinicians, and outcome assessors were blinded; if not, treat large symptom-score gains as higher inflation-risk unless supported by more objective endpoints.
A Compact Comparator Bias Potential Score
This isn’t a validated tool—just a quick heuristic to flag when you should slow down and read the methods more carefully. A practical shortcut: add +2 for unblinded plus subjective primary outcome, +2 for waitlist/no-treatment control, then +1 for vague usual care/contact imbalance, and +1 for imbalanced co-interventions/contamination (CONSORT-NPT; TIDieR). A high score doesn’t mean improvement is fake. It means the estimate may be less portable to settings with different expectations, attention, or baseline care.
Comparator-aware language to reuse: “This benefit is relative to [comparator] in [setting]; if baseline care includes [usual care elements], the expected difference may change.” Before asking “Does it work?” ask: “Compared to what, and under what conditions?”
Clinical effects don’t float in isolation. They are differences between groups, and the comparator defines what a study can honestly claim. When a paper says “symptoms improved,” the more useful translation is “improved more than what?”: placebo/sham, usual care, waitlist, or an active option (CONSORT 2010 Items 5 and 17a; ICH E9(R1)). That one step helps explain why evidence can look inconsistent: different controls often mean different questions, not true contradictions. It also protects against over-reading results when trials are unblinded and outcomes are subjective, an inflation-prone combination documented across meta-research (Wood et al., 2008; Savović et al., 2012).
Next time you see “it works,” do three quick things:
- Find the comparator and rewrite the claim as “improved more than ___.”
- Pair endpoint type with blinding (subjective + unblinded = higher inflation-risk).
- Open the methods and list what “usual care” or “control” actually included (visits, follow-ups, allowed co-interventions).
Comparator literacy isn’t cynicism. It’s decision support. Extract what each arm actually received (CONSORT; TIDieR), then interpret results relative to real baseline care and the outcomes that matter.
What’s one recent “it works” claim you’ve seen—and what was it compared to?
