How we rate the evidence
Every practice card and myth page in the Library carries a five-level evidence meter. The level is decided by a written rule from two things about each study we cite: how it was done, and which way it points. Nobody picks a level by feel, and the same rule runs on every card.
- Strong
- Promising
- Mixed
- Thin
- Contradicted
The five levels
- Strong
- Several rigorous syntheses or trials agree, including studies with school-age children.
- Promising
- Rigorous support, with nothing rigorous against it, but not yet enough to call it strong.
- Mixed
- Rigorous studies point different ways: it works in some forms or for some students and not others.
- Thin
- Little rigorous evidence either way: weak, small or inconclusive studies.
- Contradicted
- Rigorous studies find it doesn't work, and none find that it does.
Step 1: how much each study weighs
- Synthesis (counts most): a meta-analysis, a systematic review, a national panel report, or a What Works Clearinghouse practice-guide recommendation that the panel rated Strong.
- Trial: a randomized trial or a comparison-group study in real schools, or a practice-guide recommendation rated Moderate.
- Lab or classroom experiment: a randomized experiment in a lab or a single lecture, or a small controlled experiment inside a few classes. Rigorous, but too small or too far from ordinary teaching to count as a school trial.
- Context (doesn't move the rating): a narrative review, a correlational or descriptive study, or a recommendation rated Minimal. We list these because they explain the debate, not because they settle it.
Step 2: which way each study points
- Supports: it finds a positive effect for the practice.
- Mixed: positive in some forms or for some students and not others, or effects the authors themselves call weak.
- Can't tell: too few or too small studies, or a positive average that isn't statistically significant. It counts for nothing either way.
- No effect or Contradicts: a well-run study that finds the practice doesn't help, or that the claim is false.
Direction belongs to the card, not the study: one meta-analysis can support explicit instruction and count against unguided discovery.
Step 3: the rule (the first that fits decides)
- Contradicted if no rigorous study supports it and at least one synthesis, or two other rigorous studies, find no effect or the opposite.
- Mixed if rigorous studies point both ways: some support it and some find no effect or mixed results; or two or more rigorous studies are mixed; or one is mixed and another against.
- Strong if nothing rigorous is against it or mixed, and either two syntheses support it or one synthesis and two trials do, and at least one of the supporting studies was done with school-age children.
- Promising if at least one rigorous study supports it and none is against it or mixed.
- Thin otherwise: only context, only studies that can't tell, or nothing at all.
Every card shows the count behind its rating under the meter (“why this rating”), so you can check the rule's working. The rule is code, and a test holds this page and the code to the same five steps.
What the meter doesn't tell you
- How big the effect is. A practice can be rated Strong with a small effect. We give effect sizes only when the source states them, exactly as stated, and we never round them up.
- Whether it suits your class. Read the studies' samples: a lab study with college students says less about a kindergarten room than a trial in elementary schools.
- Everything ever published. We cite the strongest and most-used sources we could check, not every study. If you know a rigorous study we've missed, especially one that disagrees, tell us.
How we check a source
Before a study goes on a card, we read its title, authors, year, journal, DOI and the finding we state on the publisher's or clearinghouse's own page, or in its publisher-deposited Crossref record or its ERIC record. A study we can't confirm is cut, not guessed. Practice-guide ratings are copied from the What Works Clearinghouse's own pages. Each study card says when it was checked, and every cited address is listed with its checked date in the Library's sources file.