Methodology and sources
What we verified, and what we didn't.
A rule is only as good as the source under it. Here is how rules get written, how they are checked, and where the checking stops.
How a rule gets written
Rules are drafted from stored source text: regulations, guidance, statutes and enforcement documents. Every rule has a plain reason, a source citation, and a safer wording. It also needs at least three examples that must fire and three that must not. Those examples run as unit tests, and a rule without them will not load.
The rule-set is versioned. Every report records the version it was checked against, and the changelog lists what changed.
What verified means here
423 of 565 rules are marked verified. For those, every citation into a regulation we hold has to resolve to a real section and paragraph when the rule-set builds. If a reference points at text that isn't there, the build fails.
What is unverified
- Counsel review. Pending. No qualified reviewer has signed off the rule-set or the disclaimer. Every rule is unreviewed.
- Platform policy rules. 6 rules about ad platform policy were written from memory of pages that cannot be stored. They are marked unverified, and you should not rely on them for platform approval.
- Hand-saved sources. 133 sources were read in a normal browser and saved by hand because the site refuses scripts. We have not re-checked them against the live page, so check them yourself.
- Truth of claims. The checker compares wording with rules. It cannot tell you whether a claim is supported by evidence.
The source library
334 sources are registered: 324 stored as text, 10 link only, 0 blocked. Regulations come from the latest codified edition we could fetch, and edition dates lag the live text by months.
- The fetcher honors robots.txt and waits between requests. Where a site refuses scripted access, we record it as blocked instead of working around it. We don't fake browser headers.
- Link-only sources are registered with an address and no stored text, so no rule can be checked against them.
- Guidance documents are staff views and are not binding. Statutes and regulations are the codified text.
- A quarterly job refetches the sources and lists which rules cite anything that changed.
Link only: no text stored
- Amazon Ads: Creative Acceptance Policy (CAP 7.11 pharmaceuticals, health and beauty)
- ICH E6 Good Clinical Practice
- Example institutional IRB policy: Creighton, IRB Review of Recruitment Materials
- Example institutional IRB policy: UAMS, Subject Recruitment Materials (Policy 14.3)
- Example institutional IRB policy: UC Irvine, Advertisement and Recruitment
- Meta Advertising Standards
- Microsoft Advertising policies (disallowed content, healthcare)
- TikTok Advertising Policies: health and wellness
- Reddit Advertising Policy
- SACHRP recommendations and letters to the HHS Secretary
How accuracy is measured
Real claims come from FDA warning letters and FTC actions. A quote counts only after a script finds it, word for word, in the cited page. Quotes that cannot be confirmed are excluded. FTC pages and PDFs cannot be read by script, so most FTC quotes are excluded.
The method holds out a fixed third of the verified claims, tunes rules on the other two thirds only, and reads the held-out number as the honest one. The only held-out measurement so far covered 57 of the 150 claims. Rules blocked about 49 percent of them, at rule-set 0.4.1, and it has not been re-run. The 56 percent on the research page covers all 150. The research page has the results, including the ones that look bad.
Next step
Check a rule against its source yourself.
Every finding in the checker shows its source. Pick one that matters to you and read the paragraph it cites. If it doesn't support the rule, tell us.