Research
Reports from our own testing. None of it is an outside audit. We lead with the real-claims results because they are the hardest and the most honest.
· 25 min read
How the rules, the source library and the on-device model were built and tested, and what they find. On a sealed half of real claims quoted by regulators, the shipped stack flags 98.2% of the claims two independent reads agree are real advertising. Scored on every record it reaches 84.4%, short of our 90% target, and the report shows why.
Download the reports as PDF
From the latest report, on rule-set 0.8.0. Scored on a sealed half of real claims quoted in FDA and FTC actions, which no model trained on.
Rules alone 55.4%; On-device model (22 MB) 96.3%; Model + rules 98.2%; Teacher (DeBERTa) + rules 98.6%; Three layers combined 99.1%
Real FDA and FTC warning-letter claims (150) 93%; Held-out real violations flagged (487) 97%; False positives, held-out real acceptable (5,521) 1.7%
Scored on every record in the sealed half, including regulator text that is not advertising, the shipped stack reaches 84.4 percent, short of our 90 percent target. The report explains the gap in numbers. Results by layer.
These were measured before the latest report and were not repeated. Where a figure differs from the report above, use the report.
| Set | Size | What it is |
|---|---|---|
| Real warning-letter claims | 150 verified | Real FDA and FTC claims, checked word for word against the cited document |
| AI-written violations | 2,500 over 7 sets | Written blind by AI agents that never saw the rules |
| AI-written compliant copy | 800 over 6 sets | Careful compliant copy, including negations and tricky cases |
Source: Research report, rule-set 0.6.0, section 4.2
| Measured on a fresh set | Violations flagged | Violations blocked | Compliant copy wrongly blocked |
|---|---|---|---|
| After the broad-claims round | 80% | 52% | 3% |
| After round 1 | 85% | 71% | 6% |
| After round 2 | 81% | 68% | 10% |
| After round 3 | 80% | 67% | 10% |
Source: Research report, rule-set 0.6.0, section 4.3
Blocking improved once, then stalled while false alarms climbed. On the sets the rules were tuned on they block nearly everything, which means nothing. The fresh-set number is the one to read.
| Set | Violations with a high-confidence issue | Compliant copy with a high-confidence issue |
|---|---|---|
| omega (never tuned on) | 97% (291 of 300) | 0% (0 of 100) |
| psi (sealed) | 94% (282 of 300) | 0% (0 of 100) |
Source: Research report, rule-set 0.6.0, section 5
The first version of the prompt flagged 43 to 49 percent of compliant copy. Version 2 fixed that, and letting the AI judge each blocking rule hit cut blocking false positives on one compliant set from 10 percent to 0. It is model output and cannot be re-run to the same answer, so the record labels it reviewer-attested.
| Layer | Compliant copy wrongly flagged | On which text |
|---|---|---|
| Rules, blocking | 0 to 10% | AI-written compliant copy, 6 sets |
| AI second read, high confidence | 0% | 200 AI-written compliant snippets |
| Trained classifier, version 1 | 4 to 7% | 1,100 AI-written compliant snippets |
| Trained classifier, version 2 | 4 to 11% | 600 AI-written compliant snippets, 5 sets |
| Trained classifier, version 2 | 2% | 5,521 held-out real acceptable texts, mostly government text |
Source: Research report, rule-set 0.6.0, section 6.3 and 7, and Classifier model card, 2026-10-12, how it was tested
| Test set | Size | Classifier flags | Rules block |
|---|---|---|---|
| Real FDA and FTC warning-letter claims | 150 | 94% | 56% |
| Same, only claims sharing no five-word phrase with a training violation | 118 | 93% | not measured |
| Held-out real regulator text | 487 | 98% | not measured |
Source: Classifier model card, 2026-10-12, how it was tested (claims-v2, research prototype)
Next step
Paste an ad or a landing page and see each finding with its rule and source.