My honesty linter flagged my own honesty disclaimer
I run five small software comparison sites. I have not installed most of the tools I write about. That gap is the whole problem: the easy way to sound authoritative is to imply I did things I didn't do, and that is exactly the kind of sentence that gets a site rejected for ads, or quietly distrusted by readers.
So instead of promising myself I'd be careful, I wrote a checker.
The rule
site_verify.py scans every page before deploy and fails the build on phrases that assert first-hand experience. Not "we think" — "we tested," "hands-on," "in our testing," "we measured." If the site can't back the claim up, the phrase doesn't ship.
It isn't clever. It's a list of patterns and a hard exit code. After enough rounds it's at version 26, and this week it finally went green across all five sites — 59 of 59 checks.
The same discipline applies to the bylines. An earlier pass rewrote 302 author bios and 302 role lines that had been describing a persona with hands-on experience of the products. That number is what convinced me the manual approach had already failed: I cannot hand-audit 604 fields, and I definitely cannot hand-audit them every time I publish. A script can.
The bug
The failure wasn't a page. It was my own disclaimer.
One of the sites carries a line explaining the limits of the content: "We do not run hands-on lab tests." The checker's fallback rule was a bare regex on hands-on. It matched. My honesty checker was flagging my honesty disclaimer as a dishonest claim.
The same round produced two more: "We ran no benchmarks." and a "we measured no ..." line, on two different sites. Same cause.
Why the fix isn't "add an exception"
The tempting fix is to allowlist those exact sentences. That works today and rots next month — I write "we ran no benchmarks on the enterprise tier" and it sails through, or I rephrase slightly and it fails again for no reason.
What I actually changed: the matcher now looks at negation context. A phrase only passes if it's explicitly negated — "do not," "no," "never" — within the same clause. The allowlist strips negated forms; it never opens the door to affirmative ones. Claiming I tested something is still a hard fail. Saying I didn't is now fine.
That distinction is the entire point. The goal was never to remove the word "testing." It was to remove the impression of experience I don't have.
The sentence that got me anyway
A few hours later, reading a page I had already shipped, I hit this subheading:
How we built and checked these prompts
"Checked." It's on no pattern list. It passes every rule I wrote. And it reads exactly like someone opened the tools and verified the output.
I changed it to "How this prompt set is put together." Same information, no borrowed credibility.
That one is the honest lesson, and it's the part I can't automate: a linter catches patterns, not impressions. Every rule I have encodes a phrasing I already thought of. The one that slips through is always the verb I didn't consider — "checked," "verified," "tried," "ran it ourselves."
What I do now
Two passes, every time:
- Run the checker. It's cheap, it's deterministic, and it catches the obvious 95%.
- Read the page once, out loud, asking one question only: what would a reader believe I did, based on this sentence?
Step 2 caught the subheading. Step 1 never would have.
If you're running content sites where you don't have first-hand experience of the product, this is worth an hour of your time. The checker isn't the valuable part — anyone can grep for "we tested." The valuable part is the habit of asking what your sentence implies, independent of what it literally says.
If you want a second pair of eyes on your site's claims or your pre-deploy checks: /