Five tables, no calls between them
Every detection rule used to live split across five places: the regex and its keywords in one Python table, the severity next to it, the impact class in a second file, the vendor in a third, the revocation link in a fourth. Nothing imported anything else. Adding a rule meant editing all five, in the right order, and hoping the fifth one didn’t get forgotten — which it periodically did.
As of August 31st, each rule is one .yaml file in app/analyzers/rules/ — 381 of them today. A loader, rules_loader.py, reads every file and derives the tables the product actually needs. The five hardcoded tables are gone; what’s left are views.
One directory of self-contained files, one loader, four derived views — instead of five tables that had to be remembered.
The anatomy of a rule
Nothing else completes a rule file, and nothing else tests it — that was the whole point of the migration: per-rule coverage that simply didn’t exist before. Every file carries its name, its detection.regex and keywords, a severity and impact_class, optional vendor / revocation / validator, and two lists of examples: should_match, should_not_match.
A rule is its pattern, its prefilter, both example sets, and — where one exists — the trade-off it accepts, in writing.
The test that guards all of this, test_rules_yaml.py, deliberately does not go through the loader: it re-reads every file with yaml.safe_load, independently. A bug in the loader can’t cover for itself; only a reading that ignores it can contradict it. Four families of checks run in sequence — is the file well-formed, does its regex do what it claims, does the loader see the same thing it does, and do the four derived views stay a plain reflection of the directory, with no data that quietly grew a life of its own somewhere else.
A rule that lied to us today
While writing that test, one rule flagged this as a Sourcegraph access token:
From f39b537d4a63654ff5aebedc6e9bda1bb9167e7f Mon Sep 17 00:00:00 2001
Subject: [PATCH] docs: add README and clean up code
- README.md: … code search via Sourcegraph …
A git commit SHA-1 — 40 hex characters, character for character identical to a pre-2022 Sourcegraph token. The cause is one line: the keyword prefilter in scan() checks the whole scanned content, not the neighborhood of the actual match. The pattern accepted two shapes — sgp_…, the current prefix, and 40 bare hex characters, the format Sourcegraph retired in 2022. It only took the word “sourcegraph” appearing anywhere in the file for every SHA-1 in that same content to clear the prefilter.
The prefilter sees the whole file; the regex sees one line. The gap between the two was the hole.
The fix drops the bare-40-hex alternative. The pattern now only recognizes sgp_-prefixed tokens — the only format Sourcegraph has issued since 2022 — and the sourcegraph keyword goes with it, since nothing detectable can appear without sgp_ any more. What we give up — a legacy token, 40 bare hex characters, in some old repository — is written into the rule’s rationale field, in plain text, instead of being rediscovered the next time someone wonders why a commit got flagged.
What we’re taking away
On the migration itself. 381 rules, one file each, no hardcoded table left standing. The gitleaks cross-check that came with it found 177 rules matching by exact ID, ~17 more by rename, and 8 that looked suspicious enough to redo from scratch — plus 14 rules gitleaks had that we didn’t. About 187 of our own rules still have no gitleaks equivalent and haven’t been checked against their vendor’s documentation yet.
On the failure mode. The keyword prefilter trades precision for speed on purpose — a file with none of a rule’s keywords never runs that rule’s regex at all. The price is scope: the prefilter can’t tell “this word is near the match” from “this word is somewhere in the file.” Sourcegraph is the rule that surfaced it; nothing rules out another one doing the same.
Still open. Should the prefilter check a window around each candidate instead of the whole content — at whatever cost that adds per scan? Should a trade-off like this one live in a structured field instead of free-text rationale, so it can be listed by tooling instead of found by reading 381 files? And how do we get through the remaining ~187 unverified rules without slowing down how fast new ones get added?
The directory lives at app/analyzers/rules/, one file per rule. The discipline comes down to one sentence: a rule that’s wrong gets fixed in its own file, never anywhere else.