Tripling our secret scanner's recall: what the benchmark told us, and what we refused to fix

Five secrets out of six went straight through

LeakWatch watches for secrets exposed in public: gists, GitLab and Codeberg pushes, git history. The pipeline is two stages — a regex engine produces candidates, an LLM classifier throws out the false positives.

We had never measured the first stage properly. So we pointed it at CredData, Samsung’s benchmark: 11,457 real files annotated line by line, each secret marked T (true secret), F (known false positive) or X (ambiguous), with the exact character bounds of the value inside its line.

The first run was humbling.

TP=2561  FP(known)=512  FP(unlabelled)=1399  FN=12728
Precision=0.5727  Recall=0.1675  F1=0.2592

16.75% recall. Five secrets out of six went undetected. A false positive costs one classifier call; a secret that is never detected is gone for good.

By the end of the afternoon that number was 0.518, precision had gone up rather than down, and the scan ran faster than when we started. Here is the whole path, including the three places our intuition was flatly wrong.

Three of the five stages changed. Only one of them was on anyone’s list at the start.

First problem: you can’t fix a miss you can’t attribute

Our benchmark broke true and false positives down by rule — easy, every detection carries its own label. But misses were aggregated globally, and the code said so honestly:

for rid, r in truth_row_ids.items():
    if rid not in matched_true_rows:
        overall.fn += 1
        # we don't know which secret_type should have fired
        # (CredData's taxonomy doesn't match ours) -> global only

That is the crux of it. A false negative is by definition a non-detection: no rule fired, so there is no rule to blame. Which rule misses most? has no direct answer in the data.

The workaround: CredData annotates every secret with its own taxonomy — Key, Password, Token, AWS Client ID, NKEY Seed. It doesn’t map onto ours, which is exactly why we had ignored it, but it is enough to group misses by kind of secret and then walk back manually to the rule that should have covered each group.

We ported that into the benchmark. Without it, every later iteration would have produced a single global delta — and a gain on Key cancelling a loss on Token would have been invisible.

The result splits the world cleanly in two:

Fixed-format rules already worked. The entire deficit sat on secrets with no recognisable shape: plain keyword = value assignments. The top five categories alone accounted for 84% of all misses.

That is not an accident of the dataset, it is the boundary of what a regex can do. -----BEGIN PRIVATE KEY----- is a trivial pattern at 100% precision. password = "ywriv" has no format at all — only the surrounding words tell it apart from an ordinary variable.

One rule was causing four of the five big miss blocks

Replaying missed lines one at a time gave us the diagnosis in a single pair of results:

MISS [] | "access_token" : "rKzowpViqmFcd7DtWWUoDLOdLLTsw9OaOTN7cXFylBZhy..."
OK   ['Generic High Confidence Secret'] | access_token = rKzowpViqmFcd7DtWWUoD...

Same value, same anchor. Only the JSON syntax differs. From there the cause stops being a hunch and becomes a checkable property of the regex:

r"""(?ix)
\b                                            # (a) no prefix tolerated
(?: api[_-]?key | client[_-]?secret | access[_-]?token | … )
\s*(?:=|:|:=)                                 # (b) no closing quote allowed
\s*['"]?
([a-zA-Z0-9_\-\/+=]{24,})                     # (c) threshold above real p10
"""

Three independent defects, with very different repair costs:

(a) No prefix is tolerated before the anchor. \b(?:api[_-]?key|…) requires the keyword to start the token, which rejects certificate_key, MINIO_SECRET_KEY, SharedSecret, repokey — that is to say, most real variable names. The irony: the correct idiom, [\w.-]{0,50}?(?:vendor), was already in the same file. Every rule we imported from gitleaks uses it. It just wasn’t in the generic one.

(b) The separator is looked for immediately after the keyword. In JSON, in quoted YAML, in any JS/TS object literal, the closing quote gets in the way and the match fails.

© The {24,} threshold sits above the real 10th percentile of value lengths (16 for Token, 20 for Key).

Fix 1: ten characters of regex

Tolerate the closing quote before the separator — and, while we were there, PHP/Ruby’s => arrow, which is the same class of defect and which our own written analysis had missed:

['"`]?
# Order matters: a leading `=` would eat the `=` of `=>` and then fail.
\s*(?:=>|:=|=|:)
\s*['"`]?

Result: +90 true positives for +1 noise detection. The generic rule’s precision went from 0.38 to 0.63.

Worth pausing on, because it is the first place we were wrong: our analysis had predicted “~940 misses recovered” for this fix. The real figure was 90 — an order of magnitude out.

The error wasn’t in the counting, it was in the method. We had attributed each miss to one defect, when most of them stack several. A 20-character "api_token": "…" fails on the quote, on the missing keyword and on the length threshold simultaneously; fixing only one of the three doesn’t recover it. A per-fix volume estimate only ever reads as a ceiling, never as a forecast.

Fix 2: removing a constraint that had no reason to exist

Drop the requirement that the keyword start the token. A bare secret|key|token, any prefix:

(?: secret | key | token )

Precision, recall and F1 at each stage. Fix 2 moved recall from 0.173 to 0.518.

Key went from 0.2% to 94% recall, Secret from 2% to 86%, Token from 1% to 51%. The cost was +232 noise detections for +5,270 true positives — 23 real for every 1 spurious.

Recall by CredData category, before and after. The two faded bars at the bottom are not failures — see “What we chose not to fix”.

The part that surprised us

Precision went up 11 points while we made the rule broader. That is the opposite of what widening a pattern is supposed to do.

It is mechanical once you see it. The old version only ever fired on the bare access_token = value form, which is rare in real code, so its handful of matches were disproportionately example configs and documentation. Accepting MINIO_SECRET_KEY= reaches the seam where real secrets actually live. The denominator grew; the numerator grew a great deal more.

The rule of thumb we are keeping: on this engine, widening an anchor and losing precision are not synonyms. What costs precision is widening the accepted value — the length threshold — not the context that qualifies it.

The honest cost, which the headline number hides: annotated false positives (F/X) tripled, from 512 to 1,959. Global precision doesn’t show it because 5,270 new true positives swamp the ratio. But that is real work added to the classifier downstream.

The 29% slowdown that turned out to be two bugs

Fix 2 pushed a full scan from 8:40 to 11:13. We were ready to write that up as the price of recall. It wasn’t. Two independent defects, both fixed without changing a single detection.

The prefix we copied from gitleaks did nothing

We had lifted [\w.-]{0,50}? from the gitleaks rules sitting in the same file. But gitleaks runs on RE2, which returns the leftmost match without backtracking — there, the prefix is genuinely necessary. Python’s finditer already tries every starting position, so the KEY in MINIO_SECRET_KEY= is reached unaided, one position later.

Measured across 97 MB of CredData:

with prefix     15.33s  matches=486
without prefix   2.12s  matches=486
present only with prefix : 0
present only without     : 0

7.2× faster, zero difference in either direction. The transferable lesson: don’t import an idiom from another tool without checking that the constraint motivating it applies to you.

The rule had keywords=[]

Our engine prefilters each rule by substring before running its regex — no point running an Adafruit pattern on content that never says “adafruit”. The generic rule carried an empty list, which is falsy, so the filter never applied and the regex ran against every piece of content we had ever scanned.

Filling in ["secret", "key", "token"] — the rule’s own anchor, so it cannot possibly discard content the rule would have matched — short-circuits 25% of CredData’s files.

Full-scan wall clock. All four detection counters are bit-identical between the middle and right bars.

The scan now runs 12 seconds faster than before any of this started, while finding three times as many secrets.

Two optimisations we measured and then threw away

With both bugs fixed, the per-rule profile is flat: the generic rule is 5.7% of total time, no rule exceeds 6.3%, and the keyword prefilter alone accounts for 46%. We tried two ways to attack that, and rejected both:

• One compiled alternation of all 374 keywords — 5.33s against 2.07s. Python’s re has no multi-pattern automaton; it tests alternatives one at a time. It is also incorrect: findall consumes text, so overlapping keywords get missed and rules would be silently disabled.
• An inverted keyword → rules index — 1.73s against 2.07s, identical semantics. Only 16%, because there are just 455 keyword entries for 374 distinct keywords. Almost nothing to deduplicate.

And neither would matter in production anyway. Throughput is linear, with no fixed per-content overhead:

content   1 KB   0.32 ms/item   0.32 s/MB
content   5 KB   1.51 ms/item   0.30 s/MB
content  20 KB   6.57 ms/item   0.33 s/MB

CredData averages 28 KB per file. A gist patch or a commit diff is 1 to 5 KB. At 0.32 ms per kilobyte, our firehose ingestion is bound by the network and by the classifier, not by regex. An Aho-Corasick dependency would be complexity bought for a gain nobody would ever measure.

What we chose not to fix

Two whole categories of misses are staying broken, deliberately.

Password — 2,491 misses. A bare password is not exploitable without its target. The cases where it is — postgres://user:pass@host, curl -u, Authorization: Basic — already have dedicated rules, and those are precisely the ones with good recall (0.94 on Basic Auth). What’s left is private static String password = "ywriv" in test files: median length 8, pure alphabetic charset, no discriminator whatsoever. On public content, such a rule would mostly surface fixtures — at one classifier call each.

UUID — 2,208 misses. 64% have no keyword before the value at all. Catching them means alerting on every UUID in existence. CredData marks them T because it adjudicates on format, not on exploitability; a UUID with no context is not something an attacker can use.

Setting those aside caps our CredData recall at roughly 0.84, permanently. That is fine. A benchmark is an instrument, not a target — part of what it rewards simply isn’t what the product should do.

The noise was never where we were looking

The whole exercise had been about recall, on the comfortable assumption that false positives were somebody else’s problem downstream. The per-rule breakdown said otherwise.

True positives against false positives, for the five worst-behaved rules.

AWS ARN alone produced 1,230 of the baseline’s 1,399 unlabelled false positives, at 0.076 precision. But an ARN is a resource identifier, not a credential — there is nothing to revoke. CredData’s occurrences are AWS documentation samples with placeholder account numbers like 012345678910.

Together those five rules are around 1,480 detections for 101 true positives between them. Removing them costs zero recall and lightens the classifier by the same amount. It was the best ratio available all day — and it stayed invisible for as long as we only looked at recall.

A detour: the context-compression idea that killed itself

With the regex stage settled, attention moved to the LLM. One approach was circulating — extract only the string literals from the neighbourhood of a candidate, mask high-entropy values with a <RND> token, split camelCase, and send only that to the model instead of raw code.

The illustration is genuinely persuasive:

region            = "us-east-1"
access_key_id     = "AKIAIOSFODNN7EXAMPLE"
secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
endpoint          = "s3.amazonaws.com"

# context sent to the model:
#   s3 us-east-1 <RND> s3.amazonaws.com

Four words instead of eighty, and the model still knows it is looking at AWS.

Except that AKIAIOSFODNN7EXAMPLE / wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY is the placeholder key pair from AWS’s own documentation, sitting in tens of thousands of repositories. The correct verdict is not “AWS key” — it is false positive, documented example.

And the entropy filter replaces "AKIAIOSFODNN7EXAMPLE" with <RND>. It deletes the word EXAMPLE, which is the evidence. What survives says “AWS, region, S3 endpoint” — exactly what you need in order to conclude “real key” and be wrong.

The threshold is wedged, too. s3.amazonaws.com sits at 3.20 Shannon entropy and AKIAIOSFODNN7EXAMPLE at 3.68: half a point of margin between keeping the useful context and erasing the proof.

So we measured it properly, on 4,000 annotated CredData lines, same ±20-line neighbourhood in both cases.

Share of true-secret contexts still carrying each signal, raw code versus string literals only.

On top of that, 26.8% of contexts come out essentially empty — 86% in .txt, 66% in .c, 28% in .md, 27% in .yml.

The central flaw is structural: our anchor is not a string literal, it is a variable name. The entire regex engine is built on that — it is what both fixes above were about. In password = "ywriv", literal extraction returns ywriv and throws away password.

Three shorter objections:

• <RND> is a category error for an LLM. That preprocessing — closed vocabulary, entropy masking, camelCase splitting, length filters — comes from embedding classifiers (word2vec/fastText feeding a CNN or BiLSTM), where an open vocabulary blows up the embedding matrix. There, <RND> creates a feature. An LLM has none of those constraints, and the shape of neighbouring values is one of its best signals.
• Comments vanish entirely. 57–70% of contexts contain one, and // fake key is not a string literal.
• The saving isn’t real. Once the system prompt is counted, it comes to 9% per call.

The “survives obfuscation” argument is sound in its original domain — ProGuard’d APKs, minified JS bundles, where identifiers are destroyed and strings survive. We scan source code, where variable names are intact. And in the minified case the algorithm collapses anyway: a minified bundle is one enormous line, so radius-based expansion has nothing to walk.

What the literature actually says

Rather than keep arbitrating on instinct, we went and read what has been published. One paper has exactly our architecture.

Rahman et al., Secret Breach Detection in Source Code with Large Language Models (IEEE 2025): regex candidate extraction, then LLM classification, evaluated on SecretBench — 97,479 candidates, 818 GitHub repositories, 49 languages, 15,084 hand-verified secrets.

F1 by model and prompting strategy, from Rahman et al. (2025).

Four findings we could act on immediately:

They use 200 characters of raw context, and the ablation says that’s enough. Going from 200 to 300 characters moves LLaMA-3.1-8B from 0.9852 to 0.9895 F1. In the authors’ words: “200 characters already offering strong performance in most use cases.” We were at 1,500 — chosen by guesswork. And nobody in this literature filters context down to string literals.

Few-shot is the cheapest available gain. GPT-4o goes from 0.8428 zero-shot to 0.9392 few-shot. Nearly ten points for a handful of examples in the prompt.

Small models in zero-shot don’t hold up. Raw and zero-shot: Gemma-7B 0.335, LLaMA-3.1-8B 0.413, CodeLLaMA-7B 0.521. That is the direct answer to “why not just run a 3B locally”. Fine-tuned with QLoRA, LLaMA-3.1-8B reaches 0.9852 — and their training ran on consumer hardware (i5-13400F, RTX 4090, under 17 GB VRAM). If you want local, that’s the route.

Fine-grained typing is unreliable without fine-tuning. On multiclass, GPT-4o scores 0.178 F1 on “Generic Secret” and 0.000 on “Username”. The binary verdict is solid; the category label is not — so don’t route on it automatically. Which is awkward for us, given our highest-volume rule is literally called Generic High Confidence Secret.

On how many few-shot examples, the paper declines to say. It defines few-shot as “multiple labeled examples covering varied secret and non-secret cases” without ever publishing a count, and files the choice under threats to validity: “the subjectivity in prompt design and few-shot examples used… the selection of examples might influence model behavior, particularly for ambiguous candidates.” Diversity and representativeness are the only guidance on record.

A trap aimed squarely at our next step

From Data Leak to Secret Misses shows that SecretBench contains 69.3% exact duplicates (8.7% near-duplicates, 22% genuinely unique), and that published performance collapses once you deduplicate: Random Forest MCC 0.89 → 0.65, LSTM 0.92 → 0.77.

So before building an LLM test set on CredData, we measured it:

Far cleaner than SecretBench — but a third of the false positives are duplicates. Token::ILLEGAL, appears 116 times; "Auth Token: fake_token\n" appears 233. An LLM benchmark on the raw F set would mostly measure the ability to reject three repeated lines.

What we changed in the classifier

Two changes, both straight out of the measurements above.

Context: from a 1,500-character total window to 150 characters either side of the value. The semantics change along with the number, deliberately.

A PEM key truncated at 300 characters would have filled a 300-character total window entirely, leaving nothing around it — which is precisely the information the model needs.

Four complete few-shot demonstrations. Our system prompt already had examples — but in prose, at the level of the value alone (AKIAIOSFODNN7EXAMPLE -> false positive). That is not few-shot: the model never saw the exact shape of the question, nor of the expected answer. The new demonstrations go in as conversation turns, in the real input/output format.

The opening pair is the whole point: near-identical entropy, opposite verdicts. That is what teaches the model that entropy does not decide — something our system prompt asserted but never demonstrated. Then a maximum-entropy value that isn’t a credential, and one clear-cut leak so the set doesn’t lean toward false positives.

Prompt composition per classifier call.

The honest result: the prompt grew by 6%. Few-shot costs more than the context reduction saves. That is the price of the ~10 F1 points the paper reports — a good trade if the goal is classification quality, the wrong one if it was token economy. The two changes are independent and separately reversible.

We also deleted the four prose examples the demonstrations now cover in full, keeping the three that cover cases few-shot doesn’t (judging credentials inside a URI, the git-diff header artefact). And we bumped the cache key version — without that, verdicts rendered under the old prompt would have masked the new behaviour for the whole TTL, and we’d have concluded few-shot changes nothing.

What we’re taking away

On the results. Precision 0.573 → 0.688, recall 0.168 → 0.518, F1 0.259 → 0.591, and a scan twelve seconds faster than before while finding three times as much. The substance of it is about thirty characters of regex and one keyword list that was empty.

On method. Every number here was measured, and three of our starting intuitions were contradicted by the measurement: the per-fix volume estimates (an order of magnitude out), the CPU cost (a bug, not a trade-off), and the belief that widening a rule costs precision (it was the reverse).

On scope. Two entire families of misses are abandoned on purpose, capping our score on this benchmark at ~0.84. A benchmark measures what it measures; it doesn’t define the product.

Still on the list. Removing the five identifier rules — around 1,480 noise detections for 101 true positives — and building an LLM-classifier benchmark on CredData’s annotated lines, with deduplication, or it will mostly measure duplicates.

If you want to see the current pipeline’s output as it happens, the live feed shows what it is finding across GitHub, GitLab and Codeberg right now, every key masked and every repository withheld. And if you’d rather know about your own history than watch someone else’s, that is what the deep scan is for.

References

• Rahman, Ahmed, Wahab, Sohan, Shahriyar — Secret Breach Detection in Source Code with Large Language Models, IEEE 2025 (arXiv:2504.18784)
• From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models (arXiv:2601.22946)
• Basak et al. — SecretBench: A Dataset of Software Secrets (arXiv:2303.06729)
• FPSecretBench — 2.36M false positives from nine tools; access on request
• Wahab et al. — Secret Leak Detection in Software Issue Reports using LLMs: RoBERTa/CodeBERT reach 92.70% F1, beating prompted GPT-4o at 80.13% (arXiv:2410.23657)
• Basak et al. — A Comparative Study of Software Secrets Reporting by Secret Detection Tools (arXiv:2307.00714)

Benchmark: CredData (Samsung), 11,457 files, commit 0ad42f4. Every figure above comes from a recorded run, except the context-window sizing and the model comparison, which come from the published literature and describe a bet we have not yet verified on our own data.