# Tripling our secret scanner's recall: what the benchmark told us, and what we refused to fix — LeakWatch > One afternoon of measurement against Samsung's CredData: recall from 0.168 to 0.518, a 29% slowdown that turned out to be a bug, a promising context-compression idea killed by its own example, and what the academic literature says about regex + LLM secret detection. Source: https://leakwatch.net/blog/tripling-our-secret-scanner-recall --- [← Blog](/blog) # Tripling our secret scanner's recall: what the benchmark told us, and what we refused to fix August 15, 2026 · Cr0c0 · 29 min read ## Five secrets out of six went straight through LeakWatch watches for secrets exposed in public: gists, GitLab and Codeberg pushes, git history. The [pipeline](/blog/how-leakwatch-secret-scanning-works) is two stages — a regex engine produces candidates, an LLM classifier throws out the false positives. We had never measured the first stage properly. So we pointed it at **CredData**, Samsung’s benchmark: 11,457 real files annotated line by line, each secret marked `T` (true secret), `F` (known false positive) or `X` (ambiguous), with the exact character bounds of the value inside its line. The first run was humbling. ``` TP=2561 FP(known)=512 FP(unlabelled)=1399 FN=12728 Precision=0.5727 Recall=0.1675 F1=0.2592 ``` **16.75% recall.** Five secrets out of six went undetected. A false positive costs one classifier call; a secret that is never detected is gone for good. By the end of the afternoon that number was 0.518, precision had gone *up* rather than down, and the scan ran faster than when we started. Here is the whole path, including the three places our intuition was flatly wrong. *Three of the five stages changed. Only one of them was on anyone’s list at the start.* ## First problem: you can’t fix a miss you can’t attribute Our benchmark broke true and false positives down by rule — easy, every detection carries its own label. But misses were aggregated globally, and the code said so honestly: ```python for rid, r in truth_row_ids.items(): if rid not in matched_true_rows: overall.fn += 1 # we don't know which secret_type should have fired # (CredData's taxonomy doesn't match ours) -> global only ``` That is the crux of it. A false negative is by definition a non-detection: no rule fired, so there is no rule to blame. *Which rule misses most?* has no direct answer in the data. The workaround: CredData annotates every secret with **its own** taxonomy — `Key`, `Password`, `Token`, `AWS Client ID`, `NKEY Seed`. It doesn’t map onto ours, which is exactly why we had ignored it, but it is enough to group misses by *kind* of secret and then walk back manually to the rule that should have covered each group. We ported that into the benchmark. Without it, every later iteration would have produced a single global delta — and a gain on `Key` cancelling a loss on `Token` would have been invisible. The result splits the world cleanly in two: Category TP Misses Recall Key 7 3,812 0.002 Password 5 2,491 0.002 UUID 107 2,208 0.046 Secret 28 1,234 0.022 Token 8 573 0.014 … PEM Private Key 1,105 51 **0.96** Auth:Basic Authorization 562 39 **0.94** **Fixed-format rules already worked.** The entire deficit sat on secrets with no recognisable shape: plain `keyword = value` assignments. The top five categories alone accounted for 84% of all misses. That is not an accident of the dataset, it is the boundary of what a regex can do. `-----BEGIN PRIVATE KEY-----` is a trivial pattern at 100% precision. `password = "ywriv"` has no format at all — only the surrounding words tell it apart from an ordinary variable. ## One rule was causing four of the five big miss blocks Replaying missed lines one at a time gave us the diagnosis in a single pair of results: ``` MISS [] | "access_token" : "rKzowpViqmFcd7DtWWUoDLOdLLTsw9OaOTN7cXFylBZhy..." OK ['Generic High Confidence Secret'] | access_token = rKzowpViqmFcd7DtWWUoD... ``` Same value, same anchor. Only the JSON syntax differs. From there the cause stops being a hunch and becomes a checkable property of the regex: ```python r"""(?ix) \b # (a) no prefix tolerated (?: api[_-]?key | client[_-]?secret | access[_-]?token | … ) \s*(?:=|:|:=) # (b) no closing quote allowed \s*['"]? ([a-zA-Z0-9_\-\/+=]{24,}) # (c) threshold above real p10 """ ``` Three independent defects, with very different repair costs: **(a) No prefix is tolerated before the anchor.** `\b(?:api[_-]?key|…)` requires the keyword to *start* the token, which rejects `certificate_key`, `MINIO_SECRET_KEY`, `SharedSecret`, `repokey` — that is to say, most real variable names. The irony: the correct idiom, `[\w.-]{0,50}?(?:vendor)`, was already in the same file. Every rule we imported from gitleaks uses it. It just wasn’t in the generic one. **(b) The separator is looked for immediately after the keyword.** In JSON, in quoted YAML, in any JS/TS object literal, the closing quote gets in the way and the match fails. **© The `{24,}` threshold** sits above the real 10th percentile of value lengths (16 for `Token`, 20 for `Key`). ## Fix 1: ten characters of regex Tolerate the closing quote before the separator — and, while we were there, PHP/Ruby’s `=>` arrow, which is the same class of defect and which our own written analysis had missed: ```python ['"`]? # Order matters: a leading `=` would eat the `=` of `=>` and then fail. \s*(?:=>|:=|=|:) \s*['"`]? ``` **Result: +90 true positives for +1 noise detection.** The generic rule’s precision went from 0.38 to 0.63. Worth pausing on, because it is the first place we were wrong: our analysis had predicted *“~940 misses recovered”* for this fix. The real figure was 90 — an order of magnitude out. The error wasn’t in the counting, it was in the method. We had attributed each miss to *one* defect, when most of them stack several. A 20-character `"api_token": "…"` fails on the quote, on the missing keyword **and** on the length threshold simultaneously; fixing only one of the three doesn’t recover it. **A per-fix volume estimate only ever reads as a ceiling, never as a forecast.** ## Fix 2: removing a constraint that had no reason to exist Drop the requirement that the keyword start the token. A bare `secret|key|token`, any prefix: ```python (?: secret | key | token ) ``` PrecisionRecallF1 *Precision, recall and F1 at each stage. Fix 2 moved recall from 0.173 to 0.518.* `Key` went from 0.2% to 94% recall, `Secret` from 2% to 86%, `Token` from 1% to 51%. The cost was **+232 noise detections for +5,270 true positives** — 23 real for every 1 spurious. BeforeAfter *Recall by CredData category, before and after. The two faded bars at the bottom are not failures — see “What we chose not to fix”.* ### The part that surprised us **Precision went up 11 points while we made the rule broader.** That is the opposite of what widening a pattern is supposed to do. It is mechanical once you see it. The old version only ever fired on the bare `access_token = value` form, which is rare in real code, so its handful of matches were disproportionately example configs and documentation. Accepting `MINIO_SECRET_KEY=` reaches the seam where real secrets actually live. The denominator grew; the numerator grew a great deal more. The rule of thumb we are keeping: **on this engine, widening an anchor and losing precision are not synonyms.** What costs precision is widening the accepted *value* — the length threshold — not the context that qualifies it. The honest cost, which the headline number hides: annotated false positives (`F`/`X`) tripled, from 512 to 1,959. Global precision doesn’t show it because 5,270 new true positives swamp the ratio. But that is real work added to the classifier downstream. ## The 29% slowdown that turned out to be two bugs Fix 2 pushed a full scan from 8:40 to 11:13. We were ready to write that up as the price of recall. It wasn’t. Two independent defects, both fixed **without changing a single detection**. ### The prefix we copied from gitleaks did nothing We had lifted `[\w.-]{0,50}?` from the gitleaks rules sitting in the same file. But gitleaks runs on RE2, which returns the leftmost match **without backtracking** — there, the prefix is genuinely necessary. Python’s `finditer` already tries every starting position, so the `KEY` in `MINIO_SECRET_KEY=` is reached unaided, one position later. Measured across 97 MB of CredData: ``` with prefix 15.33s matches=486 without prefix 2.12s matches=486 present only with prefix : 0 present only without : 0 ``` **7.2× faster, zero difference in either direction.** The transferable lesson: don’t import an idiom from another tool without checking that the constraint motivating it applies to you. ### The rule had `keywords=[]` Our engine prefilters each rule by substring before running its regex — no point running an Adafruit pattern on content that never says “adafruit”. The generic rule carried an empty list, which is falsy, so the filter **never applied** and the regex ran against every piece of content we had ever scanned. Filling in `["secret", "key", "token"]` — the rule’s own anchor, so it cannot possibly discard content the rule would have matched — short-circuits 25% of CredData’s files. *Full-scan wall clock. All four detection counters are bit-identical between the middle and right bars.* The scan now runs **12 seconds faster than before any of this started**, while finding three times as many secrets. ### Two optimisations we measured and then threw away With both bugs fixed, the per-rule profile is **flat**: the generic rule is 5.7% of total time, no rule exceeds 6.3%, and the keyword prefilter alone accounts for 46%. We tried two ways to attack that, and rejected both: - **One compiled alternation of all 374 keywords** — 5.33s against 2.07s. Python’s `re` has no multi-pattern automaton; it tests alternatives one at a time. It is also *incorrect*: `findall` consumes text, so overlapping keywords get missed and rules would be silently disabled. - **An inverted keyword → rules index** — 1.73s against 2.07s, identical semantics. Only 16%, because there are just 455 keyword entries for 374 distinct keywords. Almost nothing to deduplicate. And neither would matter in production anyway. Throughput is linear, with no fixed per-content overhead: ``` content 1 KB 0.32 ms/item 0.32 s/MB content 5 KB 1.51 ms/item 0.30 s/MB content 20 KB 6.57 ms/item 0.33 s/MB ``` CredData averages 28 KB per file. A gist patch or a commit diff is 1 to 5 KB. At 0.32 ms per kilobyte, our [firehose ingestion](/blog/consuming-the-firehose-at-scale) is bound by the network and by the classifier, not by regex. An Aho-Corasick dependency would be complexity bought for a gain nobody would ever measure. ## What we chose not to fix Two whole categories of misses are staying broken, deliberately. **`Password` — 2,491 misses.** A bare password is not exploitable without its target. The cases where it *is* — `postgres://user:pass@host`, `curl -u`, `Authorization: Basic` — already have dedicated rules, and those are precisely the ones with good recall (0.94 on Basic Auth). What’s left is `private static String password = "ywriv"` in test files: median length **8**, pure alphabetic charset, no discriminator whatsoever. On public content, such a rule would mostly surface fixtures — at one classifier call each. **`UUID` — 2,208 misses.** 64% have no keyword before the value at all. Catching them means alerting on every UUID in existence. CredData marks them `T` because it adjudicates on format, not on exploitability; a UUID with no context is not something an attacker can use. Setting those aside caps our CredData recall at roughly **0.84**, permanently. That is fine. A benchmark is an instrument, not a target — part of what it rewards simply isn’t what the product should do. ## The noise was never where we were looking The whole exercise had been about recall, on the comfortable assumption that false positives were somebody else’s problem downstream. The per-rule breakdown said otherwise. True positivesFalse positives *True positives against false positives, for the five worst-behaved rules.* **`AWS ARN` alone produced 1,230 of the baseline’s 1,399 unlabelled false positives**, at 0.076 precision. But an ARN is a resource identifier, not a credential — there is nothing to revoke. CredData’s occurrences are AWS documentation samples with placeholder account numbers like `012345678910`. Together those five rules are around 1,480 detections for 101 true positives between them. Removing them costs **zero** recall and lightens the classifier by the same amount. It was the best ratio available all day — and it stayed invisible for as long as we only looked at recall. ## A detour: the context-compression idea that killed itself With the regex stage settled, attention moved to the LLM. One approach was circulating — extract only the **string literals** from the neighbourhood of a candidate, mask high-entropy values with a `` token, split camelCase, and send only that to the model instead of raw code. The illustration is genuinely persuasive: ```python region = "us-east-1" access_key_id = "AKIAIOSFODNN7EXAMPLE" secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" endpoint = "s3.amazonaws.com" # context sent to the model: # s3 us-east-1 s3.amazonaws.com ``` Four words instead of eighty, and the model still knows it is looking at AWS. Except that `AKIAIOSFODNN7EXAMPLE` / `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` is **the placeholder key pair from AWS’s own documentation**, sitting in tens of thousands of repositories. The correct verdict is not “AWS key” — it is *false positive, documented example*. And the entropy filter replaces `"AKIAIOSFODNN7EXAMPLE"` with ``. **It deletes the word EXAMPLE, which is the evidence.** What survives says “AWS, region, S3 endpoint” — exactly what you need in order to conclude “real key” and be wrong. The threshold is wedged, too. `s3.amazonaws.com` sits at 3.20 Shannon entropy and `AKIAIOSFODNN7EXAMPLE` at 3.68: half a point of margin between keeping the useful context and erasing the proof. So we measured it properly, on 4,000 annotated CredData lines, same ±20-line neighbourhood in both cases. Raw contextString literals only *Share of true-secret contexts still carrying each signal, raw code versus string literals only.* On top of that, **26.8% of contexts come out essentially empty** — 86% in `.txt`, 66% in `.c`, 28% in `.md`, 27% in `.yml`. The central flaw is structural: **our anchor is not a string literal, it is a variable name.** The entire regex engine is built on that — it is what both fixes above were about. In `password = "ywriv"`, literal extraction returns `ywriv` and throws away `password`. Three shorter objections: - **`` is a category error for an LLM.** That preprocessing — closed vocabulary, entropy masking, camelCase splitting, length filters — comes from embedding classifiers (word2vec/fastText feeding a CNN or BiLSTM), where an open vocabulary blows up the embedding matrix. There, `` *creates* a feature. An LLM has none of those constraints, and the *shape* of neighbouring values is one of its best signals. - **Comments vanish entirely.** 57–70% of contexts contain one, and `// fake key` is not a string literal. - **The saving isn’t real.** Once the system prompt is counted, it comes to 9% per call. The “survives obfuscation” argument is sound in its original domain — ProGuard’d APKs, minified JS bundles, where identifiers are destroyed and strings survive. We scan source code, where variable names are intact. And in the minified case the algorithm collapses anyway: a minified bundle is one enormous line, so radius-based expansion has nothing to walk. ## What the literature actually says Rather than keep arbitrating on instinct, we went and read what has been published. One paper has **exactly our architecture**. Rahman et al., [*Secret Breach Detection in Source Code with Large Language Models*](https://arxiv.org/abs/2504.18784) (IEEE 2025): regex candidate extraction, then LLM classification, evaluated on SecretBench — 97,479 candidates, 818 GitHub repositories, 49 languages, 15,084 hand-verified secrets. *F1 by model and prompting strategy, from Rahman et al. (2025).* Four findings we could act on immediately: **They use 200 characters of raw context, and the ablation says that’s enough.** Going from 200 to 300 characters moves LLaMA-3.1-8B from 0.9852 to 0.9895 F1. In the authors’ words: *“200 characters already offering strong performance in most use cases.”* We were at 1,500 — chosen by guesswork. And nobody in this literature filters context down to string literals. **Few-shot is the cheapest available gain.** GPT-4o goes from 0.8428 zero-shot to 0.9392 few-shot. Nearly ten points for a handful of examples in the prompt. **Small models in zero-shot don’t hold up.** Raw and zero-shot: Gemma-7B 0.335, LLaMA-3.1-8B 0.413, CodeLLaMA-7B 0.521. That is the direct answer to “why not just run a 3B locally”. Fine-tuned with QLoRA, LLaMA-3.1-8B reaches 0.9852 — and their training ran on consumer hardware (i5-13400F, RTX 4090, under 17 GB VRAM). If you want local, that’s the route. **Fine-grained typing is unreliable without fine-tuning.** On multiclass, GPT-4o scores **0.178 F1 on “Generic Secret”** and 0.000 on “Username”. The binary verdict is solid; the category label is not — so don’t route on it automatically. Which is awkward for us, given our highest-volume rule is literally called `Generic High Confidence Secret`. On *how many* few-shot examples, the paper declines to say. It defines few-shot as *“multiple labeled examples covering varied secret and non-secret cases”* without ever publishing a count, and files the choice under threats to validity: *“the subjectivity in prompt design and few-shot examples used… the selection of examples might influence model behavior, particularly for ambiguous candidates.”* Diversity and representativeness are the only guidance on record. ### A trap aimed squarely at our next step [*From Data Leak to Secret Misses*](https://arxiv.org/html/2601.22946) shows that **SecretBench contains 69.3% exact duplicates** (8.7% near-duplicates, 22% genuinely unique), and that published performance collapses once you deduplicate: Random Forest MCC 0.89 → 0.65, LSTM 0.92 → 0.77. So before building an LLM test set on CredData, we measured it: Population Samples Unique Duplicates True secrets (`T`) 15,257 13,640 10.6% False positives (`F`) 47,747 32,080 **32.8%** Far cleaner than SecretBench — but a third of the false positives are duplicates. `Token::ILLEGAL,` appears 116 times; `"Auth Token: fake_token\n"` appears 233. An LLM benchmark on the raw `F` set would mostly measure the ability to reject three repeated lines. ## What we changed in the classifier Two changes, both straight out of the measurements above. **Context: from a 1,500-character total window to 150 characters either side of the value.** The semantics change along with the number, deliberately. *A PEM key truncated at 300 characters would have filled a 300-character total window entirely, leaving nothing around it — which is precisely the information the model needs.* **Four complete few-shot demonstrations.** Our system prompt already had examples — but in prose, at the level of the value alone (`AKIAIOSFODNN7EXAMPLE -> false positive`). That is not few-shot: the model never saw the exact shape of the question, nor of the expected answer. The new demonstrations go in as conversation turns, in the real input/output format. # Value Entropy Verdict 1 `Kx92!vamos_2019` in a `.env` 3.64 **real leak** 2 `AKIAIOSFODNN7EXAMPLE` 3.68 **false positive** 3 `sha512-p8p0B+qq…` in a lockfile 5.18 false positive 4 `ghp_x7Qw9zR2kLm4…` in a shell script 5.27 real leak The opening pair is the whole point: near-identical entropy, opposite verdicts. That is what teaches the model that **entropy does not decide** — something our system prompt asserted but never demonstrated. Then a maximum-entropy value that isn’t a credential, and one clear-cut leak so the set doesn’t lean toward false positives. System promptFew-shotContext *Prompt composition per classifier call.* The honest result: **the prompt grew by 6%.** Few-shot costs more than the context reduction saves. That is the price of the ~10 F1 points the paper reports — a good trade if the goal is classification quality, the wrong one if it was token economy. The two changes are independent and separately reversible. We also deleted the four prose examples the demonstrations now cover in full, keeping the three that cover cases few-shot doesn’t (judging credentials *inside* a URI, the git-diff header artefact). And we bumped the cache key version — without that, verdicts rendered under the old prompt would have masked the new behaviour for the whole TTL, and we’d have concluded few-shot changes nothing. ## What we’re taking away **On the results.** Precision 0.573 → 0.688, recall 0.168 → 0.518, F1 0.259 → 0.591, and a scan twelve seconds faster than before while finding three times as much. The substance of it is about thirty characters of regex and one keyword list that was empty. **On method.** Every number here was measured, and three of our starting intuitions were contradicted by the measurement: the per-fix volume estimates (an order of magnitude out), the CPU cost (a bug, not a trade-off), and the belief that widening a rule costs precision (it was the reverse). **On scope.** Two entire families of misses are abandoned on purpose, capping our score on this benchmark at ~0.84. A benchmark measures what it measures; it doesn’t define the product. **Still on the list.** Removing the five identifier rules — around 1,480 noise detections for 101 true positives — and building an LLM-classifier benchmark on CredData’s annotated lines, with deduplication, or it will mostly measure duplicates. If you want to see the current pipeline’s output as it happens, the [live feed](/leaks) shows what it is finding across GitHub, GitLab and Codeberg right now, every key masked and every repository withheld. And if you’d rather know about your own history than watch someone else’s, that is what the [deep scan](/blog/deep-scan-your-whole-git-history) is for. ### References - Rahman, Ahmed, Wahab, Sohan, Shahriyar — [Secret Breach Detection in Source Code with Large Language Models](https://arxiv.org/abs/2504.18784), IEEE 2025 (arXiv:2504.18784) - [From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models](https://arxiv.org/html/2601.22946) (arXiv:2601.22946) - Basak et al. — [SecretBench: A Dataset of Software Secrets](https://arxiv.org/abs/2303.06729) (arXiv:2303.06729) - [FPSecretBench](https://github.com/setu1421/FPSecretBench) — 2.36M false positives from nine tools; access on request - Wahab et al. — [Secret Leak Detection in Software Issue Reports using LLMs](https://arxiv.org/abs/2410.23657): RoBERTa/CodeBERT reach 92.70% F1, beating prompted GPT-4o at 80.13% (arXiv:2410.23657) - Basak et al. — [A Comparative Study of Software Secrets Reporting by Secret Detection Tools](https://arxiv.org/abs/2307.00714) (arXiv:2307.00714) *Benchmark: CredData (Samsung), 11,457 files, commit `0ad42f4`. Every figure above comes from a recorded run, except the context-window sizing and the model comparison, which come from the published literature and describe a bet we have not yet verified on our own data.*