Deep scan: reading your entire git history, not just the last commit

The blind spot nobody talks about

Real-time monitoring has a structural flaw, and it’s worth stating plainly: it only sees what happens after you sign up.

LeakWatch’s continuous monitoring watches new pushes on your repositories. That’s the right tool for the leak you’re about to create. It is completely useless for the leak you created eighteen months ago, in a config.py you deleted the same evening, on a branch nobody has checked out since.

Worse: to keep the first pass cheap, monitoring only looks at the last 10 commits of a repo when you first subscribe to it. Everything older is out of frame.

That’s the gap the deep scan fills. One repo, the whole history, every ref, one pass.

And that history is not a detail. A secret committed once is in the object database forever — deleting the file doesn’t remove it, force-pushing usually doesn’t either, and every clone and fork carries a copy. As we wrote elsewhere, the only thing that actually revokes access is rotating the credential. But you can’t rotate what you don’t know is in there.

Why we clone instead of calling the API

Everywhere else in LeakWatch, we talk to forges over HTTP — that’s the whole firehose architecture. For the deep scan, we do the opposite: git clone --mirror, then read locally.

Two reasons, both decisive.

Cost. Fetching a diff over the API costs one request per commit. On a 5,000-commit repo that’s 5,000 requests — an hour of a GitHub token’s entire hourly budget, for one user, for one repo. A mirror clone costs one network operation.

Coverage. The API shows you the default branch easily and everything else painfully. --mirror brings back all refs — every branch, every tag, including the abandoned ones. That matters more than it sounds: the abandoned branch is exactly where the secret you thought you’d cleaned up is still sitting.

url = f"{CLONE_BASE[provider]}/{repo_full_name}.git"
code, _out, err = await _run_git(
    ["clone", "--mirror", "--quiet", "--", url, str(dest)], timeout=budget
)

Then we walk the history newest-first:

git rev-list --all --no-merges --count      # how much is there
git log --all --no-merges --max-count=5000  # what we'll actually read

--all for every ref. --no-merges because git show on a merge commit produces no diff anyway — counting them would inflate the progress bar without scanning anything. And newest-first is deliberate: when a history exceeds our 5,000-commit ceiling, the commits we keep are the recent ones, whose keys have the best chance of still being live. A 2014 key in a repo with 40,000 commits is a museum piece; a 2026 one is a liability.

Each commit’s diff then goes through exactly the same pipeline as the public firehose: 400+ regexes, heuristic and ML false-positive filtering, and strictly read-only validation against the provider. That part is described in detail in how the scanner works — the deep scan doesn’t reinvent any of it, it just feeds it a different source.

The clone is anonymous, on purpose

This one deserves its own section, because it looks like an oversight and isn’t.

No token is ever injected into the clone URL. Not yours, not ours. Credentials placed in a git URL end up in ps output, in git’s own error messages, and in our logs — three places where a secret has no business being. The deep scan targets public repositories, an anonymous clone is enough, and a private repo simply fails cleanly rather than making us handle a token for nothing.

git also runs in a locked-down environment, so it can neither ask for credentials nor inherit anything from the host machine:

env.update({
    "GIT_TERMINAL_PROMPT": "0",     # never prompt for login/password
    "GIT_ASKPASS": "",
    "GIT_CONFIG_NOSYSTEM": "1",     # ignore /etc/gitconfig
    "GIT_CONFIG_GLOBAL": "/dev/null",
    "GCM_INTERACTIVE": "never",
})

Without GIT_TERMINAL_PROMPT=0, a private repo doesn’t fail — it hangs, waiting forever on a password prompt nobody will ever type, holding a worker slot until the timeout kills it.

The wall: you can only scan what you own

Here’s the uncomfortable truth about this feature: a tool that clones a public repo and extracts every secret from its history is, from the outside, indistinguishable from an attack tool. The only thing separating “audit my own repo” from “harvest someone else’s” is the ownership check.

So we treat it as the critical path it is. It lives in its own module, with its own tests, with no HTTP and no Celery around it:

identity = (await db.execute(
    select(UserIdentity).where(
        UserIdentity.user_id == user.id,
        UserIdentity.provider == provider,
        UserIdentity.status == IDENTITY_STATUS_VERIFIED,
        func.lower(UserIdentity.username) == owner.lower(),
    )
)).scalar_one_or_none()

The owner is the first path segment, compared case-insensitively against a verified OAuth identity on your account. No verified identity matching that owner, no scan. And when the refusal happens, we stay deliberately vague about why: distinguishing “you have no account on this forge” from “that account isn’t yours” would tell a curious user something about other people’s linked accounts.

The check runs at launch — and again when you open the report. Between the two, the identity that justified access may have been unlinked. A page that displays secrets in plaintext doesn’t get to rely on a stale authorization.

The repo name itself is validated before anything touches the network or the disk:

REPO_FULL_NAME_RE = re.compile(
    r"^[A-Za-z0-9][A-Za-z0-9._-]{0,99}(?:/[A-Za-z0-9][A-Za-z0-9._-]{0,99}){1,4}
quot;
)

Every segment must start with an alphanumeric. That single constraint kills two classes of problem at once: .. path traversal (the name is used to build a temp directory) and segments starting with -, which git’s command line would read as options. We still pass an explicit -- to git afterwards, and we spawn it with create_subprocess_exec rather than _shell so arguments never cross an interpreter. Three barriers for one risk, because the first one is a regex and regexes are how people get owned.

Bounded everywhere, because a repo is not a friendly input

A deep scan is the only place where a user hands us an arbitrary amount of work. Every dimension of it is capped:

The diff cap is enforced by streaming, not by reading and then checking. One git show on a massive vendored-dependency import would otherwise be enough to blow up the worker’s memory:

raw = await asyncio.wait_for(proc.stdout.read(MAX_PATCH_BYTES + 1), timeout=budget)
if len(raw) > MAX_PATCH_BYTES:
    logger.info("Commit %s too large — skip.", sha[:7])
    return None

And when the 30-minute deadline hits mid-history, we don’t fail the scan. We stop, mark the result truncated, and say so in the report. What was scanned really was scanned; pretending otherwise would be the actual failure.

One more thing about the clone: it contains your secrets in plaintext, on our disk. It is removed in a finally block, so it disappears whether the scan succeeded, failed, timed out or crashed.

Watching a 30-minute job without lying to the user

A scan that shows no progress for half an hour is indistinguishable from a scan that died. So progress is published twice, to two systems with different failure modes:

• Celery task state, for fine-grained updates between database writes.
• A row in Postgres, so progress survives a page reload and a worker restart.

The write happens on whichever trigger fires first — every 25 commits, or every 10 seconds:

if (commits_done % settings.DEEP_SCAN_PROGRESS_EVERY == 0
        or now - last_flush >= settings.DEEP_SCAN_PROGRESS_SECONDS):

Two triggers because the two failure modes are opposite. The counter bounds the number of UPDATEs on a history that flies by. The clock guarantees the bar still moves when a single commit costs several seconds of LLM classification. The frontend polls every 2 seconds, and the database row is always the source of truth — Celery’s state is only used to refine the number between flushes, never to lower it.

The scan also runs on its own Celery queue, served by a dedicated worker. A 30-minute clone sharing a queue with the firehose would stall real-time detection for everyone else; queue isolation is what keeps one user’s 40,000-commit monorepo from becoming everybody’s problem.

And because a worker can always die mid-flight — OOM, redeploy, host failure — a Beat task sweeps hourly for scans stuck in queued or running past twice the timeout, and fails them. Without that sweep, a killed worker leaves an eternal running row that counts against your quota and permanently blocks you from starting another one. The unglamorous half of shipping background jobs.

A subtlety about the report

This one bit us, and the fix is worth explaining.

The report is not the list of what the scan inserted. It’s the current state of all known findings for that repository.

Why the difference matters: our pipeline deduplicates globally by content hash. If your leaked Stripe key was already caught by the public firehose three months ago, the deep scan will re-detect it, recognize it as an existing finding, and insert nothing. A report built from the worker’s return value would proudly announce “nothing found” on a repo that is actively leaking.

So the report queries findings by repo, sorts by severity, and shows you everything we know, whatever discovered it. Secrets appear in plaintext there, deliberately: you’re the verified owner, the value is already readable in your own public history, and a masked value wouldn’t tell you which of your keys to rotate.

Finishing a scan also re-scores the repo, even when it found nothing new — because scan coverage just moved from “thin” to “full”, which changes what we’re entitled to claim about that repository.

What it costs you

One deep scan per 30-day sliding window on the free plan; unlimited on a paid one. Failed scans don’t count — if our clone times out or our worker dies, that’s our problem, not your monthly quota.

The ordering of the checks is also deliberate: ownership first, quota second. Pointing at the wrong repo shouldn’t burn your one free scan, and shouldn’t incidentally teach you anything about your quota either.

In short

Continuous monitoring answers “is something leaking right now?”. The deep scan answers the question that usually matters more: “what has been leaking this whole time?”

• git clone --mirror → one network operation, every ref, including the branches you forgot.
• Anonymous clone, locked git environment, temp directory destroyed no matter what.
• Ownership verified against a linked OAuth identity, at launch and at read.
• Every dimension capped — commits, size, runtime, diff — and partial results labeled as partial.
• Progress persisted in two places, on an isolated queue, with a reaper for the days the worker doesn’t make it.

Want to know what’s in your history? Run one on your own repo — the first one is free.