# Deep scan: your whole git history — LeakWatch > Why real-time monitoring can't see the key you leaked in 2021, and how LeakWatch's deep scan clones your repo, replays every commit on every ref, and does it without ever being able to touch someone else's code. Source: https://leakwatch.net/blog/deep-scan-your-whole-git-history --- [← Blog](/blog) # Deep scan: reading your entire git history, not just the last commit July 26, 2026 · 0xCr0c0 · 9 min read ## The blind spot nobody talks about Real-time monitoring has a structural flaw, and it’s worth stating plainly: **it only sees what happens after you sign up.** LeakWatch’s continuous monitoring watches new pushes on your repositories. That’s the right tool for the leak you’re about to create. It is completely useless for the leak you created eighteen months ago, in a `config.py` you deleted the same evening, on a branch nobody has checked out since. Worse: to keep the first pass cheap, monitoring only looks at the **last 10 commits** of a repo when you first subscribe to it. Everything older is out of frame. That’s the gap the **deep scan** fills. One repo, the whole history, every ref, one pass. And that history is not a detail. A secret committed once is in the object database forever — deleting the file doesn’t remove it, force-pushing usually doesn’t either, and every clone and fork carries a copy. As we [wrote elsewhere](/blog/i-accidentally-pushed-an-api-key-to-github), the only thing that actually revokes access is rotating the credential. But you can’t rotate what you don’t know is in there — and the [live feed](/leaks) is a steady reminder of how ordinary this is: cloud, AI, payment and database credentials, all day long. ## Why we clone instead of calling the API Everywhere else in LeakWatch, we talk to forges over HTTP — that’s the whole [firehose architecture](/blog/consuming-the-firehose-at-scale). For the deep scan, we do the opposite: `git clone --mirror`, then read locally. Two reasons, both decisive. **Cost.** Fetching a diff over the API costs one request per commit. On a 5,000-commit repo that’s 5,000 requests — an hour of a GitHub token’s entire hourly budget, for one user, for one repo. A mirror clone costs *one* network operation. **Coverage.** The API shows you the default branch easily and everything else painfully. `--mirror` brings back **all refs** — every branch, every tag, including the abandoned ones. That matters more than it sounds: the abandoned branch is exactly where the secret you thought you’d cleaned up is still sitting. ```python url = f"{CLONE_BASE[provider]}/{repo_full_name}.git" code, _out, err = await _run_git( ["clone", "--mirror", "--quiet", "--", url, str(dest)], timeout=budget ) ``` Then we walk the history newest-first: ``` git rev-list --all --no-merges --count # how much is there git log --all --no-merges --max-count=5000 # what we'll actually read ``` `--all` for every ref. `--no-merges` because `git show` on a merge commit produces no diff anyway — counting them would inflate the progress bar without scanning anything. And **newest-first is deliberate**: when a history exceeds our 5,000-commit ceiling, the commits we keep are the recent ones, whose keys have the best chance of still being live. A 2014 key in a repo with 40,000 commits is a museum piece; a 2026 one is a liability. Each commit’s diff then goes through **exactly the same pipeline** as the public firehose: 400+ regexes, heuristic and ML false-positive filtering, and strictly read-only validation against the provider. That part is described in detail in [how the scanner works](/blog/how-leakwatch-secret-scanning-works) — the deep scan doesn’t reinvent any of it, it just feeds it a different source. ## The clone is anonymous, on purpose This one deserves its own section, because it looks like an oversight and isn’t. **No token is ever injected into the clone URL.** Not yours, not ours. Credentials placed in a git URL end up in `ps` output, in git’s own error messages, and in our logs — three places where a secret has no business being. The deep scan targets public repositories, an anonymous clone is enough, and a private repo simply fails cleanly rather than making us handle a token for nothing. git also runs in a locked-down environment, so it can neither ask for credentials nor inherit anything from the host machine: ```python env.update({ "GIT_TERMINAL_PROMPT": "0", # never prompt for login/password "GIT_ASKPASS": "", "GIT_CONFIG_NOSYSTEM": "1", # ignore /etc/gitconfig "GIT_CONFIG_GLOBAL": "/dev/null", "GCM_INTERACTIVE": "never", }) ``` Without `GIT_TERMINAL_PROMPT=0`, a private repo doesn’t fail — it *hangs*, waiting forever on a password prompt nobody will ever type, holding a worker slot until the timeout kills it. ## The wall: you can only scan what you own Here’s the uncomfortable truth about this feature: a tool that clones a public repo and extracts every secret from its history is, from the outside, indistinguishable from an attack tool. The only thing separating “audit my own repo” from “harvest someone else’s” is the ownership check. So we treat it as the critical path it is. It lives in its own module, with its own tests, with no HTTP and no Celery around it: ```python identity = (await db.execute( select(UserIdentity).where( UserIdentity.user_id == user.id, UserIdentity.provider == provider, UserIdentity.status == IDENTITY_STATUS_VERIFIED, func.lower(UserIdentity.username) == owner.lower(), ) )).scalar_one_or_none() ``` The owner is the first path segment, compared case-insensitively against a **verified OAuth identity** on your account. No verified identity matching that owner, no scan. And when the refusal happens, we stay deliberately vague about why: distinguishing “you have no account on this forge” from “that account isn’t yours” would tell a curious user something about other people’s linked accounts. The check runs at launch — and **again when you open the report**. Between the two, the identity that justified access may have been unlinked. A page that displays secrets in plaintext doesn’t get to rely on a stale authorization. The repo name itself is validated before anything touches the network or the disk: ```python REPO_FULL_NAME_RE = re.compile( r"^[A-Za-z0-9][A-Za-z0-9._-]{0,99}(?:/[A-Za-z0-9][A-Za-z0-9._-]{0,99}){1,4}quot; ) ``` Every segment must start with an alphanumeric. That single constraint kills two classes of problem at once: `..` path traversal (the name is used to build a temp directory) and segments starting with `-`, which git’s command line would read as options. We still pass an explicit `--` to git afterwards, and we spawn it with `create_subprocess_exec` rather than `_shell` so arguments never cross an interpreter. Three barriers for one risk, because the first one is a regex and regexes are how people get owned. ## Bounded everywhere, because a repo is not a friendly input A deep scan is the only place where a user hands us an arbitrary amount of work. Every dimension of it is capped: Limit Value Why Commits scanned 5,000 Newest-first; beyond that the result is flagged partial Clone size 500 MB Measured after cloning, before scanning Total runtime 30 min Hard deadline shared by every git subprocess Diff size 512 KB Same ceiling as the firehose; generated artifacts are noise The diff cap is enforced by *streaming*, not by reading and then checking. One `git show` on a massive vendored-dependency import would otherwise be enough to blow up the worker’s memory: ```python raw = await asyncio.wait_for(proc.stdout.read(MAX_PATCH_BYTES + 1), timeout=budget) if len(raw) > MAX_PATCH_BYTES: logger.info("Commit %s too large — skip.", sha[:7]) return None ``` And when the 30-minute deadline hits mid-history, we don’t fail the scan. We stop, mark the result `truncated`, and say so in the report. What was scanned really was scanned; pretending otherwise would be the actual failure. One more thing about the clone: it contains your secrets in plaintext, on our disk. It is removed in a `finally` block, so it disappears whether the scan succeeded, failed, timed out or crashed. ## Watching a 30-minute job without lying to the user A scan that shows no progress for half an hour is indistinguishable from a scan that died. So progress is published **twice**, to two systems with different failure modes: - **Celery task state**, for fine-grained updates between database writes. - **A row in Postgres**, so progress survives a page reload *and* a worker restart. The write happens on whichever trigger fires first — every 25 commits, or every 10 seconds: ```python if (commits_done % settings.DEEP_SCAN_PROGRESS_EVERY == 0 or now - last_flush >= settings.DEEP_SCAN_PROGRESS_SECONDS): ``` Two triggers because the two failure modes are opposite. The counter bounds the number of UPDATEs on a history that flies by. The clock guarantees the bar still moves when a single commit costs several seconds of LLM classification. The frontend polls every 2 seconds, and the database row is always the source of truth — Celery’s state is only used to refine the number between flushes, never to lower it. The scan also runs on its **own Celery queue**, served by a dedicated worker. A 30-minute clone sharing a queue with the firehose would stall real-time detection for everyone else; queue isolation is what keeps one user’s 40,000-commit monorepo from becoming everybody’s problem. And because a worker can always die mid-flight — OOM, redeploy, host failure — a Beat task sweeps hourly for scans stuck in `queued` or `running` past twice the timeout, and fails them. Without that sweep, a killed worker leaves an eternal `running` row that counts against your quota and permanently blocks you from starting another one. The unglamorous half of shipping background jobs. ## A subtlety about the report This one bit us, and the fix is worth explaining. The report is **not** the list of what the scan inserted. It’s the current state of all known findings for that repository. Why the difference matters: our pipeline deduplicates globally by content hash. If your leaked Stripe key was already caught by the public firehose three months ago, the deep scan will re-detect it, recognize it as an existing finding, and insert nothing. A report built from the worker’s return value would proudly announce “nothing found” on a repo that is actively leaking. So the report queries findings by repo, sorts by severity, and shows you **everything we know**, whatever discovered it. Secrets appear in plaintext there, deliberately: you’re the verified owner, the value is already readable in your own public history, and a masked value wouldn’t tell you *which* of your keys to rotate. Finishing a scan also re-scores the repo, even when it found nothing new — because scan coverage just moved from “thin” to “full”, which changes what we’re entitled to claim about that repository. ## What it costs you One deep scan per 30-day sliding window on the free plan; unlimited on a paid one. **Failed scans don’t count** — if our clone times out or our worker dies, that’s our problem, not your monthly quota. The ordering of the checks is also deliberate: ownership first, quota second. Pointing at the wrong repo shouldn’t burn your one free scan, and shouldn’t incidentally teach you anything about your quota either. ## In short Continuous monitoring answers “is something leaking right now?”. The deep scan answers the question that usually matters more: **“what has been leaking this whole time?”** - `git clone --mirror` → one network operation, every ref, including the branches you forgot. - Anonymous clone, locked git environment, temp directory destroyed no matter what. - Ownership verified against a linked OAuth identity, at launch *and* at read. - Every dimension capped — commits, size, runtime, diff — and partial results labeled as partial. - Progress persisted in two places, on an isolated queue, with a reaper for the days the worker doesn’t make it. Want to know what’s in your history? [Run a deep scan on your own repo](/product#deep-scan) — the first one is free. ![Gabriel Diyan, founder of LeakWatch](/brand/Photo-Gabriel-Diyan.webp) Gabriel Diyan (0xCr0c0) Cybersecurity student, founder of LeakWatch. I built and run the scanner described here — the detection patterns, the false-positive classifier and the provider validators are mine. [More about who I am](/about). [GitHub](https://github.com/Leakwatch-Scan) · [X](https://x.com/LeakwatchScan) · [LinkedIn](https://www.linkedin.com/in/gabriel-diyan-80a378375) · [GitHub (personal)](https://github.com/crocogab) LeakWatch Secrets leak into public commits every minute. This watches the forges for yours. Built and run by [Gabriel Diyan](/about), a cybersecurity student — [why LeakWatch exists](/about). Scan - [Product](/product) - [Live feed](/leaks) - [Trends](/leaks/trends) - [API docs](/docs) - [CI/CD](/docs?tab=ci) Read - [Blog](/blog) - [Changelog](/changelog) - [About](/about) Verify - [Security](/security) - [Privacy](/privacy) - [Terms](/terms) - [Legal](/legal) - [Contact](/contact) - [Status](https://status.leakwatch.net) © 2026 LeakWatch [GitHub](https://github.com/Leakwatch-Scan)[X](https://x.com/LeakwatchScan)