Introduction
Every second, GitHub, GitLab and Codeberg swallow thousands of public commits. The vast majority are harmless and dull — but every so often, an API key, a token or a .env slips in through carelessness / inattention / drunkenness.
To catch those leaks, LeakWatch has to drink from the hose: the firehose of public events from these forges. The hard part isn’t spotting a secret inside a diff (our 400+ regular expressions handle that) — it’s ingesting that volume of data without ever overloading the forges’ servers, or our own, all while staying strictly within their API limits.
This article walks through the three pillars that make that ingestion sustainable over time: ETags, rate-limit handling, and deduplication.
One important note up front: consuming the public firehose is not the core of the product. It’s our full-scale playground — it trains our algorithms and demonstrates the tool to the community. LeakWatch’s primary mission remains the targeted protection of our customers’ private repositories. The firehose is the showcase; securing your code is the long-term goal.
Three forges, three different firehoses
First trap: there isn’t one firehose, but three quite distinct mechanics.
- GitHub exposes a genuine global stream: the
/eventsendpoint. We pullPushEvents from around the world, page by page. - GitLab doesn’t let you see every event on the platform. We work around it by querying
/projectssorted by recent activity, then drilling down into each project’s commits. - Codeberg (built on Gitea) has no global stream either. We query
/repos/search?sort=updated&order=descto list freshly-modified public repos, then go fetch their commits. Given its small-scale infrastructure and tight API restrictions, we decided to only use the Codeberg firehose when a user actually needs it :))).
Three strategies — but the same three problems to solve behind each one.
Pillar 1 — ETags: don’t re-download what hasn’t moved
GitHub’s firehose returns the same events as long as nothing new has been pushed. Re-downloading 100 identical events every cycle would be terrible for our bandwidth as much as for their servers.
The solution is an under-used HTTP mechanism: the ETag. GitHub returns an ETag header with every response. On the next cycle, we send it back in If-None-Match:
if self.etag:
req_headers["If-None-Match"] = self.etag
response = await client.get(events_url, headers=req_headers)
if response.status_code == 304: # Not Modified
return [] # nothing new, we move on
self.etag = response.headers.get("ETag")
The payoff is twofold. If nothing changed, GitHub answers 304 Not Modified — an empty response, and crucially one that doesn’t count against our request quota. We save both network and rate limit, without missing a thing.
Pillar 2 — Rate limits: token rotation and circuit breakers
This is where the real fight is. An unauthenticated GitHub token is capped at 60 requests/hour — far too little; authenticated, 5000/hour. And a single cycle can require dozens of diff downloads. Two techniques let us keep the pace without ever hitting too hard.
A rotating token pool
Rather than a single token that burns out fast, we load a pool of tokens and spread requests round-robin (clever, right?):
self._token_cycle = itertools.cycle(self.tokens)
Each diff download grabs the next token. As a result, the effective concurrency becomes concurrency_per_token × number_of_tokens, which parallelizes the ingestion without ever exceeding a single token’s limit.
And when a token gets rate-limited (403 or 429), we don’t kill the whole cycle: we set it aside and switch to the others. The cycle only stops once every token in the pool is exhausted (which honestly never happens, because we take it easy so as not to overload the servers):
if patch_content == "__rate_limited__":
exhausted_tokens.add(token)
if exhausted_tokens.issuperset(self.tokens or [None]):
all_exhausted.set() # the whole pool is dry, we stop
# otherwise: we keep going with the still-fresh tokens
Codeberg’s circuit breaker
Codeberg/Gitea has a much tighter rate limit. Here, rotation isn’t enough: on the first 429 (or even a timeout), we trip a circuit breaker — a forced cooldown that pauses the collector for a set time rather than pushing harder and making things worse. In practice the collector is rarely used, because the few requests we have are spent monitoring users’ repos.
if r.status_code in (403, 429):
raise CodebergRateLimited(cooldown=self.SLEEP_ON_RATE_LIMIT)
It’s a principle we apply everywhere: at the first sign a forge is straining, we stop collecting. Better to miss a few commits from one cycle than to hammer a server that’s already under load (poor thing, spare a thought for it).
Pillar 3 — Deduplication: scan each push only once
The firehose is inherently redundant. The /events pages overlap, and the same push often comes back twice from one cycle to the next. Scanning the same diff twice means doubling the network and CPU cost for zero new information.
GitHub: an atomic, sliding-window Redis Set
Since several Celery workers run in parallel, deduplication has to be shared and race-condition-free. We lean on a genuinely cool property of Redis: SADD returns 0 if the element already existed.
def _is_already_seen(self, push_id: str) -> bool:
added = self._redis.sadd(SEEN_PUSH_KEY, push_id)
self._redis.expire(SEEN_PUSH_KEY, SEEN_PUSH_TTL) # sliding window ~1h
return added == 0
A single atomic call, no locks, and a roughly one-hour TTL that keeps the Set bounded. We check this filter before any expensive network I/O: already-seen pushes are dropped without even downloading their diff.
Codeberg: a bounded in-memory cache
Where GitHub warrants a shared Redis Set, Codeberg — lower volume — makes do with an in-memory cache of a few thousand SHAs, halved when it hits its ceiling. The right tool for the right volume.
The filters that do the rest: cut the noise before it costs anything
Even before scanning, we throw away everything with a near-zero chance of containing a real user secret — again, before any network download:
- Known bots:
dependabot[bot],renovate[bot],github-actions[bot]… An explicit list, finer than a plain[bot]test that would discard potentially interesting home-grown bots. - Giant diffs: past 512 KB, a diff is almost always a generated artifact (lockfile, vendored dependencies, build). High scan cost, weak signal — we cut it, including mid-stream so we never load the whole diff into memory:
async for chunk in resp.aiter_bytes():
buf.extend(chunk)
if len(buf) > MAX_PATCH_BYTES:
return (None, url) # bail as soon as we go over
In short
Consuming three firehoses at scale comes down to one principle: do as little work as possible, as late as possible, and back off at the first sign of strain.
- ETags → we never re-download what hasn’t moved (and the 304 costs no quota).
- Token rotation + circuit breakers → we parallelize without ever overloading a forge, and we step aside when it saturates.
- Redis/in-memory dedup + pre-filters → each push is scanned only once, and the noise is eliminated before any costly I/O.
That’s what lets us keep a steady throughput on the public showcase, while keeping the bulk of our resources where the real mission is: continuous monitoring of our customers’ repositories.
Curious what the scanner turns up on your repos? The look is free and takes a few seconds.