Branch protection worked. It lasted five seconds

At 10:08:29 UTC on 21 October, GitHub branch protection did exactly what it was there to do: it rejected an unauthorised modification to LiteLLM's release workflow, because it had not passed the required review and status checks. At 10:08:34 — five seconds later — a credential belonging to a privileged developer account overrode that protection and applied the change. Ninety minutes later there were seven poisoned wheels on PyPI. No control failed. All three were satisfied.

LiteLLM is a gateway you put in front of several language models so you can talk to all of them with one call. If anyone in your company has built something with AI in the last year, it is worth checking whether it is down there somewhere, even if it has never come up in a management meeting. On 21 October a version 1.107.0 the project had not built showed up on PyPI and Docker, with a credential stealer inside. And afterwards the team published something almost nobody publishes: the second-by-second timeline of what happened inside their own release pipeline.

That document is the interesting part, and not because of the package. It is a blueprint, minute by minute, of how you walk through a release chain that had in place every control everyone recommends. We did not read it thinking about Python; we read it thinking about the GitLab, the Jenkins or the Azure DevOps of any of the software companies whose infrastructure we run.

The five seconds, read from the official report

This is the sequence exactly as the project itself publishes it, in UTC on 21 October 2026. There is nothing here that is not in their advisory.

Read it again looking only at the three moments in bold. Three different credentials from three different people, used at three different points in the chain, and each one does something perfectly permitted for whoever holds it: override branch protection, approve a production deployment, clean up run records. The attacker broke nothing. They filled in forms.

Three controls, one single question

Here is what we take away, and it is uncomfortable because it looks a lot like what most of us have built. Those three controls are not three layers: they are the same layer repeated three times. Branch protection asks "do you hold a privileged credential that can override me?". The approval gate asks "is there a second developer credential saying yes?". And the audit log assumes whoever writes it is not the attacker. All three questions are the same question: are you who you say you are? And the answer was yes, three times, because all three identities were stolen.

And there is a fourth control that does not appear in the timeline either, because it never fired. The attack-chain document mentions it in passing, in a sentence worth an entire audit: "GitHub CLI tokens are post-2fa and not subject to GitHub's 'super-user' 2fa reauth flows." Which is to say: two-factor authentication, the thing most of us put on the table when asked whether the chain is protected, was switched on and had nothing to say here. It had already been satisfied, back on the day somebody authenticated on their own laptop.

Three doors in a row are not defence in depth if all three take the same key. What you need are doors that ask different questions, which is exactly what Zero Trust is about when it is applied for real and not as a marketing label. The question missing from the whole chain, and which no control asked at any point, is the simplest one: is this change the change we meant to make? Nobody compared the release workflow that ran against the release workflow that had been approved. Had anyone done so, the line pointing at a repository with one extra hyphen in the organisation name would have stood out on its own.

Where the credentials came from (the part that hit close to home)

The advisory explains the origin in a sentence worth reading slowly: "Claude Code will embed a user's GitHub CLI OAuth token into a response body, and in situations where response bodies are logged, these tokens may appear in LiteLLM logs." The attacker got into an internal LiteLLM deployment exposed to the public internet and pulled the tokens out of there: first the organisation owner's, then, the same day, those of two engineers with maintainer privileges.

That is the sentence to take home, because it has nothing to do with LiteLLM. If you have put a gateway in front of the models — and almost everyone doing anything serious with AI ends up with one, for cost control or to avoid handing keys around — that gateway logs requests and responses. That is its job. The day one of the tools talking to it puts a credential inside a response, your log table becomes a secret store you never declared, that nobody rotates, that is probably not encrypted at rest, and that far more people can read than can read your password manager. We wrote about the other side of this same problem when we found that the AI agent will not be in your log, whoever lent it the token will.

Two days after the incident, on 23 October, the project published a second document with the root of it all, and it is harder reading than the first. On 27 May 2026 a configuration file containing the master key to a deployment of their agent platform was committed to a public repository — LiteLLM-Labs/lite-harness. They tell it themselves: "The error was quickly identified, the commit force-pushed over, and we identified the exposed keys as requiring revocation. However, key revocation did not occur for the master key." On 10 October, 136 days later, the attacker used that key to get into the platform and pull the token out of it. Of that token the same document says two more things: that it "was not associated with a service account and was not scoped down", and that it belonged to the account of their GitHub organisation owner.

The detail that changes your triage: you do not have to use the library

Almost everyone, on hearing about it, asked the wrong question: "do any of our applications use this?". That is not the one. According to StepSecurity's technical breakdown, the poisoned package dropped three files into site-packages, and one of them moves the problem somewhere else entirely: cook.pth, twelve bytes whose entire content is import cook.

A .pth file is an old, legitimate Python mechanism: the site module, when the interpreter starts, executes any line in a .pth that begins with import. Before your code. On every interpreter start. It makes no difference whether you launch the application, run pip, run the tests, or type a three-line python -c to check something. Nobody ever has to import litellm. It is enough for the package to be installed in that virtualenv.

That moves the blast radius from your AI applications to any machine where somebody ran a pip install to try it out: the laptop of the developer who had a look on a Friday, the ephemeral CI container, the machine of someone on a work placement, the virtualenv of the proof of concept that was never switched off. And what the stealer collects, again per StepSecurity, is not modest: browser passwords and cookies, keys from ~/.ssh, GPG keys, .pem files, AWS, Azure and GCP credentials, kubectl and Docker configuration, GitHub CLI tokens, AI tool credentials, the gateway's own credentials table, and — the part least discussed — the environment of every running process, read from /proc. Which means the environment variables of services that have nothing to do with Python.

The third file, sitecustomize.py, leaves a background process that every hour downloads a chunk of code from a server and executes it. The server is not hardcoded: it reads it from a text file hosted in a public repository, so it can be changed at will. That is no longer a malicious package; it is an open door with a maintenance contract.

The real window: two hours and forty minutes

The operational question is narrower than "do we use the library" and than "did we use it back then". what did your infrastructure download between 11:39:51 and 14:20 UTC on 21 October. On PyPI the wheels were live from 11:39:51 until 13:23. On Docker Hub and GHCR, the v1.107.0, latest, main-latest and main-stable tags pointed at the poisoned image from 11:46:56 until 14:20. In Spanish time that is 13:39 to 16:20: the middle of a working Wednesday afternoon.

Two hours and forty minutes of total window. If your builds run overnight, you most likely escaped on timing alone. If you build on every merge to main, you built several times that afternoon. That difference — which has nothing to do with security and everything to do with what time your robot runs — decided who pulled the package and who did not. It is a good reminder of how non-deterministic this is: your dependency policy did not save you, cron did.

What the lesson is NOT

Half the industry's automatic reaction was "pin your versions". Pinning is good and you should do it, but it needs saying more precisely, because put that way it is false comfort: 1.107.0 was a pinnable version. Anyone with a dependency update bot configured to open and auto-merge minor bumps pulled the poisoned package in through their lockfile, not despite it. Your pin is worth exactly as much as the process that is allowed to change it.

The second reaction was "we use containers, we do not install packages by hand". That fails too: the latest and main-stable tags are moving references — pins that are not pins. Anyone who rebuilt an image that afternoon pointing at main-stable pulled the malicious content without changing a single line of their build file. What actually anchors an image is the digest, not the tag, and almost nobody uses it because it means editing the file every time you want to update. That is the real price, and it should be decided knowing what it costs.

And now the part that costs us money to say: we are not going to tell a twenty-person company to run its own PyPI mirror. It is expensive to maintain, it goes stale, and the day it breaks it breaks quietly. What is proportionate at that size is three boring things: a quarantine of 24 or 72 hours before adopting any new version (which here would have saved you from the PyPI package, which lived under two hours; from Docker's moving tags, no), an inventory of which packages and which images actually enter your builds, and taking away the dependency bot's permission to merge on its own. All three are a day of work, not a project. The private mirror is for people who build a hundred times a day, and then yes.

Seven checks, with the commands

This is what we would do, in this order, across a fleet with Python anywhere in it. The indicators are the ones published by StepSecurity and by the project itself. Before you start, a warning with consequences: on a suspect machine do not begin with pip install or pip uninstall. Every Python invocation fires the payload. You delete from the shell.

1. Look for the file, not the application. find / -name cook.pth -path "*site-packages*" 2>/dev/null, and run it across every machine with Python, not just the application servers: laptops, CI runners, lab boxes. The file is twelve bytes; the inventory is the expensive part.
2. Check whether the background process is still alive. ls -la /tmp/remote_code_loader.lock and lsof /tmp/remote_code_loader.lock to find what is holding it, and kill it. The stealer also leaves a marker, /tmp/.pc_once_72129118c59f5636c0bb, once it has managed to send the data; its absence proves nothing.
3. If it turns up, do not clean it: rebuild the environment. StepSecurity's recommendation is explicit — recreating the virtualenv or the image from scratch is better than cleaning it in place. If you must clean, that is all three files plus the matching __pycache__.
4. Hunt the version in your files and your internal mirror. grep -rniE "litellm[=<>~! ]*1\.107\.0" . --include='*.txt' --include='*.toml' --include='*.lock' --include='Pipfile*' across the repositories, lockfiles included: that is where it came in. Then check whether your package cache or mirror kept a copy, because a PyPI removal does not empty your cache.
5. Compare image digests, not tags. docker images --digests | grep -E 'berriai/litellm' and cross-check against the digests published as compromised. If you rebuilt anything that afternoon pointing at latest or main-stable, assume it got in.
6. If there was exposure, rotate as if everything went. The advisory says it plainly: the instance or task role, every provider key, the database password, and also the gateway's own master key and virtual keys, which is where this all started. Add the SSH and GPG keys of the users of those machines and any credentials saved in the browser. Rotating half the estate leaves the feeling of having acted: write down what is left unrotated and why, with a date.
7. Look backwards in your cloud audit log. The advisory recommends reviewing CloudTrail; whatever equivalent you use will do. What you are looking for is use of those keys after 11:39 UTC on 21 October from places that are not yours. And check how many days of log you actually keep, because that window decides whether this check is possible at all.

And an eighth one, which is not technical and is the one that actually changes something: sit down with whoever runs your release pipeline and ask three questions. Who can override branch protection, and whether a trace of it survives outside the system that same privilege can wipe. Whether the approval gate is satisfied by a person or by a credential, because they are not the same thing and it is almost always built as the latter. And whether anyone compares what ran against what was approved. If all three answers are uncomfortable, you have next week's work. The pattern is not new: we saw it on the npm side when we wrote about the 444 npm packages and nothing to patch, and on the build server when it turned out that the continuous integration server holds the keys to production.

What we take away

One thing in this team's favour: they published the timeline to the second, including the part where their own protection was overridden and their own records were deleted. Nobody does that for fun, and almost nobody does it at all. A good part of what we know about this incident we know because they wrote it down.

And what is left once you set aside Python, AI and the specific package is a sentence that applies to your hypervisor, your backup and your management console just as much as to your release pipeline: a control that only verifies identity is worth zero the day identity is what was stolen. Not because it is badly configured, but because it is answering a question the attacker already knows the answer to. The five seconds of 21 October are not a curious footnote: they are how long it takes a well-configured control to become decorative.

We look at this as part of consulting: an inventory of what enters your builds, who holds which privilege over the chain, and what gets recorded outside the reach of that privilege. That last point is the one we have most often found blank.

And a detection note, because there is no odd file to find here: what gives that machine away is a process waking up every hour on a box where nobody scheduled anything. A tool that watches behaviour catches that — EDR/MDR — and signature antivirus does not.

One last piece of honesty: we have not seen this package in any environment we run, and we give no figure for how many companies pulled it, because what gets counted in these cases is downloads, which is not the same as companies. Everything we state above comes from the two documents the project itself published and from StepSecurity's technical breakdown; where they write "we believe", we have not written "it was".

Sources (verified): UTC timeline, branch protection override, credentials involved, affected artefacts and remediation steps — the project's own advisory, docs.litellm.ai; technical analysis of the cook.pth, cook.py and sitecustomize.py files, scope of the theft, digests, indicators and detection commands — StepSecurity; public incident thread — BerriAI/litellm #47826. Root of the chain: the 27 May 2026 commit carrying the master key, the revocation that never happened, the organisation owner's unscoped token and the note about GitHub CLI tokens and second-factor reauth — canonical attack-chain record, published by the project on 23 October 2026.