Open Source

Surviving a Package Hijack

Surviving a Package Hijack

A hijacked open-source package is never only your incident — it is every stranger's incident who trusted the name enough to install it. That is the real argument for defending the commons with automation you have actually verified.

One of our npm packages got hijacked. A real version of @hasna/mementos, published under our name by someone who was not us. We took it back one patch release later — 0.14.33 compromised, 0.14.34 clean. That sentence is the whole incident, and it is over almost as soon as it starts. What is not over, and what a one-line commit message never shows, is what a hijack of an open package actually costs, and who it costs it to.

The uncomfortable part of publishing anything under Apache 2.0 is that a hijack is never only your incident. A private repository can only ever hurt the people who work in it. An open package can hurt every stranger who trusted the name enough to run one install command in the wrong half hour. Anyone who typed bun install -g @hasna/mementos while 0.14.33 was live got an attacker's code running with their own permissions, on their own machine, because they trusted a name I put in the world. That is the actual liability of shipping something in the open — not that it might get copied, but that it might get impersonated, and the impersonation runs on someone else's hardware, not mine.

Names being clean and guessable cuts both ways here. The whole naming convention across the packages — one plural noun, one repo, one npm scope, no confusing variants — makes a package trivially easy to typosquat or, worse, worth actually compromising at the registry, because there is exactly one canonical @hasna/mementos and everyone depending on mementos depends on that same name. But the same discipline that makes the target legible also makes tampering legible fast: there is no alternate spelling to hide a bad publish behind, no forked-name confusion to slow down the moment someone notices a version that should not exist. Clean naming raises the stakes of an attack and shortens the time it takes to notice one. Both are true at once.

So the reclaim was never just about getting our own version number back. It was about closing the exposure window for people who had never heard of us before that morning and would only ever interact with the incident as a strange postinstall script they didn't ask for. A patch release is a small fix precisely because it has to be: the priority is speed, not elegance. Publish clean, bump the version, make the fix the path of least resistance for anyone still on the compromised release. Nobody auditing a supply-chain incident wants a changelog. They want the next number to be safe.

The part of the response that never shows up in the commit history is the fleet's own exposure. We run agents across several machines that depend on our own packages, some of them updating on a schedule with nobody watching. If the reclaim had shipped clean but the update loop kept running unguarded, our own automation would have cheerfully pulled 0.14.33 back in on the next scheduled bump, vouched for by our own commit history, because a scheduled update trusts the registry by default. Fixing the registry and fixing your own blind trust in the registry are two different jobs, and only one of them shows up in the npm changelog.

Defending the commons did not start with the hijack, even if the hijack is the story people ask about. A staged scan already runs before every commit and every push, looking for exposed credentials, because the easiest way to lose a publish token is to leak it in a diff nobody read carefully before shipping. That habit is preventative, not reactive — it exists so the next hijack does not start with us handing an attacker the key instead of them finding a gap in the registry's own defenses. Reacting well to an incident matters less than not causing the next one yourself.

The instinct after an incident like this is to watch harder. The better instinct is to stop trusting your own watching until it proves itself, which is the piece of this story that rarely gets told next to the hijack itself. Alongside the supply-chain-watch loop that now gates every package update every 30 minutes, there is a separate smoke-test loop — logged under names like LOOP_SMOKE_0150, LOOP_SMOKE_0151 — whose entire job is to fire at an exact minute and reply with an exact canary string. If the canary is late, the scheduler itself is broken, and every other loop's claim to have checked something that morning is worthless. An attack does not just teach you to add a watchdog. It teaches you that a watchdog nobody has verified is barking is indistinguishable from silence.

That is the real lesson underneath defending the commons with automation: automation you cannot verify is not defense, it is a false sense of one. A supply-chain loop that silently stopped running three weeks ago looks, from the outside, exactly like a supply-chain loop that has nothing to report. The only way to tell the difference is to test the thing that runs the tests, on a schedule, with a payload specific enough that a false positive is impossible. Trust nothing you have not tested, including the automation that exists to make you feel safe.

Every security loop now posts exactly one digest per run to a shared channel, whether or not it found anything. Silence is not a valid status for a watch loop any more than it is for a coding agent — a loop that ran and reported nothing is indistinguishable, from the outside, from a loop that never ran at all, which is precisely the failure mode a hijack teaches you to fear most.

The tool that came out of the incident, shield, is itself a decision about what defending the commons means. The obvious move after getting hijacked is to write a private script, keep it to yourself, and quietly check your own packages before anyone else does. We published shield instead — an open scanner that checks any package name and version, not just ours, against known npm and PyPI supply-chain attacks, cross-referenced against real incidents like the compromises that hit axios and litellm. A pre-push hook built on the same logic blocks a push outright if it finds exposed secrets on the way out, before anything reaches a registry at all. The attack that hit us became a tool anyone depending on anyone's packages can run today. That is the actual difference between an open ecosystem and a closed one under attack: a closed shop patches itself and says nothing; an open one turns its own incident into a public defense, because the vulnerability that hit us is generic enough to hit the next person too.

None of this is theoretical vigilance aimed at a threat that might show up someday. The week a GitHub Actions supply-chain attack made real news, we already had a loop watching the disclosure and posting updates as the impact assessment changed, on a product that had nothing to do with the incident that started this discipline. The commons gets attacked because the commons is valuable, continuously, not as a one-time event you respond to and then relax. Treating our own hijack as a closed chapter would have meant building nothing more durable than the reclaim itself.

There is a version of this argument that concludes open source is the exposure and closed source is the safety. I think that gets the mechanism backwards. A closed package can be hijacked exactly the same way — through a compromised maintainer account, a stolen publish token, a dependency three levels down nobody audited — and when it happens, fewer people are positioned to notice, because fewer people are looking at the code at all. The width of the commons is what makes an attack visible fast. Ours got caught and reversed inside a release cycle precisely because enough of the surface was public enough that a version mismatch stood out. Closing the source does not remove the attack surface. It removes the audience capable of catching the attack quickly.

Vigilance does not scale to every 30 minutes forever, and that asymmetry is the actual argument for automation over watchfulness. An attacker needs to succeed exactly once, at a moment nobody happened to be looking. A human watching a registry cannot sustain that cadence indefinitely without missing the one window that matters. A loop can run every half hour without getting bored, without taking a weekend off, without deciding a headline was probably nothing. The commons does not need one person paying closer attention. It needs a schedule that does not depend on anyone's attention at all.

Fixing the registry was the easy half. The fix was not actually finished until every machine in the fleet that depended on the package had pulled the clean version, because a patch published to npm does not retroactively change what is already installed somewhere with a stale lockfile. The same ship discipline that applies to every release applied here with more urgency: commit, push, publish, then update every machine, in that order, checked off rather than assumed. A security fix that stops at the registry and never confirms it reached every consumer is a fix on paper, not a fix in practice.

The commit that closed the incident reads like every other commit in the history: lowercase, terse, no adjectives — reclaim @hasna/mementos npm at 0.14.34 after the 0.14.33 hijack. No account of what it felt like to find a version nobody on our side had published sitting live on the registry. That flatness is not indifference. It is the same discipline that makes any incident survivable: describe exactly what happened and exactly what changed, skip the narrative, let the next person reading the log understand the fix without having to read the drama first.

The reclaim itself needed almost no cleverness — bump the version, publish clean, move on. Everything that actually matters happened after: making sure the automation that changes things could no longer act on an assumption that the registry is always telling the truth, making sure the thing doing the watching could prove it was still awake, and making sure the fix did not stay a private lesson learned inside one company's incident channel. A one-patch-release reclaim is not a war story on its own. It is only worth telling because of what got built in the week after it, on the assumption that it will happen again to someone, possibly us.

The mementos hijack cost a version number and an afternoon. What it could have cost, if the response had stopped at the reclaim, was every stranger who trusted a name they had never had a reason to doubt. That is the actual stakes of publishing anything in the open, and it is not an argument for publishing less. It is the argument for making the automation around what you publish as trustworthy as the code itself. Verified, not assumed. On a schedule that does not wait for you to remember to check.

← Back to the articles

Newsletter

What we shipped, what broke,
and what we learned