All briefings
Optimus Labs · CivilizationsCRITICAL · LIVE INCIDENTS · KEY CUSTODIANS

Live incidents, as of Nov 1, 2026. Wiz has published downstream abuse of keys stolen in these breaches; Railway is still investigating and this page will be updated.

AI Provider Key Custodians Breached One After Another in October

Fireworks, OpenCode, LiteLLM and Railway hold customers' AI provider keys or environment variables, and each has been breached or backdoored. Warp lost agent trace and telemetry data. Wiz counts at least nine such platforms hit in October, including Andromeda, Zeabur, Lightning AI, CommandCode and Northflank, and has watched stolen keys resold within days. Attribution is unconfirmed.

CredentialInference

Full advisory deck

Organizations in this briefing

Fireworks AIFireworks AI
OpenCodeOpenCode
LiteLLMLiteLLM
RailwayRailway
WarpWarp
AAndromeda
ZZeabur
LLightning AI
CCommandCode
NNorthflank
NNVIDIA Brev
AnthropicAnthropic
OpenAIOpenAI
GoogleGoogle

Key takeaways

3 min read
  • Live incidents as of Nov 1, 2026. Four companies that sit between teams and AI model providers had customer secrets exposed or targeted between Oct 6 and Oct 21: Fireworks, OpenCode, LiteLLM and Railway. OpenCode and Railway tie their breaches to distillation. LiteLLM traces its release to a master key leaked in May and never revoked. Fireworks has named no motive. Wiz counts at least nine AI developer platforms that disclosed incidents in October, and saw stolen enterprise keys tested within 30 minutes and resold within a week. Revoke at the provider, not at the custodian.
  • Blast radius: Any team that stored provider keys or environment variables with one of the four: Fireworks evaluation jobs, OpenCode Console workspaces, a LiteLLM proxy or host that ran version 1.107.0, or Railway service variables. LiteLLM receives about 23 million downloads a week, and its proxy keeps every upstream provider key in one database table.
  • CRITICAL · LIVE INCIDENTS · KEY CUSTODIANS

Timeline

  1. LiteLLM master key committed to a public repository

    A config file held the master key to the internal LiteLLM Agent Platform. The commit was force-pushed over, but the key was never revoked (LiteLLM).

  2. Fireworks finds unauthorized use of internal credentials

    Environment variables in evaluation jobs were captured in a limited number of accounts. It published its bulletin on Oct 9 (Fireworks).

  3. Attacker takes LiteLLM owner GitHub token

    The Agent Platform returned an organization owner's GitHub CLI token. Throwaway workflows then posted two maintainers' tokens to the attacker (LiteLLM).

  4. LiteLLM look-alike domain registered

    litellm[.]dev was created Oct 13 and updated Oct 18, per JFrog. The real project uses litellm.ai.

  5. Andromeda discloses an incident

    Calls it part of an "industry-wide attack" (Andromeda).

  6. Zeabur discloses a breach

    Credential access; attacker activity Oct 12 to 14. Zeabur is shutting down (Zeabur).

  7. Lightning AI discloses an incident

    Credential access, per Lightning AI.

  8. OpenCode Console keys exposed and breach disclosed

    Earliest possible exposure 00:30 UTC. 1.4% of Console users, "related to a large scale distillation attack".

  9. Railway attack window opens

    Elevated latency paged on-call; Railway found privilege escalation in its build infrastructure and halted builds (Railway).

  10. LiteLLM release protections overridden

    Branch protection rejected a change to the private release workflow. Five seconds later a privileged developer credential overrode it (LiteLLM).

  11. Malicious litellm 1.107.0 live

    Seven PyPI wheels until 13:23 UTC. Docker tags latest, main-latest, main-stable and v1.107.0 were poisoned from 11:46 to 14:20 UTC.

  12. Railway disclosure and paused deployments

    Railway notified upstream providers to revoke keys and paused all deployments.

  13. Railway confirms attacker out

    Railway says this is when it confirmed the attacker no longer had access to internal systems (Railway).

  14. CommandCode discloses an incident

    Attributed to a distillation attempt (CommandCode).

  15. Northflank discloses an incident

    Posted on the Northflank status page; dates per Wiz.

  16. Wiz links abused keys to the breaches

    Wiz reports two enterprise AWS Bedrock keys stolen in one of these vendor breaches, abused for about 2.5 million calls before rotation (Wiz).

Blast radius
Any team that stored provider keys or environment variables with one of the four: Fireworks evaluation jobs, OpenCode Console workspaces, a LiteLLM proxy or host that ran version 1.107.0, or Railway service variables. LiteLLM receives about 23 million downloads a week, and its proxy keeps every upstream provider key in one database table.

Summary

Live incidents, as of Nov 1, 2026. On Oct 6, Fireworks found that someone had used internal credentials to read environment variables, including API keys, in a limited number of customer accounts. On Oct 17, OpenCode said a breach of its hosted Console may have exposed provider keys for 1.4% of Console users. On the morning of Oct 21, a poisoned LiteLLM release went out through PyPI and Docker with a credential stealer built to take the provider keys a proxy stores. The same day, Railway said someone may have read decrypted customer environment variables and paused all deployments. In its preliminary report, Railway says the attacker was after AI provider keys for distillation. LiteLLM's attack chain starts with a master key committed to a public repository on May 27 and never revoked. No one has published a root cause that links all four. Wiz counts at least nine AI developer platforms that disclosed incidents in October.

Am I affected?

Any single yes means treat this as an incident.

  • Did a host, CI runner or container of yours run litellm 1.107.0, or pull the latest, main-latest or main-stable LiteLLM image, between 11:39 and 14:20 UTC on Oct 21?
  • Do you keep API keys or registry credentials in Railway variables, run Fireworks evaluation jobs with secrets, or save provider keys in OpenCode Console (briefing #13)?
  • Did you store provider keys or environment variables on Andromeda, Zeabur, Lightning AI, CommandCode or Northflank, or use NVIDIA Brev, in October?
  • Does provider billing since Oct 6 show models, volumes or regions you do not recognise, or calls through global cross-region inference profiles you never use?

Not affected if

  • LiteLLM says its private enterprise release and every version before 1.107.0 are safe. Fireworks says no notice means it has not identified your account. Both are the vendors' own claims.

Rotate stored keys at the provider that issued them. Fireworks says removing a stored secret does not invalidate the underlying credential, and the same holds for all four custodians.

Three hosted platforms and one package

Fireworks, OpenCode and Railway are hosted platforms that store customer secrets, and an intruder reached each one. Fireworks says internal credentials were used to capture evaluation-job variables and that it found no access to customer models, datasets or inference traffic. OpenCode's Console was reached through a legacy system being retired. Railway says the attacker escalated privileges in its build infrastructure and pulled secrets in encrypted form, and that a subset of customers had secrets exposed that need rotating.

LiteLLM is different. It is the open source gateway many teams run themselves, and the poisoned release ran inside their own environments. It reads the proxy's provider-key table directly, takes cloud credentials, SSH keys and environment variables, and installs a backdoor that checks in hourly.

How the LiteLLM release was taken

LiteLLM's attack chain starts on May 27, when a config file committed to a public repository held the master key to its internal LiteLLM Agent Platform. The commit was force-pushed over, but the key was never revoked. On Oct 10 the attacker used it to read the Agent Platform, which returned a GitHub CLI token belonging to an organization owner. LiteLLM says that token was not scoped down and skips GitHub's re-authentication for sensitive actions.

With owner access, the attacker pushed throwaway workflows that posted two maintainers' tokens, stored as CI secrets, to the attacker's endpoint, and took tokens for other internal systems, with a focus on upstream inference API keys. Release approval had been relaxed that week from a device-locked account to any two engineers, so one token triggered the release and another approved it; a third deleted the run records. The build ran from a look-alike repository, Berri-AI instead of BerriAI, and the official source tag stayed clean. JFrog notes that reading the source would have missed it.

Hijacked publishing access is a repeat pattern in our own briefings: in arrayref, a compromised maintainer account pushed malicious crates to crates.io.

Why the middleman holds the prize

OpenCode and Railway tie their breaches to distillation, the pulling of a strong model's answers at scale to train a cheaper copy. Anthropic says distillers often use stolen API credentials from legitimate customers. Railway adds that the attacker appeared to have abliterated models and substantial compute. LiteLLM says the attacker's activity focused on upstream inference API tokens. Fireworks has named no motive, and no source links the four incidents to one actor.

Our read, not the vendors': a gateway or platform holds keys for many customers in one place, so one intrusion yields a pile of paid, aged accounts with normal usage history. That is plausible, not proven. Stored tokens as the way in is not new: in Salesloft Drift, attackers stole the OAuth tokens a vendor held for customers and used them against customer CRMs.

Part of a wider October

Wiz counts at least nine AI developer platforms that disclosed incidents in October: Fireworks AI, Andromeda, Zeabur, Lightning AI, OpenCode, Railway, Warp, CommandCode and Northflank. Fireworks, OpenCode and Warp describe an "agent swarm" attack, and OpenCode and CommandCode attribute theirs to distillation. Root causes differ: credential access at Fireworks, Zeabur and Lightning, build infrastructure at Railway, a third-party dependency at Warp, a legacy system at OpenCode. Every public report describes attackers hunting stored environment variables, especially AI access. Zeabur is shutting down.

Warp's CEO said on Oct 26 that an agent swarm, likely starting with a hole in a vendor dependency, reached a limited amount of agent trace and telemetry data, with an upper bound of roughly 1% of users. Warp has not reported provider keys taken, and is working with CrowdStrike and law enforcement.

NVIDIA Brev may be a tenth. International Cyber Digest reports that NVIDIA told users of unauthorised activity from Oct 4 to 20 that may have exposed the SSH keys Brev set up for their VMs. We have not seen NVIDIA's notice, so treat this as reported, not confirmed. Andromeda, Zeabur, Lightning AI, CommandCode and Northflank have published less, and no single vulnerability links them.

The other October incidents

Five more platforms disclosed incidents in October, per Wiz. Details are thin, and the dates are Wiz's.

Andromeda (Oct 9): its incident report calls the breach part of an "industry-wide attack".

Zeabur (Oct 14): its write-up puts the root cause at credential access and names attacker address 23[.]106[.]46[.]239, active Oct 12 to 14. Zeabur is shutting down because of the incident.

Lightning AI (Oct 15): its notice also points to credential access. Wiz notes Lightning was hit earlier in the Mini Shai-Hulud campaign.

CommandCode (Oct 28): its post on X ties the incident to an attempt at distillation, as OpenCode did.

Northflank (Oct 30): disclosed on its status page. No root cause is public yet.

NVIDIA Brev (reported, not confirmed): see above. NVIDIA has not published a notice we can check.

If you stored keys or environment variables on any of these, treat them as exposed and rotate them at the provider that issued them.

What happens to a stolen key

Wiz followed two enterprise AWS Bedrock keys it ties to one of these vendor breaches. Each was tested within 30 minutes of the theft by a single model call from a residential address, in a one-minute sweep across several organizations. A small checker then pinged both keys every day to confirm they still worked.

Four days later, the first key was flooded: about 800,000 Bedrock calls in 30 hours, mostly Opus models. Its policy allowed global cross-region inference profiles, so only about 6% of calls were throttled. The second key was restricted to US profiles. A reseller pool failed for hours until a checker found the working model IDs, then one address made about 1.7 million calls in 13 hours, with most of the rest throttled by regional quotas. Wiz estimates over $100,000 in usage at list price across the two.

Wiz traced checks on both keys to two open resale sites selling discounted frontier-model access, one at an address Zeabur named in its own breach. Wiz cannot tie either site to the original theft. Our read: key scope is a blast-radius control. The restricted key still leaked, but it cost the attacker a day and ran into quotas.

LiteLLM was already hit in March

On Mar 24, versions 1.82.7 and 1.82.8 reached PyPI for about 40 minutes after attackers took LiteLLM's publishing token through a poisoned Trivy scanner (Cloud Security Alliance). CloudSEK later counted more than 2,500 organizations in the captured data, which means potential exposure, not confirmed compromise. JFrog says the October release uses different payloads and infrastructure and that no evidence links it to the March actor. Both went after the same asset: the keys a gateway holds.

How it maps to OWASP

In the Agentic Applications Top 10, ASI04 Agentic Supply Chain Vulnerabilities covers LiteLLM and the hosted custodians, and ASI03 Identity & Privilege Abuse covers the stolen keys. In the LLM Applications Top 10, LLM02 Sensitive Information Disclosure names API keys as protected information, and LLM06 Unbounded Consumption covers what stolen keys are used for, including distillation theft.

The OWASP GenAI Security Project does not endorse any product, service or tool, including Optimus Labs. We cite its lists to classify these incidents.

If you hold other people's keys

Provider keys are crown jewels.

Any vendor that stores a customer's API key for them (a model gateway, an agent platform, an observability or CI tool) is one intrusion away from these same incidents. None of the four vendors has said how stored keys were protected, so none of this is a finding about them. It is the path we would expect a vendor to follow, drawn from OWASP and NIST key-management guidance.

LiteLLM adds a question for any vendor: who can change what you ship? Our read: keep an organization owner's token out of developer tooling, require approval from an account not reachable the same way, and copy workflow logs where an account cannot delete them. LiteLLM's chain also shows that a leaked key which was "cleaned up" but never revoked is still live: practices 6 and 8 apply to your own keys too. Environment variables are a favorite place to look: in the Coder registry hijack, poisoned modules scraped credentials from environment variables, config files and shell history, and no CVE was assigned. The eight practices below cover the stored keys themselves.

  1. Hold less

    1. Store less

    Use short-lived, scoped credentials or delegated access wherever a provider offers them. Many model providers still issue long-lived static keys, so a gateway often has to store them. Where the design allows, pass a key through for one request without saving it.

  2. At rest

    2. Use envelope encryption

    A unique data key per credential, AES-256-GCM, with the data key wrapped by a master key in a KMS or HSM. OWASP says not to keep encryption keys next to the secrets they protect unless those keys are themselves encrypted.

    The master key wraps the data key, and the data key seals the customer's provider key.
  3. 3. Separate customers

    One master key per customer, and bind each ciphertext to its customer and record so a copied blob will not decrypt in another row. Deleting a customer's key then makes their stored credentials unrecoverable.

  4. 4. Make stored keys write-only

    No reveal button and no export. Show the last four characters and the date it was set. Never log a key: strip authorization headers from logs and error reports.

  5. In use

    5. Decrypt per use, and watch it

    Decrypt only for the outbound call, wipe the plaintext from memory, and keep any cache short. Log every decrypt and alert on spikes, so a bulk pull stands out.

  6. Beyond the vault

    6. Retire systems completely

    Before a system is decommissioned, move or delete the credentials it holds and confirm they are gone. Any system still holding live keys is still in scope.

  7. 7. Give customers the brakes

    Per-key spend caps that stop traffic, model and IP allowlists, last-used time and source for every key, and plain revoke steps for each provider.

  8. 8. Plan the breach before it happens

    Keep a per-customer list of stored keys by last four characters, agree a revocation contact with each provider ahead of time, and tell customers in hours, not days. Anthropic and Google acted on Oct 17, within about a day of the first alert, and a pre-agreed route makes that routine.

What we still don't know

As of Nov 1, 2026, unpublished: how many Railway accounts were affected and the specific exploits used against Railway, which it says it will disclose later, how the Fireworks intruder got the internal credentials, whether the four (or the nine Wiz counts) share an actor, who ran the floods Wiz observed, and how many of the stolen keys are in use today.

Remediation

Contain now · 0-24h
  • Rotate keys at the issuing provider (Anthropic, OpenAI, Google and others), not only in the custodian's dashboard, and check billing since Oct 6.
  • LiteLLM: isolate any host or container that ran 1.107.0. Do not start with pip, because any Python start re-runs the startup hooks. Delete cook.pth, cook.py and sitecustomize.py, rebuild from verified artifacts, and rotate everything the host could reach: proxy provider keys, master and virtual keys, DATABASE_URL, the IAM role and every environment secret.
  • Railway and Fireworks: follow their notices. Railway recommends that every customer who signed up before Oct 21 rotate keys stored on Railway, including environment variables and registry credentials. Fireworks says to rotate the third-party credentials named in your notice.
Investigate · 24-72h
  • Block and search for www[.]litellm[.]dev and the GitHub dead drop in the indicators below. The marker /tmp/.pc_once_72129118c59f5636c0bb proves exfiltration if present, but its absence proves nothing.
  • Check cloud audit logs for the instance role being used from outside your infrastructure after install time. LiteLLM suggests starting with CloudTrail GetCallerIdentity.
  • AWS Bedrock: search CloudTrail for InvokeModel from unfamiliar addresses, Bedrock or EC2 quota listing, and PutAccountDataRetention calls. Wiz saw all three on stolen keys.
Harden the agent surface
  • Pin gateways and base images by digest. Do not pull a mutable tag such as latest, and verify the exact wheel, not only the source tag.
  • Keep provider keys in a secrets manager you control, one key per tool, with a hard spend cap that stops traffic. On Bedrock, deny global cross-region inference profiles unless you need them.
  • Ask every vendor that holds your keys how they are encrypted, how releases are approved, and how fast you will be told about a breach.

Indicators

  • Malicious package: litellm 1.107.0 (PyPI, all seven wheels). Last safe version 1.106.2.
  • Docker tags on Oct 21: v1.107.0, latest, main-latest, main-stable (digest sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e)
  • Files in site-packages: cook.pth, cook.py, sitecustomize.py
  • Exfiltration: hxxp://www[.]litellm[.]dev/healthcheck. Dead drop: raw[.]githubusercontent[.]com/letters9813/pythonclient-test/main/domain.txt
  • Key-checking addresses reported by Wiz (not tied directly to the original breaches): 23[.]106[.]46[.]239, 38[.]246[.]114[.]118, 104[.]243[.]32[.]50, 31[.]57[.]201[.]201
  • Lock file /tmp/remote_code_loader.lock. Look-alike source in build provenance: Berri-AI/litellm (legitimate: BerriAI/litellm)

Sources

Sourcing & confidence

Confirmed by the vendor

Assessed by Civilizations

  • That the four incidents form one pattern aimed at provider keys is our read; Wiz also groups them, with five more, as one month of attacks. OpenCode and Railway name distillation. No source links the four to one actor.
  • Counts of exposed accounts, keys and organizations are the vendors' or researchers' figures and are not independently verified. Railway is still investigating.

Related briefings

Full advisory

Download the briefing PDF

The complete advisory deck for this incident, with the attack chain diagram, indicators, and remediation bands in one file.

Optimus Labs · Civilizations

Get the next briefing first

Threat research, disclosures, and practical tips on enterprise Agentic AI attack surface management, directly in you or your agent's inbox.

Subscribe