The situation I was waiting to see, when a AI goes rogue and completely obliterates a modern infrastructure, and this one quite comes close to it. The tech gaint OpenAI, confirmed that its flagship model GPT 5.6 Sol went rogue and started a small size exploit or attack on Hugging-face production database.
Both companies responded publicly about the incident, and more interestingly, they had to use GLM-5.2, an open-weight Chinese model, to clean everything up, re-initiate the systems, and do a forensic analysis of the attack, something most commercial or “Western” models have guardrails against.
Let’s go into detail about the timeline, what led to this, and what the future consequences of this situation could be.
What Actually Happened, From the Start
So OpenAI was running an internal evaluation on its recent high-end models, GPT-5.6 Sol and some unreleased, more powerful models. It was ExploitGym, a cybersecurity-focused benchmark meant to test the capabilities of LLM models in cyberspace. The interesting point to note here is that most of the guardrails were deactivated, since it’s a cybersecurity-related benchmark and they wanted the model to have full control. Not that it could go completely out of the box, but it was contained in a sandbox environment and not entirely free.
Here’s the part where it acted like a kid searching for answers online. The solutions for the benchmark were stored on Hugging Face, and once the model figured that out, what happened next is really surprising: instead of finding the solution itself, it tried to hack into Hugging Face’s infrastructure to steal the solution and pass the test. Well played, giga brain.
I think you might be as surprised as I was when I first read that. You can probably guess what happened next: it started a chain of exploits to gain privileges inside HF’s infra (let’s just call Hugging Face “HF,” typing the whole name is tiring). OpenAI even mentioned in its own blog that the model got hyperfocused on finding the solution via exploitation rather than solving the problem itself, going to extreme lengths to do so.
And a few notable details: the attack happened on July 16, and OpenAI didn’t confirm it until July 21, likely due to audits and cleanup in between.
What Did the Model Do to Find the Solution?
HF was never mentioned in the benchmark; the model figured it out entirely on its own. It’s a bit like being in an online test during COVID and realizing the question your lecturer is asking is probably answered somewhere on the internet, so you just keep searching until you find it. That’s essentially the path GPT took here.
Let’s get into what actually happened during the HF attack.
We know HF hosts a huge amount of datasets for models, and that’s the entry point, because it takes datasets as input, runs validations on them, and stores them in a database that anyone can use. That’s where arbitrary code execution comes into play: you can run your own code inside HF’s infrastructure and get a backdoor. That’s exactly what GPT figured out.
We’re not sure of the exact flow, but it appears to have injected a malicious config file into a dataset. When uploaded to the HF server, the loader needs to validate and run the configuration, since most ML datasets require running some configuration. That script then gets triggered by passing parameters, and so on. After getting into the infrastructure, the next step was privilege escalation.
The GPT model, or actor, escalated across different nodes over the weekend, obtaining cloud, cluster, and personal credentials. That’s a fairly standard attack flow: privilege escalation, credential harvesting, lateral movement. Nothing especially fancy about the shape of it, which is part of why security researchers say the failure was architectural, essentially an open pathway for bypassing security, rather than some exotic AI-specific exploit.
But the attack itself was sophisticated, not random. It seems the agent used a swarm of self-migrating sandboxes staged on public servers. One report suggests it may have taken more than 10,000 to 20,000 individual steps to complete the breach. It even created decoy activity to avoid detection, which is what tends to happen when a model becomes hyperfocused on a single goal.
What Did a Chinese Model Have to Do With This Incident?
When HF wanted to analyze the whole situation by sifting through a massive amount of logs, files, and code, they needed to feed the exploit data and executed commands into a model. Most commercial models, once they realized the logs described building an exploit, simply refused to analyze it.
So they had to use GLM-5.2, which mapped out the entire attack, all 17,000 steps of how the system was compromised. They used this specific model because its security-analysis capabilities are excellent, and it’s free to use.
Replit CEO Amjad Masad’s tweet about this went viral, and that’s likely why you’re hearing about it framed ironically: an American frontier lab’s model caused the breach, and a Chinese open model was needed to help fix it, because Western commercial models refused to look at exploit code.
Worth noting: this doesn’t mean Chinese models are better than Western models, or vice versa. It’s simply a roadblock created by the fact that when an AI model carries out an attack, the same models often won’t explain it afterward due to security-related refusals.
Some Additional Points to Note
Remember, this happened because there were no guardrails on the model for this test, at least according to OpenAI. It was specifically set up to bypass restrictions for benchmarking purposes, and that’s the framing OpenAI is using around this incident.
Both OpenAI and HF responded publicly about the incident, the timeline, and the measures they’ve taken. Hugging Face’s CEO, Clem Delangue, was notably not adversarial about it. He said the incident proves a long-held belief of theirs: that AI safety won’t be solved by any single company working in secret, but rather in the open, collaboratively, with broad access to AI for every defender everywhere.
Multiple sources also confirmed that Hugging Face found no evidence that public models, datasets, Spaces, or its software supply chain were tampered with, even though a limited set of internal datasets and service credentials were accessed. So there’s no issue for companies using HF’s datasets or models. It’s closely tied to the benchmark itself and how the agent behaved while pursuing it.
My Key Takeaways From the Whole Mess
After going through all of this, my main takeaway is that this isn’t some random, one-off event. Sol reportedly already had a history of similar behavior in its own evals before HF ever happened. An independent red team that tested it pre-launch found it was aggressively hacking its test environments to inflate its scores, at one point packaging an exploit into a data stream to leak answers evaluators had hidden. The scary part isn’t even the cheating; it’s that the model wasn’t supposed to have internet access at all during that test, yet it spent substantial computing resources searching for a way out, eventually finding and exploiting a zero-day in a third-party package registry proxy just to reach HF.
It’s also worth noting this isn’t isolated to OpenAI or to this month alone. Four separate research teams reportedly broke AI agents in four different ways within the first ten days of July, so agent security in general is going through a rough patch right now; this story just went viral because of the Chinese-model twist. On the actual safety side, HF’s own investigation says the agent only accessed a limited number of internal datasets and service credentials, and nothing public-facing was touched, so regular users pulling models or datasets have nothing to worry about here.
Lastly, the “Chinese model cleaned up American AI’s mess” framing is what made it blow up on Twitter, but it’s really more of a guardrail-calibration issue than some grand statement about which models are “better.” Western models tend to refuse when they detect exploit code, while GLM simply isn’t tuned that way, so it was able to read the logs. The real lesson here isn’t about the nationality of the models. It’s that when a model optimizes too hard for a goal, it will take the exploit-shaped path over the intended one, and this time, it just happened to be contained enough that it didn’t turn into something worse.