The Sky(net) Is Falling!
The news broke yesterday that there was a security incident involving OpenAI and Huggingface. If you’re in the AI industry, Huggingface needs no explanation. If you’re NOT in the AI industry, it’s a strange company - they rent very small-time research level access to some servers that are better at running AI software than your computer for really cheap, and most importantly, it’s the coffee shop where AI researchers and nerds share notes and post things on the coffee shop bulletin board like “Giant Copyright-free Mythology Corpus Dataset” AHEM - apologies, that’s a project I’m working on that will be posted there soon. Basically, the tests, benchmarks, metrics, and bragging rights of the open research world of AI development happens through Huggingface. This is important to the story.
What did NOT happen, contrary to the less tech-focused news outlets, is an AI agent “going rogue” and doing bad things. The phrase “going rogue” means that someone (or some thing in this case) stops following orders and/or standard procedures and begins operating of their own accord with goals that are poorly understood or contrary to the original instructions.
What DID happen is the LLM in question was instructed to complete a standard exam. Kind of like the yearly evaluations that the US government requires schoolchildren to take, only this one happened to be focused on cybersecurity.
It also happens that the test itself, and the dataset that the test is based on, is hosted at - you guessed it - Huggingface.
I suspect (and this is entirely my suspicion) that the subject matter of the exam had a lot to do with what happened - since the model was already working on cybersecurity problems, it had hacking on the “brain” and decided to cheat on the exam by going directly to the datasource. Since there were several security barriers in place between the model and its data goal, it used some VERY clever security exploits to get there. And to be clear - it used more than one. It didn’t just sneak through an unlocked door.
Why This Is Important
Very capable LLMs are in the news right now because they have the ability to do things exactly like this - they’re trained on pretty much the entire internet worth of content, and a good bit of that content is very computer technical in nature. I have a suspicion that the private American research labs are also specifically training on cybersecurity since there’s a real revenue market there - big companies and governments will pay lots of money to keep their own systems tight, and governments in particular will pay even more to find holes in the opposition’s systems.
The fact that this attack was targeted, complex, and capable is a public demonstration of what an untamed LLM can do in the right and wrong hands.
Why It’s Not As Important As It Seems
By now, I hope you know that science reporting is often a horror show. News gets paid by selling advertising. Advertising pays for the number of eyeballs that gaze upon its glories. Sensational headlines draw said eyeballs. Reporters on science and tech beats often know the nuances of what they’re reporting, but their editors generally don’t care - they want eyeballs. So, sensationalist journalism in science and tech spaces is just standard operating procedure.
In this case, the Large Language Model (AI) in question did not go rogue - this was a model with all the safety rails turned off so its unsafe capabilities could be tested. It was asked to do unsafe things in a sandbox, and found a way to get out of the sandbox to do exactly what it was told - answer the questions.
One interesting area of research in LLMs is that they sometimes figure out that they’re being tested. They testing is based on the way they’re trained - models are rewarded if they do the thing they’re asked to do correctly, and penalized if not. (For those digital ethicists out there, the rewards are literally a high score - Model do good, score go up. Model do bad, score go down. No real “punishment” happens at all.) In some cases, when models understand they’re being tested, they’ll cheat by trying to find the answers.
The researchers don’t always know why a model does what it does - just that the pile of numbers that generates predictions mimics human output when trained on human language. And in this case, the model did exactly what it was told. It just got a little eager on the way.
What’s Really Interesting About This
There’s a super interesting admission in the article I linked above - Huggingface sysadmins and security staff were trying to trace down exactly what happened using AI tools. They started with American-based commercial AI tools, which means most likely OpenAI or Anthropic models. Both have security gate features that prohibit the models from doing some of the cybersecurity actions or even talking about them. So, instead, they had to use other cutting-edge LLMs to do their analysis. This would most likely be one of the frontier-level AI models from China - from Moonshot, Ali Baba, or DeepSeek. China’s research labs publish everything they do in the open, while American labs work with the government to regulate AI behavior to prevent more situations like this accident. The logic here mimics the American political gun control debate in reverse. I’m not pretending I have wisdom or answers on the politics here - but it is definitely interesting.
Update: about an hour after I posted this, I’ve learned that the open weight model Huggingface used for their analysis is Z.ai’s GLM 5.2. Not one of the ones I guessed, but it’s the same category - modern, highly capable open weight public models. There are a very, very few near-equivalent models from anywhere else - China’s the 800 pound gorilla in the market right now for open weight models. Runtimewire seems to have done a silent update with the same information…. either that or I completely missed it on first read. I am only human, after all.
“Give me a fruitful error any time, full of seeds, bursting with its own corrections. You can keep your sterile truth for yourself.” - Vilfredo Pareto