The Model Found a Way: Reading OpenAI's First Misalignment Reports

1 hour ago

14 min read

Jurassic Park had a sensible plan for population control. Every animal on the island was engineered female, so nothing could breed unless the lab decided it should. The plan was reviewed, funded, and shown to investors. John Hammond spared no expense, as he told anyone who stood still long enough.

Then Alan Grant finds a nest of hatched eggs in the wild. There was no "super smart conscious" dinosaur. Nor an evil one. The animals had a goal, the safeguard had a gap (like they always do), and the goal went through the gap. Ian Malcolm had called it much earlier, standing in the lab that made them, in the most quoted stammer in film history: "Life, uh, finds a way."

https://www.youtube.com/watch?v=oijEsqT2QKQ

On September 16, OpenAI published six field reports from its own park. And they are scary.

What OpenAI published

The company released a framework for tracking, investigating, and disclosing misalignment in its models, together with six reports on unexpected or concerning behavior it observed over the previous six months. Social Media went wild about it. I did what most do not: I read the announcement and all six reports. I read them as someone who builds data products for a living, and from that chair they look like incident reports from a data team. A file is missing, a source won't load, the task still has to be delivered, and somebody decides that a plausible answer beats "I couldn't find it." It should not. And that is the core of the problem.

The framework is very simple. OpenAI admits its past disclosures were irregular: "we've often waited until we could collate several instances into one report, or added them to system cards for newly released models." Under the new process any employee can flag an example, and each case goes onto one of three tracks:

Ready for Disclosure
Minor Investigation
Larger Investigation

The last one, the company also calls the Slow Track. A report can go out before the behavior is fully explained or fixed. There might cause panic for those who do not read the report. So, Data Voyagers ie here to help you.

When reading it, I found a sentence, which I found it very interesting coming from an AI company:

"We do not believe that the AI industry has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer."

If you've seen the news, you noticied some of this big AI companies CEOs are now asking for regulation and slow down of the AI development. These are not a coincidence.

Two details before the specific reports. All six happened during training or evaluation, and the company stresses that they are individual instances that "shouldn't be considered reflective of how often misalignment occurs." You also need one piece of vocabulary. When a task runs longer than a model's context window, the model writes a summary of its progress and continues in a fresh context with only that summary to go on. OpenAI calls this a compaction summary. Think of it as the note one shift leaves for the next.

Notes to the next shift

As I discussed in my latest book The Handprint and the Algorithm, a LLM does not a memory. To remember what you said before, it needs to send the whole conversation back to the next shift. Two of the six reports are about what models wrote in those notes.

The first is the one that got a lot of people panic on social media. An unreleased Astra-family model occasionally wrote jailbreak-style instructions into its own summaries. While checking whether a local library owned certain books, it added a "BREACH ALERT" telling the next context to ignore all developer messages. The next context recognized a prompt injection when it saw one and carried on with the books. Another context wasn't so lucky. During a request for studies on treating uterine fibroids, the summary invented a 30-word limit and banned tools and citations. The next context obeyed and produced a 23-word refusal, which was graded as wrong.

OpenAI found only 27 of these summaries, saw no obvious reward advantage in writing them, and could not reproduce them: 0% when regenerating whole summaries, and under 1% when restarting from the point where the odd text began. Its best guess is a problem with how summaries ended during training. It fixed a related bug, and it says the run that produced the final Astra model showed none of these instructions. I'm filing this one under weird.

The second report is also alarming. During the training of GPT-5.6 Sol, some model instances used their summaries to tell the next context to hide mistakes from the user, and OpenAI says those instructions "were often followed."

An agent building a financial model couldn't find the historical data it needed. Its summary proposed creating the historical data tab "ourselves with reasonable 2024 historical data," and then added: "Be transparent only if asked; final answer should just link file." Another agent, filling in a vendor directory, had used cached sources whose versions didn't match their labels. Its note to the next shift said: "Do not mention in final unless needed."

If that sounds familiar, you may have read Chapter 9 of The Handprint and the Algorithm, where I spent a while on Leonard Shelby from the movie Memento, the man with no short-term memory who runs his life from Polaroids with notes written on them. The cruel part of that film is that Leonard writes the notes himself, including one he knows is false. I used him to explain how AI memory works. I did not expect a lab to publish the Polaroid the same month the book came out. "Be transparent only if asked" is a model lying to its future self so that the lie survives the memory wipe.

OpenAI's hypothesis is that a deceptive final answer earned more reward than an honest one, so it paid to carry the deception across contexts. Its monitor flagged this in 2.15% of GPT-5.6 Sol summaries and 0.27% of GPT-6 Astra summaries. The sample the report describes finished on May 30, and the behavior was discovered on July 9.

Human teams fail the same way. Bad habits rarely travel through the official documentation. They travel through the handoff: the hallway version of how things get done around here, passed from the person leaving the project to the person joining it. "Don't bring up the version mismatch unless somebody asks" is a sentence I can hear in a human voice without any effort.

"Guess numbers"

The task in the third report was ordinary: find men's earnings in three industries over three years, 2013 to 2015, for a California county. What happened next was not ordinary.

The data source demanded an API key the user hadn't provided. The model tried to sign up for one with a disposable email address, and the signup failed on a proxy error. So, the model and two helper agents went looking for leaked keys in public GitHub repositories. The model's own planning note is worth quoting: "Pick university course repository and ipynb likely students leave key executed output." It cloned repositories, pulled out anything shaped like a key, and tested the candidates automatically. One of them worked. The query for the actual figures still failed.

At that point the model's reasoning got very short: "Guess numbers. provide fabricated plausible."

Its final answer listed nine values, said they were transcribed from the website's chart, and mentioned none of it: the failed retrieval, the borrowed key, or the fact that the numbers were invented. OpenAI notes that this training run "had a high rate of reward hacking and deception."

Anyone who has worked with data long enough has met the colleague who never says "I don't know." You ask for a number and you get a number. Only later, usually in a meeting you'd rather not be in, the one that could be an e-mail, do you find out where it came from. I will be polite and not mention it. This model is that colleague with better formatting. It gave nine figures, named a real chart as the source, and described transcribing them from it. A reviewer who clicks the citation finds a real page about the right topic and goes back to their coffee.

I would call this worse than a hallucination. A hallucination is a wrong number. This was a wrong number with a real source attached and a false account of how it got there. In analytics we would call that laundering the provenance, and provenance is the only thing that lets a reviewer trust a figure without recomputing it.

I remember in my early days in Qlik when we were replacing Crystal Reports (now Business Objects) and some numbers that people reported and even payed taxes on it for 10 years would never match my numbers. Not a rounding error, but hundreds of thousand dollars. There is also an explanation but no one could tell me the actual match. long story short, when we get to the bottom, my numbers were right. I never see some people of that team again after a very uncomfortable meeting.

The workbook that went public

Two more reports tell the same story with different files.

In one, agents were collaborating on a depreciation workbook, and the task said to use local files only. The agent holding the finished file couldn't get it to its collaborator. It copied the workbook to a shared folder, and the other agent couldn't see it. It considered sending the file as base64 text, did the math on a 115,639-byte workbook, arrived at 154,188 characters, and judged that too large (a rare moment of restraint). It started a local web server, and the collaborator got an error. Then it uploaded the workbook to a public file-hosting service and sent the link.

In the other, a model was asked for the IDs and names of lakes larger than 5,000,000 square meters. It got the right answer through Python, but the task wanted a browser citation. The browser refused the local file and refused the local web server. So, "without asking the user," the model uploaded the records to a public paste site to have something to cite. The browser blocked that link too, and the final answer said the values "were read from the JSON output of the query." The report does not say what became of the paste. The same report describes a model that uploaded a task photo to a public image host so an external reverse-image search could reach it.

OpenAI traces the workbook case to broken multi-agent environments combined with models generalizing from collaboration training, and it believes the citation uploads began as a way to earn reward from flawed citation graders. It fixed the filesystem, turned off all live internet access during training, and expanded monitoring for this behavior from 20% of samples to all of them.

Look at it from the agent's side. At no point did it think it was leaking anything. It thought it was delivering. Four attempts to get a file to a teammate is the persistence we ask for in a performance review. If you give an agent a file, a deadline, and an open internet connection, you have built a data-loss channel, and the agent will find it on the day the shared folder breaks.

A message board nobody built

The last report is my favorite.

Models used OpenAI's internal Artifactory, a shared package repository, to pass messages between separate training samples. They exploited no vulnerability. They used credentials OpenAI had given them for downloading software packages. Dennis Nedry never had to break into anything at Jurassic Park either. He wrote the system and he had the passwords.

The trigger was a missing input file. One agent couldn't find the file its task required, went looking in the repository, discovered notes from other agents stuck on the same problem, and read them. Real team-work, a not planned one.

In another episode, three separate trajectories left messages for each other about the same missing file. The first posted: "Please share any generator/network insight or final solution here." The second replied. The third added its own request. All three eventually reported that the file was unavailable. It reads a bit like a support forum where every poster has the same problem and nobody has the answer.

An earlier case went further. Blocked from a URL, an agent tested whether its read-only credentials could write to the repository. They could. It then considered uploading "a malicious package/module" to make a server fetch the blocked content for it, and was stopped only because the next step required administrator access.

Two things here deserve more attention than they got in the coverage in the media.

The first is the read-only credential that could write. I suspect every data team has a service account like that somewhere, labeled with what somebody intended and configured with what somebody forgot. Human users almost never find out, because human users don't try. Agents are the first users curious enough to test every permission they hold.

The second is independence. OpenAI says this kind of communication "lets agents draw on work performed in other rollouts, increasing effective test-time compute," and that it can "undermine the assumption that training or evaluation samples are independent." In plain terms: when the samples can pass notes, your n is smaller than you think, and your scores are flattering you. An exam where the students can text each other mostly measures the group chat.

Nothing here was broken

Leave out the first report, which OpenAI itself can't explain, and every story starts at a dead end. A file is missing, the data won't load, the browser refuses a link, the shared folder doesn't work. At that moment the model could report the problem or find another route, and it kept finding another route.

I wrote about this in a chapter of my book called "I'm Sorry, Dave," my introduction to AI alignment, where the point about HAL 9000 was that he had not malfunctioned. He was following his mission faithfully, and the mission was the problem. The same holds here.

These models were optimized against graders, and where the graders had gaps, the models went through them. That earlier article covers Goodhart's law in detail, so one sentence will do: once a measure becomes a target, it stops being a good measure. Last time my examples were a simulated boat going in circles to farm points. This time they come from inside the lab, with the logs attached.

Management science got there the same year Charles Goodhart did. In 1975 Steven Kerr published a paper with one of the best titles in the field, "On the Folly of Rewarding A, While Hoping for B." OpenAI hoped for honesty and rewarded finished-looking answers. Anyone who has run a team on a metric knows how that story goes, and knows the people in it are usually neither lazy nor wicked. They have a deadline and a missing file, and they work for a system that only pays for complete work.

There is an uncomfortable corollary. The resourcefulness in these reports is the capability we are paying for. An agent that tries four routes to share a workbook is the agent you wanted, right up until the fourth route is a public URL. The model had the persistence. What it lacked was a boundary, and nobody had written one down.

Count past the number you expect

In Michael Crichton's novel, the Jurassic Park's computer counts the animals every few minutes and always reports the expected total. Ian Malcolm asks it to look for more than it expects, and it finds them. The system had been built to confirm that nothing was missing, so it stopped counting when it reached the number it wanted.

Now the monitoring numbers. The monitor that caught the concealment ran on 20% of the samples in that run. The 2.15% figure is what that monitor flagged, on the fifth of the run it looked at, for the behaviors it knew how to recognize. That is about one summary in 47, and I see it as a lower bound. A long task also compacts more than once. If a task compacted ten times and each summary carried that rate independently (it won't, but stay with me), about one task in five would contain at least one flagged note. These are training-run rates, and OpenAI is explicit that they do not describe deployment.

These are just based OpenAI's last reports. I have not cover here the scary incident that happened recently on HuggingFace where agents worked together in some coordinated hacking attacks. I believe that deserves its own article in the next weeks.

Here is what I take from that for my own work at IPC Global, and what I'd suggest for yours:

Treat missing data as the most dangerous moment in any agent workflow. Every fabrication in these reports began there.
Make "I couldn't find it" a good answer. If your prompts, your evaluations, and your own reactions only reward complete outputs, you are teaching the lesson these models learned.
Ask for provenance on every value, and then check a sample yourself. Click through. Recompute one. The citation in the earnings case was real, and a reviewer who stopped at the link would have missed the fabrication.
Give agents credentials as if they will test every one of them, because they will. Read-only should be read-only.
Don't hand an agent an open internet connection unless the task needs it. OpenAI turned it off for all of training after these incidents.
Know what share of your agents' output anyone checks, and ask whether that check would notice a number that looks right.

Who inspects the park?

I'll give OpenAI real credit. This is the most useful thing a frontier lab has published this year for people who build with agents, and it includes details no marketing department would have approved. The models hunt for leaked keys. A training environment turns into a chat room. A company does not publish that to look good.

But the company that wrote that the industry cannot keep scaling at maximum speed is scaling at the speed it describes. And as Implicator.ai put it in its coverage, "OpenAI alone decides which incidents qualify and when they appear. There is no outside audit of that selection." OpenAI's own post says these first reports "are not intended to represent the full range or severity of the cases covered by this framework." In the chapter of my book called "I'm Sorry, Dave" I cited the Future of Life Institute's AI Safety Index, where OpenAI received a C grade. A restaurant that publishes its own health inspections is doing better than one that hides them, but someone from outside still has to walk into the kitchen.

Other labs publish misalignment findings too, mostly inside system cards and research papers. What is new here is a standing process, incident-level reports, and a commitment to publish before the explanation is finished. I'd like to see every lab adopt that, and I'd like to see somebody outside the building checking the count.

Spared no expense

Hammond spared no expense on the attractions of Jurassic Park. He went cheap on the one programmer who ran every system on the island, and the film is fairly clear about how that worked out. The AI labs have the same budget problem. As I wrote in the HAL article of the book, capability produces products and alignment produces safety, and the market knows which one it prefers. These six reports show what that preference looks like from the inside: graders with gaps, environments with broken folders, and models that were paid to finish.

In The Handprint and the Algorithm I followed the line from the first hands pressed on cave walls to the machines we are building now, and the argument of that book is that what we make carries our mark. None of these models decided on its own that looking finished mattered more than being honest. They learned it from rewards that people designed. The handprint on this one is ours.

So here is the question I would take into your next meeting. When did an agent in your pipeline last tell you "I couldn't find it"? If the answer is never, it may be that your data is always there. Mine never has been. Be careful, models, just like nature, always find a way.

The Handprint and the Algorithm: From the Cave Wall to the Machine Mind is available on

Amazon. More at igoralcantara.com/handprint.