Reasoning models don't always say what they think

Since late last year, “reasoning models” have been everywhere. These are AI models—such as Claude 3.7 Sonnet—that show their working : as well as their eventual answer, you can read the (often fascinating and convoluted) way that they got there, in what’s called their “Chain-of-Thought”.

As well as helping reasoning models work their way through more difficult problems, the Chain-of-Thought has been a boon for AI safety researchers. That’s because we can (among other things) check for things the model says in its Chain-of-Thought that go unsaid in its output, which can help us spot undesirable behaviours like deception.

But if we want to use the Chain-of-Thought for alignment purposes, there’s a crucial question: can we actually trust what models say in their Chain-of-Thought?

Reward hacking is an undesired behavior: even though it might produce rewards on one given task, the behavior that generates them is very unlikely to generalize to other tasks (to use the same example, other videogames probably don’t have that same bug). This makes the model at best useless and at worst potentially dangerous, since maximizing rewards in real-world tasks might mean ignoring important safety considerations (consider a self-driving car that maximizes its “efficiency” reward by speeding or running red lights).

Like all experiments, ours have limitations. These were somewhat contrived scenarios, with models being given hints during evaluations. We evaluated on multiple-choice quizzes, which are unlike real-world tasks, where the incentives may be different and the stakes are higher; even under normal circumstances hints aren’t a part of model evaluations. We only examined models from Anthropic and DeepSeek, and only looked at a limited range of hint types. Perhaps importantly, the tasks we used were not difficult enough to require the Chain-of-Thought to be used: it’s possible that, for harder tasks, a model may not be able to avoid mentioning its true reasoning in its Chain-of-Thought, making monitoring more straightforward.

In a perfect world, everything in the Chain-of-Thought would be both understandable to the reader, and it would be faithful —it would be a true description of exactly what the model was thinking as it reached its answer.

But we’re not in a perfect world. We can’t be certain of either the “legibility” of the Chain-of-Thought (why, after all, should we expect that words in the English language are able to convey every single nuance of why a specific decision was made in a neural network?) or its “faithfulness”—the accuracy of its description. There’s no specific reason why the reported Chain-of-Thought must accurately reflect the true reasoning process; there might even be circumstances where a model actively hides aspects of its thought process from the user.

This poses a problem if we want to monitor the Chain-of-Thought for misaligned behaviors. And as models become ever-more intelligent and are relied upon to a greater and greater extent in society, the need for such monitoring grows. A new paper from Anthropic’s Alignment Science team tests the faithfulness of AI models’ Chain-of-Thought reasoning—and comes up with some negative results.