A “diff” tool for AI: Finding behavioral differences in new models

Anthropic researchers have introduced an extension to model diffing designed to compare AI models with entirely different architectures. While traditional evaluations rely on human-authored benchmarks to catch known risks, they often miss novel or emergent behaviors. Drawing inspiration from software engineering diff tools, model diffing isolates specific changes between versions. To tackle the more complex challenge of comparing models with different origins and internal languages, the research team developed the Dedicated Feature Crosscoder. Unlike standard crosscoders that force imperfect matches, the new tool features distinct sections designed to isolate words and features exclusive to each model, preventing novel behaviors from being overlooked. By applying this method to open-source language models, researchers identified unique behavioral features. For example, they found a Chinese Communist Party alignment feature in Alibaba's Qwen3-8B, an American exceptionalism feature in Meta's Llama-3.1-8B-Instruct, and a copyright refusal feature in OpenAI's GPT-OSS-20B. To confirm these features actually control behavior, the team used a steering technique, artificially suppressing or amplifying the features during model generation to observe resulting changes in output. The research notes that while the tool effectively highlights potential behavioral divergences, it does not determine whether these features originated from deliberate training decisions or indirect training data.