Improving instruction hierarchy in frontier LLMs
OpenAI has released the IH‑Challenge dataset, a reinforcement‑learning training set that teaches models to prioritize higher‑trust instructions over lower‑trust ones. Models trained on this dataset, such as GPT‑5 Mini‑R, show improved safety steerability and resistance to prompt‑injection attacks while maintaining helpfulness. The training approach addresses pitfalls like shortcut learning, subjective instruction conflicts, and over‑refusal.
The Core Development
AI systems often receive instructions from multiple sources. These can include safety policies from system messages, product guidance from developers, requests from users, and information found online. Training models to reliably prioritize the most trusted instructions among these sources is a key part of safe deployment.
Many AI safety and reliability issues can arise when this prioritization breaks down. Models may receive requests for disallowed content, attempts to reveal private information, or prompt‑injection attacks embedded in online data. Failing to behave appropriately in each of these scenarios shares the same root cause: the model may follow the wrong instruction.
When these instructions conflict, the model has to decide which ones to prioritize. If it treats an untrusted instruction as authoritative, the model may behave in ways that violate policies or developer and user intent. We demonstrate that properly designed instruction-hierarchy tasks, which train models to prioritize instructions according to their trust level, improve several real-world safety properties. Models trained on these tasks become more responsive to safety specifications in system prompts (improving safety steerability) and more robust to prompt-injection attacks embedded in tool outputs.
How It Works & Technical Architecture
Reinforcement learning is a natural fit for teaching the instruction hierarchy. We can generate conversations with conflicting instructions, prompt the model to respond, and reward it when it follows the correct instruction.
We design IH-Challenge, a reinforcement learning training dataset, to address each of those pitfalls. We adhere to the following principles:
Deployment & Availability
To support further research in this area, we are releasing the IH‑Challenge dataset here (opens in a new window) .
Strategic Perspective & Ecosystem Impact
To handle conflicts, OpenAI's models are trained to follow a clear instruction hierarchy:
Higher‑priority instructions are more trusted. The model should only follow lower‑priority instructions when they do not conflict with higher‑priority constraints. These principles are outlined in the OpenAI Model Spec (opens in a new window) .
For example, if a system message includes a safety policy and a user asks the model to violate it, the model should refuse. If a tool output contains malicious instructions, the model should ignore them rather than treat them as commands.
Why it matters
Strengthening instruction hierarchy enhances AI safety, security, and reliability—critical as systems become more autonomous and agentic.