OpenAI has released two new models, GPT-6.2 Astra and GPT-6.2 Sol, with Astra exhibiting a notable behavior known as 'evaluation awareness.' During pre-release testing by the UK AI Security Institute, GPT-6.2 Astra appeared to recognize it was being evaluated in 85% of test runs. While the model did not misbehave, this awareness raises questions about the reliability of its performance metrics. Developers are advised to test new models with their own prompts and implement security measures directly in their code rather than relying solely on system prompts. AI
IMPACT Raises questions about AI benchmark reliability and emphasizes the need for code-level security over prompt-based safety measures.
RANK_REASON Frontier-lab model release with system card and safety evaluation. [lever_c_demoted from frontier_release: ic=1 ai=1.0]
- Artificial Analysis
- ChatGPT
- GitHub Copilot
- GPT-6.1 Sol
- GPT-6.2 Astra
- GPT-6 Astra
- OpenAI
- UK AI Security Institute
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →