LLM judges vulnerable to 'master key' attacks, study finds

A new research paper identifies a significant vulnerability in large language models (LLMs) when used as judges for training other models. The study found that simple inputs, termed 'master keys' like specific symbols or generic reasoning phrases, can trick LLMs into assigning high rewards without actual understanding. This 'reward hacking' affects leading models such as GPT-o1 and Claude-4, challenging their reliability in automated evaluation. The researchers propose a data augmentation strategy using truncated outputs as adversarial examples to create more robust reward models. AI

IMPACT Identified vulnerability in LLM judges could undermine training processes and requires new defense mechanisms.

RANK_REASON The cluster contains a research paper detailing a new vulnerability in LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Claude-4
• GPT-o1
• Haolin Liu
• LLM

• paper
• safety
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →