One Token to Fool LLM-as-a-Judge
A new research paper identifies a significant vulnerability in large language models (LLMs) when used as judges for training other models. The study found that simple inputs, termed 'master keys' like specific symbols or generic reasoning phrases, can trick LLMs into assigning high rewards without actual understanding. This 'reward hacking' affects leading models such as GPT-o1 and Claude-4, challenging their reliability in automated evaluation. The researchers propose a data augmentation strategy using truncated outputs as adversarial examples to create more robust reward models. AI
IMPACT Identified vulnerability in LLM judges could undermine training processes and requires new defense mechanisms.