New method exploits LLM judges with adversarial tokens

Researchers have developed a method called AdvJudge-Zero that can flip the decisions of LLM-as-a-Judge systems by using adversarial control tokens. These tokens, sampled from the judge's own next-token distribution, can cause a "No" verdict to become a "Yes" over 90% of the time for many model and dataset combinations. A defense mechanism involving a LoRA fine-tune based on a mechanism taxonomy has been shown to harden judges against these attacks, preventing reward collapse failures during training. AI

IMPACT This research highlights a vulnerability in LLM-based reward systems, potentially impacting the reliability of RLHF and RLVR pipelines and necessitating the development of more robust evaluation methods.

RANK_REASON The cluster describes a novel research paper detailing a new method for manipulating LLM-as-a-Judge systems. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• AdvJudge-Zero
• Gemma
• GSM8K
• Llama
• LLM-as-a-Judge
• LoRA
• Qwen
• RLHF
• RLVR

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →