Researchers have developed a novel approach to mitigate over-refusal in large language models (LLMs) by reframing safety training. Instead of solely preventing harmful reasoning, the new method encourages models to explore potentially unsafe thoughts as a means to better distinguish between harmful and harmless prompts. This adversarial optimization technique trains a single model to play both a reasoning role that explores unsafe strategies and an answering role that ensures a safe final output. The resulting model, SEAR, demonstrates improved safety while reducing unnecessary refusals on benign queries and enhancing defense against reasoning manipulation attacks. AI
IMPACT This approach could lead to LLMs that are both safer and more helpful by reducing over-refusal on benign prompts.
RANK_REASON The cluster contains a research paper detailing a new method for training LLMs.
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
