New LENS protocol evaluates LLM narrative unlearning effectiveness

Researchers have developed a new evaluation protocol called LENS (Level-based Evaluation of Narrative Suppression) to assess the effectiveness of machine unlearning in preventing large language models from reproducing disinformation-aligned narrative frames. The study tested four instruction-tuned models—Lapa LLM, Gemma-12B, Qwen-14B, and TAIDE-Gemma—using two specific narratives related to the Russia-Ukraine conflict and the US-Taiwan relationship. Results indicated that while unlearning could reduce narrative reproduction, it also led to unintended side effects like entity recovery, demonstrating the complexity of narrative unlearning. AI

IMPACT Introduces a new method for evaluating and potentially improving the safety of LLMs against disinformation.

RANK_REASON The cluster contains a research paper detailing a new evaluation protocol for LLM safety. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Gemma-12B
• Lapa LLM
• Large language models
• LENS
• Qwen-14B
• Russia
• TAIDE-Gemma
• Taiwan
• Ukraine
• United States

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →