New metric reveals LLM unlearning methods fail to fully forget sensitive data

A new research paper introduces \"Leak@k\", a metric designed to evaluate the effectiveness of unlearning methods in large language models (LLMs). The study found that most current unlearning techniques fail to completely remove sensitive information, as it can still be retrieved through probabilistic decoding. To address this, the paper proposes a new algorithm called \"RULE\" (Robust Unlearning under LEak@k metric) which demonstrates improved performance in preventing information leakage on benchmark datasets. AI

IMPACT Current LLM unlearning methods are insufficient for robust data removal, necessitating new techniques like RULE to ensure privacy and compliance.

RANK_REASON Academic paper introducing a new metric and algorithm for evaluating LLM unlearning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Hadi Reisizadeh
• Leak@k
• LLMs
• MUSE
• TOFU
• WMDP

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →