A new research paper published on arXiv explores the potential security risks associated with On-Policy Distillation (OPD), a technique used to transfer capabilities and improve safety in large language models. The study reveals that a backdoored teacher model, even with a low poisoning rate of 3%, can propagate malicious behavior to a clean student model with up to 70% attack success rate. The research also highlights that increasing training epochs and using top-k KL divergence can accelerate this backdoor transfer. A proposed mitigation, Lazy Defense, aims to slow down this process by making student updates less aggressive. AI
IMPACT Highlights a critical vulnerability in LLM safety alignment techniques, necessitating new security measures for model distillation.
RANK_REASON Research paper published on arXiv detailing potential security risks in LLM safety techniques. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- DagsHub
- Gotit.pub
- Hugging Face
- IArxiv
- Lazy Defense
- On-Policy Distillation
- ScienceCast
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
