New framework evaluates NLP explanation robustness in black-box enterprise systems

A new framework for evaluating the robustness of explanations in enterprise NLP systems has been proposed. This framework uses a leave-one-out occlusion method to assess how stable token-level explanations are under various perturbations. The study found that larger decoder-based LLMs, such as Llama 70B, provide significantly more stable explanations than smaller encoder-based models, with improved stability correlating with model scale. AI

IMPACT Provides a method for selecting more reliable NLP models for enterprise use, especially in compliance-sensitive applications.

RANK_REASON Academic paper proposing a new evaluation framework for NLP explanations.

Read on arXiv cs.CL →

• arXiv
• BERT
• Guilin Zhang
• Llama 70B
• Llama 8B
• Qwen 14B
• Qwen 7B
• RoBERTa

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →