New framework improves equitable system-prompt selection for LLMs

A new framework called Constrained Mixed-Strategy GroupDRO has been developed to improve the equitable selection of system prompts for large language models. This method aims to minimize the worst-case quality loss across various metrics and groups, while keeping the overall quality loss comparable to average-based selection. The framework's effectiveness was demonstrated across five LLMs on medical and consumer-finance benchmarks, showing significant reductions in worst-case quality degradation. AI

IMPACT Enhances LLM response quality by ensuring more equitable performance across diverse question phrasings.

RANK_REASON Academic paper detailing a new framework for LLM system-prompt selection. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• alphaXiv
• Average selection
• CatalyzeX
• Constrained Mixed-Strategy GroupDRO
• DagsHub
• Gotit.pub
• Hugging Face
• large language models
• Overall mean estimation of trace evidence in a two-level normal-normal model
• ScienceCast
• Worst 25% Mean

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →