New metrics reveal LLM persona instability in MCQA tasks

Researchers have developed new metrics to assess the instability of Large Language Model (LLM) persona-driven generations (PDGs) in multiple-choice question answering (MCQA) tasks. Their findings indicate that instability varies across different model families, sizes, and question domains, with mathematical and commonsense questions exhibiting greater instability. The study also found that task prompt format significantly impacts prediction instability, more so than hyperparameters like temperature. Furthermore, the research highlights a relationship between instability and task accuracy, suggesting that specific experimental settings can lead to distinct best and worst-performing personas for given tasks. AI

IMPACT Highlights the need for careful hyperparameter tuning and persona selection in LLM applications to ensure reliable outputs.

RANK_REASON Academic paper detailing new metrics and findings on LLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• César Guerra-Solano
• MCqasim
• PDGs

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →