Prompt ensembling significantly outperforms self-consistency in LLM evaluations

A study comparing prompt ensembling and self-consistency methods for LLMs found that prompt ensembling significantly outperforms self-consistency. Using five different prompts with a plurality vote achieved a 21/24 accuracy on a set of arithmetic questions, whereas self-consistency with five samples from a single prompt scored 15/24. The research highlights that prompt ensembling is effective when prompts fail on different questions, while self-consistency, which varies sampling noise, offers diminishing returns beyond a certain point and does not introduce diverse perspectives. AI

IMPACT Prompt ensembling offers a more effective strategy for improving LLM accuracy than simple sampling variation.

RANK_REASON The item describes a research finding about LLM evaluation techniques. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• prompt ensembling
• Self-Consistency In Llms

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →