LLM processing order significantly impacts multi-instance task accuracy

A new arXiv paper investigates how the order of input instances affects the performance of large language models (LLMs) in multi-instance processing (MIP) tasks. Researchers found that LLMs exhibit significant sensitivity to instance order, with accuracy varying by up to 90 percentage points depending on the arrangement of inputs. The study suggests that models tend to be more accurate when majority-label instances appear earlier in the sequence, and analysis of reasoning traces and attribution scores indicates uneven emphasis across input positions. AI

IMPACT Highlights a potential bias in LLM aggregation tasks, suggesting a need for careful input ordering or model adjustments for reliable multi-instance processing.

RANK_REASON Academic paper on LLM processing behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Hugging Face
• large language models
• multi-instance processing

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →