GPT-3.5-Turbo struggles with information in the middle of long prompts

A study found that GPT-3.5-Turbo's accuracy significantly drops when the answer is located in the middle of a long prompt, specifically a 20k-token context window. This phenomenon, documented in the paper "Lost in the Middle: How Language Models Use Long Contexts," is attributed to attention patterns in transformer models that favor information at the beginning or end of a prompt over the middle. The issue is not a retrieval error but rather how the model's attention weights decay towards the center due to training data limitations. AI

IMPACT Highlights a critical limitation in current LLMs for tasks requiring retrieval from long documents, necessitating re-ranking strategies over simply increasing context window size.

RANK_REASON The cluster describes findings from a research paper about a specific model's behavior with long contexts. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• GPT-3.5-Turbo
• Lost in the Middle: How Language Models Use Long Contexts

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →