New research suite MT-InfoSeek evaluates LLMs' ability to ask for missing information

A new research paper introduces MT-InfoSeek, a novel evaluation suite designed to assess how well Large Language Models (LLMs) can identify and request missing information when faced with underspecified user queries. The study found that while LLMs can recognize the need for more data, they consistently underestimate the amount required and often fail to acquire sufficient information to determine a unique answer. Furthermore, the order in which models ask for information can significantly impact accuracy, even if all necessary data is eventually gathered. This research highlights that multi-turn information-seeking capabilities are distinct from answer generation and are not adequately measured by current LLM evaluations. AI

IMPACT Highlights limitations in LLM information-seeking capabilities, suggesting current evaluations may not capture true multi-turn reasoning.

RANK_REASON Research paper introducing a new evaluation suite for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• CatalyzeX
• Connected Papers
• DagsHub
• Gotit.pub
• Hugging Face
• Litmaps
• LLMs
• MT-InfoSeek
• scite Smart Citations

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →