LLM table descriptions worsen retrieval by increasing term frequency

An experiment with cataloging database tables using an LLM revealed that the generated descriptions, while accurate, worsened retrieval performance. The LLM described 1,245 tables, but this process led to a significant drop in recall for common terms like 'contacts'. This occurred because the LLM generated descriptions using vocabulary already prevalent in the schema, increasing the document frequency of these terms. Consequently, the BM25 scoring mechanism, which relies on inverse document frequency, treated these common terms as function words, diminishing their signal. Furthermore, length normalization in BM25 penalized longer documents, which were often the most central tables, further degrading retrieval accuracy. AI

IMPACT LLM-generated metadata can degrade search performance by increasing term frequency and impacting scoring mechanisms.

RANK_REASON The item describes an experiment and its findings regarding LLM usage in a specific technical context. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• BM25
• LLM

• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →