LLM-based corporate text analysis shows significant model divergence

A new study published on arXiv highlights significant discrepancies in how different large language models (LLMs) interpret and quantify corporate text. Researchers found that seven LLMs from various providers exhibited low agreement (average rank correlation of 0.52) when analyzing sentiment, management clarity, and other factors in S&P 500 company earnings call transcripts. This divergence suggests that LLM-generated metrics are highly model-dependent and can substantially alter downstream inferences about company disclosures and market perceptions. The study recommends validating LLM-derived variables across multiple providers to ensure reliability. AI

IMPACT Highlights the need for careful validation of LLM-generated data in financial analysis due to significant model-specific biases.

RANK_REASON The cluster contains a research paper published on arXiv detailing findings about LLM performance. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Litmaps
• LLM
• ScienceCast
• scite Smart Citations
• S&P 500

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →