A new analysis of over 31,000 hourly Large Language Model (LLM) benchmark scores reveals significant variation in model performance. The study found that within-day score fluctuations averaged 2.8 points, while between-day variations were substantially larger, averaging 8.4 points. This suggests that sustained performance changes are more reliably detected by observing daily trends rather than hourly fluctuations, as between-day variation is approximately three times greater than within-day variation. The analysis was conducted using AIStupidLevel, an open-source system developed by the author for continuous LLM benchmarking and drift detection. AI
IMPACT Highlights the need for robust, continuous monitoring of LLM performance to distinguish genuine drift from normal operational variance.
RANK_REASON Analysis of LLM benchmark scores and performance variation. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
