PulseAugur
EN
LIVE 17:40:24

More capable LLMs make worse forecasts on specific risk-heavy tasks

By PulseAugur Editorial · [2 sources] ·

A new research paper introduces ForecastBench-Sim (FBSim), a benchmark designed to evaluate language models on forecasting tasks with superlinear growth and regime change risks. The study found that more capable language models, including Llama-3.1, tend to produce worse distributional forecasts on these specific types of problems. This inverse scaling effect, where increased capability leads to decreased accuracy in certain scenarios, was observed across simulated epidemics and real-world data from finance and public health. AI

IMPACT Highlights a potential limitation in LLM forecasting capabilities, suggesting current evaluation metrics may mask performance issues in high-risk scenarios.

RANK_REASON The cluster contains a new academic paper detailing a novel benchmark and findings about LLM performance.

Read on arXiv cs.AI →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

More capable LLMs make worse forecasts on specific risk-heavy tasks

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
The cluster contains a new academic paper detailing a novel benchmark and findings about LLM performance.
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, safety
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
128 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [2]

  1. arXiv cs.AI TIER_1 English(EN) · Nick Merrill, Jaeho Lee, Ezra Karger ·

    Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most

    arXiv:2605.22672v2 Announce Type: replace Abstract: We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance and epidemiology. On these tasks, more capable mod…

  2. arXiv cs.AI TIER_1 English(EN) · Ezra Karger ·

    Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most

    We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance and epidemiology. On these tasks, more capable models produce worse distributional forecasts. The patt…

RELATED ENTITIES

RELATED TOPICS