PulseAugur
EN
LIVE 20:07:43

Perplexity open-sources WANDR benchmark for AI agent research capabilities

Perplexity has open-sourced WANDR, an internal benchmark designed to evaluate the deep and wide research capabilities of AI agents. WANDR consists of 500 research tasks, requiring over 170,000 source-backed records across three difficulty tiers. Instead of relying on a gold solution, WANDR re-fetches cited pages to verify claims against underlying evidence, allowing for time-varying facts and providing a dense, interpretable evaluation signal. AI

IMPACT Provides a new, evidence-based evaluation framework for AI agents, potentially driving improvements in factual verification and research capabilities.

RANK_REASON Open-sourcing of an internal benchmark for AI agent evaluation.

Read on X — Perplexity →

AI-generated summary · Google Gemini · from 7 sources. How we write summaries →

Perplexity open-sources WANDR benchmark for AI agent research capabilities

COVERAGE [7]

  1. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Soft scores give partial credit. Hard scores require every component for a member to be complete and correct.

    Soft scores give partial credit. Hard scores require every component for a member to be complete and correct. The benchmark tasks and evaluation harness are available at: https://t.co/o2HX0a4RTQ

  2. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Instead of grading against a gold solution, WANDR re-fetches every cited page and checks each claim against the underlying evidence.

    Instead of grading against a gold solution, WANDR re-fetches every cited page and checks each claim against the underlying evidence. This allows certain tasks to contain time-varying facts. https://t.co/2sDRQrSoSx

  3. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    WANDR is constructed from de-identified production use cases covering people’s day-to-day research tasks like competitive research, due diligence, literature re

    WANDR is constructed from de-identified production use cases covering people’s day-to-day research tasks like competitive research, due diligence, literature review, market analysis, product comparison, talent sourcing, and more. https://t.co/y6dClkm2a6

  4. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    WANDR tests how well agents discover large sets of entities and verify specific facts about each one. It provides a dense, interpretable eval signal that reveal

    WANDR tests how well agents discover large sets of entities and verify specific facts about each one. It provides a dense, interpretable eval signal that reveals whether an agent fails, and where. The pipeline also doubles as a semi-automated factory for training data. https://t…

  5. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    WANDR represents these requirements as hierarchical, independently verifiable records.

    WANDR represents these requirements as hierarchical, independently verifiable records. It consists of 500 research tasks that require 170,495 source-backed records across three tiers of difficulty. https://t.co/OX5e1zuGbe

  6. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Wide-and-deep research requires two capabilities.

    Wide-and-deep research requires two capabilities. 1) Agents must search broadly enough to find all qualifying entities 2) Agents must investigate deep enough to support every claim with evidence. https://t.co/l9GE3z3FvW

  7. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    We’re open sourcing WANDR.

    We’re open sourcing WANDR. WANDR is an internal benchmark we built and used for building deep and wide research capabilities inside Perplexity Computer. https://t.co/gp2BWjFK4d