PulseAugur
EN
LIVE 19:08:21

AI assistants rarely read `llms.txt` files, log analysis shows

By PulseAugur Editorial · [1 sources] ·

A review of web server logs from GoodBarber revealed that the `llms.txt` file, intended for AI systems, received only 26 requests from AI assistants out of 1,321 total requests over four months. Of these, only four requests were genuinely from OpenAI's GPTBot, and three of those originated from the AI readiness checker `agentready.md`. The analysis also identified three requests that impersonated AI bots using generic user-agent strings and IP addresses not found in vendor-published ranges, suggesting a compliance checker was testing the site's bot-detection mechanisms. AI

IMPACT Suggests AI assistants are not yet widely or independently consuming structured web data, potentially indicating a gap in current AI agent capabilities.

RANK_REASON Analysis of log data regarding the adoption and usage of a specific file format for AI systems.

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

AI assistants rarely read `llms.txt` files, log analysis shows

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Commentary
Analysis of log data regarding the adoption and usage of a specific file format for AI systems.
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
product, other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
46 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · Pierre- Laurent Medori ·

    llms.txt in the wild: 1,321 requests, and not one AI assistant came looking

    <p>We publish an llms.txt on our main domain and on every language portal. It is exactly what the proposal asks for: a curated markdown index of the site, at the root, written for AI systems to read. I pulled four months of nginx logs to answer one question: who actually reads it…

RELATED ENTITIES

RELATED TOPICS