PulseAugur
EN
LIVE 13:40:59

AI assistants rarely read `llms.txt` files, log analysis shows

By PulseAugur Editorial · [1 sources] ·

A review of web server logs from GoodBarber revealed that the `llms.txt` file, intended for AI systems, received only 26 requests from AI assistants out of 1,321 total requests over four months. Of these, only four requests were genuinely from OpenAI's GPTBot, and three of those originated from the AI readiness checker `agentready.md`. The analysis also identified three requests that impersonated AI bots using generic user-agent strings and IP addresses not found in vendor-published ranges, suggesting a compliance checker was testing the site's bot-detection mechanisms. AI

IMPACT Suggests AI assistants are not yet widely or independently consuming structured web data, potentially indicating a gap in current AI agent capabilities.

RANK_REASON Analysis of log data regarding the adoption and usage of a specific file format for AI systems.

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

AI assistants rarely read `llms.txt` files, log analysis shows

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · Pierre- Laurent Medori ·

    llms.txt in the wild: 1,321 requests, and not one AI assistant came looking

    <p>We publish an llms.txt on our main domain and on every language portal. It is exactly what the proposal asks for: a curated markdown index of the site, at the root, written for AI systems to read. I pulled four months of nginx logs to answer one question: who actually reads it…

RELATED ENTITIES

RELATED TOPICS