PulseAugur
EN
LIVE 22:36:58

AI crawlers ignore proposed llms.txt file, favor robots.txt and sitemaps

By PulseAugur Editorial · [1 sources] ·

A test conducted over 30 days revealed that AI crawlers, as recognized by Cloudflare, did not access the proposed `/llms.txt` or `/llms-full.txt` files on a website. During the same period, these crawlers frequently accessed standard files like `robots.txt` and sitemaps. The `/llms.txt` file, intended to provide AI tools with a quick understanding of a website's content, was found to be unread by AI bots, suggesting that current AI crawlers prioritize traditional methods for site discovery and information gathering. AI

IMPACT Suggests that AI crawlers currently prioritize traditional methods like robots.txt and sitemaps over newer proposals like llms.txt for site understanding.

RANK_REASON The item discusses the effectiveness of a proposed method for AI crawlers, based on a personal experiment and observation, rather than a formal release or research paper.

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

AI crawlers ignore proposed llms.txt file, favor robots.txt and sitemaps

How we ranked this

Signal score
2 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Commentary
The item discusses the effectiveness of a proposed method for AI crawlers, based on a personal experiment and observation, rather than a formal release or research paper.
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Same-day
Cluster formed today. Ranking reflects the current source set at time of score.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · Yap Pat Yih ·

    Does llms.txt work? In 30 days, no AI crawler read mine

    <p><em>First published on <a href="https://yappatyih.com/writing/does-llms-txt-work/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>llms.txt is a plain text file at the root of a website that names the site, sums it up in a line and lists its main page…

RELATED ENTITIES

RELATED TOPICS