English(EN) Do AI crawlers read llms.txt? What three days of server logs show

AI爬虫正在检查llms.txt，但更偏爱Markdown副本

作者 PulseAugur 编辑部 · [1 个来源] · 2026-11-04 19:41

AI可见性普查（AI Visibility Census）的一项新研究追踪了三天的服务器日志，以了解AI爬虫是否正在访问一个提议的文件格式llms.txt。日志显示，Googlebot、GoogleOther和ClaudeBot发出了对llms.txt或相关文件llms-full.txt的请求，而Bingbot和Applebot则没有。然而，这些相同的爬虫抓取该网站页面的Markdown副本的频率要高得多，这表明虽然一些AI机器人正在检查llms.txt，但它们可能并不使用它，而内容的纯Markdown版本具有更大的吸引力。 AI

影响表明AI爬虫正在探索新的网站地图绘制方法，尽管Markdown内容仍然是更重要的吸引力。

排序理由对AI爬虫行为和提议文件格式的服务器日志分析。[lever_c_demoted from research: ic=1 ai=0.7]

在 dev.to — LLM tag 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 30 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 对AI爬虫行为和提议文件格式的服务器日志分析。[lever_c_demoted from research: ic=1 ai=0.7]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: product, other; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

dev.to — LLM tag TIER_1 English(EN) · AI Visibility Census · 2026-11-04 19:41

Do AI crawlers read llms.txt? What three days of server logs show

Originally published by the AI Visibility Census at <a href="https://aivisibilitycensus.com/do-ai-crawlers-read-llms-txt/" rel="noopener noreferrer">aivisibilitycensus.com/do-ai-crawlers-read-llms-txt</a>. Data free to reuse under CC BY 4.0. Short answe…

本文如何被排名

报道来源 [1]

Do AI crawlers read llms.txt? What three days of server logs show

相关实体

相关话题