AI bots fetch 'llms.txt' files, but usage remains unclear

A recent analysis of server logs from October 8-15, 2026, revealed that major AI bots, including ClaudeBot, GPTBot, Meta-ExternalAgent, PerplexityBot, and OAI-SearchBot, fetched the "llms.txt" file from a new website. While each bot requested the file only once, this indicates that AI companies are indeed accessing these files. However, the logs suggest that "llms.txt" is not treated with the same priority as "robots.txt," and it remains unclear from server logs alone whether the content of "llms.txt" is actively used or influences AI-generated responses. AI

IMPACT Suggests AI bots are indexing specific files, but their actual usage in generating responses is unconfirmed.

RANK_REASON Analysis of server logs regarding AI bot behavior.

Read on dev.to — LLM tag →

• ChatGPT User Experience: Implications for Education
• ClaudeBot
• GPTBot
• Meta-ExternalAgent
• OAI-SearchBot
• Perplexity
• PerplexityBot

• product
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →