AI crawlers are checking for llms.txt, but prefer Markdown copies

A new study by the AI Visibility Census tracked server logs for three days to see if AI crawlers were accessing a proposed file format called llms.txt. The logs revealed that Googlebot, GoogleOther, and ClaudeBot made requests for llms.txt or a related llms-full.txt file, while Bingbot and Applebot did not. However, these same crawlers fetched Markdown copies of the site's pages significantly more often, suggesting that while some AI bots are checking for llms.txt, they may not be using it, and clean Markdown versions of content are a more substantial draw. AI

IMPACT Suggests AI crawlers are exploring new methods for site mapping, though Markdown content remains a more significant draw.

RANK_REASON Analysis of server logs regarding AI crawler behavior and a proposed file format. [lever_c_demoted from research: ic=1 ai=0.7]

Read on dev.to — LLM tag →

• AI Visibility Census
• Applebot
• Bingbot
• ClaudeBot
• Googlebot
• GoogleOther
• GPTBot
• llms-full.txt
• llms.txt
• Perplexity

• product
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →