Meta's AI web crawlers are disproportionately impacting small, volunteer-run community websites by making an excessive number of requests, leading to increased hosting costs for the site owners. The author, who runs the LGBT History Project, found that a Meta crawler made over 230,000 requests in a day and a half, consuming nearly half of the site's bandwidth and costing them money. This behavior stems from the AI's indiscriminate crawling of all URLs, including low-value duplicates and machine-generated pages like edit histories and diffs, which a more sophisticated crawler would ignore. AI
IMPACT Highlights the externalized costs of AI development on small online communities and individuals.
RANK_REASON The item is a personal account and opinion piece about the impact of AI crawling on small websites, rather than a direct announcement or research finding.
- British Library
- LGBT History Project
- MediaWiki
- Meta
- Point of Light
- Wikipedia
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
