PulseAugur
EN
LIVE 10:51:08

Meta AI crawlers drain resources of small community websites

By PulseAugur Editorial · [1 sources] ·

Meta's AI web crawlers are disproportionately impacting small, volunteer-run community websites by making an excessive number of requests, leading to increased hosting costs for the site owners. The author, who runs the LGBT History Project, found that a Meta crawler made over 230,000 requests in a day and a half, consuming nearly half of the site's bandwidth and costing them money. This behavior stems from the AI's indiscriminate crawling of all URLs, including low-value duplicates and machine-generated pages like edit histories and diffs, which a more sophisticated crawler would ignore. AI

IMPACT Highlights the externalized costs of AI development on small online communities and individuals.

RANK_REASON The item is a personal account and opinion piece about the impact of AI crawling on small websites, rather than a direct announcement or research finding.

Read on Towards AI →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

Meta AI crawlers drain resources of small community websites

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Commentary
The item is a personal account and opinion piece about the impact of AI crawling on small websites, rather than a direct announcement or research finding.
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
Standard
On-topic for AI-industry coverage; kept in the public index.
Story freshness
47 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [1]

  1. Towards AI TIER_1 English(EN) · Jonathan Harbourne ·

    Meta’s AI keeps scraping my community website, and I get charged £25 a month for the privilege

    <h4><strong>How the AI industry has externalised its costs onto the community websites that individuals build and pay for out of their own pockets — and what the numbers actually look like when it happens to you.</strong></h4><figure><img alt="A giant vacuum cleaner hoovering up …

RELATED ENTITIES

RELATED TOPICS