Nginx config blocks AI bots ignoring robots.txt

By PulseAugur Editorial · [1 sources] · 2026-06-06 05:32

A user on Mastodon shared a configuration snippet for the Nginx web server. This code is designed to block AI bots that do not adhere to the "robots.txt" file, provided they identify themselves with a user agent string. The solution aims to help website administrators control which bots can crawl their sites. AI

IMPACT Provides website administrators with a tool to manage AI bot access.

RANK_REASON The cluster describes a technical configuration snippet for a web server, which is a tool for managing website access.

Read on Mastodon — fosstodon.org →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

COVERAGE [1]

Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] · 2026-06-06 05:32

Thanks. If an AI bot doesn't respect "robots.txt", but it doesn't conceal its user agent setting, the following "nginx" code will apparently work as well: if ($

Thanks. If an AI bot doesn't respect "robots.txt", but it doesn't conceal its user agent setting, the following "nginx" code will apparently work as well: if ($http_user_agent ~* (GPTBot|ClaudeBot|put-more-here-ai)) { return 403; } That code goes in the nginx "server" block. #ai …

COVERAGE [1]

Thanks. If an AI bot doesn't respect "robots.txt", but it doesn't conceal its user agent setting, the following "nginx" code will apparently work as well: if ($

RELATED ENTITIES

RELATED TOPICS