robots.txt

2 tools tagged "robots.txt".

Cloudflare Pages robots.txt Generator

Generate a robots.txt for a production Cloudflare Pages domain (allow + sitemap) or a pages.dev staging subdomain (disallow all).

AI Crawler Policy Generator

Paste your robots.txt, classify AI vs search crawlers, and get a suggested allow/block policy plus an llms.txt skeleton.