Why do I need three separate mechanisms just to hide one subdomain?

Each one covers a different gap: robots.txt asks well-behaved crawlers not to crawl at all, but doesn't stop a page from being indexed if it's linked to from elsewhere; X-Robots-Tag: noindex explicitly tells crawlers not to index a page even if they do crawl it; and the forced redirect stops human visitors and any crawler that ignores the first two from ever seeing staging content in the first place, sending everyone straight to production instead.

What does the trailing "!" on the _redirects rule do?

Cloudflare Pages normally serves a real static file at a matching path instead of applying a redirect rule to it. The "!" forces the redirect rule to take precedence over any static asset, ensuring the staging domain always redirects, even for paths that happen to have a matching file.

Will this break my ability to actually use the staging site for testing?

Yes, by design, once deployed, every request to the staging domain gets redirected to production, so it's no longer usable as a separate preview. If you need a working staging environment, use robots.txt and X-Robots-Tag alone (without the forced redirect) instead, or restrict access with Cloudflare Access rather than a public redirect.

Do I deploy this to the production domain or the staging domain?

All three files go with the staging/pages.dev deployment, they're what makes the staging subdomain itself redirect and de-index. Your production domain doesn't need any of these changes.

Cloudflare Staging Domain Blocker - Stop pages.dev Indexing

Overview

Introduction

A pages.dev preview or staging subdomain left open to the public is a common, easy-to-miss source of duplicate content and stray indexed pages competing with your real production domain in search results.

This tool generates the three files, robots.txt, _headers, and _redirects, that together stop that from happening, all sized to deploy on the staging domain in one pass.

What Is Cloudflare Staging Domain Blocker?

A combined generator producing a disallow-all robots.txt, an X-Robots-Tag: noindex _headers rule, and a forced 301 _redirects rule that sends every staging path to the same path on your production domain.

It's meant to be deployed together on the staging/pages.dev build specifically, not your production domain.

How Cloudflare Staging Domain Blocker Works

robots.txt gets the standard `User-agent: * / Disallow: /` block, asking crawlers not to crawl anything on the staging domain at all.

_headers adds `X-Robots-Tag: noindex, nofollow` on every path, an explicit instruction search engines respect even for pages they do manage to reach.

_redirects adds a forced (`!`) 301 rule sending `https://staging/*` to `https://production/:splat`, so any direct visit, whether from a search result, a stale bookmark, or a crawler that ignores the other two signals, ends up on production instead.

When To Use Cloudflare Staging Domain Blocker

Use it as soon as you notice (or want to prevent) a pages.dev staging URL showing up in search results or being linked to externally.

It's also useful as a standard step in a Cloudflare Pages project template, so every new project's staging environment is protected from day one.

Often used alongside Cloudflare Pages robots.txt Generator, pages.dev Duplicate Content Checker and Cloudflare Pages _redirects File Generator.

Features

Advantages

Covers all three commonly recommended layers (crawl directive, index directive, and hard redirect) in one generation step instead of three separate lookups.
The forced redirect (`!`) detail is easy to forget and silently makes the rule not apply to matching static files without it; this tool includes it automatically.
Keeps the production domain untouched, all output is meant for the staging deployment only.

Limitations

The forced redirect makes the staging domain unusable as an actual preview environment, since every request bounces to production; use Cloudflare Access instead if you still need staging to be reachable for your team.
robots.txt and X-Robots-Tag are respected by well-behaved crawlers but aren't an access control mechanism, they don't stop a determined scraper or a browser from reaching pages before the redirect applies.

Examples

Blocking my-project.pages.dev in favor of example.com

Input

staging domain: my-project.pages.dev, production domain: example.com

Output

# --- _redirects (deploy on the staging domain) ---
# Forces every path on the staging domain to permanently redirect to the same path on production.
https://my-project.pages.dev/*  https://example.com/:splat  301!

The forced (!) 301 rule ensures every path, including ones matching a real static file, redirects to the equivalent production URL.

Best Practices & Notes

Best Practices

Decide up front whether you actually need a live, reachable staging environment, if you do, skip the forced redirect and rely on Cloudflare Access plus the robots.txt/X-Robots-Tag pieces instead.
Deploy all three files together rather than picking just one, each covers a gap the others don't.
Double-check the production domain you enter is exactly correct, a typo here would redirect all staging traffic to the wrong place.

Developer Notes

The `_redirects` rule uses Cloudflare Pages' trailing `!` force-redirect syntax specifically because Pages otherwise prioritizes serving a matching static asset over applying a redirect rule for the same path; both domain inputs are normalized by stripping any protocol prefix and trailing slash before being reassembled into the generated URLs.

Cloudflare Staging Domain Blocker Use Cases

Preventing a newly created Cloudflare Pages staging subdomain from ever being indexed
Cleaning up an existing pages.dev subdomain that's already showing up in search results
Standardizing a "block staging from indexing" step across every new Pages project

Common Mistakes

Deploying only robots.txt and assuming that's sufficient, without the noindex header or redirect, already-indexed or externally linked pages can still appear in search results.
Forgetting the forced-redirect `!` suffix, causing the redirect to silently not apply to any path with a matching static file.

Tips

Run the pages-dev Duplicate Content Checker first if you're not sure whether your project already has a duplicate-content problem worth fixing this way.

Cloudflare Staging Domain Blocker