Keep growth pages crawlable
Robots rules should not accidentally block product pages, resources, tools, industry templates, or supplier directories that need Google visibility.
Free Technical SEO Tool
Robots.txt generator for export websites.
Create safe robots.txt rules that keep product, resource, tool, template, and directory pages crawlable while blocking admin and staging paths.
- Allow public SEO routes
- Block dashboard, account, admin, and staging paths
- Reference the production sitemap URL
Page highlights
Block private and low-value paths
Account, dashboard, admin, checkout, preview, staging, and internal API paths should not compete with public SEO pages.
Add sitemap discovery
A clean Sitemap directive helps crawlers find the production sitemap without guessing between environments.
FAQ
Should robots.txt block all private pages?
It can discourage crawling, but private pages still need authentication and noindex handling where appropriate.
Can robots.txt remove URLs from Google?
No. Robots.txt controls crawling, not guaranteed removal. Use noindex or removal tools for indexed pages.
Should exporters allow AI crawlers?
Decide per business policy. This generator focuses on search-engine crawlability for public SEO assets.
Related pages
Start audit
Enter one URL and run your first free audit.
Start Free