Crawler Access Diagnostic

Robots.txt Visual Tester & Generator

Test whether search engines and crawlers are blocked from crawling specific paths on your website, or generate battle-tested templates for WordPress, Next.js, and Shopify.

Robots.txt Directives

# Standard Production Robots.txt
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /cart/
Disallow: /checkout/
Allow: /

Sitemap: https://example.com/sitemap.xml

Live Crawl Access Simulator

Crawler User-Agent:

Path to Test:

ALLOWED

—