EErgoKit

SEO / GEO

Robots.txt Tester

Fetch robots.txt, inspect rules, and test whether a path can be crawled.

Server-side checkPublic web targetsResponse body hidden
Runner

Test crawler access

Submitted inputs are checked transiently by ErgoKit Workers. Results are not stored.

Enter a site URL and optional path to inspect robots.txt crawler rules.

Checks

  • robots.txt fetch status
  • User-agent groups
  • Allow and Disallow matching
  • Sitemap and crawl-delay directives

Result

  • Allowed or blocked verdict
  • Matched crawler rule
  • Parsed robots groups
  • Sitemap declarations

Guardrails

  • Public HTTP and HTTPS URLs only
  • Root robots.txt is fetched transiently
  • Response body is bounded before parsing
  • Private and local network targets blocked

Privacy

Submitted URLs are checked by ErgoKit's Worker runtime. Results show metadata only; response bodies are not displayed.