robots.txt Tester and Checker
Use this robots.txt checker to test path access against a pasted file using Google's matching behavior: equally specific user-agent groups are combined, longest path match wins, and Allow beats Disallow on a tie. Supports * and $ wildcards.
robots.txt body
Test request
Just the path, like /admin/users
Disallowed
- Path
- /admin/users
- User agent
- GPTBot
- Matched group
- gptbot
- Reason
- Longest matching rule: disallow /
Apply this with your coding agent
Get a ready-made prompt that wraps the output above with implementation steps. Paste it into your AI assistant and let it ship the change.
How to check robots.txt rules
Paste the exact file served at https://yourdomain.com/robots.txt, enter the path you need Googlebot or an AI crawler to reach, and test each relevant user agent before deployment.
This follows Google's documented matching behavior. The most specific matching user-agent token wins, duplicate groups for that token are combined, and the longest matching path rule wins, with Allow beating Disallow on a tie. Wildcards * and end-anchor $ are supported.
After publishing, verify that the live file returns HTTP 200 and review fetch or syntax errors in the Google Search Console robots.txt report.
Real usage example
Scenario
A docs team needs to confirm GPTBot can fetch /docs/getting-started before deploy.
Input
Paste the production robots.txt, choose GPTBot, and test /docs/getting-started.
Output
Allowed or blocked status with the matching rule that caused the decision.
Verify on a real URL
Run a full agent readiness scan to see how your live site responds end to end.