01
Pre-release checks
Run scans before launches or migrations to catch crawl and content regressions before they hit production.
About
About AgentScan
A specialized AI-agent content-readiness project for modern web teams. Measure, fix, and verify the machine-readable surface of your site.
AgentScan was created to solve a specific production gap: sites that rank well for humans but fail basic machine-readable expectations for autonomous agents. Our goal is to make that gap measurable and fixable in the hands of engineering teams.
We intentionally keep the default profile focused and operational. Instead of broad, generic audits, we validate the checks most likely to affect whether an agent can discover your content and consume it correctly in real-world workflows.
Core production checks
Six signals that determine agent readiness for content sites.
robots.txt quality
Ensures crawler policy exists, is parseable, and supports deterministic bot behavior.
Sitemap discoverability
Checks canonical sitemap paths and robots references to improve content indexing reliability.
Link response headers
Validates machine-discovery signals via RFC 8288-style Link relations.
Markdown negotiation
Tests whether Accept: text/markdown returns agent-friendly markdown responses.
AI bot directives
Looks for explicit AI-crawler handling or safe wildcard policy coverage.
Content-Signal rules
Checks explicit content-use preferences for training, search, and AI-input scenarios.
How teams use it
Three patterns we see most often in production engineering work.
02
Incident triage
When agents fail to parse content correctly, use scan evidence to isolate root causes quickly.
03
Ongoing governance
Use recurring scans as part of technical SEO and AI-readiness hygiene across the team.
Editorial approach
We prefer explicit, testable checks over vague heuristics. If a signal cannot be verified from observable HTTP behavior, we avoid using it in the default score. See our editorial policy for details on selection criteria and scoring.
What is not covered
- ·Private application logic behind authentication
- ·Legal compliance determinations for your jurisdiction
- ·Comprehensive security or penetration testing
- ·Performance and Core Web Vitals scoring
Apply the AgentScan baseline to your project
One copy-paste prompt that covers all six production checks. Hand it to Cursor, Claude Code, ChatGPT, or Copilot to ship the change.
What makes this different
The site content and prompts are written specifically for AI-agent interoperability work. We do not use spun text, filler templates, or syndicated policy blocks as primary content. AgentScan evaluates public HTTP behavior only and produces guidance, not legal advice.
Have a question or scan that looks wrong? Contact us and we will route it to the right team.