AI agent readiness score methodology

Pass rule: The site exposes a reachable, parseable robots.txt file at the domain root.

Agents and crawlers use this file as the first crawl-policy signal.

Pass rule: A sitemap is discoverable directly or through robots.txt and returns valid XML.

Sitemaps give agents a canonical inventory of public URLs.

Pass rule: The homepage response advertises useful machine-readable references through RFC 8288 Link headers.

Headers let agents find indexes, feeds, API catalogs, and related resources without page scraping.

Pass rule: A request with Accept: text/markdown receives a useful markdown representation.

Markdown gives agents a cleaner parsing path than visual HTML when the site can provide it.

Pass rule: robots.txt declares explicit behavior for major AI crawlers or clearly relies on wildcard rules.

Explicit bot policy reduces ambiguity around discovery, search, and on-demand fetching.

Pass rule: robots.txt includes Content-Signal preferences such as ai-train, search, and ai-input.

Content-Signal separates search visibility from model-training and answer-input preferences.