watchmycompetitor.com — Agent-Adoption Score

robots.txt served with 10 User-agent directives

robots.txt is the first file crawlers and agents check for access rules; silence defaults to blanket-allow. Per RFC 9309.

Sitemap served at https://watchmycompetitor.com/sitemap.xml (<urlset> root)

An XML sitemap is the route map agents use to find your pages. Without one they link-walk and miss deep or orphaned content.

Homepage serves a Link: header with agent-useful rel types (1 link with rel values: https://api.w.org/)

Link: response headers expose related resources — API catalogs, service docs, alternates — before an agent parses HTML. Per RFC 8288.

7 AI UAs with explicit rules + Content-Signal directive present

Per-bot robots.txt rules (GPTBot, ClaudeBot) or Content-Signal directives declare who may train on or cite your content. Missing rules allow all.

AIPREF directives present — Content-Signal (legacy, 1 directive: ai-train, search)

Content-Signal (Cloudflare) or AIPREF Content-Usage directives declare what AI systems may do post-fetch — train, search-index, or generate answers.

Web Bot Auth directory at /.well-known/http-message-signatures-directory is not valid JSON

If you operate an AI crawler, a signing-key directory lets targets verify it's you. Most sites don't publish — presence-only check. IETF draft.

robots.txt posture is not blanket-allow: 9 per-UA blanket-block groups (Amazonbot, Applebot-Extended, Bytespider, …)

A blanket-allow posture (wildcard User-agent, Allow: /, no cross-bot blocks) declares that every crawler is welcome. Informational — no pass/fail.

No llms.txt found at primary or fallback path

An llms.txt file gives agents a curated entry point into your docs — sitemap-equivalent, but sized for context windows. Per llmstxt.org.

llms-txt-valid cannot evaluate without llms.txt body

A well-formed llms.txt (H1 title, summary blockquote, linked sections) parses cleanly; a malformed one is skipped silently — worse than no file. Per llmstxt.org.

llms-txt-size cannot evaluate without a discovered llms.txt

llms.txt must fit in an agent's context window alongside the user's question. Under 50K is comfortable; over 100K gets truncated mid-section.

llms-txt-has-optional-section cannot evaluate without llms.txt body

Reports the shape of your llms.txt — Optional section, H2 count, link count — so you can tell at a glance whether agents get a skeleton or a full map.

No sitemap URLs available to sample — cannot evaluate markdown support

A .md twin alongside each HTML page (page.md ↔ page.html) gives agents an agent-readable fetch path — no HTML, ad wrappers, or modals.

Server ignored Accept: text/markdown — returned HTML instead

Accept: text/markdown negotiation serves HTML to humans and agent-readable markdown to agents from one URL — no duplicate-URL strategy.

Server-side rendering confirmed

Classifies the site as server-rendered, hydrated, or client-rendered (SPA) — what agents see without running JavaScript. A pure SPA reads as blank.

No sitemap URLs available to sample — cannot evaluate page size

Measures how much markdown each page feeds into an agent's context window. Under 50K fits cleanly; over 100K truncates mid-page — pages have context budgets too.

Server returned HTTP 200 for a non-existent path — soft-404 detected

Soft-404s (HTTP 200 on a missing page) make agents cache garbage as canonical content. An honest 4xx tells agents the URL is dead — drop it.

No sitemap URLs available to sample — cannot evaluate redirect behavior

Same-domain HTTP 3xx redirects work for agents. JavaScript redirects break agents without JS; cross-domain jumps read as tracking.

AGENTS.md not found at /AGENTS.md — non-markdown content-type (text/html)

AGENTS.md is a coding-agent convention. ETH Zurich research (2026) found it often hurts those agents; we track presence to test the effect on websites. Informational.

Homepage response carries none of Cache-Control / ETag / Last-Modified

Cache-Control, ETag, and Last-Modified headers let agents re-fetch only what changed — missing headers force full re-downloads. Informational.

api-catalog at /.well-known/api-catalog is not valid JSON

A /.well-known/api-catalog (RFC 9727) points agents at your OpenAPI specs and developer docs in one fetch — the discovery surface for programmatic integration.

Neither OAuth AS metadata nor OpenID configuration could be validated

OAuth discovery metadata at /.well-known/oauth-authorization-server lets agents locate your auth endpoints without reading docs. Per RFC 8414.

OAuth Protected Resource metadata at /.well-known/oauth-protected-resource is not valid JSON

Protected Resource metadata identifies which authorization server protects your API. Paired with oauth-discovery, agents complete auth without reading docs. Per RFC 9728.

No MCP Server Card found at any well-known path

An MCP Server Card advertises your Model Context Protocol endpoint to Claude, Cursor, and other MCP clients — the standard discovery path.

A2A Agent Card at /.well-known/agent-card.json is not valid JSON

An A2A Agent Card at /.well-known/agent-card.json describes your service to other agents — capabilities, skills, calling conventions. Per A2A v1.0.

No Agent Skills index found at primary or fallback path

An Agent Skills index at /.well-known/agent-skills/ exposes your capabilities as discrete, individually-addressable skills. Agent Skills v0.2.