dust.tt — Agent-Adoption Score

robots.txt served with 1 User-agent directive

robots.txt is the first file crawlers and agents check for access rules; silence defaults to blanket-allow. Per RFC 9309.

Sitemap served at https://dust.tt/sitemap.xml (<urlset> root)

An XML sitemap is the route map agents use to find your pages. Without one they link-walk and miss deep or orphaned content.

Homepage returned no Link header — v1 does not penalize

Link: response headers expose related resources — API catalogs, service docs, alternates — before an agent parses HTML. Per RFC 8288.

robots.txt expresses no AI-bot policy (no per-UA rules, no Content-Signal / Content-Usage directive)

Per-bot robots.txt rules (GPTBot, ClaudeBot) or Content-Signal directives declare who may train on or cite your content. Missing rules allow all.

No Content-Usage or Content-Signal directives found in robots.txt

Content-Signal (Cloudflare) or AIPREF Content-Usage directives declare what AI systems may do post-fetch — train, search-index, or generate answers.

No Web Bot Auth directory published at /.well-known/http-message-signatures-directory

If you operate an AI crawler, a signing-key directory lets targets verify it's you. Most sites don't publish — presence-only check. IETF draft.

robots.txt declares wildcard User-agent with explicit Allow: / and no cross-bot blanket blocks

A blanket-allow posture (wildcard User-agent, Allow: /, no cross-bot blocks) declares that every crawler is welcome. Informational — no pass/fail.

No llms.txt found at primary or fallback path

An llms.txt file gives agents a curated entry point into your docs — sitemap-equivalent, but sized for context windows. Per llmstxt.org.

llms-txt-valid cannot evaluate without llms.txt body

A well-formed llms.txt (H1 title, summary blockquote, linked sections) parses cleanly; a malformed one is skipped silently — worse than no file. Per llmstxt.org.

llms-txt-size cannot evaluate without a discovered llms.txt

llms.txt must fit in an agent's context window alongside the user's question. Under 50K is comfortable; over 100K gets truncated mid-section.

llms-txt-has-optional-section cannot evaluate without llms.txt body

Reports the shape of your llms.txt — Optional section, H2 count, link count — so you can tell at a glance whether agents get a skeleton or a full map.

Only 0/5 sampled URLs serve markdown (need ≥ 50%)

A .md twin alongside each HTML page (page.md ↔ page.html) gives agents an agent-readable fetch path — no HTML, ad wrappers, or modals.

Server ignored Accept: text/markdown — returned HTML instead

Accept: text/markdown negotiation serves HTML to humans and agent-readable markdown to agents from one URL — no duplicate-URL strategy.

Server-side rendering confirmed

Classifies the site as server-rendered, hydrated, or client-rendered (SPA) — what agents see without running JavaScript. A pure SPA reads as blank.

Mixed sample outcomes: 7 pass / 1 warn / 2 fail — conservative fail

Measures how much markdown each page feeds into an agent's context window. Under 50K fits cleanly; over 100K truncates mid-page — pages have context budgets too.

Correct HTTP 404 returned for non-existent path

Soft-404s (HTTP 200 on a missing page) make agents cache garbage as canonical content. An honest 4xx tells agents the URL is dead — drop it.

All 5/5 sampled URLs use same-eTLD+1 HTTP redirects or no redirect

Same-domain HTTP 3xx redirects work for agents. JavaScript redirects break agents without JS; cross-domain jumps read as tracking.

AGENTS.md not found at /AGENTS.md — HTTP 404 response

AGENTS.md is a coding-agent convention. ETH Zurich research (2026) found it often hurts those agents; we track presence to test the effect on websites. Informational.

Homepage response carries 2 of Cache-Control / ETag / Last-Modified: cache-control, etag

Cache-Control, ETag, and Last-Modified headers let agents re-fetch only what changed — missing headers force full re-downloads. Informational.

No api-catalog published at /.well-known/api-catalog

A /.well-known/api-catalog (RFC 9727) points agents at your OpenAPI specs and developer docs in one fetch — the discovery surface for programmatic integration.

no OAuth surface detected — scan inconclusive

OAuth discovery metadata at /.well-known/oauth-authorization-server lets agents locate your auth endpoints without reading docs. Per RFC 8414.

No OAuth Protected Resource metadata at /.well-known/oauth-protected-resource

Protected Resource metadata identifies which authorization server protects your API. Paired with oauth-discovery, agents complete auth without reading docs. Per RFC 9728.

No MCP Server Card found at any well-known path

An MCP Server Card advertises your Model Context Protocol endpoint to Claude, Cursor, and other MCP clients — the standard discovery path.

No A2A Agent Card published at /.well-known/agent-card.json

An A2A Agent Card at /.well-known/agent-card.json describes your service to other agents — capabilities, skills, calling conventions. Per A2A v1.0.

No Agent Skills index found at primary or fallback path

An Agent Skills index at /.well-known/agent-skills/ exposes your capabilities as discrete, individually-addressable skills. Agent Skills v0.2.