betterstack.com — Agent-Adoption Score

robots.txt served with 2 User-agent directives

robots.txt is the first file crawlers and agents check for access rules; silence defaults to blanket-allow. Per RFC 9309.

Sitemap served at https://betterstack.com/sitemap.xml (<urlset> root)

An XML sitemap is the route map agents use to find your pages. Without one they link-walk and miss deep or orphaned content.

Homepage serves a Link: header with agent-useful rel types (10 links with rel values: preload, api-catalog)

Link: response headers expose related resources — API catalogs, service docs, alternates — before an agent parses HTML. Per RFC 8288.

Content-Signal directive expresses aggregate AI-bot policy

Per-bot robots.txt rules (GPTBot, ClaudeBot) or Content-Signal directives declare who may train on or cite your content. Missing rules allow all.

AIPREF directives present — Content-Signal (legacy, 1 directive: ai-input, ai-train, search)

Content-Signal (Cloudflare) or AIPREF Content-Usage directives declare what AI systems may do post-fetch — train, search-index, or generate answers.

No Web Bot Auth directory published at /.well-known/http-message-signatures-directory

If you operate an AI crawler, a signing-key directory lets targets verify it's you. Most sites don't publish — presence-only check. IETF draft.

robots.txt declares wildcard User-agent with no Disallow rules and no cross-bot blanket blocks

A blanket-allow posture (wildcard User-agent, Allow: /, no cross-bot blocks) declares that every crawler is welcome. Informational — no pass/fail.

No llms.txt found at primary or fallback path

An llms.txt file gives agents a curated entry point into your docs — sitemap-equivalent, but sized for context windows. Per llmstxt.org.

llms-txt-valid cannot evaluate without llms.txt body

A well-formed llms.txt (H1 title, summary blockquote, linked sections) parses cleanly; a malformed one is skipped silently — worse than no file. Per llmstxt.org.

llms-txt-size cannot evaluate without a discovered llms.txt

llms.txt must fit in an agent's context window alongside the user's question. Under 50K is comfortable; over 100K gets truncated mid-section.

llms-txt-has-optional-section cannot evaluate without llms.txt body

Reports the shape of your llms.txt — Optional section, H2 count, link count — so you can tell at a glance whether agents get a skeleton or a full map.

Only 2/5 sampled URLs serve markdown (need ≥ 50%)

A .md twin alongside each HTML page (page.md ↔ page.html) gives agents an agent-readable fetch path — no HTML, ad wrappers, or modals.

Server honored Accept: text/markdown

Accept: text/markdown negotiation serves HTML to humans and agent-readable markdown to agents from one URL — no duplicate-URL strategy.

Server-side rendering confirmed

Classifies the site as server-rendered, hydrated, or client-rendered (SPA) — what agents see without running JavaScript. A pure SPA reads as blank.

Mixed sample outcomes: 1 pass / 9 warn / 0 fail — conservative fail

Measures how much markdown each page feeds into an agent's context window. Under 50K fits cleanly; over 100K truncates mid-page — pages have context budgets too.

Correct HTTP 404 returned for non-existent path

Soft-404s (HTTP 200 on a missing page) make agents cache garbage as canonical content. An honest 4xx tells agents the URL is dead — drop it.

All 5/5 sampled URLs use same-eTLD+1 HTTP redirects or no redirect

Same-domain HTTP 3xx redirects work for agents. JavaScript redirects break agents without JS; cross-domain jumps read as tracking.

AGENTS.md not found at /AGENTS.md — HTTP 404 response

AGENTS.md is a coding-agent convention. ETH Zurich research (2026) found it often hurts those agents; we track presence to test the effect on websites. Informational.

Homepage response carries 2 of Cache-Control / ETag / Last-Modified: cache-control, etag

Cache-Control, ETag, and Last-Modified headers let agents re-fetch only what changed — missing headers force full re-downloads. Informational.

api-catalog published at /.well-known/api-catalog with 3 linkset entries

A /.well-known/api-catalog (RFC 9727) points agents at your OpenAPI specs and developer docs in one fetch — the discovery surface for programmatic integration.

OAuth metadata served at /.well-known/oauth-authorization-server

OAuth discovery metadata at /.well-known/oauth-authorization-server lets agents locate your auth endpoints without reading docs. Per RFC 8414.

No OAuth Protected Resource metadata at /.well-known/oauth-protected-resource

Protected Resource metadata identifies which authorization server protects your API. Paired with oauth-discovery, agents complete auth without reading docs. Per RFC 9728.

MCP Server Card at /.well-known/mcp/server-card.json is missing required fields: transport

An MCP Server Card advertises your Model Context Protocol endpoint to Claude, Cursor, and other MCP clients — the standard discovery path.

No A2A Agent Card published at /.well-known/agent-card.json

An A2A Agent Card at /.well-known/agent-card.json describes your service to other agents — capabilities, skills, calling conventions. Per A2A v1.0.

No Agent Skills index found at primary or fallback path

An Agent Skills index at /.well-known/agent-skills/ exposes your capabilities as discrete, individually-addressable skills. Agent Skills v0.2.