Which AI actually reads your site? Two months of AI agent and LLM traffic, measured
A few months ago we published a guide to making your site readable by LLMs, a practice people now file under GEO (generative engine optimization), or AI SEO. We ended that guide with an uncomfortable admission: you don’t actually know if any of it works unless you measure things. So, Evil Martians instrumented evilmartians.com in April and has watched the server logs ever since. This post is the receipts, including the parts that might even embarrass us: find out which agents read HTML, which read Markdown, and what llms.txt actually does.
Other parts:
- Making your site visible to LLMs and AI agents: 6 techniques that work, 8 that don't
- Which AI actually reads your site? Two months of AI agent and LLM traffic, measured
- Publish agent skills like packages: discovery index, digests, and install sources
In part one, our guide to making a site discoverable to AI agents, we shipped six techniques: llms.txt, .md routes, <link> tags, a hidden hint <div>, llms-full.txt, and Accept: text/markdown content negotiation. We were honest that most of them were bets. The external log studies we cited all said the same thing: major LLM crawlers don’t fetch .md files or llms.txt on their own.
Then we put a server-side tracker on our own site and watched what AI agents actually fetched.
Updated in October 2026. The first version of this post covered May to July 2026. We’ve now repeated the measurement for September and October 2026 with one change: AI companies’ indexing and training crawlers (GPTBot, ClaudeBot, PerplexityBot, Meta’s, and the rest) no longer count as agent traffic. A crawler filling an index isn’t a person reading your page. Every number below is from September and October unless it says otherwise, and we compare with May–July where it helps. The changelog lists every edit.
TL;DR: In September and October 2026, AI agents fetching pages for a person made about 62,000 requests to our site, three for every four human pageviews. Coding agents and chat assistants want opposite things: Claude Code, now our largest AI reader, asks for Markdown 88% of the time, while ChatGPT’s fetcher reads HTML on 99% of its requests. Only one AI assistant reads llms.txt regularly: Claude Code. And the hidden hint for AI that we recommended ourselves got zero measurable hits.
How do you measure LLM traffic when the clients don’t run JavaScript?
Part one flagged a trap: AI crawlers don’t execute JavaScript, so client-side Google Analytics never sees them. So, if you want to know which agents fetch what, you need to catch the request at the edge, before a byte of HTML renders.
This means the tracking lives in server-side middleware that runs on every request. Since we’re on Netlify, that’s an edge function for us, but any request-time hook works (Nginx, a Rails or Express middleware, a Cloudflare Worker). It reads the raw User-Agent and Accept headers, classifies the client, and fires a server-side event to your analytics. In our case that’s GA4 (Google Analytics 4) through its Measurement Protocol, but any backend that accepts server-side events (or even plain log parsing) works just as well. You just get the HTTP request as it arrives, minus JavaScript, cookies, or rendering.
The core is a lookup table that matches the User-Agent against a list of patterns, each tagged with a kind:
| Kind | What it is | Examples |
|---|---|---|
ai | A fetcher acting on a person’s request | ChatGPT-User, Claude-User |
ai-crawler | An AI company’s indexing or training crawler | GPTBot, ClaudeBot |
crawler | A search engine or SEO crawler | Googlebot |
scanner | A site checker or audit tool | isitagentready |
library | A bare HTTP client | curl |
browser | A person’s web browser | Chrome, Safari |
Most AI companies ship both an ai and an ai-crawler user agent. Keeping those two apart is the most useful thing the table does.
One decision is worth copying: don’t gate on a positive “known AI” allowlist. This is because new user agents appear weekly and you’d miss them. Instead, exclude the obvious non-AI (browsers, crawlers, scanners) and count everything else that fetched a Markdown or llms.txt surface as a likely agent:
const kind = classify(userAgent);
// 'ai' | 'ai-crawler' | 'browser' | 'crawler' | 'scanner' | 'library'
const isNoise = kind === 'browser' || kind === 'crawler' || kind === 'scanner';
if (kind === 'ai' || kind === 'ai-crawler' || (servedMarkdown && !isNoise)) {
track({ agent: kind, format: servedMarkdown ? 'markdown' : 'html' });
}
The same function does the content negotiation from part one; if a client sends Accept: text/markdown, it rewrites to the .md version and sets Vary: Accept—so tracking and serving share one code path. In the reports, agent traffic means ai plus the strings we couldn’t identify. We leave out ai-crawler, because it measures a company’s index rather than its users, and library, because a bare curl request could be anything. Every number below comes from that stream, captured in September and October 2026.
One thing this setup forces you to confront immediately: classifying “LLM traffic” is the actual hard part.
How much of your traffic is even human?
In September and October, our tracker logged about 62,000 requests from AI agents acting for a person, against 84,000 human pageviews. That’s three agent reads for every four human ones.
| Reader | Requests, September–October 2026 |
|---|---|
| People (client-side pageviews) | 84,000 |
| AI assistants and coding agents fetching for a person (including 900 requests we couldn’t identify) | 61,700 |
| AI companies’ indexing and training crawlers, excluded from every count below | 143,000+ |
With the crawlers added back, software reads the site about two and a half times as often as people do. We left them out because we want to know whether a person was on the other end of each request. In those two months, Meta’s crawlers alone made more than 42,000 requests, LinkupBot more than 37,000, and Exa’s crawler and PerplexityBot more than 20,000 each. That’s companies filling their indexes, not people asking an assistant about us, and it drowns out the signal.
Removing the crawlers isn’t what changed the ratio, though. In May–July, they made only about 3,000 requests a week, and assistants alone still outread people two to one: 218,568 requests against 106,990 pageviews. What changed since then is one agent, ChatGPT, as you’ll see below.
One honest caveat is that the counts aren’t symmetric. Agent traffic is server-side, so it catches everything; human traffic is client-side analytics, which misses anyone blocking scripts or cookies. So the human side is undercounted, and the true ratio is somewhat lower than three to four. The conclusion survives anyway: a large share of what reads your site is software acting for a person, and none of it lands in the analytics dashboard you already have open.
More than half of agent reads are Markdown
With the crawlers excluded, agents fetched the Markdown version of a page on 56% of tracked reads: about 34,300 Markdown fetches against 27,200 HTML ones, plus 170 for the two llms.txt files.
| Format served | Requests | Share |
|---|---|---|
Markdown (negotiated, or a .md URL) | 34,300 | 56% |
| HTML | 27,200 | 44% |
llms.txt and llms-full.txt | 170 | 0.3% |
The first version of part one called .md routes a bet on the future, since there was no evidence that crawlers fetch them. In May–July, Markdown was 15% of agent reads, with the crawlers still counted. Two changes pushed it past half: ChatGPT’s fetcher, which reads HTML, made 92% fewer requests, and Claude Code, which asks for Markdown, grew by more than half. That’s real, recurring traffic to endpoints that skeptics expected nobody to touch.
There’s a big caveat, though: almost all of it is one client. Claude Code made 32,124 of the 34,342 Markdown reads, or 94%. Without it, the remaining named agents fetch Markdown 6% of the time. So “agents read Markdown” is true on our site because one agent reads a lot of it.
Where you draw the classification line still changes the number. Over the same two months, Exa’s search crawler fetched 13,800 Markdown pages, ElevenLabs’ and Elamur’s crawlers another 3,700, and an SEO crawler called SEOkicks 1,000. Count them as agents, and the share goes up. We count them as crawlers, so it doesn’t. Each one needed its own pattern in the table, and new ones will show up next quarter. This kind of measurement isn’t a dashboard you glance at, it’s a classification problem you have to own.
Why “optimize for LLMs” is the wrong instruction
“AI traffic” isn’t one audience. On our site, it splits into coding agents and chat assistants, and the two behave like opposites. Which company built a client matters less than which group it belongs to. Here are the five largest clients:
| Agent | Requests | Markdown fetches | Markdown share |
|---|---|---|---|
Claude Code (Claude-User) | 36,691 | 32,124 | 88% |
ChatGPT (ChatGPT-User) | 16,445 | 118 | 0.7% |
DuckDuckGo (DuckAssistBot) | 3,179 | 1 | 0.0% |
| DeepSeek | 483 | 9 | 1.9% |
Exa (ExaUser) | 456 | 456 | 100% |
Another 3,600 requests came from smaller named clients: Google’s AI fetchers, Grok, OpenCode, GitHub Copilot, Qwen, Qoder, Perplexity’s user fetcher, and about 1,200 Claude- and Anthropic-branded strings that don’t say which product sent them. Another 900 came from strings we couldn’t identify at all. When we group every named client by what it does, the pattern is clear:
| Group | Clients | Requests | Markdown share |
|---|---|---|---|
| Coding agents | Claude Code, OpenCode, GitHub Copilot, Qoder, Exa’s and Grok’s fetch tools | ~38,000 | 88% |
| Chat assistants | ChatGPT, DuckDuckGo, DeepSeek, Qwen, Perplexity, Google’s, the unlabeled Claude strings | ~23,000 | 0.9% |
ChatGPT-User, the fetcher that pulls your page when someone asks ChatGPT about it mid-conversation, reads rendered HTML almost exclusively: 16,326 HTML fetches against 118 Markdown ones. It also shrank. In May–July, it made 196,973 requests, 90% of everything agents fetched for a person. In September and October, it made 16,445, a 92% drop, and we changed nothing on our side. Our logs can’t tell us whether OpenAI changed how often ChatGPT fetches live pages or people just asked it about us less. But they do show that one quarter’s numbers for a single agent aren’t a trend. A plan built on “ChatGPT is 90% of AI traffic” would have been wrong three months later.
Now back to the first row. The traffic we labeled “Claude” is almost entirely the Claude Code coding agent, not the training crawler. It has been the largest AI reader of our site every week since late August, with 36,691 requests in two months. That’s almost 60% of the total, and up by more than half since May–July. (Claude-User also covers the claude.ai chat fetcher, which sends about one request in ten.) Claude Code asks for Markdown 88% of the time, because it sends Accept: text/markdown and our content negotiation returns the Markdown version. Claude Code alone accounts for 94% of all Markdown traffic to the site.
So here’s the real picture on our site. The biggest consumer of our Markdown is a coding agent that asks for it with a 25-year-old HTTP mechanism. Every chat assistant, ChatGPT included, reads the same HTML your visitors get. The technique that delivers the Markdown is content negotiation, the one part one bet on as the long-term survivor. Both rounds of data back that bet.
This is why “make your site visible to LLMs” is too vague to act on. The useful version is narrower: find the two or three agents in your own logs and serve what each one actually requests. For us, that’s clean HTML for the chat assistants and sharp Accept negotiation for the coding agents. Yours will differ, and the only way to know is to look.
Does anything actually read llms.txt?
Part one’s biggest open question was whether llms.txt gets used at all. Google’s John Mueller had said flatly that “no AI system currently uses llms.txt,” and the log studies we cited backed him up. Since then, Google has put it in writing that Search doesn’t use the file. Two 2026 log studies, one by Ahrefs across 137,000 domains and a 30-day audit of Adobe Experience Manager domains, each traced only 1.1% of requests for it to AI bots.
Our direct-fetch numbers look mildly encouraging at first. In September and October, /llms.txt was fetched 498 times and /llms-full.txt 95 times, about 600 in total. In May–July, it was roughly 770, but that count still included people opening the file in a browser. Modest, but not the zero the skeptics implied.
Then we broke those fetches down by client:
Who fetched llms.txt and llms-full.txt | Fetches |
|---|---|
| SEO and search crawlers (SEOJuice, Awario, PoweredBy, Idarus, Semrush…) | 139 |
| AI-readiness and bot-policy scanners (BotPolicy, PeecAudit, a “policy-surface observation” script…) | 137 |
Strings we couldn’t identify: llms.txt censuses, “research” scripts, link checkers | 120 |
Bare HTTP clients (curl 60 times, then node, got, wget, python-requests) | 75 |
| AI companies’ crawlers (Exa 41, Meta 10, ClaudeBot, GPTBot, OAI-SearchBot, YouBot, PerplexityBot…) | 75 |
| Claude Code | 43 |
| Other named assistants (ChatGPT, DuckDuckGo, OpenCode, a Claude-branded client), one fetch each | 4 |
Of the 593 direct fetches, only 47 came from named AI assistants, and 43 of those were Claude Code (35 for llms.txt, 8 for llms-full.txt). ChatGPT and Perplexity’s crawler fetched the file once each. Since we started logging in April, the total is 70 of 2,436: Claude Code 48, ChatGPT 16, and nobody else more than once.
So the file has one regular AI reader, and it’s a coding agent that a person pointed at our site. Everyone else is the same crowd as in May: search and SEO crawlers, scanners that check whether the file exists so they can score us on it, llms.txt census projects (an ElgooLlmsCensus, a discovery-v15-census, an llms-txt-collector), and 60 fetches from curl, most likely someone’s terminal. The AI assistants the file was built for are almost entirely missing. The file is popular, just not with its intended audience.
In May–July, we got excited about one more signal, and that’s where our first read of the data was simply wrong. /llms.txt showed up as the referrer on a batch of page views. We took that to mean a client was reading the file and following its links, exactly what the format was designed for. Broken down by user agent, the story fell apart. Of those 117 referred hits, 106 came from a single client with a frozen Chrome/111.0 string, a typical sign of a bot, not a browser or an AI agent.
The September–October data is even clearer. Only 57 page views had an llms.txt referrer, and all of them came from SEO crawlers:
| Client | Referred page views |
|---|---|
SEOJuice-SearchBot | 53 |
Keywords Everywhere’s PoweredByBot | 4 |
Both crawlers read the file on our secondary domains, evl.ms and evilmartians.io, and followed its links to map the site. None of the referred visits came from an AI assistant.
So our July conclusion holds, with one exception. For us, Mueller was basically right: llms.txt gets fetched by bots and crawlers, and by Claude Code 43 times in two months. The file takes five minutes to ship and it’s harmless to leave up. But apart from Claude Code, we can’t point to any meaningful AI use of it, and the “nobody meaningfully uses llms.txt” consensus held in both measurements.
What surprised us?
LLMs try URLs you don’t have, but maybe you should. Hundreds of requests again hit pages that don’t exist. Agents guessed the favicon guide’s URL as how-to-favicon-in-2024, -2025, and -2026, in both a “six files” and a “three files” variant, about 65 times; they assume we update the guide every year. Our devtools funding analysis got seven reworded versions of its real title. The predictable ones are a to-do list with the titles already picked. Your .md 404s are free signal about the pages agents expect you to have.
LLMs hallucinate your domain, too. Thousands of requests arrived for the wrong TLD—evilmartians.dev, .app, .co, .tech, .io, .eu—and even our raw origin IP. Models remember the brand and guess the extension. We own these variants and redirect all of them to evilmartians.com, which is the only reason we can see this traffic at all: we recover it instead of losing it. That’s exactly why registering the variants pays off. The extensions you don’t own are confidently wrong requests you’ll never even see.
The clever hint got zero hits. One of part one’s original techniques was a visually hidden <div> telling any AI reader where to find the Markdown version, with the URL tagged ?ref=hint so we could measure follows. Since we started counting in April, the number of agent requests carrying ?ref=hint has been zero.
We want to be fair: an agent could read the hint, strip the query string, and refetch as an untagged direct request, which we couldn’t distinguish. But the hint hands over the literal URL including the tag, so a straightforward follow would have preserved it. The honest conclusion is that we cannot attribute a single Markdown fetch to the hidden hint. It’s cheap enough to leave in place, but if you were about to invest engineering time making yours fancier, don’t. The agents that want your Markdown are asking for it with an Accept header, not reading a hidden paragraph and clicking a link. Part one has since moved it to what doesn’t work.
Not every AI user agent is an AI. About 650 requests wearing AI user-agent strings went straight for /.env, /id_rsa, and /actuator/env: ordinary vulnerability scanning in a borrowed disguise. Check the paths, not just the user agent, before you count a request as AI traffic.
The noise is relentless. The same tracking bucket that catches Claude Code also catches junk. Forty command-injection probes put their payload in the User-Agent itself (echo bcmlwx$(), expr 9000123342 - 987752) and fetched one Markdown page each to see if we’d run it. We also logged a secret-scanner/1.0, a SQLSpider, and a client called pc. About 3,000 requests asked for /chronicles/null, /events/null, and /null: someone’s URL template inserting an undefined slug and fetching the result at scale. This is the tax on measuring: your “AI traffic” report is a crime scene until you clean it up.
What we’d change, knowing all this
We’ve re-ranked part one’s priority table based on these logs. Here’s what six months of data argue for:
Keep your HTML clean, because every chat assistant only reads HTML. The chat assistants (ChatGPT, DuckDuckGo, DeepSeek, Qwen, Perplexity’s fetcher, and the rest) made 23,000 requests between them and took Markdown on 0.9% of them. Every hour you’d spend perfecting a .md pipeline for them is better spent making sure your rendered HTML isn’t 80% navigation and script tags. The Markdown readers are real, and content negotiation serves them automatically anyway.
Treat content negotiation as the load-bearing technique. It’s the mechanism that actually delivers Markdown to the clients that want it: Claude Code 88% of the time, OpenCode 93%, GitHub Copilot 100%. It needs no site-specific knowledge from the agent, and it’s standard HTTP. The first post bet this would be the survivor. Six months in, it’s the only technique on the list whose traffic keeps growing.
Demote the decorative techniques. The hidden hint earned nothing measurable, and llms.txt drew one real reader and a crowd of bots. Don’t rip either out—they’re free—but stop treating “clever signals for AI” as a category worth investing in. The techniques that worked are the boring standards-based ones.
Split your user-agent groups before you trust your numbers. A company’s crawler and its on-demand fetcher are different audiences. PerplexityBot crawled us more than 20,000 times for its index, while Perplexity-User fetched about 900 pages for people. Lumped together, “Perplexity” looks like our second-largest reader. Split, it’s under 2% of reads. Splitting “Claude” the same way showed that most of it was Claude Code, our largest reader, not the crawler. If your classification is coarse, your conclusions are wrong in ways you can’t see.
Expect the mix to move. Between May–July and September–October, ChatGPT’s fetcher fell 92% and Claude Code grew by more than half. LinkupBot appeared from nowhere, and Meta’s crawlers grew sevenfold; together, those two made 80,000 requests in September and October. In July, we wrote that the mix would look different next quarter. It did.
And the meta-point, above all the tactics: measurement comes first. Part one gave you the techniques to ship; this is the step that tells you which of them are working for you. You can’t reason about any of it—which format, which agent, which file—until you’re watching your own server logs. Every confident claim in the AI SEO discourse dissolves the moment you look at real traffic to a real site.
Want Claude to ship part one’s techniques for you? We packaged that guide as a downloadable Claude skill—every technique the data here backs, plus the eight anti-patterns to refuse.
There’s really no such thing as “AI traffic.” Underneath the label sits a handful of named agents with specific, measurable, and frankly inconsistent habits, and that mix looked different one quarter later. Winning at AI discovery won’t come from shipping the most .md files. It comes from the boring habit of reading your own logs, and then reading them again after the agents change.
FAQ
Do crawlers really fetch .md files?
Yes, some crawlers do, and so do coding agents. Among agents fetching pages for a person, 56% of reads on our site were Markdown in September and October, and 94% of those came from Claude Code asking via Accept: text/markdown. Your mix will differ, so measure your own.
Which matters more, HTML or Markdown?
Both, for different clients. On our data, HTML serves every chat assistant (ChatGPT, DuckDuckGo, DeepSeek, Perplexity) and Markdown serves the coding agents. Neglect either and you lose a real audience.
Is the hidden AI hint <div> worth adding?
We can’t show it did anything. The result was zero attributable fetches in six months. If you already have one, leaving it in is harmless, but don’t build one. Content negotiation is where the Markdown actually gets delivered.
How do you track AI crawlers and agents that don’t run JavaScript?
Server-side middleware that reads raw User-Agent and Accept headers and sends events to your analytics backend (GA4 via its Measurement Protocol, in our case—but plain log parsing or any server-side analytics works too). Client-side analytics can’t see non-rendering agents; this can.
What is GEO (generative engine optimization), and how is it different from SEO?
GEO is optimizing so LLMs and AI assistants can find, fetch, and cite your content—the same goal as classic SEO, but the audience is agents instead of search-engine crawlers. In practice the two overlap more than the hype suggests. Chat assistants like ChatGPT want exactly what SEO already asks for: clean, legible rendered HTML. The GEO-specific layer on top is serving Markdown through content negotiation for coding agents like Claude Code, now our largest AI reader. You don’t replace your SEO with GEO; you add a thin format-negotiation layer and then measure who shows up.
Is “AI bot traffic” the same as AI agent traffic?
No. AI bot traffic is everything that isn’t a person: indexing and training crawlers (GPTBot, ClaudeBot, PerplexityBot, Meta’s) plus the agents fetching a page for someone (Claude Code, ChatGPT-User, Perplexity-User). On our site, the crawlers made 143,000+ requests in September and October and the agents about 62,000, so a count that doesn’t split them is mostly crawlers. Every number in this post counts only the agents, because they’re the reads with a person on the other end.
Does llms.txt actually work?
Barely, and only for one client. In September and October, llms.txt and llms-full.txt were fetched about 600 times. Named AI assistants made 47 of those requests, and 43 of the 47 came from Claude Code. The rest came from SEO crawlers, AI-readiness scanners, llms.txt census scripts, and curl. All 57 page views the file referred came from two SEO crawlers, none from an AI assistant. So the “nobody meaningfully reads llms.txt” consensus held for us in both measurements. The file costs five minutes to ship, so leave it up—but check the user agents behind the traffic before you believe any of it is AI.
Which agents read your site, and what they pull, is a question only your own logs can answer—everything here is just what ours said. So instrument it, look, and expect to be surprised by at least one number.
Changelog
2.0.0 (2026-11-04)
- Re-measured over September and October 2026. Every count is from those two months, with the May–July figures kept where the comparison matters.
- AI companies’ indexing and training crawlers (
GPTBot,OAI-SearchBot,ClaudeBot,PerplexityBot, Meta’s,LinkupBot, Exa’s crawler, and the rest) and bare HTTP clients likecurlno longer count as agent traffic; only fetchers acting on a person’s request, and clients we can’t identify, do. - The agent table lists user-driven clients only, read as coding agents versus chat assistants: Claude Code is now the largest reader,
ChatGPT-Userfell 92%, and Markdown is 56% of agent reads, up from 15% when the crawlers still counted. llms.txtre-measured: 593 fetches, 47 from named assistants, 43 of them Claude Code; its 57 referred page views all came from SEO crawlers. The section now cites Google’s guide to generative AI features and the 2026 log studies from Ahrefs and Longato. The hidden hint is still at zero.- Replaced the GA4 screenshots with tables and added the
/nullpaths, the vulnerability probes behind AI user agents, and the command-injection probes to the surprises.
1.0.0 (2026-07-21)
- Original post published.