Which AI actually reads your site? Two months of LLM traffic, measured

In part one, we shipped six techniques: llms.txt, .md routes, <link> tags, a hidden hint <div>, llms-full.txt, and Accept: text/markdown content negotiation. We were honest that most of them were bets. The external log studies we cited all said the same thing: major LLM crawlers don’t fetch .md files or llms.txt on their own.

Making your site visible to LLMs: 6 techniques that work, 8 that don't

Making your site visible to LLMs: 6 techniques that work, 8 that don't

April 15, 2026

Read also

Then we put a server-side tracker on our own site and watched what actually happened.

TL;DR: Over two months, agents made roughly 268,000 tracked requests to our site. “AI traffic” turned out to be two very different clients that want opposite things: ChatGPT fetches rendered HTML and essentially never touches Markdown; Claude Code asks for Markdown 76% of the time and gets it through content negotiation. llms.txt, on the other hand, gets fetched almost entirely by bots that aren’t the AI assistants it’s meant for. And the one clever technique we recommended ourselves (the hidden hint for AI) got zero measurable hits.

Irina Nazarova CEO at Evil Martians

Let's run a 2-week LLM-optimization sprint with Evil Martians to improve your discoverability.

Book a call

How do you measure LLM traffic when the clients don’t run JavaScript?

Part one flagged a trap: AI crawlers don’t execute JavaScript, so client-side Google Analytics never sees them. So, if you want to know which agents fetch what, you need to catch the request at the edge, before a byte of HTML renders.

This means the tracking lives in server-side middleware that runs on every request. Since we’re on Netlify, that’s an edge function for us, but any request-time hook works (Nginx, a Rails or Express middleware, a Cloudflare Worker). It reads the raw User-Agent and Accept headers, classifies the client, and fires a server-side event to your analytics. In our case that’s GA4 (Google Analytics 4) through its Measurement Protocol, but any backend that accepts server-side events (or even plain log parsing) works just as well. You just get the HTTP request as it arrives, minus JavaScript, cookies, or rendering.

The core is a lookup table: match the User-Agent against a list of patterns, each tagged with a kind—ai, browser, crawler, scanner, or library. ChatGPT and Claude are ai; Googlebot is a crawler; isitagentready is a scanner; and so on.

One decision is worth copying: don’t gate on a positive “known AI” allowlist. This is because new user agents appear weekly and you’d miss them. Instead, exclude the obvious non-AI (browsers, crawlers, scanners) and count everything else that fetched a Markdown or llms.txt surface as a likely agent:

const kind = classify(userAgent); // 'ai' | 'browser' | 'crawler' | 'scanner' | 'library'
const isNoise = kind === 'browser' || kind === 'crawler' || kind === 'scanner';

if (kind === 'ai' || (servedMarkdown && !isNoise)) {
  track({ agent: kind, format: servedMarkdown ? 'markdown' : 'html' });
}

The same function does the content negotiation from part one; if a client sends Accept: text/markdown, it rewrites to the .md version and sets Vary: Accept—so tracking and serving share one code path. Every number below comes from that stream, which we captured between early May and early July 2026.

One thing this setup forces you to confront immediately: classifying “LLM traffic” is the actual hard part.

How much of your traffic is even human?

Over two months, our tracker logged about 268,000 agent requests against roughly 107,000 human pageviews. This means that on our our site, software now reads more than people do, by a more than two to one ratio.

One honest caveat is that the counts aren’t symmetric. Agent traffic is server-side, so it catches everything; human traffic is client-side analytics, which misses anyone blocking scripts or cookies. Treat this as an upper bound, not a precise ratio. But halve it, then halve it again, and the conclusion survives: a large and growing share of what reads your site is software, and none of it lands in the analytics dashboard you already have open.

About 15% of agent reads are Markdown

Across the window, agents pulled the Markdown version of a page on roughly 15% of tracked reads. That’s about 40,000 Markdown fetches against ~227,000 HTML ones.

Part one called .md routes “a bet on the future” with “no evidence crawlers fetch them.” One in six or seven agent reads is Markdown; not the majority, but real, recurring traffic to endpoints that skeptics said would sit untouched.

Another caveat: that 15% excludes a swarm of generic indexer bots like satoric-indexer, IbouBot, and friends. These hammer .md URLs but aren’t AI clients. Our classifier didn’t tag them yet, so they pushed the raw figure to 21%; each just needs a pattern added. Filter more aggressively and it drops to 13%. Where you draw that line is a judgment call and where you’ll have to put in the work: this kind of measurement isn’t a dashboard you glance at, it’s a classification problem you have to own.

Why “optimize for LLMs” is the wrong instruction

”AI traffic” is not one audience. On our site it’s overwhelmingly two clients, and they behave like opposites:

ChatGPT-User, the fetch that fires when someone’s ChatGPT session pulls your page mid-conversation, is 73% of all our agent traffic and reads rendered HTML almost exclusively. Out of nearly 197,000 requests, 272 were Markdown. So if you’re deciding where to spend an afternoon, keeping your rendered HTML clean and legible beats polishing a .md pipeline your single largest AI reader will never touch.

Now look at the second row. The traffic we labeled “Claude” is almost entirely the Claude Code coding agent, not the training crawler. And it asks for Markdown 76% of the time, because Claude Code sends Accept: text/markdown and our content negotiation hands it the clean version. After we drop the anonymous indexers, Claude Code alone accounts for 43% of all legitimate Markdown traffic to the site.

Here’s the real situation on our site: the biggest AI reader of our marketing content is a chatbot pulling HTML, and the biggest consumer of our Markdown is a coding agent using a 25-year-old HTTP mechanism to ask for it. Two clients, two formats. The technique that actually delivers the Markdown is content negotiation, which is the one the previous post bet on as the long-term survivor. The data agrees.

This is why “make your site visible to LLMs” is too vague to act on. The useful version is narrower: find the two or three agents in your own logs and serve what each one actually requests. For us that’s clean HTML for ChatGPT and sharp Accept negotiation for Claude Code. Yours will differ, and the only way to know is to look.

Does anything actually read llms.txt?

Part one’s biggest open question was whether llms.txt gets used at all. Google’s John Mueller had said flatly that “no AI system currently uses llms.txt,” and the log studies we cited backed him up.

The direct-fetch numbers look mildly encouraging at first. Over the two months, /llms.txt was fetched about 660 times directly and /llms-full.txt about 110. Modest, but not the zero the skeptics implied.

Then we broke those fetches down by client, and the encouragement evaporated. Of the ~770 direct fetches, only 37 came from named AI assistants: Perplexity, ChatGPT, Claude, GPTBot, and OAI-SearchBot combined. The remaining ~730 were a zoo we didn’t bother sorting into neat buckets: familiar search crawlers (Googlebot, AhrefsBot, Bingbot, Amazonbot), a long tail of llms.txt-curious fetchers (satoric-indexer, IbouBot, LLMS-Txt-Scanner, a literal “llms.txt crawler bot”), a research crawler from BCG, and a pile of bare scripts (python-requests, curl, aiohttp). What matters is what’s absent from the list: the AI assistants the file exists for. The file is popular, just not with the audience it was built for.

There was one more signal we’d gotten excited about, and this is where our first read of the data was simply wrong. /llms.txt showed up as the referrer on a batch of page views, which we initially took as a client reading the file and following the links inside it, exactly the behavior the format was designed for. So we broke that referrer traffic down by user agent, too.

The story fell apart. Of 117 referred hits, 106—more than 90%—came from a single client sending a frozen Chrome/111.0 string, the tell-tale signature of a bot rather than a browser or an AI agent. The rest were requests from raw Google Cloud IP addresses and named crawlers. Almost none of it was an AI assistant genuinely reading the file and acting on it.

So the honest conclusion flips. For us, Mueller was basically right. llms.txt gets fetched, but by bots and crawlers, not by the AI assistants it’s meant for. Only a few dozen of the ~770 direct fetches came from a named AI client, and the referrer traffic that looked like link-following was one stale-Chrome bot. The file costs five minutes to ship and it’s harmless to leave up—but we can’t point to meaningful AI use of it, and the “nobody meaningfully uses llms.txt” consensus held.

What surprised us?

LLMs try URLs you don’t have, but maybe you should. Hundreds of requests hit pages that don’t exist: how-to-favicon-in-2024, -2025, -2026 (agents assume we refresh the guide every year), plus dozens of invented lefthook-* and oklch-* slugs, often with .md appended. The predictable ones are really a to-do list; we should have redirected the favicon-year guesses to the current guide long ago. Your .md 404s are free signal about the titles agents expect you to have.

LLMs hallucinate your domain, too. Thousands of requests arrived for the wrong TLD—evilmartians.dev, .app, .co, .tech, .io, .eu—and even our raw origin IP. Models remember the brand and guess the extension. We own these variants and redirect all of them to evilmartians.com, which is the only reason we can see this traffic at all: we recover it instead of losing it. That’s exactly why registering the variants pays off. The extensions you don’t own are confidently wrong requests you’ll never even see.

The clever hint got zero hits. Part one’s technique #4 was a visually hidden <div> telling any AI reader where to find the Markdown version, with the URL tagged ?ref=hint so we could measure follows. In two months, across 268,000 agent requests, the number of fetches carrying ?ref=hint was zero.

We want to be fair: an agent could read the hint, strip the query string, and refetch as an untagged direct request, which we couldn’t distinguish. But the hint hands over the literal URL including the tag, so a straightforward follow would have preserved it. The honest conclusion is that we cannot attribute a single Markdown fetch to the hidden hint. It’s cheap enough to leave in place, but if you were about to invest engineering time making yours fancier, don’t. The agents that want your Markdown are asking for it with an Accept header, not reading a hidden paragraph and clicking a link.

The noise is relentless. The same tracking bucket that catches Claude Code also catches SQL-injection probes ((select(0)from(select(sleep(15…), agent-readiness scanners testing their own product against us, and strings of pure junk like @@0nk6U. This is the tax on measuring: your “AI traffic” report is a crime scene until you clean it up.

What we’d change, knowing all this

We’re shipping the fixes in a separate pass, but here’s what two months of data actually argues for:

Keep your HTML clean, because your biggest AI reader only reads HTML. ChatGPT-User is 73% of the traffic and never takes Markdown. Every hour you’d spend perfecting a .md pipeline is better spent making sure your rendered HTML isn’t 80% navigation and script tags. The Markdown crowd is real but smaller, and it’s served automatically by negotiation anyway.

Treat content negotiation as the load-bearing technique. This is the mechanism actually delivering Markdown to the client that wants it (Claude Code), it requires zero site-specific knowledge from the agent, and it’s standard HTTP. The first post bet this would be the survivor. The logs make that bet look good.

Demote the decorative techniques. The hidden hint earned nothing measurable, and llms.txt drew almost nothing but bots. Don’t rip either out—they’re free—but stop treating “clever signals for AI” as a category worth investing in. The techniques that worked are the boring standards-based ones.

Split your user-agent groups before you trust your numbers. Our tracker lumped every Claude string into one bucket, which nearly hid the fact that coding agents, not crawlers, drive our Markdown. If your classification is coarse, your conclusions are wrong in ways you can’t see.

And the meta-point, above all the tactics: measurement comes first. Part one gave you the techniques to ship; this is the step that tells you which of them are working for you. You can’t reason about any of it—which format, which agent, which file—until you’re watching your own server logs. Every confident claim in the AI SEO discourse dissolves the moment you look at real traffic to a real site.

Want Claude to ship part one’s techniques for you? We packaged that guide as a downloadable Claude skill—every technique the data here backs, plus the eight anti-patterns to refuse.

There’s really no such thing as “AI traffic.” Underneath the label sits a handful of named agents with specific, measurable, and frankly inconsistent habits, and that mix will look different next quarter. Winning at AI discovery won’t come from shipping the most .md files. It comes from the boring habit of reading your own logs, and then reading them again after the agents change.

FAQ

Do crawlers really fetch .md files?

Part one said no. But on our own site, some do—about 15% of agent reads are Markdown, driven mainly by the Claude Code agent via Accept: text/markdown. The external studies part one cited weren’t wrong; they measured training crawlers, which mostly don’t. Coding agents and on-demand fetchers are a different population, and they behave differently. So, measure your own.

Which matters more, HTML or Markdown?

Both, for different clients. On our data, HTML serves ChatGPT (your likely largest AI reader) and Markdown serves coding agents. Neglect either and you lose a real audience.

Is the hidden AI hint <div> worth adding?

We can’t show it did anything. The result was zero attributable fetches in two months. That said, it’s cheap, so leaving it in is harmless, but don’t prioritize it. Content negotiation is where the Markdown actually gets delivered.

How did you track clients that don’t run JavaScript?

Server-side middleware that reads raw User-Agent and Accept headers and sends events to your analytics backend (GA4 via its Measurement Protocol, in our case—but plain log parsing or any server-side analytics works too). Client-side analytics can’t see non-rendering agents; this can.

What is GEO (generative engine optimization), and how is it different from SEO?

GEO is optimizing so LLMs and AI assistants can find, fetch, and cite your content—the same goal as classic SEO, but the audience is agents instead of search-engine crawlers. In practice the two overlap more than the hype suggests. Our largest AI reader, ChatGPT-User, wants exactly what SEO already asks for: clean, legible rendered HTML. The GEO-specific layer on top is serving Markdown through content negotiation for coding agents like Claude Code. You don’t replace your SEO with GEO; you add a thin format-negotiation layer and then measure who shows up.

Does llms.txt actually work?

Not really, at least on our site. /llms.txt was fetched about 660 times directly in two months, but when we broke those fetches down by client, only about 37 came from named AI assistants (Perplexity, ChatGPT, Claude, GPTBot, OAI-SearchBot); the other ~95% were search crawlers, SEO bots, and a swarm of other automated bots. The referrer traffic looked like link-following until we segmented it too—106 of 117 hits were a single frozen Chrome/111.0 bot, the rest raw-IP crawlers and a bot called PipericBot. So the “nobody meaningfully reads llms.txt” consensus basically held for us. The file costs five minutes to ship, so leave it up—but check the user agents behind the traffic before you believe any of it is AI.

Which agents read your site, and what they pull, is a question only your own logs can answer—everything here is just what ours said. So instrument it, look, and expect to be surprised by at least one number.