The short answer
AI systems read websites as raw server-delivered HTML. In our measurements, none of them executed JavaScript or rendered pages like a browser. They also split sharply on how they get that HTML: Claude, Gemini, and ChatGPT fetch pages directly when asked, while Perplexity and Copilot answer from what their crawlers indexed earlier.
Both paths run through the same bottleneck: the HTML your server returns. If your content, structured data, and answers are present and parseable there, every AI system can use them. If they only exist after JavaScript runs, no AI system sees them. That is why source-level AI readability — not per-platform tricks — decides visibility.
How we measured it
The Kaistone AI Crawler Lab ran 95 controlled tests (June 2026) against pages we instrumented ourselves: server-side logging captured every request's headers, timing, IP, and user-agent, and we correlated each AI client's answers with the direct-origin request logs. Nineteen dimensions were tested per client — URL fetching, HTML text reading, JavaScript rendering, image processing, link following, subresource fetching, robots.txt and meta-noindex behavior, and consent-banner handling among them.
This is measurement, not inference: when we say a client did or didn't fetch a page, it is because the request did or didn't appear in our server logs. The full data set is published at the lab's comparison report.
Finding 1: Only some AI systems fetch pages live
| AI client | Direct fetch rate | Behavior |
|---|---|---|
| Claude (Anthropic) | 100% | Fetches the page directly on request |
| Gemini (Google) | 100% | Fetches the page directly on request |
| ChatGPT (OpenAI) | 89% | Fetches directly in most runs |
| Perplexity | 0% | Answers from its own crawled index instead of live-fetching |
| Copilot / Bing (Microsoft) | 0% | Answers from the Bing index instead of live-fetching |
Direct fetch rate = share of test runs where the client requested the target URL from our server when asked about it. Lab conditions, June 2026; behavior can change as vendors update their systems.
The practical consequence: for Claude, Gemini, and ChatGPT, the page you serve today is what gets read when a user asks. For Perplexity and Copilot, your visibility depends on what their index crawlers stored before the question — which makes crawlability, consistent third-party presence, and directory listings weigh more heavily for those systems.
Finding 2: No JavaScript. No rendering. HTML only.
Across every successful fetch in the study, no AI client executed JavaScript, loaded tracking pixels, or performed full browser rendering. They read the HTML the server returned, and nothing else.
This single finding explains most AI-visibility failures we see in audits:
- Client-rendered content is invisible: Text, prices, or FAQs that appear only after a JavaScript framework hydrates do not exist for an AI reader.
- JS-injected schema doesn't count: Structured data must be in the server-delivered HTML. Schema added at runtime by a tag manager is never seen.
- Content behind interactions is lost: Accordions, tabs, and consent walls that require a click to reveal content hide that content from systems that never click.
It is also why Kaistone's AI Readability score weighs server-rendered structure so heavily — it is the one surface every AI system actually reads.
Finding 3: robots.txt is not consulted at fetch time
In our tests, no AI client referenced robots.txt or meta noindex directives when fetching a target URL a user asked about. That does not make robots.txt irrelevant — it remains the control layer for training and index crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, which is precisely what determines whether you exist in the pre-crawled indexes Perplexity and Copilot answer from.
The takeaway is a two-layer model: robots.txt governs whether crawlers may store your content; your served HTML governs what any AI reads on demand. Both layers are checked in Kaistone's audit.
What this means for your site
Each lab finding maps directly onto one of the four dimensions Kaistone audits:
AI Readability
Because AI reads server HTML only: structured data in the initial response, semantic HTML, clean heading hierarchy, no content trapped behind JavaScript.
Answer-Ready Content
Because fetched pages get quoted: direct answers, FAQs, definitions, and comparison tables in plain text — the formats AI systems lift into responses.
Trust & Authority
Because models weigh sources: E-E-A-T signals, author data, citations, and brand consistency decide whether your fetched content gets used.
Platform Presence
Because index-based systems answer from what they stored: crawler permissions, directories, and third-party profiles determine how you appear where no live fetch happens.
This is the reasoning behind Kaistone's design choice: no synthetic prompt checks against individual platforms — one source-level audit of the signals every AI system relies on.
Frequently asked questions
Do AI systems execute JavaScript when they read a website?
No. In Kaistone's controlled measurements (95 tests, June 2026), none of the AI clients that fetched pages executed JavaScript, loaded tracking pixels, or performed full browser rendering. They process the raw HTML the server returns. Content that only appears after JavaScript runs — client-rendered text, JS-injected schema, content behind interactions — is effectively invisible to AI systems reading your page.
Do all AI systems fetch web pages the same way?
No — they split into two groups. In our lab tests, Claude and Gemini fetched target pages on every request and ChatGPT in 89% of runs, while Perplexity and Copilot/Bing did not live-fetch at all in the test setup, relying instead on their own pre-crawled indexes. Both paths lead back to the same requirement: the source HTML must carry your content, because that is what gets fetched directly and what index crawlers store.
Does robots.txt stop AI systems from reading a page?
Not at retrieval time. In our measurements, no AI client consulted robots.txt or meta noindex directives when fetching a target URL a user asked about. robots.txt remains the control layer for training and index crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) — which is why permitting those crawlers still matters — but on-demand retrieval behaves differently.
How do I make my website easier for AI systems to read?
Serve complete, semantic HTML from the server: structured data in the initial response, a logical heading hierarchy, answer-ready content in text form (not images or JS-rendered widgets), permitted AI crawlers in robots.txt, and an llms.txt summary. These are the signals Kaistone's audit scores across its four dimensions — AI readability, answer-ready content, trust & authority, and platform presence.
Why does Perplexity's 0% fetch rate matter for my site?
It means Perplexity's answers about your brand depend on what its crawler indexed before the question was asked. You cannot rely on a live fetch to correct the record. Being present, crawlable, and well-described in the sources these indexes draw from — your own site plus directories and third-party pages — is what determines how you appear.
Is this why Kaistone doesn't run test prompts through each platform?
Yes. Synthetic prompt checks sample stochastic answers that vary by run, region, and model version. The lab data shows what is stable underneath: every path into an AI answer — direct fetch or index — passes through your source HTML. Kaistone audits that source, so improvements benefit every AI system at once, including ones that launch after the audit.