Fundamentals · 26 August 2026

By Eddie Eastwood, Founder

How Do I Make My Website Readable by Answer Engines?

How do I make my website readable by answer engines?

Readability, in the technical sense this page covers, comes down to four checks: can the page render without JavaScript, is its markup machine-parseable, is any crawler blocked, and does the server respond cleanly. Get all four right and an AI crawler can fetch and parse your site the same way a person's browser does. Miss one and no amount of good writing fixes it, because the crawler never sees the writing in the first place.

Check these four things on your own site right now

Each check takes under a minute and needs no developer tools beyond your browser.

1. Check the raw HTML your browser receives, before any JavaScript runs. Right-click your homepage or a key service page and choose "View Page Source" (not "Inspect"). Search for your business name, phone number, or a sentence from your services list. If it is not in that raw HTML, it only exists after JavaScript runs, and a non-rendering crawler will not see it.
2. Check your robots.txt file. Visit yourdomain.com/robots.txt in a browser. Look for lines naming GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, or CCBot followed by Disallow: /. Any of those blocks that specific engine from your entire site.
3. Check for a firewall or bot-management tool separately from robots.txt. If your site runs behind Cloudflare or a similar service, log into that dashboard and check its bot-management or security settings. These tools can block a crawler at the network level even when your own robots.txt allows it, and the two settings are configured in completely different places.
4. Look up the HTTP status code your important pages return. A page that should exist but quietly serves a 500 error, or a single-page app that returns 200 for a URL that is really missing, both confuse a crawler in different ways. Either use a free header-checking tool or ask whoever manages your site to confirm.

Rendering: where your content actually lives

A page can arrive at a crawler with nothing in it. Google's own documentation calls this the app shell model (Google Search Central, Understand JavaScript SEO Basics). Google eventually renders JavaScript-heavy pages given enough crawl budget; most AI answer engines fetch a page once and move on, so a crawler that skips the rendering step reads an empty shell no matter how complete the page looks to a person. ^[1]

Whoever built your site can change this setting without touching the design. Server-side rendering or a pre-rendering step (a static snapshot generated in advance and served to bots) both solve it.

It's covered in more depth, including the exact view-source test, in our guide to whether AI can read a JavaScript website.

Markup: giving the crawler a clean structure to parse

Once a crawler can see your content at all, how it is marked up decides how easily it can be lifted out. Two markup layers do most of the work.

Semantic HTML. Real <h1>, <h2> and <h3> heading tags, genuine <table> markup for tabular data, and <ul>/<ol> for actual lists all tell a crawler what kind of content it is looking at, before it even reads the words. A page built entirely from styled <div> tags with no heading hierarchy gives a crawler no structural signal to work from; it has to guess where one topic ends and another begins. How to use that heading structure to write for extraction, rather than just build it correctly, is the subject of our companion page on content structure for AI visibility.

Structured data (schema.org markup, usually as JSON-LD). This is a block of code, invisible to visitors, that states facts about your page explicitly rather than implying them through prose: your business type, your FAQ questions and answers, your services, your address. Google's structured data documentation frames its purpose plainly: it provides "explicit clues about the meaning of a page" rather than leaving the search or AI system to infer it from formatting and context (Google Search Central, Introduction to structured data). FAQPage and Organization schema are the two types that do the most work for a service business, and both are cheap to add to an existing page without touching its visible design. ^[2]

Crawler access, part one: robots.txt

Even a fast, well-marked-up page is invisible to an engine you have accidentally shut out. GPTBot was blocked by 35.7% of the world's top 1,000 websites, per Originality.AI's 2024 analysis (Originality.AI, reported by PPC Land, Top websites increasingly block AI Web Crawlers). ^[3]

Blocking one AI crawler does not block them all, and the distinction decides whether the fix even helps you. OpenAI runs separate bots for training and for ChatGPT's search results, controlled independently in robots.txt, so a Disallow aimed at training data can leave your ChatGPT search visibility intact, or remove it, depending which name is on the line. Which AI crawlers to allow and which to refuse works through each bot family and what blocking it costs.

A robots.txt block is also not the only way to keep a page fully out of a crawler's results in the way that matters for search visibility. Google's own robots.txt documentation is careful to note the file's actual role: it "is used primarily to manage crawler traffic to your site" and "is not a mechanism for keeping a web page out of Google" (Google Search Central, Robots.txt Introduction and Guide); a disallowed page can still have its URL appear in results if other sites link to it, just without a description. For AI visibility specifically, the practical read is simpler: a Disallow: / line for an AI crawler is close to an absolute block on that engine ever citing the page's content, because there is no alternative "found it through a link" path the way there is for classic Google indexing. ^[4]

Crawler access, part two: firewalls and bot-management tools

Robots.txt is a request, not a wall. It relies on the crawler choosing to obey it. A separate way a page ends up invisible to AI crawlers is a firewall or bot-management service blocking them outright at the network level, regardless of what robots.txt says.

Cloudflare now ships a single toggle that blocks all recognised AI bots, available "for all customers, including those on our free tier" (Cloudflare, Declare your AIndependence). Cloudflare's own traffic data shows how rarely site owners reach for it. In the month it measured, AI bots accessed around 39% of the top one million internet properties on its network, and only 2.98% of those properties had taken any measure to block or challenge them. The rate climbs sharply with traffic: among the top ten properties, 40% were blocking. For a business actively trying to be found and cited by AI tools, that toggle being switched on, whether deliberately by an agency locking down a site against scraping, or accidentally inherited from a security-first default, silently undoes every other fix on this page. It is worth checking, because it will not show up in a robots.txt read and often will not show up in a Google Search Console crawl report either, since it is blocking a different set of bots at a different layer. ^[5]

The practical rule: check robots.txt and any firewall or bot-management dashboard as two separate steps, not one. They are configured in different places, by different people, often at different times, and either one alone can block a crawler the other one allows.

Server health: status codes a crawler can trust

A crawler needs to be told clearly when something has gone wrong, not left to guess. Google's JavaScript guidance puts it plainly: "Googlebot uses HTTP status codes to find out if something went wrong when crawling the page," and recommends using "a meaningful status code, like a 404 for a page that could not be found or a 401 code for pages behind a login" (Google Search Central, Understand JavaScript SEO Basics, cited above). The same guidance flags a specific trap for modern, JavaScript-routed sites: a client-side "single-page app" can quietly serve a 200 OK success code for a URL that no longer exists, because the server itself never got the chance to say otherwise; Google calls this a soft 404, and a crawler that trusts the status code has no way to know the page is actually gone. Fixing it means either redirecting a genuinely missing page to a URL the server itself answers with a real 404, or adding a noindex tag once the app-level code detects the page doesn't exist. Either fix is a small, one-off change; the trap is only costly if nobody checks for it. ^[1]

llms.txt

llms.txt is a plain-text file at a site's root that gives an AI system a short, curated map of the pages worth reading. It is not one of the four technical gates above, and the independent evidence for it is weak. Our separate guide covers what llms.txt is and whether your business needs one; the short answer there is that it is a minor tidy-up, not a priority.

FAQ

Do I need to rebuild my website to fix its technical parsing?

Usually not. A rendering problem is typically a server-side or pre-rendering setting for whoever manages your site, not a rebuild. A robots.txt block is a one-line fix once you know which crawler is blocked and why. A firewall or bot-management tool silently blocking AI crawlers is a settings change in that tool's dashboard, not a code change.

How do I know if my site has a JavaScript rendering problem?

View the page source (not the rendered page) in your browser and search for text that should be on the page, such as your business name or a service description. If it is missing from that raw source, your content is being injected by JavaScript after the fact, and a non-rendering crawler will not see it.

Which AI crawlers should I make sure are not blocked?

Check your robots.txt file for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and CCBot. If AI visibility is the goal, none of these should carry a blanket Disallow: / unless you have a specific reason to keep that engine out. Also check any firewall or bot-management tool your site runs, such as Cloudflare, separately from robots.txt, because a security setting can block a crawler that robots.txt allows.

Does adding schema markup actually help answer engines?

It gives the page an explicit, machine-readable statement of what it is about (business type, FAQ content, services) rather than requiring the crawler to infer it from formatting and prose. It is a technical parsing aid; it does not replace well-structured, answer-first writing, which is a separate, content-level layer covered on our companion page about content structure.

What is llms.txt and do I actually need one?

llms.txt is a plain-text file at your site root that gives an AI system a short, curated map of your most important pages, similar in spirit to a sitemap but written for a language model rather than a search indexer. It is not one of the four technical checks above, and independent evidence for its effect on AI citations is weak, so treat it as a minor, optional tidy-up rather than a priority.

Short version, restated

Check the raw HTML, check robots.txt, check any firewall separately from robots.txt, and check that broken pages return a real error code. All four are usually settings changes, not a rebuild.

CTA

Not sure which of the four checks your own site fails? Run a free AI visibility check →, or hand the technical fixes to us.