Schema, llms.txt, and how AI engines actually read your website

When a person visits your website, they see the design. When an AI engine visits, it sees something completely different: raw HTML, structured data blocks, and a handful of special files. Whether you appear in AI answers is decided in that invisible layer. Here's a tour of it, in plain English.

First problem: can the crawler see anything at all?

Many modern sites are JavaScript applications: the HTML that arrives is nearly empty, and the content appears only after scripts run in a browser. Google runs JavaScript when it crawls. Many AI crawlers don't — or do so inconsistently. If your content only exists after JavaScript runs, you're invisible to a chunk of the engines answering your customers' questions.

The fix is server-side rendering or prerendering: the page arrives as complete HTML, and JavaScript only adds the interactive polish. This site works exactly that way — every page is static HTML first, which is also why it loads fast.

Schema markup: your site's machine-readable ID card

Schema (structured data) is a block of JSON embedded in your pages that states facts in a standard vocabulary engines understand. For a local business, the essentials are:

• LocalBusiness / ProfessionalService — who you are, where you are, what you charge, what areas you serve.
• Offer — your actual services with actual prices.
• FAQPage — real questions and answers, which engines love to quote directly. This is the heart of AEO.
• BlogPosting, BreadcrumbList — context that helps engines understand your content's structure and freshness.

The critical rule: schema must match what's visibly on the page. Engines cross-check. Markup that claims things your page doesn't say erodes trust instead of building it.

llms.txt: the newest file in the toolbox

llms.txt is a convention — a plain-text file at your site's root that gives AI systems a concise, curated summary of who you are and where your important pages live. Think of it as a menu you hand the engine instead of making it rummage through your kitchen. It's young as standards go, adoption is still growing, and it costs almost nothing to add — which makes it one of the easiest wins available. This site has one at /llms.txt; that's also roughly what yours should look like.

robots.txt still matters

AI crawlers identify themselves — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended — and they check robots.txt for permission. Some "SEO-optimized" sites block these bots without realizing it, usually via an overzealous firewall or a copy-pasted robots file. If you want AI engines to recommend you, first make sure you're not throwing them out at the door.

The checklist

1. Content present in raw HTML (view source and check).
2. LocalBusiness + Offer + FAQ schema, matching visible content.
3. llms.txt at the root, kept current.
4. robots.txt explicitly allowing the AI crawlers you want.
5. Fast loads — crawlers have budgets, slow sites get partial reads.
6. Consistent name/address/phone across the web.

This entire layer is part of every site I build, and checking it on existing sites is a standard part of my Deep Audit. It's unglamorous work. It's also the difference between being quoted and being skipped.