Your HubSpot site is already readable by machines. The question is which machines, what they are allowed to do with what they read, and whether the parts you care about are reachable at all.
Three different kinds of crawler now hit a marketing site, and they want different things. Some collect pages to train a model. Some build a search index that an assistant queries when it answers. Some fetch a single page live, in the moment a person asks a question. They obey different rules, and a few of them obey none. Most guidance on this topic treats them as one thing and gives you a file to upload.
This is a practical guide to what a HubSpot CMS portal exposes to those crawlers, what you can control from inside HubSpot, and which of the popular tactics have now been ruled out in writing by the companies that would have to honour them. If you have already read the guide to adding llms.txt to your HubSpot website, treat this as the follow-up, because the ground moved. The work here is mostly settings rather than code, though the server-rendering section touches templates, and HubSpot CMS development covers it when the change is bigger than a paste.
In June 2026, Google added a section to its AI optimization guide in Search Central that settles several arguments at once. It is worth reading the source, but the load-bearing sentence is this one:
"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."
The same page sets out how a page becomes eligible for AI Overviews and AI Mode. It has to be indexed and eligible to appear in Search with a snippet, and it has to meet the ordinary Search technical requirements. There is no second pipeline. AI Overviews are grounded in the regular Search index, with the model firing off related queries behind the scenes and pulling results for each.
Two practical consequences follow, and both contradict advice that circulates widely.
The first concerns structured data. Google states that it is not required for generative AI search and that there is no special schema.org markup to add. Keep your structured data for rich results, which is a real benefit, but adding schema will not buy you a place in an AI answer.
The second concerns Google-Extended, which is regularly described as an AI Overviews opt-out. It is not. Google-Extended controls whether your content trains Gemini models and grounds answers in Gemini Apps and Vertex AI. Google states plainly that it does not affect inclusion in Search and is not a ranking signal, and its documentation does not mention AI Overviews at all. If you want to limit what Google shows from your pages inside AI features, the controls are the snippet directives: nosnippet, data-nosnippet, max-snippet, and noindex. Blocking Google-Extended and expecting to disappear from AI Overviews will leave you exactly where you started.
Once you stop treating AI crawlers as a single category, robots.txt becomes a set of deliberate decisions rather than a guess.
GPTBot from OpenAI, ClaudeBot from Anthropic, meta-externalagent from Meta, and CCBot from Common Crawl collect pages that may be used to train foundation models. All four document that they obey robots.txt. Google-Extended sits in this group in intent, though it works differently: it has no user agent of its own, and the token is a control applied to crawling that Googlebot already did.
This is the group that decides whether you can be cited. OAI-SearchBot feeds ChatGPT search, and OpenAI is explicit that sites opted out of it will not appear in ChatGPT search answers. Claude-SearchBot does the equivalent for Claude. PerplexityBot surfaces and links sites in Perplexity results, and Perplexity states it is not used to crawl for foundation models. All three obey robots.txt.
OpenAI also makes the point that the settings are independent. You can allow OAI-SearchBot so you show up in answers while disallowing GPTBot so your pages are not used for training. That is the split most publishers want, and it is one robots.txt away.
ChatGPT-User, Claude-User, Perplexity-User, and meta-externalfetcher retrieve a page because someone asked for it in that moment. OpenAI and Perplexity both document that robots.txt rules generally do not apply to these, because a user initiated the action. Blocking them is possible at the network layer, but robots.txt will not do it, and Anthropic notes that disabling Claude-User may reduce your visibility in user-directed search.
One more is worth knowing about because it turns up in logs and behaves differently from the rest. Applebot-Extended controls whether Apple uses crawled data for foundation model training, and Apple documents something none of the others do: if your robots instructions say nothing about Applebot but do mention Googlebot, Applebot follows the Googlebot rules. Apple also honours the schema.org isAccessibleForFree property. A page marked as not free stays in search results, but Apple will not use it as context when generating output. That is the one place where structured data demonstrably gates AI usage.
HubSpot gives you a text box and no opinions. There is no per-bot toggle anywhere in the product, so every rule below is typed by hand.
Step 1. Click the settings gear, then go to Content and Pages in the sidebar, then open the SEO and Crawlers tab. The Robots.txt editor is there.
Step 2. Choose the domain you want from the domain dropdown at the top, then select the option to override the default settings. If you skip this, you are editing the default that applies to every connected domain, which is rarely what you want when one of them is a demo site and another is the live one.
Step 3. Write the rules. A configuration that keeps you eligible for citation while opting out of training looks like this:
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / Sitemap: https://www.yourdomain.com/sitemap.xml
Invert the Disallow lines if you would rather be in the training data. There is a reasonable argument for either position and no evidence that being in a training set produces referral traffic, so decide it on principle rather than on hope.
Step 4. Save, then load the file in a browser and read what came back. HubSpot appends a block of its own after whatever you type, disallowing preview and preferences URLs. That behaviour is not documented, so check rather than assume, and be aware that changes on OpenAI's side can take around a day to take effect.
One HubSpot-specific warning from their own knowledge base: do not pair a robots.txt Disallow with a noindex meta tag on the same page. Blocking the crawl stops the crawler from ever seeing the noindex, so the page can linger in an index you were trying to leave.
Look at the last line of the block above. HubSpot generates a sitemap for every connected domain at /sitemap.xml, and it does not add a Sitemap directive to robots.txt. That directive is the standard way a crawler discovers your sitemap without being told where to look.
This site is missing it as I write this, which is how the section earned its place. Load your own /robots.txt right now and check. It is one line, and it is the cheapest discovery win available.
While you are in there, know what the sitemap does and does not include. Website pages and blog posts are added automatically. Landing pages are not, so if you run paid traffic to landing pages you want indexed, add them by hand from the Sitemap tab under Content, then Domains and URLs. HubDB detail pages are included only when their parent listing page is.
Here is the asymmetry that decides whether any of this is worth doing. Google processes content inside JavaScript, and Apple documents that Applebot renders pages in a browser. OpenAI, Anthropic, and Perplexity document no JavaScript rendering at all. If your content only exists after a script runs, a large part of the group you are trying to reach sees an empty page.
HubSpot is on the right side of this by default. HubL is processed on the server and cached at the CDN, so the HTML that arrives already contains your text. You inherit that without doing anything, which is the single biggest advantage the platform gives you here.
It is worth confirming rather than trusting, and the check takes a moment. Open one of your pages, right click, and choose View Page Source. That is the HTML as it arrived, before any script ran, which is close to what a crawler without a browser sees. Search that source for a full sentence from the middle of your page using the browser find shortcut.
If the sentence is there, the text is in the HTML and a crawler will find it. If the source contains your navigation and a wall of script tags but not that sentence, the page is being built in the browser and those crawlers get nothing. The same check runs from a terminal with curl and grep if you prefer.
If your headings and body copy come back, you are fine. If you get navigation and nothing else, something on that page is client-rendered and worth revisiting. The usual culprits are JavaScript building blocks and HubDB data read through the browser API rather than rendered by HubL. Page speed feeds the same outcome, since a crawler that times out reads nothing, and fixing Core Web Vitals on HubSpot CMS covers that side.
Every HubSpot blog publishes a feed at /your-blog-slug/rss.xml, with variants per tag and per author. If summaries are switched off, that feed carries the complete body of every recent post inside a content:encoded element. It is a clean, structured, markup-light version of your writing, which is close to ideal for anything parsing your site.
Check the setting under Content, then Blog, then your blog, then the Subscriptions tab. The option controlling this is "Show summaries in RSS", and leaving it unchecked publishes full content. Raise the post count in the same place while you are there.
The tag and author feeds are the underused part. A feed scoped to one topic is a tidier answer to "what does this company know about X" than a generic firehose, and it costs nothing to link.
This site publishes an llms.txt and a longer llms-full.txt, and the guide to setting one up is the most-read thing here. So this section is an update to my own advice rather than a swipe at someone else's.
The state of it, as of now. Google has said in its own documentation that Search ignores these files, that creating them will neither harm nor help visibility, and it lists them among tactics to skip. OpenAI, Anthropic, and Perplexity document nothing about llms.txt anywhere, in either direction. All of them publish one for their own developer documentation, which is a pleasing detail and evidence of nothing, since publishing a file is not the same as reading one.
The best measurement available comes from an Ahrefs study in May 2026 covering a large sample of domains. Most published files received no traffic at all in the study window, the majority of requests that did arrive came from generic bots rather than named AI tools, and no crawler was observed probing for the file on sites that did not have one. That last point is the telling one. Nothing is looking for it.
So: the file costs an hour, harms nothing, and is currently ignored by everything that matters for visibility. Publish it if you want a human-readable map of your site, and do not present it to anyone as the deliverable. Spending that hour on the sitemap directive and a server-rendering check will do more.
If you already have one on HubSpot, there is a failure mode worth checking. HubSpot has no way to place a file at the domain root, so the file lives under /hubfs/ and a URL redirect points the root path at it. That redirect is easy to add for one file and forget for the other. On this site, /llms.txt resolved correctly and /llms-full.txt returned a 404 for months, because only the first redirect was ever created. Check both paths, and remember a redirect means a hop that a naive fetcher may not follow.
HubSpot's own AI is worth separating from the rest, because the documentation is narrower than the marketing.
Breeze Customer Agent is the feature that indexes your content. Its documented sources are knowledge base articles, website pages, landing pages, blogs, uploaded files, manually written question and answer pairs, and external public URLs, with a toggle that crawls other URLs on the same domain. Knowledge base articles re-sync when they change and everything else re-syncs weekly. HubDB is not a documented source.
The detail people trip on: the Customer Agent crawler obeys robots.txt, and its user agent is HubSpot Crawler. If you write a restrictive robots.txt and then wonder why your own agent cannot read your own site, that is the reason. HubSpot documents the error message and the fix.
Knowledge vaults are the other mechanism, holding files, CRM objects, and HubSpot content for Breeze to draw on, without crawling public URLs. Brand voice reads your existing posts or a URL to derive tone. Breeze Assistant works from CRM records, connected apps, and conversation history, and does not document indexing your website as a retrieval corpus. Breeze Intelligence is a separate product for enriching CRM records and reading buyer intent, which watches people visiting your site rather than reading your pages.
Two reports exist now, and neither needs a third-party tool.
Inside HubSpot, traffic analytics has a source called AI Referrals, covering visits from assistants including ChatGPT, Claude, Perplexity, Copilot, Gemini, and others, with a per-platform breakdown when you click through. This is the number that tells you whether being readable turned into anyone arriving.
In Search Console, the Generative AI performance report shows impressions and clicks from Google's AI features. Impressions there mean you were surfaced as a source. Expect the click rate to look poor compared to ordinary Search, because an assistant that answers the question well removes the reason to click. Read those impressions as evidence of being read, and judge them on their own scale rather than against your organic numbers.
Point robots.txt at your sitemap. Decide separately whether you want to be trained on and whether you want to be cited, then write both decisions into robots.txt by hand, because HubSpot will not offer you a switch. Confirm your pages render on the server, since a large share of AI crawlers do not run JavaScript. Turn off RSS summaries so your feed carries full text. Publish llms.txt if you like the tidiness, while knowing that nothing currently reads it. Then watch AI Referrals and the Search Console report to see whether it moved.
None of this is a growth tactic and none of it will manufacture demand. It removes the reasons a machine would fail to read you properly, which is a smaller claim and a more durable one.