Why JavaScript-Only Links Hide Your Pages From AI Crawlers

A page that can only be reached through a link JavaScript adds after the page loads is, for most AI crawlers, a page with no link to it at all. These crawlers request the raw HTML, read the <a href> tags that are already in it, and move on without running your scripts. So if your navigation, pagination or "related posts" block is assembled in the browser, the pages behind it depend on the crawler finding their URLs some other way. Often it doesn't.

This matters more for AI answer engines than for Google, because Google renders JavaScript and most of the others, as far as anyone has measured, do not.

What has actually been measured

Three sources we could read in full on 2026-09-24, each with its own date, because a crawler measurement ages like any other.

Vercel and MERJ, published 17 Dec 2024. Vercel's engineering team and the consultancy MERJ analysed a month of crawler traffic on Vercel's network and tested rendering against Next.js and non-Next.js sites. Their summary is blunt: "none of the major AI crawlers currently render JavaScript". The list they give covers OpenAI's three agents (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic's ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. The two exceptions were Gemini, which uses Googlebot's infrastructure, and AppleBot, which renders through a browser-based crawler. The same post notes that GPTBot and Claude's crawler each spent roughly a third of their fetches on 404 pages. That is a sign of crawlers guessing at URLs rather than following a clean link graph. (vercel.com/blog/the-rise-of-the-ai-crawler, read 2026-09-24.) This measurement is now almost two years old. We cite it because it is the largest public one, not because it is current.

EdgeComet's OpenSeoTest experiment, published 15 Jan 2026, updated 17 Feb 2026. A fresh domain with no history, observed through server logs. When ChatGPT-User was sent to a URL, "It fetched only the raw HTML document": no CSS, no scripts, no media. GPTBot arrived later and did download JavaScript files, but across hundreds of requests the authors saw a single AJAX call fire. Their reading is that the downloads are ingestion, not rendering. (edgecomet.com, OpenSeoTest results, read 2026-09-24.) It is one site and one vendor's experiment, and the authors say so themselves.

Bing's own Webmaster Guidelines, which now cover Copilot and grounding results as well as search. They tell site owners to make each important URL reachable through standard HTML anchor links. They also name critical content that only appears after client-side rendering as something to avoid, and warn that content Bing cannot reliably render may not be selected for grounding. (bing.com/webmasters/help/webmaster-guidelines-30fba23a, read 2026-09-24.) That is a search engine operator stating a requirement, not an outside observation.

A study we could not read

Search Engine Land has published an experiment on exactly this question: half of a site's internal links in HTML, half injected by script, with crawler requests logged. It is being widely summarised. On the day we tried to read it, the page returned a bot-verification wall to our client, so we have not read the original. We are therefore not repeating any of its figures second-hand here. If you can open it, it is the most direct test of the link question, as opposed to the rendering question, that we know of.

Why links and rendering are two different problems

People often collapse these into one, and the fix for one does not fix the other.

• Rendering decides what a crawler sees on a page it has already fetched. If your article body is injected by a client-side framework, a non-rendering crawler fetches an empty shell.
• Links decide which pages a crawler fetches at all. A perfectly server-rendered article is still invisible if the only paths to it are a JavaScript menu, an infinite-scroll listing or a "load more" button.

The second is easier to miss. Every page on the site can pass a "view source" check and the site can still be half-invisible, because nobody checked whether the links to those pages are in the source too.

The usual culprits:

1. Navigation built from a JSON call. The header renders, then fetches the menu and paints it. The raw HTML has an empty <nav>.
2. Click handlers instead of anchors. A <div onclick="location.href=..."> or a button that calls a router. A crawler reading HTML sees no href to follow.
3. Infinite scroll and "load more". Page one of your blog is in the HTML; posts eleven onwards only exist after a scroll event.
4. Client-side related-content widgets. The block that would have linked your older posts to each other is fetched and inserted after load.
5. Tabs and accordions filled on open. Links inside a panel whose content is requested only when someone clicks it.

A sitemap softens this, since a crawler that reads it gets the URLs without following links. It is still worth having for that reason. But a sitemap tells a crawler a page exists. It does not tell it how that page relates to the rest of your site, which is part of what internal links carry.

The self-check: ask for the page the way a crawler does

You do not need a tool for the first pass. Request the page with a crawler's user-agent and look at what comes back before any script runs:

curl -s -A "GPTBot" https://example.com/blog/ | grep -o 'href="[^"]*"' | sort -u

Then compare that list with the links you can see when you open the same page in a browser. Anything visible in the browser and missing from the curl output is a link that exists only after JavaScript runs. Three refinements make the check honest:

• Count with grep -o … | wc -l, not grep -c. Server-rendered pages are often one or two very long lines, and grep -c counts matching lines, not matches. It will report 2 for a page with 80 links.
• Check the page that links, not only the page linked to. If /blog/ carries no anchors to your posts in the raw HTML, the posts themselves passing a view-source check does not help.
• Try a second user-agent. Some sites serve different HTML to different agents, sometimes by accident through a CDN rule. ClaudeBot and PerplexityBot are worth one request each.

Our free GEO checklist covers the rendering half. It fetches one page's raw HTML and reports whether the body text is there without JavaScript, whether robots.txt lets AI answer bots in, and whether a sitemap is published. It does not follow or count your links, so the curl comparison above is still the check for the link half. If you have server logs, the AI crawler log analyzer shows which AI agents actually requested which URLs. That is the ground truth this whole argument is about.

What to change

In order of how much they usually buy you:

1. Put primary navigation in the HTML. The header, footer and category links should be real <a href> elements in the server response. This one change reconnects most sites.
2. Replace click handlers with anchors. If something navigates, make it a link. Frameworks' link components render real anchors. Custom onClick navigation usually does not.
3. Give listings real pagination. Keep infinite scroll for people if you like, but also emit /blog/page/2-style URLs with anchors between them.
4. Server-render related-content blocks. They are among the most valuable internal links you have, and they are often the ones built in the browser.
5. Keep the sitemap complete and current. It is the safety net for everything above, and Bing's guidelines ask for it separately.

None of this means removing JavaScript from the site. Interactive features, personalisation and widgets can stay client-side. The rule is narrower: any route to content you want found should exist in the HTML before a script runs.

What this does and does not fix

Getting fetched is necessary, not sufficient. Once a crawler can reach a page, whether an answer engine uses it depends on what the page says and who else answers the same question. Bing and Google both frame this as ordinary SEO work, which we cover in what Google and Bing now say about GEO. To see whether engines cite you today, run the AI visibility checker. A page that is reachable and still not cited has a content problem, not a crawling one, and it is worth knowing which of the two you have before spending time on either.