Letting the right robots in

Two thirds of Google searches now end without anybody clicking anything at all. People ask, they get their answer on the page or in a chat window, and they never arrive. You cannot fix that by wishing, and you certainly cannot fix it by writing more blog posts into the void.

What you can do is make sure that when an answer gets assembled somewhere, your site is one of the sources it gets assembled from. Last month I wrote about keeping AI crawlers out and promised you the receipts this month, so here they are: what to change in Joomla, where to find it, the one five-second test that matters more than everything else put together, and how to tell afterwards whether any of it worked.

First, a quick look back. In the previous article The robots have changed, and your robots.txt hasn't I went through who is crawling your Joomla site these days and how you tell the ones you do not want to go away. Sign on the door, lock on the door, and a small pile of competing standards for saying something more precise than no. I finished by spending one of my two Oscar Wilde quotes, on the grounds that there is only one thing in the world worse than being talked about. This month is about that worse thing.

I also flagged a date, and it has now passed. The Cloudflare default change landed on 15 September. On pages carrying advertising, Training and Agent bots are now blocked by default, with Search bots still allowed, for every new domain arriving at Cloudflare. Existing sites keep whatever they already had, and can opt out of the new defaults in their Security settings. If you looked beforehand and set it the way you wanted, there is nothing to do. If you did not, it is worth two minutes now in those same settings to see where you stand, because it is the Agent category that decides whether an assistant can go and read your page when somebody asks it a question about you.

Right. Onward.

Why this half matters more for most of us

That two thirds figure deserves its actual numbers. SparkToro found that 68.01% of American Google searches ended without a click in the first four months of this year, against 60.45% in 2024. Turn it round and it stings more: the share of searches that produced a click of any kind, to anywhere at all, fell by 9.51 points in those two years. That is close to a quarter of the clicks gone. Rand Fishkin is careful to say the two studies drew on different panels, so take the trend as sound and the exact gap as approximate. The trend is quite bad enough.

Last month's article suited a photographer or an illustrator, whose pictures are their living and for whom the whole question really is how to keep people out. This month is for everybody else: the local business, the charity, the society with a committee and a newsletter. Being read too much was never their problem.

First, and most importantly: can they read your pages at all?

This is the least glamorous item on the list and by some distance the most important.

The major AI crawlers do not run JavaScript. Vercel and MERJ measured this across their network at the end of 2024 and published the results: GPTBot fetches JavaScript files in 11.5% of its requests and ClaudeBot in 23.8%, and neither of them ever executes what it downloads. They read the raw HTML your server sends and nothing else. Perplexity's crawler is the same. Googlebot does render, with a real browser engine, and so does Applebot, but they are the exceptions rather than the rule.

For Joomla this is genuinely good news, and I think we should say so more often. A standard Joomla site builds its pages on the server and sends finished HTML. We are, by default, in exactly the right shape. That is not true of every way of building a website in 2026, and it is a quiet advantage we have not been shouting about.

The risk is in what we bolt on. Content inside JavaScript-driven tabs and accordions. Article text that loads when you scroll. A "load more" button on a blog listing. Anything where a JavaScript framework assembles the page in the browser rather than on the server. If your words are not in the HTML that comes back from the server, then as far as ChatGPT and Claude are concerned, your words do not exist.

There is a test for this that takes five seconds and needs no tools:

curl -s https://yoursite.example/your-page | less

If you cannot find your article text in what comes back, neither can they. That is the whole test. Do it on your most important page before you do anything else in this article.

llms.txt: let me save you some time

You will have seen this recommended. A file at /llms.txt, written in Markdown, listing your important pages with descriptions so that AI systems can understand your site. It was proposed by Jeremy Howard in September 2024, there are already several Joomla extensions that will generate one for you, and it is being sold fairly hard as the new essential.

So, honestly: nobody who matters reads it.

No major AI company has said it uses the file in production. Gary Illyes of Google was reported from Search Central Live in July 2025 as confirming that Google does not support it and has no plans to. His colleague John Mueller was blunter, comparing it to the old keywords meta tag: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag: this is what a site-owner claims their site is about... At that point, why not just check the site directly?" And if you are old enough to remember the keywords meta tag, you will remember exactly how that story ended. OpenAI's own published advice on controlling crawlers is robots.txt, front to back, although its developer documentation site does quietly serve an llms.txt of its own.

There is one place it may genuinely earn its keep, and it is not marketing. If you publish technical documentation, a curated index of your docs saves an assistant wading through your navigation to find the page it needs. The evidence that coding assistants actually fetch these files is anecdotal rather than published, so treat it as plausible rather than proven.

So: if you run a documentation site, add one. Otherwise it is a harmless twenty minutes that will not hurt you and will not help you either, and anyone telling you it is an AI ranking factor is selling you a keywords meta tag with a new haircut.

Bing is the back door to ChatGPT

Here is a much less exciting suggestion that will do considerably more.

Microsoft Copilot runs on Bing's index, and ChatGPT's search leans on it heavily too, alongside OpenAI's own crawler. Which means Bing, the search engine we have all spent seventeen years cheerfully ignoring, is now a direct route into the AI tools most people actually use.

Which brings me back to the sitemap I left hanging last month. Pointing at it from robots.txt is the easy half. The half that actually does the work is submitting it, in both Google Search Console and Bing Webmaster Tools, and verifying your site in each. Most people do the first and never get round to the second, and the second is now the one wired into ChatGPT.

While you are in there, set up IndexNow. It is an open protocol that inverts the usual arrangement: instead of waiting to be crawled, you host a small key file on your site and then notify the search engines the moment something changes. Bing, Yandex, Seznam, Naver and Yep all take part. Google does not.

One detail worth knowing while you are setting it up: the field in an XML sitemap that earns its keep is lastmod, the last modified date. The old priority and changefreq fields are largely ignored now, whatever your extension's settings screen implies. And keep lastmod truthful, which I will come back to shortly.

Now. Last month I invited you to open my own sitemap and have a look at it, and I promised I would come back and admit what is wrong with it. Here we are. If you clicked, you will have found 339 entries, every one of them carrying a priority value that Google ignores, and every one of them carrying the same lastmod date: 18 August 2026, top to bottom. Which is my sitemap cheerfully announcing that every page on my site changed on the same day. It did not. That is the date the file was generated, and I have just told you two paragraphs above to keep this field truthful.

I am not going to pretend I noticed that before writing this article. Take it as evidence for the main argument: this stuff drifts, most of us set it up once and never look again, and the defaults our tools give us were chosen for a different web. Mine is now on the list. Go and look at yours.

Bing has also, rather to everyone's surprise, got ahead of Google on measurement. In February it launched an AI Performance report showing which of your URLs are being cited in Copilot and in Bing's AI answers, which queries pulled them in, and how that changes over time. Google announced its own AI performance report on 3 June, starting with a handful of UK sites and reaching everybody on 31 August, so you have that too now. What Google's does not give you is clicks or queries, only impressions, which is the difference between knowing you were cited and knowing it did you any good. So if you want to know whether any of this is working, Bing will currently tell you more.

I did not have that sentence on my bingo card either.

Structured data, which we already have

Structured data means adding a machine-readable summary to a page saying what it is: this is an article, this is who wrote it, this is when it was updated, this is an event on this date at this place. The standard vocabulary is schema.org, and the usual format is JSON-LD, a small block of JSON sitting in the page.

Google is clear about this, and worth quoting exactly: "You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add." So this is not a trick to get cited. What it does do is remove the ambiguity, for anything reading your HTML rather than looking at your page, about what the thing on it actually is.

And here is a thing worth celebrating: Joomla 5 shipped this in core. When Dan Atrill wrote his excellent Explore the Core: Native SEO Options in these pages in June 2023, he correctly noted that Joomla had no structured data support. That changed, and I am not sure we made enough noise about it at the time.

It takes about three minutes, and rather less than you would expect, because most of it has already been done for you. The plugins ship switched on: System - Schema.org, and nine type plugins in the schemaorg group covering Article, BlogPosting, Book, Custom, Event, JobPosting, Organization, Person and Recipe. Joomla's 5.0 announcement called it "automatic activation", and it meant it.

What they do not ship with is a configuration, and until they have one they do nothing at all.

1. Go to System, then Plugins, and open System - Schema.org.
2. Set your Base Type to Organization or Person, add your site name and your social profile URLs, and save.
3. Filter the plugin list by Type schemaorg and satisfy yourself the nine are enabled. They should be. This is a look, not a job.

A Schema tab then appears when you edit an article. And here is the gotcha: it is the parent plugin that puts it there. Leave it unconfigured and the tab tells you to go and configure the plugin first; let somebody disable it and the tab never appears at all, no matter what else is switched on, and you will conclude the feature is broken.

Writing so a machine can quote you

There is a research literature on this now, which is more than can be said for most SEO advice. The founding paper is "GEO: Generative Engine Optimization" by Aggarwal and colleagues, presented at KDD in 2024. They built a benchmark of ten thousand queries and tested nine different ways of rewriting a page.

The three that worked best were adding citations to reliable sources, adding credible quotations, and adding relevant statistics, which delivered a 30 to 40 per cent relative improvement on the paper's main visibility measure, and 15 to 30 per cent on its softer one. Two methods that were nothing more than making the writing clearer, improving fluency and making the text easier to understand, gave 15 to 30 per cent on their own.

And the method that performed worst, out of all nine? Keyword stuffing. The paper's own summary is that "simple methods like Keyword Stuffing traditionally used in SEO don't perform well". I find that quietly delightful. Twenty years of people ruining perfectly good sentences, and the machines would rather you just wrote properly.

Practically, for those of us who write for the web:

• Answer first. The first forty to sixty words under a heading should be a complete answer that stands on its own. These systems pull out passages, not whole pages, and a paragraph that only makes sense once you have read the three before it gets left behind.
• Make your headings the question. If somebody would type it, use it as the heading, and put the answer directly underneath.
• Cite things and count things. A named source and a real number survive being rewritten. An enthusiastic adjective does not.
• Keep your dates honest. Do not have your CMS stamp today's date on content that has not changed. It is a short-term trick with a long-term cost.

You will notice none of that is a trick. It is just clear writing with its sources shown, which is what we should have been doing anyway.

The same protocol, pointed the other way

I want to finish somewhere more hopeful, because both of these articles have been fairly defensive so far.

In February, David Jardin wrote in these pages about Evolving Joomla - Joomla CMS MCP Server: Opening the Door to AI-Powered Administration, and if you have not read it, please do, because it is genuinely the other half of this story.

MCP stands for Model Context Protocol. In plain terms, it is an agreed way for an AI assistant to connect to an outside system, ask what that system can do, and then do it, safely and with permission. The important word is agreed. Rather than an AI poking at your site from the outside and guessing, your site hands it a list: these are the operations that exist, this is who is allowed to run them, this is what each one needs.

The work David describes came out of a discovery sprint completed in January this year, run with Martin Kopp, Stefan Wendhausen and Niels Braczek. They built a working proof of concept: a backend interface for generating access tokens, an MCP endpoint on the site, and successful test calls that retrieved system information and purged the cache. Authentication starts with MCP-specific tokens managed in the Joomla backend, with a generic Joomla OAuth server named as the next step. It is a prototype, and the article argues it belongs in core rather than running as a standalone service, which tells you where this is meant to end up.

What I like most about it is the stated aim, which is that it should not be a "super admin only" feature but "easy to set up, lightweight, and accessible to a broad range of users". And the examples are all the jobs we put off: generating titles and meta descriptions in bulk, applying extension updates across multiple sites, building landing pages from a brief.

Now hold that next to everything in these two articles. The MCP server is about an AI helping you run your site. Everything else here has been about AI reading your site. The same underlying shift, pointed in opposite directions, and Joomla is currently a good deal further ahead on the first than on the second.

Which is worth noticing, because there is a related idea called NLWeb, from Microsoft, that turns a website into something an AI can hold a conversation with. Every NLWeb instance is also an MCP server. And it is built on things we already have: schema.org, and RSS feeds. The building blocks are sitting in our CMS right now, in the plugins I asked you to go and configure three sections ago.

The short version

1. curl your most important page and confirm your content is actually in the HTML. Nothing else on this list matters if it is not.
2. Verify in Google Search Console and Bing Webmaster Tools. Submit your sitemap to both.
3. Set up IndexNow. It is free, and Bing's index is what Copilot runs on and what ChatGPT's search leans on.
4. Configure the Schema.org plugin in Joomla 5 or 6. It is already switched on and doing nothing until you set a Base Type.
5. Write answer-first, under headings shaped like questions.
6. Cite your sources and use real numbers.
7. Add an llms.txt only if you publish documentation. Otherwise, do not lose sleep over it.
8. Check the Bing AI Performance report in a month and see whether any of it worked.

Over to you

I have been writing about AI in this magazine since 2023, when the question was whether these tools were any use for building sites at all. Three years on, the question has turned round completely, and it is now about what they do with what we have already built.

Yann Gomiero's Safeguarding Your Joomla Content Against AI Crawlers told us how to keep them out in May 2024, and that was the right article for that moment. David Jardin's Evolving Joomla - Joomla CMS MCP Server: Opening the Door to AI-Powered Administration showed us in February where the other end of it is going. I hope these two have been the right ones for this moment, which is that the answer is not the same for everybody, and that the choice is worth making deliberately rather than inheriting.

Which leaves two things I keep turning over, and I would genuinely like other people's view on them. Joomla ships a robots.txt whose last meaningful change was in 2020, which is to say it predates every crawler in these two articles. Should core ship something that at least acknowledges the web of 2026? Not a policy, because that is not ours to choose on anybody's behalf, but perhaps a commented block laying out the options, so that every new Joomla site starts with its owner having made a decision rather than inherited one. And given that the Schema.org plugins are already enabled on every site and sit inert until somebody opens them and picks a Base Type, should that question be asked during installation, next to the site name we are already typing in?

I do not know. But those are decisions the project could actually take, and they are exactly the kind of thing that gets decided by whoever turns up. If you have a view, the magazine takes comments, the forums are open, and there is a Mattermost channel with the door propped open.

Come and join in. The robots certainly have.