The robots have changed, and your robots.txt hasn't

There is a file on your Joomla site you have almost certainly never opened. Joomla put it there the day you installed, it has sat quietly ever since, and it is currently telling every crawler on the internet what it may and may not look at. It was written for a web where the only things crawling your site were search engines wanting to send you visitors.

That web has gone. In June this year, more than half of all crawler requests across Cloudflare's network were AI related, and non-human traffic passed fifty per cent of all internet traffic for the first time. Your robots.txt has not been told.

This is the first of two articles about who is knocking now, what they each want, and how you decide who gets in. There is also a date in September that some of you will want to act on before it arrives, so I will not bury it at the end.

Go and look at the robots.txt file on your Joomla site. I will wait.

If you have never touched it, and most of us never have, it says something very close to this:

User-agent: *
Disallow: /administrator/
Disallow: /api/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

That is the file Joomla ships. It is sensible, it is tidy, and every line in it is about tucking away the plumbing so a search engine does not waste its time. Not one line of it imagines a visitor that reads your work and never sends you a reader.

So the file is not wrong, exactly. It is just answering a question nobody is asking any more.

Think of this article as the appetiser. Here I want to go through what these files actually do, properly and from the beginning, because I think a lot of us have a fuzzy idea of which one is for what. Then who is knocking these days, and how you say no to them if that is what you want.

Next month is the main course: the how-to, with all the receipts. The step by step, the settings to change, the checks to run, and how to tell afterwards whether any of it worked. I am doing it in that order deliberately, because a list of instructions you do not understand is just a list of instructions you will get wrong.

First, though, that date.

One date, before anything else

On 15 September 2026, Cloudflare changes its defaults for AI traffic. On pages that carry advertising, two categories of bots get blocked automatically, while search crawlers stay allowed. It applies to new customers, to new sites, and to everyone on the free tier.

If your site sits behind Cloudflare, and a great many Joomla sites do, a decision is about to be made on your behalf. It may well be the one you would have made anyway. But it is worth five minutes before then to open Security, then Bots, and see what is set. Exact position changes as they add and regroup features, but if you are in your domains area, then the left hand side menu has Security -> Settings, and there you will find it.

That is the only thing here with a deadline on it. The rest of this article explains what those settings mean, so that when you look, you know what you are looking at. I will come back to the date at the end, once the words make sense.

A quick tour of the furniture

Let us start with the things you already have, because none of the new stuff makes sense without them.

robots.txt is a sign, not a lock

A crawler, sometimes called a bot or a spider, is just a program that fetches web pages automatically, follows the links it finds, and does something with what it collects. robots.txt is a plain text file at the root of your domain that tells those crawlers which parts of your site you would rather they left alone.

The single most important thing to understand about it, and the thing almost everyone gets wrong, is this: it controls crawling, not indexing. Those are different. Crawling is fetching the page. Indexing is putting it in the list of results. Google says so in plain English in its own documentation: a page disallowed in robots.txt can still be indexed if other sites link to it. You will get a result with your URL, no description, and a slightly embarrassed note where the summary should be.

If you genuinely need something kept out of search results, robots.txt is the wrong tool. You want a noindex tag, a password, or the page not to exist.

The other thing to know is that it is advisory. It is a polite notice on the door. A well-behaved crawler reads it and obeys. A badly behaved one reads it to find out where the interesting folders are. There is no enforcement in it at all, and there never has been.

It did become an official standard eventually. Martijn Koster proposed it in 1994, and it was published as RFC 9309 in September 2022, a mere twenty-eight years later.

Two practical notes. It only works at the root of the domain. And you can extend it with lines that are not part of the standard but are widely understood, the most useful being Sitemap:.

.htaccess is the lock

If robots.txt is the sign, .htaccess is the door with a lock on it. It is an Apache configuration file, and Joomla ships it as htaccess.txt for you to rename. On nginx the same job is done in the server configuration, and on Cloudflare in the firewall.

The difference matters enormously. robots.txt asks a crawler not to visit. .htaccess makes the server refuse. You match on the visitor's user agent, the short bit of text every browser and bot sends to identify itself, and return a 403 Forbidden. Nothing gets through, whether it reads your robots.txt or not. It is also cheap: serving a 403 costs almost nothing, so this is how you deal with a crawler hammering your site and slowing it down for real people.

One weakness, and remember it because I will come back to it: .htaccess still trusts what the visitor tells you it is. A bot arriving as Chrome on a Mac looks exactly like Chrome on a Mac.

Sitemaps, both of them

Here is a muddle worth clearing up, because these two things have almost the same name and completely different jobs.

An XML sitemap is a machine-readable list of the URLs on your site, with the date each one last changed. It is for crawlers, it does not make you rank better, and it helps things get found, particularly pages buried a long way down.

An HTML sitemap is a page on your site, for humans, listing where everything is. It helps visitors who cannot find something in your menu, and it helps people using screen readers. Its search benefit is a side effect: it is a flat block of internal links, which gives a crawler a short path to pages otherwise hidden behind pagination or a mega-menu.

You want both. Neither replaces the other. And Joomla ships neither, which surprises people coming from other systems, so you will need an extension. There are plenty to choose from in the Site Map category of the Joomla Extensions Directory, free and paid, and most of the serious ones will produce both formats for you.

The one thing to do here and now is point at your XML sitemap from robots.txt, since that is the file we are already looking at. Here is the line from my own:

Sitemap: https://www.softforge.co.uk/sitemap.xml

Do open that link if you have never seen a sitemap in the raw. It is not glamorous. It is a few hundred <url> entries and very little else, which is rather the point: it exists to be read by something that does not care how it looks.

Getting it in front of the right search engines is a bigger job than one line, and it turns out to matter more than you would think, so I will come back to it properly next month. Along with the bit where I admit what is wrong with mine.

The robots meta tag, per page

robots.txt works on folders. The robots meta tag works on individual pages, and it is the one that says noindex. In Joomla it is under System, Global Configuration, Site tab, Metadata, as a setting called Robots with four options: Index Follow, No index Follow, Index No follow, No index No follow. That is the site-wide default, overridable on any menu item and on any article under its Publishing tab.

The classic Joomla injury, and I have done it myself, is building a site with this set to "No index, no follow" so the client's half-finished pages stay private, and then forgetting to change it back at launch. That has always been bad. As we are about to see, it is now worse.

There is also an HTTP header version, X-Robots-Tag, which does the same job for files that are not HTML pages. If you run a document library, that is the only way to say "please do not index these PDFs". And while we are here: noarchive and nocache, still recommended in older hardening guides, are deprecated. Google no longer uses them.

So what has actually changed?

All of that is the world as it was. Here is what happened to it.

The mistake nearly everyone makes, including most of the articles you will find on this, is to treat "AI bot" as a single thing to be allowed or banned. It is not one thing. Cloudflare has settled on a way of describing the traffic that I think is the most useful framing available, and it splits into four jobs:

• Search. It reads your content so it can be found later and cited, with a link back to you. This is a search engine wearing a new coat.
• Agent. It fetches a page right now because a person just asked a question and is waiting for the answer. That is closer to a visitor than a robot.
• Training. It collects your content in bulk to train a model. It sends you nobody. It cites you nowhere. You will never hear from it again.
• Classic search. Googlebot and Bingbot, still doing what they always did, except that what they collect now also feeds AI answers.

Once you see it that way, the question stops being "AI, yes or no" and becomes something you can actually answer. Most site owners, once they think about it, want the search and agent traffic, are lukewarm about the training, and would quite like all of it to be honest about who it is.

And here is the consequence, which is the single most important paragraph in either of these articles:

Blocking the training bots costs you nothing in visibility. Blocking the search bots is how you accidentally disappear.

Two years ago the advice everywhere was "block the AI". People copied lists of user agents into their robots.txt, in good faith, and a lot of those lists have since had search bots quietly added to them. If you followed that advice in 2024 and have not looked since, there is a reasonable chance you have told ChatGPT's search crawler to go away while leaving the training crawler wide open. That is precisely the wrong way round.

Who is actually knocking

So it helps to know the names. These are the tokens you put after User-agent: in robots.txt.

OpenAI now runs four, and keeps them deliberately separate:

• GPTBot trains the models.
• OAI-SearchBot powers search inside ChatGPT. This is the one to allow if you want to be cited.
• ChatGPT-User fetches a page because a user asked. OpenAI states it is not used to decide search inclusion.
• OAI-AdsBot checks advertising landing pages and is not used for training.

Anthropic runs three:

• ClaudeBot for training.
• Claude-User for user-initiated fetches.
• Claude-SearchBot for search indexing. 

The old anthropic-ai and claude-web names are retired. Note that blocking ClaudeBot, which is what most 2024 guides tell you to do, does nothing whatsoever about the other two.

Google has Googlebot, which does everything, and Google-Extended, which is training and grounding for Gemini only.

Apple has Applebot and Applebot-Extended, the second being purely a training opt-out.

Then a long tail: PerplexityBot and Perplexity-User, CCBot (Common Crawl, whose archive feeds an awful lot of models), Bytespider, Amazonbot, Meta-ExternalAgent, cohere-ai, DuckAssistBot, MistralAI-User, and more arriving monthly.

Do they behave? Mostly. OpenAI, Anthropic, Google, Apple and Perplexity all state that their crawlers respect robots.txt. Bytespider and Meta's crawlers are widely reported not to, which means for those two a Disallow line is decoration rather than a control.

If you want to keep them out

Say you have decided you do not want your writing used as training material. That is a perfectly reasonable position and I have clients who hold it firmly. Here is the honest ladder, weakest to strongest.

One: ask. Add the training bots to robots.txt.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Note what is not in that list. No OAI-SearchBot, no Claude-SearchBot, no Googlebot. You have opted out of the training and kept every route by which somebody might actually find you.

Two: insist. Put the ones that ignore you into .htaccess and let the server refuse them.

Three: get help. That single Cloudflare toggle I mentioned at the top, under Security, Bots, is called "AI Scrapers and Crawlers". It is on the free plan, it is maintained for you, and for most people reading this it is a better answer than hand-editing lists you will never update again.

And then the honest caveat, which I think we owe each other. In August 2025 Cloudflare accused Perplexity of stealth crawling: when its declared crawler was blocked, switching to undeclared addresses and a user agent claiming to be an ordinary Chrome browser on a Mac, running to millions of requests a day. Cloudflare removed it from its list of verified bots. Perplexity denied it and called the report a publicity stunt. I have no way to adjudicate that from here and I am not going to try.

But it tells you the shape of the problem. robots.txt is a sign. .htaccess is a lock on the front door. Only something that verifies who a visitor really is, rather than believing what it says, deals with someone coming in through a window. Which is why the interesting work is now happening in cryptography: an IETF working group is building Web Bot Auth, where a bot signs its requests and proves its identity instead of merely asserting it. That is the actual fix. It is coming. It is not here yet.

The awkward bit about Google

Now for the part that nobody enjoys.

You might reasonably assume Google-Extended keeps you out of those AI Overviews at the top of the results page. It does not. Google's documentation is explicit: Google-Extended governs training and grounding for Gemini. AI Overviews and AI Mode are features of Search, and to appear in them a page needs only to be indexed and eligible to show a snippet. In Google's own words, "there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary".

What does work on AI Overviews are the ordinary snippet controls: nosnippet, data-nosnippet for a single passage, max-snippet to cap the length, and noindex for the nuclear option. Google's position, stated plainly, is that "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control".

Which means the only way to stay out of Google's AI answers is to damage your ordinary presence in Google. That is not a loophole anyone has failed to spot. That is the deal, and it is worth stating out loud rather than pretending there is a clever setting.

Four different people trying to write the same rule

Beyond simple yes and no, there are now four separate attempts to let you say something more precise. They overlap, they compete, and they get confused with each other constantly, so here they are in one place.

Content Signals, from Cloudflare, launched in September 2025, is an extra line in your robots.txt:

Content-Signal: search=yes, ai-train=no

Three signals: search for building a search index, ai-input for feeding your content into a model to answer a question in real time, and ai-train for training. Cloudflare switched it on by default across more than 3.8 million domains, and pointedly left ai-input unset rather than guess on anybody's behalf. It also attaches a claim, in capital letters, that restrictions expressed this way are express reservations of rights under Article 4 of the EU copyright directive.

TDMRep, from a W3C community group, is the one Yann Gomiero covered here in May 2024 in Safeguarding Your Joomla Content Against AI Crawlers. A file at /.well-known/tdmrep.json, or a meta tag, with the same legal hook: a machine-readable reservation of rights under European law. It has real traction in Europe.

RSL, or Really Simple Licensing, arrived in September 2025 and was finalised that December. Modelled on RSS, it goes past permission into actual terms: a subscription fee, a pay-per-crawl fee, or a fee each time your content is used in an answer. Reddit, Yahoo, Medium and Quora have backed it. Whether anybody pays is another question entirely.

AIPREF, at the IETF, is the grown-up attempt to replace all three with one standard, using a Content-Usage header and a robots.txt rule. It is on the standards track. It is not finished.

My honest read: none is enforceable today, all are statements of intent, and the European ones have the sharper edge because they hook into copyright law rather than good manners. If you want to put your position on the record, the Cloudflare line costs you one row in a text file.

Back to that date

So now the September change should make sense.

From 15 September, on pages carrying advertising, Cloudflare blocks Training and Agent by default, and keeps allowing Search. Read against the four jobs: your content stops being collected for model training, and it also stops being fetched live when somebody asks an assistant about you and it goes looking. The crawlers that can cite you with a link still get through.

For a lot of sites that is a sensible default you can happily leave alone. But notice the middle one. If you want the agent traffic, because you would rather an assistant read your actual opening hours than guess at them, the new default is not what you want, and the time to change it is before the fifteenth.

I am not going to tell you which way to set it. There is no single right answer, and it differs enormously between a charity that wants every possible route to being found and a photographer whose pictures are their living. Only that a default is about to be chosen for a great many sites, and it is better to choose it yourself.

The short version, so far

1. Look at your robots.txt. It is probably still the shipped default, with no Sitemap: line and nothing about AI at all.
2. Check Global Configuration is set to Index, Follow. Yes, really check.
3. Get a sitemap extension, and publish both the XML and the HTML version.
4. Decide about training bots deliberately. Block them if you want to. It costs you nothing.
5. Do not block the search bots. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are how anybody finds you at all.
6. Add a Content-Signal: line if you want your position on the record.
7. Look at your Cloudflare settings before 15 September.

Next month

That is the defensive half: who is knocking, and how to say no politely, firmly, or with a lawyer's phrasing.

Most of us can manage at least one Oscar Wilde quote at a party. I am going to spend one of mine here, because he got to the point about a hundred and thirty years before any of this did:

There is only one thing in the world worse than being talked about, and that is not being talked about.

Lord Henry says it in the first chapter of The Picture of Dorian Gray, and he says it to talk a reluctant painter into exhibiting his work. Which is almost too neat, given where we have ended up.

Because everything in this article has been about the first half of that sentence. Who is talking about you, on what terms, and how you tell them to stop. And for a photographer or an illustrator, that is genuinely the whole question.

But for most of the sites we build, the local business, the charity, the society with a committee and a newsletter, the risk is squarely the second half. Not being read too much. Not being read at all. Two thirds of Google searches now end without anybody clicking anything, and that number is going the wrong way.

So next month, the receipts. How to make sure that when an answer gets assembled somewhere, your site is one of the sources it is assembled from. There is one technical thing that matters more than all the rest put together, and it is not the thing currently being sold to you. There is a feature Joomla 5 quietly added that we never made enough noise about. There is a way to tell whether it is working, and it is not where you would look for it. And there is work happening in Joomla right now, which David Jardin wrote about here in February in Evolving Joomla: Joomla CMS MCP Server, Opening the Door to AI-Powered Administration, that points at where all of this ends up.

To be continued…