Are your Cloudflare AI crawler settings blocking ChatGPT?

Short answer: if your website sits behind Cloudflare, log in and check the AI bot controls this week. On 1 July 2025 Cloudflare changed its default so that every new domain added to the platform has known AI crawlers blocked unless the owner opts in. There is also no longer one switch to check. Cloudflare's crawler controls now distinguish bots by purpose, training, AI assistants and AI search, each with its own setting, so an owner who flipped a single toggle some time ago may still be blocking the others. A crawler that cannot read your pages has nothing of yours to quote.

This is written for owners and managers of Klang Valley firms with 2 to 50 staff and no in-house technical team: clinics, dental and veterinary practices, law and accounting firms, freight forwarders, tuition centres, property agencies. Disclosure: it comes from Geo One, a Generative Engine Optimisation (GEO) and Answer Engine Optimisation (AEO) agency in Kuala Lumpur. We sell this work, so every factual claim below links to its source.

One reason almost nobody checks this: network-level crawler settings are nobody's job until something goes quiet.

Why does a Cloudflare rule block more completely than robots.txt?

Because robots.txt is a polite request and a Cloudflare rule is a locked door.

OpenAI's own developer documentation makes this concrete. Its bots page states that because actions by its ChatGPT-User crawler are initiated by a user, "robots.txt rules may not apply." So robots.txt is not a reliable control in one direction. In the other direction, editing robots.txt does nothing to restore access that is being refused at the network edge, before the request ever reaches your server.

Scale is the other factor. Cloudflare sits in front of a large share of the web, commonly estimated at around a fifth of all sites. If your site was set up with Cloudflare at any point, this is where the decision is actually being made, not in a text file on your server.

The 30-second test before you log in anywhere

Open ChatGPT, paste your homepage URL and ask it to summarise the page. If it replies that it cannot access the site, treat that as a strong hint that something is refusing the request, and run the checklist below. It is a hint rather than a diagnosis: a slow server, a login wall or a page built entirely in JavaScript can produce the same reply. If ChatGPT summarises the page accurately, its user-triggered fetcher can reach you, but that covers one bot category only, and the others are set separately. The same test works in Claude, which we walk through in Can Claude read your website at all?

What should you check, and in what order?

Six steps. None of them require a developer, though step four sometimes does.

1. Confirm whether you are on Cloudflare at all. Many Malaysian SME sites were built years ago by an agency, a freelancer or a staff member's friend, and the domain and DNS still sit in that person's account. Ask whoever registered the domain, or check your name servers. If they point at Cloudflare, this article applies to you.

2. Open the bot and AI crawler controls in the dashboard. These settings live inside each individual domain, not at account level, so if you run a main site plus a separate booking or campaign domain, you have to check each one. The menu names move around; Cloudflare's bot documentation is the current reference for crawler controls and bot management.

3. Check all three AI categories separately. Training, agents and search are different decisions. Plenty of owners are happy to block model training while still wanting AI search engines to read the site. Our position, for what it is worth: unless you publish something genuinely proprietary, original research, a database, a course you sell, blocking training buys a 10-person clinic or law firm almost nothing, while the search and agent categories are the ones that decide whether you get named when someone asks for a dentist in Petaling Jaya. Blocking all three and then asking why ChatGPT never mentions you is a contradiction worth resolving on purpose.

4. Check your firewall rules and rate limiting too. A blanket rule that blocks non-browser user agents, or an aggressive rate limit, can catch named crawlers even when the AI toggles say allow. Anthropic publishes its crawler and user-agent details in Anthropic's crawler note, and OpenAI publishes its own on the bots page linked above. Match the names in your rules against those pages rather than against memory.

5. Then fix robots.txt. It still governs crawlers that respect it, so make it consistent with your Cloudflare settings. Two systems disagreeing is how sites end up half visible.

6. Write down what you changed and when. Record the date, the setting and the person in the same place you keep your hosting login. A site migration, a new developer or a fresh domain added after 1 July 2025 will all reset this quietly, and the default is blocked.

How this plays out for a typical KL firm

Picture a Petaling Jaya dental practice with three domains on Cloudflare: the main site, an old campaign domain from a promotion two years ago, and a booking subdomain run by a third-party system. The main site was added in 2022, so it predates the new default and allows crawlers. The booking subdomain was added in August 2025 and blocks everything. An assistant asked "can I book online with this clinic" can read the services page but not the booking page it points to. Nobody made a decision; the date of setup made it for them.

The technical fix takes minutes once the right login is found. Finding the login is often the slow part, because the domain sits in a former freelancer's or ex-employee's account. That is the usual shape of this job: the change is trivial, and the admin around it is not. If the person who set up your site has moved on, start with account recovery before anything else.

What does this not fix?

Unblocking a crawler is a necessary condition for AI visibility, not a sufficient one. If your pages have no clear answers to common questions, thin content, or no signals that establish your business as credible and local, a crawler that can now reach you will still find little to quote. The related question of why AI assistants do not name a business even when they can reach its site is covered separately in Why doesn't ChatGPT mention my business in Malaysia?

Work with Geo One: explore our services or get in touch to get started.