Why ChatGPT May Block Your Website—and What Businesses Can Do

A website can follow standard crawler rules and still be blocked from ChatGPT’s browsing tools. A Search Engine Journal investigation describes a separate OpenAI site-safety check that can prevent an AI agent from opening a site, regardless of robots.txt settings or a user’s permissions.

That matters to business owners who want customers and AI tools to access their web pages. It also matters if your business has rules about whether its content can be used to train AI models. New shared opt-out plans may help with that second issue, but they do not control browsing access.

Two different questions: can an AI crawl or browse your site?

Website owners often discuss AI access as though it were one setting. In practice, the source describes separate crawlers and browsing tools, with different roles and rules.

OpenAI lists GPTBot for model training, OAI-SearchBot for search features, and ChatGPT-User for direct user requests. These crawlers publish IP addresses and follow standard web rules. But they operate separately from the safety rules that govern AI agents browsing the web.

The reported safety check, called Aura in OpenAI’s systems and codebase, acts as another gate. It decides whether autonomous tools can visit a link. If the site is flagged, tools such as ChatGPT live search and Codex Browser Use may refuse to open it. A permissive robots.txt file does not override that decision.

This distinction is useful when diagnosing a problem. Crawler access, user-requested visits, AI search visibility and agent browsing are not necessarily the same thing. A site may receive crawler visits while an agent still cannot load its pages.

What the reported block looks like

The Search Engine Journal report describes an international media publisher whose website ChatGPT repeatedly failed to load from May 2026. Server logs showed that ChatGPT traffic never reached the site. Tests with Codex tools also returned a message identifying site-safety policy as the reason for refusal.

The publisher had open robots.txt rules and had set Chrome permissions to “Always allow.” Neither helped. In comparison tests, competing newspaper and magazine sites loaded, as did another publication page using the brand’s technology but hosted on a different domain. The report says the tested pages remained blocked weeks later.

The article describes a technical status check at `chatgpt.com/backend-api/aura/site_status`. A block decision is saved locally for 24 hours, according to the report. It also says that explicit blocks are strict: custom allow rules within a company’s systems do not override OpenAI’s internal decision.

OpenAI’s help information, as described in the report, says agents may be blocked for security, safety and compliance reasons using a global blocklist. The report says OpenAI has not publicly explained how the system works in detail or provided a public way to learn why a site was blocked or request a formal review. It notes that the publisher’s issue was being checked by specialist support teams.

The report does not establish why the publisher was blocked. It says some researchers think prompt histories might accidentally trigger site-wide blocks, but presents this as a possibility, not a confirmed cause. Business owners should not assume that a block means their site is unsafe or that a robots.txt change will solve it.

What this means for your business

If your business depends on people finding information through AI tools, a browsing block could make some pages unavailable to those tools. That is different from a search ranking change, and different from whether a crawler can visit your site. The report does not say how common these blocks are; it says most websites do not encounter them, while some established companies have.

The publisher’s crawler figures underline why the distinctions matter. Its site received about 5,000 successful connections from OpenAI’s training crawlers over the preceding 30 days, compared with Google at 477,000 and Anthropic at 564,000. Those figures concern crawler connections, not agent browsing. They do not tell you whether your own site is accessible or visible.

Treat access as one part of your website health, not as a substitute for clear content or a good user experience. A website audit can help you review your site’s structure and identify issues that affect ordinary visitors. It cannot determine or change OpenAI’s internal safety decision.

It is also worth separating site access from training preferences. In a separate Search Engine Journal report, Google, Microsoft and OpenAI said they planned to honor a shared robots.txt line for AI training preferences. The proposal came from an Internet Draft submitted to an IETF working group. The draft is not an official IETF standard, and the three companies did not yet support the line at the time of the report.

The proposed `__usage_ai-training` policy token uses ordinary Allow and Disallow rules to express a training preference. It does not control crawling. Existing robots.txt rules do not count as a shared training preference, and a file without the new policy group means no preference under the draft. Google and Microsoft also described different plans for handling the control. These details may change as the proposal develops, so do not treat it as an active, universal control today.

Practical steps to take now

Start with evidence, not assumptions. If an AI tool cannot open your site, check whether the request reached your web server. Save the exact error message, the affected URL and the date. Compare access from the tool with ordinary browser access, and note whether the issue affects one page, a subdomain or the whole domain.

Then check the basics you control. Review robots.txt and your site’s access settings, but remember that these do not override the reported OpenAI safety block. Keep a record of relevant changes so you can distinguish a site configuration issue from an upstream restriction.

If you have a support channel for the platform, provide concise evidence and ask whether a site-safety restriction is active. The report describes support escalating the publisher’s case to specialist teams, but does not describe a public appeal process or promise a resolution. Avoid repeatedly changing settings without evidence that they are responsible.

For content training preferences, monitor the shared opt-out proposal and the providers’ own documentation. Do not assume that an existing robots.txt rule expresses the new shared preference. If AI use of business content raises policy questions, AI compliance support can help organise evidence and map policies to controls for expert review.

What to do next

1. Test the problem carefully. Record the tool, URL, error and date. Check server logs to see whether a request arrived.
2. Separate the access type. Identify whether the issue involves a crawler, a user request, search features or an agent browsing a page.
3. Review what you control. Check robots.txt, page availability and site permissions, while recognising their limits.
4. Contact support with evidence. Ask whether a safety restriction is active, and keep a record of the response.
5. Track training preferences separately. Follow the shared opt-out proposal and confirm provider support before relying on it.

The key lesson is simple: a site can be technically open to crawlers and still be inaccessible to an AI browsing agent. Clear records and a careful diagnosis will help you understand which access problem you are actually facing.