20/10/2026
Added OpenAI's crawler to your robots.txt and assumed ChatGPT can't read your pages anymore? We hate to be the ones to tell you, but that assumption is probably wrong.
There are actually three different OpenAI agents, and they don't all play by the same rules. GPTBot and OAI-SearchBot honour robots.txt. But the ChatGPT-User fetch bot — the one that grabs a live page when someone shares or asks about your link — treats that as a user action, not a crawl. So your disallow line may simply not apply.
If you really want to control what ChatGPT reads, you have to enforce it somewhere the bot can't ignore: your web server, a WAF, or authentication. We wrote up what works, what doesn't, and how to do it without accidentally locking out Googlebot and hurting your search traffic. Have a read here:
https://tpc-hosting.com/blog/block-chatgpt-fetch-bot-robots-txt