GPTBot

GPTBot is OpenAI's web crawler that collects publicly available pages which may be used to train OpenAI's models; you can allow or block it in robots.txt with the user agent name GPTBot.

Also searched as  GPTBot user agent · OpenAI crawler · block GPTBot · GPTBot robots.txt

Updated

What it means for a Dubai business

OpenAI runs more than one bot and they do different jobs. GPTBot is about training. Blocking it tells OpenAI not to use your pages for future models. OpenAI documents that it is separate from OAI-SearchBot, the crawler behind ChatGPT search, so blocking GPTBot alone does not take you out of ChatGPT’s live search results.

Many Dubai sites block GPTBot without meaning to, through a plugin or a template robots.txt copied years ago.

Example

A robots.txt with these lines blocks training but leaves search open:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

A wealth manager who does not want its research used for training but does want to appear in ChatGPT search could choose exactly this.

Common mistake

Blocking GPTBot and believing ChatGPT will now never mention you. Or allowing it and believing ChatGPT will now recommend you. Neither follows. What an assistant says about you comes from many sources, most of them outside your own site.

Related terms

Guides for your field

From the journal

Every term, A to Z