What it means for a Dubai business
Since July 2025 Cloudflare has blocked known AI crawlers by default for new domains. Other hosts and security plugins offer similar switches. Your robots.txt can say “welcome” while the server says “no”. The bot never sees the robots.txt at all.
For a business that wants to be read by assistants, this is the most common silent problem we find. It is also one of the quickest to fix once you know where the setting lives.
Example
In our scan of five Dubai villa interior design studio sites, one refused AI crawlers at the server. Its robots.txt allowed them. The block sat in the hosting layer, so the studio’s project pages were invisible to the assistants a homeowner might ask.
Common mistake
Turning off all bot protection to fix it. You can allow the specific AI search and user agents you want and keep protection against abusive traffic. Ask your developer or host to allow those named agents rather than opening everything.