AI answers

I checked five Dubai accounting firms. Two block the crawler built to read them.

Five Dubai accounting firm sites were scanned on 24 September 2026. Two block an AI crawler. The rest have big LinkedIn followings and few reactions.

Dominik · Published · Updated

2 of 5 Dubai accounting firm homepages turned away at least one AI crawler. Our own scan of five homepages, 24 September 2026
Source: Our own scan of five homepages, 24 September 2026

TL;DR

Five Dubai accounting and audit firm homepages were scanned on 24 September 2026 using the GPTBot, ClaudeBot and PerplexityBot user agents with JavaScript off. Three read normally. One returned a Cloudflare challenge page to every non-browser request, robots.txt notwithstanding. One returned an error to GPTBot's exact user agent while ClaudeBot, PerplexityBot and an ordinary browser all got the page. Of the three that also link out to social accounts, LinkedIn followings ran from 12,618 to 19,191 people, with one to fourteen reactions on a typical post.

Ask how an accounting firm in Dubai gets clients and most partners will say the same thing: referral, reputation, the client who was happy enough to mention your name to a friend. That is still true. It is also no longer the whole answer and the gap in it is measurable today, not next year.

How do accounting firms get clients?

Mostly referral and reputation and a growing share of buyers now put a question to an assistant somewhere in that chain, before they call. On 24 September 2026 I scanned five Dubai accounting firm homepages the way a crawler sees them, JavaScript off and two were unreachable by at least one of the crawlers that assistant would use.

That is not a small technical footnote. It is the exact channel a referral now travels through on its way to becoming a call.

What was measuredResult across five firms
Readable to GPTBot, JavaScript off3 of 5
Blocked or throttled for at least one AI crawler2 of 5
Genuine llms.txt file served2 of 5
Headings phrased as the client’s own question0 of 5
Homepage links out to Instagram and LinkedIn3 of 5

Firms are not named here. This is a category finding, not a report card on anyone in particular and naming a firm’s weak result in public marketing is not something we do.

Can an accounting firm block one AI crawler without knowing it?

Yes and this is the finding that surprised me most in this scan. On 24 September 2026, one Dubai auditor’s homepage answered ClaudeBot, PerplexityBot and a plain browser request normally. The identical request, made with GPTBot’s exact published user agent string, came back with an error both times it was tried.

Its robots.txt allowed every crawler, no exceptions listed. So the block was not a policy decision anyone made on purpose. It sat somewhere further down the stack, in a security or rate limiting layer nobody had reason to check against a user agent name, because nobody was looking for it.

A second firm went further still. Its homepage returned a security challenge page to GPTBot, to ClaudeBot and to a plain browser making the same non-interactive request, every single time. Robots.txt on that site also said everyone was welcome. What the server actually served said otherwise and no amount of reading robots.txt would have told the firm that.

Does a big LinkedIn following mean an accounting firm gets found?

Not by itself. Three of the five firms link out to LinkedIn from their own homepage, so I checked those pages directly on 24 September 2026, reading the follower count, the posting cadence and the reactions on the most recent handful of updates for each one, the same way a prospective client scrolling past would see them.

LinkedIn followingPosting cadenceReactions on a typical post
19,191about once a week1 to 3
13,106several times a week7 to 14
12,618about once a week1 to 6

Between twelve and nineteen thousand people follow these pages. None of the three has stopped posting, which rules out the easy explanation. What they share instead is a following that scrolled past without stopping, on a platform whose own algorithm rewards exactly the reaction these posts are not getting. A page an algorithm stops showing to people is quietly the same problem as a page a crawler cannot read. Different layer, same result: nobody sees it.

Is SEO dead for accounting firms?

No, though the question itself is now a genuine search, not a rhetorical one and it deserves a straight answer instead of a dismissal. Ranking still routes a real share of traffic. What has changed is that ranking alone no longer guarantees a click, because a growing number of searchers read the summary above the results and stop there.

The clearest independent number on that comes from the Pew Research Center, published 22 July 2025, tracking 68,879 queries from 900 US adults in March 2025. With an AI summary present, users clicked a traditional result on 8% of visits, against 15% when no summary showed. Only 1% clicked a link inside the summary itself. Google disputes the methodology and that objection is worth stating rather than leaving out, since the sample is American and the data predates this year.

What that number does support, without needing the disputed parts: being ranked and being the one named in the summary are two different jobs now and a firm doing well at one can still be doing nothing at the other.

What would you check first?

Start with what a crawler actually receives, not what your robots.txt claims it allows. The gap between the two, on the auditor above, was the whole story. A firm can pass every policy check it wrote for itself and still be unreadable to the exact software reading it on a buyer’s behalf.

If you want to see what your own site and social pages actually show, the free AI check runs a version of this same scan and takes about ten seconds.

The honest caveat

A scan like this tells you what a crawler receives and what a LinkedIn page shows a stranger today. It does not tell you whether an assistant will name your firm when a client asks, because nobody outside those companies publishes how that selection works and anyone who claims to have cracked it is selling something.

What it does tell you is whether you have given either audience, the crawler or the follower, a reason to stop. Five Dubai accounting and audit firms here. Two closed a door they likely never noticed was there. Three kept a large room mostly quiet.

Both are fixable and neither needed a bigger following or a faster crawler to start.

Sources: Direct scan of five Dubai accounting and audit firm homepages, 24 September 2026, requested with the GPTBot, ClaudeBot and PerplexityBot user agents and JavaScript disabled; Public LinkedIn company pages for three of the five firms, checked 24 September 2026; Pew Research Center, 'Google users are less likely to click on links when an AI summary appears in the results', 22 July 2025

FAQ

How do accounting firms get clients?

Mostly referral and reputation and increasingly a search or a question put to an assistant somewhere before that referral is confirmed. A scan of five Dubai accounting and audit firms on 24 September 2026 found two of them unreachable by at least one of the AI crawlers a buyer's assistant would use to read the firm first.

Can an accounting firm block one AI crawler without knowing it?

Yes. On 24 September 2026 one Dubai auditor's site answered ClaudeBot, PerplexityBot and a plain browser normally, then returned an error to GPTBot's exact user agent string on two separate requests. Nothing in its robots.txt named such a rule. The block sat in a security layer the firm most likely never checked.

Does a big LinkedIn following mean an accounting firm gets found?

Not on its own. Three Dubai firms checked on 24 September 2026 carried LinkedIn followings between 12,618 and 19,191 people, all posting roughly weekly, all drawing single figures to the low teens in reactions per post. A following that does not react is not the audience an algorithm keeps handing the next post to.