Googlebot vs ChatGPT: who is really crawling your online shop?

Google: steady, constant and thorough

Google visits with a family of crawlers, each with its own job:

With a few smaller Google fetchers added, that comes to about 105,000 genuine Google requests a day, or roughly 73 a minute, day and night.

Googlebot's traffic is remarkably flat. Its quietest hour, in the small hours of the morning, still saw 60% of the traffic of its busiest hour. Google crawls continuously and doesn't come in bursts.

Does Google download images?

It depends on which Google crawler you mean.

• Googlebot (Search) is almost entirely about pages: 86% HTML pages, 9% CSS and JavaScript (used to render the page as a browser would) and 4% images.
• Googlebot-Image is the opposite: 95% images.
• Storebot-Google, the Shopping crawler, renders product pages in full: 55% images, 38% CSS and JavaScript and 7% pages. It checks that your product photos and prices match your feed.
• GoogleOther fetched 58% images.

Across the whole Google family, about a third of all requests were for images.

One more detail: about a third of Googlebot-Image's requests got a "404 Not Found". Google remembers image addresses for a long time, well after a product photo has been renamed or deleted.

OpenAI: three crawlers, three behaviours

OpenAI runs three separate crawlers, and they behave very differently:

That is about 17,300 genuine OpenAI requests a day, roughly one sixth of Google's volume.

What each one fetches

• OAI-SearchBot is an image-heavy crawler: 64% images and 30% pages. ChatGPT search shows product pictures in its answers, and this crawler collects them.
• GPTBot fetched 64% pages and 35% images.
• ChatGPT-User fetched 98% pages. When someone asks ChatGPT about your product, it reads the text of the page and leaves the pictures alone.

When they come

The hourly patterns tell a story.

• GPTBot arrives in bursts. 81% of its ordinary requests landed between 01:00 and 06:00, and its daily total swung from under 100 to more than 18,000. It works through a queue and moves on.
• OAI-SearchBot is fairly steady around the clock, like Googlebot.
• ChatGPT-User follows a human day. It is quiet overnight, picks up from about 08:00 and stays busy until late evening, with 79% of its requests between 08:00 and 23:00. Its daily total barely moved, from 1,685 to 1,925. Each of these requests means a real person asked ChatGPT something that led it to your page, so this is the closest thing to "ChatGPT traffic" you will find in your logs.

The crawler that kept clicking "add to basket"

The figures above leave out one thing. On top of its 4,300 ordinary requests a day, GPTBot requested add-to-basket links about 404,000 times a day, almost all of them on three of the shops. That is 99% of everything GPTBot asked for during the week.

Many shop category pages carry a link behind each "add to basket" button, and those links include a random number so that browsers never reuse an old one. To a crawler, every random number makes a new address, so the supply never runs out. GPTBot followed them by the hundreds of thousands, and nine times in ten it was served a full page each time. Perplexity's crawler did the same on another shop, about 12,300 times a day.

Google and Bing also visited a lot of extra addresses on some shops, but those were the product-variant links from the shops' own Google Shopping feeds, which they are meant to check. We left both kinds out of every other figure in this article.

The fix is one line in robots.txt telling crawlers to stay out of basket links, for example Disallow: /*cart=ADD. It saves server time and points the crawler back at your real pages.

A word on fake bots

Any program can call itself "Googlebot" or "ChatGPT". It only has to say so in the User-Agent header it sends. So before counting anything, we checked every request against the official IP address lists that Google, OpenAI, Microsoft, Perplexity, Apple and Common Crawl publish.

Seven in ten requests calling themselves ChatGPT-User did not come from OpenAI.

The impostors were not after product pages. Across all the names we could check, fake requests ran to about 46,700 a day, and:

• 62% were hunting for secrets, such as stored passwords, configuration files and hidden admin folders. No genuine search or AI crawler asks for those.
• 88% came from rented Google Cloud servers. Anyone can rent one, so an address there tells you nothing about Google's own crawlers, and fake Googlebot traffic came from Google Cloud too.
• Only 15% got a normal response. Most were turned away or asked for files that do not exist.

The same addresses turned up under name after name. Of the 25 busiest addresses behind fake GPTBot traffic, 17 were also among the busiest behind fake ChatGPT-User, Applebot and Common Crawl traffic. It looks like a single scanning operation rotating through a dozen well-known bot names, hoping one of them gets waved through.

Takeaway: never let a bot through on its name alone. Check its address against the list its owner publishes. Every other figure in this article counts genuine traffic only.

The other AI and search crawlers

For context, here is the rest of the field across all the sites:

A few things stand out.

• Bing outpaces OpenAI. Bing's index also feeds Microsoft Copilot and several other AI assistants, so it matters more than its search market share suggests.
• Apple and Perplexity each crawl more than all three OpenAI crawlers put together.
• Several crawlers were mostly turned away. About ten of the sites, including several of the busiest, refuse ClaudeBot, Bytespider and Amazonbot at the server, and nine refuse GPTBot. Blocking AI training crawlers is a business decision, and each site owner makes it for themselves.
• Treat unchecked names with care. For the crawlers we could not check, a large part of the traffic carried the scanner's pattern. Of the requests using Anthropic's newer Claude-User and Claude-SearchBot names, about 7,800 a day, almost three in four were hunting for configuration files.

What this means for your shop

1. Google is still the main event. Its crawlers make about six times as many requests as OpenAI's, and they come all day, every day. Fast, clean, crawlable pages still matter more than anything else.
2. Images matter to both Google and ChatGPT, through specific crawlers. Search crawlers read your text. Shopping and AI-search crawlers, Storebot-Google and OAI-SearchBot, collect your product images. Clear file names, alt text and stable image addresses pay off in both places.
3. ChatGPT-User is your "real person" signal. If you want to know whether ChatGPT is sending people your way, look for it in your logs. It keeps human hours because it acts for humans.
4. Keep crawlers out of your basket. If your category pages have add-to-basket links, a robots.txt rule stops AI crawlers burning hundreds of thousands of requests on them.
5. Treat bot names with suspicion. Up to seven in ten requests using a ChatGPT name were fake, and most fakes were looking for passwords. Any security rule that lets bots through by name should check the address too.
6. You control the AI training crawlers. You can allow or block GPTBot, ClaudeBot and the rest separately in robots.txt without affecting Google Search or ChatGPT search. Blocking GPTBot, the training crawler, leaves OAI-SearchBot, which powers ChatGPT search, free to visit, and the reverse is also true.