Key takeaway
Google added a guide on November 17, 2026 explaining how to let Google-Agent, its user-requested AI agents, through your CDN, firewall and bot protection.
On November 17, 2026, Google published a new technical guide explaining how to make sure Google-Agent, the user agent for its AI agents that browse the web on a user's behalf, can reach your website without being blocked by your CDN, firewall or bot-detection service.
What changed
Google's crawling documentation changelog lists the November 17 entry as "Added guidance on allowing Google-Agent". The new page, Allow Google-Agent traffic, explains how to let these requests through while keeping your existing security controls in place.
Google-Agent isn't new. Google's user-triggered fetchers page describes Google-Agent as being "used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." What's new is a practical guide to allowing it. The page covers three ways Google-Agent identifies itself:
- User agent. The User-Agent header contains "Google-Agent." Google lists both a mobile and a desktop user agent string.
- IP addresses. Requests come from the IP ranges Google publishes in its user-triggered-agents.json file.
- Signed requests. Google says it is experimenting with Web Bot Auth, using the identity https://agent.bot.goog, so some requests are signed. Those signed requests use HTTP Message Signatures (RFC 9421) and include Signature-Agent, Signature and Signature-Input headers.
Google also notes on its user-triggered fetchers page that, because these fetches are requested by a user, these fetchers generally ignore robots.txt rules. So if you want to control this traffic, robots.txt isn't the lever. Your CDN, firewall and bot rules are.
Who it affects
This matters most if your site sits behind a CDN, web application firewall or bot-protection service, which is common for ecommerce stores, booking sites and any business that has tightened security against scrapers. It also matters if your customers might use an AI agent to search, compare, book or buy on your site. If those requests get blocked or hit a CAPTCHA, the agent can't finish the task for the person who asked it to.
If you run a simple brochure site with no special bot rules, there may be nothing to change. It's still worth a quick check with whoever manages your hosting.
What to do now
Google's guide recommends using your provider's built-in verification for Google-Agent if it has one. If it doesn't, Google suggests verifying the user agent string, checking the source IP against the published JSON file, verifying signatures when they're present, and skipping bot challenges for verified requests. The troubleshooting section adds a few things I'd check with your developer or host:
- Rule order. Make sure your allow rules run before broad ASN blocks, managed bot rules or any policy that blocks AI agents.
- Challenges, not just blocks. Google says to skip CAPTCHAs and JavaScript challenges for verified requests, not only hard HTTP blocks.
- The whole journey. Allow rules should cover search, filtering, detail pages, cart and booking steps, plus the XHR, GraphQL or API endpoints behind them.
- Rate limits. Exempt or account for verified Google-Agent IP ranges in per-IP rate limits.
- Keep IP lists current. Keep your IP rules in sync with Google's JSON file rather than hard-coding a list once.
- Proxies and load balancers. Make sure the client IP, User-Agent and signature headers are passed through so your rules can see them.
If you'd like a second pair of eyes on this, it's the kind of thing I look at in technical SEO work and a technical SEO audit. If you're already digging through your logs, my post on how Google says to spot Search traffic in analytics and logs may help too.
Was this helpful?

Written by
Jen Cannizzo
Founder of Growthopedia, an Albany, NY SEO and digital marketing consultancy. She has run technical SEO, content and reporting programs for enterprise brands and local Capital Region businesses.

DK Creative Design ServicesCreative Studio
CribpicksBaby Gear Comparison Site
Normanside Country ClubGolf Course
Leafy PeaksCannabis Dispensary
Kurver KremeIce Cream Shop
Upstate Wine & SpiritsLiquor Store
Beer Bones TaproomBar & Restaurant
Lily & Willow Co.Houseplant & Gardening Store
Dr. Alexandra SullivanProfessional Portfolio
Side Pies PizzaPizza Delivery Service



