AI accessibility

How can I determine if my website is unknowingly blocking AI services like Claude and Perplexity?

Learn how to check if your website is blocking AI tools like Claude or Perplexity, why it happens, and get actionable steps to allow access for improved analytics and growth.

Mohamad Eldib

Mohamad Eldib

— 2 min read
Share

Quick Answer

Your website may be blocking AI services like Claude and Perplexity if firewall, CDN, or security plugins are rejecting their user agents or IPs. To check, monitor server logs for those AI tools, test accessibility, and adjust security rules to allow legitimate requests. Ensuring access enables operational efficiency and sustainable growth.

Why This Happens

Overly aggressive security configurations—such as bot mitigation, firewalls, or CDN rules—often block AI user agents or IP ranges, mistaking them for threats. Lack of detailed logging on these blocks keeps you unaware of restricted AI access.

Step-by-Step Solution

  1. Enable Detailed Logging
    Configure server and CDN logs to capture full user-agent strings and source IP addresses. Focus on identifying attempts from known Claude and Perplexity user agents or ranges.
  2. Audit Security Rules
    Review firewall, CDN, and security plugin settings. Temporarily disable or relax bot prevention and rate-limiting rules to test if AI access improves.
  3. Whitelist Legitimate AI Traffic
    Add explicit allow rules or whitelist entries for Claude and Perplexity's user agents and published IP ranges in your firewall and CDN tools (Cloudflare, AWS WAF, ModSecurity, etc.).
  4. Actively Test AI Access
    From command line or HTTP clients, emulate Claude and Perplexity requests using their headers. Alternatively, use their APIs to attempt accessing your endpoints and confirm successful responses.
  5. Automate Monitoring
    Deploy workflows in tools like n8n or Make.com to periodically test AI accesses and alert you on failed attempts or new blocks.

ROI

By granting AI services like Claude and Perplexity reliable access, you gain better analytics, automated insights, and streamlined operations. This can increase efficiency and cost optimization potential by ~20% through smarter data-driven decisions and automation capability.

Watch Out For

Broad whitelisting may open up exploitable traffic paths if not tightly scoped. Silent failures can persist if you do not log or monitor these access attempts continually.

When You Scale

Doubling your traffic volume increases the challenge of distinguishing legitimate AI requests from general bots. You'll likely need an API gateway or advanced bot management solution to maintain secure, selective access for trusted AI tools.

FAQ

Q: How do I know if my firewall is blocking AI crawlers?

A: Check your firewall and CDN logs for denied requests with Claude or Perplexity user agents or known IPs. If you see '403 Forbidden' or similar errors linked to those services, they are likely being blocked.

Q: What user agents or IPs do Claude and Perplexity use?

A: Both services publish documentation listing their official user agents and, sometimes, IP ranges. Consult their respective websites or support to obtain current identifiers for allow-listing in your systems.

Q: Will allowing these AI services compromise my site security?

A: Not if you restrict access to only the official documented user agents and published IPs, and continue to monitor traffic. Overly broad rules can increase risk, so scope your allowances carefully.

Read more