Skip to main content

Boteraser | Website and Server Security Solutions

ScryBot

Bot User-Agent: scrybot

🤖 Overview

ScryBot is a legitimate crawler operated by Scry, the company behind the Scry AI search product, documented at https://scry.ai/bot. It discovers publicly accessible pages that feed Scry's search index and retrieval-augmented answer pipeline; it is not user-triggered and does not execute JavaScript or submit forms. Scry's public bot page is the authoritative source for its current user-agent and opt-out instructions.

🌐 Technical Behavior

ScryBot sends HTTP GET requests over TLS, follows redirects, and fetches HTML plus limited linked resources for indexing. Scry publishes no fixed CIDR block; official docs direct validation by User-Agent, forward-confirmed reverse DNS, and Scry-owned cloud infrastructure rather than hard-coded IP ranges. It identifies itself on every request and uses per-host politeness, adaptive concurrency, and backoff on 429 or 503 responses. Requests come from distributed cloud egress addresses, so source IPs change as capacity scales.

📋 robots.txt Compliance

Scry's documentation states ScryBot honors robots.txt Disallow rules and will not intentionally fetch disallowed paths. Operators can exclude it with a targeted User-agent: ScryBot group or wildcard rule; Scry recommends robots.txt for preferences and firewall rules for immediate enforcement. The policy cites RFC 9309 for parsing and caching. Scry documents that robots.txt rules are fetched and cached per host and rechecked periodically.

🔍 Detection Indicators

The main fingerprint is the literal ScryBot token in User-Agent, commonly ScryBot/1.0 (+https://scry.ai/bot). Scry relies on no secret header, so combine UA matching with verified DNS/IP ownership and request-rate patterns because UA strings are spoofable. No CVE or security advisory is associated with ScryBot in NVD as of this writing; it is a crawler, not an exploitation tool.

📊 Data Usage

Pages fetched by ScryBot build and refresh Scry's search index and support AI-generated answers with citations. Scry's documentation describes indexing publicly available content for search retrieval; it is not an analytics beacon or advertising tracker. Site owners can review Scry's bot page for opt-out and data-use details.

⚙️ Rate Limiting Policy

ScryBot is rate-limited because unconstrained crawling can consume origin bandwidth, CPU, and database capacity; threshold blocking with 429 responses and temporary IP/UA blocks protects availability while allowing indexing to resume after backoff. The rationale is predictable crawl traffic and site-specific enforcement without permanently blocking Scry's crawler. Operators should return 429 with Retry-After when possible so ScryBot can back off gracefully.

Free Bot Analysis

Is Your Site Under Bot Attack Right Now?

Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics.

Run Free Bot Scan →

No credit card required  ·  Results in minutes

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓