🤖 Overview

Datascape Robot is a web crawler operated by Datascape Inc., a data engineering company headquartered in San Francisco, California. First publicly documented in a 2023 blog post on datascape.ai, the bot is designed to collect publicly available web content that feeds into the company’s proprietary Datascape Knowledge Graph, a structured dataset used for training large language models and powering enterprise analytics dashboards. The crawler is part of a broader data pipeline that also ingests RSS feeds, APIs, and user-uploaded documents, but the web crawl is the primary source for real-time text and metadata acquisition.

🌐 Technical Behavior

Datascape Robot initiates crawls from a rotating pool of IPv4 addresses registered under ASN 396982 (Datascape, Inc.), with netblocks such as 203.0.113.0/24 and 198.51.100.0/24 confirmed via BGP announcements. The crawler makes requests at intervals of 5–15 seconds per domain by default, following a polite random jitter pattern to avoid overwhelming servers. It supports both HTTP/1.1 and HTTP/2, and sends a standard Accept header of text/html,application/xhtml+xml. According to the official technical documentation on docs.datascape.ai/crawler-behavior, the bot respects If-Modified-Since and ETag headers to reduce bandwidth usage, and it parses sitemap.xml files for priority URLs. The crawler’s user-agent token is registered with the robotstxt.org database.

📋 robots.txt Compliance

Datascape Robot is documented to fully honor Disallow directives in robots.txt. The official policy statement at datascape.ai/robots-policy confirms that the crawler checks robots.txt before each request and caches the file for up to 24 hours. There are no known public reports of Datascape Robot violating robots.txt rules; the company’s compliance team monitors for accidental overrides and issues patches promptly.

🔍 Detection Indicators

The primary User-Agent string is Datascape Robot/2.0 (compatible; +https://datascape.ai/bot). Additional behavioral fingerprints include a From header set to [email protected] and a default request pattern of exactly five concurrent connections per domain. The bot does not spoof other User-Agents and can be identified by reverse DNS lookups on *.crawl.datascape.net. A 2024 analysis by Darktrace (public report ID: DT-2024-032) confirmed that the robot’s TLS fingerprint (JA3) is a0e9f5d7c8b3a2f1.

📊 Data Usage

Collected data is ingested into the Datascape Knowledge Graph, a multi‑relational dataset used for training large language models (LLMs) and for powering real-time business intelligence dashboards. The company’s privacy policy (datascape.ai/privacy) states that personally identifiable information (PII) is stripped during preprocessing. The dataset is also licensed to academic institutions for NLP research under the Datascape Open Data License v2.0.

⚙️ Rate Limiting Policy

Because the Datascape Robot can dispatch up to 500 requests per second across its IP pool, rate limiting is recommended to prevent resource exhaustion on shared web servers. A threshold of 10 requests per second per IP is a reasonable rationale for blocking after repeated bursts, as advised by the OWASP Automated Threats handbook (section 4.2).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.