🤖 Overview

ScooperBot is a web crawler operated by Scooper, Inc., a news aggregation and personalized content platform headquartered in San Francisco. First documented in 2019, the bot systematically indexes news-oriented websites, blogs, and RSS feeds to collect article metadata, headlines, and full-text content for the Scooper news application and its recommendation algorithms. It is a legitimate automated agent and is not associated with malicious activity.

🌐 Technical Behavior

ScooperBot uses a breadth-first crawl strategy with a default Crawl-Delay of 10 seconds, adjustable via robots.txt. It employs HTTP/1.1 and HTTP/2 protocols, originating from IPv4 addresses in the 104.28.0.0/14 netblock (Cloudflare IP space). The crawler respects the If-Modified-Since header and sends GET requests with headers such as Accept: text/html,application/xhtml+xml. It does not execute JavaScript or load images/CSS, focusing solely on text. The bot operates 24/7 but adapts its rate based on server response times.

📋 robots.txt Compliance

ScooperBot fully complies with the Robots Exclusion Standard, honoring both Disallow and Allow directives. According to Scooper’s public documentation (https://scooper.news/robots.txt), the bot also respects Crawl-Delay values. There are no reported violations; the company maintains an official policy page.

🔍 Detection Indicators

Identify ScooperBot by the User-Agent string Mozilla/5.0 (compatible; ScooperBot/1.0; +https://scooper.news/bot) or simply ScooperBot/1.0. It sets a custom header X-Scooper-Bot: true and a From header with [email protected]. Behavioral fingerprints include a 10-second default interval and a tendency to fetch RSS feeds (.xml/.rss) before article URLs.

📊 Data Usage

Data collected by ScooperBot powers the Scooper news aggregation platform, including personalized feeds, trend analysis, and recommendation models. It indexes titles, dates, authors, and full text to build a searchable index. Scooper states that collected data is not used for LLM training or sold to third parties; it solely supports real-time news summaries and topic clustering for end users.

⚙️ Rate Limiting Policy

While ScooperBot is legitimate, rate limiting is recommended to protect server resources. A threshold of 20 requests per minute per IP is advised; sustained rates above 50 requests per minute should trigger a temporary block. The policy ensures equitable access for human users and other crawlers, preventing small sites from being overwhelmed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.