🤖 Overview

RedesScrapy is a legitimate web crawler operated by Redes Informáticas S.L., a Spanish technology firm, built on the open-source Scrapy framework (version 2.5+). Its purpose is to index publicly available news articles and blog posts for the proprietary Redes Insights trend analysis platform. First observed in public logs around 2020, the bot is documented in the Scrapy community with source code partially available on GitHub at github.com/redes/redes-scrapy-crawler. It identifies itself clearly and respects site owner preferences, distinguishing it from malicious scrapers.

🌐 Technical Behavior

RedesScrapy uses distributed Python workers across EU-West cloud servers (AWS eu-west-1, eu-south-2) with a default crawl delay of 10 seconds between pages on the same domain, configurable via Scrapy’s DOWNLOAD_DELAY. It sends HTTP/1.1 keep-alive requests with an Accept: text/html header and respects Cache-Control. IP ranges belong to 185.201.64.0/22 registered in RIPE for Redes Informáticas, published at redes.es/crawler-ips. Maximum 500 requests per domain per hour with exponential backoff on HTTP 429 responses.

📋 robots.txt Compliance

According to official documentation, RedesScrapy fully parses robots.txt directives for its own user-agent token, falling back to the wildcard rule. It does not honor Crawl-Delay directives, instead using its own fixed delay. This policy is documented in Scrapy’s framework as a design choice by the bot’s maintainers to manage their own rate limits.

🔍 Detection Indicators

The primary User-Agent string is RedesScrapy/1.0 (compatible; +https://redes.es/crawler). Additional indicators include the From: [email protected] header and X-Redes-Crawler: true custom header. IP addresses have PTR records crawler*.redes.es and can be verified via reverse DNS.

📊 Data Usage

Collected data powers Redes Insights for media monitoring, sentiment analysis, and keyword tracking for enterprise clients. Content is not sold to third parties nor used for AI training. Annual transparency reports at redes.es/transparency detail crawl volumes and domains.

⚙️ Rate Limiting Policy

Despite its legitimacy, RedesScrapy’s automated requests can strain server resources. Rate limiting via threshold-based blocking (e.g., 100 requests per minute per IP) is recommended to protect application performance while allowing access, as per the bot’s own guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.