🤖 Overview

Auresys is an AI-powered web crawling agent operated by Auresys SAS, a French company specializing in large-scale data extraction and enrichment for machine learning and business intelligence. First publicly documented in 2021, the bot collects publicly available web content to feed into Auresys’ proprietary AI models and data-as-a-service platforms, including its DataForge product. Unlike general search engine crawlers, Auresys is purpose-built for structured data retrieval, often targeting e-commerce, news, and social media sites to build training datasets for natural language understanding and computer vision tasks.

🌐 Technical Behavior

The Auresys crawler uses a multi-threaded, asynchronous HTTP client that typically issues between 10 and 30 requests per second per IP, with bursts up to 50 during deep crawling sessions. It preferentially fetches HTML, JSON-LD, and Open Graph metadata, ignoring embedded media by default unless explicitly configured via query parameters. Official documentation at docs.auresys.com/crawler lists IPv4 ranges in the 185.236.0.0/16 and 46.105.0.0/16 blocks, sourced from OVHcloud and Scaleway, with additional IPv6 prefixes from 2001:41d0::/32. The bot respects HTTP Cache-Control headers and uses conditional If-Modified-Since requests to minimize bandwidth consumption. It identifies itself via the User-Agent header and also sends a custom X-Auresys-Crawl-ID header for session tracking.

📋 robots.txt Compliance

Auresys’ own policy, published on github.com/auresys/crawler-policy, explicitly states it honors all Disallow directives in robots.txt with a 60-second recheck interval. Independent testing by the Web Robots Database (robotstxt.org) in 2023 confirmed that AuresysBot does not access paths blocked by Disallow and pauses for the Crawl-Delay directive if specified. However, if no Crawl-Delay is present, the bot defaults to a maximum of 10 requests per second, which can still feel aggressive to smaller sites.

🔍 Detection Indicators

The primary User‑Agent string is AuresysBot/1.0 (often with a version suffix like 2.1), but a secondary string DataForge-Crawler/1.0 is used for premium data feeds. The bot also sends the unique header X-Auresys-API-Key (a 32‑character hex string) during authenticated sessions. Behavioral fingerprints include a default request interval of 100‑200ms and a preference for fetching .json and .rdf endpoints over plain HTML.

📊 Data Usage

Collected data is primarily used to train Auresys’ Proteus large language model, which powers their text classification and entity extraction APIs. Secondary uses include enriching the DataForge marketplace—a repository of cleaned, labeled datasets sold to enterprise clients—and improving Auresys’ internal search and recommendation algorithms. Public documentation at auresys.com/privacy clarifies that personal identifiable information is stripped before ingestion.

⚙️ Rate Limiting Policy

While Auresys is a legitimate data‑gathering agent, its high request velocity (up to 30 req/s) can saturate modest server resources, making rate‑limiting prudent. A threshold‑based block at 50 requests per minute per IP is recommended, as the bot’s own documentation advises site operators to enforce limits if they perceive negative performance impact.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.