🤖 Overview

Attrakt is a web crawler operated by Attrakt GmbH, a German technology company specializing in AI-driven content discovery and semantic search solutions. First documented in public forums around 2023, this bot is deployed to collect structured and unstructured web data for training proprietary large language models (LLMs) and enhancing the company’s Attrakt AI Search Engine product, which focuses on contextual relevance and entity extraction. According to Attrakt’s official documentation (available at https://attrakt.com/robots), the crawler is designed to index high-quality public websites for non-commercial research and model improvement, similar in purpose to OpenAI’s GPTBot but with a narrower scope of German/European-language content.

🌐 Technical Behavior

The crawler employs an asynchronous, multi-threaded architecture that respects the Robots Exclusion Protocol but may send bursts of up to 10 requests per second per IP address under default settings, as noted in a 2024 GitHub repository (https://github.com/attrakt/crawler-config). It identifies itself via the User-Agent string AttraktBot/1.0 (+https://attrakt.com/bot) and uses HTTP/1.1 with keep-alive connections. IP ranges are allocated from a dedicated pool registered under ASN AS206092 (Attrakt GmbH), primarily in German data centres (Frankfurt and Berlin). The bot follows only GET requests and does not perform POST or form submissions. Crawl depth is limited to three levels by default, and it avoids binary files (e.g., .exe, .zip) unless explicitly allowed in robots.txt. According to a technical whitepaper on Attrakt’s developer site, the crawler uses a custom probabilistic scheduling algorithm to avoid overloading small sites, reducing request frequency when response times exceed 2 seconds.

📋 robots.txt Compliance

Attrakt explicitly advertises full compliance with robots.txt directives, as stated in its official policy page (https://attrakt.com/about/crawling). The bot checks for Disallow rules before each request and pauses crawling if a Crawl-Delay directive is present, honouring the value specified (e.g., 30 seconds if set). No evidence of ignoring robots.txt has been reported in security advisories or webmaster forums; however, a minor bug in version 1.0.3 (fixed in 2024) caused it to ignore Allow overrides for subdirectories – this was patched and documented in Attrakt’s changelog.

🔍 Detection Indicators

The primary User-Agent string is AttraktBot/1.0 (+https://attrakt.com/bot). Additionally, the crawler sends a custom HTTP header X-Attrakt-Crawl: true to assist server administrators with logging. In logs, it resolves to hostnames like crawler-01.attrakt.com and uses a reverse DNS PTR record of *.attrakt.com. A behavioral fingerprint is the use of ETag headers for cache validation, reducing redundant downloads. Server operators can verify authenticity via a verified WHOIS entry under Attrakt GmbH.

📊 Data Usage

Collected data is aggregated to train Attrakt’s semantic search models, improve entity recognition for German-language content, and power the company’s AI Assistant feature, which offers contextual answers from indexed web pages. According to their privacy policy (https://attrakt.com/privacy), raw page content is retained for up to 90 days before being anonymised and used solely for model training – not for re‑selling or advertising. Attrakt also publishes a monthly transparency report listing the top 1000 crawled domains.

⚙️ Rate Limiting Policy

Because AttraktBot can exceed typical crawl frequencies during initial indexing of large sites, administrators should apply rate‑limiting thresholds (e.g., 20 requests per minute per IP) to protect origin servers, as recommended in Attrakt’s own rate‑limiting guide. The policy rationale is to balance legitimate data collection with avoiding service degradation, using HTTP 429 responses with a Retry-After header to throttle the bot gracefully.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.