🤖 Overview

TransgenikBot is operated by Transgenik AI, a research organization focused on large language model training. The bot indexes publicly accessible web content to enhance Transgenik's proprietary NLP models.

🌐 Technical Behavior

TransgenikBot performs HTTP GET requests over IPv4 and IPv6, primarily from Amazon Web Services and Google Cloud Platform IP ranges. According to the Transgenik developer documentation (https://transgenik.ai/docs/crawler), it requests a maximum of 50 pages per minute per domain to avoid overloading servers. It supports gzip compression and respects If-Modified-Since and ETag headers for incremental crawling.

📋 robots.txt Compliance

TransgenikBot fully honors robots.txt directives. The official GitHub repository (https://github.com/transgenik/crawler) states that the crawler waits for a Crawl-Delay directive if specified and stops immediately upon encountering a Disallow rule. It does not ignore Allow overrides.

🔍 Detection Indicators

The primary User-Agent string is TransgenikBot/1.0 with a secondary identifier TransgenikCrawler. Additionally, the bot sets a custom HTTP header X-Bot-Identity: TransgenikBot and includes a comment linking to its opt-out page (https://transgenik.ai/block). Fingerprint: low request concurrency and consistent IP ranges from 34.0.0.0/8 and 35.0.0.0/8.

📊 Data Usage

Collected data is used exclusively for training Transgenik's generative AI models and improving search relevance. The data is not sold to third parties and is processed in compliance with GDPR and CCPA, as per their privacy policy (https://transgenik.ai/privacy).

⚙️ Rate Limiting Policy

Administrators are advised to rate-limit TransgenikBot if its requests exceed 100 per minute per IP to prevent server resource exhaustion. This threshold is based on Transgenik's own rate limiting documentation and is considered a reasonable safety measure while allowing legitimate crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.