🤖 Overview

RepoMonkey is a legitimate web crawler operated by RepoMonkey Inc. (founded 2023, headquartered in San Francisco, CA). Its primary purpose is to systematically index publicly accessible source code repositories (GitHub, GitLab, Bitbucket) and software documentation websites to build a comprehensive dataset for AI code-generation model training. The product that consumes this data is CodeForge AI, a proprietary code completion and generation service. According to official documentation at https://repomonkey.io/about, the bot was first publicly announced in March 2024 and has been in continuous operation since.

🌐 Technical Behavior

RepoMonkey employs a distributed crawling architecture using a pool of AWS EC2 instances (primarily us-east-1 and eu-west-1 regions). Official IP ranges are published in a JSON file at https://repomonkey.io/ips.json — currently 64 distinct /24 subnets. The crawler uses HTTP/1.1 with keep-alive connections, sending requests at a rate of approximately 2 requests per second per IP, but scaling up to 10 concurrent requests during initial site indexing. It follows Canonical URLs and respects Last-Modified headers to avoid re-crawling unchanged content. The bot also parses sitemap.xml files and supports gzip compression. In official public statements, RepoMonkey Inc. states that the crawler runs a 24‑hour delay between full revisits of the same repository.

📋 robots.txt Compliance

RepoMonkey fully respects the robots.txt protocol as documented in its own User‑Agent: RepoMonkey/1.0 section. The development team explicitly states on the official website that “we honour all Disallow, Crawl‑Delay, and Allow directives”. The bot also respects the X‑Robots‑Tag HTTP header and meta robots tags. Notably, RepoMonkey includes a Crawl‑Delay value of at least 5 seconds from the robots.txt file, below which it will not reduce its interval.

🔍 Detection Indicators

The primary User-Agent string used is RepoMonkey/1.0 (compatible; +https://repomonkey.io/bot). In some cases, a secondary header X-RepoMonkey-ID is sent with a unique identifier per crawl session. The bot also sends a Via header containing the string “RepoMonkey crawler platform”. Behavioral fingerprints include: requesting /robots.txt before any other page, preferring HTML content over raw files when both are available, and always including a Referer header pointing to the previous page crawled.

📊 Data Usage

All collected public code and documentation are ingested into the CodeForge AI training pipeline. The data is used to train large language models (LLMs) for code completion, bug detection, and automated refactoring. Additionally, metadata (licenses, stars, commit history) is aggregated for statistical analysis. RepoMonkey Inc. publishes a quarterly transparency report at https://repomonkey.io/transparency detailing the number of repositories indexed and the jurisdictions covered.

⚙️ Rate Limiting Policy

RepoMonkey is rate‑limited because its distributed IP pool can generate high request volumes that may overwhelm smaller servers. The recommended practice is to set a Ceiling threshold of 50 requests per minute per IP, as documented in the official rate-limit guidelines at https://repomonkey.io/rate-limits. Blocking the bot entirely is not advised — instead, implement exponential backoff responses (HTTP 429) after exceeding the limit to allow legitimate crawling while protecting server resources.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.