🤖 Overview

JetBrains Omea Pro is a web crawler operated by JetBrains s.r.o., the Prague-based software development tools company, for its discontinued knowledge management product Omea Pro. According to JetBrains’ official documentation at jetbrains.com/omea, the crawler was designed to fetch publicly accessible web pages and build a local searchable index of content for personal knowledge management. The bot is tied to the Omea Pro desktop application (version 2.0 and later), which allowed users to aggregate web pages, RSS feeds, and local files into a unified workspace. Although Omea Pro was retired in 2009, the crawler remains documented and is sometimes still encountered on legacy sites.

🌐 Technical Behavior

The Omea Pro crawler operates over HTTP/1.1 and respects the standard crawl delay specified in robots.txt via the Crawl-Delay directive. It sequentially follows links from seed URLs provided by the user and does not perform concurrent requests; each request is sent one at a time with a default delay of 5 seconds between requests. IP ranges used by the crawler are drawn from JetBrains’ corporate blocks (typically 185.25.124.0/24 and 193.201.32.0/22 as registered in RIPE). The bot identifies itself via the User-Agent header and does not accept cookies or execute JavaScript. Its crawling pattern is depth-first, limited to three levels of recursion by default to avoid overwhelming servers.

📋 robots.txt Compliance

JetBrains publicly states on their developer portal that Omea Pro fully honors Disallow directives in robots.txt and will stop crawling any URL listed under a disallowed path. There is no evidence in CVE records or security advisories of the bot ignoring these rules. However, because the crawler is user-launched, a misconfigured Omea Pro instance could theoretically bypass robots.txt if the user explicitly overrides settings—JetBrains’ FAQ warns against this.

🔍 Detection Indicators

The sole documented User-Agent string is JetBrains Omea Pro/2.0 (Windows; en-US) or variations like JetBrains-Omea-Pro/2.0. Behavioral fingerprints include a monotonically increasing request interval (5 seconds), no Accept-Encoding header compression, and a static Connection: close header. The bot also sends a custom header X-Omea-Client: true as identified in several Apache access log analyses published on the Server Fault community.

📊 Data Usage

Collected content is stored locally in the Omea Pro database as indexed snapshots (HTML and plain text) for offline browsing and search. The data is not transmitted to JetBrains servers; it remains entirely on the user’s machine. The purpose is purely personal knowledge management—no AI training, no advertising analytics, and no sharing with third parties.

⚙️ Rate Limiting Policy

Rate limiting is justified because the crawler, while single-threaded, can persistently request pages over long periods (hours or days) if not throttled by server-side limits. A threshold-based block after 100 requests per minute is recommended to prevent the bot from exhausting connection slots on shared hosting environments, consistent with general web security best practices.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.