🤖 Overview

onetszukaj is a web crawler operated by Onet.pl, the largest Polish web portal owned by Grupa Onet (part of Ringier Axel Springer Polska). Its primary purpose is to index content for the Onet search engine (szukaj.onet.pl) and potentially feed data into other Onet services like news aggregation. First documented in 2006, it remains active as of 2025, with updates to its user‑agent string over time.

🌐 Technical Behavior

The crawler follows standard HTTP/1.1 and HTTP/2 protocols, making GET requests with a default interval of 1–2 seconds between requests. IP ranges are distributed across Polish and European datacenters, primarily from the ASN belonging to Onet’s infrastructure (AS 15967). It respects robots.txt and Crawl-Delay directives, and may send conditional GET requests with If-Modified-Since headers to reduce bandwidth. Onet.sk, a related Slovak service, also uses a similar crawler. No JavaScript execution or cookie storage is performed during crawling.

📋 robots.txt Compliance

Based on community observations and Onet’s own documentation, onetszukaj strictly adheres to robots.txt directives, including Disallow and Allow rules. It also respects the Crawl-Delay directive if specified by webmasters, making it one of the more polite Polish search crawlers.

🔍 Detection Indicators

The primary User‑Agent string is onetszukaj/1.0 or onetszukaj/2.0, often with additional comments like “Onet.pl search bot”. No unique custom HTTP headers are documented; it identifies solely via the User‑Agent field. Behavioral fingerprint: consistent request intervals, no random delays, and no JavaScript execution. The bot typically requests HTML pages, not images or CSS.

📊 Data Usage

Collected data is used exclusively for search indexing on Onet’s search engine, providing Polish‑language results. Onet does not publicly disclose use for AI training or advertising, making it a pure search crawler similar to legacy bots like Slurp or FAST. Indexed pages appear in Onet’s search results within a few days.

⚙️ Rate Limiting Policy

While entirely legitimate, rate limiting is advised because its crawl speed can impact server performance on small or shared hosting sites. A standard threshold‑based blocking strategy (e.g., >100 requests per minute) is appropriate to prevent resource exhaustion without blocking legitimate indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.