🤖 Overview

Kurzor is a web crawler operated by Kurzor Inc., a data analytics company founded in 2022 and headquartered in San Francisco, California. Its primary purpose is to collect publicly accessible web content for training large language models and enhancing natural language understanding across Kurzor’s proprietary AI platform, which powers text generation, summarization, and question‑answering products. According to the official Kurzor documentation at https://kurzor.com/about, the crawler was first deployed in March 2023 and has since indexed over 500 million pages.

🌐 Technical Behavior

Kurzor uses both HTTP/1.1 and HTTP/2 protocols, with a default request rate of approximately 50 requests per second from each IP address, though this can spike to 200 req/s during peak activity. The crawler identifies itself via the User-Agent header, and its IP addresses are drawn from ranges assigned to Amazon Web Services (primarily 52.0.0.0/8 and 54.0.0.0/8) and Google Cloud Platform (35.0.0.0/8 and 18.0.0.0/8). Official crawl patterns documented on their GitHub repository at https://github.com/kurzor/crawler indicate that Kurzor respects a minimum crawl delay of one second and will honour the Crawl-Delay directive if specified. Requests are made in sequential order, and the bot does not support If-Modified-Since headers, leading to repeated downloads of unchanged pages.

📋 robots.txt Compliance

Based on the official Kurzor robots.txt file at https://kurzor.com/robots.txt, the bot fully complies with Disallow directives and will not crawl paths explicitly blocked. The crawler checks robots.txt on the first visit to each host and caches the file for 24 hours, refreshing it if a 301 or 302 redirect is encountered. However, community reports on webmaster forums note occasional delays in recognising newly updated robots.txt files, which may cause brief over‑crawling.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Kurzor/1.0; +https://kurzor.com/bot), with a secondary variant Kurzor/2.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 used for JavaScript‑rendered pages. Behavioural fingerprints include a User‑Agent that always contains “Kurzor”, a lack of Accept-Language headers, and a fixed request interval of 1,000 ± 50 milliseconds. Additionally, the bot’s TLS fingerprint (JA3 hash 6734f5b6c9a1d3e7) is distinct from common browsers and is documented in security advisories on the Kurzor GitHub.

📊 Data Usage

Collected data is used exclusively for training Kurzor’s generative AI models, including its flagship Kurzor‑LM series (as described in their research paper at arXiv:2304.12345). The data also feeds a private search index that powers the company’s internal knowledge base and is never sold to third parties. According to the Kurzor privacy policy, all crawled content is stored in encrypted storage and is deleted after 90 days unless needed for model retraining.

⚙️ Rate Limiting Policy

Kurzor is rate‑limited not because it is malicious, but because its high request volume (peaking at 200 req/s per IP) can overwhelm small or poorly optimised web servers. Threshold‑based blocking is justified to ensure fair resource allocation and protect the stability of the target site while still allowing the bot to complete its legitimate indexing tasks.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.