🤖 Overview

qweery_robot is the web crawler operated by Qweery, a French search engine company headquartered in Paris. Its primary purpose is to index French-language and European web content for the Qweery search engine, which launched publicly in 2019. The bot collects publicly accessible HTML pages, PDFs, and images to populate Qweery's search index, which serves approximately 2 million queries per month as of 2023. Official documentation available at qweery.com/bot describes the crawler as a GDPR-compliant agent that respects user privacy by not storing cookies or tracking identifiers.

🌐 Technical Behavior

The crawler uses HTTP/1.1 requests with a default crawl delay of 10 seconds between consecutive requests to the same host, as specified in its robot's.txt policy. It operates from a dynamic pool of IPv4 addresses allocated to Qweery's ASN (AS207722), primarily originating from France (FR) and Germany (DE). Requests are made over port 80 and 443, with a preference for HTTPS when available. The bot does not execute JavaScript or render pages; it fetches raw HTML and follows internal links up to a depth of 5 levels. According to Qweery's official technical documentation, the bot sends a User-Agent header of "Mozilla/5.0 (compatible; QweeryBot/1.0; +https://www.qweery.com/bot)" and includes a custom "From" header with a contact email address for webmaster inquiries.

📋 robots.txt Compliance

Qweery_robot fully honors robots.txt directives as per its published policy. It checks the robots.txt file at the start of each crawl session, caching the result for up to 24 hours. The bot respects both "Disallow" and "Allow" rules, including wildcard patterns. Evidence from Qweery's community forum indicates that webmasters can also specify a crawl-delay directive to further throttle the bot, which it adheres to.

🔍 Detection Indicators

The primary identification string is "QweeryBot/1.0" with a secondary token "qweery_robot" appearing in some logs. The bot's requests typically include a "From" header of "[email protected]" and a "Accept-Language: fr,en;q=0.9" header. Behavioral fingerprinting shows the bot only requests text-based content types (text/html, application/pdf) and skips images above 1MB. It also sets a "X-Robots-Tag" header to "noarchive" when encountered.

📊 Data Usage

Collected web pages are indexed for search results in the Qweery search engine. The company states that content is not used for AI training or any other purpose beyond search indexing. Qweery stores crawled data for a maximum of 60 days, after which it is re-fetched if the page is still accessible. The bot does not archive pages or share data with third parties.

⚙️ Rate Limiting Policy

Rate limiting for qweery_robot is recommended because the bot's default crawl delay of 10 seconds can still generate significant load on smaller websites. A threshold-based approach (e.g., blocking if exceeding 100 requests per minute) ensures fair resource allocation while allowing legitimate indexing.

txt_checkbot;

🤖 Overview

txt_checkbot is a specialized web crawler operated by CheckBot Ltd, a UK-based cybersecurity and SEO tool provider. It was created to validate the accessibility and correctness of robots.txt files across websites for use in the company's "Robots Checker" SaaS product. The bot systematically fetches robots.txt, verifies its syntax, and checks whether its own user-agent (and others) are blocked or allowed, helping webmasters audit their crawl permissions. Official details are published at checkbot.io/txtcheck and in a GitHub repository (github.com/checkbot/txt_checkbot).

🌐 Technical Behavior

txt_checkbot operates with a very low request rate of one request per 60 seconds per domain to minimize impact. It uses a fixed pool of IPv4 addresses belonging to ASN 20473 (UK-based cloud provider), with IPs ranging from 185.151.120.0 to 185.151.127.255. The bot only requests the single file "/robots.txt" via HTTP GET on both port 80 and 443, and does not crawl any other page unless the robots.txt contains an "Allow" directive for itself. It respects the "Crawl-Delay" directive if encountered in the robots.txt it fetches. The bot sends a custom header "X-CheckBot-Version: 2.3" to identify itself as part of the checker tool. Behavior is documented in the project's README on GitHub.

📋 robots.txt Compliance

txt_checkbot is inherently designed to be robots.txt‑compliant; it only fetches the robots.txt file itself and does not crawl any other URLs. However, it does not parse or honor Disallow directives within the robots.txt for its own subsequent requests because it has no subsequent requests — it only retrieves the file. This design is intentional: the bot’s purpose is to verify the file, not to index content. The CheckBot documentation explicitly states that webmasters who do not wish to receive the check can block the IP range.

🔍 Detection Indicators

The User-Agent string is "txt_checkbot/2.3 ( compatible; +https://checkbot.io/txtcheck )". Additional identifying features: the "From" header is "[email protected]", and the bot uses a fixed "Accept: text/plain, */*;q=0.1" header. It never includes a Referer header. Log entries for this bot will show a single request to /robots.txt per domain, often with a 404 or 200 response. No session cookies are sent.

📊 Data Usage

The data collected (robots.txt content, HTTP status codes, response headers) is used solely to generate reports for paying customers of the Robots Checker service. The reports include syntax errors, missing files, and potential blocks for important crawlers. CheckBot states that no content is stored beyond 7 days, and no data is sold or shared. The tool does not use the collected data for AI training or any other purpose.

⚙️ Rate Limiting Policy

Rate limiting for txt_checkbot is unnecessary for most sites due to its extremely low request frequency (one per domain per 60 seconds). However, if a webmaster observes multiple checks from different IPs in the same range, a threshold of 10 requests per minute from the same subnet can be applied to prevent accidental overload, though this bot is considered one of the least aggressive legitimate crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.