🤖 Overview

check&get is a legitimate web crawler operated by the Czech company Check&Get s.r.o., primarily designed for automated website monitoring and change detection. Its purpose is to periodically crawl web pages to detect content updates, broken links, or structural modifications, feeding the collected data into the company’s “Check&Get Monitor” SaaS product used by webmasters and SEO professionals.

🌐 Technical Behavior

The bot employs a scheduled crawl cycle with a default request frequency of once every 24 hours per target domain, though users can configure intervals as short as 1 hour. It sends requests over HTTP/1.1 and HTTP/2, with a maximum of 5 concurrent connections per domain to avoid server overload. IP ranges are documented in the operator’s official whitepaper (available at check-and-get.com/ip-ranges) and include blocks from 185.185.0.0/16 (AS 197068) and 91.213.0.0/16, all geolocated in the Czech Republic. The crawler respects If-Modified-Since and ETag headers to reduce unnecessary downloads, and it limits each request to 5 MB of content.

📋 robots.txt Compliance

According to the operator’s publicly posted robots.txt policy (verified on GitHub at github.com/checkandget/crawler-policy), check&get fully honors all Disallow directives. The bot checks robots.txt for every new domain and caches it for up to 48 hours. If a site returns a 403 or 429 status, the bot backs off exponentially up to a maximum of 7 days before retrying.

🔍 Detection Indicators

The primary User-Agent string is check&get/1.0 (compatible; +https://check-and-get.com/bot). A secondary identifier appears as CheckGet/2.1 (Linux x86_64) in older versions. The bot also sets a custom X-Crawler-ID header with a hexadecimal hash value. Behavioral fingerprints include a consistent crawl interval within ±2 minutes of the configured schedule and a request order that follows sitemap links when available.

📊 Data Usage

Collected data is stored in the Check&Get Monitor platform and used exclusively for generating change alerts, SEO reports, and uptime dashboards for paying subscribers. The operator explicitly states in its privacy policy (check-and-get.com/privacy) that no content is used for AI training, search indexing, or third-party distribution.

⚙️ Rate Limiting Policy

While check&get is a legitimate, non‑malicious agent, its periodic crawling can consume bandwidth if many users target the same site simultaneously. Rate limiting is applied via threshold‑based blocking when a single IP exceeds 100 requests per minute, ensuring fair resource usage for all websites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.