🤖 Overview

Cloud mapping is a legitimate web crawler operated by Cloud Mapping LLC, a company specializing in website architecture analysis and SEO auditing. First publicly documented in 2022, the bot systematically traverses public web pages to create topological maps of site structures, which are then used to generate actionable reports for webmasters and digital marketers. Its primary product, the Cloud Mapping Dashboard, provides insights on page hierarchy, internal linking efficiency, and crawl budget optimization. The bot is not affiliated with any cloud infrastructure provider but rather uses a proprietary distributed crawling network.

🌐 Technical Behavior

The Cloud mapping bot crawls at a default rate of 10 requests per second per domain, though this may increase to 50 requests per second for large sites during initial mapping. It respects the Crawl-Delay directive in robots.txt and reduces its frequency if the server returns 429 or 503 status codes. The bot makes requests over both HTTP/1.1 and HTTP/2, using a randomized user-agent suffix to avoid simple pattern blocking. Its IP ranges belong to ASN 394256 (Cloud Mapping LLC) and are published at https://cloudmapping.io/ip-ranges.txt. The crawler follows all rel="nofollow" and rel="nofollow" link attributes and does not execute JavaScript, relying solely on static HTML parsing. According to the official documentation, it also supports the Accept-Language header to request English content by default.

📋 robots.txt Compliance

Based on a 2023 analysis by Cloud Mapping's published compliance report, the bot fully honors Disallow directives in robots.txt. It also respects the Allow directive for overriding restrictions. The bot checks robots.txt at the start of each crawl session and caches it for up to 24 hours. Any changes to robots.txt are reflected within that caching window, and the bot does not crawl disallowed paths even if they are referenced from allowed pages.

🔍 Detection Indicators

The primary User-Agent string for the Cloud mapping bot is Mozilla/5.0 (compatible; CloudMapping/1.0; +https://cloudmapping.io/bot). A secondary string CloudMapping/2.0 (compatible; +https://cloudmapping.io/bot) is used for HTTPS-only requests. The bot also sends a custom HTTP header X-CloudMapping-Crawl: true which can be used for detection. Behavioral fingerprints include a consistent request interval of 100 milliseconds between consecutive page fetches and a preference for crawling .html, .htm, and / (root) URLs first.

📊 Data Usage

Data collected by the Cloud mapping bot is used exclusively for generating site structure maps, internal link analysis, and SEO recommendations for paying subscribers. The company explicitly states that no personal data is stored and that all collected URLs are aggregated into anonymous topological graphs. The bot does not train generative AI models; instead, it feeds a deterministic algorithm that identifies orphan pages, deep linking issues, and crawl depth problems. According to their privacy policy at https://cloudmapping.io/privacy, the raw crawl data is retained for 30 days before being anonymized.

⚙️ Rate Limiting Policy

Because the Cloud mapping bot can send up to 50 requests per second during initial site mapping, administrators should rate-limit it to protect server resources and maintain quality of service for human visitors. A threshold-based block at 100 requests per second per IP is recommended to allow legitimate crawling while preventing unintended denial-of-service conditions, as the bot does not negotiate its own rate beyond the Crawl-Delay directive.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.