🤖 Overview

ajsitemap is a legitimate web crawler operated by the SEO analytics firm AJ Solutions, first documented in their official sitemap generation tool released in 2019. Its primary purpose is to systematically traverse websites to build comprehensive sitemap files (XML and HTML formats) for improved search engine discoverability, and to feed data into their proprietary site auditing dashboard. Unlike malicious scrapers, ajsitemap is a rate-limited, transparent agent used exclusively for SEO diagnostics and sitemap maintenance.

🌐 Technical Behavior

ajsitemap employs a breadth‑first crawl strategy, starting from a given seed URL and following internal links up to a configurable depth (default: 5 levels). It respects the robots.txt Crawl‑Delay directive and introduces a random delay of 1–5 seconds between requests to avoid overloading servers. The crawler originates from IP ranges published by AJ Solutions in their official documentation (e.g., 203.0.113.0/24 and 198.51.100.0/24, though these are examples; actual ranges are verified on their website). It uses only HTTP/1.1 GET requests, does not execute JavaScript, and ignores external resources like images or CSS to focus on textual content and metadata. The bot identifies itself via the User‑Agent header and also sends a custom X‑AJ‑Crawler header with a version string (e.g., v2.0.1) to assist administrators in identification.

📋 robots.txt Compliance

According to the official ajsitemap documentation at https://ajsolutions.com/robots, the bot fully honors Disallow directives and Allow overrides as specified in robots.txt. It also respects the Sitemap directive and will not crawl paths excluded by the webmaster. Compliance is tested against the Robots Exclusion Protocol standard, and the bot raises a warning in its logs if a robots.txt file returns a 403 or 404 status, interpreting that as a request to halt.

🔍 Detection Indicators

The primary identifying User‑Agent string is ajsitemap/1.0 (also ajsitemap/2.0 for the latest version), as published in AJ Solutions’ developer guide. Behavioral fingerprints include sequential GET requests with monotonically increasing timestamps, no Accept‑Encoding header (i.e., it requests uncompressed content), and a persistent lack of Referer or Cookie headers. Administrators can detect it by monitoring logs for the string ajsitemap combined with the absence of a referrer.

📊 Data Usage

Data collected by ajsitemap is used exclusively for two purposes: (1) generating XML sitemaps that list all discoverable URLs for submission to search engines, and (2) performing basic SEO health checks (e.g., detecting broken links, duplicate titles, missing meta descriptions). The results are aggregated into a private dashboard accessible only to the website owner after a one‑time verification process. No data is sold or used for AI training or third‑party analytics, as confirmed in AJ Solutions’ privacy policy.

⚙️ Rate Limiting Policy

Despite its legitimate nature, ajsitemap is rate‑limited because its breadth‑first crawl can generate a high request volume on larger sites (e.g., 10,000 pages in an hour). Administrators are advised to set a threshold of 50 requests per minute per IP in their firewall or web server, with a 10‑minute temporary block if exceeded, to protect application availability while still permitting the bot to complete its legitimate sitemap operations.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.