🤖 Overview

aiHitBot is a web crawler operated by aiHit Ltd., a data intelligence company headquartered in London, UK, established in 2015. Its primary mission is to systematically crawl publicly accessible web pages to collect structured data for training AI models and populating aiHit's sales intelligence platform, which offers company enrichment and lead generation services. According to official documentation published at https://www.aihit.com/bot, the bot is designed to operate transparently and respectfully.

🌐 Technical Behavior

aiHitBot employs a polite crawling strategy with a configurable delay between requests, typically between 2 and 10 seconds, as advised in their guidelines. The bot uses IPv4 addresses sourced from Amazon Web Services (AWS) and Google Cloud Platform (GCP) data centers, though specific CIDR ranges are not publicly listed. It sends conditional HTTP GET requests using If-Modified-Since and ETag headers to minimize data transfer. The crawler follows standard link traversal from sitemaps and anchor tags, and it does not execute JavaScript. It requests primarily text-based content such as HTML, XML, and JSON, with an Accept header preferring these MIME types. The bot also respects Cache-Control directives and uses a consistent request interval to avoid overwhelming servers.

📋 robots.txt Compliance

aiHitBot strictly adheres to the Robots Exclusion Protocol as confirmed by aiHit's official site. It checks robots.txt before each crawl session and will not access URLs that are disallowed at any path level. Additionally, it honors Crawl-Delay directives if specified, and there have been no documented instances of non-compliance. Site owners can verify this by examining their server logs; the bot always fetches robots.txt first.

🔍 Detection Indicators

The definitive User-Agent string is aiHitBot/1.0 (or aiHitBot/2.0 for newer deployments). It also sends a Referer header typically set to https://www.aihit.com. Behavioral fingerprints include a steady, predictable request rate of one request every few seconds and the absence of JavaScript or CSS resource requests. Server administrators can identify aiHitBot by filtering logs for these User-Agent strings and by noting the robotic pattern of requesting only textual content without accepting image or media types.

📊 Data Usage

The data harvested by aiHitBot feeds into aiHit's proprietary AI training pipeline to improve natural language understanding and entity extraction models. It is also used to update aiHit's Intelligence Platform, which provides sales teams with enriched company profiles, technology stacks, and contact information. According to their privacy policy, aiHit does not intentionally collect personally identifiable information (PII) and allows opt-out via robots.txt. The company emphasizes that data is stored securely and used only for internal product development.

⚙️ Rate Limiting Policy

aiHitBot is rate-limited because its systematic, high-volume crawling can consume significant server resources if left unchecked. The policy rationale is to allow website owners to set thresholds (e.g., 100 requests per minute) that temporarily block the bot when it exceeds a reasonable load, preventing performance degradation while still enabling the beneficial data collection that powers aiHit's AI services.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.