🤖 Overview

FollowSite is a legitimate web crawling agent operated by FollowSite Ltd, a company specializing in website change detection and monitoring services. Its primary purpose is to systematically scan publicly accessible web pages to detect modifications, updates, or deletions, feeding data into its proprietary alert system that notifies subscribers of changes. The bot is not involved in AI training or search indexing but focuses on real-time content monitoring for users who need to track competitor updates, regulatory changes, or news article revisions.

🌐 Technical Behavior

FollowSite's crawler operates on a scheduled basis, typically revisiting monitored pages at intervals configurable by the service subscriber — from minutes to days. It uses HTTP/1.1 and HTTP/2 protocols, sending GET requests with standard headers including User-Agent: FollowSite/1.0 and Accept: text/html,application/xhtml+xml. The bot's IP addresses are drawn from a documented range published at https://followsite.com/ip-ranges, which includes IPv4 blocks such as 192.0.2.0/24 and 203.0.113.0/24 (example ranges; actual ranges vary). It does not execute JavaScript or load external resources, focusing solely on static HTML content to minimize server impact. Crawl frequency per domain is limited to one request every 10 seconds by default, as stated in the official documentation at https://followsite.com/crawler-policy.

📋 robots.txt Compliance

According to the official FollowSite crawler policy published at https://followsite.com/robots, the bot fully honors Robots Exclusion Protocol directives. It respects Disallow rules and will pause crawling for any URL path listed in a site’s robots.txt. The bot also checks for Crawl-delay directives and adheres to them, allowing webmasters to control the frequency of requests. This compliance is verified through independent testing by webmaster forums such as WebmasterWorld.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: FollowSite/1.0 (compatible; FollowSite Bot; https://followsite.com/bot). Additional behavioral fingerprints include a consistent request interval pattern, lack of JavaScript execution, and a unique From header containing [email protected]. The bot also appends a X-Followsite-Request header with a timestamp for internal tracking. Web server logs show these headers distinctly, making identification straightforward for site administrators.

📊 Data Usage

Collected data — specifically the HTML content and its hash fingerprint — is used exclusively for change detection. FollowSite does not store full page copies beyond a configurable retention period, typically 30 days, nor does it share data with third parties. The service generates alerts when differences exceed a defined threshold, and subscribers receive notifications via email or API. No machine learning models are trained on the crawled content, as stated in the privacy policy at https://followsite.com/privacy.

⚙️ Rate Limiting Policy

FollowSite is rate-limited because its periodic revisits can generate sustained request volumes on monitored domains, potentially impacting server performance if unthrottled. The recommended practice is to apply threshold-based blocking only if the bot exceeds the documented crawl-delay of 10 seconds per page, as its behavior otherwise is non-aggressive and fully compliant with site controls.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.