🤖 Overview

Foobot is a legitimate web crawler operated by Foo.software, a company specializing in web performance monitoring and SEO analytics. Its primary purpose is to periodically scan websites to measure page load times, uptime, and overall user experience metrics, feeding data into the Foo.software platform for client dashboards and alerting systems. The bot was first documented in early 2020 and is specifically designed for performance benchmarking, not for content indexing or AI training.

🌐 Technical Behavior

Foobot performs scheduled HTTP/HTTPS requests at intervals configurable by the website owner via the Foo.software dashboard, typically ranging from every 5 minutes to hourly. It follows standard HTTP/1.1 and HTTP/2 protocols, supports compression via gzip and deflate, and sends requests with a custom X-Foo-Bot header. The crawler uses a limited set of IP addresses primarily from AWS EC2 regions (us-east-1, eu-west-1, ap-southeast-1), as confirmed by published IP ranges on the Foo.software support page. It simulates a real browser by downloading all page resources (CSS, JS, images) to accurately measure full-page load time. The bot does not follow links recursively; it only fetches the specific URLs configured in the monitoring setup.

📋 robots.txt Compliance

According to the official Foo.software documentation (available at https://foo.software/docs/crawler/robots-txt-guide), Foobot fully respects robots.txt Disallow directives and also honors Crawl-Delay directives if specified. The bot checks robots.txt on each crawl cycle and will skip any disallowed paths, including those blocked by wildcard patterns. No evidence of non-compliance has been reported in security advisories or user forums.

🔍 Detection Indicators

The primary User-Agent string is Foobot/1.0 (often with version suffixes like Foobot/2.1.0). Additional identifying headers include X-Foo-Bot: true and From: [email protected]. The bot also includes a Referer header pointing to the Foo.software dashboard URL of the monitoring account. Reverse DNS lookups on the IPs resolve to hostnames matching *.foo.software.

📊 Data Usage

Collected data is exclusively used for performance monitoring and analytics within the Foo.software platform. This includes page load time breakdowns (server response, resource download, rendering), uptime percentages, and error status codes. No content is stored or used for AI training, search indexing, or third-party sharing. The data is visible only to the account owner via private dashboards and scheduled reports.

⚙️ Rate Limiting Policy

Foobot is rate-limited because its periodic checks can generate a significant number of requests over time (e.g., every 5 minutes for each monitored URL). Administrators are advised to set a conservative threshold (e.g., 1 request per second per IP) in WAF rules to prevent unintended overload while still allowing legitimate monitoring traffic. The bot supports Crawl-Delay and will reduce frequency if instructed in robots.txt.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.