🤖 Overview

Inspirebot is a web crawler operated by Inspire, a health and wellness community platform headquartered in Arlington, Virginia, first documented in 2015. Its purpose is to collect publicly accessible web content to power internal search features, content recommendations, and user discovery within the Inspire community. Official documentation is available at inspire.com/robots.txt and inspire.com/about/crawler.

🌐 Technical Behavior

Inspirebot uses a breadth-first crawl strategy starting from health-related seed URLs. According to server logs from major publishers, it requests at an average rate of one per 12 seconds with bursts up to 6 per minute. It uses IPv4 addresses from ranges owned by Inspire, primarily 104.248.0.0/16 and 159.203.0.0/16 (DigitalOcean), confirmed via whois. The crawler sends standard User-Agent and From headers, respects ETag and Last-Modified for conditional requests, and uses HTTP/1.1 only, without HTTP/2 or IPv6 support. It ignores crawl-delay in robots.txt but slows down upon receiving HTTP 429 responses and checks X-Robots-Tag headers.

📋 robots.txt Compliance

Inspirebot fully adheres to Disallow directives. Inspire’s documentation states the bot reads robots.txt at session start and caches rules for up to 24 hours. No incidents of ignoring allow or disallow patterns have been reported.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Inspirebot/1.0; +http://inspire.com/bot). Additional identifiers include header From: [email protected], custom header X-Inspire-Bot: true, and consistent Accept-Language: en-US,en;q=0.5. Reverse DNS lookups often resolve to hostnames containing inspire.

📊 Data Usage

Collected data is used exclusively for internal search indexing and content discoverability within Inspire. The platform does not use crawled data for AI model training or external sales. Inspire’s privacy policy (inspire.com/privacy) confirms temporary storage and retention only as needed for index updates.

⚙️ Rate Limiting Policy

Although legitimate, Inspirebot can become aggressive on large sites, causing server load spikes. Rate limiting with threshold-based blocking using HTTP 429 responses is recommended to prevent resource exhaustion, as the bot is designed to respect such signals.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.