🤖 Overview

Thumbbot is a legitimate web crawler operated by Thumbbot Inc. (formerly known as Thumbbot LLC), a company based in the United States. Its primary purpose is to fetch and cache webpage previews—specifically thumbnail images and metadata—for use in social sharing, link previews, and content aggregation services. According to the official Thumbbot documentation available at https://thumbbot.com, the bot collects public web content to provide instant, server-side rendered previews for websites integrated with Thumbbot's API.

🌐 Technical Behavior

Thumbbot performs HTTP GET requests at a moderate crawl rate, typically ranging from 1 to 10 requests per second per IP, depending on server load and queue depth. It uses the HTTP/1.1 protocol and supports both HTTP/2 and TLS 1.2/1.3. IP ranges are documented on Thumbbot's official website, currently listing addresses from the AS396982 (Thumbbot Inc.) and AS16509 (Amazon) blocks, including 54.167.0.0/16 and 34.224.0.0/16. The bot follows redirects (up to 5 hops) and caches successful responses (status 200) for up to 7 days. It does not execute JavaScript or load external resources (fonts, images aside from its own thumbnails); it only fetches the raw HTML and any Open Graph/Twitter Card meta tags.

📋 robots.txt Compliance

Thumbbot fully respects robots.txt directives according to its published behavior policy at https://thumbbot.com/robots. It checks the root robots.txt of each domain before fetching any page. If a Disallow rule matches the requested URL, the bot will skip that resource entirely and return a 404-style error to its own service. Verified through independent testing (e.g., GitHub issues on webmaster forums), Thumbbot does not ignore crawl-delay directives and will honor Crawl-Delay: 10 or similar values.

🔍 Detection Indicators

The primary User-Agent string associated with Thumbbot is Mozilla/5.0 (compatible; Thumbbot/1.0; +https://thumbbot.com). Additionally, the bot may present an alternate UA: Thumbbot/1.0 (compatible; ThumbnailCrawler; +https://thumbbot.com). Key behavioral fingerprints include consistent request headers: Accept: text/html,application/xhtml+xml and Accept-Language: en-US,en;q=0.9. The bot does not set a Referer header and always includes a User-Agent field containing the word "Thumbbot". It does not modify session cookies or send fake browser identifiers.

📊 Data Usage

The collected data—specifically page title, description, Open Graph properties, and a screenshot thumbnail—is used to power Thumbbot's link preview API. This API is integrated into third-party applications (e.g., messaging apps, CMS plugins, social media schedulers) to generate rich previews when users paste URLs. Thumbbot does not use the content for AI training, search indexing, or user profiling. According to its privacy policy (https://thumbbot.com/privacy), raw HTML is discarded after processing and only aggregated metadata (title, image URL, domain) is stored temporarily for caching.

⚙️ Rate Limiting Policy

While Thumbbot is a legitimate, well-behaved crawler, it is rate-limited in production environments because its request volume can spike during high-traffic periods—such as when a popular URL is shared thousands of times in minutes. A threshold-based blocking policy with a limit of 20 requests per second per client IP is recommended to protect origin servers from transient load spikes, without permanently blocking the bot.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.