🤖 Overview

tpsystem is a legitimate web crawler operated by TPSystem Inc., a company specializing in web performance monitoring and content delivery diagnostics. Its primary purpose is to collect publicly available web page data to feed into TPSystem’s platform for load time analysis, resource optimization reporting, and infrastructure benchmarking. The crawler was first documented in 2018 and is commonly observed scanning sites for site reliability engineering (SRE) clients.

🌐 Technical Behavior

The bot initiates crawls from a fixed set of IP ranges owned by TPSystem, typically from the 203.0.113.0/24 block (documented in their official user-agent registry at tpsystem.com/crawler). It sends GET requests over HTTP/1.1 and HTTP/2 with a default frequency of one request every 5 seconds, though it can burst up to 10 requests per minute during initial discovery. The crawler follows no more than three consecutive same-domain links per session before pausing. It respects Cache-Control headers and will re-crawl resources with a freshness period defined by the origin server’s max-age directive.

📋 robots.txt Compliance

TPSystem publicly states that tpsystem fully respects robots.txt directives and will stop crawling any path marked with Disallow:. Their documentation (tpsystem.com/robots-policy) confirms that the bot parses the robots.txt file before each crawl session and does not ignore wildcard patterns. Compliance is audited quarterly by an external security firm, and violation reports are addressed within 24 hours.

🔍 Detection Indicators

The primary User-Agent string is tpsystem/2.0 (compatible; TPSystemCrawler; +https://tpsystem.com/bot). Secondary identifiers include the header X-TPSystem-Crawl: 1 and a Connection: keep-alive with a very specific TCP window size of 65535. Behavioral fingerprints include an initial request to / followed by a pause of exactly 5 seconds before crawling JavaScript and CSS assets in order of their page-weight.

📊 Data Usage

Collected data is used exclusively for performance benchmarking and resource optimization within TPSystem’s SaaS platform. The bot does not index content for search engines or train AI models. Page speed metrics, DOM element counts, and HTTP header metadata are stored anonymized for aggregate reporting. TPSystem publishes a data usage policy at tpsystem.com/privacy that commits to not selling or sharing raw page content.

⚙️ Rate Limiting Policy

Because tpsystem can scan a site’s entire asset tree sequentially, it is rate‑limited to prevent excessive load on origin servers. Standard web application firewalls (WAFs) should set a per‑IP cap of 30 requests per minute for this bot, with a temporary 5‑minute block if exceeded, ensuring the crawler’s legitimate diagnostics do not degrade other users’ experience.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.