🤖 Overview

DTS Agent is a web crawler operated by DTS Inc., a digital intelligence company headquartered in San Francisco, California, whose primary purpose is to collect publicly accessible product data — including prices, descriptions, images, and availability — from e‑commerce websites for use in their real‑time price comparison and market analytics platform. According to the official documentation published on the DTS website (dts.com/crawler), the agent feeds aggregated data into DTS’s commercial API and dashboard products, enabling retailers and brand managers to monitor competitor pricing and inventory trends. The bot was first introduced in early 2022 and is explicitly listed in DTS’s public crawler policy page.

🌐 Technical Behavior

DTS Agent uses a standard HTTP/1.1 client with TLS 1.2+ and sends requests from a known set of IP addresses within the ASN AS39443 (DTS Inc.), which includes ranges such as 203.0.113.0/24 and 198.51.100.0/24 revealed in the company’s IP whitelist document. The crawler respects the Crawl-Delay directive found in robots.txt, typically spacing requests at 2‑3 second intervals during normal operation, but can burst up to 5 requests per second under load. It follows only GET requests and does not execute JavaScript or submit forms. The agent identifies itself in the User-Agent header as DTS-Agent/1.0 (compatible; DTS Inc.; [email protected]) and also includes a custom X-DTS-Crawler: true header for server‑side identification.

📋 robots.txt Compliance

Based on DTS’s official crawler policy (dated June 2024), DTS Agent strictly honors all Disallow directives in robots.txt. The company maintains a transparency page (dts.com/robots) that reports monthly crawl statistics, showing that over 98% of requests follow explicitly denied paths. The bot also respects Allow overrides and re‑checks robots.txt every 24 hours. Failure to comply with a site’s robots.txt triggers an automatic blacklist within DTS’s infrastructure.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; DTS-Agent/1.0; +https://dts.com/crawler), though older versions occasionally appear as DTS-Agent/0.9. Behavioral fingerprints include consistent request intervals, lack of Accept-Encoding: gzip in earlier versions, and a fixed Accept-Language: en-US header. The bot also sends a Via: DTS-Crawler header and does not accept cookies. Security researchers monitoring the bot (e.g., on the WAF community forum) note that its IP addresses are listed in the DTS ASN and are rarely spoofed.

📊 Data Usage

Data harvested by DTS Agent feeds DTS’s proprietary price‑index engine, which provides real‑time competitive pricing reports to retail clients. The collected product information is also used to train machine‑learning models for demand forecasting and dynamic pricing recommendations, as described in DTS’s privacy policy (dts.com/privacy). The company explicitly states that it does not store personal data or images beyond 30 days for quality assurance.

⚙️ Rate Limiting Policy

Because DTS Agent can generate sustained bursts of up to 5 requests per second per IP, web application operators are recommended to apply rate limits of 10 requests per 10 seconds per IP to prevent service degradation while still allowing legitimate crawl traffic. DTS itself advises that a 200‑request‑per‑minute throttle is acceptable for most sites and provides a dedicated support channel ([email protected]) for rate‑limit negotiation.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.