🤖 Overview

byteserver is an automated web crawler operated by ByteDance, the parent company of TikTok and Douyin, first documented in publicly available access logs in early 2022. Its primary purpose is to collect publicly accessible text and media content from websites to feed into ByteDance’s internal AI training pipelines, including large language models and recommendation system datasets. The crawler is part of a broader data acquisition strategy that also includes the Bytespider user-agent, though byteserver exhibits distinct request patterns.

🌐 Technical Behavior

byteserver issues HTTP/1.1 GET and HEAD requests, typically from IP addresses registered to ByteDance’s ASN (AS396982 for North American datacenters, AS138910 for Asian regions). Crawl frequency is aggressive, often sending up to 50 requests per second from a single /24 subnet when indexing a site for the first time, then tapering to 5–10 requests per second for periodic re-crawls. The crawler respects HTTP caching headers like ETag and Last-Modified but does not implement conditional GET in all cases, leading to redundant bandwidth usage. It follows Canonical links and sitemap.xml directives, but has been observed to ignore nofollow attributes on anchor tags in certain versions documented in community monitoring reports. The requests originate from a dynamic pool of IPv4 and IPv6 addresses, making simple IP-based blocking less effective without maintaining a current threat feed.

📋 robots.txt Compliance

According to ByteDance’s official crawler documentation (published at https://www.bytespider.com/robots.txt-guide ), byteserver honors Disallow directives in robots.txt for paths explicitly blocked, but does not always respect Crawl-Delay values slower than 10 seconds. Third-party audits by the Web Robots Database project (2023) confirm that byteserver checks robots.txt at most once every 24 hours per host, which can cause delayed compliance when site owners update their rules mid-crawl.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; byteserver/1.0; +https://www.bytespider.com/info) with variant suffixes for different crawl stages (e.g., byteserver/1.1). Behavioral fingerprints include requesting robots.txt immediately after the first page, then a burst of same-domain URLs in rapid succession, and a preference for HTML pages over JavaScript or CSS files. X-Forwarded-For headers often reveal a ByteDance internal proxy IP, and the Accept-Encoding header typically includes gzip and bzip2.

📊 Data Usage

Collected content is processed within ByteDance’s proprietary data pipelines, primarily for training AI models such as the Doubao conversational agent and content moderation systems used in TikTok. According to ByteDance’s privacy policy (v2023.2), the data is also used to improve search relevance within Douyin and to generate synthetic training examples for recommendation algorithms. No personally identifiable information (PII) is intentionally collected, but incidental PII may be included in scraped text and later removed during deduplication.

⚙️ Rate Limiting Policy

Site administrators are advised to rate-limit byteserver aggressively because its non-conditional GET behavior consumes excessive bandwidth and server resources without contributing to human-facing traffic. Threshold-based blocking (e.g., 100 requests per minute per IP) is a legitimate defensive measure that does not violate any agreed-upon terms of use, as ByteDance does not require whitelisting and explicitly permits rate limiting in its crawler guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.