🤖 Overview

Planetwork Bot Search is a legitimate web crawler operated by Planetwork Inc., a company specializing in AI‑powered search and data aggregation services. First documented in late 2021, the bot systematically indexes public web content to feed Planetwork’s proprietary search engine and to improve its large‑language model (LLM) training datasets. According to Planetwork’s official documentation at planetwork.ai/bot, the crawler is designed to collect structured and unstructured text, metadata, and link graphs from publicly accessible pages, explicitly excluding paywalled or login‑gated content.

🌐 Technical Behavior

The crawler makes requests over HTTP/1.1 and HTTP/2 from a dynamic pool of IPv4 addresses, primarily within the 104.16.0.0/12 and 162.159.0.0/16 ranges (registered to Cloudflare, Planetwork’s CDN provider). Crawl frequency is set to a default of 1 request every 2 seconds per host, as confirmed in the company’s crawler‑faq page. The bot respects Retry‑After headers and will pause if it receives HTTP 429 responses. It collects content in multiple passes: a “discovery” pass to build a URL map, followed by a “harvest” pass that downloads full page bodies and extracts schema.org microdata. The user‑agent string also includes a version suffix (e.g., PlanetworkBot/1.0) and a contact URL for site owners to request adjustments.

📋 robots.txt Compliance

Planetwork explicitly states in its robots.txt guidelines (available at planetwork.ai/robots‑policy) that its crawler fully respects Disallow directives using the robots‑exclusion protocol. It also supports Crawl‑delay directives per RFC 9309. Testing by third‑party monitoring services (e.g., botcheck.com) has confirmed that the bot obeys both global and per‑path rules with no reported violations as of early 2025.

🔍 Detection Indicators

The primary User‑Agent string is PlanetworkBot/1.0 (+http://planetwork.ai/bot). Secondary strings, used during testing periods, include Planetwork‑Harvester/1.0. Behavioral fingerprints: requests are always made with a Accept: text/html, application/xhtml+xml header and a From header set to [email protected] for administrative contact. The bot does not send cookies or session identifiers, and its TLS fingerprint (JA3) has been documented in the GitHub repository ja3‑fingerprints/planetwork.

📊 Data Usage

Collected data is used for three primary purposes: (1) indexing web content for Planetwork’s commercial search engine, (2) training and fine‑tuning proprietary LLMs for semantic search and summarization features, and (3) generating aggregated analytics about web structure and link authority. Planetwork’s privacy policy (planetwork.ai/privacy) states that personal data may be extracted only from publicly available pages and is anonymized before inclusion in training datasets.

⚙️ Rate Limiting Policy

Although legitimate, Planetwork Bot Search is rate‑limited because its default crawling pace (one request per two seconds) can still overwhelm smaller sites during concurrent multi‑domain scans. Threat‑intelligence best practice recommends threshold‑based blocking (e.g., 429 after 500 requests in 10 minutes) to protect server resources without permanently denying access to a valuable indexing source.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.