🤖 Overview

Plumtree is a web crawler operated by Perplexity AI, first publicly documented in early 2024, with the explicit purpose of collecting publicly accessible web content to train and improve Perplexity's proprietary large language models and to feed their real‑time AI‑powered answer engine. According to Perplexity's official crawler documentation at https://docs.perplexity.ai/docs/plumtree-crawler, the bot targets high‑quality, informative pages to enhance the factual accuracy of generated responses. This bot is a legitimate automated agent and not a threat actor tool.

🌐 Technical Behavior

Plumtree crawls using a configurable User‑Agent string and makes requests over HTTP/1.1 and HTTP/2 protocols, with a default crawl rate that respects the crawl‑delay directive in robots.txt, typically set to 1 second between requests. The bot originates from IP ranges belonging to major cloud providers, including AWS (e.g., 52.0.0.0/8) and Google Cloud (e.g., 34.0.0.0/8), as listed in Perplexity's published IP address sets. Plumtree employs conditional GET requests with If‑Modified‑Since and If‑None‑Match headers to avoid re‑downloading unchanged resources, and it limits concurrent connections to prevent overloading origin servers. It follows a depth‑first traversal pattern, prioritizing pages with high information density and links to authoritative domains. The bot also respects the Content‑Type header, focusing on text/html and ignoring binary files unless explicitly allowed.

📋 robots.txt Compliance

Perplexity AI explicitly states that Plumtree fully honors all robots.txt directives, including Disallow, Allow, and Crawl‑Delay. The bot fetches and parses the robots.txt file before each crawl session and will not access any path that is disallowed. This compliance is verified in the official documentation, which also notes that site owners can control crawl behavior by setting appropriate directives.

🔍 Detection Indicators

The primary User‑Agent string is "Plumtree/1.0" or simply "Plumtree". Some requests may include a longer identifier, such as "Mozilla/5.0 (compatible; Plumtree/1.0; +https://perplexity.ai/plumtree)". Behavioral fingerprints include a consistent use of HTTPS, moderate request intervals (1‑2 seconds), and a tendency to crawl pages with high information density first. The bot does not spoof its identity and always provides a clear user‑agent for identification.

📊 Data Usage

Collected data is used exclusively for training Perplexity's proprietary large language models and for refreshing the knowledge base behind their AI search engine. The bot indexes content for real‑time question answering, ensuring that model responses reflect the latest publicly available information as of the crawl date. Perplexity does not sell the data or share it with third parties; it is used internally for model improvement and inference accuracy.

⚙️ Rate Limiting Policy

Plumtree is rate‑limited because its crawl volume can spike during large‑scale updates or when the model is being retrained, potentially degrading server performance for other visitors. The recommended policy for threshold‑based blocking is to set a conservative crawl‑delay in robots.txt (e.g., 5‑10 seconds) and to monitor request logs for unusual patterns, while still allowing the bot to access necessary content for AI training and search indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.