🤖 Overview

pwebot is a web crawler operated by PWeb Technologies, a company specializing in large-scale web data extraction for AI model training and aggregated analytics. First documented in public forums in early 2024, pwebot is used to feed curated public web content into proprietary NLP models and benchmarking datasets.

🌐 Technical Behavior

pwebot performs sequential HTTP GET requests with a configurable crawl delay, defaulting to 10 seconds between consecutive requests. It respects the robots.txt file and follows Crawl-Delay directives when present. The bot uses IPv4 addresses from a block registered to PWeb Technologies (e.g., 203.0.113.0/24) and rotates User-Agent strings to avoid simple pattern blocks. It requests both HTML and JSON content, occasionally sending Accept-Language: en-US headers. The crawler does not execute JavaScript and retrieves only static pages, making it less resource-intensive than headless browsers.

📋 robots.txt Compliance

According to official documentation published at pwebot.com/robots, the bot fully honors Disallow rules and will not crawl paths starting with /admin, /private, or any path explicitly forbidden in the site’s robots.txt. It also respects Allow overrides and custom crawl-delay values, as verified by community reports on GitHub (issue #47 in the pwebot repo).

🔍 Detection Indicators

The primary User-Agent string is pwebot/1.0 (compatible; +https://pwebot.com/bot), sometimes accompanied by a secondary string Mozilla/5.0 pwebot. Requests typically include the header X-Robots-Tag: pwebot for identification. The bot also sends a From field with a contact email ([email protected]) in compliance with RFC 9309.

📊 Data Usage

Collected data is used to train PWeb’s custom language models for industry-specific tasks, such as legal document summarization and financial trend analysis. Aggregated crawl data also feeds a public research dataset called WebCorpus-v2, available under a permissive license on the company’s GitHub page (github.com/pwebot/corpus). No personally identifiable information is retained beyond 30 days.

⚙️ Rate Limiting Policy

While pwebot is not malicious, it is rate-limited because its request volume (up to 10,000 requests per hour per IP) can overwhelm smaller sites. Threshold-based blocking at the reverse‑proxy level is recommended to ensure fair resource allocation for all visitors.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.