🤖 Overview

The phpDig bot is a web crawler originating from the open‑source PHP search‑engine framework phpDig, maintained by the phpDig project (hosted on SourceForge and GitHub). It is designed to index web content for private or community‑based search engines, allowing operators to build local search functionality without relying on third‑party services. The bot is used by individuals and organizations that deploy the phpDig script on their own servers, making it a decentralised, non‑commercial crawler.

🌐 Technical Behavior

phpDig performs recursive depth‑first crawling starting from a seed URL, following all discovered links within the allowed scope. Its default request frequency is configurable but can be as high as several requests per second if not throttled by the operator. The bot does not use fixed IP ranges; it originates from the IP address of the server running the phpDig script. It supports both HTTP and HTTPS protocols and includes a built‑in parser for HTML, PDF, and plain text documents. The crawler respects robots.txt directives and meta robots tags, and it can be configured to add a crawl delay via the script’s $crawl_delay parameter. Official documentation on the phpDig website notes that it sends a User‑Agent string of the form PhpDig/1.8.x (http://phpdig.net) by default.

📋 robots.txt Compliance

According to the official phpDig documentation and the source code published on SourceForge (project: phpdig), the crawler reads robots.txt before every crawl session and honours Disallow directives. It also checks for X‑Robots‑Tag headers and meta name="robots" content="noindex" tags. However, because the bot is deployed by individual operators, misconfigurations can occur; the default script does not override a site’s robots.txt rules.

🔍 Detection Indicators

The most distinctive identifier is the User‑Agent string: PhpDig/1.8.x (http://phpdig.net) with the version number varying. It may also appear as PHPdig/1.8.x or simply PhpDig. Another fingerprint is the Referer header, which often contains the previous URL crawled. The bot does not send any custom X‑headers. Behavioural indicators include rapid sequential requests from a single IP for linked pages with a short delay, and a lack of JavaScript execution.

📊 Data Usage

The collected data is used to populate a local search index managed by the phpDig script. This index stores page titles, meta descriptions, body text, and outgoing links. The operator can then run keyword searches against the index via a web interface. No third‑party or commercial use is implied; the data is used solely for the operator’s private search engine.

⚙️ Rate Limiting Policy

Because phpDig is under the control of its operator and can be configured to crawl aggressively (e.g., zero delay), it is rate‑limited by default in many web application firewalls. The policy rationale is that, while legitimate, a misconfigured phpDig instance can consume considerable server resources and may inadvertently trigger denial‑of‑service conditions. Threshold‑based blocking (e.g., >10 requests per second from a single IP) is recommended to protect site stability without outright blocking the bot.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.