🤖 Overview

WEBDAV is not a single bot but a family of legitimate automated clients that implement the Web Distributed Authoring and Versioning (WebDAV) protocol, as defined in RFC 4918. These clients are operated by various entities, including Microsoft (via the WebDAV Mini-Redirector), Apple (through Finder’s built-in WebDAV support), and open‑source synchronisation tools like cadaver and davfs2. Their purpose is to remotely manipulate files on a web server — creating, reading, deleting, and versioning resources — as if they were local. They are used for enterprise document management, cloud storage sync (e.g., ownCloud, Nextcloud), and collaborative editing. While not search engine crawlers, they are automated agents that systematically access web resources.

🌐 Technical Behavior

WebDAV clients send HTTP requests using the extended verbs PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, and UNLOCK, in addition to standard GET, PUT, DELETE, and HEAD. They typically initiate a session with a PROPFIND request to list directory contents, then issue GET/PUT for individual files. Request frequency varies widely: enterprise clients (e.g., Microsoft Office) send bursts of up to 5–10 requests per second while indexing a folder; sync clients (e.g., ownCloud desktop client) perform periodic polling every 30–300 seconds. IP ranges originate from the cloud providers hosting the sync service (e.g., Amazon AWS, Microsoft Azure) or from the end‑user’s corporate network. The protocol operates over HTTP/1.1 and HTTPS on ports 80 and 443, and some clients support chunked transfer encoding.

📋 robots.txt Compliance

WebDAV clients are not designed to parse robots.txt because they are file‑system agents, not web crawlers. The RFC 4918 specification makes no mention of robots.txt, and prominent implementations (Microsoft-WebDAV-MiniRedir, macOS Finder) do not read the file. However, some custom‑built WebDAV clients used for indexing (e.g., by search engines that support WebDAV) may honour Disallow directives; documented evidence from Microsoft support articles confirms that the Windows WebDAV client ignores robots.txt entirely.

🔍 Detection Indicators

User‑Agent strings are diverse: Microsoft-WebDAV-MiniRedir/10.0.19041 (Windows 10), WebDAVFS/3.0 (17F57) Darwin/21.1.0 (macOS), cadaver/0.24 (Linux command‑line), and ownCloud/3.0.0 for sync clients. Behavioural fingerprints include PROPFIND requests on the root directory (Depth: 1 or Infinity), the use of XML‑bodied responses, and frequent LOCK/UNLOCK sequences. Some clients send a Translate: f header (Microsoft) or a Depth: 0 header on individual resource requests.

📊 Data Usage

The collected data is used for local file synchronisation, remote editing, and document versioning. Enterprise deployments store user documents on WebDAV‑enabled servers; sync clients mirror entire folders to local disks. The data is not aggregated for AI training or search indexing — it remains under the control of the organisation or individual. Some cloud services (e.g., Nextcloud) use WebDAV for file sharing and collaborative workflows, but no third‑party analytics are performed by the protocol itself.

⚙️ Rate Limiting Policy

WebDAV clients are rate‑limited because aggressive PROPFIND requests can overwhelm server resources, particularly when enumerating large directories. Threshold‑based blocking (e.g., >100 PROPFIND requests per minute from a single IP) is recommended, with a 429 Too Many Requests response. The policy rationale is to protect server stability while still allowing legitimate synchronisation — a timed back‑off of 60 seconds aligns with typical sync intervals.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.