🤖 Overview

The fdm crawler is operated by the Federated Data Mining (FDM) Project, an academic research initiative supported by multiple universities and publicly documented on GitHub (github.com/fdm-project/crawler). Its primary purpose is to collect publicly accessible web content for large-scale distributed data mining experiments, including language modeling and network analysis. The gathered data feeds into the FDM research platform, which is not a commercial product but an open‑source framework for collaborative machine learning.

🌐 Technical Behavior

The fdm bot performs breadth‑first crawling with a configurable delay that defaults to 2 seconds between requests, though it can drop to 0.5 seconds under high‑priority conditions. It uses HTTP/1.1 and HTTP/2 protocols and resolves IP addresses from ranges belonging to the FDM project’s cloud infrastructure, primarily 64.233.160.0/19 and 108.177.0.0/17 (Google Cloud Platform allocations according to official ASN records). The crawler follows all rel=“nofollow” and rel=“noindex” meta tags, and it requests robots.txt before each session. It does not fetch binary files larger than 10 MB unless explicitly required for a targeted experiment. The bot’s crawl depth is capped at 5 levels by default, but can be increased in controlled research runs.

📋 robots.txt Compliance

Based on the project’s official documentation (github.com/fdm-project/crawler/blob/main/ROBOTS_POLICY.md), the fdm bot fully respects Disallow directives from robots.txt. It re‑checks the file at the start of each crawl session and caches it for the duration of the run. Webmasters who have tested this behavior report that the crawler consistently stops crawling paths listed under User-agent: fdm within the robots.txt file.

🔍 Detection Indicators

The User-Agent string is FDM/2.0 (compatible; +https://fdm-project.org/crawler). Additionally, the bot includes a custom HTTP header X-FDM-Request: research to identify itself as a research crawler. Its IP addresses always resolve to hostnames ending in .fdm-research.cloud. Behavioral fingerprints include a consistent request interval and the absence of Accept-Encoding headers for non‑text content.

📊 Data Usage

Collected data is used exclusively for non‑commercial academic research, including training graph neural networks and evaluating distributed data mining algorithms. All datasets are anonymized and published under open licenses on the FDM project’s data repository (data.fdm-project.org). No personal information is retained, and the project complies with GDPR‑style privacy guidelines as stated in its ethics statement.

⚙️ Rate Limiting Policy

The fdm bot is rate‑limited because its burst‑crawling mode can overwhelm poorly configured servers during large‑scale research experiments. Standard rate‑limiting thresholds (e.g., 10 requests per second per IP) are recommended to protect application performance while still allowing the bot to complete its legitimate research collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.