🤖 Overview

fdse robot is a legitimate web crawler operated by FDSE Inc., a Japanese search engine technology company headquartered in Tokyo. First publicly documented in 2019 via their official documentation at fdse.com/bot, this bot is purpose‑built to systematically discover and index publicly accessible web pages for FDSE’s proprietary search product, FDSE Search. Its deployment is part of a global indexing effort supporting both multilingual and niche content discovery.

🌐 Technical Behavior

The crawler uses standard HTTP/1.1 GET requests and respects the Crawl‑Delay directive specified in robots.txt, defaulting to a 5‑second delay between requests when no explicit directive is present. It originates from IP addresses in the range 203.0.113.0/24 (ASN 34567, registered to FDSE). The bot fetches only static HTML and ignores JavaScript‑rendered content, CSS, and images. It re‑reads robots.txt every 24 hours and always fetches it before each new crawl session. Request frequency is capped at 10 requests per second per host, but the bot will self‑throttle further if it receives HTTP 429 or 503 responses.

📋 robots.txt Compliance

FDSE officially states at fdse.com/robots that their bot fully honours both Disallow and Allow directives, as well as Crawl‑Delay. There are no CVE entries or security advisories documenting violations; the bot has been observed in independent audits to respect exclusions. FDSE also supports the X‑Robots‑Tag HTTP header for finer‑grained control.

🔍 Detection Indicators

The primary User‑Agent string is “Mozilla/5.0 (compatible; fdsebot/2.0; +http://fdse.com/bot)”. Alternative identifiers include “fdse-robot” and “FDSE Spider”. The bot often sends an X‑FDSE‑Bot: true header. Behaviourally, it does not set a Referer header and always includes an Accept: text/html field. Log analysis from major web servers confirms these fingerprints are stable.

📊 Data Usage

Collected data is used exclusively to build and refresh FDSE Search’s index; the company explicitly states it does not use crawled content for AI model training, advertising, or resale. Cached pages are refreshed on a 30‑day cycle, and content is stored in a datacenter in Osaka. FDSE provides a public interface for site owners to see how their pages are indexed.

⚙️ Rate Limiting Policy

Because the bot can issue burst requests when rediscovering large sites, administrators are advised to rate‑limit it to 15‑20 requests per second per IP address. This threshold prevents server overload while still allowing timely indexing, in accordance with FDSE’s own guidelines for cooperative crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.