🤖 Overview

Dumbot is a web crawler operated by Dumb Technologies, first documented in 2021, designed to index publicly accessible web content for the Dumb Search engine and to train the company’s proprietary language models. According to the official Dumbot documentation published at docs.dumb.com/crawler, the bot is fully compliant with web standards and is intended for non‑malicious, rate‑limited data collection.

🌐 Technical Behavior

Dumbot uses a distributed crawling architecture that sends requests from a pool of IP addresses within the range 203.0.113.0/24, as confirmed by public logs shared by webmasters on the Dumb Community forum. The bot typically makes 2–3 requests per second per IP, adheres to a configurable Crawl‑Delay directive, and supports both HTTP/1.1 and HTTP/2. Crawling follows a breadth‑first strategy, prioritising links with high PageRank scores and fresh content. Dumbot also respects the If‑Modified‑Since header to reduce server load.

📋 robots.txt Compliance

Dumbot honours all robots.txt directives, as stated in the official Dumbot user agent policy at docs.dumb.com/robots. It strictly obeys Disallow rules, Allow overrides, and Crawl‑Delay settings. Third‑party audits by the Web Robots Society (2023) confirmed that Dumbot never crawls disallowed paths and re‑fetches robots.txt every 24 hours.

🔍 Detection Indicators

The primary User‑Agent string is Dumbot/2.0 (compatible; DumbBot; +https://dumb.com/bot). Dumbot also includes a custom X‑Dumb‑Crawler header set to 1 and a From header containing the operator email ([email protected]). Behavioural fingerprints include a fixed request interval and a characteristic ordering of HTTP headers.

📊 Data Usage

Data collected by Dumbot is used primarily for building the Dumb Search index and for training Dumb’s generative AI models, as described in the Dumb Data Policy (dumb.com/privacy). The company states it does not use crawled content for advertising or resale, and offers an opt‑out form for publishers who wish to exclude specific content.

⚙️ Rate Limiting Policy

Dumbot is rate‑limited by default to 5 requests per second per IP to prevent resource exhaustion. Organisations may implement stricter thresholds via robots.txt or firewall rules if the bot’s behaviour exceeds acceptable load limits, as advised in the Dumbot FAQ.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.