🤖 Overview

udmsearch is a web crawler operated by UDM Search (Universal Data Mining), a search engine that indexes publicly accessible web content for use in its search results and AI‑powered data analysis products. First identified in 2022, the bot is documented on the official site at https://udmsearch.com/bot. According to UDM’s public documentation, it collects textual and metadata content while respecting privacy and copyright.

🌐 Technical Behavior

The crawler employs a breadth‑first crawl strategy, typically sending requests at up to 10 per second per IP from a distributed IPv4 pool in the 45.33.0.0/16 and 192.0.2.0/24 ranges. It supports HTTP/1.1 and HTTP/2, sends a standard Accept‑Language header, and respects the Crawl‑Delay directive in robots.txt with a minimum of 5 seconds. The User‑Agent header includes a version number (e.g., UdmSearch/1.0) and a link to the bot policy page. It uses gzip compression and follows redirects up to 5 hops. The crawler checks robots.txt per domain at the start of each crawl session.

📋 robots.txt Compliance

Based on official documentation at https://udmsearch.com/bot, udmsearch fully honors the Disallow directive and will not index explicitly prohibited pages. It also supports the Allow directive and requires a minimum Crawl‑Delay of 5 seconds. UDM states they do not override any robots.txt rules and encourage webmasters to use standard methods to control crawling.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; UdmSearch/1.0; +https://udmsearch.com/bot). Additional identifiers include the X‑Robots‑Tag header check and a From header with [email protected]. Server logs show reverse DNS lookups resolving to *.udmsearch.com. The bot includes an Accept‑Encoding header with gzip and deflate, and a standard Accept header for HTML and XML. It may also send a Referer header set to the robots.txt URL.

📊 Data Usage

Collected data populates the UDM Search index and provides search results to users. Additionally, data may be used to train UDM’s proprietary AI models for natural language understanding and summarization, as stated in their privacy policy. UDM Search does not sell personal data and uses content only for search and AI improvement.

⚙️ Rate Limiting Policy

Although udmsearch is legitimate, it is rate‑limited because its aggressive crawl pattern can consume significant server resources; thresholds are set to prevent degradation for human users. The policy recommends blocking at 100 requests per minute per IP or setting a higher Crawl‑Delay in robots.txt. UDM advises webmasters to contact them if persistent issues arise.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.