🤖 Overview

SurdotlyBot is the official web crawler operated by Surdotly, a web directory and ranking service originally launched in 2016. Its primary purpose is to index publicly accessible web pages and collect metadata—such as page titles, descriptions, load times, and link structures—to populate the Surdotly directory and generate site popularity scores similar to the former Alexa ranking system. According to Surdotly’s official bot page at http://surdotly.com/bot.html, the crawler is designed to be non‑intrusive and respects standard web crawling conventions.

🌐 Technical Behavior

SurdotlyBot crawls using a multi‑threaded HTTP/1.1 client that follows 302 redirects and parses HTML, CSS, and JavaScript content to evaluate page structure. It typically issues requests at a rate of one per 2–5 seconds per domain, but can burst up to 10 requests per second when initial discovery is performed on new sites. The bot uses IPv4 addresses drawn from a documented range: 5.188.62.0/24 and 159.69.0.0/16 (as confirmed by Surdotly’s support documentation and third‑party reverse‑DNS lookups). It does not support HTTP/2 or QUIC, and always fetches pages over TCP port 80 or 443, honoring robots.txt crawl‑delay directives when specified.

📋 robots.txt Compliance

SurdotlyBot fully honors Disallow and Crawl-Delay directives found in robots.txt, as documented on Surdotly’s official bot page. It also respects X‑Robots‑Tag HTTP headers and meta robots tags (noindex, nofollow). However, the bot does not automatically reduce its crawl rate in response to 503 or 429 status codes unless a Crawl-Delay is explicitly set; webmasters are advised to use these directives to control access.

🔍 Detection Indicators

The primary User‑Agent string is surdotlybot/1.0 (+http://surdotly.com/bot.html). Secondary variants include SurdotlyBot/1.0 (Android‑based crawler) and surdotlybot-mobile/1.0. The bot does not present a custom Via or X‑Forwarded‑For header but always includes a From header with the email address [email protected] on request. Traffic originates from the ASN AS24940 (Hetzner) and AS51167 (Contabo).

📊 Data Usage

Collected data (page content, meta tags, outgoing links, response times) is used to build Surdotly’s public directory and generate site ranking scores displayed on surdotly.com. This includes aggregated analytics on page speed, uptime, and popularity—similar to legacy services like Alexa or SimilarWeb. No raw content is stored for AI training or sold to third parties; instead it is used exclusively for Surdotly’s own ranking algorithm and browser extension data.

⚙️ Rate Limiting Policy

While SurdotlyBot is legitimate and abides by standard crawling ethics, its burst behavior can overwhelm smaller servers. Therefore, rate limiting is recommended: set a Crawl-Delay of 5–10 seconds in robots.txt and block IPs from the known ranges if the bot exceeds 20 requests per minute. This ensures server stability without unfairly penalizing a cooperative crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.