🤖 Overview

Fatbot is a web crawler operated by FatRank (formerly part of Search Engine Genie), a company specializing in SEO rank tracking and competitive analysis tools. First documented in public robots.txt discussions around 2016, Fatbot is designed to systematically scan websites to collect keyword rank data, page metadata, and link profiles for its search engine position monitoring service. The crawler feeds into FatRank’s dashboard, which provides clients with daily positional reports across major search engines.

🌐 Technical Behavior

Fatbot exhibits a moderate crawl rate, typically issuing one request every 1–2 seconds per domain, though bursts may occur during initial indexing. It follows standard HTTP/1.1 and HTTPS protocols, and its requests include a Fatbot/1.0 User‑Agent string plus an X‑Robots‑Tag header that can be used for selective blocking. According to publicly accessible server logs and the official FatRank crawler documentation (published at https://www.fatrank.com/crawler), Fatbot sources its IP addresses from a mix of dedicated datacenter blocks owned by companies like DigitalOcean and OVH, as well as AWS EC2 elastic IPs. It does not perform JavaScript rendering and only fetches HTML, CSS, and plain text content. The crawler respects Cache‑Control and Expires headers to reduce unnecessary re‑crawls.

📋 robots.txt Compliance

Fatbot fully obeys the Robots Exclusion Protocol as confirmed by FatRank’s own documentation and multiple independent tests reported in SEO forums. The crawler parses Disallow directives and also respects Crawl‑Delay instructions when present in robots.txt. However, if a site returns a 429 Too Many Requests status, Fatbot will back off for a longer period rather than retry immediately.

🔍 Detection Indicators

The primary identification is the User‑Agent string Fatbot/1.0, sometimes appearing as Mozilla/5.0 (compatible; Fatbot/1.0; +https://www.fatrank.com/crawler). Behavioral fingerprints include a consistent request interval (no random jitter), an Accept‑Language header set to en‑US, and a tendency to start crawling from a site’s homepage before traversing internal links in breadth‑first order. Server logs may also show the Referer header set to the Fatrank domain.

📊 Data Usage

Collected data—such as keyword rankings, page titles, meta descriptions, and outbound link counts—is compiled into FatRank’s subscription‑based analytics platform. This data is used solely for search engine rank tracking and competitive benchmarking; it is not employed for AI model training or any form of content republishing.

⚙️ Rate Limiting Policy

Although legitimate, Fatbot’s crawling can be aggressive when monitoring high‑value client domains, making rate limiting essential to prevent resource exhaustion. A threshold‑based block (e.g., triggering at 10 requests per 10 seconds) is recommended to align with standard polite crawling practices while still allowing the bot to retrieve necessary ranking data.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.