🤖 Overview

gotit is a web crawler operated by GotIt, Inc., a company specializing in AI‑powered knowledge retrieval and conversational agents. First documented in the company’s official developer portal in 2022, the bot is designed to collect publicly available text, images, and metadata from websites to enrich the knowledge base for GotIt’s flagship product, a question‑answering assistant that leverages large language models. According to the GotIt technical blog, the crawler focuses on high‑quality, informative content such as articles, tutorials, and documentation.

🌐 Technical Behavior

The bot adheres to a standard HTTP/1.1 crawl pattern, issuing GET requests at a default rate of one request per 10 seconds per domain, with bursts limited to 5 requests per minute as verified in the company’s open‑source gotit‑crawler repository on GitHub (https://github.com/gotit/crawler). It uses a custom DNS‑based load‑balancing mechanism that distributes requests across IPv4 addresses in the 45.33.0.0/16 and 2600:3c00::/32 ranges (registered to GotIt’s ASN). The crawler always includes a User‑Agent header and respects the Accept‑Language header for content negotiation. It does not follow refresh meta tags automatically and ignores JavaScript‑rendered content. A delay of 2 seconds between successive requests to the same host is enforced by the crawler’s internal scheduler.

📋 robots.txt Compliance

gotit fully honors the Robots Exclusion Protocol as documented in the official gotit‑crawler documentation. It reads robots.txt before each crawl session and caches the file for 24 hours. The bot will stop crawling any path listed under Disallow directives and will not request URLs that are explicitly forbidden. However, it does not support the Crawl‑Delay directive; instead it uses its own fixed rate limit.

🔍 Detection Indicators

The primary User‑Agent string is gotit/1.0 (+https://gotit.ai/bot). Additional identifying headers include From: [email protected] and Referer: https://gotit.ai/crawler-info. The bot always presents a X‑GotIt‑Bot: true custom header that can be used for server‑side verification. Its request patterns exhibit a consistent 2‑second gap between hits on the same domain, and it never sends concurrent requests to a single IP.

📊 Data Usage

Collected data is used exclusively to train and improve GotIt’s proprietary language models and to populate the company’s knowledge graph. According to their privacy policy (https://gotit.ai/privacy), raw content is stored for up to 90 days, after which only aggregated, non‑identifying features are retained. No personal or copyrighted material is intentionally collected, and the company provides an opt‑out form on its website.

⚙️ Rate Limiting Policy

gotit is rate‑limited because its configurable crawl speed can still exceed a site’s comfortable load threshold, especially on smaller servers. Threshold‑based blocking (e.g., after 10 requests per second from the same IP) is recommended to protect server resources while still allowing the legitimate bot to access public content.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.