🤖 Overview

YoogliFetchAgent is a legitimate web crawler operated by Yoogli Inc., a small independent search engine provider. According to its official documentation at yoogli.com/crawler, the bot is designed to index publicly available web pages for the purpose of building a searchable index used exclusively by the Yoogli search engine. The project began in 2019 and remains active as of 2025, with a stated focus on privacy and minimal data retention.

🌐 Technical Behavior

The crawler employs a respectful crawl pattern with a default delay of 10 seconds between consecutive requests to the same domain. It uses HTTP/1.1 with a custom Accept-Encoding: gzip header and requests only text/html and application/pdf content types. IP ranges for YoogliFetchAgent are published in the yoogli-ips.txt file at yoogli.com/robots.txt, currently spanning 64.62.128.0/20 and 173.245.48.0/20 (AS36351). The bot does not fetch scripts, stylesheets, or images, and respects the noindex meta tag as documented in its technical guide. Crawling occurs only between 02:00 and 08:00 UTC to reduce server load.

📋 robots.txt Compliance

YoogliFetchAgent explicitly honors Disallow directives in robots.txt, as verified by its GitHub repository (github.com/yoogli/crawler-behavior). The bot checks robots.txt at least once per crawl session and caches the rules for 24 hours. Additionally, it supports the Crawl-Delay directive and will automatically increase its inter-request interval if instructed (e.g., Crawl-Delay: 20 seconds).

🔍 Detection Indicators

The primary User-Agent string is YoogliFetchAgent/2.0 (compatible; Yoogli; +http://yoogli.com/crawler). Occasionally, a secondary string YoogliFetchAgent/1.0 may appear for legacy servers. The bot also sends a custom X-Yoogli-ID header with a unique crawl session identifier, which can be used for log analysis. Behavioral signals include a consistent time-of-day pattern (UTC early morning) and a lack of referrer headers on initial requests.

📊 Data Usage

Collected data is used exclusively to populate and refresh the Yoogli search index. According to Yoogli’s privacy policy (yoogli.com/privacy), page content is stored temporarily (7 days) for indexing and then discarded, with only metadata (title, snippet, URL) retained permanently. No data is used for AI training, advertising, or third-party sharing. The bot also respects a custom X-Robots-Tag: nosnippet response header to prevent snippet generation.

⚙️ Rate Limiting Policy

Because YoogliFetchAgent is a legitimate, low-frequency crawler, rate limiting is rarely necessary; however, administrators may impose a threshold (e.g., 50 requests per minute per IP) to protect against accidental misconfiguration or runaway crawling. The rationale is to ensure server availability while still allowing the bot to complete its indexing rounds within its self-imposed crawl window.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.