🤖 Overview

Hooblybot-Image is a legitimate web crawler operated by Hoobly, a classified advertisements platform similar to Craigslist. Its primary purpose is to fetch and index images from third-party listings hosted on external websites, feeding data into Hoobly’s image search and listing verification systems. According to publicly available user-agent registries and Hoobly’s own technical documentation, the bot was introduced to improve the accuracy and richness of visual content in classified results.

🌐 Technical Behavior

The bot exclusively requests image files (e.g., JPEG, PNG, GIF, WebP) and ignores non-image resources. It sends sequential GET requests with a delay of 5–10 seconds between requests to avoid overloading servers, as documented in Hoobly’s robot policy. The crawler originates from IP ranges associated with Hoobly’s infrastructure, which are listed in the ASN for Hoobly (AS397677). It uses HTTP/1.1 with standard headers, including a custom From field containing a contact email ([email protected]). The bot does not follow redirects to external domains unless explicitly allowed.

📋 robots.txt Compliance

Hooblybot-Image fully obeys robots.txt directives, including Disallow rules for specific paths and files. The official Hoobly bot policy (published at hoobly.com/robots.txt) states that the bot checks the file and respects Crawl-Delay instructions. There is no documented evidence of the bot ignoring Disallow entries; it adheres to both the standard and extended directives.

🔍 Detection Indicators

The primary User-Agent string is Hooblybot-Image/1.0 (extensible with version numbers). Additional headers include a From field with the email [email protected] and a Accept header limited to image MIME types. The bot does not use obfuscated or randomized user-agent strings, making it straightforward to identify in access logs. It also includes a Referer header pointing to the Hoobly platform.

📊 Data Usage

Collected images are stored and processed for Hoobly’s image search engine, which allows users to search classifieds by visual similarity. The data is also used for automated content moderation, such as detecting duplicate or inappropriate images in listings. No personally identifiable information is extracted from the images themselves; the bot only processes publicly accessible files.

⚙️ Rate Limiting Policy

While Hooblybot-Image is not malicious, its bulk image fetching can strain server resources if left unchecked. Rate limiting is recommended at 5 requests per second per IP, with a threshold block after 50 consecutive requests without the required delay, ensuring fair resource usage for all users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.