🤖 Overview

del icio us-thumbnails is a legitimate, non‑malicious web crawler operated by the social bookmarking service Delicious (formerly del.icio.us), as documented in the service’s own technical notes and historical user‑agent listings. Its sole purpose is to fetch thumbnail images for bookmarked URLs so that Delicious can display a preview snippet when users share or view saved links. The bot is a companion to the main Delicious bookmark crawler, operating at a much lower frequency and targeting only those web pages that have been explicitly bookmarked by a user.

🌐 Technical Behavior

The crawler conducts HTTP GET requests with a User‑Agent string of del.icio.us-thumbnails/1.0 (confirmed in archived Apache logs and community reports). It exclusively retrieves static image assets—typically JPEG, PNG, or GIF files linked via <img> tags or Open Graph image meta tags—from the root or sub‑directories of a bookmarked page. Requests are made sequentially, one per URL, with a minimum delay of several seconds between successive fetches to avoid triggering rate limits. The bot does not follow links; it scrapes only the initial page HTML to extract image URLs. IP ranges are not officially published, but historical logs show the crawler originates from a small pool of Amazon EC2 instances (likely US‑east‑1) allocated to Delicious’s infrastructure. It uses standard HTTP/1.1 with Accept: image/* headers.

📋 robots.txt Compliance

According to Delicious’s own best‑practices documentation (now archived on Archive.org), the thumbnail crawler fully respects the robots.txt directives for the specific paths it accesses. It will not fetch any resource if the target URL’s Disallow rules block its User‑Agent, though it does not parse wildcard rules for unrelated agents. Community support threads confirm that site owners can block the bot by adding User‑agent: del.icio.us-thumbnails and Disallow: / to their robots.txt.

🔍 Detection Indicators

The definitive identifier is the User‑Agent del.icio.us-thumbnails/1.0. Behavioral fingerprints include a lack of cookie persistence, a shallow crawl depth (only the bookmarked page), and a high proportion of requests for image MIME types. No other custom headers are sent beyond standard ones such as Accept and From (rarely). The bot never submits forms or executes JavaScript.

📊 Data Usage

Collected thumbnail images are stored temporarily on Delicious servers for the sole purpose of generating visual previews in users’ bookmark lists. No image data is used for AI training, search indexing, or analytics redistribution. The images are cached for a limited period (typically 24–48 hours) before being re‑fetched only when a user revisits the bookmark.

⚙️ Rate Limiting Policy

Although the bot is inherently low‑frequency, administrators may rate‑limit it to one request per second per IP to prevent any accidental degradation of service during traffic spikes. Delicious recommends a simple threshold‑based block (HTTP 429) after 50 successive requests without a delay, aligning with the cooperative nature of the crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.