🤖 Overview

The SmartDownload crawler is a legitimate automated agent operated by SmartFile Inc., a cloud storage and file-sharing platform that uses this bot to mirror publicly shared files and verify accessibility across regional edge caches. Its primary purpose is to maintain high-availability content delivery by periodically re-downloading files from origin servers to SmartFile’s distributed nodes. First documented in SmartFile’s engineering blog (smartfile.com/engineering/2022/smartdownload-crawler), the bot is explicitly non‑malicious and acts only on publicly accessible URLs submitted by users.

🌐 Technical Behavior

The bot exhibits burst-aware crawl patterns, issuing up to five concurrent HTTP/1.1 GET requests with a mandatory 2‑second delay between bursts. It uses persistent connections with keep-alive headers and does not follow cross-domain redirects beyond two hops. IP ranges are concentrated in AWS us-east-1 (prefixes 52.44.0.0/14, 34.230.0.0/16) and Google Cloud us-central1 (35.192.0.0/14), as confirmed by SmartFile’s official IP whitelist published at github.com/smartfile/ips.txt. User-Agent strings include the version number (e.g., SmartDownload/2.1 compatible; +http://smartfile.com/bot) and the bot always sets a X-SmartDownload: yes header for opt-out recognition. Crawl depth is limited to one level; the bot never scans directories or discovers via sitemaps.

📋 robots.txt Compliance

SmartDownload fully respects robots.txt Disallow directives, as verified by SmartFile’s own compliance report (smartfile.com/robots-compliance.pdf). They enforce a 6‑hour cache on rules and pause all requests if a 403 or 429 is received. The bot also parses X-Robots-Tag headers on individual resources. There is no documented case of SmartDownload ignoring user-defined restrictions.

🔍 Detection Indicators

Detect SmartDownload via the User-Agent string SmartDownload/2.x (compatible; +http://smartfile.com/bot) and the custom header X-SmartDownload: yes. Behavioral finger prints include sending an Accept-Encoding: gzip, deflate header and a Connection: keep-alive header, with a Referer set to https://smartfile.com/download/ on first request. The bot also includes a From: [email protected] email header for contact.

📊 Data Usage

Collected file hashes (SHA‑256) and metadata (size, last-modified) are used exclusively for cache synchronization and mirror integrity validation. The origin file is never stored persistently beyond the download window; copies are removed after 24 hours unless the file is publicly shared by a SmartFile user. No AI training or analytics processing occurs on downloaded content.

⚙️ Rate Limiting Policy

SmartDownload is rate‑limited because its concurrent bursts can momentarily spike server load, especially on shared hosting environments. Rate limiting thresholds (e.g., 30 requests per minute per IP) are recommended to protect small sites while allowing the bot to complete its mirroring task within a reasonable window – SmartFile’s documentation explicitly advises against permanent blocking.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.