🤖 Overview

FileHound is a legitimate web crawler operated by FileHound Ltd., a UK‑based enterprise document management and workflow automation company founded in 2005. Its purpose is to scan publicly accessible web directories and document repositories to index files (primarily PDF, DOCX, XLSX, PPTX) for integration into FileHound’s cloud‑based document management platform, enabling automated classification and search. Official documentation from filehound.com/crawler states the bot is designed solely for indexing public documents, not for AI training or data resale.

🌐 Technical Behavior

The crawler employs HTTP/1.1 and HTTPS protocols, with a documented maximum rate of 5 requests per second per IP (according to the FileHound Robotics Policy). It follows a breadth‑first crawl pattern, starting from a seed URL and only following links that point to document‑extension URLs or directory listings. IP ranges are not publicly fixed but have been observed originating from AWS EC2 (us‑east‑1, eu‑west‑2) and UK‑based dedicated servers. The crawler does not execute JavaScript or parse dynamic content. Each request includes a custom X‑FileHound‑Bot header set to “true” and a Contact‑email header referencing [email protected]. The bot respects Cache‑Control and Last‑Modified headers to avoid redundant re‑crawls.

📋 robots.txt Compliance

FileHound’s crawler fully respects robots.txt directives, including Disallow, Crawl‑Delay, and Allow rules. The company provides a dedicated opt‑out mechanism via a configuration file at filehound.com/opt‑out, where webmasters can permanently block all scans by adding their domain. This compliance is documented in the FileHound Crawler FAQ, which states that the bot checks robots.txt before every crawl and honors wildcard patterns.

🔍 Detection Indicators

The primary detection indicator is the User‑Agent string: FileHound/1.0 (compatible; +https://filehound.com/crawler). Secondary fingerprints include the X‑FileHound‑Bot header, a consistent Accept header of “application/pdf,application/msword,application/vnd.openxmlformats‑officedocument.*”, and a Referer header set to “https://filehound.com/bot”. Behaviorally, the bot never requests HTML pages unless they are directory indices, and it stops immediately upon receiving a 403 or 429 response.

📊 Data Usage

Collected document metadata (filename, size, modification date) and full text content are stored in FileHound’s encrypted cloud storage for use by enterprise customers. The data powers search indexing, automatic document classification, and workflow routing within the FileHound platform. No data is shared with third parties, and the company’s privacy policy (filehound.com/privacy) guarantees deletion upon written request from the domain owner.

⚙️ Rate Limiting Policy

Rate limiting is recommended because FileHound’s crawler, while polite, can generate significant load on small servers if multiple document directories are present simultaneously. A threshold of 30 requests per minute per IP is standard, with a 429 Too Many Requests response triggering an exponential backoff period of 60 seconds. FileHound’s own documentation supports this approach as a best practice to prevent resource exhaustion. Use of mod_evasive or similar tools is advised for high‑traffic environments.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.