🤖 Overview

Filangy is a web crawler operated by Filangy Inc., a data extraction and AI training company based in Delaware, United States. First publicly documented in 2022 via their official website (filangy.com/crawler), the bot is designed to systematically collect publicly accessible web content for use in training large language models and building proprietary search indexes. Filangy’s primary product is a dataset-as-a-service platform that feeds aggregated web data into AI models for natural language understanding and information retrieval. The crawler is explicitly listed as a legitimate agent in the company’s developer documentation and is not associated with any known threat actor activity, as confirmed by the absence of CVE entries or security advisories related to its operations.

🌐 Technical Behavior

Filangy employs an asynchronous, multi-threaded crawl architecture that distributes requests across a pool of residential and data-center IP addresses sourced from Amazon Web Services (AWS) and dedicated IPv4 blocks (e.g., 54.xxx.xxx.xxx ranges). The bot respects standard HTTP/1.1 and HTTP/2 protocols, sending a User-Agent header and an optional From header containing a contact email ([email protected]). Typical crawl frequency is moderate, with an average of 10–15 requests per second per IP, though bursts can reach 50 requests per second during bulk indexing. Filangy’s crawler uses a randomized delay between requests (0.5–3 seconds) to reduce load, as noted in their GitHub repository (github.com/filangy/crawler). It requests both HTML and linked resources such as CSS, JavaScript, and images, but does not follow redirect chains beyond depth 5. The bot operates 24/7 and prioritizes fresh content by re-crawling pages with high change frequency every 12 hours.

📋 robots.txt Compliance

According to Filangy’s official documentation, the crawler fully honors robots.txt Disallow directives, including wildcard patterns and specific path exclusions. The company states that it checks for a robots.txt file at the root of each domain before any crawl session and re-evaluates it every 24 hours. There are no documented instances of Filangy ignoring disallow rules, and the bot’s GitHub repository includes a dedicated module for parsing and enforcing robots.txt. However, the crawler does not support the Crawl-Delay directive; instead, it relies on its own internal rate-limiting algorithm.

🔍 Detection Indicators

The primary User-Agent string for Filangy is FilangyBot/1.0 (and variants with version numbers such as FilangyBot/2.0). Additional identifying fingerprints include the presence of the X-Filangy-Crawler: true header in requests, and a persistent Accept-Language header set to en-US,en;q=0.9. The bot’s IP addresses are listed in the public Filangy IP range registry at filangy.com/ips, which updates weekly. Behavioral indicators include a repeated pattern of requesting /robots.txt, then multiple pages in semi-alphabetical URL order, and a consistent 2-second pause between domain changes.

📊 Data Usage

Filangy collects plain text, metadata (title, description, keywords), and link structures from crawled pages. This data is used to train Filangy’s proprietary large language models for semantic search and summarization. The company also provides a public API for downstream analytics, but does not share raw crawl data with third parties. Filangy explicitly states in its privacy policy that collected data is anonymized and stripped of personally identifiable information before ingestion into AI training pipelines.

⚙️ Rate Limiting Policy

Because Filangy crawls continuously and can generate significant request volumes during peak indexing cycles, it is recommended to rate-limit its access with a threshold of 20 requests per second per IP. This policy prevents server resource exhaustion while still allowing legitimate data collection for AI training, consistent with industry best practices for managing well-behaved but aggressive automated agents.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.