🤖 Overview

Funnelback is an enterprise search crawler developed by Funnelback Pty Ltd, an Australian company acquired by Missionsky in 2022. Its primary purpose is to index web content, documents, and intranets for use in Funnelback’s enterprise search platform, delivering relevant search results to organizations (source: docs.funnelback.com).

🌐 Technical Behavior

The Funnelback crawler performs scheduled, incremental crawls using standard HTTP/1.1 requests and respects Cache-Control and Last-Modified headers to avoid redundant downloads. It uses a configurable crawl delay (default 1 second) and follows robots.txt directives. IP ranges are not publicly documented; crawls originate from the organization’s outbound IP pool, often varying by deployment. The crawler supports both depth-first and breadth-first strategies, with a maximum crawl depth of 20 by default (source: docs.funnelback.com/crawler-configuration). It can also authenticate via forms, NTLM, or SSO for intranet crawling.

📋 robots.txt Compliance

Funnelback fully honors robots.txt Disallow directives and supports the Crawl-Delay directive. Official documentation explicitly states that the crawler reads robots.txt at the start of each crawl cycle and caches it for the session (source: docs.funnelback.com/robots-txt). Non-compliance would violate the intended enterprise search functionality.

🔍 Detection Indicators

The primary User-Agent string is Funnelback/1.0, often preceded by a Mozilla compatibility token: Mozilla/5.0 (compatible; Funnelback/1.0; +https://www.funnelback.com/). The crawler may also appear as Mozilla/5.0 (compatible; Funnelback; +https://www.funnelback.com) (source: user-agents.net). It does not set unique custom headers; identifying it relies on the User-Agent field and consistent request intervals.

📊 Data Usage

Collected content is indexed into Funnelback’s search engine for internal or external enterprise search applications. The data is not used for AI training or public aggregation; it serves only the organization that owns the Funnelback instance. The crawler’s logs are available to administrators for auditing (source: docs.funnelback.com/crawl-logs).

⚙️ Rate Limiting Policy

Rate limiting is recommended for Funnelback because its configurable crawl rate can be set too high for smaller servers, causing performance degradation. A typical threshold is 10 requests per second per IP, adjusted based on server capacity and the Crawl-Delay value in robots.txt. This ensures equitable access without disrupting other services.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.