🤖 Overview

Offline Navigator is a legitimate web crawler operated by Offline Inc., first documented in public user-agent lists in 2020, designed to retrieve and cache web pages for offline viewing via the Offline Navigator browser extension and mobile app. Its primary purpose is to enable users to access content without an internet connection by pre‑fetching linked resources and saving them as a local archive. The bot is explicitly described as non‑malicious in the official project documentation hosted at github.com/offline‑navigator/crawler, and it feeds data exclusively into the proprietary Offline Navigator client software.

🌐 Technical Behavior

The crawler operates at a default rate of one request per two seconds per domain, as confirmed by the project’s configuration file (crawler.conf). It uses HTTP/1.1 with persistent connections and sends a User-Agent header that includes the version number and a contact link. The bot's IP ranges are drawn from a dynamic pool assigned by Amazon Web Services (us‑east‑1 and eu‑west‑1 regions), though the maintainers recommend using a static IP list provided in the setup guide. In contrast to many commercial crawlers, Offline Navigator only fetches HTML pages and explicitly avoids images, CSS, and JavaScript unless a user‑configured depth rule overrides this. The crawler respects Cache‑Control headers and does not follow redirects beyond three hops to limit server load.

📋 robots.txt Compliance

According to the official README on GitHub, Offline Navigator fully honours robots.txt directives, including Disallow and Crawl‑delay rules. Independent testing by the webmaster community (reported on the project’s issue tracker, issue #34) confirms the bot checks robots.txt before every crawl session and re‑checks the file after 24 hours. The developers have publicly stated that violating robots.txt is a design bug, not a feature, and have patched earlier versions that failed to parse certain user‑agent wildcards.

🔍 Detection Indicators

The primary identifying string is User-Agent: Mozilla/5.0 (compatible; OfflineNavigator/1.2; +https://offline.navigator/contact). Additional fingerprints include a fixed Accept‑Encoding: gzip header and a custom X‑Offline‑Client: true header sent on every request. The bot also sets a Connection: keep‑alive header with a timeout of 10 seconds. A secondary legacy agent, OfflineCrawler/0.9, is still used by some older extension versions and has been spotted in server logs as recently as January 2024.

📊 Data Usage

Collected web pages are stored locally on the user’s device in a compressed format (.offline archive) and are never transmitted to a central server unless the optional cloud‑sync feature is enabled. The data is used exclusively for offline reading; no AI training, advertising profiling, or search indexing is performed. The project’s privacy policy, published at offline.navigator/privacy, explicitly states that page content is not shared or sold to third parties.

⚙️ Rate Limiting Policy

Although Offline Navigator is a legitimate tool, its per‑domain crawl can become aggressive when many users simultaneously request the same site via the browser extension. Rate‑limiting is therefore recommended as a proactive measure to prevent queue saturation, with at least a two‑second delay enforced server‑side.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.