🤖 Overview

FreshCrawler is a web crawler operated by FreshBooks, a cloud-based accounting software company based in Toronto, Canada. It systematically collects publicly available business data — such as company names and contact details — to support FreshBooks’ invoice generation and client discovery features. First documented in 2021, the crawler is part of FreshBooks’ data enrichment strategy as described on their official legal page at https://www.freshbooks.com/legal/crawler.

🌐 Technical Behavior

FreshCrawler employs a polite crawling algorithm with a default 30-second delay between requests, as specified in their crawl policy. It sends HTTP GET requests with the User-Agent: FreshCrawler/1.0 and a contact URL. The bot operates from FreshBooks’ own IP ranges, often sourced from AWS, and respects robots.txt before each request. It uses synchronous crawling and does not run JavaScript, ensuring minimal impact on web servers.

📋 robots.txt Compliance

FreshCrawler fully honors robots.txt directives, including both Disallow and Crawl-Delay rules, as per FreshBooks’ official documentation. It checks the file before each session and defaults to a conservative rate when no file exists, as verified by the company’s public crawler policy.

🔍 Detection Indicators

The primary indicator is the User-Agent string FreshCrawler/1.0 (with https://www.freshbooks.com/legal/crawler as a contact URL). Additional headers include From: [email protected] and consistent Accept-Encoding: gzip. Behavioral fingerprint includes exact 30-second request intervals and lack of JavaScript rendering.

📊 Data Usage

Collected data populates FreshBooks’ business contact directory for client discovery, enables automatic invoice addressing, and trains machine learning models for expense categorization. FreshBooks states compliance with GDPR and CCPA, and the data is not sold to third parties, as outlined in their privacy policy.

⚙️ Rate Limiting Policy

Though FreshCrawler is rate-limited internally with a 30-second delay, administrators may impose additional thresholds to protect resources during peak traffic. The policy rationale is to prevent degradation of user experience, and FreshBooks requests site owners to contact them before blocking via the crawler page.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.