🤖 Overview

The Isara crawler is operated by Isara AG, a Swiss company focused on semantic search and AI-driven data aggregation, first publicly documented in 2021 according to the official website isara.ch. Its purpose is to collect publicly accessible web content to feed into Isara’s proprietary semantic search engine and machine learning models for context-aware information retrieval and training datasets.

🌐 Technical Behavior

The Isara crawler employs a combination of breadth-first and curated crawl strategies, requesting pages at a rate of 2–5 requests per second with random jitter to reduce server load, as noted in the crawler documentation at isara.ch/crawler. It preferentially crawls HTTPS sites and respects Content-Type headers, focusing on HTML, PDF, and XML feeds. IP ranges are primarily assigned from ASN 62161 (Isara AG) and include subnets 185.123.100.0/24 and 185.124.200.0/24, verified via whois records from RIPE NCC. The crawler uses HTTP/1.1 and HTTP/2 protocols and includes a From header with a contact email: [email protected].

📋 robots.txt Compliance

According to Isara’s official crawling policy, the bot fully respects Disallow directives and also honors the Crawl-Delay directive when specified, as documented in their robots.txt sample at isara.ch/robots.txt. Documentation confirms that the crawler does not index content blocked by robots.txt and will re-evaluate changes on a weekly basis, ensuring compliance with site owner preferences.

🔍 Detection Indicators

The User-Agent string is Isara/1.0 (compatible; +https://isara.ch/crawler) with a secondary identifier IsaraBot, as listed on useragentstring.com. Behavioral fingerprints include a consistent request header ordering and a 60-second timeout between crawl batches, plus a custom X-Isara-Crawl: 1 header. The crawler also sends an Accept-Language: en-US,en;q=0.9 header.

📊 Data Usage

Collected data is used to train Isara’s semantic search algorithms and improve the relevance of search results in their public search engine, as stated in the privacy policy at isara.ch/privacy. Additionally, content is processed for AI summarization and entity recognition models, with data retained for up to 90 days before anonymization.

⚙️ Rate Limiting Policy

While entirely legitimate, the Isara crawler can be aggressive during initial scans of large domains, generating up to 500 requests per minute according to observed patterns. Rate limiting is recommended to prevent resource exhaustion, with a threshold of 10 requests per second providing a reasonable balance between allowing legitimate crawling and protecting server performance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.