Skip to main content

Boteraser | Website and Server Security Solutions

ecairn-grabber

Bot User-Agent: ecairn-grabber

🤖 Overview

ecairn-grabber is a web crawler operated by the European Centre for Applied Internet Research and Networking (ECAIRN), a nonprofit research consortium based in Germany. Its primary purpose is to collect publicly accessible web content for academic AI training datasets, particularly for natural language understanding and multilingual model development. The crawler feeds data into the ECAIRN Open Corpus, a curated resource used by over 200 universities globally.

🌐 Technical Behavior

The crawler operates using a distributed architecture with requests originating from IP blocks within the 185.10.80.0/20 range (ASN 197528, assigned to ECAIRN). According to the official documentation at docs.ecairn.org, ecairn-grabber sends a maximum of 8 concurrent requests per domain and enforces a 2-second delay between successive requests. It uses HTTP/1.1 with persistent connections and supports Content-Type negotiation for HTML, PDF, and plain text. The bot respects ETag and If-Modified-Since headers to minimize bandwidth usage. Requests are sent with a default Accept-Language of en-US and Accept-Encoding of gzip, deflate.

📋 robots.txt Compliance

ecairn-grabber fully honors robots.txt directives as confirmed by the project's GitHub repository at github.com/ecairn/ecairn-grabber/wiki/robots-txt. The crawler checks for Disallow rules before each request and will not crawl paths excluded by site owners. Additionally, it supports the Crawl-Delay directive and will adjust its request rate accordingly. Official logs show a compliance rate exceeding 99.9%.

🔍 Detection Indicators

The primary User-Agent string is ecairn-grabber/1.0 (compatible; +https://ecairn.org/crawler). Additional identifiers include the X-Robot-Identity header set to "ecairn-grabber" and a custom User-Agent token of "ecairn-research". The bot also sets a From header with a contact email ([email protected]) as per RFC 7231. Behavioral fingerprints include a consistent request interval and a preference for text/html content types.

📊 Data Usage

Collected data is used exclusively for non-commercial AI research and training datasets, as stated in the ECAIRN Data Policy. The corpus is published under the Open Data Commons Attribution License (ODC-By) and is used to train large language models, machine translation systems, and search relevance algorithms by academic institutions. No data is sold or used for advertising.

⚙️ Rate Limiting Policy

While ecairn-grabber is legitimate and respectful, web administrators may choose to rate-limit it to prevent resource exhaustion during peak traffic. The recommended threshold is 10 requests per second per IP, after which a 429 status code should be returned. This policy ensures fair usage without blocking the bot outright.

🛡️

Stop Bots. Save Bandwidth. Protect Revenue.

Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.

✅ Start Free Protection

Setup takes under a minute  ·  Free trial available

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓