ecairn-grabber
ecairn-grabber is a web crawler operated by the European Centre for Applied Internet Research and Networking (ECAIRN), a nonprofit research consortium based in Germany. Its primary purpose is to collect publicly accessible web content for academic AI training datasets, particularly for natural language understanding and multilingual model development. The crawler feeds data into the ECAIRN Open Corpus, a curated resource used by over 200 universities globally.
The crawler operates using a distributed architecture with requests originating from IP blocks within the 185.10.80.0/20 range (ASN 197528, assigned to ECAIRN). According to the official documentation at docs.ecairn.org, ecairn-grabber sends a maximum of 8 concurrent requests per domain and enforces a 2-second delay between successive requests. It uses HTTP/1.1 with persistent connections and supports Content-Type negotiation for HTML, PDF, and plain text. The bot respects ETag and If-Modified-Since headers to minimize bandwidth usage. Requests are sent with a default Accept-Language of en-US and Accept-Encoding of gzip, deflate.
ecairn-grabber fully honors robots.txt directives as confirmed by the project's GitHub repository at github.com/ecairn/ecairn-grabber/wiki/robots-txt. The crawler checks for Disallow rules before each request and will not crawl paths excluded by site owners. Additionally, it supports the Crawl-Delay directive and will adjust its request rate accordingly. Official logs show a compliance rate exceeding 99.9%.
The primary User-Agent string is ecairn-grabber/1.0 (compatible; +https://ecairn.org/crawler). Additional identifiers include the X-Robot-Identity header set to "ecairn-grabber" and a custom User-Agent token of "ecairn-research". The bot also sets a From header with a contact email ([email protected]) as per RFC 7231. Behavioral fingerprints include a consistent request interval and a preference for text/html content types.
Collected data is used exclusively for non-commercial AI research and training datasets, as stated in the ECAIRN Data Policy. The corpus is published under the Open Data Commons Attribution License (ODC-By) and is used to train large language models, machine translation systems, and search relevance algorithms by academic institutions. No data is sold or used for advertising.
While ecairn-grabber is legitimate and respectful, web administrators may choose to rate-limit it to prevent resource exhaustion during peak traffic. The recommended threshold is 10 requests per second per IP, after which a 429 status code should be returned. This policy ensures fair usage without blocking the bot outright.
Similar Threats
🛡️
Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.
✅ Start Free ProtectionSetup takes under a minute · Free trial available
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.