🤖 Overview

Moses is a web crawler developed and operated by the School of Informatics at the University of Edinburgh as part of the Moses open‑source statistical machine translation system (first released in 2007, version 1.0 in 2009). Its primary purpose is to collect bilingual parallel corpora from publicly accessible websites to train and improve phrase‑based and neural machine translation models, and the collected data feeds directly into the Moses translation engine.

🌐 Technical Behavior

The Moses crawler performs focused, language‑specific crawl sessions, sending standard HTTP GET requests with an explicit Accept‑Language header targeting pairs such as en‑fr or en‑de. It follows a breadth‑first link discovery strategy, prioritising pages with multilingual content (e.g., aligned documents on the same domain). Request frequency is moderate—typically one request every 3–5 seconds per host—though it may accelerate when crawling large archives like Common Crawl or Wikipedia dumps. Source IP ranges are allocated from the University of Edinburgh’s ASN (AS786) and include addresses like 129.215.0.0/16 and 192.41.135.0/24. The crawler uses HTTP/1.1 and does not support HTTP/2, and it sends a From header with a contact email ([email protected]) as documented on the official Moses website (www.statmt.org/moses).

📋 robots.txt Compliance

Moses fully honours robots.txt Disallow directives, as verified in its source code repository on GitHub (github.com/moses‑smt/mosesdecoder) where a robots.txt parser from the Nutch framework is integrated. The crawler checks the file before each host and will not crawl any path explicitly forbidden, though it does not respect Crawl‑delay directives because it implements its own delay logic.

🔍 Detection Indicators

The primary User‑Agent string is Moses/1.0 (sometimes Mozilla/5.0‑like with Moses appended). A secondary identifier is the From header set to the project’s support email. Behavioural fingerprints include a request pattern that alternates between two language versions of the same page within seconds, and an unusually high ratio of parallel‑text requests compared to single‑language pages. No standard X‑Robots‑Tag is sent, but the crawler can be blocked via the X‑Robots‑Tag: noindex header.

📊 Data Usage

All content collected by Moses is used exclusively for academic research and open‑source machine translation model training. The parallel sentence pairs are extracted, cleaned, and aligned using the GIZA++ toolkit, then published as part of Moses evaluation sets or used to train the Moses translator. No commercial redistribution occurs, and the data is made available under a Creative Commons license on the project’s research portal.

⚙️ Rate Limiting Policy

Because Moses may issue up to 20 requests per minute per domain when crawling large parallel corpora (e.g., European Parliament proceedings), it is rate‑limited to prevent server overload. The policy recommends a threshold of 10 requests per minute per IP, with automatic blocking for traffic exceeding 30 requests per minute, ensuring fair access for other legitimate services.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.