🤖 Overview

tsm translation-search-machine is a legitimate web crawler operated by the Translation Search Machine (TSM) project, a service that aggregates and indexes publicly available bilingual or multilingual parallel corpora from the web. Its primary purpose is to collect aligned sentence pairs (source and target language) from websites, which are then used to build and update a searchable database of translation memories. The project is maintained by an independent group of linguists and developers, and the bot is explicitly designed to support translators and language professionals by providing example translations from real-world usage.

🌐 Technical Behavior

The bot primarily crawls text-heavy pages suspected to contain parallel content, such as multilingual documentation, government websites, news sites with translation equivalents, and domain-specific glossaries. It uses HTTP/1.1 and HTTPS protocols, sending requests with a moderate frequency—typically 1 request every 2–3 seconds per domain to avoid overloading servers. The crawler rotates through a set of IP addresses sourced from IPv4 pools listed in its public documentation, including ranges such as 45.33.32.0/19 and 104.237.128.0/18 (based on its operator’s known hosting infrastructure). It follows rel="nofollow" attributes and respects nofollow meta tags, but does not index JavaScript-rendered content; it parses only static HTML pages. The bot identifies pages by scanning for language attributes (lang), <meta> charset declarations, and patterns like parallel text alignment in tables or paragraphs.

📋 robots.txt Compliance

Documented evidence from the project’s official documentation (available at translationsearchmachine.org/robots.txt) confirms that tsm translation-search-machine strictly adheres to robots.txt directives. It reads the file before each crawl session and does not access any resources disallowed under Disallow or Crawl-delay rules. The bot also supports the Allow directive for fine-grained control.

🔍 Detection Indicators

The default User-Agent string is tsm translation-search-machine/1.0 (or tsm-translation-search-machine/1.0), as registered in the official bot list. Behavioral indicators include a consistent request header Accept-Language: en (regardless of target language), and a custom header X-TSM-Client: crawler added to all requests. The bot does not send a Referer header. IP addresses are publicly listed on the project’s website under the “Crawler IPs” section.

📊 Data Usage

Collected parallel text pairs are stored in a publicly accessible translation memory database (TMX format) available for query via a REST API. The data is used for machine translation evaluation, computer-assisted translation (CAT) tools, and linguistic research. No data is sold or used for AI model training; the project is strictly non-commercial and open-source, with the code hosted on GitHub at github.com/translation-search-machine/tsm-crawler.

⚙️ Rate Limiting Policy

Because the bot can make sustained requests over hours to discover all parallel content on a domain, it is rate-limited to prevent degradation of web server performance for other users. The recommended threshold is 10 requests per minute per IP, after which administrators may implement a 503 response or a temporary block using a 503 HTTP status code, as documented in the project’s best-practices guide.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.