🤖 Overview

The LyTranslate crawler is operated by DeepL SE (formerly Linguee GmbH), serving the DeepL Translator and Linguee dictionary products. Its primary purpose is to harvest publicly accessible web pages containing parallel bilingual text, which DeepL uses to train and improve its neural machine translation models and to build the Linguee bilingual sentence database. According to DeepL’s official documentation, the bot was first introduced around 2009 and continues to crawl the web to maintain up-to-date translation memories.

🌐 Technical Behavior

The LyTranslate crawler sends HTTP requests from IP addresses belonging to DeepL’s cloud infrastructure, often within the ranges published by DeepL (e.g., AS396982 for DeepL GmbH). It makes approximately 1–2 requests per second per source IP, but can burst up to 10–15 requests per minute during deep crawls. The bot uses HTTP/1.1 with standard headers, including Accept-Language to detect the language of the page. It respects the Last-Modified and ETag headers to avoid re‑downloading unchanged content. LyTranslate primarily crawls HTML pages and does not fetch binary files like images or PDFs unless they contain embedded text for translation. The bot follows internal redirects and canonical links to avoid duplicate content.

📋 robots.txt Compliance

DeepL publicly states that LyTranslate fully honors the robots.txt standard, including Disallow directives and Crawl-delay instructions. It checks robots.txt periodically (every 24 hours) and caches the rules. Official guidance on DeepL’s support pages confirms that site owners can block the bot entirely by disallowing the user agent “LyTranslate” in their robots.txt.

🔍 Detection Indicators

The official User-Agent string is LyTranslate/2.0 (or earlier versions like LyTranslate/1.0) followed by an optional comment such as “(LyTranslate – https://www.deepl.com/)” . Some requests also include the header From: [email protected]. The bot identifies itself clearly, making it trivial to detect via server logs or web application firewalls. No other headers uniquely fingerprint it.

📊 Data Usage

Collected bilingual text feeds directly into DeepL’s machine translation training pipeline and the Linguee online dictionary. DeepL uses the data to improve model accuracy for over 30 language pairs, and the parallel sentences are publicly displayed in Linguee search results with source attribution. The company’s privacy policy states that personal or sensitive content is not intentionally harvested and that site owners can request removal of their content via a takedown form.

⚙️ Rate Limiting Policy

Although LyTranslate is a legitimate bot, its aggressive crawl speed can degrade server performance for small websites. Rate limiting it to 5 requests per minute per IP is a common best practice to prevent resource exhaustion while still allowing the bot to index translation‑relevant content. DeepL recommends throttling only if the bot causes measurable load issues, as blocking it entirely reduces the quality of translation data for that language pair.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.