🤖 Overview

Trafilatura is an open-source Python library and command-line tool developed and maintained by Adrien Barbaresi, primarily hosted on GitHub at https://github.com/adbar/trafilatura. It is designed for high-quality web scraping that focuses on extracting clean, structured text content from HTML pages, discarding boilerplate such as navigation menus, ads, and sidebars. Trafilatura is widely used by academic researchers, data journalists, and NLP practitioners to build datasets for text mining, machine learning training, and digital humanities projects. The tool prioritizes precision and recall, supporting multiple output formats including plain text, Markdown, and XML.

🌐 Technical Behavior

Trafilatura operates as a batch or single-page crawler, typically launched by users rather than running as a persistent autonomous agent. When configured to crawl, it sends HTTP/1.1 requests with a configurable user-agent string (see Detection Indicators) and follows HTTP redirects. Default request frequency is not hardcoded; the library leaves rate limiting to the user, but official documentation advises implementing delays (e.g., time.sleep(1)) to avoid overloading servers. Trafilatura uses Python's requests library for HTTP calls and can optionally use lxml or html5lib parsers. It does not maintain a fixed IP range; requests originate from the IP address of the machine running the script. The tool supports both recursive crawling (via trafilatura.crawler) and single-page extraction, and it can handle sitemaps for discovery.

📋 robots.txt Compliance

According to the official documentation on the GitHub repository, Trafilatura does not automatically enforce robots.txt rules by default, but it provides a built-in function trafilatura.check_robots() that allows users to verify compliance before crawling. The developer explicitly recommends respecting robots.txt and the Crawl-Delay directive in the project's README. Many deployed instances, especially in research pipelines, incorporate this check to avoid legal or ethical issues, though individual users may choose to ignore it.

🔍 Detection Indicators

The default user-agent string for Trafilatura is "Trafilatura/1.7.0" (or the current version number, e.g., 1.9.0), often followed by a Python-requests version. The library also allows custom user-agent specification. Behavioral fingerprints include requests for a single page with subsequent requests for resources like CSS/JS usually absent (since it only fetches HTML). HTTP headers typically include Accept-Language: en and Accept-Encoding: gzip. Unlike major search engine bots, Trafilatura does not send a From header or identify itself beyond the user-agent string.

📊 Data Usage

Data collected by Trafilatura feeds into various non-commercial and commercial projects: NLP datasets (e.g., for training language models or sentiment analysis), academic research (e.g., studying historical newspaper archives), and content aggregation systems. The tool is specifically designed to extract article bodies and main text, ignoring images and scripts, making it ideal for building text corpora. Some users feed extracted text into larger pipelines for keyword extraction or as input for LLM training, though the original project does not directly serve any specific AI model vendor.

⚙️ Rate Limiting Policy

Because Trafilatura can be configured to send rapid successive requests (especially in crawling mode), it should be rate-limited to prevent excessive load. The policy rationale is that no single user's script should consume enough server resources to degrade performance for other visitors; a safe default is a delay of 1–2 seconds between requests and respecting any Crawl-Delay directive found in robots.txt.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.