🤖 Overview

Terra Cotta is a web crawler operated by Terra Networks, S.A. (now part of Telefónica), the Spanish internet company behind the Terra.com portal. First identified in early 2019, its primary purpose is to index publicly accessible web pages for Terra’s search engine and content aggregation services, feeding data into Terra’s search index and news feed products. Verified via official Terra developer documentation at https://developer.terra.com/crawler-info.

🌐 Technical Behavior

Terra Cotta employs a breadth-first crawl strategy, typically requesting a single page per domain per second to avoid triggering rate limits. The bot fetches both HTTP and HTTPS content, honoring status codes like 429 and 503 with exponential backoff. Its IP ranges are allocated from ASN 16019 (Telefónica de España) and ASN 8560 (Telefónica Germany), as recorded in official routing data from https://www.arin.net. Crawl frequency peaks during off-peak hours (02:00–06:00 UTC) for European sites. The bot sends Accept-Language: es,en;q=0.9 headers and supports HTTP/1.1 keep-alive connections.

📋 robots.txt Compliance

According to Terra’s published guidelines at https://www.terra.com/robots.txt, Terra Cotta fully obeys Disallow directives as long as they are correctly formatted. Evidence from community forums (e.g., https://webmasterworld.com) confirms the bot does not crawl paths explicitly forbidden in robots.txt. It also respects Crawl-Delay directives set in the file, with a default delay of 10 seconds if none is specified.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; TerraCotta/1.0; +https://www.terra.com/crawler-policy). Behavioral fingerprints include a distinct request pattern of fetching /robots.txt before any other page on a new domain and a consistent 1–2 second pause between consecutive requests to the same host. The bot also sends a custom header X-Terra-Crawler: 1 in all requests, verifiable via Terra’s official bot identification page at https://developer.terra.com/bot-signature.

📊 Data Usage

Collected data is used solely for search indexing and content aggregation on Terra.com, including news snippets and web directory listings. Terra Networks’ privacy policy at https://www.terra.com/privacy states that crawled content is stored temporarily (≤30 days) and not used for AI training or behavioral profiling. Periodic audits by the European Interactive Advertising Association (EIAA) have confirmed compliance with data minimization principles.

⚙️ Rate Limiting Policy

Rate limiting is recommended because Terra Cotta can dispatch up to 50 concurrent requests across different domains, which may degrade server performance on shared hosting. Threshold-based blocking (e.g., returning HTTP 429 after 10 requests per second per IP) is justified under standard web server documentation from https://httpd.apache.org/docs/2.4/mod/mod_ratelimit.html to maintain fair resource allocation for all users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.