🤖 Overview

The Lydia Entity Spider is a legitimate web crawler operated by Lydia AI (formerly part of the Lydia Technologies group), a company specializing in entity extraction and knowledge graph construction for enterprise AI applications. First documented in 2022, its primary purpose is to systematically collect publicly available web content—including news articles, blog posts, product pages, and technical documentation—to build a structured entity database used to train and improve Lydia’s proprietary entity resolution and relationship extraction models. The bot feeds data into Lydia’s core product suite, which includes the Lydia Entity API, the Lydia Knowledge Graph service, and downstream AI analytics platforms used by clients in finance, healthcare, and cybersecurity.

🌐 Technical Behavior

The spider employs a standard breadth-first crawl pattern, starting from seed URLs provided via a configurable sitemap-based discovery mechanism. It respects the Crawl-Delay directive in robots.txt when present, defaulting to a delay of 2 seconds between requests if no directive is specified. Requests are made using HTTP/1.1 with the Accept-Encoding: gzip header to compress responses. The bot uses a rotating set of public IP addresses primarily drawn from AWS EC2 (us-east-1, eu-west-2) and Google Cloud (us-central1, europe-west1) ranges, as documented in the official Lydia AI crawling policy page at lydia.ai/crawler. Concurrent requests are limited to 5 per domain to balance thoroughness with politeness, and the crawler skips resources larger than 10 MB to avoid straining servers.

📋 robots.txt Compliance

According to the official Lydia AI Crawler Documentation (lydia.ai/robots.txt), the Lydia Entity Spider fully honors Disallow directives and respects both the User-agent: * block and bot‑specific User-agent: LydiaEntitySpider rules. Independent analyses by WebmasterWorld (2023) and the enterprise SEO site BotDaddy.io confirm that the spider never accesses URLs listed in disallowed paths and also obeys Allow overrides when present. The same documentation explicitly states that the crawler will cease all requests for a domain if it receives a 403 or 429 response.

🔍 Detection Indicators

The primary User-Agent string used is LydiaEntitySpider/1.0, although variations such as Lydia-Entity-Spider/2.1 have been seen in wild samples from HTTP Archive and CrowdSec honeypots. Additional identifying headers include X-Lydia-Entity: 1 and From: [email protected], which are logged by the bot on all requests. Behavioral fingerprints include a consistent Accept: text/html,application/xhtml+xml header and a TLS cipher suite favoring modern ECDHE and AES‑GCM ciphers, distinguishing it from generic scrapers.

📊 Data Usage

All collected data is processed through Lydia’s proprietary entity extraction pipeline, which uses deep learning models (based on BERT and RoBERTa architectures) to identify entities—such as people, organizations, locations, and products—and their relational links. The resulting structured knowledge graph is refreshed weekly and employed in Lydia’s AI training datasets (for both supervised and unsupervised entity recognition) as well as in their real‑time entity resolution API. According to a whitepaper published on lydia.ai/publications (2024), the spider’s output is also used to benchmark entity linking systems against community standards like the CoNLL‑2003 and TAC‑RED corpora.

⚙️ Rate Limiting Policy

The Lydia Entity Spider is rate‑limited because its systematic, multi‑threaded crawl pattern can inadvertently overload smaller websites or those with limited server capacity if left unchecked. A threshold‑based blocking policy—typically 20 requests per 60 seconds per IP—is recommended by the Lydia documentation to protect web servers while still allowing the legitimate crawler to complete its indexing within a reasonable timeframe.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.