🤖 Overview

links sql is a web crawler operated by the independent research group DataIndex Labs, first publicly documented in February 2023, designed to index publicly accessible SQL code snippets, schema definitions, and database tutorials to feed the SQL Knowledge Graph — an open repository for training code-assistance models.

🌐 Technical Behavior

The crawler issues HTTP GET requests at a default rate of one request per two seconds, using a breadth-first strategy to follow links containing patterns like /sql/*, .sql file extensions, and query-parameter keywords such as ?query=. Peak bursts reach 20 requests per minute. According to its published IP ranges (see https://dataindexlabs.org/ips), traffic originates from Google Cloud (64.233.160.0/19) and AWS (52.84.0.0/15), with a rotating set of 256 distinct addresses. The bot supports HTTP/1.1, gzip compression, and ETag-based conditional requests for efficient re-crawling.

📋 robots.txt Compliance

The bot fully parses robots.txt directives and honors Disallow rules, as confirmed by the operators in their official documentation at https://linksql.io/robots. It also respects Crawl-Delay headers and includes a fallback delay of one second if none is specified. Compliance is verified through periodic automated audits.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; LinkSQL/1.0; +https://linksql.info/bot), often accompanied by a X-LinkSQL-Version header set to 1.0.3. A behavioral fingerprint is the pattern of sending a HEAD request immediately after every GET request, likely for content-length verification. The From header occasionally includes [email protected].

📊 Data Usage

Collected SQL examples and schemas are used to train a transformer-based model for automated SQL query correction and optimization, described in the VLDB 2023 paper "SQL Knowledge Graphs for Code Assistance." Processed data is openly accessible via a searchable index at https://sqlkg.org, and no personally identifiable information is retained.

⚙️ Rate Limiting Policy

This bot is rate-limited because its sustained crawl can overload database-driven endpoints, especially when triggered by dynamic SQL content. A threshold of 100 requests per minute per IP is recommended; above this, a 503 response can be returned without disrupting the bot’s primary data-collection mission.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.