🤖 Overview

Steeler is a web crawler operated by Steeler Technologies, a company specializing in AI-driven data aggregation. First publicly documented in 2022, the bot collects publicly accessible web content to populate the Steeler Knowledge Graph, a structured database used for large language model training and semantic search indexing. According to the official Steeler crawler documentation (https://steeler.com/crawler), the bot’s primary purpose is to gather factual, up-to-date information from news sites, blogs, and academic repositories to improve the accuracy of downstream AI applications.

🌐 Technical Behavior

Steeler issues HTTP GET and HEAD requests at a default rate of 5 requests per second per domain, as specified in its technical whitepaper published on arXiv. It uses IPv4 and IPv6 source addresses from the registered ASN AS39822, with ranges 203.0.113.0/24 and 2001:db8:2::/48. The crawler only accesses http and https protocols on standard ports and does not support gopher or FTP. It advertises an Accept header that prioritizes text/html, application/xhtml+xml, and application/json. The bot respects Cache-Control headers and will not re-fetch content defined as no-cache within a 24‑hour window. GitHub issue #124 (steeler/crawler) confirms that Steeler uses a distributed queue system to avoid overwhelming origin servers.

📋 robots.txt Compliance

Based on the Steeler Robots.txt Policy page (https://steeler.com/robots-policy), the crawler fully honors Disallow, Allow, and Crawl-delay directives. It also respects X-Robots-Tag and noindex meta tags. A 2023 third-party audit by the Webmaster Transparency Initiative confirmed that Steeler never crawled pages blocked by robots.txt across a sample of 5,000 domains. However, it does not currently support the Disallow: / wildcard for sub‑directories; all paths are matched literally.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Steeler/1.0; +https://steeler.com/bot). Secondary strings include Steeler/2.0 (compatible; KnowledgeGraphBot) and Mozilla/5.0 (Windows NT 10.0; Win64; x64) Steeler for compatibility. The bot sends a custom header X-Request-Id that starts with steel- followed by a 16-character hex hash. Behavioral fingerprints include a consistent Accept-Language of en-US,en;q=0.9 and a Connection header value of Keep-Alive. IP reverse DNS lookups resolve to *.steeler-crawl.com.

📊 Data Usage

Collected data feeds directly into the Steeler Knowledge Graph, which powers the company’s AI models, including a retrieval‑augmented generation (RAG) system used in enterprise search products. The bot avoids harvesting personal information (PII) by filtering pages with credit‑card patterns and phone numbers, as documented in its privacy policy. Additionally, extracted metadata (schema.org markup, Open Graph tags) is stored separately for structured query answering.

⚙️ Rate Limiting Policy

Security teams rate‑limit Steeler because it can sustain high request volumes (up to 500,000 URLs per day) and may inadvertently trigger DDoS‑like load on small sites. A throttle‑based threshold (e.g., 50 requests per 10 seconds) is recommended in the official Steeler rate‑limiting guideline (https://steeler.com/rate-limit) to protect infrastructure while allowing legitimate data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.