superlumin-downloader
superlumin downloader is a legitimate web crawler operated by Superluminal AI, a company focused on building large-scale training datasets for machine learning models. First documented in early 2024, its primary purpose is to systematically download publicly accessible web content, including text, images, and structured data, which is then used to train foundation models and improve natural language understanding. The bot feeds data into Superluminal's proprietary training pipeline, as described in their official documentation at https://superluminal.ai/crawler.
Superluminal Downloader employs a distributed crawling architecture that can issue requests from a pool of IP addresses within the ASN ASXXXXX (Superluminal Inc.), with ranges including 203.0.113.0/24 and 198.51.100.0/24. The bot respects a crawl delay of 10 seconds between requests to the same host, as specified in its own configuration. It uses HTTP/1.1 and HTTP/2 protocols, and sends a From header containing a contact email address ([email protected]) to facilitate communication with webmasters. Requests are made with a default timeout of 30 seconds and follow redirects up to five hops. The crawler does not execute JavaScript and only fetches static HTML and linked resources (CSS, images, PDFs) as needed for content extraction.
According to Superluminal's official guidelines, the bot fully honors robots.txt directives, including Disallow and Crawl-delay rules. It checks the robots.txt file for each domain at the start of a crawl session and caches the rules for up to 24 hours. Evidence from multiple webmaster forums confirms that the bot stops crawling paths explicitly prohibited in robots.txt.
The primary User-Agent string is Mozilla/5.0 (compatible; Superluminal-Downloader/1.0; +https://superluminal.ai/crawler). Secondary patterns include a User-Agent of Superluminal/2.0 (research) in later versions. The bot also sends an X-Robots-Tag header with value noarchive when accessing content that should not be cached. Behavioral fingerprints include consistent request intervals and a preference for non-robots.txt blocked paths.
Collected data is used exclusively for training Superluminal's large language models and retrieval-augmented generation systems. The company states that it does not sell data to third parties and that all content is processed in compliance with copyright laws, including the use of a fair-use filter that excludes paywalled or explicitly copyrighted material. The training datasets are not publicly released.
Although Superluminal Downloader is legitimate, its high request volume can impact server resources, especially on small sites. Therefore, webmasters are advised to rate-limit the bot using standard throttling mechanisms (e.g., 5 requests per second per IP) to ensure equitable resource allocation while still allowing the crawler to access content.
Free Traffic Analysis
Discover which unwanted bots are being blocked on your site, how often they hit, and where they come from — real data from your own traffic, not guesswork.
🔍 Scan My Site FreePowered by JA4 fingerprinting, honeypot traps & behavioral analysis
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.