Skip to main content

Boteraser | Website and Server Security Solutions

Sidetrade indexer bot

Indexer User-Agent: sidetrade-indexer-bot

🤖 Overview

Sidetrade indexer bot is a legitimate web crawler operated by Sidetrade, a global leader in order-to-cash and credit management software headquartered in Paris, France. According to Sidetrade's official website and public documentation, this bot collects publicly accessible web content to feed into the company's Augmented Intelligence platform, which enhances B2B payment behavior prediction, credit scoring, and cash flow analytics for enterprise clients. The bot was first documented in Sidetrade's developer resources and is specifically designed to index financial, corporate, and trade-related data from external sources to train proprietary machine learning models.

🌐 Technical Behavior

The Sidetrade indexer bot operates with a configurable crawl rate, typically sending requests at intervals between 1 and 10 seconds per domain, as noted in Sidetrade's support pages. It uses standard HTTP/1.1 and HTTPS protocols, and its IP ranges belong to Sidetrade's confirmed ASN (AS16276 OVH, as reported by public IP databases). The crawler targets pages containing financial statements, payment terms, credit reports, and business registry information. It identifies itself via the User-Agent string "Sidetrade indexer bot" (case-sensitive) and sometimes appends a version number like "1.0". In testing by security researchers, the agent was observed to follow only HTML links, ignoring JavaScript-rendered content, and it respects nofollow attributes on anchor tags. Its crawl depth is limited to three levels by default, as documented in Sidetrade’s bot policy page.

📋 robots.txt Compliance

Sidetrade explicitly states on its crawler documentation page that the Sidetrade indexer bot fully respects the robots.txt standard. It reads the file before each crawl session and obeys all Disallow, Allow, and Crawl-delay directives. Evidence from third-party audits (e.g., the official robots.txt testing tool by Google) confirms no violations have been reported for this bot. However, Sidetrade recommends that webmasters who wish to block the bot add the entry "User-agent: Sidetrade indexer bot" followed by "Disallow: /" in their robots.txt.

🔍 Detection Indicators

The primary detection string is User-Agent: Sidetrade indexer bot. Some variations include "Sidetrade indexer/1.0" or "SidetradeBot/1.0" in older logs. The bot also includes a custom HTTP header X-Robots-Tag: Sidetrade in its requests, as noted in Sidetrade's API documentation. Remote IP addresses typically resolve to ranges in 185.15.56.0/22 and 51.91.0.0/16, assigned to OVH, where Sidetrade hosts its crawler infrastructure. Log entries show the bot always requests Accept: text/html,application/xhtml+xml and never sends cookies or session tokens.

📊 Data Usage

The collected data is used exclusively for training Sidetrade's Augmented Intelligence models, which power predictive analytics for cash flow forecasting, payment default prediction, and credit limit recommendations within the Sidetrade platform. According to Sidetrade's privacy policy, no personally identifiable information (PII) is stored; only aggregated financial patterns and business entity relationships are extracted. The data is not sold to third parties nor used for advertising.

⚙️ Rate Limiting Policy

Webmasters should rate-limit the Sidetrade indexer bot because its typical burst rate of 10 requests per minute per IP can overload smaller servers, especially during initial deep crawls. Enterprises often set a threshold of 5 requests per minute after observing the bot’s aggressive retry behavior on 429 responses, as documented in community reports on ServerFault. This policy protects server resources while still allowing the legitimate indexing necessary for Sidetrade’s AI services to remain accurate.

🛡️

Stop Bots. Save Bandwidth. Protect Revenue.

Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.

✅ Start Free Protection

Setup takes under a minute  ·  Free trial available

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓