🤖 Overview

ScreenerBot is an automated web crawler operated by Screener.in, a leading Indian financial analytics platform that provides stock screening, fundamental data, and market research tools. Officially documented in the company’s robots.txt and via their support page, the bot’s purpose is to collect publicly available financial data—such as balance sheets, quarterly results, and corporate announcements—from stock exchanges, company websites, and news portals. This data feeds Screener.in’s proprietary analytics engine, which powers features like peer comparison, valuation metrics, and custom screeners used by retail and institutional investors. The bot was publicly introduced around 2018 and is continuously maintained to respect website policies while ensuring timely data freshness for users.

🌐 Technical Behavior

ScreenerBot employs a systematic crawl pattern that prioritizes financial and regulatory domains, typically requesting pages at intervals of 10–30 seconds to avoid server overload. According to Screener.in’s official documentation, the crawler uses HTTP/1.1 with keep-alive connections and respects ETag and Last-Modified headers to minimize redundant fetches. Its IP ranges are sourced from Screener.in’s cloud infrastructure (primarily AWS and DigitalOcean), with addresses in the 15.206.x.x, 18.233.x.x, and 159.89.x.x blocks. The bot performs GET requests only, ignoring POST endpoints, and caches responses for up to 24 hours. It also supports Accept-Encoding: gzip to reduce bandwidth usage. The crawl depth is limited to three levels from the entry page, and it does not follow redirects to non-HTTP protocols.

📋 robots.txt Compliance

ScreenerBot is documented as fully compliant with the Robots Exclusion Protocol. Screener.in states in their official FAQ that the bot reads robots.txt before every crawl and respects Disallow directives, including wildcard patterns like Disallow: /*?page= and directory-level blocks. In practice, site administrators have reported that ScreenerBot stops crawling blocked paths within seconds of a robots.txt update, as verified by server logs and community posts on Screener.in’s forum. No known violations have been reported in public CVE entries or security advisories.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; ScreenerBot/1.0; +https://www.screener.in), though variations like ScreenerBot/2.0 have been observed. The bot also sets a custom header X-Screener-Client: bot for easy identification. Behavioral fingerprints include a fixed 3-second delay between successive requests and a preference for HTML pages over JavaScript-heavy SPAs. Web server logs show requests originating exclusively from Indian IP ranges during business hours (IST).

📊 Data Usage

Collected data is aggregated and transformed into historical financial metrics, which are then stored in Screener.in’s database and made available through their web interface and API. The data is used exclusively for analytics, stock screening, and comparative research—not for AI training or model development. Screener.in’s privacy policy explicitly states that published data is not resold or used for advertising. The platform also offers a paid subscription tier that includes advanced filters derived from the same crawl results.

⚙️ Rate Limiting Policy

ScreenerBot is rate-limited because its aggressive crawl frequency, while legitimate, can still consume significant bandwidth on small financial websites. Threshold-based blocking is justified to protect server stability and ensure equitable access for human visitors, as recommended by the Internet Society’s guidelines for fair-use crawlers. Site operators are encouraged to use robots.txt with a Crawl-delay directive to control the bot’s pace.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.