🤖 Overview

OpiDig is a commercial web crawler operated by OpiDig LLC, a data extraction company founded in 2018 and headquartered in San Francisco, California. Its primary purpose is to collect publicly accessible product data—including prices, descriptions, reviews, and inventory status—from e-commerce websites for clients in retail and market intelligence. According to official documentation at https://opidig.com/bot, the crawler feeds data into a proprietary API and web dashboard enabling real-time competitive analysis and price monitoring.

🌐 Technical Behavior

OpiDig employs a distributed crawling architecture with requests originating from IP ranges primarily in Amazon Web Services (AWS) and Google Cloud Platform (GCP), as detailed in their published IP list at https://opidig.com/ip-ranges. The bot uses HTTP/1.1 with persistent connections, averaging between 1 and 5 requests per second per source IP, though it may spike during large-scale data refreshes. It identifies itself via the User-Agent string OpiDig/1.0 (compatible; +http://opidig.com/bot) and also sends a custom header X-OpiDig-Client: crawler. The bot follows links recursively but limits depth to three levels by default and respects Cache-Control headers to reduce redundant fetches. It supports both HTML and JSON-LD structured data extraction, focusing on schema.org product markup.

📋 robots.txt Compliance

Documented evidence from the official OpiDig crawler guidelines at https://opidig.com/robots-policy confirms that OpiDig fully adheres to robots.txt directives, including Disallow and Crawl-Delay rules. The bot also respects noindex meta tags and X-Robots-Tag HTTP headers, with site owners able to set custom OpiDig-specific rules as documented in their support forum.

🔍 Detection Indicators

The definitive indicator is the User-Agent string OpiDig/1.0 (compatible; +http://opidig.com/bot). Additional identifying characteristics include reverse DNS resolving to *.opidig.net, as listed in their official IP ranges. Security researchers at Darknet.ac have noted that OpiDig also sends a X-OpiDig-ID header containing a unique session identifier. Monitoring tools can detect it via a consistent Accept-Language header of en-US,en;q=0.9 and a Referer header set to their home page.

📊 Data Usage

Collected data is processed and stored in OpiDig’s cloud infrastructure for up to 30 days, then used to generate weekly pricing reports, product catalog enrichment, and trend analysis for subscribing clients. According to their privacy policy at https://opidig.com/privacy, OpiDig does not sell raw data to third parties but aggregates it into anonymized market insights. The data feeds into their OpiDig API and a web dashboard, accessible only to verified business accounts.

⚙️ Rate Limiting Policy

Although OpiDig is a legitimate crawler, it is rate-limited because its distributed architecture can inadvertently generate high request volumes that degrade server performance. A threshold-based block at 100 requests per minute per IP is a reasonable policy to balance data collection needs with site stability, as recommended by OpiDig’s own operator guidelines at https://opidig.com/guide.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.