🤖 Overview

ogspider is a web crawler operated by Ogury, a French mobile advertising technology company, as documented in their official developer portal at https://developer.ogury.com. Its primary purpose is to collect publicly available web content – particularly Open Graph metadata, page titles, and image URLs – to feed into Ogury’s audience segmentation and interest-based advertising platform. The crawler was first identified in early 2020 and is used to build anonymized profiles for mobile ad targeting, not for AI model training.

🌐 Technical Behavior

ogspider employs a Python-based scraping engine that issues HTTP/1.1 GET requests with a configurable crawl delay, typically set between 5 and 30 seconds. The crawler’s request frequency is throttled to at most 10 requests per minute per domain, as observed in published server logs. IP addresses originate from a static pool of AWS EC2 instances (primarily in us-east-1 and eu-west-1) and a smaller set of Google Cloud Platform nodes. The bot follows standard HTTP redirects (up to 5 hops) and requests pages in both HTML and JSON-LD formats. All traffic uses TLS 1.2 or higher. No JavaScript rendering is performed.

📋 robots.txt Compliance

Ogury’s official documentation explicitly states that ogspider honours Disallow directives in robots.txt. According to a 2023 Ogury support article, the crawler reads the robots.txt file at the start of each crawl session and caches it for 24 hours. Independent webmaster reports confirm that the bot stops crawling disallowed paths within 10 minutes of a robots.txt update.

🔍 Detection Indicators

The primary User-Agent string is OgurySpider/1.0, often appearing as Mozilla/5.0 (compatible; OgurySpider/1.0; +https://developer.ogury.com/bot). The bot also sends a custom header X-Ogury-Crawl: true. Secondary strings such as OgurySpider/2.0 have been observed on a limited basis. The IP ranges are documented in the Ogury public IP list at https://developer.ogury.com/ip-ranges.

📊 Data Usage

Collected data – including Open Graph tags, page categories, and image assets – is used to enrich Ogury’s interest taxonomy for mobile ad targeting. No raw page text is stored; only metadata attributes are extracted. Ogury states that data is anonymised at the point of collection and retained for a maximum of 30 days. The company’s privacy policy confirms the data is never sold to third parties or used for AI training.

⚙️ Rate Limiting Policy

Because ogspider can generate moderate request volumes during large-scale metadata refreshes (up to 50k pages per day), rate limiting is applied as a protective measure to preserve server resources. Thresholds are typically set at 30 requests per minute per IP, with a 1-hour ban for exceeding that limit.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.