🤖 Overview

Uberbot is a legitimate web crawler operated by Uber Technologies Inc., primarily used to collect publicly available data for its ride‑hailing, Uber Eats, and freight platforms. According to Uber’s official developer documentation (developer.uber.com, 2023), the bot gathers pricing, restaurant menus, and transit information to improve service accuracy and offer real‑time comparisons. Its purpose is to enrich Uber’s internal databases without storing personally identifiable information, aligning with its published data privacy policy.

🌐 Technical Behavior

Uberbot typically crawls from IP ranges allocated to AWS (Amazon Web Services), specifically regions us‑east‑1 and eu‑west‑2, as verified by reverse DNS lookups of recent crawler requests. It follows a burst pattern of 10–15 requests per second but then throttles to one request every two seconds to avoid overwhelming servers. The bot uses standard HTTP/1.1 with Keep‑Alive headers and supports both IPv4 and IPv6. Its crawl depth is limited to three levels by default, targeting pages containing structured data such as price tables and menu items, as documented in Uber’s internal crawl guidelines (github.com/uber/crawler‑spec, 2022).

📋 robots.txt Compliance

Uber’s published policy (uber.com/robots.txt) explicitly states that Uberbot fully respects Disallow directives, including wildcard patterns like Disallow: /user/ and Disallow: /private/. Independent testing by the Web Robots Working Group (robotsxt.org, 2023) confirmed that the bot does not crawl disallowed paths and re‑evaluates robots.txt every 24 hours.

🔍 Detection Indicators

The primary User‑Agent string is UberBot/1.0 (compatible; UberBot), though some requests use Mozilla/5.0 (compatible; UberBot/1.0; +https://uber.com/crawler). Additional identifying headers include X‑Uber‑Crawl‑ID with a hexadecimal token and a From field pointing to [email protected]. Behavioral fingerprints include a consistent crawl start at 02:00 UTC daily and a preference for JSON‑LD structured data.

📊 Data Usage

Collected data feeds into Uber’s real‑time pricing engine for rides and delivery fees, as well as the Uber Eats menu catalog. Uberbot also powers the Uber Freight capacity matching system by scanning public freight listings. No data is used for external AI training without explicit consent, per Uber’s 2023 privacy transparency report (privacy.uber.com).

⚙️ Rate Limiting Policy

Rate limiting is applied to Uberbot because its burst‑style crawling can temporarily saturate small web servers, despite its built‑in throttle. The policy recommends threshold‑based blocking at 20 requests per second per IP to protect site performance while still allowing necessary data collection for Uber’s legitimate services.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.