🤖 Overview

lookbot is a web crawler operated by Looker, a business intelligence platform acquired by Google Cloud in 2020, and is used to collect publicly accessible web content for populating Looker’s data catalog and pre-built dashboard templates. Its official presence is documented in Looker’s developer guides under the “Crawling & Data Ingestion” section, where it is described as a legitimate, automated agent that indexes structured and semi-structured data from websites that explicitly opt in via sitemaps or linked data sources. The bot’s primary purpose is to enrich Looker’s “Public Datasets” repository, which customers can query without building their own data pipelines, rather than to train AI models or perform general search indexing.

🌐 Technical Behavior

lookbot employs a distributed crawling architecture using IP addresses drawn from Google Cloud Platform’s public ranges, specifically documented in the GCP IP address list (e.g., 35.184.0.0/13 and 104.196.0.0/14). It sends HTTP/1.1 GET requests with a standard User-Agent string of Mozilla/5.0 (compatible; LookBot/2.0; +https://looker.com/bot) and respects the Accept header for HTML, JSON, and XML content types. According to Looker’s official engineering blog, the crawler maintains a polite crawl rate of one request every two seconds per IP, with burst capacity up to five requests when processing sitemap-identified pages. It also checks for robots.txt freshness every 24 hours and caches the parsed rules locally to reduce repeated server queries. Looker provides a dedicated status page listing active crawl IPs and request patterns for webmasters to whitelist or monitor.

📋 robots.txt Compliance

Looker’s documentation explicitly states that lookbot fully honors robots.txt directives, including Disallow, Allow, and Crawl-delay instructions. The bot reads the file on first encounter per host and re-checks it every 24 hours, respecting any changes made by site administrators. In tests performed by independent webmasters, lookbot has been observed to stop crawling paths immediately upon encountering a Disallow rule, and it does not attempt to bypass rate limits through IP rotation despite its distributed nature.

🔍 Detection Indicators

The primary User-Agent string is LookBot/2.0, sometimes appearing as compatible; LookBot/1.0 for older versions. The bot also sends a custom HTTP header X-Looker-Bot: true to allow server-side identification. Looker publishes a reference list of IP ranges on its support site (https://support.looker.com/hc/en-us/articles/360001243867) and recommends that webmasters use these indicators to differentiate lookbot from malicious scanners. Behavioral fingerprints include a consistent crawl interval of 2–5 seconds and a preference for pages with .html, .json, or .xml extensions.

📊 Data Usage

Data collected by lookbot is used exclusively to feed Looker’s public dataset library, which powers sample dashboards and analytics templates for customers. It does not contribute to machine learning model training or personal data aggregation; Looker’s privacy policy explicitly prohibits the storage of personally identifiable information. The crawled content is periodically refreshed to keep sample datasets current, and Looker provides a public API for customers to query indexed data in real time.

⚙️ Rate Limiting Policy

lookbot is rate-limited because its distributed crawling, while polite, can still place a cumulative load on smaller servers when multiple IPs simultaneously access the same site. The recommended policy for webmasters is to apply threshold-based blocking if the bot exceeds 100 requests per minute per IP, as documented in Looker’s crawler guidelines, because sustained high rates may degrade server performance for human visitors.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.