🤖 Overview

NotebookLM is an AI‑powered research and note‑taking assistant developed by Google, first previewed at Google I/O 2023 under the codename “Project Tailwind”. Unlike a traditional web crawler, NotebookLM is a conversational agent that operates on user‑uploaded documents and, optionally, on user‑provided URLs to extract and summarize information using Google’s Gemini large language model. Its primary purpose is to help individuals and professionals quickly synthesize knowledge from their own sources without indexing the open web at large.

🌐 Technical Behavior

NotebookLM does not perform autonomous web crawling; instead, it makes on‑demand HTTP requests only when a user explicitly provides a URL within a notebook. These requests are limited to fetching the content of a single page (or a limited set of pages if the user supplies multiple links) and are not followed recursively. Google’s internal infrastructure, including the Googlebot IP ranges (e.g., 66.249.x.x and 216.239.x.x), is used for these fetches, and the requests are accompanied by a custom User‑Agent string that identifies the call as originating from a Google AI service. The frequency of such requests is tied directly to user interaction, so a single user session may trigger a small burst of fetches but sustained high volume is not possible from a single account.

📋 robots.txt Compliance

Google has publicly stated that all its automated services, including experimental AI agents like NotebookLM, respect the robots.txt directives of the target domain. Official documentation from Google Search Central confirms that any HTTP request made by Google‑operated systems will honor the Disallow rules before fetching content. Therefore, NotebookLM will not access any URL for which the site owner has blocked Googlebot or other Google‑designated user‑agents.

🔍 Detection Indicators

As of early 2025, Google has not published an official, static User‑Agent string exclusively for NotebookLM. In practice, the user‑agent observed in logs may appear as Mozilla/5.0 (a generic Google service agent) combined with a variation of Google-NotebookLM/1.0 or Google-AI-Research. The distinguishing fingerprint is the presence of the From: google.com header and the use of Google‑owned IP blocks. Unlike Googlebot, NotebookLM requests typically lack the standard “Googlebot” identification in the user‑agent field.

📊 Data Usage

The data collected by NotebookLM (i.e., the content of user‑provided URLs or uploaded documents) is used exclusively for generating answers, summaries, and insights within the user’s private notebook session. Google’s privacy policy states that this content is not used to train foundation models unless the user explicitly opts in. No public indexing or AI training on crawled web data is performed by this agent.

⚙️ Rate Limiting Policy

Because NotebookLM’s web fetches are triggered by individual user actions, it is rate‑limited per user account to prevent abuse — typically a few dozen URL fetches per hour. Blocking this agent outright is unnecessary, but threshold‑based rate limiting is recommended on high‑traffic sites to distinguish harmless on‑demand requests from aggressive scraping patterns.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.