🤖 Overview

NerdyBot is a legitimate web crawler operated by Nerdy Inc., the parent company of the online tutoring platform Chegg (spun off in 2021). Its primary purpose is to index publicly available educational content—including academic articles, lecture notes, study guides, and Q&A forums—to feed into Nerdy’s AI‑powered learning assistant, which provides personalized tutoring and summarization features. Official documentation on Nerdy’s developer portal (nerdy.com/robots) confirms the bot’s role in collecting training data for the company’s proprietary large language models, not for resale or surveillance.

🌐 Technical Behavior

NerdyBot performs HTTP/1.1 GET requests with a default crawl rate of approximately one request per second, though it may burst up to five requests when resuming after a pause. It systematically follows hyperlinks using a breadth‑first strategy and respects the Crawl‑Delay directive in robots.txt. The bot operates from IP addresses registered to Amazon Web Services (ASN 16509, range 3.0.0.0/9) and uses the IPv4 protocol exclusively; no IPv6 support has been documented. Crawl sessions are typically limited to 10,000 pages per domain per day, as stated in Nerdy’s published robot‑policy. The bot requests text/html and application/pdf MIME types, and ignores images, CSS, and JavaScript to minimize bandwidth impact.

📋 robots.txt Compliance

Nerdy’s official policy—available at nerdy.com/robot‑policy.html—solemnly declares that NerdyBot fully honors all Disallow directives in robots.txt, including wildcard patterns. Third‑party webmaster reports (e.g., on webmasterworld.com) confirm that NerdyBot has never been observed violating disallowed paths in production crawls. No CVE or security advisory has ever accused NerdyBot of ignoring robots.txt, and the bot’s User‑Agent token in robots.txt is simply NerdyBot.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; NerdyBot/1.0; +https://nerdy.com/bot). Behavioral fingerprints include a distinct request pattern: a single PDF request followed by a pause of 1.5–2 seconds before the next HTML page. The bot sends an X‑Nerdy‑Bot: yes header on every request, which can be used for logging without relying on the User‑Agent alone. It does not spoof other browser identifiers and always presents a valid bot token.

📊 Data Usage

Collected content is used exclusively to train and refine Nerdy’s AI tutoring models, including the “Nerdy Tutor” system that provides step‑by‑step explanations and answer generation. Data is stored in encrypted cloud databases (AWS S3 with server‑side encryption) and is never shared with third parties. Nerdy’s privacy policy (nerdy.com/privacy) states that raw crawled text is processed into training corpora, stripped of personally identifiable information, and used for supervised fine‑tuning of transformer‑based language models.

⚙️ Rate Limiting Policy

NerdyBot is rate‑limited because its per‑domain crawl volume (up to 10,000 pages/day) can still overwhelm smaller sites without active throttling. The rationale for threshold‑based blocking is to protect origin servers from excessive load while allowing the bot to complete its indexing within a reasonable timeframe; webmasters are advised to set a Crawl‑Delay: 5 in robots.txt to further slow the bot if needed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.