🤖 Overview

NetLyzer is a web crawler operated by the Social Media Lab at Toronto Metropolitan University (formerly Ryerson University), established in 2012 by Dr. Anatoliy Gruzd. Its purpose is to collect publicly available social media and web forum data for academic research, feeding into the Netlytic.org text and social networks analyzer platform used by researchers globally.

🌐 Technical Behavior

The crawler uses a Python-based framework sending sequential HTTP GET requests at a rate of one request every 2–5 seconds, adjustable via Crawl-Delay. It originates from IP ranges within 141.117.0.0/16 (Toronto Metropolitan University's block) and supports HTTPS only for static HTML and API endpoints, never executing JavaScript. The bot follows redirects up to 5 hops and backs off exponentially upon receiving HTTP 429 responses, as documented in its GitHub repository at https://github.com/Netlytic/NetLyzer.

📋 robots.txt Compliance

NetLyzer fully respects robots.txt directives, including explicit Disallow rules and Crawl-Delay values, as verified by the official Netlytic documentation. Site owners can block the bot entirely by adding User-agent: NetLyzer/1.0 and Disallow: / to their robots.txt file.

🔍 Detection Indicators

The primary identifier is the User-Agent string NetLyzer/1.0 (case-sensitive). Behavioral fingerprints include a fixed Accept-Language: en-US,en;q=0.5 header, no Referer header, and no cookies. IP addresses reverse-resolve to *.torontomu.ca or *.ryerson.ca, and requests target API endpoints like /search?q= sequentially.

📊 Data Usage

All collected data is used solely for academic research and non‑commercial purposes, processed within Netlytic.org for social network analysis, content categorization, and sentiment analysis. Anonymized datasets may be shared with other researchers but are never sold or used for advertising; the project adheres to ethical guidelines published by the Social Media Lab.

⚙️ Rate Limiting Policy

NetLyzer is rate‑limited because its research mission requires thorough data collection that could overload small servers. Threshold‑based blocking (e.g., 10 requests per minute) is recommended, as the crawler obediently respects HTTP 429 responses and backs off when encountering such limits.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.