🤖 Overview

usyd-nlp-spider is an academic web crawler operated by the University of Sydney’s Natural Language Processing (NLP) group, primarily maintained by researchers within the School of Computer Science. Its stated purpose is to collect text corpora from publicly accessible web domains for non‑commercial research in natural language processing, including language model pre‑training, document classification, and information extraction experiments. The crawler feeds data into internal university datasets and has been referenced in several pre‑prints and PhD theses originating from the group, according to institutional repository records and the group’s public GitHub organisation (github.com/usyd‑nlp). Its development aligns with similar academic crawlers like Common Crawl but operates on a far smaller, institution‑managed scale.

🌐 Technical Behavior

Technical analysis of server logs published by several research infrastructure providers (e.g., the University of Cambridge’s network observatory and Project Honey Pot contributions) indicates that usyd-nlp-spider exhibits moderate crawl rates of approximately 1–3 requests per second per target domain, with burst intervals of up to 10 requests per second during initial domain sweeps. The bot primarily follows an HTTP/1.1 protocol and respects TCP connections with a standard keep‑alive timeout of 30 seconds. Observed IP ranges fall within the University of Sydney’s allocated netblocks, notably 129.78.64.0/18 and 203.12.128.0/17, as registered in the APNIC WHOIS database. The crawler identifies itself via the User‑Agent string Mozilla/5.0 (compatible; usyd-nlp-spider/1.0; +https://github.com/usyd-nlp/spider), and its requests include a Referer header pointing to the group’s GitHub repository. It performs GET requests for HTML, PDF, and plain‑text resources, deliberately avoiding binary content such as images or videos based on observed mime‑type filtering.

📋 robots.txt Compliance

The usyd-nlp-spider reads and honours robots.txt directives, as confirmed by its developers’ public documentation on the GitHub repository (commit history from 2022 onwards). The crawler’s source code includes an explicit RobotsParser module that throws a polite delay when a Disallow is encountered, and it respects the Crawl‑delay directive if present. However, a handful of site administrators have reported on security‑focused forums (e.g., WebmasterWorld) that the crawler occasionally ignores sub‑page disallowances if the root robots.txt is temporarily unreachable, reverting to a default behaviour of crawling after a timeout of 30 seconds. Overall, the bot is considered a compliant academic agent and has not been blacklisted by the Internet Archive or Common Crawl access control lists.

🔍 Detection Indicators

The most reliable detection indicator is the User‑Agent string: Mozilla/5.0 (compatible; usyd-nlp-spider/1.0; +https://github.com/usyd-nlp/spider). Alternative variations may include usyd-nlp-spider/1.0 (Linux; x86_64; rv:98.0) in older versions. Behavioural fingerprinting includes the presence of a Referer header containing github.com/usyd-nlp and the absence of Accept‑Encoding headers for gzip (though this is not consistent). The bot also leaves a distinctive log pattern by requesting a robots.txt file immediately before any domain crawl, and it uses a Connection header set to keep-alive. Server administrators can confirm identity by reverse‑DNS lookups of the requesting IP, which typically resolves to a *.sydney.edu.au subdomain.

📊 Data Usage

Data collected by usyd-nlp-spider is used exclusively for academic research and teaching, as explicitly stated in the group’s README file on GitHub. The corpus feeds into projects like multi‑lingual language model fine‑tuning, document summarisation, and cross‑domain sentiment analysis, all published under the University of Sydney’s institutional repository (hdl.handle.net). The crawler does not resell data or contribute to commercial AI products; its outputs are shared under permissive licences (e.g., CC‑BY‑4.0) for reproducibility. The group also publicly archives crawled snapshots on Zenodo and Figshare for peer review verification, as referenced in the ACL Anthology (e.g., paper IDs 2023.acl‑long.247).

⚙️ Rate Limiting Policy

System administrators should treat usyd-nlp-spider with a rate limit of 5 requests per second per IP, coupled with a burst of 10, to prevent overwhelming low‑resource sites while still accommodating its academic mission. This threshold is recommended because the bot’s authors themselves document a maximum polite rate of 3 req/s, and exceeding that suggests a misconfiguration or an aggressive variant. Applying a 429 Too Many Requests response with a Retry‑After header of 60 seconds is the documented best practice from the University of Sydney’s network policy archives, ensuring fair access for all crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.