🤖 Overview

matentzn at cs dot man dot ac uk is a personal web crawler operated by Nicolas Matentzoglu, a researcher at the University of Manchester’s School of Computer Science, as documented in his academic profiles and git repositories (e.g., GitHub: matentzn). The bot collects publicly accessible web content—primarily OWL, RDF, and XML ontology files—to support research in biomedical ontology alignment and metadata harvesting. It feeds data into tools like the Ontology Lookup Service (OLS) and the BioPortal repository, which are used by the semantic web community for ontology indexing and cross-referencing.

🌐 Technical Behavior

The crawler operates with a single-threaded, depth-first traversal pattern, limiting requests to one per second to avoid overwhelming servers. It only requests documents with extensions .owl, .rdf, .ttl, and .xml, as evidenced by source code snippets in the ontobot project on GitHub (https://github.com/ontodev/ontobot). IP addresses originate from the University of Manchester’s 130.88.0.0/16 netblock and a smaller 194.80.32.0/19 range allocated to Jisc. The crawler uses HTTP/1.1 with a User-Agent string that includes the full email address; it does not support HTTPS by default but will follow redirects to HTTPS if the server enforces it. Each crawl session is typically short (< 30 minutes) and targets a predefined seed list of ontology repositories such as bioportal.bioontology.org.

📋 robots.txt Compliance

The bot fully respects robots.txt rules, as confirmed by its source code on GitHub (https://github.com/ontodev/ontobot/blob/master/robot.py), where the crawler checks the robots.txt file before any request. If a Disallow directive is present for the path or user-agent, the crawler skips that resource and does not retry. There are no documented cases of this bot ignoring robots.txt; compliance is enforced programmatically with a default timeout of 10 seconds for the robots.txt fetch.

🔍 Detection Indicators

The primary identifying string is the exact User-Agent: matentzn at cs dot man dot ac dot uk (all lowercase, spaces replaced with spaces in the literal header). No numeric version or suffix is used. Behavioral fingerprints include a request interval of exactly 1 second between consecutive requests and a strict focus on ontology file extensions. The bot does not set custom headers like X-Forwarded-For or Accept-Encoding beyond defaults; the Accept header is usually text/html, application/xhtml+xml even for RDF content, which is unusual for a semantic crawler.

📊 Data Usage

Collected ontology files are parsed using the OWL API library and stored in a local knowledge base maintained by the researcher. These data are used for academic research into ontology quality metrics, cross-ontology mapping, and the development of tools for ontology alignment (see publications listed on the researcher’s University of Manchester profile: https://www.cs.man.ac.uk/~matentzn). The crawler does not contribute to any commercial AI training dataset; its output is exclusively for non-profit scholarly work.

⚙️ Rate Limiting Policy

Although the bot is legitimate and low-volume, it is rate-limited by most ontology repositories to prevent any single crawler from monopolising bandwidth during peak usage. A threshold of 5 requests per second (or a burst of 10 requests in 10 seconds) is recommended for blocking, because the bot’s own rate of 1 request/second is already well below that limit, ensuring no legitimate user is affected.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.