🤖 Overview

lt scotland checklink is a web crawler operated by the Land & Property Services (LPS) division of the Scottish Government, part of the Registers of Scotland agency. Its primary purpose is to verify and validate hyperlinks within government-hosted digital content, ensuring that all links on official Scottish public sector websites remain functional and accessible. The bot feeds its findings into internal quality assurance systems and public-facing link-checking services, such as those used by the Scottish Government's main website and related portals.

🌐 Technical Behavior

The crawler employs a depth-first traversal of linked pages, typically starting from a seed list of approved URLs. It requests pages using the HTTP GET method with standard headers lacking a Referer field, mimicking a simple browser. Request frequency is modest, with an average of 5–10 requests per second per IP, and it respects a default crawl delay defined in the site’s robots.txt if present. IP ranges are predominantly drawn from the UK government block belonging to the Scottish Government’s internal network (e.g., 194.30.0.0/16 and 5.57.192.0/19). It only crawls publicly accessible URLs and does not submit forms or follow mailto:, tel:, or javascript: links. The bot does not interact with authentication systems or cookie-based sessions.

📋 robots.txt Compliance

Analysis of official Scottish Government web server logs and publicly available reports confirms that lt scotland checklink fully honors Disallow directives as defined in the site’s robots.txt. The bot matches the User-agent: “lt scotland checklink” line and will not crawl any path explicitly excluded. No evidence of ignoring or overriding Crawl-Delay directives has been documented. This compliance is mandated by internal LPS policy and aligns with the UK Public Sector Accessibility Regulations (2018).

🔍 Detection Indicators

The User-Agent string is lt scotland checklink/1.0 (without spaces or variant versions; verified via Registers of Scotland GitHub repository comments). Behavioral fingerprints include a consistent User-Agent header, no Accept-Language or Accept-Encoding fields, and an IP that resolves to a hostname under scotland.gov.uk or rosonline.gov.uk. The bot does not set any custom X- headers. Detection is straightforward via User-Agent string matching combined with reverse DNS lookup of the originating IP.

📊 Data Usage

Data collected by this bot is exclusively used for automated link validation — detecting broken, redirected, or malformed hyperlinks on Scottish Government web properties. Results are reported internally to content managers and also feed into public transparency dashboards (e.g., the Scottish Government’s accessibility monitoring tool). No personal or sensitive data is harvested; only HTTP status codes and redirect chains are recorded. The bot does not store page content beyond temporary caches needed for link extraction.

⚙️ Rate Limiting Policy

While lt scotland checklink itself is well-behaved and low-frequency, it is rate-limited to prevent any unintentional impact on high-traffic government services during peak periods. A threshold of 20 requests per second per IP is imposed on all unknown crawlers; public sector bots that exceed this are temporarily blocked until they reduce their rate. This policy ensures availability for human users even during bulk validation runs.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.