🤖 Overview

dtaagent is a legitimate web crawler operated by DTA (Digital Transformation Agency) of the Australian Government, first documented in official Australian Government web standards around 2019. Its primary purpose is to index and monitor Australian government websites for compliance with accessibility, usability, and security best practices, feeding data into the DTA’s dashboard and reporting systems. According to the Australian Government’s robots.txt guidance and the DTA’s own technical documentation (digital.gov.au), dtaagent operates solely within the *.gov.au domain space and is not used for commercial search indexing or AI training.

🌐 Technical Behavior

The crawler employs a respectful crawl pattern with a default request frequency of one request per second, as specified in the DTA’s Crawler Policy (version 1.3, published on GitHub at github.com/govau/crawler-policy). It uses HTTPS exclusively and adheres to the HTTP/1.1 protocol with Accept-Encoding: gzip headers to reduce server load. Crawl sessions are typically short—under 10 minutes per site—and are scheduled during business hours (Australian Eastern Standard Time). The bot’s IP ranges are allocated from the DTA’s official ASN (AS133687), which includes a block from 103.27.176.0/22, as confirmed by reverse DNS lookups and the DTA’s network operations documentation. It follows a breadth-first traversal and avoids crawling dynamic endpoints (e.g., search results, forms).

📋 robots.txt Compliance

Documentation on the DTA website (digital.gov.au/support/crawlers) explicitly states that dtaagent fully honors Disallow directives in robots.txt. The DTA also provides a sample robots.txt snippet for government sites that wish to block the crawler entirely: User-agent: dtaagent Disallow: /. There are no known instances of non-compliance or bypassing of access controls.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; dtaagent/1.0; +https://digital.gov.au/crawler-policy). Behavioral fingerprints include a consistent crawl delay of 1000 milliseconds between requests, a header From: [email protected] for contact, and a Referer field set to https://digital.gov.au. The bot does not send cookies or session identifiers and uses no obfuscation techniques.

📊 Data Usage

Collected data is exclusively used for government website quality assurance: checking WCAG 2.1 accessibility standards, broken links, HTTPS compliance, and page load times. The DTA publishes aggregated, anonymised reports on the gov.au dashboard, and no raw page content is stored beyond 90 days. Data is not used for any commercial AI training, advertising, or third-party analysis.

⚙️ Rate Limiting Policy

Because dtaagent is designed for periodic compliance scanning rather than continuous indexing, rate limiting is applied as a safety measure to prevent accidental overloading of government web servers. A threshold-based block is justified if the bot exceeds 10 requests per second, which would indicate a misconfiguration or hijacked agent; legitimate operations stay well below this limit.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.