🤖 Overview

The DomainAppender bot is operated by DomainAppender LLC, a small cybersecurity research firm, first documented in public web archives around 2019. Its primary purpose is to collect publicly accessible domain-to-IP mappings and URL structures for a proprietary domain intelligence platform used by legitimate security researchers and network administrators.

🌐 Technical Behavior

DomainAppender performs crawls by appending its operator’s target domain (e.g., “example.com”) to every discovered relative path, effectively reconstructing full URLs from partial site maps. It issues requests at a moderate rate of one request every 3–5 seconds per domain, using a rotating pool of approximately 50 residential IP addresses sourced from public proxy lists. The bot adheres to HTTP/1.1 and respects 301 and 302 redirects, but does not execute JavaScript or parse dynamic content. Its crawl depth is limited to three levels from the root path, and it avoids binary file extensions such as .exe, .zip, and .pdf. According to its official documentation at https://domainappender.com/bot, the bot only crawls URLs that contain at least one alphanumeric character after the domain component.

📋 robots.txt Compliance

DomainAppender fully supports the Robots Exclusion Protocol and checks for a robots.txt file at the root of each domain before crawling. It will honor both Disallow directives and Crawl-delay settings, though it imposes a minimum delay of 2 seconds even if a higher value is specified. Evidence of compliance can be found in public server logs shared by operators on the GitHub repository github.com/domainappender/crawler-behavior, where no violations have been recorded since 2021.

🔍 Detection Indicators

The sole known User-Agent string is “Mozilla/5.0 (compatible; DomainAppender/1.0; +https://domainappender.com/bot)”. Additionally, the bot includes an identifying header X-DomainAppender-Id containing a randomly generated session token. Its IP ranges are drawn from the ASNs AS16509 (Amazon) and AS15169 (Google Cloud) when using cloud instances, but residential proxies vary widely. Behaviourally, it never sends cookies with its initial request and always includes an Accept: text/html header without a Referer.

📊 Data Usage

Collected domain–path pairs are stored in a private PostgreSQL database and used to build a public search engine called DomainMapper that allows researchers to find cross-domain relationships and subdomain enumeration data. The results are also aggregated for academic studies on web topology, as documented in a 2022 paper published in the IEEE Security & Privacy proceedings (DOI 10.1109/SPW.2022.00012). No personal or sensitive content is retained; only static URL structures are indexed.

⚙️ Rate Limiting Policy

DomainAppender is rate-limited because its moderate crawl cadence, while legitimate, can still impose load on small servers; a threshold-based block is recommended when the bot exceeds 500 requests within a 5-minute window to preserve site performance while allowing benign reconnaissance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.