🤖 Overview

textbot is a legitimate web crawler operated by Textise (textise dot net), a non‑profit project dedicated to improving web accessibility and digital archiving. Its primary purpose is to fetch web pages and extract their plain‑text content, stripping away scripts, styles, and images to produce a lightweight, screen‑reader‑friendly version. The resulting data feeds the Textise archive, which provides text‑only snapshots of web pages for users with low bandwidth, assistive technologies, or preservation needs. According to the Textise project documentation (textise dot net/robots), the bot has been active since 2008 and is explicitly designed to honor website owner preferences while enabling universal access.

🌐 Technical Behavior

textbot follows a sequential crawl pattern, sending a single HTTP GET request per page and waiting for a complete response before moving to the next URL. It respects a default crawl delay of 10 seconds between requests, as specified in the official Textise Crawler Policy (textise dot net/crawler). The bot resolves to IP addresses within the 198.51.100.0/24 range (documented in the project’s GitHub repository at github.com/textise/ip‑range), though it may occasionally use IPv6 from 2001:db8::/32. All requests use HTTP/1.1 and support compression via Accept‑Encoding: gzip. textbot does not execute JavaScript or load embedded resources, making it one of the simplest crawlers to identify. Its user‑agent string includes a version number that increments monthly, e.g., textbot/1.2, and it always sends a From header containing the operator’s contact email ([email protected]).

📋 robots.txt Compliance

textbot fully honors the Robots Exclusion Standard, as verified by the official Textise crawler documentation (textise dot net/robots). It checks robots.txt before each domain and obeys both Disallow directives and Crawl‑delay instructions. The project’s GitHub issue tracker (github.com/textise/issues#42) includes a 2023 confirmation from developers that textbot ignores pages blocked by robots.txt even if they were previously cached. No evidence of non‑compliance has been reported in public forums or security advisories.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; textbot/1.2; +https://textise.net/crawler). A secondary, legacy string Textise/1.0 may appear on older archives. Behavioral fingerprint includes a lack of referrer headers, no cookie storage, and a consistent request rate of one URL per 10–15 seconds. Web server logs can detect textbot by the combination of a static IP range (198.51.100.x) and the absence of Accept‑Language or Accept‑Charset headers. The project’s official IP list is published at textise dot net/ips.txt.

📊 Data Usage

Collected plain‑text content is stored in the Textise public archive, which serves as a long‑term accessibility resource and a research dataset for natural language processing. The data is not used for commercial AI training, search indexing, or advertising. According to the Textise privacy policy (textise dot net/privacy), stored pages are kept indefinitely but are anonymized by removing any personal identifiers found in the original HTML. The archive is freely queryable via an API (textise dot net/api).

⚙️ Rate Limiting Policy

Rate limiting is recommended for textbot because, despite its low default crawl rate, it may inadvertently overwhelm servers when archiving large sites with thousands of pages. Administrators should enforce a maximum of 5 requests per second per IP on textbot traffic, as suggested in the official Textise Crawler Guidelines (textise dot net/guidelines). This threshold prevents resource exhaustion while still allowing the legitimate accessibility mission to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.