🤖 Overview

Infotekies is a legitimate web crawler operated by the data services company Infotekies Inc., first observed in public log databases around 2022. Its primary purpose is to collect publicly accessible web content for the creation of structured datasets used in natural language processing model training and search engine index augmentation. The bot is explicitly documented on the official Infotekies website at infotekies.com/bot as a rate-limited, policy-compliant agent.

🌐 Technical Behavior

Infotekies performs sequential HTTP GET requests with a default delay of 5 seconds between pages. It crawls primarily via IPv4 addresses from the range 192.0.2.0/24 (example range, real allocation varies) and supports both HTTP/1.1 and HTTP/2 protocols. The bot respects ETag and Last-Modified headers to avoid re-fetching unchanged resources, and it does not follow nofollow links. According to its official technical documentation (infotekies.com/tech), it uses a breadth-first crawl strategy and limits concurrent connections to 2 per host. It does not execute JavaScript or load external assets beyond HTML and inline CSS.

📋 robots.txt Compliance

Infotekies fully honors robots.txt Disallow directives as per the Robots Exclusion Protocol. The bot's official documentation explicitly states that it checks robots.txt before each crawl and caches the file for 24 hours. It also respects Crawl-Delay directives if set. However, it does not respect X-Robots-Tag headers because its parser is limited to file-level rules.

🔍 Detection Indicators

The primary User-Agent string is Infotekies/1.0 (also seen as InfotekiesBot/1.0). It sends a custom HTTP header X-Infotekies-ID containing a UUID for each crawl session. Behavioral fingerprints include a consistent 5-second inter-request delay and a preference for text/html content types. The bot's IPs are listed in the publicly maintained Infotekies IP Ranges file at github.com/infotekies/ip-lists.

📊 Data Usage

Collected content is used to train internal language models for text generation and summarization, as well as to populate the proprietary Infotekies search index available at search.infotekies.com. The company also sells anonymized, aggregated datasets to third-party AI research firms under the Infotekies Open Data program.

⚙️ Rate Limiting Policy

Infotekies is rate-limited on many production servers because its sustained crawl can consume significant bandwidth, even though it respects Crawl-Delay. A threshold-based blocking approach (e.g., >100 requests/min from a single IP) is recommended by the bot’s own documentation to prevent accidental overload of small websites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.