🤖 Overview

axfeedsbot is a legitimate web crawler operated by Axfeeds (axfeeds.com), a South Korean news aggregation service, designed to collect news articles and web content for real-time indexing and aggregation into the Axfeeds platform. According to the official Axfeeds documentation, the bot's primary purpose is to gather news headlines, summaries, and metadata to populate a feed distributed to users via the Axfeeds app and website. It is not used for AI training or any malicious activity, and it fully respects webmaster controls.

🌐 Technical Behavior

The crawler employs a polite crawl pattern with a default delay of 10 seconds between requests, as specified in the Axfeeds crawler guidelines and confirmed via third-party logs. It issues standard HTTP GET requests with gzip encoding and conditional GET using If-Modified-Since headers to reduce server load. The bot follows links only within the same domain and does not execute JavaScript or parse dynamic content. IP ranges are primarily associated with South Korean ISPs, notably AS4766 (Korea Telecom) and AS9318 (SK Broadband). The crawl frequency can be adjusted via the Crawl-Delay directive in robots.txt, and the bot sets its User-Agent to include a contact URL for website owners.

📋 robots.txt Compliance

Axfeedsbot fully honors robots.txt directives, including both Disallow rules and Crawl-Delay settings, as documented on the Axfeeds website. There are no known instances of this bot ignoring robots.txt; it checks the file before each crawl session and caches it for a short period. Webmasters can reliably use standard directives to control its access.

🔍 Detection Indicators

The primary User-Agent string is axfeedsbot/1.0, often extended with a contact URL such as axfeedsbot/1.0 (http://www.axfeeds.com/bot). It also sends a custom HTTP header X-Axfeeds-Bot: true. Behavioral fingerprints include consistent request intervals (no burst traffic), lack of referrer headers, and absence of cookies or session identifiers. The bot never makes concurrent requests and does not simulate human browsing patterns.

📊 Data Usage

Collected data is used exclusively for news aggregation, including headlines, short summaries, and the first few sentences of articles, displayed with attribution to the original source. Axfeeds does not sell collected data or use it for training large language models; the aggregated feed is provided to end users free of charge with optional premium features. The service indexes content from thousands of news outlets globally.

⚙️ Rate Limiting Policy

Although axfeedsbot is legitimate and non-malicious, webmasters may rate-limit it to prevent excessive server load, especially on smaller sites. The policy rationale for threshold-based blocking (e.g., >10 requests per minute) is to protect server resources while allowing the bot to still access content at a reasonable pace, balancing index freshness with server capacity.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.