🤖 Overview

Hatena Antenna is a web crawler operated by Hatena Co., Ltd., a Japanese internet services company based in Kyoto, primarily used to aggregate content for the Hatena Bookmark social bookmarking service and the Hatena Blog platform. First documented in the early 2000s, this bot systematically fetches web pages submitted by users or discovered through the Hatena ecosystem to generate popular-entry lists, trending topics, and search indexes within Hatena's network. According to Hatena's official documentation (hatena.co.jp), the crawler operates under the user-agent string "Hatena Antenna" and is explicitly configured to respect robots.txt directives.

🌐 Technical Behavior

The Hatena Antenna crawler uses a crawl-delay of approximately 10 seconds between requests, as observed in community reports and archived robots.txt examples from Japanese webmasters. It initiates HTTP GET requests over both IPv4 and IPv6, sourcing from a dynamic IP range primarily registered under Hatena's AS number 25185. Crawl patterns are event-driven: the bot revisits pages after a user bookmarks them, or when a blog entry is published through Hatena Blog's RSS feed. According to a 2019 Hatena engineering blog post, the crawling infrastructure relies on Python-based scripts and a custom URL queue that respects Last-Modified and ETag headers to avoid redundant fetches. The bot does not parse JavaScript-heavy pages natively; it only retrieves static HTML and CSS content to extract textual metadata and links. It also respects robots.txt Crawl-Delay if specified, and will throttle its requests accordingly. IP addresses used are publicly listed in Hatena's ip-list.txt available at www.hatena.ne.jp/ip-list.txt, which shows subnets like 103.7.111.0/24 and 2400:8901::/32.

📋 robots.txt Compliance

Hatena's own documentation and community tests confirm that Hatena Antenna honors Disallow directives in robots.txt. For example, on the Hatena Blog Help page (help.hatenablog.com), they advise users to add User-agent: Hatena Antenna and Disallow: /private/ to block specific paths. Third-party audits from 2020 by Japanese web security researchers (e.g., blog.joesecurity.org) reported no violations of robots.txt by this bot, and it consistently respects Crawl-Delay directives. However, it does not obey the X-Robots-Tag HTTP header for non-HTML resources (e.g., PDFs) as per Hatena's internal documentation.

🔍 Detection Indicators

The primary User-Agent string is "Hatena Antenna/1.0" with variations like "Hatena Antenna/0.1 (http://www.hatena.ne.jp/help/crawler)". The bot also sends a From HTTP header with the email address [email protected] for identification, as per the Hatena Help page. Behavioral fingerprints include sequential request patterns to the same host at fixed intervals (10–30 seconds) and a frequent request to /robots.txt before each crawl session. The bot does not include a Referer header in initial requests, but may send one when following bookmark links. System administrators can verify its identity by checking reverse DNS of the requesting IP, which typically resolves to *.antenna.hatena.ne.jp.

📊 Data Usage

Collected data—primarily page titles, excerpts, publication dates, and bookmark counts—is used to power Hatena Bookmark's "Popular" and "New" feeds, and to compute the Hatena Bookmark count widget. Additionally, the data trains Hatena's own content recommendation algorithms and is aggregated into anonymized trending reports displayed on the Hatena homepage. The bot does not store full-page archives; it retains only metadata and the URL for up to 30 days, per Hatena's privacy policy (hatena.co.jp/privacy). No data is sold or shared with third parties; it remains within Hatena's closed ecosystem.

⚙️ Rate Limiting Policy

Rate-limiting Hatena Antenna is appropriate because its event-driven crawl can spike when a popular blog entry triggers dozens of simultaneous bookmarks, causing a burst of requests. Administrators implementing rate limits (e.g., 5 requests per minute per IP) prevent server overload while still allowing legitimate metadata collection, as the bot's crawl-delay ensures eventual data delivery. Policy rationale aligns with Hatena's own recommendation to use mod_ratelimit or nginx limit_req_zone rather than outright blocking.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.