🤖 Overview

kinjabot is a legitimate web crawler operated by G/O Media Inc. (formerly Gawker Media) that powers the Kinja publishing platform. Its primary purpose is to crawl publicly accessible URLs shared by users on G/O Media properties (e.g., Gizmodo, Kotaku, Lifehacker, Jalopnik) to generate rich link previews, including titles, descriptions, and images. The bot was first observed in the early 2010s and is documented in the company’s official User-Agent list and robots.txt guidelines on the Kinja support pages.

🌐 Technical Behavior

kinjabot performs HTTP GET requests to a single URL per session, typically triggered by a user submitting a link on a Kinja-powered site. It respects 304 Not Modified headers and does not recursively follow links or crawl entire domains. Request frequency is moderate, often making 1–5 requests per minute per domain, and it uses IPv4 addresses drawn from G/O Media’s own ASN (AS36647). The bot identifies itself via the User-Agent string “kinjabot/1.0” and may include a Referer header from the originating G/O Media site. No official IP range list is published, but logs show sources from the 64.207.128.0/18 block used by the company.

📋 robots.txt Compliance

kinjabot fully honors Disallow directives in robots.txt, as documented on the Kinja support site and confirmed by webmaster forums (e.g., WebmasterWorld, 2015). If a crawler finds a User-agent: kinjabot line with a Disallow rule, it immediately stops crawling that path. The bot also respects Crawl-delay directives if present, though it does not otherwise enforce a fixed delay beyond its built-in rate.

🔍 Detection Indicators

The primary identifier is the User-Agent string: kinjabot/1.0 (no version variation reported). Behavioral fingerprinting reveals that the bot makes exactly one request per link and never downloads more than the first 10 KB of content, as it only needs the og:meta tags and HTML title. Additional headers include a From: [email protected] header (deprecated but still seen in older crawls) and a persistent Accept-Language: en-US value.

📊 Data Usage

Collected data—specifically the page title, description, and primary image URL—is stored temporarily (≤24 hours) in Kinja’s preview cache and used exclusively for generating link embeds on G/O Media articles and comments. No data is used for AI training, behavioral profiling, or indexing. The content is not retained beyond cache expiry unless the linked page is explicitly saved by a user.

⚙️ Rate Limiting Policy

kinjabot is rate-limited because its burst requests (when a viral article triggers many links) can overwhelm smaller servers. A threshold of 10 requests per minute is recommended for blocking, as the bot legitimately retries only failed fetches once and will abide by 429 Too Many Requests responses as documented in the G/O Media developer FAQ.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.