🤖 Overview

infolink is a web crawler operated by InfoSpace LLC (now part of Blucora, Inc.), originally developed to support the InfoSpace metasearch engine and related vertical search products. Its primary purpose is to collect publicly available web content for indexing and aggregating search results across multiple categories, including news, images, and web pages. The bot feeds data into InfoSpace’s internal search infrastructure, which powers aggregators such as Dogpile, Metacrawler, and WebCrawler, as documented on InfoSpace’s legacy product pages and robot.txt guidance at infospace.com.

🌐 Technical Behavior

The infolink crawler follows standard HTTP/1.1 protocols and typically sends requests with a User‑Agent string of Infolink or InfolinkBot. It uses IPv4 addresses from the InfoSpace‑owned netblocks, including 209.204.0.0/16 and 208.87.0.0/16, though these ranges may have been reassigned after Blucora’s restructuring. The bot performs breadth‑first crawling, respecting a default crawl delay of 10 seconds between requests unless overridden by a Crawl‑Delay directive. It does not support HTTP/2 or HTTPS prioritization and will retry failed requests up to three times with exponential backoff, as observed in legacy logs from webmasters. The crawl frequency varies per site, but typical instances show up to 5,000 requests per day for large domains, with a strong preference for text/html content and XML sitemaps.

📋 robots.txt Compliance

According to InfoSpace’s publicly archived documentation, the infolink bot reliably honors Disallow directives in robots.txt and respects Allow exceptions when present. It also reads the Crawl‑Delay directive and defaults to 10 seconds if none is specified. While no known violations have been reported, the bot may ignore meta robots tags on pages that are not blocked by robots.txt, as it relies primarily on the file‑level rules.

🔍 Detection Indicators

The primary User‑Agent string is Infolink (e.g., Mozilla/5.0 (compatible; InfolinkBot/1.0; +http://www.infospace.com/infolinkbot)). Additional fingerprints include a distinct From header set to [email protected] and a low request rate per IP. Reverse DNS lookups on the source IP often resolve to *.infospace.com or *.blucora.com.

📊 Data Usage

The collected data is used exclusively for building and updating InfoSpace’s search index, which serves aggregated search results across its metasearch properties. No evidence suggests that the crawled content is used for AI training or commercial resale; the index is refreshed periodically to maintain relevance for end‑users performing web searches through Dogpile, Metacrawler, and similar portals.

⚙️ Rate Limiting Policy

Despite its legitimate purpose, infolink is rate‑limited by many admins due to its historical pattern of aggressive crawling on large sites, sometimes exceeding 10,000 requests per day before throttling. Implementing threshold‑based blocking (e.g., 50 requests per minute per IP) helps prevent unnecessary server load while still allowing the bot to index essential content.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.