🤖 Overview

Goldfire Server is a legitimate automated agent operated by IHS Markit (now part of S&P Global) as part of the Goldfire Innovation Intelligence platform, first documented in 2014. It systematically crawls public websites to gather technical literature, patent filings, and market data, feeding into the Goldfire semantic search and analytics engine used by R&D teams.

🌐 Technical Behavior

The crawler uses a multi-threaded architecture, sending requests at a rate of up to 10–15 per second per IP, though it respects a default crawl delay of 2 seconds when specified in robots.txt. It originates from a dynamic IP range belonging to IHS Markit’s AWS infrastructure (e.g., 54.174.0.0/16 and 52.90.0.0/15) and employs HTTP/1.1 keep-alive connections. It preferentially requests HTML and PDF content, ignoring images and scripts unless explicitly needed. The agent uses a polite throttle by default, reducing its request volume during peak server hours based on response headers (Retry-After and 429 status codes).

📋 robots.txt Compliance

Goldfire Server is documented to fully honor Disallow directives and Crawl-delay settings in robots.txt, according to IHS Markit’s official administrator guide (available at goldfire.ihsmarkit.com/docs/crawler-policy). It also respects noindex meta tags and X-Robots-Tag HTTP headers, as verified in internal compliance reports published by S&P Global.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Goldfire Server/2.1; +https://goldfire.ihsmarkit.com/bot), though older versions may appear as Goldfire/1.0. It sends a custom X-Goldfire-Bot: true header and a persistent cookie GF-Session for session tracking. Behavioral fingerprinting reveals a consistent 2‑second minimum delay between requests to the same domain when no crawl-delay is set.

📊 Data Usage

Collected content is processed through Goldfire’s natural-language pipeline to build a semantic knowledge graph used for patent prior‑art search, competitive intelligence, and trend analysis. The data is not used for generative AI training but is stored in the Goldfire platform’s indexed corpus accessible to subscribing enterprise clients.

⚙️ Rate Limiting Policy

Goldfire Server is rate‑limited because its sustained multi‑threaded crawl can still overload smaller sites; administrators are advised to apply a threshold of 100 requests per 60 seconds per IP and respond with HTTP 429 to enforce polite behavior, which the bot will recognize and back off from.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.