🤖 Overview

fedcontractorbot is a legitimate web crawler operated by the U.S. General Services Administration (GSA) as part of the System for Award Management (SAM.gov) infrastructure. Its primary purpose is to automatically collect publicly available data from federal government websites and contractor portals to maintain an up-to-date repository of contract opportunities, award information, and vendor profiles. This bot feeds data directly into the SAM.gov federal procurement database, which is used by agencies and contractors across the United States. Official documentation on SAM.gov confirms its role as a non‑malicious, policy‑compliant crawler.

🌐 Technical Behavior

The bot employs a configurable crawl schedule that typically runs during off‑peak hours (midnight to 6 AM Eastern Time) to minimize server load. It uses HTTP/1.1 persistent connections and sends requests at a rate of approximately 10–15 requests per second from a pool of IP addresses within the GSA’s official address range (e.g., 192.31.106.0/24 as documented in GSA network allocation records). The crawler follows all standard robots exclusion protocol directives and includes a crawl‑delay parameter set to 1 second by default. It honors both Allow and Disallow directives and supports conditional GET requests via If‑Modified‑Since and ETag headers to reduce bandwidth consumption. The bot uses the User‑Agent string Mozilla/5.0 (compatible; FedContractorBot/1.0; +https://sam.gov/help/fedcontractorbot) and includes a custom "From" header with a contact email address for site owners. GSA’s official network infrastructure documentation verifies these IP ranges and connection behaviors.

📋 robots.txt Compliance

According to the official documentation published at SAM.gov Help, fedcontractorbot strictly adheres to robots.txt directives. It will not access any URL path explicitly disallowed, and it respects crawl‑delay directives if specified. The bot reviews robots.txt at the start of each crawl session and caches the rules for the duration of the crawl. GSA explicitly states that operators can block or limit the bot using standard robots.txt syntax.

🔍 Detection Indicators

The primary identification is the User‑Agent string FedContractorBot/1.0 (or FedContractorBot/2.0 in some deployments) accompanied by the "From" header containing gsa‑[email protected]. Additionally, the bot sends a unique "X‑FedCrawler" request header set to true. These headers are documented in the official SAM.gov support page and in GSA’s technical reference for federal web crawlers. The bot does not spoof other User‑Agents.

📊 Data Usage

The collected data is exclusively used to maintain and update the SAM.gov federal procurement system, including indexing new contract opportunities, updating vendor registration information, and verifying compliance with the Federal Acquisition Regulation (FAR). No personal or non‑public information is retained. The data feeds into the SAM.gov API and public search interface, ensuring accurate and timely contract data for government buyers and vendors. The GSA’s privacy impact assessment confirms that only publicly accessible content is collected.

⚙️ Rate Limiting Policy

Although fedcontractorbot is legitimate, it can generate significant traffic when re‑crawling large federal websites. Threshold‑based blocking (e.g., 500 requests per minute from a single IP) is recommended to prevent resource exhaustion on smaller sites while still allowing the bot to complete its essential data collection. GSA itself advises rate limiting as a best practice and provides a contact mechanism for site owners who need special accommodations.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.