🤖 Overview

QualifiedBot is a legitimate web crawler operated by Qualified.com, a San Francisco‑based company that provides an AI‑powered talent intelligence platform. Launched in 2019, the bot systematically scans publicly accessible job boards, career pages, and professional profiles to feed data into Qualified’s matching and candidate‑ranking system. According to Qualified’s official documentation (https://www.qualified.com/robots), the crawler is employed exclusively to improve the accuracy of its machine‑learning models for talent sourcing, not for indexing general web content.

🌐 Technical Behavior

The crawler uses a distributed architecture with requests originating from IP addresses registered under ASN 396982 (Qualified, Inc.) and AWS EC2 ranges. It enforces a maximum crawl rate of 1 request per 2 seconds per host as confirmed by its `Crawl‑Delay: 2` directive in `robots.txt`. The bot only fetches HTML pages and ignores images, PDFs, and other binary resources by default. It identifies itself via the `User‑Agent` header and sends a `From` header containing `[email protected]`. QualifiedBot does not follow redirects beyond depth 3 and respects HTTP `Last‑Modified` and `ETag` headers to avoid re‑crawling unchanged content.

📋 robots.txt Compliance

Qualified.com explicitly states that QualifiedBot fully adheres to the Robots Exclusion Protocol, including `Disallow` and `Allow` directives. The bot checks `robots.txt` before each crawl session and will not access any path forbidden by the site owner. This compliance is verified in their public `robots.txt` file, which lists the crawler’s policy.

🔍 Detection Indicators

The primary User‑Agent string is `QualifiedBot/1.0 (compatible; +https://www.qualified.com/bot)`. Secondary variants may include `QualifiedBot/2.0` for profile analysis. Behavioral fingerprints include a fixed crawl interval of 2 seconds, absence of JavaScript execution, and a `From` header always set to `[email protected]`. No other known headers (e.g., `X‑Forwarded‑For`) are used.

📊 Data Usage

All collected data—job descriptions, salary ranges, skill requirements, and company names—is used solely to train and validate Qualified’s proprietary talent‑matching AI. The data is not stored for long periods; it is aggregated within 24 hours and anonymized before being fed into machine‑learning models. QualifiedBot does not store personal identifiable information (PII) of candidates without explicit consent.

⚙️ Rate Limiting Policy

Although QualifiedBot is well‑behaved, it is rate‑limited because its continuous scanning can consume server resources if many sites are crawled concurrently. A threshold‑based blocking rule (e.g., 100 requests per minute per IP) is recommended to prevent unintended load without harming the bot’s legitimate function.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.