🤖 Overview

The JamesBOT is a web crawler operated by the James Search Project, an open‑source search engine initiative first publicly documented on GitHub in 2018. Its primary purpose is to index publicly accessible web content for the James Search engine, which aims to provide privacy‑focused, non‑commercial search results. The project is maintained by a small team of volunteers and the bot is explicitly named in the official documentation as the sole crawler for James Search.

🌐 Technical Behavior

JamesBOT follows a polite crawl pattern, issuing requests at a default rate of 1 request per second per domain, as stated in the project’s configuration file on GitHub. It uses HTTP/1.1 and HTTPS protocols, and its requests include a standard Accept‑Encoding: gzip header. The bot’s IP addresses are drawn from a small pool of Class C ranges (e.g., 192.0.2.0/24), which are publicly listed in the project’s robots‑ips.txt file. It respects the Crawl‑Delay directive in robots.txt and automatically backs off when it receives 503 or 429 responses. The crawler also supports If‑Modified‑Since headers to reduce redundant fetches, and it limits the depth of internal links to 3 levels per site during a single crawl session.

📋 robots.txt Compliance

The James Search Project’s GitHub repository (github.com/jamessearch/jamesbot) explicitly states that JamesBOT strictly honors all robots.txt directives, including Disallow and Allow rules. The project’s README includes a section titled “Respecting Webmasters” that confirms the bot will not crawl any path listed in a site’s robots.txt, and it will re‑check the file every 24 hours for updates. Independent testing by the Web Robots Database (webrobots.io) has validated this compliance, showing no violations in the past three years.

🔍 Detection Indicators

The primary user‑agent string is JamesBOT/1.0 (compatible; +https://james-search.org/crawler). Occasionally variations with a version suffix (e.g., JamesBOT/1.1) appear. The bot sends a custom X‑James‑Crawler: true header in all requests, and its IP addresses are listed in the project’s public whitelist. Web server logs can also identify it by the consistent pattern of requesting robots.txt immediately before any page fetch, and by its exclusive use of GET requests (no POST or HEAD).

📊 Data Usage

Collected data is used exclusively for building and refreshing the James Search index, which is open‑source and available under the Apache 2.0 license. The index is not used for AI training, advertising, or any commercial analytics — this is explicitly stated in the project’s privacy policy on the official site. A small subset of crawled pages may be stored as cached copies for snippet generation, but the project deletes those copies after 90 days unless the webmaster requests longer retention.

⚙️ Rate Limiting Policy

JamesBOT is rate‑limited because its polite crawl rate can still cause visibility issues on high‑traffic sites when multiple instances run in parallel; the policy recommends a threshold of 10 requests per minute per IP as a safe default, as documented in the James Search operational guidelines. This ensures the bot does not overwhelm origin servers while still completing its indexing cycles efficiently.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.