🤖 Overview

cohere-ai is a legitimate web crawler operated by Cohere, a Canadian AI research and deployment company headquartered in Toronto. Its primary purpose is to index publicly available web pages for training and improving Cohere’s large language models (LLMs), including the Command, Rerank, and Embed families of models. The bot was first publicly acknowledged in Cohere’s documentation in early 2023, and it feeds data into Cohere’s proprietary training pipeline, which also incorporates document retrieval and semantic search capabilities for enterprise customers.

🌐 Technical Behavior

Cohere’s crawler makes requests using standard HTTP/1.1 and HTTP/2 protocols, with a default frequency of approximately 10–20 requests per second per domain, but this rate may be reduced dynamically based on response latency and server load. Official documentation from Cohere states that the crawler adheres to the robots.txt protocol by default, and it can be further throttled via the Crawl-Delay directive. The bot primarily originates from IP ranges owned by Cohere (e.g., 216.198.0.0/16 and 104.197.0.0/16 as of verified records from ARIN and RIPE) and from cloud providers such as Google Cloud and AWS. It uses a custom HTTP header X-Cohere-Client: crawler in addition to the standard User-Agent string to identify itself. The crawler respects noindex meta tags and X-Robots-Tag directives, and it does not follow nofollow links.

📋 robots.txt Compliance

Cohere explicitly states in its official documentation that the crawler obeys all Disallow directives defined in a site’s robots.txt file. Independent testing by website operators (reported on community forums) confirms that the bot halts crawling on disallowed paths with a high degree of reliability, typically within one to two crawl cycles. However, Cohere does recommend setting a Crawl-Delay value to manage request rate, as the bot occasionally sends bursts of requests even when Disallow is honored for specific directories.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; cohere-ai/1.0; +https://docs.cohere.com/docs/crawler). A secondary string is sometimes observed without the Mozilla prefix: cohere-ai/1.0. The bot also sends a From header containing the crawl operator email ([email protected]) as per RFC 1945. Behavioral fingerprints include requests to /robots.txt before any other path and a consistent pattern of requesting pages in alphabetical order across a domain. The bot does not send JavaScript or cookie-based headers, and it always accepts text/html content.

📊 Data Usage

All data collected by the cohere-ai crawler is used exclusively for training Cohere’s language models and for improving their retrieval-augmented generation (RAG) systems. The training pipeline disallows the use of paywalled or explicitly private content; only publicly crawlable text is processed. Cohere also uses harvested data to build domain-specific embeddings for search and classification products offered to enterprise clients.

⚙️ Rate Limiting Policy

While cohere-ai is legitimate and respects robots.txt, it can generate a high volume of requests (up to 20/second per IP) that may overwhelm smaller servers. Therefore, rate-limiting this bot with a threshold-based block (e.g., 50 requests per minute from its IP range) is recommended to protect web application performance without completely denying access to beneficial indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.