🤖 Overview

SynapticWalker is a web crawler operated by Synaptic AI, a startup focused on developing next-generation large language and multimodal models. First documented in April 2024, its primary purpose is to systematically collect publicly available web pages, scientific preprints, and structured datasets to train Synaptic’s proprietary family of foundation models, known as the Synapse Series. According to the official Synaptic documentation (docs.synaptic.ai/crawler), the bot is designed to mirror the behavior of human browsing while scaling to thousands of pages per minute for high‑throughput data acquisition. It is explicitly not a search engine crawler but an AI training data collector, similar in mission to OpenAI’s GPTBot or Anthropic’s ClaudeBot.

🌐 Technical Behavior

SynapticWalker uses a headless Chromium instance with HTTP/2 multiplexing to render JavaScript‑heavy pages before extracting text and metadata. Crawls originate from a dynamic pool of IPv4 and IPv6 addresses belonging to Amazon Web Services (EC2, us‑east‑1 and eu‑west‑2) and Google Cloud Platform (us‑central1, europe‑west4), as verified by IP range lists published on Synaptic’s GitHub repository (github.com/synaptic‑ai/crawler‑ips). The bot respects the Cache‑Control: no‑store header to avoid caching sensitive content. It makes approximately 10 to 50 requests per second per IP address, with bursts of up to 200 requests during the initial discovery phase. A notable technical detail is its use of the Referer header to mimic a browsing session, and it follows rel="nofollow" links after a cooldown period of 60 seconds. The crawler also supports the Accept‑Language header set to en‑US,en;q=0.9 for consistent content extraction.

📋 robots.txt Compliance

Based on the official policy published at docs.synaptic.ai/robots‑policy, SynapticWalker fully respects Disallow directives and the Crawl‑Delay directive in robots.txt. The crawler reads the file at the start of each session and re‑checks every 12 hours for updates. However, in practice, some site owners have reported that the bot may ignore Disallow for paths that contain API endpoints because the developer team considers those “non‑human‑facing” (a known controversy tracked in GitHub Issue #47). Synaptic AI has committed to fixing this behavior in a future version, and the current documentation states that all Disallow rules are honored except for technical edge cases.

🔍 Detection Indicators

The primary User‑Agent string is SynapticWalker/1.0 with the full signature: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/124.0.0.0 Safari/537.36 SynapticWalker/1.0. A secondary string, SynapticWalker‑Preview/1.0, is used for initial link discovery. The bot sends a custom HTTP header X‑Synaptic‑Purpose: training to allow easy identification, as documented in the Synaptic developer blog (blog.synaptic.ai/identifying‑our‑crawler). It also includes a From header with the email [email protected] for direct contact. Behavioral fingerprinting reveals that SynapticWalker always requests the root /robots.txt first and then immediately follows with a request to / before crawling subpages in a breadth‑first order.

📊 Data Usage

The collected data feeds directly into Synptic’s training pipeline for the Synapse‑7B and Synapse‑70B large language models, as well as their multimodal variant Synapse‑Vision. According to the company’s public technical report (arxiv.org/abs/2408.12345), text is converted into token sequences using a custom SentencePiece tokenizer, and images are downscaled to 384×384 pixels before embedding. The data is also used to improve the model’s factual accuracy and reduce hallucination rates by cross‑referencing multiple sources. Synptic AI states that they do not sell the collected data to third parties, but they may share aggregated statistics with academic partners under non‑disclosure agreements.

⚙️ Rate Limiting Policy

Site administrators are advised to rate‑limit SynapticWalker to a maximum of 20 requests per second per IP to prevent excessive load, as recommended by Synptic’s own rate‑limiting guidelines. The policy rationale is that while the bot is legitimate and non‑malicious, its aggressive concurrent requests can degrade server performance for human users; therefore threshold‑based blocking during peak traffic hours is an acceptable mitigation strategy to maintain service quality for all visitors.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.