amfibibot
Bot User-Agent:amfibibot
🤖 Overview
amfibibot is a web crawler operated by Amfibia Inc., a data company based in San Francisco, first announced in August 2023 on their official blog at amfibia.com. Its primary purpose is to gather publicly accessible web content — including text, images, and structured data — to train Amfibia's large language models for tasks such as summarization, question answering, and dialogue generation.
🌐 Technical Behavior
The bot uses a distributed crawling system with IP addresses registered under Amfibia's own ASN (AS399182), primarily in the United States and European Union. According to their documentation, the default crawl rate is one request per two seconds, and it respects the Crawl-Delay directive in robots.txt. It uses HTTP/1.1 Keep-Alive connections with the User-Agent string "amfibibot/1.0", follows HTTP redirects, and handles gzip and deflate content encoding. The bot downloads CSS and JavaScript files to assess page structure but does not execute JavaScript. It always fetches robots.txt before crawling a new host and respects meta robots tags.
📋 robots.txt Compliance
Amfibia states on amfibia.com/crawler that amfibibot honors all Disallow directives. Webmasters can block it entirely by adding User-agent: amfibibot followed by Disallow: /. The crawler refreshes its cached robots.txt every 24 hours, and no violations have been documented. Amfibia provides a feedback form for handling any issues.
🔍 Detection Indicators
The primary User-Agent string is "amfibibot/1.0". The bot also sends a From header with the email address "[email protected]". IP ranges are published at amfibia.com/ip-ranges. Reverse DNS lookups often show hostnames ending in "amfibia.net". Behavioral indicators include consistent request intervals, lack of JavaScript execution, and the initial robots.txt fetch.
📊 Data Usage
Collected data trains Amfibia's language models via supervised fine-tuning and reinforcement learning from human feedback. PII is automatically redacted during preprocessing. The data improves model reasoning and factual accuracy. Amfibia does not resell raw data to third parties.
⚙️ Rate Limiting Policy
Similar Threats
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
