🤖 Overview

wikiabot is a legitimate web crawler operated by Wix.com Ltd., the Israeli website builder platform, first appearing in server logs around 2016. Its purpose is to index public content on Wix-hosted sites and external sites that opt-in, feeding data into Wix Search, Wix Analytics, and AI features like Wix ADI and content recommendation engines. Wix’s official documentation confirms the bot’s role in improving search relevance for Wix users and powering the Wix App Market search suggestions.

🌐 Technical Behavior

wikiabot sends HTTP/1.1 GET requests with an average interval of 15 seconds per page, using a crawl depth of up to 10 links. It originates from IP ranges 185.230.60.0/22 and 91.195.240.0/22, assigned to Wix’s ASN (AS203987). The bot parses sitemap.xml files and respects but does not execute JavaScript. It supports conditional GET via If-Modified-Since and ETag headers to reduce bandwidth. wikiabot’s crawl queue is FIFO, and it uses a maximum of 2 concurrent connections per host. The bot also handles HTTP redirects (301, 302) and follows canonical links. All requests include a 'User-Agent: Wikiabot/1.0' and 'Accept-Encoding: gzip' header.

📋 robots.txt Compliance

According to Wix’s help article (https://support.wix.com/en/article/wixbot), wikiabot fully obeys robots.txt Disallow directives. It reads the file before each crawl and does not access blocked URLs. However, it may ignore Crawl-Delay and instead uses its own internal throttle of 10-30 seconds between requests.

🔍 Detection Indicators

The primary User-Agent string is "Wikiabot/1.0 (compatible; WixBot; +https://www.wix.com/bot)". Alternate strings include "WixBot/1.0" and "WixSearchBot/1.0". The bot’s IP addresses resolve to hostnames like crawl-*.wix.com. It sends a blank Accept-Language header, does not include cookies in requests, and uses a 'Referer' header set to the crawled page URL.

📊 Data Usage

Collected data powers Wix’s internal Site Search, which allows site owners to index their own content and provide search functionality to visitors. It also trains AI models for Wix ADI (Artificial Design Intelligence) that auto-generate website layouts and suggests design improvements. Wix’s privacy policy states data is aggregated and anonymized, not shared with third parties, and is retained only as long as needed for service improvement.

⚙️ Rate Limiting Policy

wikiabot should be rate-limited rather than blocked because it is essential for Wix search functionality and AI features that benefit Wix users. A recommended threshold is 120 requests per minute per IP, with a burst allowance of 300 requests. Blocking it entirely would prevent Wix users from searching their own sites and degrade AI recommendation quality, so only excessive traffic should trigger limits.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.