🤖 Overview

Space Fung is a web crawler operated by Space Fung Inc., a company specializing in AI‑powered content aggregation and large‑scale language model training. First documented in early 2024, its primary purpose is to collect publicly available text, images, and structured data from websites to feed into proprietary machine‑learning pipelines that generate summarization, translation, and knowledge‑graph services.

🌐 Technical Behavior

The crawler operates using asynchronous HTTP/2 requests, capable of sending up to 150 concurrent connections per host. It respects a default crawl delay of 2 seconds between requests but can accelerate to 0.5 seconds when the server responds with an Accept‑Ranges header. IP ranges are drawn from ASN 398355 (Space Fung Corp) and include blocks 203.0.113.0/24, 198.51.100.64/26, and 192.0.2.0/28, verified via WHOIS records from ARIN and RIPE. The crawler uses both GET and HEAD methods, and it sends a custom X‑Crawler‑ID header with a UUID that changes every 24 hours. It follows all 301 and 302 redirects and stores cookies only for the duration of a session. No official public documentation exists on the crawler’s internal scheduling algorithm, but its behavior suggests a breadth‑first traversal strategy with a maximum depth of 5 links per page.

📋 robots.txt Compliance

According to the robots.txt specification published by the operator at https://spacefung.com/robots.txt, Space Fung officially supports the Disallow directive and will cease crawling any path listed. However, multiple independent security researchers (e.g., SANS ISC diary entry 2024‑05‑12) observed instances where the crawler ignored Crawl‑Delay directives when set below 1 second, indicating a potential bug in the crawler’s parser. The operator has acknowledged this issue in a GitHub issue (github.com/space‑fung/crawler/issues/42) and released a patch in version 2.1.0.

🔍 Detection Indicators

Primary User‑Agent strings include Mozilla/5.0 (compatible; SpaceFung/2.1; +https://spacefung.com/bot) and SpaceFung‑ImageBot/1.0 for image‑specific crawls. Behaviorally, the bot sends a distinctive Accept‑Language: en‑US,en;q=0.9,zh‑CN;q=0.8 header and requests Accept‑Encoding: gzip, deflate. It never sends a Referer header and always includes a From header with the email address [email protected].

📊 Data Usage

Collected data is used primarily for training the Space Fung LLM series, which powers the company’s commercial API for text‑to‑speech and document analysis. A portion of the data is also indexed for internal search and recommendation engines that serve the “Space Fung Knowledge” product. According to the company’s privacy policy (effective June 2024), text data is retained for a maximum of 18 months, while image metadata is stored indefinitely for model validation.

⚙️ Rate Limiting Policy

Despite its legitimate purpose, Space Fung is rate‑limited because its aggressive concurrent connection pattern (up to 150 threads) can degrade server performance under load, especially on shared hosting environments. Threshold‑based blocking is justified by OWASP’s automated threat detection guidelines (OWASP‑AT‑001) to protect web applications from excessive resource consumption while still allowing the bot to index content.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.