zlib-rs stack overflow during decompression with malicious input
A denial of service vulnerability was found in zlib-rs, triggered by specially constructed input. This input causes a stack overflow, resulting in the process using zlib-rs to crash.
Impact
Due to the way LLVM handles the zlib-rs codebase, tail calls were not guaranteed. This caused certain input patterns to result in a large number of stack frames being required, quickly resulting in a stack overflow. These are unlikely to occur in practice, but a dedicated attacker can construct malicious input files.
After stack overflows were found by @inahga with a fuzzer, we dove into the assembly, and found some cases where the stack grew
```asm
.LBB109_326:
mov rdi, rbx
call zlib_rs::inflate::State::type_do
jmp .LBB109_311
.LBB109_311:
lea rsp, [rbp - 40]
pop rbx
pop r12
pop r13
pop r14
pop r15
pop rbp
.cfi_def_cfa rsp, 8
ret
```
LLVM wants to centralize the cleanup before the return (many other blocks jump to LBB109_311), thereby invalidating a tail call to type_do. We were not able to get rid of this call without introducing one elsewhere: we just don't currently have the power to tell LLVM what we want it to do.
So, we switch back to loop+match waiting for changes to rust to make a more efficient implementation possible. Performance-wise, the damage is relatively minimal: we're just slower in cases where we already were slower than C. We are faster in cases where the relevant code is barely touched (in these cases the logic quickly moves into a hot inner loop and just spends most of its time there).
Patches
Version 0.4.0 patches the problem and is no longer vulnerable.
Workarounds
Users of zlib-rs should upgrade to the latest version. Users could alternatively run zlib-rs in a separate process to prevent a stack overflow crashing the entire program. In some situations a signal handler can be used to catch a stack overflow happening.
