Deno: Miller-Rabin Primality Test Allows Zero Rounds
Summary
node:crypto.checkPrime(candidate[, options][, callback]) and crypto.checkPrimeSync(candidate[, options]) ran no Miller-Rabin rounds at all when the caller left options.checks at its default of 0. In that mode, the only test applied to the candidate was trial division by the primes up to 17,863. Any composite whose smallest prime factor exceeds that bound — for example the product of two primes just above it, such as 17,881 × 17,891 — was reported as true ("probably prime").
The same divergence affected the lower-level op_node_check_prime / op_node_check_prime_bytes paths that the polyfill calls into.
Node.js itself does not have this problem: it forwards checks = 0 to OpenSSL's BN_check_prime, which substitutes a sensible default number of rounds based on the candidate's bit length (per FIPS 186-4 Appendix C.3 Table C.1). Deno's Rust implementation had no equivalent fallback, so count = 0 meant "skip the loop entirely."
Affected APIs
crypto.checkPrime(candidate) (callback form, default options)crypto.checkPrime(candidate, { checks: 0 }, callback)crypto.checkPrimeSync(candidate) (default options)crypto.checkPrimeSync(candidate, { checks: 0 })
Callers who explicitly passed checks >= 1 were less affected, the loop ran the number of rounds they asked for, but were still receiving fewer rounds than Node would have applied for the same bit length. With the patched version they get at least the FIPS minimum.
Not affected
- Deno's prime *generation* (
crypto.generatePrime, crypto.generatePrimeSync, and the DH parameter generation path). Those routes go through Prime::generate_with_options in ext/node_crypto/primes.rs, which hardcodes 20 Miller-Rabin rounds and never reads a user-controlled checks value, so the bug never reached them. - Any other Deno-internal use of primality testing —
is_probably_prime is not called from elsewhere in the runtime with count = 0. - Web Crypto (
crypto.subtle.*), which uses entirely separate code paths and does not expose a primality test.
Impact
The realistic exposure is application-level: a Deno program that calls crypto.checkPrime (or its sync variant) with default options to validate an externally-supplied bignum, for example checking a peer-provided Diffie-Hellman prime, validating a prime read from configuration, or sanity-checking an RSA factor, will accept crafted composites as prime. The composite is trivial to construct: any product of two primes greater than 17,863 works.
Downstream consequences depend on what the program does with the "verified" prime. If the prime is fed into a key exchange, signature verification, or factorization-style check, the security guarantees of that protocol collapse to whatever the attacker engineered into the composite.
The CVSS impact is bounded by the requirement that the victim application both (a) calls checkPrime with default options and (b) acts on the result for security-relevant input it does not control.
Reproduction
```ts
import { checkPrimeSync } from "node:crypto";
// 17881 and 17891 are both prime and both above the trial-division
// ceiling used by Deno's implementation.
const composite = 17881n * 17891n;
// Affected versions print true; the patched version prints false.
console.log(checkPrimeSync(composite));
```
The same result is reproducible from Rust against the internal helper:
```rust
use num_bigint::BigInt;
let composite = BigInt::from(17881u32) * BigInt::from(17891u32);
assert!(!is_probably_prime(&composite, 0)); // fails on affected versions
```
Fix
PR [#34391](https://github.com/denoland/deno/pull/34391) introduces a
helper min_miller_rabin_rounds_for_bits(bits) that returns the FIPS
186-4 Appendix C.3 round counts, matching the defaults OpenSSL uses
inside BN_check_prime. is_probably_prime then clamps the loop bound
to count.max(min_miller_rabin_rounds_for_bits(n.bits())). The
probabilistic loop now always executes, regardless of what checks
value the caller supplied, with a round count strong enough to keep the
false-positive probability below 2^-80. Callers that pass a larger
explicit checks still get exactly that many rounds.
Unit tests under ext/node_crypto/primes.rs cover the
17,881 × 17,891 case, a larger 64-bit composite, and the FIPS lookup
table itself.
Workarounds
If you cannot upgrade immediately:
- Pass an explicit
checks value when calling crypto.checkPrime or crypto.checkPrimeSync. A value of 64 is conservative for any reasonable bit length and keeps the loop running. - Do not rely on
crypto.checkPrime to validate attacker-influenced bignums in security-critical paths until you are on the patched release.
