Forge: Denial of Service via infinite loop in BigInteger.modInverse() with negative input (incomplete fix of CVE-2026-33891)

Summary

BigInteger.prototype.modInverse() in node-forge's bundled jsbn library enters an infinite loop when the value or the modulus is negative, for example when a negative public exponent is passed to forge.pki.rsa.generateKeyPair(). This is an incomplete fix of CVE-2026-33891 (GHSA-5m6q-g25r-mvwx), which handled only a zero value. Fixed in node-forge 1.4.1.

Details

bnModInverse() (lib/jsbn.js, exposed as forge.jsbn.BigInteger.prototype.modInverse) implements the binary extended Euclidean algorithm (Handbook of Applied Cryptography, Algorithm 14.61), which is defined for positive integers. With a negative operand the loop cannot terminate: rShiftTo is an arithmetic shift, so a negative value never moves towards zero, and the loop's exit condition (u reaching zero) is never met. The calling thread spins at 100% CPU indefinitely; in Node.js this blocks the event loop. Memory does not grow.

node-forge 1.4.0 fixed CVE-2026-33891 (GHSA-5m6q-g25r-mvwx) by returning zero early when this.signum() == 0, although its changelog describes the check as "<= 0". Negative values were not covered, and a negative modulus was never handled. Every negative input that did terminate already returned zero.

Within node-forge, RSA key generation is the only path to a negative operand: the caller's public exponent e becomes BigInteger.fromInt(e), and state.e.modInverse(state.phi) is called with it. gcd() works on absolute values, so the coprimality checks pass and the loop is reached. Any exponent other than 3 or 65537 uses the pure JavaScript implementation, even when forge.options.usePureJavaScript is not set. RFC 8017, Section 3.1 requires the RSA public exponent to be at least 3.

Values that node-forge parses (DER, PEM, JWK, PKCS#12, X.509, TLS) are always read as unsigned integers and cannot produce a negative operand.

The fix makes modInverse() return zero when the value or the modulus is zero or negative, as it already does when no inverse exists. Results for positive inputs are unchanged.

This issue was reported by wzc.

Patches

Fixed in node-forge 1.4.1. modInverse() now returns zero when the value or the modulus is zero or negative.

Workarounds

• Pass only fixed, valid RSA public exponents (e.g. 65537) to key generation, or validate caller-supplied exponents first: an odd integer >= 3.
• Do not call BigInteger#modInverse() with negative operands. Reduce the value modulo a positive modulus first (x.mod(m).modInverse(m)).

Proof of concept

A negative public exponent passed to RSA key generation never returns (also with the callback form and with createKeyPairGenerationState() / stepKeyPairGenerationState()):

var forge = require('node-forge');
forge.pki.rsa.generateKeyPair({bits: 256, e: -1});  // hangs at 100% CPU

Calling modInverse() directly with a negative value or modulus hangs the same way:

var BigInteger = forge.jsbn.BigInteger;
new BigInteger('-1').modInverse(new BigInteger('7'));  // hangs
new BigInteger('3').modInverse(new BigInteger('-7'));  // hangs

With the fix, these calls return zero. The regression tests are in tests/unit/jsbn.js.

Impact

Applications are affected only if they:

• pass an untrusted value as the RSA public exponent e to forge.pki.rsa.generateKeyPair(), forge.pki.rsa.createKeyPairGenerationState() or forge.pki.rsa.stepKeyPairGenerationState(), or
• call forge.jsbn.BigInteger#modInverse() directly with negative values derived from untrusted input.

An attacker who can supply such a value makes the call hang indefinitely at 100% CPU, which blocks the Node.js event loop (denial of service). Keys and other data parsed by node-forge (DER, PEM, JWK, PKCS#12, X.509, TLS) cannot trigger this issue, and the documented usage of key generation passes a constant exponent (65537). There is no effect on confidentiality or integrity.