MessagePack: Unbounded memory leak in msgpack C extension on strict_map_key validation failure

Impact

The msgpack-python C extension (msgpack._cmsgpack) leaks a decoded map value when deserialization rejects a key with strict_map_key=True (the default). Repeated malformed inputs can cause unbounded process memory growth and denial of service in applications that deserialize attacker-controlled MessagePack data. Both unpackb() and the streaming Unpacker API are affected. The pure-Python fallback does not exhibit this leak in the reported tests.

The report measured approximately 4 MiB retained per failed parse and 100 MiB after 25 failed parses on Windows with Python 3.14 and msgpack 1.2.2. These are reporter-provided results.

Additional impact: stale parser-stack references

The reference-ownership audit performed while fixing this leak found an additional memory-safety issue in the streaming Unpacker.

After unpack() completed a nested container, an inactive parser-stack slot could retain a non-owning pointer to the decoded object. A subsequent skip() could reactivate that slot. If parsing then failed, context cleanup treated the stale pointer as an owned reference and decremented it.

This corrupts the object's reference count and can result in a use-after-free or double Py_DECREF when its actual owners later release it.

Although few users use this API, but those who do may experience crashes or unexpected errors due to a use-after-free.

Details

In msgpack/unpack.h, unpack_callback_map_item() rejects keys that are neither str nor bytes. At this point the map value has already been constructed. The early error return does not release the value reference.

In msgpack/unpack_template.h, the value is held in the local obj in unpack_execute(). The CT_MAP_VALUE error path jumps to _failed without decrementing that reference. Cleanup of the parser stack does not recover the value because it is not stored in the stack. The fallback checks the key before decoding the value.

Affected versions

The report reproduces the issue on 1.2.2 and reports it on development commit 475c1b7. The affected-product field lists the reproduced release only; it is not a claim that earlier releases are unaffected.

The earliest affected release needs confirmation: the report's summary says 1.1.x through 1.2.2, while its root-cause analysis attributes introduction to commit cd81370 (PR #673). Resolve this discrepancy before publishing a definitive affected-version range.

Reproduction

Run in a disposable local process with the C extension installed:

import msgpack

payload = msgpack.packb({42: b"A" * (4 * 1024 * 1024)}, use_bin_type=True)
msgpack.unpackb(payload)  # Raises ValueError; the report observes a 4 MiB leak.

Streaming API trigger:

import msgpack

payload = msgpack.packb({42: b"A" * (4 * 1024 * 1024)}, use_bin_type=True)
u = msgpack.Unpacker()
u.feed(payload)
list(u)  # Same strict_map_key error path.

Workarounds

The pure-Python fallback is reported not to leak on this path and may be used as a temporary workaround after application compatibility and performance checks. Request-size and rate limits may reduce the impact but do not eliminate cumulative leakage.