Rustls: Brotli certificate decompression can allocate an attacker-controlled amount of memory up to 1 GiB before authentication.

Summary

The built-in (but disabled by default) Brotli certificate decoder bounds the output buffer but accepts an input-selected history window. A header of five bytes can specify a 1 GiB window allocation. A peer can abort a process whose allocator cannot satisfy that request. A client must enable Brotli and connect to the peer; a server must enable Brotli and request a client certificate. Neither direction requires a trusted attacker certificate.

Details

rustls calls BrotliDecompress which enables Brotli's large-window extension. rustls applies a limit to certificates, but this applies to the decompressed result, and not to any internal allocations the process requires.

Impact

Client applications that enable the brotli feature will be vulnerable to malicious servers. Server applications that enable the brotli feature and enable client authentication will be vulnerable to malicious clients.

Potential for denial of service due to memory exhaustion from peers sending malicious Brotli-encoded certificates, for applications that enable Brotli decompression.

Acknowledgement

Thanks to OpenAI OutboundDisclosures@openai.com for the disclosure.