Newly Disclosed OpenSSL Vulnerability Underscores Reliance on Foundational Libraries
Recent findings have unveiled a vulnerability that starkly highlights the reliance of digital infrastructure on critical libraries. A team from Okta Red Team has identified a Denial of Service (DoS) flaw, dubbed “HollowByte,” within OpenSSL. This vulnerability enables a remote, unauthenticated attacker to compel a server to allocate disproportionate memory before the initiation of any security handshake, utilizing a payload as small as 11 bytes.
The TLS (Transport Layer Security) handshake kicks off with a ClientHello message, wrapped in a record format. Each handshake message is accompanied by a 4-byte header that specifies the size of the incoming message body. In older versions of OpenSSL, the mechanism for allocating a receive buffer is triggered based on the attacker-declared length, occurring before any actual data has been received. Consequently, when the malicious 11-byte payload arrives, the TLS state machine interprets the header and initiates an unchecked pre-allocation according to its 3-byte length declaration.
The sequence of operations, as follows, outlines the vulnerability:
- Read Header →
grow_init_buf() →OPENSSL_clear_realloc() →malloc(attacker_size)
Due to the absence of payload validation at this early stage, malloc() allocates memory—up to 131 KB—based solely on the unverified claim made by the attacker. This situation results in the worker thread entering a state of indefinite blocking, awaiting data that never arrives. This tactic, of maintaining open connections to deplete server threads, mirrors strategies used in previous attacks like Slowloris. However, HollowByte exacerbates the situation with additional complications stemming from how glibc, the GNU C Library, manages memory.
When a connection is terminated, OpenSSL does free the allocated buffer. Yet, glibc does not swiftly return smaller to medium-sized memory allocations back to the operating system; it keeps them for possible reuse. By orchestrating numerous connection attempts with random size claims, an attacker can hinder the allocator’s ability to reuse freed memory chunks. This leads to severe heap fragmentation, causing the server’s Resident Set Size (RSS) to steadily increase.
Once the attacker disconnects, the server’s memory remains bloated, necessitating a complete process termination to restore normal functionality. Testing conducted on both unpatched and patched OpenSSL versions under NGINX has illuminated the severity of the threat. In environments with 1 GB of RAM, the unpatched server was subjected to an Out of Memory (OOM) kill at 547 MB of frozen, fragmented memory. In a scenario with 16 GB of RAM, the attack managed to seize 25% of total system memory while navigating comfortably beneath conventional connection thresholds, indicating that standard connection-limiting defenses are ineffective against this particular threat.
Since OpenSSL is widely used and incorporated in various systems, this vulnerability poses risks to an array of web servers—including Apache and NGINX—as well as language runtimes like Node.js, Python, Ruby, and PHP, and even databases such as MySQL and PostgreSQL.
In response to this significant vulnerability, OpenSSL has addressed the issue by implementing incremental buffer growth, as indicated by merged pull requests (PRs) #30792, #30793, and #30794. The resolution was quietly included in OpenSSL version 4.0.1, with backports made available for earlier versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21. Instead of outright trusting the declared header size, OpenSSL has adjusted its approach to grow buffers only as actual bytes are received over the network, thereby mitigating the server’s burden from empty claims. Remarkably, OpenSSL has characterized this update as a hardening fix, refraining from issuing a formal Common Vulnerabilities and Exposures (CVE) advisory.
Given the widespread implications of this vulnerability, it is imperative for security teams to prioritize upgrading OpenSSL packages without delay, especially on servers accessible from the internet. Conventional connection-limiting strategies will not offer adequate protection against the HollowByte attack, rendering an immediate response essential for safeguarding digital assets and maintaining operational integrity.