HollowByte Vulnerability in OpenSSL Triggers Memory Leak via 11-Byte Packet

Okta’s Red Team researchers have identified a severe vulnerability in the OpenSSL library that enables unauthorized attackers to trigger denial-of-service (DoS) conditions on servers. Dubbed HollowByte, the flaw is triggered by sending just an 11-byte packet, which forces the server to allocate excessive memory without actual data transfer.

The vulnerability stems from how TLS handshakes are processed in affected OpenSSL versions. Servers allocate memory based on the declared packet size in headers, without verifying the actual data volume. Attackers can send packets with inflated length values, causing memory fragmentation and uncontrolled growth. Even after the attack ends, servers remain in a high-resource consumption state until processes are restarted.

OpenSSL is widely used across most Linux distributions, popular web servers (NGINX, Apache), programming environments (Node.js, Python, PHP), and databases (MySQL, PostgreSQL). Okta’s testing revealed that the attack can disable low-resource servers, while high-end systems lose up to 25% of available memory without triggering monitoring alerts.

OpenSSL developers have patched the issue in versions 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21. The fix modifies memory allocation: buffers now expand only upon actual data arrival, ignoring header-declared values. Though classified as a 'security enhancement' rather than a critical flaw, experts urge immediate library updates.