CyberSecurity SEE

Cloudflare Container Vulnerability May Expose Data from Other Customers’ Workloads

Cloudflare Container Vulnerability May Expose Data from Other Customers’ Workloads

Cloudflare Addresses Significant Cross-Tenant Data Exposure Vulnerability

Cloudflare has recently taken significant steps to rectify a cross-tenant data exposure vulnerability found in its Containers platform. This flaw had the potential to allow one customer’s workload to recover residual data from other customers operating on the same infrastructure. The implications of this vulnerability extend beyond just the Containers platform, affecting Cloudflare Sandboxes and the Browser Run service associated with Browser Rendering, which relies on the same underlying disk structure.

Discovery and Response

The concerning issue was initially reported on September 4, 2026, by Oren Yomtov from the firm Accomplish, through Cloudflare’s bug bounty program. Following the report, Cloudflare declared that it had implemented fleet-wide remediation measures and found no evidence to suggest that the vulnerability had been exploited maliciously. Notably, the company assured its customers that they would not need to take any action regarding this matter.

Technical Details of the Vulnerability

The root of the vulnerability lay in Cloudflare’s utilization of Linux device mapper thin provisioning, specifically known as dm-thin, to manage writable container root disks. Each individual Cloudflare Container functions within a dedicated Firecracker virtual machine, where its root disk is presented under the designation /dev/vdc. Unfortunately, the affected infrastructure shared 64 KB thin-provisioning blocks across multiple customer accounts, leading to possible data exposure.

The underlying storage setup had enabled the skip_block_zeroing option. This configuration prevented dm-thin from erasing a physical block that had been previously utilized before reallocating it to a new container. As a result, a newly assigned container could inadvertently write data to an unmapped region, resulting in the allocation of a recycled 64 KB block. While the new data would overwrite part of the block, unaltered bytes could still harbor data from prior tenants.

Research conducted in this area revealed alarming findings: writing a 4 KB-aligned block into ext4 free-space regions could expose the residual 60 KB of data contained within the recycled block through a direct raw disk read. This issue represents a significant tenant-isolation failure, heightening concerns about data privacy and security.

Potential Impact on Customers

A customer operating on the Workers Paid account could, under certain circumstances, potentially retrieve a wealth of sensitive information such as filesystem metadata, directory listings, SQLite database pages, Chromium profiles, .env files, and various credential-related artifacts left by previous workloads. Compounding the issue, the proof-of-concept for this exploit did not target any specific victim, customer, host, or live disk. Instead, it was contingent upon workload placement and whether a previously utilized disk block was reassigned to the attacker’s container.

During controlled testing across six production placements, researchers identified an alarming 2,700 distinct foreign directory inodes. They discovered residual data on 18 out of 24 placements across 20 of the 22 underlying nodes stretched across four continents. The study employed ext4 directory checksums to differentiate between foreign blocks and their own test data, focusing on aggregate counts, offsets, checksums, and truncated hashes to avoid directly submitting third-party content.

Mitigation Strategies

In response to the vulnerability, Cloudflare undertook substantial mitigation efforts. These actions involved the removal of the skip_block_zeroing option from dm-thin pools, thereby restoring the default process of clearing blocks before they are exposed to containers. Additionally, the company retired existing container disks, drained hosts, rebooted virtual machines, and deleted cached OCI image snapshots created prior to the configuration adjustment. These actions were deemed necessary as previously mapped blocks and cached layers could still retain residual data, even with the reinstatement of block zeroing.

Cloudflare’s review of historical disk I/O telemetry only revealed activity correlated with the researchers and approved internal validation efforts. The incident serves as a stark reminder that multi-tenant isolation can fail due to various factors, including not only hypervisor or container escapes but also vulnerabilities at the storage-layer level and unsafe block-reuse configurations.

Conclusion

As the tech landscape continues to evolve, the significance of robust security measures cannot be overstated. The actions taken by Cloudflare not only demonstrate their commitment to securing customer data but also underline the importance of constant vigilance in the realm of cybersecurity. With the introduction of updated protocols and comprehensive remediation strategies, Cloudflare is taking necessary steps to ensure that they provide a safer environment for all their clients in the future.

Source link

Exit mobile version