Cloudflare faced a disruption in its 1.1.1.1 DNS resolver service on June 27, 2024. This disruption lasted for several hours and was caused by a combination of BGP hijacking and a route leak. As a result, users around the world experienced difficulties accessing the service, with some unable to reach it at all and others facing high latency.

The incident began at 18:51 UTC when AS267613 (Eletronet) initiated the announcement of the 1.1.1.1/32 prefix to its peers and upstream providers. This unauthorized announcement led to the misrouting of traffic intended for Cloudflare’s DNS resolver. Shortly after, at 18:52 UTC, AS262504 (Nova) leaked the 1.1.1.0/24 prefix upstream to AS1031 (Peer-1 Global Internet Exchange), further exacerbating the impact of the incident.

Cloudflare’s internal monitoring systems detected the issue at 20:03 UTC, prompting immediate action. By 20:08 UTC, Cloudflare had disabled a partner peering location with AS267613 and engaged with the network to address the problem. The route leak was resolved on June 28, 2024, when AS262504 ceased the unauthorized announcements. However, the impact on users persisted throughout the incident, with some experiencing high latency and others unable to access the 1.1.1.1 service.

The incident’s root cause was identified as a mix of BGP hijacking and route leaks. BGP hijacking occurs when a network announces IP prefixes it does not own, resulting in misrouted traffic. In this case, AS267613 announced the 1.1.1.1/32 prefix, which was accepted by multiple networks, leading to traffic blackholing. Route leaks occur when a network incorrectly announces prefixes it has learned from one provider to another. AS262504’s leak of the 1.1.1.0/24 prefix to AS1031 further propagated the issue.

The incident impacted users in various countries, including Germany and the United States. While the overall percentage of affected users was relatively low, the disruption was significant for those relying on the 1.1.1.1 service for DNS resolution. Cloudflare took steps to mitigate the impact by disabling peering with AS267613 in multiple locations and engaging with all networks involved.

To prevent similar incidents in the future, Cloudflare is advocating for the adoption of RPKI for route origin and AS path validation. RPKI helps limit the spread of hijacked BGP prefixes by securely storing and sharing ownership information. The company also promotes ASPA objects for BGP to prevent route leaks by signing AS paths with authorized provider networks.

The outage highlighted vulnerabilities in the current BGP system, and Cloudflare is actively enhancing its security measures to reduce the risk of future incidents. Users are encouraged to check if their ISPs enforce RPKI origin validation. Cloudflare remains committed to providing reliable and secure DNS resolution services while working towards improving the resilience of its network infrastructure.

Source link