CrowdStrike’s Root Cause Analysis (RCA) of the faulty software update that caused a major outage affecting 8.5 million Windows machines on July 19 has provided a detailed explanation of what went wrong. The 12-page report delves deeper into the incident than the Preliminary Post-Incident Review (PIR) released earlier, shedding light on the specific technical issues that led to the outage.
According to CrowdStrike, the initial error that triggered the outage occurred back in February when the company released a new sensor version (7.11) that included a new Template Type for Windows interprocess communication (IPC) mechanisms. The integration code that invoked the Content Interpreter with Channel File 291’s Template Instances failed to provide the required 21 input parameter fields, resulting in a mismatch that went undetected during validation and testing processes.
On July 19, two additional IPC Template Instances were deployed, one of which introduced a non-wildcard matching criterion for the 21st input parameter. This change exposed a latent out-of-bounds read issue in the Content Interpreter, leading to a system crash when the interpreter attempted to access the 21st input value despite expecting only 20 values.
As a result of this incident, CrowdStrike has outlined six key changes it will implement to prevent similar outages in the future. These include validating the number of input fields in the Template Type at sensor compile time, adding a runtime array bounds check for Content Interpreter input fields, and implementing staged deployment for template instances with customer control over rollout.
In addition to addressing the technical aspects of the outage, CrowdStrike also acknowledged the need to adapt its approach to Windows Kernel Driver usage. The company plans to move kernel driver functions to less-sensitive user space as Windows evolves to support more security functions in user space, reducing the reliance on kernel drivers for core functionality.
CEO and founder of CrowdStrike, George Kurtz, issued an apology for the impact of the outage on customers and emphasized the company’s commitment to customer protection and trust. However, legal battles surrounding the outage are far from over, as Microsoft has entered the fray by challenging Delta Airlines’ handling of the incident.
According to Microsoft, Delta’s prolonged recovery from the outage was exacerbated by the use of non-Microsoft systems for critical IT functions, such as crew-tracking and scheduling. The company refuted Delta’s claims and warned of potential litigation if Delta chooses to pursue legal action.
As the fallout from the CrowdStrike outage continues to unfold, it remains to be seen how the involved parties will navigate the legal and reputational challenges stemming from this high-profile incident.