Last week, CrowdStrike faced a unprecedented setback when an update to its Falcon software caused 8.5 million Windows machines to crash. The company has since published a detailed post-incident review, attributing the issue to a bug in their test software, which failed to validate the content update properly.
The problematic update was a 40KB Rapid Response Content file intended to gather telemetry on potential novel threat techniques but instead led to widespread system crashes.
CrowdStrike’s Falcon software is a crucial tool for businesses worldwide that provids malware and security breach management for millions of Windows machines. Typically, the company issues two types of updates: Sensor Content, which directly updates the Falcon sensor at the kernel level, and Rapid Response Content, which adjusts the sensor’s behaviour to detect malware. It was the latter that caused the chaos last Friday.
A bug in the Content Validator allowed the problematic content to pass through undetected. When this content was loaded into the sensor’s Content Interpreter, it triggered an out-of-bounds memory exception, leading to Windows crashing with a Blue Screen of Death (BSOD). CrowdStrike has said that while they perform both automated and manual testing on Sensor Content and Template Types, their testing of Rapid Response Content was not as thorough.
To prevent such incidents from recurring, CrowdStrike is enhancing its testing protocols. The company plans to implement local developer testing, content update and rollback testing, stress testing, fuzzing, fault injection and stability and content interface testing for Rapid Response Content. They are also updating their cloud-based Content Validator to include additional checks to prevent problematic content from being deployed.
In addition, CrowdStrike will enhance the error handling capabilities of the Content Interpreter and adopt a staggered deployment process for Rapid Response Content. This approach will ensure updates are gradually rolled out to a larger portion of their install base rather than being pushed to all systems at once.
_20240918083333_original_image_44.webp)