CrowdStrike is actively working to resolve a defect in a content update that struck about 8.5 million Windows machines on July 19 – and continues to disrupt many Windows hosts days later.
In a recent update, the cybersecurity company said it has “tested an update to the remediation that was deployed on Friday, July 19, 2024 05:27 UTC. The update has accelerated our ability to remediate hosts. Customers are encouraged to follow the Tech Alerts for latest updates as they happen.”
The organization has also “published a video outlining the steps required to self-remediate impacted remote Windows laptops. We will continue to provide updates here as information becomes available and new fixes are deployed”, denoted the official response. In the update, CrowdStrike explains that this incident is not a cyberattack and they have already “identified and isolated, and a fix has been deployed” for mitigation.
CrowdStrike has previously introduced a new technique aimed at expediting system remediation for impacted systems. The company is currently in the process of operationalizing an opt-in option for this method. Customers are encouraged to stay informed by following CrowdStrike’s Tech Alerts for timely updates and will receive notifications when action is necessary.
Microsoft has also released a fix for the faulty CrowdStrike update, which resulted in bugcheck and “blue screen of death” (BSOD) errors on millions of Windows hosts.
Delta Airlines was one noteworthy company struggling to recover from the outages, and was still canceling about 20% of its flights as of early afternoon Eastern U.S. time on Monday, July 22. CrowdStrike shares (CRWD) have plunged more than 20% since the incident, erasing roughly $15 billion in market cap.
CEO George Kurtz has assured customers that the faulty update was not due to a cyberattack and that Falcon platform systems remain unaffected.
CrowdStrike Outage Response and Customer Support
The defective update stemmed from a Windows sensor-related content deployment, specifically a channel file in the CrowdStrike directory, which has sparked widespread discussion in the cybersecurity industry about how to ensure that software updates and rollouts are safer and more reliable.
CrowdStrike CSO Shawn Henry took to LinkedIn to apologize for the incident:
“On Friday, though, we failed. The past two days have been the most challenging 48 hours for me over 12+ years. The confidence we built in drips over the years was lost in buckets within hours, and it was a gut punch.
But this pales in comparison to the pain we’ve caused our customers and our partners. We let down the very people we committed to protect, and to say we’re devastated is a huge understatement.
I, and the entire company, take that personally. Thousands of our team members have been working 24/7 to get our customer systems fully restored. The days have been long and the nights have been short, and that will continue for the immediate future. But that is part of the promise we made to all of you when you put your trust and protection in our hands.”
The company quickly mobilized its resources to assist affected customers. A new technique to accelerate system remediation was tested in collaboration with clients, with an opt-in process being implemented. CrowdStrike is providing regular updates through its support portal and social media channels, urging customers to verify communication with official representatives.
Kurtz emphasized the company’s commitment to transparency and customer trust. “Nothing is more important to me than the trust and confidence that our customers and partners have put into CrowdStrike,” he stated. The CEO promised full disclosure on the incident’s cause and preventive measures for the future.
Technical Details and Remediation Steps
For systems still experiencing crashes, CrowdStrike recommends rebooting to download the reverted channel file – multiple times, if necessary. If issues persist, manual or automated remediation options are available, including the use of a bootable USB key for automated fixes.
In response to the widespread issues caused by the faulty update of the CrowdStrike Falcon agent on Windows-based clients and servers, Microsoft released its own recovery tool to help system administrators and IT staff. The updated Microsoft recovery tool offers two repair options – Recover from WinPE (Windows Preinstallation Environment) or Recover from Safe Mode – and also includes guidance for recovering BitLocker encryption keys, if necessary.
As the situation evolves, CrowdStrike continues to prioritize customer support and system restoration, even as the issue of who will pay for the restoration efforts remains unresolved. The company acknowledges the impact of the incident and says it is working tirelessly to regain customer confidence through transparent communication and effective problem-solving.
Shawn stated in his post, “I know I speak for the women and men of CrowdStrike when I say thank you to every customer and partner who has also been working around the clock. You are the real heroes in all of this. We are committed to re-earning your trust by delivering the protection you need to disrupt the adversaries targeting you. Despite this setback, the mission endures.”
