In a week that saw a cryptocurrency exchange lose $235 million to hackers and a botched migration that led to the takeover of four crypto company domains, CrowdStrike (CRWD) topped them all with possibly the biggest self-own of all time.
CrowdStrike basically bricked the internet overnight with a single faulty channel update file for Windows machines, in the process grounding flights, knocking banks offline and wreaking havoc with healthcare and other critical infrastructure.
With nearly 9 million machines affected, it is likely the biggest-ever cyber event – or if you prefer, “technical outage.” And as the fix requires each machine to be manually rebooted, it will likely take days to clean up the damage. Pity your local IT admins, because they may be headed for a long weekend of cleanup. One assessment of how bad the cleanup is going to be was viewed on X more than 2 million times:
It will take time to assess the full damage and costs, but as CrowdStrike stock lost about $8 billion in market cap on Friday, Wall Street traders are certainly expecting some lasting fallout. It’s not clear how much of this cyber insurance might cover (and what will happen to premiums as a result), but given the scale of affected machines and organizations, the cost will be astronomical at least in terms of lost productivity and staff time. And CrowdStrike’s terms and conditions may push most of the cost onto the company’s customers.
In the meantime, the CrowdStrike incident has at least been worth some good memes:
CrowdStrike’s Reach Is Part of the Problem
Interestingly, a similar incident hit McAfee in 2010 – when CrowdStrike CEO George Kurtz was CTO of McAfee. He apparently got the inspiration to launch CrowdStrike from there.
CrowdStrike is, in my analysis, a Top 5 Endpoint Detection and Response (EDR) vendor, based on years of following independent testing from MITRE and other organizations. What sets the company apart is its incident response capabilities – and overburdened security teams value rapid cleanup more than they do top security.
CrowdStrike has moved into many other cybersecurity markets over the years, and its ability to market itself as a leading-edge security vendor has enabled it to land some very large customers with equally large security needs. Among the company’s 29,000 customers are 298 of the Fortune 500, 8 of the top 10 technology companies, 8 of the 10 largest financial services firms, and 6 of the 10 largest healthcare providers, and 7 of the 10 largest manufacturers.
When you look at it that way, it’s easy to see why a single improperly formatted file could bring down the global internet.
So What the Heck Happened in the CrowdStrike Outage?
According to CrowdStrike’s own explanation of events, a single channel file (“C-00000291*.sys” with timestamp of 0409 UTC) led to all the problems.
That file has been submitted to VirusTotal – and no security vendors have flagged it as malicious yet, supporting CrowdStrike’s claim that the invalidly formatted file is not the result of a cyber attack.
However, to assuage customers, CrowdStrike will have to explain how the file got past QA checks – and what the company will do to avoid a similar incident in the future.
Interestingly, CrowdStrike may share one thing in common with another one of the week’s facepalm events – the Squarespace crypto domain hijacking – as rushed development processes that weren’t given adequate quality and security checks may well be a central factor in both events.
What led to the $235 million crypto theft has yet to be adequately explained.
Fixing the CrowdStrike BSOD
CrowdStrike offers these instructions for Windows machines hit by the “blue screen of death” (BSOD):
- Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys” and delete it.
- Boot the host normally.
- Boot Windows into Safe Mode or the Windows Recovery Environment
Bitlocker-encrypted hosts may require a recovery key. As one observer wryly noted, “As a bonus, today we learn who responsibly manages Bitlocker recovery keys.”
Update: Microsoft has released a tool to help with CrowdStrike BSOD recovery – including guidance on BitLocker key recovery.
Microsoft Azure Hit in Separate Incident
Not surprisingly, Microsoft was hit by a number of incidents across Windows machines and Microsoft 365 apps, services and Cloud PCs.
At the same time, Microsoft was hit by a separate Azure outage, as the company noted in a service update.
“We are aware of an issue that started on July 18, which resulted in customers experiencing unresponsiveness and startup failures on Windows machines using the CrowdStrike Falcon agent, affecting both on-premises and various cloud platforms (Azure, AWS, and Google Cloud),” the Azure update said.
“It’s important to clarify that this incident is separate from the resolved Central US Azure outage (Tracking Id: 1K80-N_8). Microsoft is actively providing support to assist customers in their recovery on our platforms, offering additional guidance and technical assistance.”
Microsoft explained the cause of the Azure service degradation: “We determined that a backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks hosted on impacted storage resources.”
What’s Next After the CrowdStrike Outage?
The CrowdStrike cleanup will be involved and costly, and once it’s done, customers – and those affected by CrowdStrike customers – will want a thorough accounting of the incident, along with concrete steps to make sure it doesn’t happen again.
Until then, expect to see more memes – including the company’s own marketing materials used against it, as in this Mastodon post from Microsoft security researcher Kevin Beaumont:
Published on July 19, 2024, and updated on July 22, 2024 to add Microsoft’s recovery tool and information on the scope and cost of the outage
