CrowdStrike Outage Updates , Incident & Breach Response , Security Operations

CrowdStrike Cleanup: Vast Majority of Systems Restored

On day five of the Windows outages due to a faulty CrowdStrike update, there is cautious optimism as IT experts report significant restoration of downed systems.

One U.S.-based IT asset management platform, Sevco Security, said that as of Monday evening, 93% of affected CrowdStrike systems across its customer base appeared to have been fixed, up from 89% as of Friday night.

Microsoft on Sunday estimated that 8.5 million Windows hosts comprising, clients, servers and Hyper-V virtual machines, crashed due to a faulty CrowdStrike Falcon endpoint detection and response software "content update." CrowdStrike pushed the faulty update at 4:09 UTC on Friday - just after midnight on the East Coast of the U.S. - and replaced it with a fixed update just over an hour later, at 5:27 UTC. Any system running CrowdStrike Falcon that was online during that time was potentially affected.

Microsoft said the disrupted systems comprised less than 1% of active Windows systems, but the impact has been much more significant than that suggests, since many of the CrowdStrike EDR product's customers are larger organizations in critical infrastructure sectors. This was highlighted by the mix of sectors disrupted, including hospitals and doctors' offices, major airlines, railways, and banking and stock exchanges (see: CrowdStrike Disruption Restoration Is Taking Time).

Some organizations continue to struggle more than others. Major U.S. airline Delta has faced days of disruption, including the cancellation of 30% of its flights on Monday and delays for half of the remaining flights, according to flight tracking site FlightAware.

Systems that received the faulty code update ended up stuck in a constant loop: crashing out to a Windows "blue screen of death" due to the code, rebooting and then encountering the bad code again and crashing again.

"We are still experiencing reachability problems with most of our Windows-based AWS EC2 instances despite multiple reboots," said "Le7emesens" in a post to the r/crowdstrike subreddit, asking if there might be any upcoming, automatic updates for AWS and CrowdStrike that could fix affected instances.

"Basically we would prefer not to do any more gruesome hand work of the manual recovery," Le7emesens said. "Our weekend was already screwed up. I'm sure I'm echoing a lot of folks here :)."

Many IT personnel report having to manually recover systems. Both CrowdStrike and Microsoft have released utilities that can help. For physical Windows hosts, running those utilities typically requires loading them onto a bootable USB drive and plugging it into the affected system, or else plugging a device into the network and targeting it using a server-based Preboot Execution Environment for recovery. As a result, remote or hybrid workers may have to go to the office to get the fix.

For any system that uses full-disk encryption, such as Microsoft's BitLocker, to protect the data being stored on the hard drive, the recovery process will take extra time, since the recovery key must be entered before the recovery utility can run. Some organizations may store copies of such keys centrally, although experts say in many cases they do this only for their most critical servers and other systems, and thus need to compile them for affected end users' systems.

Individual users may also be able to access the recovery key for their system by logging into their Microsoft account on a different device.

To help involve more end users in the recovery process, CrowdStrike on Monday released a five-minute video on YouTube to guide them through the process, requesting that they first get their IT department's permission to do so.

"During this process, some users may encounter a screen asking for a 'BitLocker' recovery key. Please reach out to your IT department to get this information," the video states. "This is a 48-digit numerical key that you may need to enter multiple times throughout the process. Unfortunately, this may take some time."

Not all affected Windows systems are easy to access, as demonstrated by photographs circulating on social media of IT personnel climbing ladders to reach connections for airport kiosks and other "in the field" systems. Cybersecurity experts say the rise in outsourcing in recent years means some organizations may have fewer IT staff at the ready to deploy and quickly reach these systems.

Despite the challenges, multiple cybersecurity experts lauded CrowdStrike for moving quickly once the problem came to light, offering transparency into the remediation process and putting senior executives front and center to apologize (see: CrowdStrike's Response to Outage Will Minimize Lost Business).

How much the publicly traded cybersecurity firm may be on the hook for the outages tied to its faulty software content update remains to be seen.

"We continue to work with impacted customers to fully restore their systems," CrowdStrike said Monday in a Form 8-K filing to the U.S. Securities and Exchange Commission. "This is an evolving situation. We continue to evaluate the impact of the event on our business and operations."