CrowdStrike Falcon Down: How a single security update shutdown Windows worldwide

Today, on July 19, 2024, a CrowdStrike code update led to global issues affecting Windows computers. The operating systems crashed repeatedly, displaying Blue Screen of Death (BSoD), keeping them in a non-usable loop state.

This outage caused around 1400 flights to be cancelled, as well as numerous services to be stuck in a dysfunctional state: banks, airports, train stations, broadcasters, and even online game servers. It’s one of the largest IT service outages that has happened in recent times.

The culprit was the Falcon Sensor, a component of CrowdStrike made to block attacks and record system activity. The CrowdStrike suite is very popular in large businesses relying on their Windows infrastructure & end-user devices.

Technical resolution of the issue

As for the cause, it has all been caused by a faulty channel file. This can be resolved in a few ways, depending on the scale.

Small-scale systems

If you’re dealing with a handful of computers, this can be fixed the classic way:

1. Launch Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete the file matching C-00000291*.sys
4. Reboot the system as usual

Large-scale systems

Now, if you’re working on a large scale and have to manage thousands of machines, this might get tricky. Here’s a quick guide for IT professionals to bring your business back to life:

1. Grab an appropriate Windows Assessment and Deployment Kit (ADK)
2. Mount the WinPE file with wimlib, or use Microsoft’s tools
3. Edit startnet.cmd and add the following lines:
   del C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys
exit
4. Save the startnet.cmd & unmount the image
5. Copy the image to a PXE server or to a pendrive
6. Boot into the impacted system via the image & enjoy the automation

Cloud systems

Now, regardless of scale, cloud environments have their own specification, thus they require a different approach.

1. Detach the operating system disk volume from the impacted virtual server
2. Create a snapshot or backup of the disk volume
3. Attach the volume to a new virtual server
4. Navigate to C:\Windows\System32\drivers\CrowdStrike
5. Locate the file matching C-00000291*.sys and delete it
6. Detach the volume from the new virtual server
7. Reattach the fixed volume to the impacted virtual server

Parting words

With all of the above in mind, we’ve exhausted one of the largest IT service outages to date. Thank you for reading, and if you have any questions or stories to share, feel free to post them in the comments to this article!