Down Left Arrow Logo
Logo

What we can learn from the CrowdStrike outage

First, mistakes happen, this just happens to be a monumental mistake affecting millions of users and services worldwide. In cases like this it's easy to get angry and point fingers, looking for someone to blame. Instead, let's have some sympathy for both parties involved (even if it's breif) and look at what lessons we can take from this monumental gaffe.

Be careful with updates

With any new update to critical systems, these should be handled with care. Ideally you would have a test environment that mimics your live environment so you can perform tests before rolling them out to your live systems. We're not all so lucky to have such a thing, so it is generally acceptable to update one (1) system at a time, ideally long after the update has been released from Microsoft to let early adopters find the bugs before you expose it to your critical systems. Once you've updated one system, monitor for issues, then move on to the next. Preferably, you'd start with the least vital system if possible.

Test your backups

You can have the best backup systems on the planet, with robust storage and months of redundancy. But what if your backups are corrupt? Or you're not backing up the right things? Or your backups don't work with the hardware you have earmarked for recovery.

Backup your Bitlocker Keys

Or whatever encryption program you happen to be using. Personally, I backup all my client system bitlocker keys to Azure AD. While that's super convenient and easy, there's no saying something like the CrowdStrike outage couldn't hit Microfot Azure systems. I'm not saying not to use Azure AD for your Bitlocker keys, but having an offline backup of your keys might just save your butt, or at least prevent some major headaches.

For example, the fix for this CrowdStrike outage is to boot the system into Safe Mode, then delete the affected driver. Problem here is your system will prompt you for the Bitlocker recovery key when attempting to do this. Good luck getting those keys if they are backed up on a affected Windows Server. I imagine a number of System Admins would give a large portion of their salaries to have their Bitlocker Keys readily available to them.