The global IT outage from late July, which grounded planes, shut down TV stations, disrupted payments, and cancelled surgeries, was blamed on a faulty update in CrowdStrike's EDR tool Falcon on Microsoft's operating system.
The defect caused Windows computers with Falcon installed to crash without fully loading. Microsoft has revealed that the outage affected 8.5 million devices or 1% of Windows computers worldwide.
Unfortunately, there was no "quick fix" for those affected and rather a more manual and time-consuming process ensued with manually rebooting each individual endpoint. Below are the key takeaways and lessons learned from this outage.
1. Test before you push updates across your organization.
A proper patch management process is not only fun and challenging to say 3 time fast, but also part of a well-rounded information security program. Two important parts of the patching process are 1) testing and 2) rollback procedures (if needed). Testing the patches to see how nicely they play with existing system configurations should help minimize the blast radius of any major releases. And having a solid rollback plan can help remediate any issues that may arise shortly after the releases, hopefully reducing any interferences from the end-user's perspective.
2. Have a backup plan
A business continuity and disaster recovery plan are table stakes now-a-days for any business. Without them you are risking your service level agreements (and potentially your whole business or at least 12% market share like CrowdStrike post outage). Even though this was a technical issue, many affected businesses and industries suffered because they did not have a well thought out or tested BC/DR plan forcing them to scramble to enact analog workarounds. Whether you are a cash-free caf, multinational airline, or hospital network, you need to have a documented and tested BC/DR plan and your employees need to be trained on how to deal with a sudden change in business operations.
3. Know your third-parties and perform security reviews on them annually.
For individuals outside of the Infosec world, CrowdStrike might not have meant much to them on July 18
th. But Microsoft and Windows operating system were likely easier brands to recall from memory. So whether you are a smaller third party that supports enterprise clients or if you are an organization that relies on third-parties to support your operations, be aware of your upstream and downstream service providers. You should periodically review your third-party relationships from a security perspective. Don't be the next headline like the
recent hacking of third-party pathology provider Synnovis that led to many London hospitals canceling all non-emergent appointments.
In conclusion, implement the procedures to help prevent outages like the CrowdStrike outage in July, educate your employees on the BC/DR plan, and know and review your third-party relationships. Ad when all else fails, rely on the internet to provide comic relief like the meme below.
