Daily Archives: 2024-07-19

CrowdStrike issue is causing massive computer outages worldwide

Posted on 2024-07-19 by pappp

Source: OSNews

Article note: Oh man, again? Bolt-on third party "security" company, of the appeals to Csuite-types for outsourcing liability style (run by a former McAfeee exec, the hustle never changes for these people), has a kernel driver on all their WinNT clients to enable file-scanning and monitoring (and remote shell and...). Apparently their Linux client is also failing but in a slightly less absurd way. This time (as opposed to when it was Solarwinds. Or Okta. Or...), instead of getting their infrastructure hacked in a multilevel supply-chain attack, they're apparently just grossly incompetent and pushed an automated update to the scanner definition file which breaks the parser - which is running as privileged code - killing the kernel module and blue-screening then bootlooping the system. 'Somehow' they didn't catch this in testing before deploying to half of the global enterprise market because their test setup is probably to spin a reference VM, apply the update, see that it applied, then automatically wipe the whole thing, because more than that would be expensive. And all their customers, because they're primarily a compliance tool, have automatic updates turned on so they don't have to explain their update test/hold/deploy scheme to regulators, so everyone, everywhere, all at once got this update. I've been hearing years of "Maximize homogeneity" "Continuous, Silent, Automatic update everything" and "Outsource your monitoring and Auth to security professionals" as best practice and uh... how's that goin? Minor global catastrophe? Again? Yea. Presumably ZScaler, their largest competitor, will have a good time until they inevitably do the same kind of bullshit because the whole product category is mostly a scam. Glad I'm not working in IT this week.

Well, this sure is something to wake up to: a massive worldwide outage of computer systems due to a problem with CrowdStrike software. Payment systems, airlines, hospitals, governments, TV stations – pretty much anything or anyone using computers could be dealing with bluescreens, bootloops, and similar issues today. Open-heart surgeries had to be stopped mid-surgery, planes can’t take off, people can’t board trains, shoppers can pay for their groceries, and much, much more, all over the world.

The problem is caused by CrowdStrike, a sort-of enterprise AV/monitoring software that uses a Windows NT kernel driver to monitor everything people do on corporate machines and logs it for… Security purposes, I guess? I’ve never worked in a corporate setting so I have no experience with software like this. From what I hear, software like this is deeply loathed by workers the world over, as it gets in the way and slows systems down. And, as can happen with a kernel driver, a bug can cause massive worldwide outages which is costing people billions in damages and may even have killed people.

There is a workaround, posted by CrowdStrike:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it. 
  4. Boot the host normally. 

This is a solution for individually fixing affected machines, but I’ve seen responses like “great, how do I apply this to 70k endpoints?”, indicating that this may not be a practical solution for many affected customers. Then there’s the issue that this may require a BitLocker password, which not everyone has on hand either. To add insult to injury, CrowdStrike’s advisory about the issue is locked behind a login wall. A shitshow all around.

Do note that while the focus is on Windows, Linux machines can run CrowdStrike software too, and I’ve heard from Linux kernel engineers who happen to also administer large numbers of Linux servers that they’re seeing a huge spike in Linux kernel panics… Caused by CrowdStrike, which is installed on a lot more Linux servers than you might think. So while Windows is currently the focus of the story, the problems are far more widespread than just Windows.

I’m sure we’re going to see some major consequences here, and my – misplaced, I’m sure – is that this will make people think twice about one, using these invasive anti-worker monitoring tools, and two, employing kernel drivers for this nonsense.

Posted in News | Leave a comment