Here's an idea: don't give one company kernel level access to the OS of millions of PCs that are necessary to keep whole industries functioning.
I mean, Microsoft themselves regularly shits the bed with updates, even with Defender updates. It's the nature of security, they have to have that kind of access to stop legit malware. That's why these kind of outages happen every few years. This one just got to much coverage from the banking and airline issues. And I'm sure future outages will continue to get similar coverage.
But the Crowdstrike CEO was also at McAfee in 2010 when they shit the bed and shut down millions of XP machines so it seems like he needs a different career...
The problem is the monoculture. We are fucking addicted to convenience and efficiency at all costs.
A diverse ecosystem, if a bit more work to manage, is much more resilient, and wouldn't have been this catastrophe.
Our technology is great, but our processes suck. Standardization. Just in time. These ideas create incredibly fragile organizations. Humanity is so short sighted. We are screwed.
That seems like a pretty hardcore doomer view for an event that didn't really do much in the grand scheme of things. I wouldn't have even known it happened if it wasn't all over the internet, and I work in tech to boot.
Time is money. Training all of the staff needed to manage not just one system in multiple areas, but multiple systems in multiple areas is a horrible idea. Sure for a one off issue like this it would save your bacon. But how often does this really happen?
This happened to me in December 2022/January 2023. Pretty similar problem. Just a regular Windows update caused it. Weirdly it didn't affect everyone (and I'm not on any sort of beta channels). Installing KB5021233 keeps causing BSOD 0xc000021a.
After installing KB5021233, there might be a mismatch between the file versions of hidparse.sys in c:/windows/system32 and c:/windows/system32/drivers (assuming Windows is installed to your C: drive), which might cause signature validation to fail when cleanup occurs.
I’m not sure you can blame the CEO. As much as I despise C-level execs this seems like a failure at a much lower level. Now the question of whether this is a culture failure is a different story because to me that DOES come from the CEO or at least that level.
How difficult would it be for companies to have staged releases or oversee upgrades themselves? I mostly just use Linux but upgrading itself is a relatively painless processing and logging into remote machines to trigger an update is no harder. Why is this something an independent party should be able to do without end user discretion?
So we should have five different cyber security solutions at any given site? That wheezing is the sound of every it person on the planet queuing to swing a sock full of nickles at you.
Crowdstrike was near ubiquitous because it was the best tool out there. And plenty of threats were prevented because of it.
The answer isn't to force every single site to manage everything themselves. It is to increase oversight on ci/CD models
Crowdstrike was near ubiquitous because it was the best tool out there.
I understand the reason for it, but that ubiquity comes with potential dangers, as we saw on Friday. But, no, I don't think the solution is "five different cyber security solutions" at every site. However, different cyber security solutions for different industries might not be such a bad idea. Or, I suppose the root of the problem might be the ubiquity of the OS. Should every PC be running the same jack of all trades but master of none OS?
Again, all you are doing is increasing complexity and punting it to a support staff who are likely unqualified to even know what crowdstrike did.
This was one of those rare cases of capitalism working. There are many options. There was one that was miles ahead of all the others and that dominated.
How did capitalism as in private ownership structures help here?
I want to spin up a separate thread here if that’s okay.
Please give me an example of any EDR solution produced through “public ownership structures”. I don’t think such a thing exists, but I welcome being proven wrong.
Private ownership and investment of capital created Crowdstrike as a profit-seeking venture. It also created MS Defender, SentinelOne, trellix, carbon black, etc. Competition in the marketplace (and there was/is lots of competition) forced these products to be as good as they could, and or self-stratify into pricing tiers. Crowdstrike, being the best (and most expensive) is the most widely-used. Note that not every enterprise requires that level of security, and so while CS is widely used, it is not ubiquitous. This outage could have been significantly worse.
I'm actually pretty excited to go to work on Monday.
We have spent the past few years hardening our security and simplifying our critical systems. One way to doing that was to move a much off Microsoft as possible.
And since I've been on vacation for the past week, I'm either going to walk into a nightmare shit show or everyone is going to be cheering that we are fully operational since we don't depend on Microsoft.
I don't get how you're not sure what you will find if you don't use Windows.
They haven't moved everything off windows, just as much as they could.
I'm not on that team. For all I know, our password manager might be on a random window server. Or some middleware.
In a major company where each team does their own thing and communicate through endpoints, It's impossible to know every configuration.
Was thinking bout this (a week later).
Following up, our partner/affiliate sites were down. Each partner connects to us to submit data, and half were government contracts that were down. It didn't affect our systems, but it affected how we provide services to them.
So it was a mild shit show.
Well that's not so bad, all things considered. Glad it wasn't worse anyway
Crowdstrike shit the bed a few months ago with linux systems
Good luck. I hope that even if it is chaos when you get there, that it's at least popcorn-tier chaos.
C-suite to experts: Are the future risks short term or long term? Specifically longer term than my golden parachute?
This is why "they are the biggest" isn't a good reason to pick a vendor. If all these companies had been using different providers or even different OS, it wouldn't have hit so many systems simultaneously. This is a result of too much consolidation at all levels and one issue with the Microsoft OS monopoly.
The issue, in this case, is more about Crowdstrike's broad usage than Microsoft's. The update that crippled everything was to the Crowdstrike Falcon Sensor software, not to the OS.
Funnily enough, they had a similar issue with an update to the Linux version of the software a few months ago, that didn't have these broad-reaching consequences largely due to the smaller Linux user base. Which means this is starting to look like a pattern, and there are going to need to be some serious process changes at Crowdstrike to prevent things like this in the future.
Anybody's guess if those changes happen or not.
The real surprise to me is not the software, company or OS issues, but rather so many companies just blindly pushing untested updates to their prod environments, this was and will continue to be a risk associated with anything they do trust so implicitly. Feels like the security folks just totally failed Dev 101.
I know in at least some of the BSOD cases that it was an automatic update that wasn't possible to delay. An acquaintance of mine told me that they have previously complained to their IT support about the disruption of auto-updates at inopportune times, but IT said it's out of their hands for security updates because of regulatory requirements.
Or the security folks are doing the best they can with a shoestring budget.
This would have been a fun MIR had my systems be impacted.
Still say not allowing untested updates in a production environment fixes this. I don’t care if it’s a README file, don’t update without testing.
Welcome to the News community!
Rules:
1. Be civil
Attack the argument, not the person. No racism/sexism/bigotry. Good faith argumentation only. This includes accusing another user of being a bot or paid actor. Trolling is uncivil and is grounds for removal and/or a community ban. Do not respond to rule-breaking content; report it and move on.
2. All posts should contain a source (url) that is as reliable and unbiased as possible and must only contain one link.
Obvious biased sources will be removed at the mods’ discretion. Supporting links can be added in comments or posted separately but not to the post body. Sources may be checked for reliability using Wikipedia, MBFC, AdFontes, GroundNews, etc.
3. No bots, spam or self-promotion.
Only approved bots, which follow the guidelines for bots set by the instance, are allowed.
4. Post titles should be the same as the article used as source. Clickbait titles may be removed.
Posts which titles don’t match the source may be removed. If the site changed their headline, we may ask you to update the post title. Clickbait titles use hyperbolic language and do not accurately describe the article content. When necessary, post titles may be edited, clearly marked with [brackets], but may never be used to editorialize or comment on the content.
5. Only recent news is allowed.
Posts must be news from the most recent 30 days.
6. All posts must be news articles.
No opinion pieces, Listicles, editorials, videos, press releases, or celebrity gossip will be allowed. All posts will be judged on a case-by-case basis. Mods may use discretion to pre-approve videos or press releases from highly credible sources that provide unique, newsworthy content not available or possible in another format.
7. No duplicate posts.
If an article has already been posted, it will be removed. Different articles reporting on the same subject are permitted. If the post that matches your post is very old, we refer you to rule 5.
8. Misinformation is prohibited.
Misinformation / propaganda is strictly prohibited. Any comment or post containing or linking to misinformation will be removed. If you feel that your post has been removed in error, credible sources must be provided.
9. No link shorteners or news aggregators.
All posts must link to original article sources. You may include archival links in the post description. News aggregators such as Yahoo, Google, Hacker News, etc. should be avoided in favor of the original source link. Newswire services such as AP, Reuters, or AFP, are frequently republished and may be shared from other credible sources.
10. Don't copy entire article in your post body
For copyright reasons, you are not allowed to copy an entire article into your post body. This is an instance wide rule, that is strictly enforced in this community.