[News] Mass IT Outage

[Iggle Piggle]

100% agree with this

I missed out the bit where they will also employ a PR firm of consultants on 1k per day each to explain how they should handle this and basically come up with something anyone on here could have suggested for free.

 

[WATFORD zero]

I missed out the bit where they will also employ a PR firm of consultants on 1k per day each to explain how they should handle this and basically come up with something anyone on here could have suggested for free.

Shirley you're not suggesting listening to your own staff for free when you could be spending big money getting a consultantancy to interview your staff and then tell you what they've been trying to tell you for years

 

[dwayne]

To be fair the only way you mitigate this is to test the arse off any updates before they hit the systems. It is obvious in some companies that crowdstrike is the single point of failure ( anti virus software is always robust but throwing thousands of developers at it using their own take on risk is again a multiple point of failure.

when you have a malware software that in effect mimics AI by infecting itself into the very coding level of AWS that is a recipe for disaster and a ball ache for you and your AWS colleagues.

is there an answer, reign in these large software houses in India and the far east and instil in them a mantra of TEST TEST TEST not just GET IT OUT AS CHEAP AS YOU CAN.

you must have seen this first hand using AWS

Problem is that crowdstrike are a 3rd party so we can't test their releases ! As I said in a previous post big companies exposure to 3rd parties is pretty scary these days.

 

[nickbrighton]

Crowdstrike: How China swerved worst of global tech meltdown

Very few organisations will buy software from the US firm, partly thanks to its criticism of Beijing.

www.bbc.co.uk

So it appears that China was largley unaffected by the problem, why? because they, by and large dont use global storeage, software, systems etc, but their own Chinese software , hardware, and cloud storeage

This seems an emminently sensible approach to take, in a similar way that the Suez Canal blockage affected world wide trade for months and woke everyone up to the folly of all the eggs in ons supply chain basket, this outage must make governments think of diversifying critical systems away from the one size fits all approach.

Covid, Suez, and now crowdstrike surely shows that we need to have a more diverse and robust infrastructure, be it physical, or digital, the days of relying on a world wide single entity approach must be over, and individual nations, have their own

 

[Springal]

Crowdstrike: How China swerved worst of global tech meltdown

Very few organisations will buy software from the US firm, partly thanks to its criticism of Beijing.

www.bbc.co.uk

So it appears that China was largley unaffected by the problem, why? because they, by and large dont use global storeage, software, systems etc, but their own Chinese software , hardware, and cloud storeage

er no - it’s because no western security company would sell to a Chinese entity

 

[nickbrighton]

er no - it’s because no western security company would sell to a Chinese entity

er-that doesnt negate my point, nor did i say why they didnt use the tech. The article says it was the Chinese not buying it rather than the company not selling it.The point is that by not using that tech(for whatever reason) they largly avoided the problem. All the planet using one system is, as shown by what happened , inherently very risky. A mistake in one file led to massive issues across the globe, which will probably take weeks to rectify, and its not just the inconvienence of not being able to tweet, or look at porn, its GPs being unable to access systems , banking disrupted, and the like.

Its not too far of a stretch to extrapolate how much worse this could have been had someone deliberatly set out to do it, and hidden the faulty code .

All Im saying is maybe having everything so interconnected isnt quite the great idea it may seem

 

[Springal]

er-that doesnt negate my point, nor did i say why they didnt use the tech. The article says it was the Chinese not buying it rather than the company not selling it.The point is that by not using that tech(for whatever reason) they largly avoided the problem. All the planet using one system is, as shown by what happened , inherently very risky. A mistake in one file led to massive issues across the globe, which will probably take weeks to rectify, and its not just the inconvienence of not being able to tweet, or look at porn, its GPs being unable to access systems , banking disrupted, and the like.

Its not too far of a stretch to extrapolate how much worse this could have been had someone deliberatly set out to do it, and hidden the faulty code .

All Im saying is maybe having everything so interconnected isnt quite the great idea it may seem

Companies like CrowdStrike exist to protect organisations from the Chinese though. It’s a bit of a nonsense argument. They’re not going to buy something that stops most of what cyber threats come out of china

 

[nickbrighton]

Companies like CrowdStrike exist to protect organisations from the Chinese though. It’s a bit of a nonsense argument. They’re not going to buy something that stops most of what cyber threats come out of china

You are completly missing my point,it matters not why China didnt have the system,the fact that they didnt meant they were basically immune to the effects of this particular outage. China was just an example of a nation that wasnt as interconnected and therefore protected from it. The point is that having so much interconnected by a single process is what caused the problem to be so wide spread. Had a similar isssue occured in china, it wouldnt have affected the rest of the planet in the same way
All im saying, is that having global systems so interconnected and reliant on a very few processes is inherent with risk to global operations from a single incident. If it happened once, it can, and likley will happen again. Perhaps next time it wont be so easy to resolve, and maybe, diversifying systems so that while they still work together, but not be reliant on the same software, would mitigate the effects of issues like the one that occurred with crowdstrike

 

[dwayne]

You are completly missing my point,it matters not why China didnt have the system,the fact that they didnt meant they were basically immune to the effects of this particular outage. China was just an example of a nation that wasnt as interconnected and therefore protected from it. The point is that having so much interconnected by a single process is what caused the problem to be so wide spread. Had a similar isssue occured in china, it wouldnt have affected the rest of the planet in the same way
All im saying, is that having global systems so interconnected and reliant on a very few processes is inherent with risk to global operations from a single incident. If it happened once, it can, and likley will happen again. Perhaps next time it wont be so easy to resolve, and maybe, diversifying systems so that while they still work together, but not be reliant on the same software, would mitigate the effects of issues like the one that occurred with crowdstrike

China are like a world of their own though. So when something goes wrong there it's probably the same difference.

 

[Snowflake]

You are completly missing my point,it matters not why China didnt have the system,the fact that they didnt meant they were basically immune to the effects of this particular outage. China was just an example of a nation that wasnt as interconnected and therefore protected from it. The point is that having so much interconnected by a single process is what caused the problem to be so wide spread. Had a similar isssue occured in china, it wouldnt have affected the rest of the planet in the same way
All im saying, is that having global systems so interconnected and reliant on a very few processes is inherent with risk to global operations from a single incident. If it happened once, it can, and likley will happen again. Perhaps next time it wont be so easy to resolve, and maybe, diversifying systems so that while they still work together, but not be reliant on the same software, would mitigate the effects of issues like the one that occurred with crowdstrike

I get your point, but I think you are perhaps trying to explain it in the wrong way by highlighting China.
most apps failed due to a lot of business using specific affected DNS. That was affected by the issue rather than being directly affected. In a similar way to the Facebook outage a year or so ago.

There is a lot more to come out about crowdstrike. it’s not so much a security software as opposed to analytics and control. There was also some odd stock exchange activity and social media chat in advance of the “issue”. May be a coincidence.
I get your general point though about not relying on one system. Although how do you stop that. Create thousands of individual apps?

I personally think it’s more about individual testing and change management. How did one app get updated across the board without the issue being spotted,

I am keeping more cash in my pocket from now on….

 

[Cotton Socks]

er-that doesnt negate my point, nor did i say why they didnt use the tech. The article says it was the Chinese not buying it rather than the company not selling it.The point is that by not using that tech(for whatever reason) they largly avoided the problem. All the planet using one system is, as shown by what happened , inherently very risky. A mistake in one file led to massive issues across the globe, which will probably take weeks to rectify, and its not just the inconvienence of not being able to tweet, or look at porn, its GPs being unable to access systems , banking disrupted, and the like.

Its not too far of a stretch to extrapolate how much worse this could have been had someone deliberatly set out to do it, and hidden the faulty code .

All Im saying is maybe having everything so interconnected isnt quite the great idea it may seem

I've just said on the 'Oven' thread that I don't like the fact my home phone line is now dependant on the internet. I'll always have a gas hob as a back up in a power cut..
We rely so much on the internet for everything and I find that slightly worrying.

 

[Springal]

Cba

 

[Barnet Seagull]

In my view, the problems are due to incredibly poor management of risk. Mission critical systems should always have a level of update lag built in to prevent against security false positives and bad updates.

 

[Springal]

In my view, the problems are due to incredibly poor management of risk. Mission critical systems should always have a level of update lag built in to prevent against security false positives and bad updates.

And an over reliable of critical infrastructure running on Windows

 

[beorhthelm]

You are completly missing my point,it matters not why China didnt have the system,the fact that they didnt meant they were basically immune to the effects of this particular outage. China was just an example of a nation that wasnt as interconnected and therefore protected from it. The point is that having so much interconnected by a single process is what caused the problem to be so wide spread. Had a similar isssue occured in china, it wouldnt have affected the rest of the planet in the same way
All im saying, is that having global systems so interconnected and reliant on a very few processes is inherent with risk to global operations from a single incident. If it happened once, it can, and likley will happen again. Perhaps next time it wont be so easy to resolve, and maybe, diversifying systems so that while they still work together, but not be reliant on the same software, would mitigate the effects of issues like the one that occurred with crowdstrike

your point is flawed because China dont have the choice to use this software, and secondly, Crowdstrike are not dominant having about 15-17% of the market. running multiple versions of security system would double the risk (two lots of updates that could go pear shaped), running parallel systems would be vastly expensive and not even practical (GP surgeries having two PC on their desk?).

 

[Publius Ovidius]

I get your point, but I think you are perhaps trying to explain it in the wrong way by highlighting China.
most apps failed due to a lot of business using specific affected DNS. That was affected by the issue rather than being directly affected. In a similar way to the Facebook outage a year or so ago.

There is a lot more to come out about crowdstrike. it’s not so much a security software as opposed to analytics and control. There was also some odd stock exchange activity and social media chat in advance of the “issue”. May be a coincidence.
I get your general point though about not relying on one system. Although how do you stop that. Create thousands of individual apps?

I personally think it’s more about individual testing and change management. How did one app get updated across the board without the issue being spotted,

I am keeping more cash in my pocket from now on….

Us change mangers always were seen as an interfering bunch of busybodies stopping Trekkie wizzkids putting through their changes on the fly with impunity.

i had great reservations about automated changes that the original coding was presented to CAB ago be approved when only the Indian tekkies understood what was being shown, and then the shysters would subtly change the code before it went into production.

” so why did this automated change fail and take down our critical infrastructures “
” well you signed it off”
” so was there any changes made to the coding before it was implemented”
”no”
” are you sure”
“ yes”
” very sure”
” yes”
” so what is this line of code that wasn’t in the original change plan”
” erm…ask my supervisor”


does that sound familiar dealing with the sub continent?

I was once told on a CAB meeting not to ask a direct question to our Indian tekkies as culturally they couldn‘T lose face therefore will never give you a straight answer. This came from a senior technical manager at INFOSYS!

 

[Springal]

removed

 

[Westdene Seagull]

I sincerely hope that one singular person isn't blamed for this thing.

A f**k up of this magnitude should be looked at in terms of systems and process and testing methodology rather than "Dave, ya f**kin' eejit! Look at the mess you've made."

Disagree - I wouldn't want any front line engineer to take any blame but I would expect an executive to explain why proper UAT wasn't in place and then fall on their sword.

 

[Tom Hark Preston Park]

Disagree - I wouldn't want any front line engineer to take any blame but I would expect an executive to explain why proper UAT wasn't in place and then fall on their sword.

Crowdstrike Test Manager is TOAST - assuming the company doesn't file for bankruptcy first. As for its customers, going to be plenty of thuggish scumbag CEOs currently bullying their techies to 'get it farkin fixed!'

 

[tedebear]

Anyone buying Crowdstrike stock?