[News] Mass IT Outage

[ozzygull]

As some one who works in IT, when I see these outages on the news, the sense of relief felt knowing I had nothing to do with it, is massive. I just imagine the fallout within that company trying to find out who is ultimately to blame for the catastrophic error. I used to support software on for mobile telecoms operators in the early 2000's. the stress of upgrading their systems which one mistake could bring down their networks and affect millions of customers was massive. even though it had been tested to death and the implementation and documented role back plan was in place, I used to still feel sick until everything was back online and tested working.

 

[dejavuatbtn]

So - that's why I had to pay cash on the ferry this morning! Wondered why the ticket office staff were looking harassed.

Yes, part of their disaster recovery planning was to teach staff about this thing called cash.

 

[BBassic]

Imagine being the person who signed that one off for release. Knowing that you effectively shut down large chunks of the Western world, shutting down airports, access to healthcare, financial services.

Wow.

I sincerely hope that one singular person isn't blamed for this thing.

A f**k up of this magnitude should be looked at in terms of systems and process and testing methodology rather than "Dave, ya f**kin' eejit! Look at the mess you've made."

 

[Dick Head]

Installing Linux was the second best choice I ever made in my life.

 

[raymondo]

As some one who works in IT, when I see these outages on the news, the sense of relief felt knowing I had nothing to do with it, is massive. I just imagine the fallout within that company trying to find out who is ultimately to blame for the catastrophic error. I used to support software on for mobile telecoms operators in the early 2000's. the stress of upgrading their systems which one mistake could bring down their networks and affect millions of customers was massive. even though it had been tested to death and the implementation and documented role back plan was in place, I used to still feel sick until everything was back online and tested working.

And one can be as thorough as possible, but still not think of every possible scenario.
Yeah, ive felt your stress , different business environment but still...

 

[raymondo]

Yes, part of their disaster recovery planning was to teach staff about this thing called cash.

GP surgeries hand writing prescriptions with 'paper and pen ' ... whatever they are, pah

 

[Publius Ovidius]

This, totally agree. WTF is DR?

Disaster recovery. BC = business continuity.

remember the commercial union bombing in London back in the 70’s? It all came out of that where it became obvious that if companies had some sort of scheme whereby if a disaster occurs, they can relocate to somewhere else, restore the pre disaster backups in their systems, and carry on.

IRA bombs were designed to disrupt business, and the Bank of England made all banks and finance houses get DR contracts and test their systems once a year….that was my job!

 

[dwayne]

Disaster recovery. BC = business continuity.

remember the commercial union bombing in London back in the 70’s? It all came out of that where it became obvious that if companies had some sort of scheme whereby if a disaster occurs, they can relocate to somewhere else, restore the pre disaster backups in their systems, and carry on.

IRA bombs were designed to disrupt business, and the Bank of England made all banks and finance houses get DR contracts and test their systems once a year….that was my job!

Dave you still haven't told me how you mitigate the risk with something like this

Crowdstrike just pushes the updates out willy nilly. Unpicking it in AWS has been complex for some systems I deal with. Has taken a rebuild, not fast. Also quite bespoke not something that can be sorted with additional AZs or regions.

 

[US Seagull]

Interestingly, Crowdstrike CEO George Kurtz was once the CTO of McAfee. In fact, he was their CTO when this happened:

Defective McAfee update causes worldwide meltdown of XP PCs

Oops, they did it again. Early this morning, McAfee released an update to its antivirus definitions for corporate customers that mistakenly deleted a crucial Windows XP file, sending systems into a reboot loop and requiring tedious manual repairs. It's not the first strike for the company...

www.zdnet.com

Call me old fashioned, but I think you only get to cause massive, global PC outages of critical infrastructure once.

 

[Springal]

Crowdstrike just pushes the updates out willy nilly. Unpicking it in AWS has been complex for some systems I deal with.

that’s kind of why they’ve been so efficient for so long from my experience. Sadly with any software there is this kind of risk

 

[sir_gullahad]

This is getting serious, ITVX isn't letting me watch Tour de France highlights!

Sack them all.

 

[WATFORD zero]

from US, in interview with Crowdstrike CEO he was asked why all the systems dont have backups. like thats his problem too.

I saw that. Looks a bit shell shocked. Don't envy his job today.

And that's why he gets paid the big bucks

I retired from IT 15 years ago, but one thing always rang true 'a junior person cannot make a massive cockup, you need to be senior to do that'. It's the truth but not always reflected in the outcome

 

[Publius Ovidius]

Dave you still haven't told me how you mitigate the risk with something like this

Crowdstrike just pushes the updates out willy nilly. Unpicking it in AWS has been complex for some systems I deal with. Has taken a rebuild, not fast. Also quite bespoke not something that can be sorted with additional AZs or regions.

To be fair the only way you mitigate this is to test the arse off any updates before they hit the systems. It is obvious in some companies that crowdstrike is the single point of failure ( anti virus software is always robust but throwing thousands of developers at it using their own take on risk is again a multiple point of failure.

when you have a malware software that in effect mimics AI by infecting itself into the very coding level of AWS that is a recipe for disaster and a ball ache for you and your AWS colleagues.

is there an answer, reign in these large software houses in India and the far east and instil in them a mantra of TEST TEST TEST not just GET IT OUT AS CHEAP AS YOU CAN.

you must have seen this first hand using AWS

 

[Tom Hark Preston Park]

Installing Linux was the second best choice I ever made in my life.

Not about, say, the twentieth or fiftieth best choice you ever made in your life? Bit sad

 

[Rogero]

I have just been in a shop and they cannot take any card transactions. .

 

[Scappa]

As some one who works in IT, when I see these outages on the news, the sense of relief felt knowing I had nothing to do with it, is massive. I just imagine the fallout within that company trying to find out who is ultimately to blame for the catastrophic error. I used to support software on for mobile telecoms operators in the early 2000's. the stress of upgrading their systems which one mistake could bring down their networks and affect millions of customers was massive. even though it had been tested to death and the implementation and documented role back plan was in place, I used to still feel sick until everything was back online and tested working.

Transpose that to a food safety setting - and the potential for something really dreadful to happen - and that resonates with the recent ecoli outbreak: sorrow and sympathy for those who were seriously effected, but a selfish, abject kind of relief that me or my team were in no way implicated

 

[Iggle Piggle]

I sincerely hope that one singular person isn't blamed for this thing.

A f**k up of this magnitude should be looked at in terms of systems and process and testing methodology rather than "Dave, ya f**kin' eejit! Look at the mess you've made."

As someone that has worked Corporate for 30 years in IT, that is exactly what will happen. An unpopular junior will take the fall along with a senior manager who had nothing to do with it who they wanted to get rid of anyway.

Some clueless bell will have a look at the processes and no doubt make things worse whilst claiming they've improved things.

The senior management will congratulate themselves on a job well done whilst pocketing massive bonuses.

Don't you just love capitalism?

 

[Publius Ovidius]

As someone that has worked Corporate for 30 years in IT, that is exactly what will happen. An unpopular junior will take the fall along with a senior manager who had nothing to do with it who they wanted to get rid of anyway.

Some clueless bell will have a look at the processes and no doubt make things worse whilst claiming they've improved things.

The senior management will congratulate themselves on a job well done whilst pocketing massive bonuses.

Don't you just love capitalism?

100% agree with this

 

[beorhthelm]

To be fair the only way you mitigate this is to test the arse off any updates before they hit the systems.

there's the problem, looks like people dont test updates and just allow them to roll out upon issue. having some passing involvement in security policy i wager its because policy, usually with too little IT input, dictates this to ensure arse covering. i remember when sysadmins would have every release go through their test systems first before even hitting dev servers.

 

[jcdenton08]

What’s the latest on the ground? Everything fixed?