BA global IT system failure

[dejavuatbtn]

They need to say Ta Ta to their IT department. It's amazing just how many UK firms have outsourced to India. Personally I think the outsourcing of IT abroad is dangerous from a UK security position.
Having said that, we went to the US two weeks ago with BA and their service was faultless. The only aggro we had was US customs breaking open our suitcases in a random search of baggage. Caused a bit of panic as we weren't able to check contects thoroughly until we got home. I don't have a problem with their levels of security and for once their immigration officer was quite pleasant and chatty!

 

[happypig]

Speaking as someone who works in the, what used to be called , DR, disaster recovery business, something doesn't quite fit here.

BA Have more that 1 backup site, in fact they have a series of failover sites that are tested to death on a regular basis.
All CER sites have N+1 as standard failover and in the case of the bigger sites, we have n+ 4 or 5 ( yes I know that is not a technical term but you know what I am getting at) depending on the generator capacity.
All cer sites are covered by UPS systems that are now linked to systems that monitor flow and report on power outages, spikes, blips or whatever you want to call them.
To suggest everything went down because one centre had a power issue I said nonsense.
To suggest it is cutting costs, is not wholly plausible. Yes they have outsourced to India, the Philippines and really really cheap places, but if it is a power issue, that is covered at th local site, let's say Didcot for example. Whilst the itSystems will be looked after by offshore companies, the power systems and CER maintanance wil all be done, and is, locally at the shop floor.
ALl engineers are highly trained and constantly undergo disaster scenario training so this must have been a hell of a catastrophic failure to take out as much as it did.

Something doesn't add up.

I was saying exactly this to the Sainsbury's man earlier. I work with the links between data centres. All the fibre terminations now have two power supplies. All the data centres I have dealt with are dual-parented on separate exchanges.
Resilience and redundancy are key in these places and I don't believe that a single point of failure could take the whole system down.

As you say, something funny about this...

 

[ozzygull]

I have seen this many times, people who run business take a stable IT environment for granted. They see no problems so assume nothing is done, and any joe blog can do it. They then think why do we pay all these staff, when outsourcing is so cheep. They then want the outsourcing company to do a thing that the local IT person would have done without thinking, but to the outsourcing company this is an extra. After many of these instances they find that their infrastructure is not as stable and the costs of all the extras costs more then having own local staff. They break contract with outsourcing company and hire local IT, until new boss **** arrives and the whole thing happens again.

 

[JCL666]

I really don't get BA. Me and Mrs W needed flights to Oz, one way only, Emirates was £600 odd each, BA was £2k each! Both economy fares, and I understand that Emirates are subsidised but not to that degree. A basket case that can only go bust if it continues that pricing policy.

The main reason is that Emirates can utilise Dubai as a hub so they can offer cheaper flights due to connection frequency.

Also if you look at return flights BA will have a better price than their own one-way fares (still not as competitive as Emirates). This is because they prefer to sell return tickets.

 

[Dave the OAP]

They need to say Ta Ta to their IT department. It's amazing just how many UK firms have outsourced to India. Personally I think the outsourcing of IT abroad is dangerous from a UK security position.
Having said that, we went to the US two weeks ago with BA and their service was faultless. The only aggro we had was US customs breaking open our suitcases in a random search of baggage. Caused a bit of panic as we weren't able to check contects thoroughly until we got home. I don't have a problem with their levels of security and for once their immigration officer was quite pleasant and chatty!

Not just uk firms either....ours has and we are a us company.

 

[hampshirebrightonboy]

Exactly, I don't know anything about the BA infrastructure, but having systems that can cause that much trouble susceptible to a power failure, is either rubbish or irresponsible.

I don't believe the power failure excuse. BA saying systems not yet back to full capacity. And we are into the third day!
Looks to me more like the people trying to sort it out aren't up to it.
Airline systems are hugely complex and it takes many years to build up the knowledge required to develop and support them. Not something you should send out to India. Especially with the added issues around communication.
Senior managers don't seem to value knowledge and experience. Maybe in BAs case they might start doing so. Unfortunately reports suggest they have already got rid of a lot of experienced staff.

 

[Beezer]

Interesting as about to be outsourced to the Indian company involved and the first thing they are doing is cutting the workforce of my department by 50%

 

[Titanic]

Speaking as someone who works in the, what used to be called , DR, disaster recovery business, something doesn't quite fit here.

BA Have more that 1 backup site, in fact they have a series of failover sites that are tested to death on a regular basis.
All CER sites have N+1 as standard failover and in the case of the bigger sites, we have n+ 4 or 5 ( yes I know that is not a technical term but you know what I am getting at) depending on the generator capacity.
All cer sites are covered by UPS systems that are now linked to systems that monitor flow and report on power outages, spikes, blips or whatever you want to call them.
To suggest everything went down because one centre had a power issue I said nonsense.
To suggest it is cutting costs, is not wholly plausible. Yes they have outsourced to India, the Philippines and really really cheap places, but if it is a power issue, that is covered at th local site, let's say Didcot for example. Whilst the itSystems will be looked after by offshore companies, the power systems and CER maintanance wil all be done, and is, locally at the shop floor.
ALl engineers are highly trained and constantly undergo disaster scenario training so this must have been a hell of a catastrophic failure to take out as much as it did.

Something doesn't add up.

Indeed it doesn't add up... to me, it looks more like a catastrophic database failure and probably compounded by some poor initial recovery actions.

 

[Herr Tubthumper]

What's the latest at Heathrow? I'm supposed to be flying there for a work thingy tomorrow. Thinking it might be best to cancel.

 

[Gary1]

I don't believe the power failure excuse. BA saying systems not yet back to full capacity. And we are into the third day!
Looks to me more like the people trying to sort it out aren't up to it.
Airline systems are hugely complex and it takes many years to build up the knowledge required to develop and support them. Not something you should send out to India. Especially with the added issues around communication.
Senior managers don't seem to value knowledge and experience. Maybe in BAs case they might start doing so. Unfortunately reports suggest they have already got rid of a lot of experienced staff.

Senior management at any company don't care for knowledge and experience these days. All they are worried about is how much can they slash costs and pay themselves extra from the savings made. Scum of the earth are senior management.

 

[Tom Hark Preston Park]

I have seen this many times, people who run business take a stable IT environment for granted. They see no problems so assume nothing is done, and any joe blog can do it. They then think why do we pay all these staff, when outsourcing is so cheep. They then want the outsourcing company to do a thing that the local IT person would have done without thinking, but to the outsourcing company this is an extra. After many of these instances they find that their infrastructure is not as stable and the costs of all the extras costs more then having own local staff. They break contract with outsourcing company and hire local IT, until new boss **** arrives and the whole thing happens again.

This, completely. Have worked on a number of major IT projects where there's been outsourcing offshore to India (or to make it sound more palatable to shareholders, bringing Indian staff onshore). Got nothing but praise for the young supersmart Indian graduates that are brought onboard. Most of them are lovely kids being exploited by their parent company. I've made many lasting friendships there. But the middle-managers are generally incompetent, constantly buck-pass and knock back each any every request for change, however small, as out of scope, for which an outrageously inflated cost will be quoted. There's serious wear and tear on the offshore workers who are expected to put in a start time of Indian office hours, followed by a full UK business hours day. So some of these kids are putting in a sixteen hour day, six or seven days a week by the time they're done. Indian public holidays rarely correspond with UK public holidays, so there's double the number of days work lost there. And conference calls are invariably a nightmare of mutual non-comprehension. And after a period of 'Knowledge Transfer' where everything is built into a cast in stone process, there is no leeway whatsoever for lateral thinking by expert in-house techies of many years experience on the company systems to get round an issue. So the company is stuffed basically. It's a false saving, despite a short term effect on the bottom line and a bumper bonus for the guys at the top, who then bail out quick before the shit hits the fan.

 

[WATFORD zero]

My twopenneth -

Before I retired, I was responsible for the IT function of large numbers of acquisitions, often with offshore contracts in place. The only places I have seen offshore work effectively is on run-off business, where no changes are being made and it is simply continuing to run infrastructures and systems to closedown. Also I have had success with outsourcing infrastructure service and support where there is little change.

One other instance was where a small software house had three Russian developers who worked for them for (I think) about 6 or 7 years and had significant knowledge and experience with the software. They wanted to go home and an offshore contract was set up for them to run a development team from Siberia. Over the next few years they increased this team and became the sole development and support team for the software. As i said, I inherited that situation and it ran very effectively until we sold the business (and is probably still running effectively).

Without fail, every other instance of outsourcing on a core system that was being developed and expanded was an unmitigated disaster. Each time, the decision had been made by people without sufficient knowledge of the technologies and people they were dealing with (and were often in senior IT positions).

I would bet the issues were design and application/data based, compounded by lack of knowledge when trying to recover.

 

[SeagullDubai]

 

[Springal]

BA new Fly system has been a disaster since rolled out and made the main platform. Constant problems and outages. Also compounded by their CEO Alex Cruz who is ruthlessly cutting costs on everything involved with BA.

BA with a low cost carrier very soon