NSC's unscheduled downtime

[Bozza]

Apologies for the unscheduled and unannounced downtime. The Blue Square data centre (BlueSquare) which hosts NSC's dedicated server suffered a power outage. The UPS power supplies provided by UPS power systems - Uninterruptible Power Supplies and Dieselec Generators :: Power at our fingertips didn't do what they should meaning our server lost power.

Going down in such an ungraceful manner, when the database is mid-update, caused the NSC DB to be rendered corrupt.

Fortunately we've been able to repair the affected tables with the exception of one - PostIndex - which records the poster and all words (of 4 or more letters) from each post so when we use the search facility we get some meaningful results. With a repair having taken 14 hours and still not finished, we took the decision to kill it off. This means search will NOT work on any of the 2.3m or so posts that make up NSC right now. I'll rebuild it in batches over-night but it will take some time.

Thanks, as ever, to Nail-z for his help in tinkering under NSC's hood to restore normal service. I owe him (yet another) cider outside the round system.

 

[hitony]

Apologies for the unscheduled and unannounced downtime. The Blue Square data centre (BlueSquare) which hosts NSC's dedicated server suffered a power outage. The UPS power supplies provided by UPS power systems - Uninterruptible Power Supplies and Dieselec Generators :: Power at our fingertips didn't do what they should meaning our server lost power.

Going down in such an ungraceful manner, when the database is mid-update, caused the NSC DB to be rendered corrupt.

Fortunately we've been able to repair the affected tables with the exception of one - PostIndex - which records the poster and all words (of 4 or more letters) from each post so when we use the search facility we get some meaningful results. With a repair having taken 14 hours and still not finished, we took the decision to kill it off. This means search will NOT work on any of the 2.3m or so posts that make up NSC right now. I'll rebuild it in batches over-night but it will take some time.

Thanks, as ever, to Nail-z for his help in tinkering under NSC's hood to restore normal service. I owe him (yet another) cider outside the round system.

Well done for getting it back online

 

[Cian]

DB corruptions are a whore to recover (as I have to do at least once a week, but on a far smaller scale than NSC's 2M+ records in one table...), congrats on that one, was expecting another "Year Zero" for here!

 

[severnside gull]

well done - nice to have the board back

 

[Hiney]

Hoo-f***ing-Ray

 

[footychick]

Phew, glad it's back, it was almost like losing an arm, not that I know what that's like

 

[Bozza]

I've kicked off the re-build of the search index and it's going pretty quickly c3250 records per minute. Should mean c1,600,000 will be done by the time I get up.

 

[nail-Z]

I've kicked off the re-build of the search index and it's going pretty quickly c3250 records per minute. Should mean c1,600,000 will be done by the time I get up.

Then they'll just be another 26,000,000 to go.

 

[Fungus]

Thanks, I've missed it!

 

[steward 433]

Well done Bozza nice to have you back

 

[beorhthelm]

sterling work i reckon to get it back in the state at which it went.

i too have found out first hand that datacentre's claims surrounding 100% power are worth crap all, at a major datacentre in docklands we had 3 outages in about as many years and there was another on the floor that just missed us.

 

[Spearritt of the Albion]

DB corruptions are a whore to recover (as I have to do at least once a week, but on a far smaller scale than NSC's 2M+ records in one table...), congrats on that one, was expecting another "Year Zero" for here!

Use DB2 on a mainframe. Much more reliable
Can deal with tables with 1bn rows and never needs recovering! Servers are not a patch on the power of a mainframe. The dinosaur lives!

If you ever have to recover data in DB2 it is usually through user error. "Oops I deleted all the rows from my table and I really wanted to select them, can you recover the 500m rows for me today please?"!

 

[bright1064]

Good work Bozza, top man! It's nice to be able to wake up to NSC again

 

[desprateseagull]

good to have you back, NSC'ers. i see there are a couple of yahoo groups that have been setup as an alternative, though i wonder if even yahoo can handle us!

 

[Tom Hark Preston Park]

Well done to Bozza and all his little elves for getting NSC back online

 

[Simster]

You *are* CLEVER, Darren.

Thanks.

 

[Bozza]

Nail-z's the clever one. He beats me at pool generally too.

 

[Cian]

Use DB2 on a mainframe. Much more reliable
Can deal with tables with 1bn rows and never needs recovering! Servers are not a patch on the power of a mainframe. The dinosaur lives!

If you ever have to recover data in DB2 it is usually through user error. "Oops I deleted all the rows from my table and I really wanted to select them, can you recover the 500m rows for me today please?"!

I'll recommend that to the next doctors surgery that has problems, are there any models of mainframe that'll fit on a GPs desk? Mini-Cray would be great, they like shiney things