So that sucked.

Post by **FishPants** » Tue Nov 15, 2016 9:36 pm

So here's some learning points:

1. Rip was right. So very right. I feel shame.
2. Database won't convert, I'll open a ticket.. We stay on the old version.
3. First restore took about 30 hours.. Then I screwed it up again (whoops, wrong script).
4. I made some tweaks (!) to the config, and the next restore took exactly 15 minutes.
5. No that's not a joke.. 15 minutes.

Please report any funkiness. I'm going to go feel shame with a glass of red wine.

Bad Demographic · Post by **Bad Demographic** » Tue Nov 15, 2016 9:46 pm

No funkiness, but I had to look in the Books Read thread to figure out the correct spelling of my username - and I had to guess at my password, too.

(I've written them down now - for the future)

Zaxxon · Post by **Zaxxon** » Tue Nov 15, 2016 9:51 pm

No shame. We're back, we appear to be whole, and you live to fight another day. Lessons learned, and all that.

Post by **FishPants** » Tue Nov 15, 2016 9:53 pm

On the plus side the entire database is in RAM now, and the "write" threads is up to 16 -- so that means restores are WAY faster.

At least now I can reload the backup to another DB and test over there, even on a clean install the phpbb3 script to upgrade the database has a weird failure.. I may need to update our MySQL version to a more modern version. That's for another day.

gbasden · Post by **gbasden** » Tue Nov 15, 2016 9:54 pm

No shame at all! Thanks so much for all the effort you put in!

Kraken · Post by **Kraken** » Tue Nov 15, 2016 9:55 pm

Yay, normality is restored, and 10% better than it was before.

Smoove_B · Post by **Smoove_B** » Tue Nov 15, 2016 10:25 pm

Have you patented the methodology that you used to get from 30 hours to 15 minutes, because that seems astoundingly more efficient?

Thanks for all your hard work. Have a drink on me!

Exodor · Post by **Exodor** » Tue Nov 15, 2016 10:29 pm

gbasden wrote:No shame at all! Thanks so much for all the effort you put in!

Speak for yourselves.

Thanks for all your hard work.

Post by **FishPants** » Tue Nov 15, 2016 10:31 pm

LawBeefaroni · Post by **LawBeefaroni** » Tue Nov 15, 2016 10:37 pm

Enjoy the well deserved wine.

cheeba · Post by **cheeba** » Tue Nov 15, 2016 10:39 pm

As always, thanks for all you guys do with the site! I especially appreciate the updated page and not just coming to a 404 or something.

Carpet_pissr · Post by **Carpet_pissr** » Tue Nov 15, 2016 10:40 pm

Smoove_B wrote:Have you patented the methodology that you used to get from 30 hours to 15 minutes, because that seems astoundingly more efficient?

Unpossible. Pied Piper already holds patents on the middle out compression solution that saved us all from internet purgatory.

Daehawk · Post by **Daehawk** » Tue Nov 15, 2016 10:46 pm

Bad Demographic wrote:No funkiness, but I had to look in the Books Read thread to figure out the correct spelling of my username - and I had to guess at my password, too.
(I've written them down now - for the future)

Strange as Im still logged in.

Fish thank you so much for it all. Sorry it went badly. umm better luck next time?

Post by **FishPants** » Tue Nov 15, 2016 10:51 pm

Ehh it wasn't that stressful to be honest.. I had solid backups, it didn't involve a lot of my time other than check it once in awhile and update the HTML file for status.. I'm just really pissed off that the performance tweaks I found were something I could have used 10 years ago -- and it was through a forum post that referred to an academic that posted some crazy whitepaper, and a link to a TL;DR version of it with 5 settings for InnoDB tables. I set those (based on RAM) and viola -- fast restore. Crap default values, and obscure articles on how to tune to something reasonable -- frustration!

Anyways now I need to figure out how to create a mirror site, restore the DB there and test the upgrade script there.. I'm wondering if people hitting the site while it was upgrading was the problem (I didn't turn apache off, so I could serve up the status page..).

Daehawk · Post by **Daehawk** » Tue Nov 15, 2016 11:04 pm

I only checked every 5 minutes so it wasn't me.

Post by **FishPants** » Tue Nov 15, 2016 11:06 pm

Daehawk wrote:I only checked every 5 minutes so it wasn't me.

dbt1949 · Post by **dbt1949** » Tue Nov 15, 2016 11:14 pm

Without OO I don't think I need the internet. Just boring Trump crap.

Post by **FishPants** » Tue Nov 15, 2016 11:16 pm

You know we have a core group of people that love this place when Smoove is logged in before I am.. and I'm the guy bringing the site back online.. "Oh look the restore is finished.. let me change this config file.. ok done.. ok lets restart the web server and login.. hey there's smoove!".

Not to mention another 67 "guest users" (we seemed to dump most cached login information on the restore). Sickos! The lot of ya!

Isgrimnur · Post by **Isgrimnur** » Tue Nov 15, 2016 11:24 pm

What do you expect me to do all day? Work?!

Blackhawk · Post by **Blackhawk** » Tue Nov 15, 2016 11:51 pm

cheeba wrote:As always, thanks for all you guys do with the site! I especially appreciate the updated page and not just coming to a 404 or something.

Or being told you're banned for being a spammer!

Zarathud · Post by **Zarathud** » Wed Nov 16, 2016 12:24 am

Very happy to see the harmony of OO restored to the Universe.

A forced break was good, but there were serious withdrawal symptoms.

Daehawk · Post by **Daehawk** » Wed Nov 16, 2016 1:00 am

Im feeling much better now.

Archinerd · Post by **Archinerd** » Wed Nov 16, 2016 1:38 am

Hooray, tomorrow is going to feel like a vacation day at work now!

Moat_Man · Post by **Moat_Man** » Wed Nov 16, 2016 3:06 am

What happened to BlackPearl?

Post by **FishPants** » Wed Nov 16, 2016 3:06 am

Some interesting database issues with mods. All fixed, no more touching for tonight/this morning.

Moat_Man · Post by **Moat_Man** » Wed Nov 16, 2016 3:14 am

FishPants wrote:Some interesting database issues with mods. All fixed, no more touching for tonight/this morning.

I'm back in Black. Time to hit sack.

Daehawk · Post by **Daehawk** » Wed Nov 16, 2016 3:25 am

Hmm came back just now and it reset me to default silver style. Had to reset it to prosilver 2.

Post by **FishPants** » Wed Nov 16, 2016 3:29 am

Yeah in short what I did was take the current DB, restore it to another DB (under a fictitious URL/domain).. I accessed it (by editing my hosts file) and ok great I have a mirror.. I read documentation on how to do the upgrade that's been failing (more forum posts).. Hey it says disable all styles and mods.. so I do.. then I realize I am using the same production config so I'm modifying this DB..

Whoops..

it's late.. my bad. restored some mod tables, seems to have fixed it.

Jeff V · Post by **Jeff V** » Wed Nov 16, 2016 10:41 am

It seems that every time there's an extended OOutage, someone goes missing and Daehawk looks well-fed. Not that it's proof of anything, just an observation.

Post by **ImLawBoy** » Wed Nov 16, 2016 11:16 am

FishPants wrote:All fixed, no more touching for tonight/this morning.

You're not the boss of me.

Rip · Post by **Rip** » Wed Nov 16, 2016 11:26 am

Smoove_B · Post by **Smoove_B** » Wed Nov 16, 2016 11:45 am

FishPants wrote:You know we have a core group of people that love this place when Smoove is logged in before I am.. and I'm the guy bringing the site back online.. "Oh look the restore is finished.. let me change this config file.. ok done.. ok lets restart the web server and login.. hey there's smoove!".

The real story here is who told me that it was up and running.

Rip · Post by **Rip** » Wed Nov 16, 2016 12:56 pm

Putin?

Post by **FishPants** » Wed Nov 16, 2016 1:45 pm

So the good news is that I was able to convert the database (after I stopped screwing up this one) without an error. Apparently having you clowns hammering the system while I was converting it was causing problems (I would have thought the conversion script would frigging lock the DB.. I guess not).

So after the holidays I will try again, but will have to disable the web server (so no status page). The good news is that it only took about an hour to do (and if it gets hosed, recovery is 30 mins). That sure beats a couple days of restore time!

Zaxxon · Post by **Zaxxon** » Wed Nov 16, 2016 2:58 pm

Excellent! Are there any user-facing benefits with 3.1?

Post by **FishPants** » Wed Nov 16, 2016 3:01 pm

Zaxxon wrote:Excellent! Are there any user-facing benefits with 3.1?

Take a look at our test instance

I really like it, but it's different -- the masses may revolt!

Rip · Post by **Rip** » Wed Nov 16, 2016 3:04 pm

The lack of something with a black background is certainly revolt worthy.

Has black done something wrong?

Blackhawk · Post by **Blackhawk** » Wed Nov 16, 2016 3:36 pm

No. Badhawk.

DD* · Post by DD* » Wed Nov 16, 2016 4:22 pm

Rip wrote:The lack of something with a black background is certainly revolt worthy.

Has black done something wrong?

stessier · Post by **stessier** » Wed Nov 16, 2016 4:54 pm

Thank you for the frequent updates during the downtime. It made the withdrawals a lot easier to handle knowing someone was keeping tabs on the situation.