Peppermint OS

General => General Discussion => Topic started by: josephd on March 01, 2020, 11:11:03 pm

Title: A note on Sunday's server downtime
Post by: josephd on March 01, 2020, 11:11:03 pm
Hi everyone,

During multiple attempts to renew the security certificate the Peppermint web server crashed. This resulted in the need to restore our from a backups made around 4 UTC yesterday. Any posts, DMs in have been lost as a result.

We're currently running the web server on Ubuntu 14.04 which as you know reached end of life last year. Three separate attempts to perform in place upgrades to 16.04 resulted in MySQL errors. Although ironically updated the encryption certificate for about 15 minutes until the server crashed near the end of the process.

I will need to research this issue a little further before making another attempt. In the meantime please be aware that traffic on this forum and the Peppermint OS website may not be encrypted or has weak encryption. That basically means your surfing of this website could be intercepted by third parties. Particularly if you're on a open wifi network.

I apologize for the lost posts and DMs made yesterday.

Thank you for your understanding as I attempt to resolve these issues.
Title: Re: A note on Sunday's server downtime
Post by: stevesveryown on March 02, 2020, 06:01:38 am
Joseph,

Thanks for the update.  It is unfortunate that things like this keep happening.  On my YouTube Channel I am being asked if this is the end for Peppermint.  I keep telling them to hold on and give the team a chance to recover from their loss.  Also some big proponents of Peppermint in the YouTube community have already jumped ship.  I will be here and patiently await to see what happens.  If there is anything you would like passed on to the YouTube community please let me know.
Title: Re: A note on Sunday's server downtime
Post by: The Omen on March 02, 2020, 09:40:51 am
Hi folks,

I, for one, am going to stick with Peppermint as I realize that anytime there is a "death in the family", that things are never going to be quite the same ever again and there are always adjustment problems to deal with.

This is just one of those adjustment problems is all.

Mr. Greaves would want us all to stick it out and be supportive.

This is not doomsday or the end of Peppermint, just an adjustment for all of us to "keep calm and carry on" while everyone is adjusting to the new situations as they arise and hopefully learning that the "keys to the kingdom" need to be held not by just one person but also entrusted to others so that smoother transitions can take place in future.

Warm Regards to all, especially those that are now taking the lead in rectifying these otherwise minor teething problems as we all need to grow and accept the situation as it is.

The Omen



Title: Re: A note on Sunday's server downtime
Post by: spence on March 02, 2020, 10:50:33 am
Also some big proponents of Peppermint in the YouTube community have already jumped ship.  I will be here and patiently await to see what happens.

Might you alert us to who is jumping ship so the rest of #teampeppermint may make contact @stevesveryown? 

 ;)
Title: Re: A note on Sunday's server downtime
Post by: josephd on March 02, 2020, 11:12:36 am
Thanks for your continued support @stevesveryown. Mark was very good at what he did in this project and no one person can replace such a dedicated project lead. We didn't have a second person on the website, I had the opportunity a year ago and passed. Its a personal regret of mine to not have worked with Mark on the website and help him along when I could have. He appreciated my early enthusiasm but I got busy with work and life. You know how it is.

Anyway all that aside, Mark ran a very tight ship and learning his processes will take some time. If you do see the website go down in the meantime know that its probably me on the other end trying to resolve certificate issue :)
Title: Re: A note on Sunday's server downtime
Post by: stevesveryown on March 02, 2020, 12:57:07 pm
Thanks for your continued support @stevesveryown. Mark was very good at what he did in this project and no one person can replace such a dedicated project lead. We didn't have a second person on the website, I had the opportunity a year ago and passed. Its a personal regret of mine to not have worked with Mark on the website and help him along when I could have. He appreciated my early enthusiasm but I got busy with work and life. You know how it is.

Anyway all that aside, Mark ran a very tight ship and learning his processes will take some time. If you do see the website go down in the meantime know that its probably me on the other end trying to resolve certificate issue :)

With the support and encouragement from Mark over the last 2 years how can I not continue support for this project?
Title: Re: A note on Sunday's server downtime
Post by: stevesveryown on March 02, 2020, 12:59:12 pm
Also some big proponents of Peppermint in the YouTube community have already jumped ship.  I will be here and patiently await to see what happens.

Might you alert us to who is jumping ship so the rest of #teampeppermint may make contact @stevesveryown? 

 ;)

Sure!
Title: Re: A note on Sunday's server downtime
Post by: rayzer on March 02, 2020, 01:56:32 pm
i expected all these issues to arise as ive seen it happen on another forum i used to frequent. the owner passed and he didnt pass any of the forum portal or hosting logins onto any of his staff as he was kinda paranoid about the security of it (long story). the forum simply just went offline as time went on and nobody could do anything about it. Thankfully the staff here are on top of things even if it is a difficult time.
Title: Re: A note on Sunday's server downtime
Post by: spence on March 02, 2020, 03:19:52 pm
I seem to recollect a discussion about that @rayzer, mind refreshing my memory about which distro that was?
Title: Re: A note on Sunday's server downtime
Post by: rayzer on March 03, 2020, 10:27:25 am
was a music forum spence, nothing to do with linux
Title: Re: A note on Sunday's server downtime [solved]
Post by: josephd on March 03, 2020, 10:10:16 pm
The security certificate problem has been solved. Thank you everyone for your patience while I worked on this issue.

(https://www.linuxbookpro.com/wp-content/uploads/2020/03/phew-allan-rickman-galaxy-quest.gif)
Title: Re: A note on Sunday's server downtime [solved]
Post by: VinDSL on March 04, 2020, 05:55:17 am
The security certificate problem has been solved. Thank you everyone for your patience while I worked on this issue.

We must have been on the site at the same time. The cert was still borked at 10:00 PM, so I went about doing some daily maintenance.

About 10:15 I started getting 404s. I figured you were working on it.

Then, around 10:30 the connection(s) returned, and the SSL was working.

I was gonna shoot you a congrats via Proton, but we've been-there-done-it before, just to have it crash a few minutes later. I thought... why tempt fate!?  LoL

Anyway, good job, Detective Joe. Onward and upward    8)
Title: Re: A note on Sunday's server downtime [solved]
Post by: stevesveryown on March 04, 2020, 06:30:48 am
The security certificate problem has been solved. Thank you everyone for your patience while I worked on this issue.

(https://www.linuxbookpro.com/wp-content/uploads/2020/03/phew-allan-rickman-galaxy-quest.gif)

Awesome, Joseph!
Title: Re: A note on Sunday's server downtime
Post by: TheAkantor on March 04, 2020, 06:42:28 am
Nice work Josephd , next time you go drink a beer put it in my account  8)
Title: Re: A note on Sunday's server downtime
Post by: jlschwartz on March 04, 2020, 06:54:38 am
Thanks Josephd.  :)
Title: Re: A note on Sunday's server downtime
Post by: josephd on March 04, 2020, 08:26:53 am
Thanks all,
Yeah, I would love to put up a maintenance notice but what's the point when the server goes down and no one is there to read it.

VinDSL, that few minutes where it was down was a result of some sort of Apache mismatch for the certificate during those few minutes. I reran the process and by the time you returned it was fixed. I'll be honest I screamed in joy. My wife must have thought I was watching Super Tuesday results. But alas no. Just Peppermint Tuesday  :D
Title: Re: A note on Sunday's server downtime [solved]
Post by: spence on March 04, 2020, 10:42:35 am
(https://www.linuxbookpro.com/wp-content/uploads/2020/03/phew-allan-rickman-galaxy-quest.gif)

Thank you so much for your diligence and patience upon the task @josephd! I'll even loan you the color purple I usually reserve for @PCNetSpec, for I'm sure he's smiling out there somewhere, now that you've helped #teampeppermint leap the first test hurdle he laid for us.

perhaps now we should make a few posts on our social media outlets?  

Title: Re: A note on Sunday's server downtime
Post by: josephd on March 04, 2020, 03:21:41 pm
Marcus has the social credentials I believe, Possibly Zeb. If you want to take on social media in any form I don't think you'll get any objections.

Just a quick note, I'm going to look into upgrading us to PHP 7.4.3 this evening so if you notice any downtime around 7PM-11PM Pacific that's me.