Posted 5 years ago
·
Author
Well, after 2.5 weeks of downtime, we are finally back. Really though, being away from all of you was extremely painful. All of our staff have been stressed as hell trying to bring our community back up, but we finally did it. Fuck yeah!
The Bad News:
* We lost 2 weeks worth of site updates. (all updates we made to our site between December 15th - Jan 1st have been lost)
* We lost 1 week of data. (all private messages sent, orders purchased, and accounts registered between December 24th - Jan 1st have been lost)
The Good News:
Our staff is busy rapidly re-engineering the lost updates made to our site. Things might look a little wonky on our site for the next week or two, but you can expect everything to go back to normal rather quickly. Lost orders will be re-delivered. We still have the mafia account names, payment details, and number of products each person purchased between December 24th - Jan 1st. All 133 orders purchased during that time will be re-delivered. We will be contacting each and every one of the 133 customers who purchased products between those dates to make sure you receive the products you paid for, delivered with the full 110% mafia service our customers have come to know and love. We are deeply sorry for any inconvenience this delay may have caused you.
Technical details on why our site was down for almost 3 weeks:
There was an accident at our main datacenter. The datacenter's networking infrastructure was badly damaged on Jan 1st. Our primary server had to be shipped to our secondary datacenter. The primary server arrived at the secondary datacenter on Jan 8th. As of the morning of Jan 10th, we hooked the server back up and networked it into the datacenter only to find that the server's hardware had been damaged during shipping. According to our engineers, the hardware failure was not worth fixing. Our staff then spent a day discussing what steps / options we could take next to bring our community back online. As of the evening of Jan 11th, we decided to setup three new servers; a new primary server, a new secondary backup server, and a new proxy / load balancing server. Work immediately started on the new servers; with all funding for the new servers immediately invested without a second thought. Jan 16th we brought the site back up and immediately took it back down for additional maintenance needed to fix priority level bugs on the new servers. Finally, on the night of Jan 17th we officially brought our site back online.
The Bad News:
* We lost 2 weeks worth of site updates. (all updates we made to our site between December 15th - Jan 1st have been lost)
* We lost 1 week of data. (all private messages sent, orders purchased, and accounts registered between December 24th - Jan 1st have been lost)
The Good News:
Our staff is busy rapidly re-engineering the lost updates made to our site. Things might look a little wonky on our site for the next week or two, but you can expect everything to go back to normal rather quickly. Lost orders will be re-delivered. We still have the mafia account names, payment details, and number of products each person purchased between December 24th - Jan 1st. All 133 orders purchased during that time will be re-delivered. We will be contacting each and every one of the 133 customers who purchased products between those dates to make sure you receive the products you paid for, delivered with the full 110% mafia service our customers have come to know and love. We are deeply sorry for any inconvenience this delay may have caused you.
Technical details on why our site was down for almost 3 weeks:
There was an accident at our main datacenter. The datacenter's networking infrastructure was badly damaged on Jan 1st. Our primary server had to be shipped to our secondary datacenter. The primary server arrived at the secondary datacenter on Jan 8th. As of the morning of Jan 10th, we hooked the server back up and networked it into the datacenter only to find that the server's hardware had been damaged during shipping. According to our engineers, the hardware failure was not worth fixing. Our staff then spent a day discussing what steps / options we could take next to bring our community back online. As of the evening of Jan 11th, we decided to setup three new servers; a new primary server, a new secondary backup server, and a new proxy / load balancing server. Work immediately started on the new servers; with all funding for the new servers immediately invested without a second thought. Jan 16th we brought the site back up and immediately took it back down for additional maintenance needed to fix priority level bugs on the new servers. Finally, on the night of Jan 17th we officially brought our site back online.