eUSA Forums Update - July 17, 2014 [UPDATED 10:23 PM EST]

Day 2,431, 05:43 • Published in USA USA • by PigInZen

(Update at the bottom)

A Little Background

Hi again, eRepublik friends and colleagues. For those of you that don't know, I am the primary administrator of the off-game forum for the eUSA community. Recently there was some extended downtime. I apologize for the interruption in service and the inconvenience this caused to the eUSA players and government as well as to the host of others that visit the forum on a regular basis for fun and entertainment. I pay for and maintain this server because it's my strong belief that the eUSA needs a place to discuss in-game matters and build friendships. I try to do it in a fair and impartial manner and the moderation at the forums is community-run. I have a strong dedication in keeping the forums active and dedicated as a tool for the eRepublik eUSA community and maintain personal control over the server and forum mostly to prevent the forum from being used by admins for political favoritism and unethical opportunity.

What the Hell Happened - Why We Were Offline

Our old hosting service, Burst.net, went out of business. I was informed of this back in mid June. Burst.net sold their accounts to another provider, Hostwinds, and I believed, mistakenly, that the contents of our current server would be migrated to another server managed by Hostwinds. Fair enough, all I should have had to do was change the domain name registration and update the IP address.

Users of the forum began noticing that performance was terrible about a week ago. I failed to connect this poor service to the fact that Burst was closing shop. That poor performance was most likely tied to Burst turning off service to us or to the mass-migration of server data from other Burst clients to new servers. I spent two days or so troubleshooting what I thought was more configuration problems or a continuation of configuration problems that led to the last server rebuild in May. During those few days troubleshooting I finally noticed that I had to arrange for the migration by notifying the new hosting service, Hostwinds. I did so last Friday. The old server data was migrated on Saturday.

Here's where Real Life™ intervened. Some of you are well aware that I am the father of two sets of twins. Well, early-to-mid July where we live is 4H Fair preparation time and our kids have many projects to complete and submit for judging. It just so happened that this past weekend was that time. One of our sons was working on a rather complex model of a Sopwith Camel WWI fighter plane and needed my help. ALL DAY ON SATURDAY AND MOST OF SUNDAY. This meant that I didn't get around to working on the new server until Sunday evening, late, after we arrived back home from the fairgrounds and eating dinner out (at Chili's, that's what all fine suburban families eat, right?).

Upon connecting to the new server I noticed that something wasn't familiar. Now, I'm not the world's most advanced IT guy, but I know my stuff, mostly. The old server ran a variant of Linux provided by Debian. The new server wasn't Debian. It was a different variant, CentOS. Big deal, you think. Well, the different variants have different ways of dealing with providing all the services that run a webserver, including different locations for configuration files and where data should go. That means the old locations for our database, web files, etc., were not correct. And bigger yet, I have experience with Debian. I have none with CentOS. While they're both technically UNIX, I could set up a Debian server much faster than setting up a CentOS server. I tried to make the best of it but after a couple of hours doing config and setup I punted and went to bed.


The next morning I contacted Hostwinds support, which I must say was great. They advised me to backup the current server and that they would install Debian. So I did the backup Monday evening and let it run overnight. Why the delay? Well, I work for a major international apparel and shoe manufacturer not named Nike and we block most outgoing communications I need to connect to the server (SSH, SFTP) to do the work I needed to get things running. This means that I cannot easily work on the server during normal work hours in the Eastern Time Zone. The following morning (Tuesday) I put in a support ticket with Hostwinds to re-image the server with the Debian OS. They did so.

So Far So Good

So Tuesday evening after arriving home from work, guess what? More 4H projects, this time all four kids had projects due during the day on Wednesday, July 16. This meant that I couldn't get connected to the server to do config and restoration work and instead was focused on helping the younger twin girls with their baking projects and the older boys with their snack foods project and business plans. Tons of work. So much that I took Wednesday morning off from work to finalize the older boys' project. Then I went to work. More delays.

So Wednesday evening. I began my configuration work, hardening the services (web, database, ssh, sftp) and installed various monitoring software. I went to restore the database and oticed that the database files themselves weren't in the comprehensive backup I made before reimaging the new server with Debian. They should have been there but weren't. Crap. I knew it was bad and when the upload of the most recent database backup from May 15 failed I went to bed. It was midnight and I had to be up at 5:30 AM for work.

So this morning. Headed into work early and jumped onto our testing connection to get access to the server. When I do this I don't get corporate email, IMs, or access to anything inside the network. Whatever, I needed to get this damn thing running. I restored the recent database backup from May 15 and turned it on. It works.

Implications

Since my negligence as a sysadmin meant that I had not scheduled regular database dumps we have lost all changes to the forum after May 15, 2014. Most of the last terms of Congress and the Presidency. All of the discussions about alliances. All of the party discussions, as well as the Off-topic gen discussion areas. All of the new user accounts since May 15 as well as any changes made to existing profiles since then. All moderation, bannings, etc. This is my fault. I've tried to keep this forum running without major interruption since December 2011. I feel really sad and disappointed about this. It was avoidable. This is why you do backups, kids, and I have worked in IT for TWENTY YEARS and should have known better. Hell, I did know better and just didn't make it a priority. I apologize.

Thanks for reading. Hope this helps all of you to make sense of what happened and gain a window of understanding into what became a perfect storm that led to a week of downtime and the loss of nearly two months' of data.

The Forum

The eUSA Forums are located at eusaforum.com. That domain is currently pointing elsewhere but I will have that changed tonight, Thursday, July 17, 2014. It will take 24-48 hours for the domain name change to fully propagate globally so if you want to access the server, please do so via the IP address: http://23.254.129.242

Thanks for reading. Cue Ajay Bruno/Ronald Gipper Reagan saying something about how I better watch my kids or... something might happen to them.


UPDATE, 10:23 PM July 18, 2014: DNS change has been made and is successful for my host (Comcast) and many others globally (checked via https://www.whatsmydns.net/#A/eusaforums.com). There are some servers that have yet to update. Some of you will still need to use the IP address but many of you should be able to use the domain name for access: http://eusaforums.com