eUSA Forums Update - July 17, 2014 [UPDATED 10:23 PM EST]
PigInZen
(Update at the bottom)
A Little Background
Hi again, eRepublik friends and colleagues. For those of you that don't know, I am the primary administrator of the off-game forum for the eUSA community. Recently there was some extended downtime. I apologize for the interruption in service and the inconvenience this caused to the eUSA players and government as well as to the host of others that visit the forum on a regular basis for fun and entertainment. I pay for and maintain this server because it's my strong belief that the eUSA needs a place to discuss in-game matters and build friendships. I try to do it in a fair and impartial manner and the moderation at the forums is community-run. I have a strong dedication in keeping the forums active and dedicated as a tool for the eRepublik eUSA community and maintain personal control over the server and forum mostly to prevent the forum from being used by admins for political favoritism and unethical opportunity.
What the Hell Happened - Why We Were Offline
Our old hosting service, Burst.net, went out of business. I was informed of this back in mid June. Burst.net sold their accounts to another provider, Hostwinds, and I believed, mistakenly, that the contents of our current server would be migrated to another server managed by Hostwinds. Fair enough, all I should have had to do was change the domain name registration and update the IP address.
Users of the forum began noticing that performance was terrible about a week ago. I failed to connect this poor service to the fact that Burst was closing shop. That poor performance was most likely tied to Burst turning off service to us or to the mass-migration of server data from other Burst clients to new servers. I spent two days or so troubleshooting what I thought was more configuration problems or a continuation of configuration problems that led to the last server rebuild in May. During those few days troubleshooting I finally noticed that I had to arrange for the migration by notifying the new hosting service, Hostwinds. I did so last Friday. The old server data was migrated on Saturday.
Here's where Real Life™ intervened. Some of you are well aware that I am the father of two sets of twins. Well, early-to-mid July where we live is 4H Fair preparation time and our kids have many projects to complete and submit for judging. It just so happened that this past weekend was that time. One of our sons was working on a rather complex model of a Sopwith Camel WWI fighter plane and needed my help. ALL DAY ON SATURDAY AND MOST OF SUNDAY. This meant that I didn't get around to working on the new server until Sunday evening, late, after we arrived back home from the fairgrounds and eating dinner out (at Chili's, that's what all fine suburban families eat, right?).
Upon connecting to the new server I noticed that something wasn't familiar. Now, I'm not the world's most advanced IT guy, but I know my stuff, mostly. The old server ran a variant of Linux provided by Debian. The new server wasn't Debian. It was a different variant, CentOS. Big deal, you think. Well, the different variants have different ways of dealing with providing all the services that run a webserver, including different locations for configuration files and where data should go. That means the old locations for our database, web files, etc., were not correct. And bigger yet, I have experience with Debian. I have none with CentOS. While they're both technically UNIX, I could set up a Debian server much faster than setting up a CentOS server. I tried to make the best of it but after a couple of hours doing config and setup I punted and went to bed.
The next morning I contacted Hostwinds support, which I must say was great. They advised me to backup the current server and that they would install Debian. So I did the backup Monday evening and let it run overnight. Why the delay? Well, I work for a major international apparel and shoe manufacturer not named Nike and we block most outgoing communications I need to connect to the server (SSH, SFTP) to do the work I needed to get things running. This means that I cannot easily work on the server during normal work hours in the Eastern Time Zone. The following morning (Tuesday) I put in a support ticket with Hostwinds to re-image the server with the Debian OS. They did so.
So Far So Good
So Tuesday evening after arriving home from work, guess what? More 4H projects, this time all four kids had projects due during the day on Wednesday, July 16. This meant that I couldn't get connected to the server to do config and restoration work and instead was focused on helping the younger twin girls with their baking projects and the older boys with their snack foods project and business plans. Tons of work. So much that I took Wednesday morning off from work to finalize the older boys' project. Then I went to work. More delays.
So Wednesday evening. I began my configuration work, hardening the services (web, database, ssh, sftp) and installed various monitoring software. I went to restore the database and oticed that the database files themselves weren't in the comprehensive backup I made before reimaging the new server with Debian. They should have been there but weren't. Crap. I knew it was bad and when the upload of the most recent database backup from May 15 failed I went to bed. It was midnight and I had to be up at 5:30 AM for work.
So this morning. Headed into work early and jumped onto our testing connection to get access to the server. When I do this I don't get corporate email, IMs, or access to anything inside the network. Whatever, I needed to get this damn thing running. I restored the recent database backup from May 15 and turned it on. It works.
Implications
Since my negligence as a sysadmin meant that I had not scheduled regular database dumps we have lost all changes to the forum after May 15, 2014. Most of the last terms of Congress and the Presidency. All of the discussions about alliances. All of the party discussions, as well as the Off-topic gen discussion areas. All of the new user accounts since May 15 as well as any changes made to existing profiles since then. All moderation, bannings, etc. This is my fault. I've tried to keep this forum running without major interruption since December 2011. I feel really sad and disappointed about this. It was avoidable. This is why you do backups, kids, and I have worked in IT for TWENTY YEARS and should have known better. Hell, I did know better and just didn't make it a priority. I apologize.
Thanks for reading. Hope this helps all of you to make sense of what happened and gain a window of understanding into what became a perfect storm that led to a week of downtime and the loss of nearly two months' of data.
The Forum
The eUSA Forums are located at eusaforum.com. That domain is currently pointing elsewhere but I will have that changed tonight, Thursday, July 17, 2014. It will take 24-48 hours for the domain name change to fully propagate globally so if you want to access the server, please do so via the IP address: http://23.254.129.242
Thanks for reading. Cue Ajay Bruno/Ronald Gipper Reagan saying something about how I better watch my kids or... something might happen to them.
UPDATE, 10:23 PM July 18, 2014: DNS change has been made and is successful for my host (Comcast) and many others globally (checked via https://www.whatsmydns.net/#A/eusaforums.com). There are some servers that have yet to update. Some of you will still need to use the IP address but many of you should be able to use the domain name for access: http://eusaforums.com
Comments
Namaste
I guess you learned your lesson.. backup weekly at worst.
I would think at least monthly at the end of each Congressional Session....
I'm still offering FREE service, as I stated in a message to you on fb and on my article.
We've got tons of cloud space, yours for the taking. I'll even connect you directly to my hosting guy, I'm just the salesperson. The free service is only for you, however. Anyone else has to pay.
Kids come first, bro. And I ain't talking about THESE kids - lol
Please, just duck my sick and move on!
You are awesome.
Feel free to discard my two months while you're at it... they were uneventful anyway.
no they werent, there was more greeling
Yeah but once you've read one greeling post you've read them all
He's a little bit less fumb as duck then you are!
Thanks a lot.
You Sir, are a masochist, and I mean that in the nicest way possible. Two sets of twins, werk full time, and still trying to fit in switching a forum over to a new server on a tight deadline.
*Hands you a case of your favorite alcoholic beverage*
Hi.
Thanks for your hard work PiZ! We appreciate all that you do for the community!
Drunk ninja presdienting ftw! 😛
Seriously though, thank you for all your time and effort on behalf of our community. Sometimes shit happens and this is one of those times. We will all cope. We know you aren't required to do what you do and it IS appreciated, both during the good times and the rough times.
So Pig works for Nike?
NOT NAMED NIKE 🙂
This man truly works with everyone's best interest at heart.
Anyone who bitches about this 'downtime or lack of a backup' should be chastised, castrated and blacklisted by congress.
Thank you, PiZ
really, Pig, you don't owe Ajay or Snake or anyone any explanation.
the rest of us appreciate your efforts.
Namaste
two sets of twins o7
People make mistakes man
l like a man who knows he screwed the pooch and admits it, the sUSA Forums run smoothly for the most part and PiZ is the reason they do so; since RL > GL maybe someone with no life should maintain the forums....
2 sets of twins...wow...and I thought raising 2 girls was hard..!
😛ipe:
Can I get an Under Armor Tee Shirt please?
Also, you do this game a service, if someone bitches give them the finger and tell them if they are not happy to go out and try to do better.
It's adidas, damnit
Can I get a soccer ball then?
Would you like it signed?
http://i.gyazo.com/ee9483aba592df6a313543511ee5af23.png
I could probably arrange that, yes.
[removed]
I FN love Adidas, almost as much as Puma or Wilson; actually anyone besides the swoosh...I have some REALLY old Adidas tennis shoes, could you sign them for me?
I'll be happy to sign them but my sig ain't worth dick.
Thanks PIZ for the hard work!!'
Thanks for all your work
At least you don`t work for Nike!
LMAO Lemi
Something something joke about Carbonite Online Backup which is irrelevant to the actual discussion here, only because it's stuck in my head from an ad on the radio.
Glad the forums are back at least.
gdi I told Pub to do db dumps regularly
I still have some super old ones from when I was Admin if you want
Nah, we have everything from before Nov. 15. Pubs was doing dumps, they didn't make it over to the new server though. I will be putting in place an automated dump/post to external storage process.
I'm gonna automate my dumps.
On your face.
Fluffy, you're an a$$hat & you know it. Go now to the forum to insult regular visitors!
tl;dr: my post count has been lowered.
I worked so hard on that. SO HARD.
I know right? I'm under 3000 again 😨
You know what they say about big post counts.
No, I don't, what do they say?
Wow, you're a busy dude...
And it probably goes without saying, but thanks for all your efforts to keep the loony bin working as smoothly as possible.
Thanks for the summary, PigInZen! Shit happens and admitting is noble.
Were the kids successful with the projects after all?
Girls: cookies, first year 4H foods project. Blue ribbons both for snickerdoodles and peanut butter cookies. They're showing their guinea pigs (!!!) on Sunday for pocket pets. Boys: snack foods & accompanying business plan for seventh year 4H foods projects. Blue ribbons both for excellent nutty trail mix (with honey & pistachios!) and protein bar. They have an art project and model of Sopwith Camel WWI fighter plane respectively that we're waiting to find out how judging went (tomorrow night).
Thanks for asking. I love talking up my kids.
You are an admirable father! It's good to know your priorities ^__^
Summary??? It's a Dostoyevsky piece tl;dr shite!
Meh, it happens. Did your kids place in any of the events?