Being the middle of the summer in the northern hemisphere, and with the Database Corruption Challenge over, I thought I would cover something a bit lighter, and not as hard core as database corruption.
Share Your DBA on Vacation Stories
If you have been a DBA for more than a year or two you probably have some story about going on vacation, and getting the call from the office with some database problem. It might have been that someone forgot their password, or something more serious like a corrupt database. Either way, the simple or more complex issues are not why you went on vacation.
Here is a picture of a DBA on vacation in Hawaii. Notice the empty chair… They should be sitting in the sun sipping a tropical drink, but instead they are back in the hotel room with a VPN connection to the office dealing with some emergency (or perceived emergency) SQL Server issue instead of enjoying the beach.
DBA on Vacation (photo by Steve Stedman)
If you work for a big company with many DBAs, with different shifts, and plenty of coverage, this may not be as much of an issues. For the small company with only 1 or 2 DBAs, a vacation can have a significant impact on the business if something goes wrong.
I am looking for stories from DBAs on what you have been called for when you were out on vacation, holiday, or just away for a long weekend.
I will kick it off with a short story of my experience:
Many years ago I took off with my family for a weekend at Lake Chelan (eastern Washington State USA) at the end of September for a last boating weekend in the warm weather before the season was over. Where we were staying was out of cell phone range for 4 days. When we returned to cell phone range, I had over 20 voice mail messages, and over 1000 text messages. This was before the days of the smart phone, so the incoming text messages took over an hour of continuous beeping to come in. Every time a message arrived it interrupted me trying to get to voice mail, or to read the text message. When the mesages finally stopped, and I was able to get to my voice mail, I found out that the primary database for a web SaaS service had a RAID controller failure, and nobody knew how to restore the database, and due to a glitch in the firware, then RAID redundancy didn’t do any good. Well, my employer also didn’t have any support contracts in place due to the budget crunch after the dot com bubble, so we couldn’t get any help from Dell. A think that I worked for 36 hours straight when I returned from that vacation until I was finally able to rebuild things, undo what had been tried while I as out, and finally get the production system running again. By the time that 36 hours was over I needed another vacation to recover. My reaction was to eventually leave that job for several reason, one of which was the none of the hardware was under maintenance, and eBay was the preferred vendor for servers.
Please share your DBA on vacation stories here.
Please post a response below with a short summary of what you have been called for when you were on vacation. Please include how you reacted, and how you wanted to react. Also I would love to hear what you might have done to prevent this type of vacation intrusion in the future.
More from Stedman Solutions:
Steve and the team at Stedman Solutions are here for all your SQL Server needs.
Contact us today for your free 30 minute consultation..
We are ready to help!
This story will be probably like a lot of other stories but this is what I experienced.
I went on vacation with some friends to the mountains of France for two weeks of snowboarding. When we arrived at the hotel and finally settled in we got out to some drinks to start the vacation. A few beers later I walked back around 1 in the morning and my phone rings.
It was my personal phone and I didn’t recognize the mobile number in the screen. I answered the call and it was my boss. Before I could say anything he screamed in the phone why I didn’t answer my work phone and told him I was on vacation. It was not a very patient man and didn’t have a lot of social skills in my opinion. I wasn’t in the mood to get yelled at especially through the phone, on vacation and by my work on my personal phone. How did they get this number!
With a stern tone I answered that I didn’t like to be yelled at and if the tone didn’t change I would hang up. This wasn’t a good reaction but everything together, the booze, the fact that I got yelled at didn’t make it easier to react.
Besides the fact that I got yelled at I gave all the work to one of my colleagues. Nothing special would happen during my leave so monitoring everything would have been the only thing they would have to do.
My boss calmed down, fortunately.. I still have a job, and I asked him what the problem was. It turned out that the production database for one of our customers was down after a change executed by one of my colleagues, who is not a DBA. The change was planned to be executed when I got back but the customer wanted it to be executed right away.
The change was still in the test stage and we had some bugs in the code. The customer wanted the change to be executed in production and told my colleague that all tests were successful. That, that wasn’t true would be very clear after the change. I would have asked for the test documentation but my colleague didn’t ask for it.
Fortunately I took my laptop with me on vacation. I told my boss I would contact the guy who executed the change and that I would call him when I got connected and inventoried the damage.
Try connecting to a VPN in a hotel where the WiFi literally sucks. I got disconnected over and over and it took me hours to inventory all that was wrong.
It turned out that the data in the database was scrambled due to the bugs in the code. Columns were named differently because it should still have been test code.
All together the database was a mess.
I checked for the last backups and prayed that my colleagues checked them during my absence. Yes!… we have good backups of the database before the change. We’ll only loose about 4 hours of data but unfortunately I couldn’t do anything else.
I called my boss and told him what I found. He was furious that the change got executed on production without good testing.
I restored the last backup and everything worked again but the customer lost a lot of data.
The thing I learned from this is that I shouldn’t have reacted to my boss like that during the first call. I should have just listened and should have kept my cool.
The next thing I should have done is tell everybody that no change was going to be executed on the database servers until I got back. Even if the customer wanted the change to be done immediately they should have contacted me beforehand.
At the end this was a learning experience for the company and me and the customer didn’t have too much problems with the incident.