Being the middle of the summer in the northern hemisphere, and with the Database Corruption Challenge over, I thought I would cover something a bit lighter, and not as hard core as database corruption.
Share Your DBA on Vacation Stories
If you have been a DBA for more than a year or two you probably have some story about going on vacation, and getting the call from the office with some database problem. It might have been that someone forgot their password, or something more serious like a corrupt database. Either way, the simple or more complex issues are not why you went on vacation.
Here is a picture of a DBA on vacation in Hawaii. Notice the empty chair… They should be sitting in the sun sipping a tropical drink, but instead they are back in the hotel room with a VPN connection to the office dealing with some emergency (or perceived emergency) SQL Server issue instead of enjoying the beach.
DBA on Vacation (photo by Steve Stedman)
If you work for a big company with many DBAs, with different shifts, and plenty of coverage, this may not be as much of an issues. For the small company with only 1 or 2 DBAs, a vacation can have a significant impact on the business if something goes wrong.
I am looking for stories from DBAs on what you have been called for when you were out on vacation, holiday, or just away for a long weekend.
I will kick it off with a short story of my experience:
Many years ago I took off with my family for a weekend at Lake Chelan (eastern Washington State USA) at the end of September for a last boating weekend in the warm weather before the season was over. Where we were staying was out of cell phone range for 4 days. When we returned to cell phone range, I had over 20 voice mail messages, and over 1000 text messages. This was before the days of the smart phone, so the incoming text messages took over an hour of continuous beeping to come in. Every time a message arrived it interrupted me trying to get to voice mail, or to read the text message. When the mesages finally stopped, and I was able to get to my voice mail, I found out that the primary database for a web SaaS service had a RAID controller failure, and nobody knew how to restore the database, and due to a glitch in the firware, then RAID redundancy didn’t do any good. Well, my employer also didn’t have any support contracts in place due to the budget crunch after the dot com bubble, so we couldn’t get any help from Dell. A think that I worked for 36 hours straight when I returned from that vacation until I was finally able to rebuild things, undo what had been tried while I as out, and finally get the production system running again. By the time that 36 hours was over I needed another vacation to recover. My reaction was to eventually leave that job for several reason, one of which was the none of the hardware was under maintenance, and eBay was the preferred vendor for servers.
Please share your DBA on vacation stories here.
Please post a response below with a short summary of what you have been called for when you were on vacation. Please include how you reacted, and how you wanted to react. Also I would love to hear what you might have done to prevent this type of vacation intrusion in the future.