A Wandering Mind

Update: Mike sent me a link to an article he wrote for SQL Server Central on this same subject, and I wanted to include it here: http://www.sqlservercentral.com/articles/Troubleshooting/66134/

When I was sixteen I became one of the youngest EMTs in the state of Pennsylvania. I was proud to volunteer my time, and found I greatly enjoyed helping people. It was both incredibly humbling and empowering at the same time: on one hand, there were times when you were utterly helpless and could do nothing to help your patients, but other times you clearly saw the immediate impact you were able to have. I can say without question my experiences, both good and bad, changed me as a person.

Fast forward fifteen years, and I find that fellow SQL tweeter Mike Walsh (blog | twitter) also has some background in emergency services. We had a nice little chat about our experiences and how they ultimately help us in our daily work. Thus, I became inspired to write about how my time riding around in an ambulance helps me be a better DBA.

Checklists And Procedures Are Essential

When we arrived on the scene of a medical emergency, we were drilled ruthlessly about a series of steps to assess our patient’s condition in rapid fashion. One of the first tasks was what we called the ABCs- Airway, Breathing, and Circulation. Basically, is the patient have a clear airway (choking, etc), are they breathing, and do they have a pulse. It was simple and easy to remember; two essential things when you’re in an adrenaline charged situation with someone’s life on your hands.

Once we were past the initial basics, we had a written list of additional information to gather: current medications, allergies, existing conditions, injuries, etc. Personally I carried that pad of paper with me in my pocket at all times while on duty, and used it without fail on every call. I don’t know about you but I would not trust someone’s life to my memory.

The same should be true about dealing with SQL Server emergencies (I use that term loosely as you’ll see later). Do you have a documented, step-by-step process to perform an assessment of the issue at hand, and take corrective measures? Is it readily available, clearly written, and has it been practiced? What about recovery procedures? If you get paged at 3AM (after a night at the bar of course) and have to perform a restore of a corrupted page, will you be going off memory or will you be walking down your checklist? Heck, forget you, what about that junior DBA who joined your team a month ago and is taking his first on-call rotation? Do you really want him flying by the seat of his pants?

Sometimes A Calm Voice Is The Best Medicine

I once was called out to a house where we found a young boy who had fallen and broken his arm. Immediately it was apparent that the injury was fairly minor (at least as minor as a broken limb could be), and that the situation was being made far more chaotic by the boy’s hysterical parents. They were yelling and screaming at each other, blaming one another for letting this happen, yelling at us for not taking proper care of their son, etc.

That night it happened that my driver was also my supervisor, a man with over ten years of experience in EMS. He quickly brought the two parents together off to the side, and said in a quiet but authoritative voice, “Ma’am, sir, I promise you we are doing everything we can to help your boy out. What I need from you two right now is to listen carefully to my medic’s questions so we can get all the information the doctors will need. Can you do that for me please?”

That simple request, combined with his soothing tone of voice, settled them instantly and brought them out of their hysteria. Once they had something to focus on they were cooperative, and we were able to get the information we needed and transport the boy to the hospital for treatment.

Similarly, the next time someone (be it a developer, a business user, or whomever) runs up to your desk in a panic, try speaking to them in a calming manner. Assure them that everything will be done to fix their situation, then ask them to help you by answering some simple questions. Is the application totally down, or is there just some functionality that is not working? How many users are affected? Are there any financial or other risks present due to the ongoing presence of the issue? In a lot of cases, those answers may even prove that the situation really isn’t as urgent as they thought.

It’s Not Life And Death

Mike and I both agreed strongly on this one: our time as volunteers gave us a healthy dose of perspective when it came to dealing with IT “Emergencies”. After all, it’s not like we’re dealing with broken bones, blood, or the prospect of a patient dying.

I don’t say this to belittle the problems we deal with as DBAs. We are paid to take care of our users and our servers, and we need to take our work seriously. But at the same time, I think we need to make sure we don’t lose track of the bigger picture of life happening around us. A server going down is not worthy of raising your blood pressure, nor is dealing with “that annoying developer” again worth losing your cool. Ultimately, life will go on, the sun will come up, and you can go home to hug your loved ones at the end of the day.

Note: I do want to single out and commend those of us who truly support mission-critical platforms, such as those in hospitals, power plants, etc, where lives may really be at stake. You have my utmost respect as a fellow data professional.

Maybe some day I’ll return to volunteering my time to help those who have become victims of life. In the mean time, I take solace knowing that the lessons I learned in my time in EMS are with me today, helping me to truly be a better DBA.

We’ve all had this happen to us at one time or another. You wake up in the middle of the night, thoughts rushing through your head at a mad pace. You try to take the zen-like approach of “letting them go”, but it’s hopeless. You toss and turn, but the harder you try to sleep, the more awake you are. This, friends, is what I call “Racing Brain Syndrome”.

There are two main forms of this nasty little bug, which we’ll call “Stress Induced” and “Excitement Based”. In this post, we’ll look at the first variety in more detail.

Stress Induced

As the name implies, this version is caused by an excess of built up stress that has yet to be dealt with. Common symptoms (not inclusive of the other variant of this syndrome) include racing pulse, pounding heartbeat, cold sweats, and possibly (in extreme cases) delusions of persecution or general paranoia.

Now stress, as you well know, can come from many sources, including the practice of keeping things in your head, nagging concerns over projects left un-planned, fear of upcoming regulatory audits (a favorite of us IT folks), and of course the ever present conflict between the Ego and the Id caused by an underlying need for affection, complicated by an Oedipus complex.

Whoops, I must apologize for that last one. This post has me reverting to my old psychobabble style of writing. Ignore that one, will you please?

When dealing with this variety of RBS, one’s best course of action is tri-fold:

1. Determine if the cause of the stress is a rational one. That is, are you feeling stressed because you’ve fallen off your good practice of keeping things out of your head, or are you suddenly having a sinking feeling that you’ve left your torrent bot up and running at work, and the folks from InfoSec are, at this very moment, hot on your trail? Ok, that’s an extreme example, but you get my drift.
2. If the former, your best bet is to take a few minutes and put some thoughts down on paper around what is bothering you. You don’t have to answer every question out there; just make sure every question is written down so you are confident it won’t get lost in the shuffle.
3. If the latter, you would be advised to fall back on a technique I used to teach to the children at the mental health clinic I worked at out of college, called (in it’s most complex form) Cognitive-Behavioral Therapy. The basic idea is that you write down the thoughts that are causing you discomfort, such as “I’m afraid the ninjas from security are after me”, then examine them in a critical, analytical fashion. For instance, what evidence is there that you’re really about to be attacked in your sleep? Do you even have a torrent bot on your work computer? These techniques are usually used by patients with more severe mental health issues such as depression or anxiety disorder, but they serve RBS sufferers equally well.

On a sidenote, if you’re actually interested in a more clinical view of CBT, I’d heartily recommend the book The Feeling Good Handbook by a fellow named David Burns.

Next time we’ll examine the milder, and perhaps more pleasant variety of RBS, “Excitement Based”.

Today was my first day back from work after nearly a week off.  Surprisingly, I had only around 300 e-mails to process in my inbox; I usually receive that amount each day, not counting automated alerts and the like.  After reviewing everything and making sure all my lists were up to date, I began working on what was the most important, urgent matter at the time.

By about 9:30AM, several fires had appeared on the horizon.  Each demanded much attention from my team, and it quickly became obvious that my day would not be very productive, at least from the perspective of crossing off a lot of items.

In the past situations such as this would cause my blood pressure to rise almost instantly.  Not because of any physical danger (“When Servers Attack” anyone?), but because subconsciously, I immediately began worrying about what I was not doing.  That is, if I was spending all my effort to correct some immediate problem, what was getting implicitly pushed off?

So what happened today?  Certainly at first, I felt a slight twinge and a brief rise in my pulse.  But after taking a breath and calmly reviewing everything in my “Next Actions” list, I was able to definitely know that I was indeed directing my time appropriately.  The feeling was one I am certainly not used to!

This state of being was possible only because of the following pre-existing conditions:

1. I have been incredibly good in terms of making sure that each and every possible “open loop” is in my trusted system, no matter how small or how insignificant it may seem.
2. I have developed the habit that whenever one of those “it needs to be done now” (usually stated by whomever reports the problem) type of problems hits my plate, I do the following:
   1. Use soothing words and a little empathic listening to get the person to remain calm.
   2. Ask pointed questions about the true severity of the issue. How many, what’s the workaround, etc.
   3. Take note of all the information for future reference. Moleskins are great for this.
   4. Politely let the person know that I am going to review the problem with my team and get back to them when more information is available, and as soon as time permits.
   5. If I believe the problem is truly urgent, I’ll immediately decide on a next action and put it in my list. Otherwise, I’ll throw some reminder into my “in bin” for later perusal.
3. I review my lists with near religious regularity, constantly reminding myself to not lose sight of the larger picture.

All of these combine to enable me to do my daily work with agility and perspective, such that at any given point, I feel very confident that I am doing exactly what  I should be.

The most difficult part?  Maintaining an even keel and and objective frame of reference in the face of perpetual problems.  That is a skill that I am still very much honing, and one that can only be developed through daily practice and a strong sense of your emotions.

How do you maintain a broad view of your responsibilities, while still responding to the fires that fall on you unexpectedly?