A Wandering Mind

When I was a DBA (and, to a lesser extent, now as well) I frequently got into a scenario where I’d have multiple groups clamoring at me to accomplish a request from them on short notice. This sometimes became a problem, such as when there was only x hours in the day and I needed y hours to finish everything that folks were requesting. Or, when I had multiple people demanding immediate service (that never happens, right? Oh wait, it just happened today, that’s why I’m writing this. #Facepalm). Earlier in my career I would tend to get worked up, let my blood pressure go up, and perhaps engage in some creatively worded conversations with the people competing for my time. But after realizing that (a) this really wasn’t helping things, since people still want what they want; (b) all the yelling wasn’t helping my health, I came up with a couple rules on how to handle these, let’s say, explosive situations. In both cases, they’re remarkably effective at both alleviating the confusion over relative priority and lowering my stress level.

First: don’t even try to work out on your own which request is more important. Sure, you can ask questions and try and rank things in your mind according to things like which client things are for, how long things take, etc. Ultimately though, you’re going to be wrong in someone’s eyes and that someone is going to be unhappy with it. And moreover, unless you’re a manager, you’re not paid to determine things like that. You’re paid to get things done and write code, period.

Second: let the competing groups do just that – compete! Send an e-mail to the various requestors stating something like this:

Hi there folks, this is your friendly neighborhood (DBA | Architect | Database Developer) speaking. I thank you all for putting in your requests to me, and I acknowledge that each of you has requested that I handle them immediately. Unfortunately I can only work on one task at a time (unless someone’s developed cloning technology, which would be super!), so I will be doing these in the order which they were received. Before you begin thinking of arguments to get me on your side, I will warn you that this decision is not mine to make, so please do not send me justification of why you need to come first. If you feel that your request should take priority, please work with the owner(s) of those requests in front of yours to adjust their priority. If you all can amicably agree to shuffle things around I’m happy to oblige. I’ve listed the relevant work items below in order along with their owners. If I do not hear back I will assume you are all fine with this and will proceed in the order shown. Thanks!

Let the various people battle among themselves for the lead. Yes, this can sometimes get a little bloody, so you should probably check with your boss to make sure this is allowed before doing it. Which brings us to our third point…

Third: let your boss handle it. Managers are paid to, well, manage! If people cannot agree peaceably (or otherwise) on a pecking order, let your boss hear each of their cases and then tell you what to do. This will take the pressure off you to make a decision that is clearly one at a level above yours. Certainly your boss may (and perhaps should) ask for your input in the matter, but the decision should rest with them.

Since I’ve put the above rules in place I’ve never been more than slightly annoyed when these situations inevitably come along. It may not stop (or even lower) the volume of whining coming your way, but it will let you deflect and delegate all the stressful aspects of the problem.

Update: Mike sent me a link to an article he wrote for SQL Server Central on this same subject, and I wanted to include it here: http://www.sqlservercentral.com/articles/Troubleshooting/66134/

When I was sixteen I became one of the youngest EMTs in the state of Pennsylvania. I was proud to volunteer my time, and found I greatly enjoyed helping people. It was both incredibly humbling and empowering at the same time: on one hand, there were times when you were utterly helpless and could do nothing to help your patients, but other times you clearly saw the immediate impact you were able to have. I can say without question my experiences, both good and bad, changed me as a person.

Fast forward fifteen years, and I find that fellow SQL tweeter Mike Walsh (blog | twitter) also has some background in emergency services. We had a nice little chat about our experiences and how they ultimately help us in our daily work. Thus, I became inspired to write about how my time riding around in an ambulance helps me be a better DBA.

Checklists And Procedures Are Essential

When we arrived on the scene of a medical emergency, we were drilled ruthlessly about a series of steps to assess our patient’s condition in rapid fashion. One of the first tasks was what we called the ABCs- Airway, Breathing, and Circulation. Basically, is the patient have a clear airway (choking, etc), are they breathing, and do they have a pulse. It was simple and easy to remember; two essential things when you’re in an adrenaline charged situation with someone’s life on your hands.

Once we were past the initial basics, we had a written list of additional information to gather: current medications, allergies, existing conditions, injuries, etc. Personally I carried that pad of paper with me in my pocket at all times while on duty, and used it without fail on every call. I don’t know about you but I would not trust someone’s life to my memory.

The same should be true about dealing with SQL Server emergencies (I use that term loosely as you’ll see later). Do you have a documented, step-by-step process to perform an assessment of the issue at hand, and take corrective measures? Is it readily available, clearly written, and has it been practiced? What about recovery procedures? If you get paged at 3AM (after a night at the bar of course) and have to perform a restore of a corrupted page, will you be going off memory or will you be walking down your checklist? Heck, forget you, what about that junior DBA who joined your team a month ago and is taking his first on-call rotation? Do you really want him flying by the seat of his pants?

Sometimes A Calm Voice Is The Best Medicine

I once was called out to a house where we found a young boy who had fallen and broken his arm. Immediately it was apparent that the injury was fairly minor (at least as minor as a broken limb could be), and that the situation was being made far more chaotic by the boy’s hysterical parents. They were yelling and screaming at each other, blaming one another for letting this happen, yelling at us for not taking proper care of their son, etc.

That night it happened that my driver was also my supervisor, a man with over ten years of experience in EMS. He quickly brought the two parents together off to the side, and said in a quiet but authoritative voice, “Ma’am, sir, I promise you we are doing everything we can to help your boy out. What I need from you two right now is to listen carefully to my medic’s questions so we can get all the information the doctors will need. Can you do that for me please?”

That simple request, combined with his soothing tone of voice, settled them instantly and brought them out of their hysteria. Once they had something to focus on they were cooperative, and we were able to get the information we needed and transport the boy to the hospital for treatment.

Similarly, the next time someone (be it a developer, a business user, or whomever) runs up to your desk in a panic, try speaking to them in a calming manner. Assure them that everything will be done to fix their situation, then ask them to help you by answering some simple questions. Is the application totally down, or is there just some functionality that is not working? How many users are affected? Are there any financial or other risks present due to the ongoing presence of the issue? In a lot of cases, those answers may even prove that the situation really isn’t as urgent as they thought.

It’s Not Life And Death

Mike and I both agreed strongly on this one: our time as volunteers gave us a healthy dose of perspective when it came to dealing with IT “Emergencies”. After all, it’s not like we’re dealing with broken bones, blood, or the prospect of a patient dying.

I don’t say this to belittle the problems we deal with as DBAs. We are paid to take care of our users and our servers, and we need to take our work seriously. But at the same time, I think we need to make sure we don’t lose track of the bigger picture of life happening around us. A server going down is not worthy of raising your blood pressure, nor is dealing with “that annoying developer” again worth losing your cool. Ultimately, life will go on, the sun will come up, and you can go home to hug your loved ones at the end of the day.

Note: I do want to single out and commend those of us who truly support mission-critical platforms, such as those in hospitals, power plants, etc, where lives may really be at stake. You have my utmost respect as a fellow data professional.

Maybe some day I’ll return to volunteering my time to help those who have become victims of life. In the mean time, I take solace knowing that the lessons I learned in my time in EMS are with me today, helping me to truly be a better DBA.

I’ve blogged a bit over the life of this site about my use of a life-management framework called GTD and how profoundly it’s affected my life. Sometimes when I talk to people about it, a common response is “but why would I want to spend so much time keeping track of what I have to do instead of just doing it?”

The answer is that having everything in those magic lists gives you a kind of ultimate peace of mind that’s, well, hard to describe for those who haven’t experienced it. Now I know that makes it sounds like some kind of hippy-ish drug-induced trip; but really, the simple confidence of knowing all your responsibilities (“agreements” as David Allen, author of GTD calls them) will be there waiting for you when you come back is nothing short of priceless.

As I was reading my blogs today I found a great entry from fellow GTD’er and SQL community member Brent Ozar (blog | twitter). I think the passage below beautifully describes this “mind like water” (another David Allen phrase) state:

At around 5PM, when I’m not on the road, i leave my home office and my tasks behind.  I walk Ernie (our dog), get the house ready for Erika’s return from work, and leave the workday problems behind.  I’ll still check email from my ozone when we’re not doing anything, and I’ll respond to quick questions, but I won’t do work.

And I won’t care.

I won’t stress out about things I have coming tomorrow, won’t get worried about what a client’s server is doing, won’t work late trying to “get ahead” – because there’s no such thing.  As a knowledge worker, I’m going to be behind for the rest of my life.  The better I am at accomplishing stuff, the more work people will give me.  At 5PM, I have to change contexts because I won’t ever be caught up in my home life either.

via GTD: Why Things Have Been Quiet Around Here | Brent Ozar – Too Much Information.

Thanks for helping spread the word Brent. It’s always great to see others having the same experience and speaking eloquently about it to the world.

Now if you’ll excuse me, I’ve been neglecting that @home list of mine.

So as I previously wrote, the new gig is turning out to be a little bit more intense than I expected. And while I’d love to say that I have it all under control, that, in truth, would be a flat out lie.

The last few days I’ve done my best to keep track of how many interruptions I get on a given day; today I lost count at over ten. Ten in eight hours of work! That means I averaged more than one interruption per hour. Of all of them, only one was actually worthy of breaking my concentration. The rest were either simple questions or related to non-emergency work, such as ongoing testing or requests in queue.

I have to say, I’m at a loss how to put an end to these, or at the very least, reduce their volume. I’ve tried gently redirecting people to send the team an e-mail, forcefully redirecting them (which is usually met with a resentful glare or a “but it will only take 30 seconds”), even putting on my headphones and doing my damndest to ignore anyone who walks up to my desk. It’s gotten so bad that my senior DBA has called a SWAT meeting next month with all of my unit’s team leads to discuss how this situation can be rectified. Without something on the side of a drastic change, my team is going to have a really hard time getting things done.

Even trying to triage everything in queue is difficult, because inevitably, everyone argues valiantly (and rather stubbornly at times) that their issue is the most important and must be looked at first. And most have valid reasons: client impact, loss of functionality, etc. The problem is, there are so many to pick from at a given time that we simply cannot take “My problem has client impact so it must be looked at ASAP” at face value. Otherwise, we’d be constantly shifting in work and never able to focus on our larger goals. Here again, I’m really hoping next month’s meeting will help to clarify and ease this nasty task. At the very least, we need to have some kind of objective framework by which to determine the true priority of requests / issues, similar to the various trauma scales I learned years ago as an EMT.

Yes, I really did just compare my work to dealing with traffic accidents, gunshots, and various other forms of critical illness. The way things are around here, you’d think we were really dealing with life and death scenarios, instead of mere dollars and cents.

His heart was pounding, racing at a pace that surely would make any competent doctor raise an immediate concern for the safety of his patient. Beads of sweat rolled slowly down his forehead, wiped away by a hurried swipe of a forearm, only to be replaced by fresh ones. He was hunched over his keyboard, fingers flicking over its surface in a frantic blur of activity. His eyes gleamed with fury; two dark spots whose glare was entirely focused on the screen before him.

It was anything but an elegant fight that they waged, the operator and his silicon-based opponent. They darted back and forth, each one probing the other for any sign of weakness. Even as one seemed to gain the upper hand, the other would counter with a move both swift and unexpected. It was a war of attrition waged by two stubborn parties: one driven by his need for dominance, the other because it did not know how to do anything but respond in kind.

Words and numbers trailed endlessly across the screen, a seemingly meaningless stream of nonsense, uttered by a mind bereft of all reason. Yet to the operator it all made perfect sense; perfect nonsense, that is. As his nemesis endlessly spewed chunks of data, his eyes sprinted back and forth, trying as best he could to collect and analyze it all. But try as he might, it seemed always that his enemy held the upper hand. Just as he gleaned a tiny bit of understanding, some small nugget of progress, a new wrinkle would appear, more confounding than the last.

As his frustration grew he felt his temples begin to throb; gently at first, but increasingly as time passed. It was as if his body was trying to tell him “Back off, you’re in dangerous territory.” But he was past all reason, so completely consumed he was with his task at hand. On and on he pushed, till the throbbing became outright agony. He flung open the drawer to his right, digging feverishly through its contents until a small bottle was found. Pawing it open, he quickly extracted two small white pills and swallowed them hurriedly, without water. They offered little hope of relief, but perhaps they would give him the edge he so desperately desired, no, needed.

So, I’ll give you three guesses around what the above narrative is about. A piece from my first attempt at an epic sci-fi perhaps (yes, that is very much on my someday list)?

I’ll give you a hint: been following the saga of Urban Meyer, the embattled yet highly acclaimed football coach of the Florida Gators? If you have, you know that he’s basically been told by his doctors that the very thing that makes him one of the most successful coaches in history may also cause him to prematurely leave this Earth: his singular focus and intensity.

When I first read this story, my immediate thought was “Wow, that could be describing me to a T, minus the arachnoid cyst pressuring his skull and all.” I am probably one of the more driven individuals you’ll ever meet; when my mind is set on solving some problem, it’s a nearly useless task to try and pull me away. Just ask my wife, she’ll be happy to oblige you with all sorts of evidence supporting that statement.

On the one hand, this trait is an asset: it lets me persevere when others would stop, long discouraged by setbacks and seemingly insurmountable challenges. It’s probably the major reason why I’ve been so successful in a field that, up until around three years ago, I knew nothing about. But there is a dark side to this unending focus. It manifests as the (not so at times) occasional splitting headache, the missed personal responsibility, the midnight awakenings when nothing short of a blow to the skull will seemingly quiet my mind.

Faced with the prospect of enduring this escalating chain of physical and emotional malaise, I’ve made the decision, much as Meyer did, to step back. No, I’m not taking a leave of absence, but I am engaging in a concerted effort to become more aware of my emotions and the physical manifestations thereof. Journaling, reminder alarms to “check in” with my state of mind, even asking others to notice when my symptoms begin to show themselves.

I love my work, but I love my family and my health above all else.

Update: I’ve been following the comments on this post with a mixture of surprise, amusement, and a little bit of nervousness. It’s great how everyone is commenting, but based on the somewhat “medical advice-y” nature of some of the comments (and perhaps the post itself) I feel compelled to explicitly point out the following: My naming of “Racing Brain Syndrome” is purely anecdotal and should not be considered any kind of official medical diagnosis. I came up with this name purely from my own experience as described in the post. If you are experiencing any kind of severe or disturbing symptoms, including severe forms of anything described in this post, please consult with a licensed professional. I am not a doctor or psychologist and my advice here should not be taken for medical prescription. I’m just a guy with a fidgety brain trying to relax and be productive.

We’ve all had this happen to us at one time or another. You wake up in the middle of the night, thoughts rushing through your head at a mad pace. You try to take the zen-like approach of “letting them go”, but it’s hopeless. You toss and turn, but the harder you try to sleep, the more awake you are. This, friends, is what I call “Racing Brain Syndrome”.

There are two main forms of this nasty little bug, which we’ll call “Stress Induced” and “Excitement Based”. In this post, we’ll look at the first variety in more detail.

Stress Induced

As the name implies, this version is caused by an excess of built up stress that has yet to be dealt with. Common symptoms (not inclusive of the other variant of this syndrome) include racing pulse, pounding heartbeat, cold sweats, and possibly (in extreme cases) delusions of persecution or general paranoia.

Now stress, as you well know, can come from many sources, including the practice of keeping things in your head, nagging concerns over projects left un-planned, fear of upcoming regulatory audits (a favorite of us IT folks), and of course the ever present conflict between the Ego and the Id caused by an underlying need for affection, complicated by an Oedipus complex.

Whoops, I must apologize for that last one. This post has me reverting to my old psychobabble style of writing. Ignore that one, will you please?

When dealing with this variety of RBS, one’s best course of action is tri-fold:

1. Determine if the cause of the stress is a rational one. That is, are you feeling stressed because you’ve fallen off your good practice of keeping things out of your head, or are you suddenly having a sinking feeling that you’ve left your torrent bot up and running at work, and the folks from InfoSec are, at this very moment, hot on your trail? Ok, that’s an extreme example, but you get my drift.
2. If the former, your best bet is to take a few minutes and put some thoughts down on paper around what is bothering you. You don’t have to answer every question out there; just make sure every question is written down so you are confident it won’t get lost in the shuffle.
3. If the latter, you would be advised to fall back on a technique I used to teach to the children at the mental health clinic I worked at out of college, called (in it’s most complex form) Cognitive-Behavioral Therapy. The basic idea is that you write down the thoughts that are causing you discomfort, such as “I’m afraid the ninjas from security are after me”, then examine them in a critical, analytical fashion. For instance, what evidence is there that you’re really about to be attacked in your sleep? Do you even have a torrent bot on your work computer? These techniques are usually used by patients with more severe mental health issues such as depression or anxiety disorder, but they serve RBS sufferers equally well.

On a sidenote, if you’re actually interested in a more clinical view of CBT, I’d heartily recommend the book The Feeling Good Handbook by a fellow named David Burns.

Next time we’ll examine the milder, and perhaps more pleasant variety of RBS, “Excitement Based”.

Today I caught myself getting rather ticked off after spending around 2.5 hours trying to script out changes to around 25 SQL Server stored procedures. To explain in a non-geeky (if that’s possible coming from me) way, I had to comment out a common line in all 25 procs that needed to be disabled in order to use the proc in my test system. (OK, that failed only slightly in the non-geeky department.)

I’ll be the first to admit that once I get focused on solving an issue, it pretty much consumes my attention, to the point of near obsessiveness. This was no exception, with me getting more and more worked up as attempts to automate this change (rather than cracking the code open on all twenty plus one at a time) failed. Finally I threw up my hands in disgust and walked away, having undoubtedly raised my blood pressure a notch or two in the process.

A short time later, it occurred to me that most of what I was trying to accomplish was completely outside of the work necessary to accomplish my pre-stated goals for this project. In effect, I was trying to change twenty plus bits of code, when this particular effort required just one change. Yep, you heard me, just one. So why even make the attempt to automate the process?

I suppose it’s mostly because I’m a lazy coder by nature; if chances are better than 50-50 that I’ll need to repeat some action in the future, I’ll probably at least take a stab at scripting or otherwise automating the process. Plus, in this case, it presented a worthy challenge, which is always more than enough to entice me into jumping in over my head. I’m a sucker for challenge, almost to the point of it being a character flaw.

In hindsight, perhaps if I’d stopped at the onset and thought things out in a rational, methodical manner, I would have saved myself not only a brooding headache, but almost two hours of time as well. Two hours. Think of all the things you could get done in two hours time.

Next time, I’m going to follow my own advice, and make sure I properly define what my work is going to be, lay out all the time and effort needed, and only then consider adding scope to my plate. Sure, automation is a wonderful thing, but only when used with discretion. Spending two hours to avoid doing something that ultimately took around 10 minutes to accomplish manually just isn’t worth it.