A Wandering Mind

Brent Ozar (blog | @BrentO) wrote a great post the other day that I just read that touched on the oft-seen differences between management’s priorities and our own. Even having only been a DBA for a mere month (damn, has it been that long?) I’ve already encountered situations like this all too frequently.

Troubleshooting and digging into data are two of my favorite things to do when it comes to DBA work. I love pulling numbers out and slicing-and-dicing them every which way until a pattern emerges. In this light, I’ve been really excited lately about the prospect of building a kind of DBA data warehouse: a place where aggregated performance statistics from all our servers are kept and can be analyzed in detail. It’s not a novel concept by any means, just one that’s never been implemented at work. With this in place we could engage in detailed benchmarking of our environment, and proactively find developing “hot spots” before they become real fires.

The trouble is, we can never seem to find time to work on it. There’s always other things to work on, whether it be dealing with issues or building more servers to add capacity. This is really tough to handle because as a team, we pride ourselves on keeping things running at peak efficiency, like a well-trained pit crew for a championship racing team. We can clearly see the value in this project and how much it would help us.

But all that being the case, I won’t be spending my free time working on that. If I do spend time outside of business hours working (something I try to avoid whenever possible), it’ll be on those other, more important items.

Yes, you read correctly: “more important”. “But Josh”, you say, “you just said that stability is your team’s top priority, and to boot that you find your biggest intiative in that area incredibly exciting. Why the heck are you calling the other things ‘more important’?”

The answer is simple: because that’s what my clients want from me. Like it or not, right now the word is that the primary focus must be on building for new initiatives and responding rapidly to the changing needs of our industry. Granted that stability was ranked right behind that, and I am certainly going to stake a case to management about the need for some time to be allocated to our bechmarking project. But as I commented on Brent’s post (modified slightly):

I think in this case you have two ways to go: you can continue to play the role of the a-hole that is always arguing about priorities and complaining that he has to work on weekends to get his “important” work done, or you can accept that your client’s priorities are what they are, and try and work within them. Well, I guess there’s a third choice: quit and start blogging about SQL Server full time, but most of us probably aren’t that brave in this economy.

So thanks for the reminder Brent; while I sure do love building warehouses and crunching numbers, I won’t be doing any of that outside of 8AM – 5PM Monday – Friday anytime soon.

Recently I was chatting with my sister-in-law, who happens to be a product owner at a rapidly growing startup company. We were talking about the nature of “flow” and how important it was for people to be able to focus on doing their work without being interrupted. Partway through the conversation she had a great quote (paraphrased roughly here):

Google has a great description for the role of managers: they should be sh*t umbrellas for their team. Their most important role is to protect their team from all the sh*t that comes flying at them so they can focus on doing their job: writing code. That’s my job in a nutshell and I take it very seriously.

The best reference I could find to that term is here, where GMail product manager Todd Jackson was quoted as saying “You can either be a sh*t funnel or a sh*t umbrella.” The article goes on to describe how a good manager protects their team from unnecessary garbage, in effect being an “umbrella” to the storm of things (requests, questions, stupid meetings) that rain down on their staff. But some managers take on more of the role of being a “funnel”, in that they simply filter and then pass on the things that come at their team.

I suppose one might say, “Well at least in the case of the funnel, the people aren’t directly exposed to the onslaught of sh*t.” That’s true, but I’d counter that depending upon how the manager passes on the stuff they deal with, the end result could be the same. If at once-weekly meetings everything that’s been triaged is laid out and assigned, that’s fine, and in fact I’d argue that’s really playing the umbrella role. But, on the other hand, if the manager simply sends everything off towards their team with the convenient “FW:” tag, well then what’s the point of the filter at all?

Personally at work I’d love to have some way of shielding my team from the daily barrage of (mostly) crap work that comes our way. I still believe the idea of a “handler” and holding designated office hours would provide immediate and noticeable relief. In effect, the handler becomes the umbrella, handling the collecting and triaging of incoming requests and only interrupting the rest of the team when absolutely necessary. Hopefully we can get some buy in from management on the idea, and time will prove our strategy to be one that greatly increases our ability to get work done.

So how do you shield your team from flying sh*t? Are you a funnel, an umbrella, or some combination thereof?

So as I previously wrote, the new gig is turning out to be a little bit more intense than I expected. And while I’d love to say that I have it all under control, that, in truth, would be a flat out lie.

The last few days I’ve done my best to keep track of how many interruptions I get on a given day; today I lost count at over ten. Ten in eight hours of work! That means I averaged more than one interruption per hour. Of all of them, only one was actually worthy of breaking my concentration. The rest were either simple questions or related to non-emergency work, such as ongoing testing or requests in queue.

I have to say, I’m at a loss how to put an end to these, or at the very least, reduce their volume. I’ve tried gently redirecting people to send the team an e-mail, forcefully redirecting them (which is usually met with a resentful glare or a “but it will only take 30 seconds”), even putting on my headphones and doing my damndest to ignore anyone who walks up to my desk. It’s gotten so bad that my senior DBA has called a SWAT meeting next month with all of my unit’s team leads to discuss how this situation can be rectified. Without something on the side of a drastic change, my team is going to have a really hard time getting things done.

Even trying to triage everything in queue is difficult, because inevitably, everyone argues valiantly (and rather stubbornly at times) that their issue is the most important and must be looked at first. And most have valid reasons: client impact, loss of functionality, etc. The problem is, there are so many to pick from at a given time that we simply cannot take “My problem has client impact so it must be looked at ASAP” at face value. Otherwise, we’d be constantly shifting in work and never able to focus on our larger goals. Here again, I’m really hoping next month’s meeting will help to clarify and ease this nasty task. At the very least, we need to have some kind of objective framework by which to determine the true priority of requests / issues, similar to the various trauma scales I learned years ago as an EMT.

Yes, I really did just compare my work to dealing with traffic accidents, gunshots, and various other forms of critical illness. The way things are around here, you’d think we were really dealing with life and death scenarios, instead of mere dollars and cents.

Before I was a DBA I used to bother them. It always seemed like they never had time for my work, never wanted to come to my meetings, never wanted to be a part of my projects. It was awful and I cursed them.

Now that I am a DBA, however, I’m finding the roles starkly reversed. My inbox is being pounded with requests to attend meetings; people are constantly walking up to my desk insisting that their project is the most important thing on my schedule, and that I must look at it immediately. I’m quickly developing a ruthless demeanor when it comes to these types of things. The hard truth is, I have to be ruthless, or I would never get anything done. I truly believe that is the case, and I have a whole new level of respect for the other guys on the team because of what they’ve had to deal with.

The good news is I have a good base of skills with which to wade through the river of work and choose what to do in a methodical and measured fashion. I’m becoming even more disciplined with my “not in my list, I don’t do it” rule, and I have mini reviews several times per day at a minimum. If at any point I’m feeling flustered or out of control, I simply take a deep breath, refocus, and look through my lists once again. It’s so simple and yet so effective; why do anything else?

So in a bit of personal news, I was recently moved into a new role at work, namely that of a full time database administrator. In my previous role I had quite a bit of interaction with my new team, and gathered a decent amount of knowledge around the workings of SQL Server, at least from the application interacting parts.

It’s been two weeks now, and I must say the change is startling. Already I’ve learned more than I have in the last six months in my old role. There is always something new to learn about SQL, be it the inner workings of the underlying engine, the new features such as compression, encryption, and partitioning (not a new feature, but made much better from what I can see), or digging into some new problem we are facing. I am kept constantly busy, with new demands being thrown at me at a pace even higher than the one I previously faced. A month ago I would have said that wasn’t possible; now I see that I was barely seeing the tip of the amount of effort these guys put into their job when I was working with them.

I’m blessed to have a team of seasoned veterans to work with. Combined they have more than 30 years of experience with SQL Server, networking, server administration, and other areas of IT. They have welcomed me in and made sure to teach me at every possible turn. I’m tremendously grateful to them, and can’t wait to continue learning.

As I do learn and grow into this new job, I thought it might be interesting to post my thoughts and “lessons learned” as time goes on. Who knows; maybe years from now when I myself am a grizzled old pro, I’ll look back on these musing and laugh.