A Wandering Mind

A short while ago, I read a great post by Kendra Little (@Kendra_Little on Twitter) on how using some common consulting tricks can be useful even for those not in a consulting role. One item that resonated in particular was the idea of tracking your time, for the purposes of finding places where your time is probably not being used wisely (Kendra uses the example of not planning for disaster recovery, a clear “must-do” for a good DBA). I’ve tried various methods of tracking my time before, including everything from home-grown applications (I suck at GUI programming on a side note) to simple notepad paper. Everything failed miserably for one of two reasons.

It Was Too Hard To Use

My cruddy GUI programming skills aside, I had never found something lightweight enough that I could really use it consistantly without slowing down. In my work I am constantly switching tasks, so whatever method I used needed to not require ten clicks to switch from one task to another. If it was anything less than really easy, I would simply give up after a few days.

It Didn’t Give Useful Data

Recording time is one thing, but of equal importance is the metadata associated with that slice of time. What project was I working on (if there was a defined project, not something that’s always true)? For what client? What “tags” or other little flags were associated with a piece of work (for instance, was my time spent dealing with a “walk-up”, or was it unplanned troubleshooting)? Without this enriched data, simply knowing that I spent from 8 AM to 9:45AM working on task ‘XYZ’ really isn’t very useful.

Well, thanks to Kendra’s suggestion, I’ve finally found a tool that I’m sticking with. Toggl is nothing short of fantastic! It’s easy to use, quick, and allows me to easily tag and classify work without a lot of extra effort. It’s been downright fascinating seeing how my time is really spent, and I’m finding it’s often quite different than what I perceive. I might feel like I’ve spent every waking moment working on a particular effort, only to find that my judgement is skewed simply because I find that particular work unpleasant (nothing to throw off your gut feeling like hating what you’re doing).

Most importantly, it’s enabled true, accurate (at least mostly so) transparency with my superiors and customers. When I report weekly to my boss, I can say with integrity what I have (and conversely, have not) been spending my time (and therefore the company’s money) on. It’s been invaluable in tweaking my workload to better suit our client’s needs, but also in identifying easy automation / delegation opportunities. Clearly a win for both sides!

I was just sitting here looking over some old code projects of mine, and thinking “Man this stuff is junk. How’d I write this?! And why’d I waste my time on it?” These included a Java library for parsing tweets for data mining (boy that was going to be a great open source project), my first .NET application that was for work and used Microsoft Access as a back-end, and various other half-started works that never really took hold. At first this was rather discouraging, since it reminded me of my genetic pre-disposition to not following through on my projects (a topic for another day for sure).

Then it hit me: it’s only by writing all this crap (and believe me, most of it really is steaming piles of crap), trying out different routes and ideas, and ultimately letting them fall off that I’ve been able to improve my skills as much as I have. Picture the proverbial writer sitting at the typewriter with a pile of crumpled papers next to them, head in hands. But then, one day, something clicks, and out comes a masterpiece.

It’s not important that all we produce is wonderful, glittery, and perfect. No, what’s really crucial is that we keep going and pushing ourselves, especially when it seems like all we churn out is junk. That junk is gold, because it’s what teaches us to do better. As long as we keep learning from our mistakes and bad ideas, then we grow as professionals and human beings. And sooner or later, you might just produce that golden egg.

Lately I’ve seen several folks on Twitter talking about “updating their 30, 40, and 50,000 foot GTD goals.”. This made me realize that while I’ve really been cruising along in the day-to-day arena, I’ve pretty much neglected some higher level thinking. When you’re working 9 hour days and the to-do lists are always full, it’s pretty easy to lose sight of considering larger goals and visions. Or at least it is for my ADD riddled brain.

50,000 Feet – Life’s Purpose

If I talk about my big time life goals, I would say at the moment they reside in three categories:

Be a wonderful father and husband (and son) for my family

I want to provide us with enough to live comfortably, but not work so hard as to interfere with spending time with my son and wife. Family time is extremely important to me.

Become a well known and respected member of the SQL Server community

After knocking around for years in various technical roles, I really feel that I’ve finally found my niche. I love working with SQL Server; the internals, the various moving pieces, the never ending amount of information to learn. And as I’ve continued to learn and grow, I’ve really become enamored with the SQL community; not in a weird man-crush kind of way, but in a “wow the combined amount of knowledge and experience present here is amazing” kind of way. I really want to become more engaged in being a part of that.

Become a published author

Ever since I was a kid I’ve loved writing. It’s always been a dream of mine to write a best selling novel. Not one of those cheap drug store romance pieces, but something really deep and dark, that gives its readers a real emotional reaction. I’ve picked up more books this year than any year since high school, and the end result has been a real return of that hunger to create. When I read stories like Ender’s Game and The Deathly Hallows, and watch some phenomenal visual narratives like Battlestar Gallactica and Lost (except for the last season, which I think overall was pretty bad), I’m continuously left with a feeling of “I want to write that.” By no means an easy thing to do, but I’m keeping it on the list.

40,000 feet – 3-5 year goals

Present at least once at a SQL community event

I still consider myself very much a rookie in the SQL Server realm, so I’m giving myself a couple of years yet before I start expecting to be at a level where I can present. I’m constantly coming up with ideas, which are getting filed away in a safe place (i.e. anywhere but my brain) for later use. That way, when I feel I’m ready, I’ll have plenty of material.

MCITP Certification

I’ve only recently begun studying for my 70-433 and 70-432 exams, and while in a lot of cases I’m finding that I already know the material, there’s also a lot to learn. As you’ll see later I’m hoping to pass those two exams within a year, but to get the full MCITP certs I’m giving myself a little longer, both for experience and financial reasons (at least for the moment it looks like I’ll be paying my own way).

Take on some side consulting work

Make no mistake: I love my job and the company I work for, and I’m not looking to quit and move into full time consulting. But I would like to start entering into that world, both for experience and, to be totally honest, the extra money. I don’t know how feasible this will be, since I won’t be able to commit to anything more than 8-12 hours per week. I’m hoping I can find some “Remote DBA” style opportunities, and maybe some performance analysis / tuning ones.

30,000 Feet – 1 year goals

Servers at 100% Patch Level

When I inherited my current environment of around 35 SQL Servers, one of the first things I did was to assess their health. Overall I found them in pretty poor condition, especially in the realm of patch level. Many were at least one major service pack behind, which is completely unacceptable. This year, that will change.

Build a complete replica of production

Right now our development environment is scattered and disjointed; some servers have many components (i.e. Analysis Services, Reporting Services), while others have only the Database Engine components. This does not make for valid testing, since it looks nothing like production. By the end of the year, that will change, and I will have a complete setup (including network topology) mirroring our production systems.

Start a SQL community at work

Because I’m very much a “teach a man to fish, don’t catch the fish for him” kind of guy, I want to start helping to empower the folks around me to be better SQL Server users. I’ve decided to do this via series of learn-over-lunch sessions where we’ll all get together and study up on the latest trends in SQL. Sometimes we’ll just be viewing webinars, others we might jointly look over some troublesome code and decide how to fix it. Having already had one such session the response so far has been very positive, and I’m excited to keep on learning.

---

That’s about it for now. I’m sure things will come on and move off over the course of the year, but I’m really going to try and tie my actions to these higher goals as much as possible. From time to time I think I’ll check in and see where I stand, maybe once a quarter.

What are your big goals for the year?

Update: Mike sent me a link to an article he wrote for SQL Server Central on this same subject, and I wanted to include it here: http://www.sqlservercentral.com/articles/Troubleshooting/66134/

When I was sixteen I became one of the youngest EMTs in the state of Pennsylvania. I was proud to volunteer my time, and found I greatly enjoyed helping people. It was both incredibly humbling and empowering at the same time: on one hand, there were times when you were utterly helpless and could do nothing to help your patients, but other times you clearly saw the immediate impact you were able to have. I can say without question my experiences, both good and bad, changed me as a person.

Fast forward fifteen years, and I find that fellow SQL tweeter Mike Walsh (blog | twitter) also has some background in emergency services. We had a nice little chat about our experiences and how they ultimately help us in our daily work. Thus, I became inspired to write about how my time riding around in an ambulance helps me be a better DBA.

Checklists And Procedures Are Essential

When we arrived on the scene of a medical emergency, we were drilled ruthlessly about a series of steps to assess our patient’s condition in rapid fashion. One of the first tasks was what we called the ABCs- Airway, Breathing, and Circulation. Basically, is the patient have a clear airway (choking, etc), are they breathing, and do they have a pulse. It was simple and easy to remember; two essential things when you’re in an adrenaline charged situation with someone’s life on your hands.

Once we were past the initial basics, we had a written list of additional information to gather: current medications, allergies, existing conditions, injuries, etc. Personally I carried that pad of paper with me in my pocket at all times while on duty, and used it without fail on every call. I don’t know about you but I would not trust someone’s life to my memory.

The same should be true about dealing with SQL Server emergencies (I use that term loosely as you’ll see later). Do you have a documented, step-by-step process to perform an assessment of the issue at hand, and take corrective measures? Is it readily available, clearly written, and has it been practiced? What about recovery procedures? If you get paged at 3AM (after a night at the bar of course) and have to perform a restore of a corrupted page, will you be going off memory or will you be walking down your checklist? Heck, forget you, what about that junior DBA who joined your team a month ago and is taking his first on-call rotation? Do you really want him flying by the seat of his pants?

Sometimes A Calm Voice Is The Best Medicine

I once was called out to a house where we found a young boy who had fallen and broken his arm. Immediately it was apparent that the injury was fairly minor (at least as minor as a broken limb could be), and that the situation was being made far more chaotic by the boy’s hysterical parents. They were yelling and screaming at each other, blaming one another for letting this happen, yelling at us for not taking proper care of their son, etc.

That night it happened that my driver was also my supervisor, a man with over ten years of experience in EMS. He quickly brought the two parents together off to the side, and said in a quiet but authoritative voice, “Ma’am, sir, I promise you we are doing everything we can to help your boy out. What I need from you two right now is to listen carefully to my medic’s questions so we can get all the information the doctors will need. Can you do that for me please?”

That simple request, combined with his soothing tone of voice, settled them instantly and brought them out of their hysteria. Once they had something to focus on they were cooperative, and we were able to get the information we needed and transport the boy to the hospital for treatment.

Similarly, the next time someone (be it a developer, a business user, or whomever) runs up to your desk in a panic, try speaking to them in a calming manner. Assure them that everything will be done to fix their situation, then ask them to help you by answering some simple questions. Is the application totally down, or is there just some functionality that is not working? How many users are affected? Are there any financial or other risks present due to the ongoing presence of the issue? In a lot of cases, those answers may even prove that the situation really isn’t as urgent as they thought.

It’s Not Life And Death

Mike and I both agreed strongly on this one: our time as volunteers gave us a healthy dose of perspective when it came to dealing with IT “Emergencies”. After all, it’s not like we’re dealing with broken bones, blood, or the prospect of a patient dying.

I don’t say this to belittle the problems we deal with as DBAs. We are paid to take care of our users and our servers, and we need to take our work seriously. But at the same time, I think we need to make sure we don’t lose track of the bigger picture of life happening around us. A server going down is not worthy of raising your blood pressure, nor is dealing with “that annoying developer” again worth losing your cool. Ultimately, life will go on, the sun will come up, and you can go home to hug your loved ones at the end of the day.

Note: I do want to single out and commend those of us who truly support mission-critical platforms, such as those in hospitals, power plants, etc, where lives may really be at stake. You have my utmost respect as a fellow data professional.

Maybe some day I’ll return to volunteering my time to help those who have become victims of life. In the mean time, I take solace knowing that the lessons I learned in my time in EMS are with me today, helping me to truly be a better DBA.

I think pretty much everyone who knows me would say that one of my defining characteristics is my singular focus. Once I put my mind to solving some problem or accomplishing something, I damn well poke at it until it’s finished. Now, in many cases this trait has served me quite well; it’s what allowed me to learn programming and SQL Server with no formal training or even so much as a seminar (thank you Internet and amazon.com), and why I consider myself an above average problem solver.

One of my favorite teachers once told me that at times I “missed the forest for the trees.” He went on to explain that at times my work tended to find it’s conclusion too early, then ignore all other points of view. Narrow focus is often good in writing, but not when it excludes other angles of approach completely.

The same could be said of how I function as a DBA. Take a project I am working on: the design and build of an enterprise data warehouse, storing performance data for every SQL server in our environment. It’s been fascinating and a tremendous learning experience, but also at time a real challenge.

Early on one of the requirements was to collect performance data on a second-by-second basis, essentially forming a “baseline” of data to review and use for later comparison. Now being the aggressive lad I am, I expanded upon this as follows: the warehouse should now store second-by-sec0nd data for every server at all times, with a retention of 2 years. Yes, you read that correctly. Two years of data, with points at every second for fifty plus servers. That’s a lot of data folks (think 1,800 rows per minute, per server, or around 129,600,000 rows per day).

So, I set out to prove it could be done. Here’s a brief list of some of the challenges I’ve hit along the way:

• PerfMon’s built-in direct-to-SQL functionality scales downright awfully. The structure of the tables would make any decent DBA shudder (the primary key of the main, and largest, table has a GUID as it’s first column, as an example), the use of bulk inserts (not using MS’s own optimization schemes), and statistics issues to name a few.
• After abandoning that approach we are going with dumping the performance data into flat files, which are then picked up by a SSIS package at intervals, which required a custom script component acting as a data flow source. This is due to the fact that PerfMon records its data in .csv files with a dynamic and unpredictable number and order of columns. Custom logic had to be written to essentially perform a reverse pivot of the data into a more normalized form.
• Even fully optimized, the package simply could not keep up with the incoming flood of files. They would eventually pile up and consume disk space, not to mention having stale data in the warehouse. This was attacked by custom coding a script task in a parent SSIS package that would dynamically spawn multiple threads running a child package to import one file each, up to ten at once (I’m shocked by the way that this isn’t either a standard part of SSIS or that a suitable aftermarket component could not be found). Naturally this promptly brought the server to its knees, being a poor little VM with only four CPUs. But the files did import faster!

Now, throughout all of this, keep in mind that I was going far beyond the original and still stated requirements given to me. Why? Because dammit, I was out to prove that such a huge amount of data could be handled by a little VM running SQL 2008 Standard Edition.

Well, today I cried Uncle. In an effort to actually get some usable data into the system, we’re (read, I’m) reducing our goals to only baselining one server at a time, with others collecting a mere four data points per minute. And damn, I am disappointed we haven’t gotten there yet. But you can believe I’m not giving up. But at the same time, I look back and consider all the time I spent chasing what, pride?

My old teacher was right: I need to stop staring at the one big tree in front of me and enjoy the woods a bit more. Luckily, there’s this little methodology I follow called GTD, and something tells me it’s going to be very useful in this upward movement of focus.

More often than I’d like, I get e-mails from various folks late in the evening (the record is about 11pm) somewhere along the line of: “This is really urgent and needs to be done EOD today.” Now, I have to assume that either one of two things is going on here:

1. Their End Of Day is different than mine. Not in the time zone sense (though I do occasionally deal with folks offshore), but just in the sense of they work hours beyond the normal workday.
2. They’re being clever and trying to be the first mail in my inbox the next morning.

Now in the case of #1, obviously they’re not readers of my blog. Otherwise, they’d see my prior post where I was pretty clear about my policy with regards to working after hours: high value and emergency work only. That’s not to say there have not been some cases where the work met one or both of those criteria; just merely that most of the time it doesn’t.

In the case of #2, for one I’d laugh because with the sheer volume of e-mail I get, no matter where you end up it’s most likely not at the top of anything. That, and even if you do somehow manage to be the first thing I read in the morning, that in no way affects where in the queue of actions you’ll end up. Guess they’re not subscribers to the GTD style of organization, huh?

Recently I’ve started using the “Estimate” tag on RTM as a way to organize tasks by how long I think they will take. I’ve found it’s been a real help in deciding what to do at a given time, especially when I have short breaks in between meetings or larger tasks. After trying it out for a bit, I quickly decided that it was now a rule that all recorded next actions must have a time estimate attached to them.

One of the really nice features of RTM is its flexible search system. For example, let’s say I had about 20 minutes to spare, and wanted to see if there was any low hanging fruit available. I simply type in “timeEstime:’<20min’ and voila, a list of tasks that I could easily get out of the way. I’ve found it to be quite a useful trick when faced with 15 – 30 minutes of useful time on my hands. It doesn’t happen that often, but I’ve found that it definately has decreased the number of simple one-off tasks malingering on my lists.

So what do you think? Are time estimates worth their effort, and helpful in planning your daily work? Or do you find them inaccurate and a waste of time, especially for small, low-level tasks?

Today was my first day back from work after nearly a week off.  Surprisingly, I had only around 300 e-mails to process in my inbox; I usually receive that amount each day, not counting automated alerts and the like.  After reviewing everything and making sure all my lists were up to date, I began working on what was the most important, urgent matter at the time.

By about 9:30AM, several fires had appeared on the horizon.  Each demanded much attention from my team, and it quickly became obvious that my day would not be very productive, at least from the perspective of crossing off a lot of items.

In the past situations such as this would cause my blood pressure to rise almost instantly.  Not because of any physical danger (“When Servers Attack” anyone?), but because subconsciously, I immediately began worrying about what I was not doing.  That is, if I was spending all my effort to correct some immediate problem, what was getting implicitly pushed off?

So what happened today?  Certainly at first, I felt a slight twinge and a brief rise in my pulse.  But after taking a breath and calmly reviewing everything in my “Next Actions” list, I was able to definitely know that I was indeed directing my time appropriately.  The feeling was one I am certainly not used to!

This state of being was possible only because of the following pre-existing conditions:

1. I have been incredibly good in terms of making sure that each and every possible “open loop” is in my trusted system, no matter how small or how insignificant it may seem.
2. I have developed the habit that whenever one of those “it needs to be done now” (usually stated by whomever reports the problem) type of problems hits my plate, I do the following:
   1. Use soothing words and a little empathic listening to get the person to remain calm.
   2. Ask pointed questions about the true severity of the issue. How many, what’s the workaround, etc.
   3. Take note of all the information for future reference. Moleskins are great for this.
   4. Politely let the person know that I am going to review the problem with my team and get back to them when more information is available, and as soon as time permits.
   5. If I believe the problem is truly urgent, I’ll immediately decide on a next action and put it in my list. Otherwise, I’ll throw some reminder into my “in bin” for later perusal.
3. I review my lists with near religious regularity, constantly reminding myself to not lose sight of the larger picture.

All of these combine to enable me to do my daily work with agility and perspective, such that at any given point, I feel very confident that I am doing exactly what  I should be.

The most difficult part?  Maintaining an even keel and and objective frame of reference in the face of perpetual problems.  That is a skill that I am still very much honing, and one that can only be developed through daily practice and a strong sense of your emotions.

How do you maintain a broad view of your responsibilities, while still responding to the fires that fall on you unexpectedly?