A Wandering Mind

I think pretty much everyone who knows me would say that one of my defining characteristics is my singular focus. Once I put my mind to solving some problem or accomplishing something, I damn well poke at it until it’s finished. Now, in many cases this trait has served me quite well; it’s what allowed me to learn programming and SQL Server with no formal training or even so much as a seminar (thank you Internet and amazon.com), and why I consider myself an above average problem solver.

One of my favorite teachers once told me that at times I “missed the forest for the trees.” He went on to explain that at times my work tended to find it’s conclusion too early, then ignore all other points of view. Narrow focus is often good in writing, but not when it excludes other angles of approach completely.

The same could be said of how I function as a DBA. Take a project I am working on: the design and build of an enterprise data warehouse, storing performance data for every SQL server in our environment. It’s been fascinating and a tremendous learning experience, but also at time a real challenge.

Early on one of the requirements was to collect performance data on a second-by-second basis, essentially forming a “baseline” of data to review and use for later comparison. Now being the aggressive lad I am, I expanded upon this as follows: the warehouse should now store second-by-sec0nd data for every server at all times, with a retention of 2 years. Yes, you read that correctly. Two years of data, with points at every second for fifty plus servers. That’s a lot of data folks (think 1,800 rows per minute, per server, or around 129,600,000 rows per day).

So, I set out to prove it could be done. Here’s a brief list of some of the challenges I’ve hit along the way:

• PerfMon’s built-in direct-to-SQL functionality scales downright awfully. The structure of the tables would make any decent DBA shudder (the primary key of the main, and largest, table has a GUID as it’s first column, as an example), the use of bulk inserts (not using MS’s own optimization schemes), and statistics issues to name a few.
• After abandoning that approach we are going with dumping the performance data into flat files, which are then picked up by a SSIS package at intervals, which required a custom script component acting as a data flow source. This is due to the fact that PerfMon records its data in .csv files with a dynamic and unpredictable number and order of columns. Custom logic had to be written to essentially perform a reverse pivot of the data into a more normalized form.
• Even fully optimized, the package simply could not keep up with the incoming flood of files. They would eventually pile up and consume disk space, not to mention having stale data in the warehouse. This was attacked by custom coding a script task in a parent SSIS package that would dynamically spawn multiple threads running a child package to import one file each, up to ten at once (I’m shocked by the way that this isn’t either a standard part of SSIS or that a suitable aftermarket component could not be found). Naturally this promptly brought the server to its knees, being a poor little VM with only four CPUs. But the files did import faster!

Now, throughout all of this, keep in mind that I was going far beyond the original and still stated requirements given to me. Why? Because dammit, I was out to prove that such a huge amount of data could be handled by a little VM running SQL 2008 Standard Edition.

Well, today I cried Uncle. In an effort to actually get some usable data into the system, we’re (read, I’m) reducing our goals to only baselining one server at a time, with others collecting a mere four data points per minute. And damn, I am disappointed we haven’t gotten there yet. But you can believe I’m not giving up. But at the same time, I look back and consider all the time I spent chasing what, pride?

My old teacher was right: I need to stop staring at the one big tree in front of me and enjoy the woods a bit more. Luckily, there’s this little methodology I follow called GTD, and something tells me it’s going to be very useful in this upward movement of focus.

More often than I’d like, I get e-mails from various folks late in the evening (the record is about 11pm) somewhere along the line of: “This is really urgent and needs to be done EOD today.” Now, I have to assume that either one of two things is going on here:

1. Their End Of Day is different than mine. Not in the time zone sense (though I do occasionally deal with folks offshore), but just in the sense of they work hours beyond the normal workday.
2. They’re being clever and trying to be the first mail in my inbox the next morning.

Now in the case of #1, obviously they’re not readers of my blog. Otherwise, they’d see my prior post where I was pretty clear about my policy with regards to working after hours: high value and emergency work only. That’s not to say there have not been some cases where the work met one or both of those criteria; just merely that most of the time it doesn’t.

In the case of #2, for one I’d laugh because with the sheer volume of e-mail I get, no matter where you end up it’s most likely not at the top of anything. That, and even if you do somehow manage to be the first thing I read in the morning, that in no way affects where in the queue of actions you’ll end up. Guess they’re not subscribers to the GTD style of organization, huh?

Recently I’ve started using the “Estimate” tag on RTM as a way to organize tasks by how long I think they will take. I’ve found it’s been a real help in deciding what to do at a given time, especially when I have short breaks in between meetings or larger tasks. After trying it out for a bit, I quickly decided that it was now a rule that all recorded next actions must have a time estimate attached to them.

One of the really nice features of RTM is its flexible search system. For example, let’s say I had about 20 minutes to spare, and wanted to see if there was any low hanging fruit available. I simply type in “timeEstime:’<20min’ and voila, a list of tasks that I could easily get out of the way. I’ve found it to be quite a useful trick when faced with 15 – 30 minutes of useful time on my hands. It doesn’t happen that often, but I’ve found that it definately has decreased the number of simple one-off tasks malingering on my lists.

So what do you think? Are time estimates worth their effort, and helpful in planning your daily work? Or do you find them inaccurate and a waste of time, especially for small, low-level tasks?

Today was my first day back from work after nearly a week off.  Surprisingly, I had only around 300 e-mails to process in my inbox; I usually receive that amount each day, not counting automated alerts and the like.  After reviewing everything and making sure all my lists were up to date, I began working on what was the most important, urgent matter at the time.

By about 9:30AM, several fires had appeared on the horizon.  Each demanded much attention from my team, and it quickly became obvious that my day would not be very productive, at least from the perspective of crossing off a lot of items.

In the past situations such as this would cause my blood pressure to rise almost instantly.  Not because of any physical danger (“When Servers Attack” anyone?), but because subconsciously, I immediately began worrying about what I was not doing.  That is, if I was spending all my effort to correct some immediate problem, what was getting implicitly pushed off?

So what happened today?  Certainly at first, I felt a slight twinge and a brief rise in my pulse.  But after taking a breath and calmly reviewing everything in my “Next Actions” list, I was able to definitely know that I was indeed directing my time appropriately.  The feeling was one I am certainly not used to!

This state of being was possible only because of the following pre-existing conditions:

1. I have been incredibly good in terms of making sure that each and every possible “open loop” is in my trusted system, no matter how small or how insignificant it may seem.
2. I have developed the habit that whenever one of those “it needs to be done now” (usually stated by whomever reports the problem) type of problems hits my plate, I do the following:
   1. Use soothing words and a little empathic listening to get the person to remain calm.
   2. Ask pointed questions about the true severity of the issue. How many, what’s the workaround, etc.
   3. Take note of all the information for future reference. Moleskins are great for this.
   4. Politely let the person know that I am going to review the problem with my team and get back to them when more information is available, and as soon as time permits.
   5. If I believe the problem is truly urgent, I’ll immediately decide on a next action and put it in my list. Otherwise, I’ll throw some reminder into my “in bin” for later perusal.
3. I review my lists with near religious regularity, constantly reminding myself to not lose sight of the larger picture.

All of these combine to enable me to do my daily work with agility and perspective, such that at any given point, I feel very confident that I am doing exactly what  I should be.

The most difficult part?  Maintaining an even keel and and objective frame of reference in the face of perpetual problems.  That is a skill that I am still very much honing, and one that can only be developed through daily practice and a strong sense of your emotions.

How do you maintain a broad view of your responsibilities, while still responding to the fires that fall on you unexpectedly?