A Wandering Mind

I was just sitting here looking over some old code projects of mine, and thinking “Man this stuff is junk. How’d I write this?! And why’d I waste my time on it?” These included a Java library for parsing tweets for data mining (boy that was going to be a great open source project), my first .NET application that was for work and used Microsoft Access as a back-end, and various other half-started works that never really took hold. At first this was rather discouraging, since it reminded me of my genetic pre-disposition to not following through on my projects (a topic for another day for sure).

Then it hit me: it’s only by writing all this crap (and believe me, most of it really is steaming piles of crap), trying out different routes and ideas, and ultimately letting them fall off that I’ve been able to improve my skills as much as I have. Picture the proverbial writer sitting at the typewriter with a pile of crumpled papers next to them, head in hands. But then, one day, something clicks, and out comes a masterpiece.

It’s not important that all we produce is wonderful, glittery, and perfect. No, what’s really crucial is that we keep going and pushing ourselves, especially when it seems like all we churn out is junk. That junk is gold, because it’s what teaches us to do better. As long as we keep learning from our mistakes and bad ideas, then we grow as professionals and human beings. And sooner or later, you might just produce that golden egg.

It’s that time again, when SQL bloggers from all over join forces to blog for a common theme. That’s right: T-SQL Tuesday!

Background: T-SQL Tuesday is the brainchild of Adam Machanic (Blog|Twitter) and is described as “the SQL Server blogosphere’s first recurring, revolving blog party. The idea is simple: Each month a blog will host the party, and about a week before the second Tuesday of the month a theme will be posted. Any blogger that wishes to participate is invited to write a post on the chosen topic.”

This month’s theme: “Beach Time”, hosted by Jason Brimhall (blog | twitter). Basically, we’re being asked to write about “What do you do prior to ‘Beach Time’ to ensure that the beach time will not involve work.”

This is actually a good topic for me, as last month I took the longest consecutive vacation of my working life at two full weeks. It was challenging to get everything in order before leaving, and I did get a phone call once (ironically on the first day off), but all in all my team did great work and kept all my projects in good order while I was gone (with one minor exception, but that was at least partially my fault as you’ll see).

So how did I pull this extended absence off?

I documented the status of all my projects, including what was outstanding from others and what needed to be done while I was gone.

We have a wonderful SharePoint site where we keep all our projects listed and have detailed status notes on all of them. This means that if any one of us ever had to suddenly be out for an extended period of time, someone else could just glance over their list and pick up where they left off.

This reminds me strongly of one of my GTD habits: keep things in lists and not in your head. By forcing us (and yes, I do sometimes neglect it a little, hence my first (but only) phone call) to keep this updated, we don’t have to rely on our memory to keep track of what’s on our plates.

I made sure my runbooks were up to date.

Just before I left, I closed out a major phase of a project to build a massive data warehouse containing PerfMon data for all the SQL servers we support. It’s a complex system with multiple inputs and moving pieces. Because of this, I made sure the write up a thorough document detailing the ins and outs of the system, including architecture and troubleshooting steps. No way was I being called when I was away because this thing broke.

Now as it turns out, the system did fail while I was gone, but was unnoticed! And while my teammates sheepishly admitted that they missed the alerts and should have at least dug into them a little further, I also took responsibility for not ensuring they (the alerts) were a) urgent enough to be noticed, b) clear enough to tell the reader where to get more information.

Once I was gone, I made damn sure I was very hard to get in touch with.

While I am the first to admit I have a pretty bad addiction to my Crackberry, when I’m on paid vacation I strictly enforce the “no e-mail rule”. I go so far as to actually turn off the mail synchronization on my phone and make it clear in my out of office that I will not be checking my mail at any time during my absence. Only my teammates know my phone number and they are told they can call me if absolutely necessary, but that I may not immediately respond. While I do tend to take my work laptop, that’s only because it’s also my playground for testing SQL related things, and I do not connect to the office.

Being selfish? Maybe a little, but hey, it’s my vacation time and I’m damn well going to enjoy it and not waste it working. That being said, on this last vacation I did end up working for about 3 hours, but that was wholly my fault for not properly documenting something before I left.

I have a great, strong team around me.

While this may not have much directly to do with me, I consider it a crucial factor in vacation success. If your team can’t hold down the fort, even given good documentation, then you’re pretty much sunk. I’ve been in those situations before, and let’s be honest: it sucks. There’s few things in the world more maddening than getting called on vacation because “Mr. So and So is here and he says you told him X, but that doesn’t make sense?”, especially when your conversations with Mr. So and So are explicitly detailed in writing, and readily accessible to your teammates. That’s a real example by the way, from my time as a retail store manager. Let’s just say that individual got politely redirected to where I kept a log of all my conversations, along with a gentle reminder that I was not to be disturbed. Thankfully, this is not the case where I am now, and I am eternally grateful for the hard work and dedication of my fellow DBAs.

So today I spent basically the entire day working on troubleshooting an issue that is related to a little SQL Server technology called MemToLeave. Now I’m not going to go into extreme technical detail on what exactly this is; instead I’ll direct you towards some other excellent resources:

http://sqlblogcasts.com/blogs/christian/archive/2008/01/07/sql-server-memtoleave-vas-and-64-bit.aspx

http://sqlblog.com/blogs/jonathan_kehayias/archive/2009/07/07/understanding-the-vas-reservation-aka-memtoleave-in-sql-server.aspx

http://blogs.msdn.com/khen1234/archive/2005/05/08/415501.aspx

Basically MemToLeave is an area of reserved memory outside of the buffer pool, which is commonly used for such operations as extended stored procedures, CLR, and linked servers. It only affects 32 bit systems, is limited to a maximum amount of 512MB, and can be set using the “-g” startup parameter.

In my short experience as a DBA, this is far and away the most complex and mind-bending piece of investigative work I’ve done by a landslide. After spending nearly three hours Googling  and digging into various DMVs, I had made very little headway. This is real down-the-rabbit-hole stuff; we’re talking about understanding and trying to determine what is happening deep within the internals of SQL Server. Even my senior DBA, who has a tremendous wealth of knowledge and more than a decade of experience, was largely at a loss to explain what was going on. I even went so far as to use the great Twitter #sqlhelp hashtag (and got some good information, thanks David Levy (twitter | blog)) to try my best to get some insight. As it turned out, the Twitter community helped to push us in what we believe to be the right direction.

While I’ll leave the technical details for another post, our basic conclusion after nearly a day’s worth of work is this: due to a tremendous amount of statements in the procedure cache (the application using this DB makes a maddening number of adhoc calls, in such a way that each is stored as its own plan, even with forced parameterization on), and the fact that a large number of these plans were stored in multi-page arrangements (see Jonathan’s blog post above), MTL became suddenly and rapidly depleted. Normally we’d be forced to stop and start SQL in order to clear this up, but in this case, all we ended up having to do is to a) wait for the application to cease its (adhoc) insert heavy behavior, then run a DBCC FREEPROCCACHE. Before running the command, the return result of xp_memory_size was around 3MB; after it immediately jumped to ~21MB. Also, we saw that the CACHESTORE_SQLCP memory clerk type went from consuming around 180MB to around 1/100th of that (we used the queries on Brent McCraken’s blog here, plus some home-brewed ones of various DMVs). The server stayed quiet the rest of the day.

While I’m far from convinced, the evidence is pretty strong that we’ve found at least a contributing factor. The next bulk series of imports into the system happens overnight, so in the morning we’ll be checking in to see how we are doing.

That’s all for now, but once we have more data I’ll probably be posting a followup with the exact queries and results. What a day!

More often than I’d like, I get e-mails from various folks late in the evening (the record is about 11pm) somewhere along the line of: “This is really urgent and needs to be done EOD today.” Now, I have to assume that either one of two things is going on here:

1. Their End Of Day is different than mine. Not in the time zone sense (though I do occasionally deal with folks offshore), but just in the sense of they work hours beyond the normal workday.
2. They’re being clever and trying to be the first mail in my inbox the next morning.

Now in the case of #1, obviously they’re not readers of my blog. Otherwise, they’d see my prior post where I was pretty clear about my policy with regards to working after hours: high value and emergency work only. That’s not to say there have not been some cases where the work met one or both of those criteria; just merely that most of the time it doesn’t.

In the case of #2, for one I’d laugh because with the sheer volume of e-mail I get, no matter where you end up it’s most likely not at the top of anything. That, and even if you do somehow manage to be the first thing I read in the morning, that in no way affects where in the queue of actions you’ll end up. Guess they’re not subscribers to the GTD style of organization, huh?

Brent Ozar (blog | @BrentO) wrote a great post the other day that I just read that touched on the oft-seen differences between management’s priorities and our own. Even having only been a DBA for a mere month (damn, has it been that long?) I’ve already encountered situations like this all too frequently.

Troubleshooting and digging into data are two of my favorite things to do when it comes to DBA work. I love pulling numbers out and slicing-and-dicing them every which way until a pattern emerges. In this light, I’ve been really excited lately about the prospect of building a kind of DBA data warehouse: a place where aggregated performance statistics from all our servers are kept and can be analyzed in detail. It’s not a novel concept by any means, just one that’s never been implemented at work. With this in place we could engage in detailed benchmarking of our environment, and proactively find developing “hot spots” before they become real fires.

The trouble is, we can never seem to find time to work on it. There’s always other things to work on, whether it be dealing with issues or building more servers to add capacity. This is really tough to handle because as a team, we pride ourselves on keeping things running at peak efficiency, like a well-trained pit crew for a championship racing team. We can clearly see the value in this project and how much it would help us.

But all that being the case, I won’t be spending my free time working on that. If I do spend time outside of business hours working (something I try to avoid whenever possible), it’ll be on those other, more important items.

Yes, you read correctly: “more important”. “But Josh”, you say, “you just said that stability is your team’s top priority, and to boot that you find your biggest intiative in that area incredibly exciting. Why the heck are you calling the other things ‘more important’?”

The answer is simple: because that’s what my clients want from me. Like it or not, right now the word is that the primary focus must be on building for new initiatives and responding rapidly to the changing needs of our industry. Granted that stability was ranked right behind that, and I am certainly going to stake a case to management about the need for some time to be allocated to our bechmarking project. But as I commented on Brent’s post (modified slightly):

I think in this case you have two ways to go: you can continue to play the role of the a-hole that is always arguing about priorities and complaining that he has to work on weekends to get his “important” work done, or you can accept that your client’s priorities are what they are, and try and work within them. Well, I guess there’s a third choice: quit and start blogging about SQL Server full time, but most of us probably aren’t that brave in this economy.

So thanks for the reminder Brent; while I sure do love building warehouses and crunching numbers, I won’t be doing any of that outside of 8AM – 5PM Monday – Friday anytime soon.

We’ve all had this happen to us at one time or another. You wake up in the middle of the night, thoughts rushing through your head at a mad pace. You try to take the zen-like approach of “letting them go”, but it’s hopeless. You toss and turn, but the harder you try to sleep, the more awake you are. This, friends, is what I call “Racing Brain Syndrome”.

There are two main forms of this nasty little bug, which we’ll call “Stress Induced” and “Excitement Based”. In this post, we’ll look at the first variety in more detail.

Stress Induced

As the name implies, this version is caused by an excess of built up stress that has yet to be dealt with. Common symptoms (not inclusive of the other variant of this syndrome) include racing pulse, pounding heartbeat, cold sweats, and possibly (in extreme cases) delusions of persecution or general paranoia.

Now stress, as you well know, can come from many sources, including the practice of keeping things in your head, nagging concerns over projects left un-planned, fear of upcoming regulatory audits (a favorite of us IT folks), and of course the ever present conflict between the Ego and the Id caused by an underlying need for affection, complicated by an Oedipus complex.

Whoops, I must apologize for that last one. This post has me reverting to my old psychobabble style of writing. Ignore that one, will you please?

When dealing with this variety of RBS, one’s best course of action is tri-fold:

1. Determine if the cause of the stress is a rational one. That is, are you feeling stressed because you’ve fallen off your good practice of keeping things out of your head, or are you suddenly having a sinking feeling that you’ve left your torrent bot up and running at work, and the folks from InfoSec are, at this very moment, hot on your trail? Ok, that’s an extreme example, but you get my drift.
2. If the former, your best bet is to take a few minutes and put some thoughts down on paper around what is bothering you. You don’t have to answer every question out there; just make sure every question is written down so you are confident it won’t get lost in the shuffle.
3. If the latter, you would be advised to fall back on a technique I used to teach to the children at the mental health clinic I worked at out of college, called (in it’s most complex form) Cognitive-Behavioral Therapy. The basic idea is that you write down the thoughts that are causing you discomfort, such as “I’m afraid the ninjas from security are after me”, then examine them in a critical, analytical fashion. For instance, what evidence is there that you’re really about to be attacked in your sleep? Do you even have a torrent bot on your work computer? These techniques are usually used by patients with more severe mental health issues such as depression or anxiety disorder, but they serve RBS sufferers equally well.

On a sidenote, if you’re actually interested in a more clinical view of CBT, I’d heartily recommend the book The Feeling Good Handbook by a fellow named David Burns.

Next time we’ll examine the milder, and perhaps more pleasant variety of RBS, “Excitement Based”.

Recently I’ve started using the “Estimate” tag on RTM as a way to organize tasks by how long I think they will take. I’ve found it’s been a real help in deciding what to do at a given time, especially when I have short breaks in between meetings or larger tasks. After trying it out for a bit, I quickly decided that it was now a rule that all recorded next actions must have a time estimate attached to them.

One of the really nice features of RTM is its flexible search system. For example, let’s say I had about 20 minutes to spare, and wanted to see if there was any low hanging fruit available. I simply type in “timeEstime:’<20min’ and voila, a list of tasks that I could easily get out of the way. I’ve found it to be quite a useful trick when faced with 15 – 30 minutes of useful time on my hands. It doesn’t happen that often, but I’ve found that it definately has decreased the number of simple one-off tasks malingering on my lists.

So what do you think? Are time estimates worth their effort, and helpful in planning your daily work? Or do you find them inaccurate and a waste of time, especially for small, low-level tasks?

I was having a debate with a developer today about a situation similar to the following:

When reviewing the build for a new SSIS / SSAS deployment, I noticed that because of the way their build was structured, a failure could occur in some critical underlying processes, while on the surface, everything appears normal. <geekery> This was due to 1) a bug in how SQL Agent jobs work in SQL 2005 (see here), and 2) the way their process was calling the SQL Agent job to process an Analysis Services cube. </geekery>

Now, immediately in my mind, this was a clear showstopper. I mean, if I can’t tell for sure if an automated process failed or not, how on Earth am I supposed to support it? I luughed at suggestions such as manually checking on a daily basis or just telling business “it’s broken, but if you don’t care we won’t fix it”. Ultimately, the developer’s final comeback was basically that “I followed conventions of how this was supposed to be set up, so you can’t require me to make this change before we go live.”

Huh? Since when does “it’s broken in other places, so I can leave it broken here” work as a reason not to fix something, especially before it’s actually running for real? How about showing the business we’re looking out for their best interests and ensuring a quality client experience?