A Wandering Mind

Lately at work I’ve been on a bit of a SQL kick, as I ease into a new role as “Data Process Expert”. While this won’t totally replace the other functions of my job (project / implementations manager, production support specialist, occasional code geek), I do see it becoming a large share of my time. Analysis / Reporting Services is a fairly hot topic these days, since we’re engaging in a concerted effort to centralize all reporting in the enterprise. It’s a young initiative, sure to have some bumps along the way, but I am enjoying being a part of it.

Anyway, I thought I would post my thoughts on two new skills I’ve been working on: Analysis Services design and Data Warehousing. If trends at work are indicative of anything elsewhere, BI and data analysis are on a lot of people’s lists. Being a total newbie, I think I’m coming in with a bit of a fresh perspective.

This time I’ll be talking about Data Warehousing. In no particular order, here are my thoughts:

• Doing DW well is really tough. You have to have both a really good picture at a 30,000 foot level of where all your data is coming from, as well as a runway level view to understand the structure and format of individual fields.
• Depending on your sources, you may spend a good amount of effort just cleaning up invalid data. In my case, I was aggregating metadata from several document management systems, and when forced into a system with real referential integrity, I was amazed as how many holes there were. Assume your data has gaps and design ways to handle them, whether it’s ignoring them or adding them to some “unknown” category.
• If you’re aggregating data from different instances of the same software, think from the start about how your going to do so. For instance, are you going to want to separate the data for analysis, or will you want to see it all at once (or both)? Will you load it all in one shot, or separately?
• This is blurring the lines a bit between this post and a future one (Analysis Services), but determining how granular you want your data to be in terms of time aggregation should be an early task. I had to basically redesign my extract process about halfway through after figuring out one of my data points would be better seen at a minute by minute basis.
• Once you make that decision, put specific steps in your ETL process to ensure your data fits within those boundaries. You can’t assume because you see the first thousand rows in neat 30 minute intervals that all the data meets that standard. Again, spoken from late game experience.

Overall I’ve gained a great deal of respect for the folks who do good work in data warehousing, and lost respect for those that try and do it with a shoehorn mentality.

One of my latest projects has been to get a VPN link up between my parent’s house and ours, so that I can help them out with computer issues without having to make the trek down to West Chester. Not that I don’t love visiting them (shamless parental plug, they do read this after all), but what with Taylor’s arrival and all we don’t get out much these days. Anyway, after thinking about how best to accomplish this I decided I should use a DD-WRT (open source alternative firmware) flashed Linksys router for the job, for a number of reasons:

• Thanks to the generousity of a local gentlemen I know from Twitter I acquired one of these devices for free. Don’t ever say Twitter can’t get you stuff!
• It’s low power and unobtrusive. While I love my big ol’ quad core monster, I didn’t really want to take up space at my parents house.
• With DD-WRT, it functions great as an OpenVPN client.

Now, the setup was to look something like this:

(Please excuse the rather awful drawing, I’m afraid I’m used to Visio)

Once I had everything setup, everything appeared to work, except I couldn’t ping anything at my parent’s end of the tunnel from my home network (vice versa worked fine). Well, normally I might try and sniff the traffic at the two endpoints, but this was a little more complicated than you might think, since the router at my parents doesn’t support such an operation.

In then end, I was able to collect some data after hacking a solution (<geekery>I used an already compiled version of tcpdump and manually moved it to the router</geekry>), and figured out that for some reason the VPN router at my paren’t house was performing what’s known as “masquerading”, or “NATing”. What this means is that any traffic that passed out of it was translated such as to appear that it came directly from the router itself, and not some machine behind it. This novel concept is actually the basis for how pretty much every home router functions; but in this case it was bad, in that it made the tunnel between our networks essentially one way.

So what did I do? Well get royally annoyed of course, proceed to spend the next several hours pouring over the setup of both ends, trying various hacks, googling like crazy, and perhaps questioning my intellect (or that of the programmers of OpenVPN, DD-WRT, and Linux in general) a few times. Finally after giving up for a day, I decided I’d look at a tutorial on the OpenVPN site itself I remembered. It took awhile to find, but there was the answer, right in front of me.

Apparently OpenVPN needs a little special tweaking to allow for traffic from both the “client” (my parent’s house) and “server” (my house) networks to fully talk to each other:

<geekery>

Next, we will deal with the necessary configuration changes on the server side. If the server configuration file does not currently reference a client configuration directory, add one now:

client-config-dir ccd

In the above directive, ccd should be the name of a directory which has been pre-created in the default directory where the OpenVPN server daemon runs. On Linux this tends to be /etc/openvpn and on Windows it is usually \Program Files\OpenVPN\config. When a new client connects to the OpenVPN server, the daemon will check this directory for a file which matches the common name of the connecting client. If a matching file is found, it will be read and processed for additional configuration file directives to be applied to the named client.

The next step is to create a file called client2 in the ccd directory. This file should contain the line:

iroute 192.168.4.0 255.255.255.0

This will tell the OpenVPN server that the 192.168.4.0/24 subnet should be routed to client2.

Next, add the following line to the main server config file (not the ccd/client2 file):

route 192.168.4.0 255.255.255.0

Why the redundant route and iroute statements, you might ask? The reason is that route controls the routing from the kernel to the OpenVPN server (via the TUN interface) while iroute controls the routing from the OpenVPN server to the remote clients. Both are necessary.

Yep, you read correctly. “Both are necessary.” Long story short, I only had one of the two necessary configurations in place. Basically, while the normal

route 192.168.a.b 255.255.255.0

command got the packets as far as the OpenVPN tunnel, the software required that extra

iroute 192.168.a.b 255.255.255.0

configuration line to correctly send the traffic on its merry way from there. Without it, the traffic would just appear to go out the proper interface, but actually go to that awful place where packets disappear and are never seen again.

</geekery>

Hi everyone, my name is Josh, and yes, I admit it, I should have read the… oh hell you know what the rest means.

So you might remember a previous post where I lamented my lack of a “Someday” list. Well, since then, things seem to have swung around to the opposite extreme. I’ve now got a healthy selection of projects-in-waiting, both for work and personal. Partly it’s been because I’ve been better at cleansing my action and project lists on a weekly basis; anything that has laid static for more than a few weeks gets archived and tagged “Someday”. But I’d say with confidence that the growth is largely attributable to a shear lack of time.

Ever since the birth of my son Taylor, things at home have been, well, busy to say the least. It’s been a struggle just to keep up with the day to day work, such as keeping the house clean, the laundry done, and the trash empty. With such little time and energy left over for personal projects, strict adherence to priorities and ruthless cutting of scope has been the rule of thumb. So much for things like learning Perl (though I’m sneaking this in at the gym on the treadmill), building a new Snort server, or even non-geeky work like painting the exterior windows on the house.

Work, while slightly less crazy, has been quite a whirlwind as well. After some purging of wartime troop levels, the remaining force has been tasked with a “lights on” mantra. That’s all fine and good, except you’d be ludicrous to call what we do “keeping the lights on”. Software still needs to be updated, security maintained, systems administered. Yes, some of the excess fluff has been removed: no more long troubleshooting of user issues (is it replicated on a clean system? If not, guess what, you get to re-image your computer) or extra out-of-scope work. But still, there is no shortage of necessary tasks to be had. Combine that with taking on a new product and expanding my role to include some levels of data-guru, and you’ve got a packed agenda.

On the one hand, it is more than a little frustrating to see the mounting list of “not yets” and “maybe somedays”. But at least I can be secure that everything is safely tucked away, waiting for the day when changing diapers isn’t an hourly occurence, or treading water at work less the norm.

At work I’m fairly religious about such tenets as proper scope definition and change management in running my projects. Without this level of control, things would quickly spin out of hand and projects would never be completed in a timely fashion. Flip through any project management guide (the ubiquitous PMP Handbook comes to mind) and you’ll find pages of information on managing scope creep and keeping stakeholders in check.

But what about personal projects? In many cases, projects are more loosely defined, and lack specific objectives and milestones. The end result is that work gets added or priorities shifted mid-stream, which inevitably causes delays in completion, frustration, and an ever increasing pile of uncompleted projects. Not an ideal place to be!

Here’s a good example. One of the recent additions to the projects list is to build a website for my son, where we can showcase pictures, stories, and movies of his growth. Now at first, this seemed a pretty easy task, given that the steps might look like the following:

• Select a hosting provider and setup account
• Design simple iWeb template
• Upload pictures to Picasa
• Upload video to YouTube
• Add content to pages, initial list to include:
  • Birth
  • The First Week
  • Pictures and Video
• Send notification to immediate family

Easy enough, yeah? Well, being my geeky self, while carrying out these actions I decided to add some interesting learning opportunities to the mix:

• Instead of hosting, use a dynamic DNS service and host the web site on my VM server at home. It is cheaper and since this site is not financially bound (i.e. if it goes down I won’t loose money), high availability is not a requirement.
• Install WordPress as the CMS for the site, to allow for RSS feeds.
• Find and install custom plugins allowing for photo gallery and embedded video functionality.

Getting a bit more complicated, aren’t we! Well, being the enterprising sort, I thought I might want to add even more complexity, so I considered these as well:

• Setting up a VM to act as a gateway between my home network and the Internet, so as to allow the Internet exposed VMs to be on a separate network subnet from the rest of my internal LAN.
• Setup of an IDS such as Snort to monitor traffic on this DMZ.
• Setup of a custom VPN solution using OpenVPN so as to allow grandparents and other close relatives the ability to see additional content over a secure link.

Now, unless you are one hard-core geek (and not a parent of a new baby), you are probably going “What is this guy thinking?!” And you’d be right; while one could argue that the first set of “enhancements” adds some value to the overall project goals (a simple, easy to update, followable web presence), I think you’d be hard pressed to argue the same for that second list.

True, separating Internet and non-Internet facing network zones is a standard best practice, and a solution such as Snort would probably show all sorts of useful data. But really, what is the chance that a) a hacker is going to take the time to scan my network when it’s not widely advertised, b) there is some undisclosed vulnerability in WordPress or PHP, c) once gaining non-root access to the web server the hacker proceeds to succesfully brute force my strong, random 40 character password? Yeah, not likely I think. Definitely not worth the considerable effort for those last three tasks.

Given that my time is going to be rather short for the forseeable future, it appears I’m going to have to curb my knack for adding needless complication to personal initiatives. Just like with real projects, when time is short and resources tight, it’s more important than ever to ensure the work you do is adding real, tangible value to your projects.

Lately things have been a little crazy at work. Between a fair amount of vacations (some planned, some not) and a never ending stream of project work, my days have been pretty much filled to capacity. I’m a guy who prides himself on being efficient and squeezing the most out of my time, but damn, the difficulty of getting everything done has risen to a new level.

One of the most challenging aspects has been the sheer volume of requests that come in, requiring some form of response / work. Mind you, it’s not that I am stressing out because I can’t find the time to do the work; no, it’s become stressful just taking the time to explain to people that I can’t do the work! My management is excellent at backing us when we tell our business that a given request / project needs to wait or will take an extended period of time, but that doesn’t avoid the necessity of managing a constant stream of incoming and outgoing communications.

What I’d really like to do is put up an auto-responder that would sound something like this:

“Hello, thank you for e-mailing me. Please be aware that due to a sizeable project load, I will not have sufficient time to take on any additional ad-hoc requests for the next month. If your e-mail is regarding an urgent matter that cannot wait (Severity 1 / 1.5 critical outages), please call my extension to discuss. If your e-mail is regarding an existing project, don’t worry, I will respond to you in the next 24 hours. Otherwise, please be aware that it may be an extended period of time before I respond. Thank you for your understanding.”

A little extreme? Yes, I’ll admit that. But seriously, I can’t begin to think how much more productive I would be if I could eliminate this needless “Hi So and So, thanks but I won’t be able to do that for at least a month.”

Given that a proposal to my boss to implement this would likely be met with hysterical laughter (followed by one of those glowering, “Are you serious” looks), here’s some possible alternatives I’m considering, as a kind of stop gap measure if you will:

1. E-mail (Outlook, GMail, all of the above) is closed, not minimized, except for a once per hour check to process incoming items. Every hour my inbox needs to be cleared and all e-mails either placed in the “Needs Reply” folder, the “Reference” folder, or logged as an to-do in Remember The Milk. Truth be told I’m just not doing well at ignoring checking e-mail with Outlook only minimized, despite turning off those annoying pop-up notices.
2. Dedicate a period of the day to responding to e-mails, outside of which only critical issues would be answered. I’m thinking 1.5 hours in the afternoon would be sufficient.
3. Put up an auto-responder for any e-mails marked urgent with the following:”Hello, thank you for e-mail me. Because your e-mail was marked as urgent, you are receiving this automated response to inform you that I only check e-mail approximately once every hour, and respond to the majority of e-mails once daily at (insert chosen time here). If you are reporting a critical (Severity 1 / 1.5) issue, or feel you need a response before the time noted above, please call me directly at extension xxxx, or call my team line at xxxx. Thank you.”

What do you think? Still too extreme?

Today I caught myself getting rather ticked off after spending around 2.5 hours trying to script out changes to around 25 SQL Server stored procedures. To explain in a non-geeky (if that’s possible coming from me) way, I had to comment out a common line in all 25 procs that needed to be disabled in order to use the proc in my test system. (OK, that failed only slightly in the non-geeky department.)

I’ll be the first to admit that once I get focused on solving an issue, it pretty much consumes my attention, to the point of near obsessiveness. This was no exception, with me getting more and more worked up as attempts to automate this change (rather than cracking the code open on all twenty plus one at a time) failed. Finally I threw up my hands in disgust and walked away, having undoubtedly raised my blood pressure a notch or two in the process.

A short time later, it occurred to me that most of what I was trying to accomplish was completely outside of the work necessary to accomplish my pre-stated goals for this project. In effect, I was trying to change twenty plus bits of code, when this particular effort required just one change. Yep, you heard me, just one. So why even make the attempt to automate the process?

I suppose it’s mostly because I’m a lazy coder by nature; if chances are better than 50-50 that I’ll need to repeat some action in the future, I’ll probably at least take a stab at scripting or otherwise automating the process. Plus, in this case, it presented a worthy challenge, which is always more than enough to entice me into jumping in over my head. I’m a sucker for challenge, almost to the point of it being a character flaw.

In hindsight, perhaps if I’d stopped at the onset and thought things out in a rational, methodical manner, I would have saved myself not only a brooding headache, but almost two hours of time as well. Two hours. Think of all the things you could get done in two hours time.

Next time, I’m going to follow my own advice, and make sure I properly define what my work is going to be, lay out all the time and effort needed, and only then consider adding scope to my plate. Sure, automation is a wonderful thing, but only when used with discretion. Spending two hours to avoid doing something that ultimately took around 10 minutes to accomplish manually just isn’t worth it.

When I first started up this blog, I had a fairly clear vision of where I wanted to go with it. Having browsed the musings of established GTD bloggers like Merlin Mann of 43folders.com and Andrew Mason of Did I Get Things Done, I wanted to follow in their mold of becoming an established voice in the field. But then along the way, it seemed that I hit a wall. You see, one of my goals has always been to have some useful, original content to post; trouble is, GTD is such a hot topic these days that original thought or advice is rather hard to come by.

Now granted I could talk about how to set up your “trusted system”, or how to maintain discipline with starting tasks, etc. But the truth is, someone has probably already written about it, and with far better clarity and experience than I could claim to have. I’ve been at this game for just under two years now; hardly enough time to call myself a guru of any kind. So what’s a disenchanted productivity geek to do?

In short, I’ve decided to take the blog in a bit of a new direction. Instead of trying to write some authoritative pieces on how you should integrate the concepts of GTD into your life, I’m going to keep it rather simple, and just talk about me. Now before you ask, this is not going to turn into some ego-maniacal diatribe on why Josh Is God or some such nonsense. No, instead, I’m just going to write as what I am: a guy coming from a world of disorganization and ADD-exacerbated messiness, trying his damndest to learn how to keep everything together (or at least keep appearances of such, but don’t tell my boss).

From now on, it’s going to be a more journalistic approach. I.e. “I read this really interesting post the other day, here’s how I’ve tried to incorporate it, here’s why it worked / didn’t work.” I’ll be happy to tell you all about my struggles and triumphs, and maybe even those of others I meet or talk to. But from now one, the one thing I’ll try not to do is tell you how to do things. If you ask (comments still encouraged), I’ll be happy to give an opinion, but outside of that, the advice column is closed.

Beyond that, I’m also going to try and expand a bit on the non-GTD topics that are holding my interest. I think a while ago I wrote a pretty decent introduction to a series on e-mail encryption… maybe it’s time to pick that up again, eh?

Here’s to a little diversity and change in focus. Hopefully it will be just the thing to help break out of a bit of writer’s block and get this blog up and running again.

Yes, it’s been a while since I last posted. Truth be told, I’ve a) been really busy, both at work and at home, b) been in a bit of a creative funk, c) fallen of the GTD wagon. There, I said it. “Hi there everyone. My name is Josh and I keep stuff in my head.”

Thinking back over the last few weeks, I can see a couple places where I’ve fallen short in my practices:

• Processing Inbox To Zero: I’ve been doing quite a lot of what Merlin Mann calls “Emergency Scanning” lately. Usually my rule is that once I touch my inbox, everything has to be processed. Instead, I’ve been flipping through items, making sure there weren’t any brewing fires, then only processing everything every other day or so. Alas, as a wise man (probably) once said, “A nuisance today is a fire tomorrow.”
• Reviewing Religously: I don’t think I need to espouse the necessity of reviewing what’s on your plate early and often. For the past two weeks or so, however, it’s been a pretty constant fire-fight, with the latest really being the most urgent. It’s a draining place to be in, and I hope soon that the firestorms will die down and I can get back to being a bit more proactive.
• The Weekly Review: Already one of my week spots, and I’m afraid I’ve completely skipped over this in favor of the same emergency “quick-scan” mentality. I’ve been so focused on one or two projects that outside of those two, I haven’t really cared what else got pushed off.

So, like a golf player whose swing has gone awry, I’m returning to the fundamentals to break out of the slump. Here’s a few self-imposed rules:

1. Every day must end with an empty, processed inbox. NO EXCEPTION
2. Any e-mail that arrived prior to 3:00PM must either be answered, delegated, or responded to with a “I don’t know at the moment but I’ll get back to you” note and subsequent logging of action.
3. Sunday nights from 8-9PM are set aside for weekly review. My wife has already been recruited to enforce this policy, on penalty of, well, horrible things (the mind of a pregnant woman is a dangerous thing).

Oh, and I plan on posting to Twitter and this blog over the next week to log my progress getting back in the saddle. Truth be told I do very much miss the “mind-like-water” feeling that always accompanied knowing exactly what’s on my plate at any given time.

Ever had one of those tasks that just seems to sit endlessly on your to-do list? Every day you see it, waiting there, just screaming for some progress. But the end of the day comes, and there it still sits, without any update or progress to report. What is it about these kinds of tasks that makes them so unpleasant to start that they remain in perpetual limbo?

Well, among other things, according to the folks at PsyBlog this can be caused by our tendency to think at too abstract a level. Citing a study by Piers Steel at the University of Calgary, the referenced post does a great job of explaining the difference between abstract and concrete thinking. While I’m not going to go into detail about the study or its results (if you’re a former psychology nerd such as myself, you’re welcome to read it on your own), the basic finding was that by forcing people to think in a constructive, concrete manner, they were able to cut the average time to complete a seemingly meaningless task nearly in half (20.5 days down to 12.5 days on average). That’s an impressive result to anyone.

I found this to be true in my own experience, but was never quite able to verbalize it. I’d find that there were certain things on my list which would simply sit, incessantly staring back at me day after day. Whenever I tried to start working on them, I instantly lost motivation and inevitably moved on to something else. It was not until after I read this post that the light bulb finally went on: I wasn’t being concrete enough in my definition of the task at hand.

Here’s a good example: I might have a task called “Update documentation for product X.” Well, at first glance, you might call that a fairly specific item. But thinking further, I might begin to wonder “What exactly do I need to update about the documentation?” This invariably led to an even more detailed analysis, fostering questions such as “What screenshots do I need?” or “How can I prove out that my updates are correct?”

What I began to discover was that “fairly specific” task was in fact an entire project! All this time, my resistance had been due to what a project manager might diagnose as a failure to break work into small enough chunks to be easily tracked and defined. Once the task was broken down into chunks taking no more than a half hour or so, it suddenly became far easier to begin working on the item at hand.

When resources are tight, folks such as myself in the IT world are often asked to “do more with less”, to use a rather contrite phrase. With layoffs looming for many and already a reality for some, it goes without saying that in the coming year, so-called “knowledge workers” will be asked to stretch their capacity as far as humanly possible. So outside of working insane hours and sacrificing quality time with your family, how do we meet this goal and satisfy our clients?

Now, if you’re reading this blog, chances are you are already a productivity minded person, which puts you at an advantage above those around you. But beyond practicing our GTD skills of constant capture and ruthless review, an ability to single out and automate those routine, time-wasting tasks can prove equally as useful.

As a programmer / business analyst in my previous job (and even now as a product specialist / implementations manager), this mindset was constantly hammered into my way of thinking. There’s a saying that programmers are a lazy bunch, and I certainly won’t deny it; I will however argue that is a good thing. If you’re paying a programmer by the hour, would you rather they spend extra time re-inventing the wheel, or make efficient use of existing code? Similarly, if you’re working with a B.A., wouldn’t you appreciate it if they noticed some repeatable tasks you were paying staff to do that could be easily automated. Granted it’s not always quite that simple, since with automation often comes cost as well. It’s always a give-and-take situation, where the pros and cons must be carefully weighed. If it’s going to take 400 hours and $20,000 to automate a process that takes one person an hour a week to do, that’s probably not a good value.

But really, that’s not what I’m talking about here. The fact is, there are people paid lots of money with fancy initials like “P.M.P” after their name whose job it is to do analysis on larger business processes. No, what I’m suggesting is that you take a closer look at your own daily work, and see if there are any easy targets to be scripted or scheduled, so as to free up your time to take on higher value work. Here’s are three adjectives and phrases that describe good candidates for automation:

• Repeatable
  
  You perform the same task on a hourly, daily, weekly, or monthly basis, with very little change. It might be running a report and sending it to a client, doing some number crunching in an Excel sheet, or perhaps pouring over one of those dreadful “green bar” mainframe reports and pulling out some data.
• Rules Driven
  
  The task performed is based on hard business rules. For example, you open a report, and if a certain number is off by more than a given percentage, you have to send it to someone.
• Number Based
  
  Number crunching, by definition, is highly adaptable to automation. If you spend any amount of time manually punching in calculations or summary statistics, you could probably take care of all the work via something like macros (in MS Office, for example).

How far you go with this is totally up to you. I’m a pretty avid scripting geek, so I’ve been known to write VBScripts, AppleScripts, and even (for you hardcore geeks) bash scripts to do just about anything. Over the next week or so I’ll be looking for good resources on how to script / automate common tasks and tweeting them, so tune in! Here’s a few to get started:

• Recording And Customizing An Excel Macro - good for you Excel whiz kids.
• Beginners Guides: Understanding and Creating Batch Files - The do-it-all of Windows / DOS, batch files can be used to automate lots of simple tasks.