Jim Hall's blog

Maybe you like to read the comics page in the local paper. For myself, I prefer to check out comics on the web. I follow a lot of them - almost 25. I used to have bookmarks to all the webcomics I like to read, but it took a lot of time to visit each site and find that day's comic. Over 25 webcomics, that took up a lot of of my mornings. (Fortunately, I read them during breakfast, but still ... it took extra time that I could use to get ready in the morning.)

So a few years ago, I wrote some scripts to put all my favorite webcomics on one page at my personal web site. These were very simple scripts, one line to download the latest webcomic from a specific URL and put it on my page. Over time, I updated some of these scripts to add error checking, or to accommodate webcomics that moved their images to different locations, whatever.

A few weeks ago, I realized that several of my scripts had been silently failing for weeks. Upon reviewing them, I found that the scripts all worked differently, because I had updated them at different times. The basics were supposed to be the same, but the implementation had diverged and was different in every script.

I needed to completely re-architect how I collected all 25 webcomics on my web page. Rather than have a different script for each comic, I created a single, universal script to automatically fetch a web page, find the webcomic in that page, then get the comic. I put error checking at each step, so I would know exactly what failed if the script is unable to get a new webcomic. When I was done, the script was less than 90 lines long and worked for every webcomic.

Of those 90 lines, 20 are comments, 10 lines set variables, another 10 lines do all the fetching, and the rest (30 lines) are error checking and tests. Now, to update each webcomic, I run a single command through my crontab every morning at 6:00AM.

I'm only a manager, and these days my scripting skills are pretty rusty. But if I can do this kind of automation, I'm sure you can do much more with the systems we support in OIA. I'm looking to everyone in Operations & Infrastructure to use the lens of Simplify, Standardize, Automate to take a second look at what you do. Figure out ways to simplify how the work gets done, automate the tasks to the extent possible, and standardize on that method to every system you manage. Only by doing this can we support the growing number of systems that we are asked to support.

Posted by Jim Hall at 1:49 PM | Permalink

Right now, we're going through a change in OIT. We're working to streamline everything that we do, so OIT can provide more and better services to our customers across campus. With the current economic and budget situation at the University, we can expect more departments to look to OIT for help in providing services.

But a critical component to ensuring we can meet those needs is to standardize on services. Nothing hurts us more than when someone in OIT, especially part of our own teams, makes a special "one-off" arrangement with a customer to deliver some service. Because when we do that, it sets a new standard that all of OIT needs to deliver on, or we risk other customers asking "why can't you do that for me?" - or worse, "you just said 'no' to me, but you're doing it for that other department."

I know it's hard for us to say no - it's sometimes hard for me, too. As IT people, we often measure our productivity by the things we've done or built. When we go home at night, it may not seem like we've done "work" if all we did was respond to server issues, or write a design document. (Yet, in an enterprise support role, we need to recognize the documentation of the work is just as important as the work itself.) As a result, when a customer contacts you and asks if you can work on some special "one-off" project, there's a tendency to say "yes" so that, at the end of the day, you can reference something as "I did that."

But it's important to realize that doing all these "one-off" projects is hurting OIT in the long run. The key behind simplify, standardize, automate is to run all our systems more or less the same. We manage over 1,100 servers across all of OIT - almost all of them supported directly through Operations & Infrastructure. That number will only grow; in a year, we may manage more than 2,000 systems. If those systems have the same or similar system setup - systems administration, databases, production automation, storage, etc. - and if we have the same expectations for customer roles vs. OIT roles, we will be able to effectively manage those servers. If they all have different configurations - different applications, different ways to manage storage, different expectations for the customer & OIT - then we will fail in meeting our mission to the University.

I'm asking for your help. If a customer asks you for a special configuration, or to take on an "application" task for them, consider what that means to the rest of your team, and to OIT as a whole. Avoid the temptation to say "yes" because you think that's what the customer wants to hear. Remember that every "one-off" project you commit to is another variation in how we support the University.

Posted by Jim Hall at 3:25 PM | Permalink

I'd consider trying to get myself removed from this spam's mailing list, but they continue to send me these great "gems" that I like to post here, in my work blog. The latest helpful spam, trying to sell me a project management tool:

Too often, projects seem to need "just a little more time" to complete. The most important question for any project is "How much is really done and how much is really left to do?"

[...]

But how can you confidently estimate how much time each
project still needs? By following these steps:

1. Break the project down into tasks that must be
completed. Then break these tasks down into smaller tasks
until no task will take more than a day or two to complete.

It is much easier to estimate the duration of a small
task than a big one. The biggest cause of failure in
estimating how long a project will take is not properly
identifying ALL of the tasks that are involved. A task
like "Updating the website" is too broad to estimate
accurately. Breaking it up into tasks for each of 20
pages that have to be updated will lead to a much more
accurate estimate.

2. Decide which tasks must be completed before others
can be started.

This will affect how many people you can profitably have
working on the project. If each step has to be done before
another can start, adding more people won't help. On the
other hand, if each task is independent of the others,
then adding people to work side-by-side will speed
things up.

3. Assign people to work on each task.

Don't assign more work that can reasonably get done in
the time to any one person. Allow for other work the
person may also have. If a person can only work half-time
on a task, you will need to double the number of days it
will take to complete it.

A number of us in OIA occasionally act in a "project manager" role, even if it is only for a "mini" project. Yet the practices to effectively manage a project or set of tasks are the same whether the project is large or small or "mini". The next time you manage a task, follow the above advice and you'll find yourself much likely to hit your target on time, and within budget.

Posted by Jim Hall at 1:23 PM | Permalink | Comments (3)

Another example of how I am applying Simplify, Standardize, Automate to my own stuff.

A lot of you know that, outside of my day job, I work on the FreeDOS Project. For a long time, I've had a kind of "blog" there where I'd write about various happenings, things that didn't really fit as a "news item" on the main page. This was a "blog" in the lowest sense - to add a new item to the page, I had to edit the page.

Just like how I used to do this blog, I created a simple shell script that automatically generated my FreeDOS "blog" pages based on posts I kept as html files in a particular directory. Recently, I decided the shell script was a good start, but what I really needed was a blogging system. Fortunately, SourceForge (where www.freedos.org is hosted) now offers application hosting for SF users. One of those applications is Wordpress, a popular blogging system.

Wordpress has a neat feature: categories. Using categories means I can use one blog to write about several different topic areas: FreeDOS updates, personal stuff, my work with Linux, etc.

But here's my problem: pretty much everyone on the FreeDOS Project would be uninterested in my personal stuff, most people who follow my Linux work won't want to hear about FreeDOS. Similarly, my friends and family aren't really DOS geeks, so they won't want to see the FreeDOS stuff either. That means no one will likely read my blog as a whole, just the bits they care about. Sure, people could simply follow my Wordpress blog for just the category they want to see - but as I mentioned earlier, I cannot change the (ugly) web design.

How to solve this? In the past, I've kept a separate web page for FreeDOS stuff, another for "friends and family", and another for Linux stuff. But now, I wanted to use Wordpress for blogging, rather than update web pages manually.

Scripting to the rescue! I wrote a shell script that scrapes my Wordscraper blog for a particular category, and displays it at each of my other web sites. Without posting the whole script (let me know if you want it), here's what happens:

• the blog category is fetched as an html file, using "wget"
• an in-line AWK script parses the html, to extract just the blog entries for that category
• another AWK script extracts the link(s) to the blog archives for that category

The tricky bit is that AWK script. Here it is for #2. Behold the power of AWK!

/<div class="post hentry category-/ { blog=1 }
/<div/ { if (blog==1) {div++} }
{ if ( (blog==1) && (div>0) ) {print} }
/<\/div/ { if (blog==1) { if (--div == 0) {blog=0} } }

I expect every person who works on the UNIX systems to understand what this means, as well as predict the output when run against this:

... </div>

<div class="post hentry category-uncategorized">
<h3 id="post-71"><a href="http:// ... /wordpress/jhall1/2009/02/05/test/" rel="bookmark" title="Permanent Link to test">test</a></h3>
<small>Thursday, February 5th, 2009</small>

<div class="entry">
<p>This is a test post to see if non-SF users can leave comments.</p>

<p><em>Update: looks like you need to have a SF account to leave comments.</em></p>
</div>

<p class="postmetadata"> Posted in <a href="http:// ... /wordpress/jhall1/category/uncategorized/" title="View all posts in Uncategorized" rel="category tag">Uncategorized</a> | <a href="http:// ... /wordpress/jhall1/2009/02/05/test/#respond" title="Comment on test">No Comments »</a></p>

</div>

<div class="navigation">
...

With this script in place, my personal blogs will magically appear at my "friends and family" web site, and my Linux comments will go to my "Linux" web site. And I just have to update a single blog.

This is the kind of scripting and level of automation that we need to use in Operations & Infrastructure, especially as we work to automate everything that we do.

Posted by Jim Hall at 5:35 PM | Permalink | Comments (2)

Sometimes, I love the spam that I get. Because even though they're trying to sell me on some product, there's an occasional gem that contains good advice. I've quoted spam before, when talking about Setting the context.

This week, I got a spam about "Four Easy Steps to a Standard Procedure". Ignoring the product they're trying to sell, it's a great message. In part, the spam said:

The easiest way to achieve this [operational efficiency] is to create a standard procedure for all of the operations you repeat every day: processing orders, shipping product, servicing field calls, making sales calls, and so on. Then make sure everyone follows it. THIS BUYS YOU THREE THINGS:

1. YOU ARE MORE PRODUCTIVE. By optimizing the procedure, everyone does the job in the most efficient way.

2. YOUR QUALITY IS HIGHER - YOU MAKE FEWER MISTAKES. You get the same predicable outcome, no matter who does the job.

3. YOUR WORKFORCE IS MORE FLEXIBLE. You can easily train more people to do the job.

With increased productivity, better quality and a more flexible workforce you can [..] lower your costs and still get the same (or more) work done.

[and the steps are...]

1. Document
2. Optimize
3. Communicate
4. Execute

Make sure the people doing the job follow the same optimized procedure. You'll get increased output, consistent results and easily be able to train new hires and other employees to do the job so that you aren't dependent on a key person to get the job done.

If this sounds similar to Simplify, Standardize, Automate - it should! You should also see some echoes of Making an impact in the steps.

We need to put a strong focus on Simplify, Standardize, Automate. I'm constantly pushing on the managers to do all they can in these 3 areas, particularly with automation. Scripting tasks, removing manual steps, will allow us to support larger systems, and more systems, than we have previously. And we'll do it with less stress on everyone who supports these systems.

Posted by Jim Hall at 10:55 AM | Permalink | Comments (1)