« Why it's hard to say no | Main | Operations & Infrastructure team meeting »

Automation is good

Maybe you like to read the comics page in the local paper. For myself, I prefer to check out comics on the web. I follow a lot of them - almost 25. I used to have bookmarks to all the webcomics I like to read, but it took a lot of time to visit each site and find that day's comic. Over 25 webcomics, that took up a lot of of my mornings. (Fortunately, I read them during breakfast, but still ... it took extra time that I could use to get ready in the morning.)

So a few years ago, I wrote some scripts to put all my favorite webcomics on one page at my personal web site. These were very simple scripts, one line to download the latest webcomic from a specific URL and put it on my page. Over time, I updated some of these scripts to add error checking, or to accommodate webcomics that moved their images to different locations, whatever.

A few weeks ago, I realized that several of my scripts had been silently failing for weeks. Upon reviewing them, I found that the scripts all worked differently, because I had updated them at different times. The basics were supposed to be the same, but the implementation had diverged and was different in every script.

I needed to completely re-architect how I collected all 25 webcomics on my web page. Rather than have a different script for each comic, I created a single, universal script to automatically fetch a web page, find the webcomic in that page, then get the comic. I put error checking at each step, so I would know exactly what failed if the script is unable to get a new webcomic. When I was done, the script was less than 90 lines long and worked for every webcomic.

Of those 90 lines, 20 are comments, 10 lines set variables, another 10 lines do all the fetching, and the rest (30 lines) are error checking and tests. Now, to update each webcomic, I run a single command through my crontab every morning at 6:00AM.

I'm only a manager, and these days my scripting skills are pretty rusty. But if I can do this kind of automation, I'm sure you can do much more with the systems we support in OIA. I'm looking to everyone in Operations & Infrastructure to use the lens of Simplify, Standardize, Automate to take a second look at what you do. Figure out ways to simplify how the work gets done, automate the tasks to the extent possible, and standardize on that method to every system you manage. Only by doing this can we support the growing number of systems that we are asked to support.