Gojko Adzic

It’s not uncommon for “star” programmers to be an order of magnitude more productive than their colleagues. I believe that a large part of that productivity gap comes not from doing tasks faster, but from not doing them at all. Good programmers make the machine sweat instead.

I recently had a very unpleasant talk with one of our support developers. She was very annoyed, in a defensive mood, arguing that she is already too busy and cannot take on a bit more work. After twenty minutes of talking, she finally calmed down, and we came to the root of the problem. It’s not the programming part of her work that is killing her, but the fact that she has to go through several terminal servers to deploy changes to client environments, and that she has to merge all her changes into several revisions. And she does that for every single change, several times a day. So, in fact, she spent most of her time logging on to different computers and copying files between them. Although she understood the problem herself, the thought of automating those silly tasks never came to her mind.

It’s incredible how, for a profession based around telling machines how to perform dull and error-prone tasks instead of humans, so few programmers think about how to automate their own work. This is especially noticeable with the “Windows generation”, people who never effectively used the command line. GUI programs are, without a doubt, more user friendly and easier to use, but they are bloody hard to automate. When a task has to be done fifty times, or performed across the network, words “ease of use” take on a completely different meaning. That is the point when good programmers break from the usual practices and find better ways to do the job, and bad programmers stick to their old habits.

Breaking from the old habits

A few years ago, I did some consultancy work for a company whose application generated huge log files on clustered servers. To find the cause of any problem, programmers and support staff transferred gigabytes of data from client’s web servers to their computers in order to analyse them. Each transfer took at least 10 to 15 minutes. There was no special reason for pulling all the log files to local machines, people were just used to doing their job like that. I was even more amazed to see that even some “senior” programmers in that organization repeatedly (tried) to open monstrous log files in Notepad and than wasted the next 20 minutes waiting for their machine to unfreeze. All that just to find a few lines of text, which could have been done in a minute or two using findstr on the server.

When a solution clearly does not scale to the problem, we look for a better one, or a different way to implement it. The solution often requires using a different approach, or different tools. The same holds for non-programming tasks. We must look for better ways to perform daily tasks when the usual tricks don’t work. In this case, the proper solution was to use the command line. Programs like tail and grep were developed at a time when network connections were bad, and computing power was scarce, so they don’t require a lot of typing and clicking. Although available bandwidth and computing power have improved greatly over the last 20 years, such tools are still irreplaceable for processing large text files, especially on remote computers. Sadly, I meet more and more developers who never heard of those tools. If you are reading these lines and thinking to yourself “what the heck is grep“, here is just a brief introduction to some common tasks that these tools do really, really well:

grep

• get all lines containing a string (or a regular expression) from a large file - for example, get all log items for a particular session or user
• count the number of occurrences of a string (or regular expression) in a file - for example, find how many times someone has logged yesterday
• find all files that contain a string (or regular expression) - for example, list log files that contain a particular username

more

• view the start of a large file - for example, see the IIS log file format which prints column headers in the first line

tail

• quickly view the last few lines of a large file - for example, check out the last exception
• monitor how a log file is changing while other processes are writing into it - great for troubleshooting a live application, because it does not lock the file

awk

• extract lines from a large file and reformat them - for example build URLs by appending fields from a tab-separated log file to the site prefix, or create a block of SQL statements to populate tables from some data export file
• calculate statistics on file content - find top 20 client IPs by web bandwidth used and print the number of requests and total bandwidth used by each one
• split a file into several other files, based on line content - for example, extract multi-line exceptions from a log file
• quickly analyse a comma-separated, tab-separated or any other file format with multiple fields in a line

sed

• “replace all” for large files and sets of files

Working on Windows is often an excuse for not using command-line tools, but that is just an excuse. There are several free sets of tools which bring power to the Windows command line. For example, Cygwin package allows most popular Unix tools to run from Windows. Windows Resource Kit from Microsoft contains tail and qgrep (a fairly usable variant of grep). PsTools, another package now owned by Microsoft, contains gems like psexec that can run a program on a remote computer. Some tools, like GNU Awk have been directly ported to Windows by the GNU foundation.

On any variant of Unix, you can expect these tools to be already installed. Some trimmed-down replacements exist even for the standard Windows command line (like findstr which can be used instead of grep if you don’t look for regular expressions). However, most of these tools can just be copied to a remote machine and started (no need to “install” them), so you don’t require any special privileges to run them on client’s web servers, for example. You can just drop them on a remote computer and start using them. So don’t even try pulling the entire log file from the customers web site just to do a simple search: copy gawk there and process the log on the server.

In addition to being very efficient, these tools can easily be automated (scripted). It does not really matter whether shell scripts, batch files or some scripting language like vbscript, python or perl is used. Storing a grep or awk command line along with all the strange parameters will save you from the pain of writing them again tomorrow.

Three strikes, and you are out

“Don’t repeat yourself” is one of fundamental guidelines of Extreme Programming. Most good programmers today take care not to repeat the code, the ones that are even better find similarities in design and consolidate them, but very few apply the same rule to their habits or non-coding related work. I’m not just talking about half-hour tasks like sorting e-mail and analysing debug logs, but also small jobs like checking whether the client database is on-line or replacing seven occurrences of a field name in a script.

Apart from saving time, using replace all will also prevent errors, but you have to test it first before updating a 200 MB file on a client’s environment. The same tasks have to be applied for replacement in a 5 KB script file. There is always some overhead involved in automating the tasks, but when does it begin to pay off, and when is it unjustified? The rule I stick to is Roberts’ rule of Three (from Fowler’s Refactoring: Improving the Design of Existing Code), another version of DRY:

The Rule of Three by Don Roberts

The first time you do something you just do it. The second time you do something similar, you wince at the duplication, but you do the duplicate thing anyway. The third time you do something similar, you refactor.

This is also a good rule of thumb for tasks outside of coding. Read it like this: If you catch yourself doing something for the third time, chances are that you will have to do it again, so automate it. Use the replace all if you have to replace the same string more than two times. More importantly, stick to the rule when things get complicated! If you are not dealing with plain strings, but have to re-format the data a bit, don’t fall back to manual work: utilise regular expressions (if you don’t know how to use regular expressions, now is the time to learn). If this is the third time this week that you are doing the same replacement, don’t use the regular expressions manually, but write a script. When you catch yourself running that script every day (week, month) add it to the scheduled tasks list (or cron jobs).

Automating tasks is especially important when your client has to perform them. Having a nice automated installer instead of a two-page word document explaining all the steps required to set up the software goes a long way. It makes the task less error-prone so the customers will be happier, and it leaves us with more time for the fun stuff.

Image credits: D Lee/SXC

Add to Del.Icio.Us bookmarks

Trackback URI | Comments RSS