Well, not the glamorous type of rescue. Just the kind that can only fall on the shoulder of a geek: Saving your fellow collegues from the mishaps of a buggy mission-critical application.

I did more than my fair share of it and it was fun. That was probably as close as I will ever get to feeling what a fireman must feel on his way to a fire.

Except I was the pyromaniac too.

For a long time, I considered a test environment some sort of superfluous luxury reserved to engineers with too much time on their hands. I fixed – and created – bugs directly on production applications because it was easy and fast. I designed systems that were like ticking time-bombs (who knew hard-drives had limited capacity?)

I didn’t know better and I learned the hard way.

Now I am running a commercial web-based service and I am not looking forward to middle-of-the-night emergencies.

The truth is that I am still a sloppy programmer, but I have taken steps to protect my business from myself.

• First, the whole site is under a version control system (subversion).

• The production site is checked out of the repository and never modified directly.

• The development site is also checked out. It is the working copy.

A typical update of the site works like this:

1. FTP changes to the development site.
2. Tests in the development environment.
3. Commit changes to the repository.
4. Update production site from the repository.

The commit operation is neither quick nor easy – intentionally. I use subversion clients like TortoiseSVN for other projects, but not for my work on the Form Assembly. To commit I first need to establish a VPN to the server, then open a terminal session and enter the command line to commit the changes.

It takes long enough that I am not tempted to commit changes prematurely. I get to think twice about what I’m doing.

Then I can update the production site from the repository. That is just one more command, and one more chance to reconsider what I’m doing. I purposely removed the FTP access to the site to force me to go through the version control system.

Bugs still manage to find their way in the application, but I feel much more productive and empowered knowing that I have a safety net, a log of all my changes and a safe environment to develop.

I am still looking for ways to improve my procedures, so feel free to chime in if you want to share your own experiences and setup.

I know of something else that only engineers with too much time on their hands use – Test Cases. One day it will strike me as the new indispensable tool, hopefully before another emergency ride on a Vespa is needed.