Repairing Software is Easier than Repairing a Car

My loyal fan base – which I currently estimate at approximately 1 (plus or minus 1… Hi Mom) – will note that I haven’t posted here since before the Labor Day holiday. That’s because absolutely nothing happened in the last 4 months. No, that’s not really true. The more accurate (but probably still a bit lame) reason (ok, excuse) I can offer is that I was consumed by my job during this time. But I did make notes on many ideas that I do still intend to write about (really Mom, I will).

That introduction was necessary because despite the current holiday being the New Year’s celebration, I’m going to write about Labor Day. I will attempt to explain the relationship between software development and this photo:

and why debugging and fixing software is so much easier than debugging and fixing hardware.

On the Thursday before Labor Day I was expecting to enjoy a relaxing and leisurely extended weekend. Then my son informed me that his 1991 Mazda roadster (with 170K miles, but otherwise in great shape and a steal for $2K) could not be driven.

Technically it was still running, but he wasn’t willing to drive it due to a problem that he had diagnosed (correctly, it turns out) as a collapsed valve lifter. Now the cost of a new lifter is about $12, but because inserting this new part requires removing the valve cover, camshafts, timing belt and timing gears (among other things), the labor costs to have it repaired by a professional mechanic were more than half the purchase price of the car. We decided to repair it on our own.

Now some of you may have had the experience of going to the repair shop for a $30 oil change but then paying $800 or so to replace your timing belt (or in more modern cars, a timing chain), pressured into making this repair by stories from your mechanic that if this belt/chain breaks it will completely ruin your engine.

Those stories are true (insert dramatic music here).

Well, those stories are true for the majority of all modern cars (built using an interference engine design). Fortunately for us the engine in this car was a non-interference design.  This meant that even if we really screwed things up it was unlikely to result in catastrophic engine failure (insert music here to resolve the previous dramatic chords).

This non-interference design is partially accomplished through the magic of a part called a hydraulic lash adjuster, which is the aforementioned $12 lifter part. There are 16 such lifters total, since this is a 4 cylinder engine with 2 intake and 2 exhaust valves per cylinder. The diagnosis was that 1 lifter had failed and should be replaced with a new part, and the remaining 15 lifters would be cleaned and rebuilt for good measure (keeping the parts cost at $12 instead of $192). For any (fellow-)geeks out there who are fascinated by ingenious mechanical devices you can find a good description of how these lifters work here.

Here is another view of the engine disassembled to allow access to half of the lifters:

What you see here is what lies under the valve cover, with one camshaft and timing gear still in place (closer to you in the photo) and the other camshaft and timing gear removed. This exposes the lifters, which are the shiny silver cylinders partially visible here, but fully shown in the photo below (note that full-resolution photos are available by clicking on the embedded images).

One way we saved a lot of time and effort in this repair was to not fully remove the timing belt and water pump, knowing that the previous owner had replaced them both relatively recently. This is why you see the plastic zip ties holding the timing belt in the right position on the gear cogs (this can be seen even better in the first photo above). Allowing the belt to move even 1 cog on any gear throws the timing completely off (which is also the reason the bungee cord is holding tension on the gear that is removed so the belt doesn’t slip on the timing gear on the crankshaft).

This photo shows the lifter rebuild in progress:

The valve cover is on the right side of the photo, the camshaft is the long steel rod, and the silver cylinders are the lifters. There is 1 disassembled lifter in the jar of cleaning solvent, and another just to the right of the jar (the smaller part fits inside the larger part, but you (fellow-)geeks already know this).

OK, if anyone other than my Mom or a thoroughly bored auto-mechanic is still reading at this point, I’m sure you are wondering what this could possibly have to do with software development, and if there is a connection, when I’m going to get around to explaining it. Well your perseverance has paid off because we finally reached that point.

There were multiple points in the repair process where we weren’t entirely certain what to do. How much oil to prime the lifters with was the biggest issue, and in spite of reading all the instructions we could find on the internet it was never quite clear how to manage this.

What occurred to me was that if this was a software system and not a hardware system it would be easy to just try something, rebuild the software and run some tests, and then make adjustments and iterate until it works as expected. No harm, no foul.

In the case of the lifter repair, rebuilding the hardware system was a multiple hour endeavor. And if the test failed the iteration process is likewise hours long. Advantage software.

The disadvantage is that because fixing software is so easy (for the software engineers), they are not nearly as concerned about product quality as hardware engineers are. You don’t find many car companies that use their customers as beta-testers. Advantage hardware.

By the way software engineers, just because it is easy for you to make repairs, it doesn’t mean that it is easy for your customers to deploy them. Quality is paramount in software too.

My final point is related to hiring practices and the value of experience. In general I believe hiring managers place too much emphasis on specific skills that the candidate already has (their experience), and not enough on their ability to acquire new knowledge and new skills (their potential). I stand by that assessment, but I’m going to demonstrate the value of experience as the final point.

Having no experience in hydraulic-lash-adjustment-lifter-rebuilding (do you suppose there is a doubly-long German word for that?), we made our best guess about how much oil to fill them with and put everything back together for a test drive.

After starting the engine the first thing that became clear was that instead of having 1 noisy lifter, we now had 16 noisy lifters.  It appeared that there wasn’t enough oil in the lifters.

Based on the mechanics of the lifter design we decided that running the engine at a higher speed (and higher oil pressure) might fill the lifters.  We went for a test drive.

The lifter noise did not abate, and it became clear that the timing was also off (due to a small slip of one timing gear). Partially due to the timing issue, and partly due to a known crankshaft issue that plagues this particular model, before we got back home the pulley for the water pump sheared off.

A little bit of experience on how to rebuild a lifter would have prevented these problems, which resulted in having to repeat the entire repair process again. Of course we gained some valuable experience, and as is so often the case the most costly mistakes you make provide the most valuable experiences.

The good news is that the second attempt at the lifter repair was successful, and of course the process was much faster the second time (again, the value of experience).

So you can see that I still know how to get my fingers dirty, even though my main role was supervising rather than doing (just like my real job). Now if I just knew a good way to clean the grease from underneath my nails.  Oh wait, I do!

Best Wishes for 2012.


This entry was posted in Dirty Jobs I've Had. Bookmark the permalink.

One Response to Repairing Software is Easier than Repairing a Car

  1. Pingback: Motivation, Determination, Procrastination, Destination | A Few Things I Learned in the Butcher Shop

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s