This is the first post where meat and software meet. It deals with the topic of quality, and how poor quality in the butcher shop can bring a company to halt in the same way that poor quality in the software shop can bring a company to a halt. In both cases producing high quality output from the start results in higher efficiency, lower costs and a more satisfied customer. Alternatively, when quality problems are discovered at the very latest stage – by your customers – the cost to mitigate immediate damages as well as the cost to correct the underlying root cause of the problem can be orders of magnitude higher than detecting the problem at an earlier stage.
The USDA is responsible for the meat inspection process. There is a licensed Federal or State inspector on site who has the authority to bring the entire operation to a standstill if there is any issue with quality. Prior to anyone starting any work activity for the day, there is a site inspection covering everything from the cleanliness of the grounds outside the building, to ensuring the proper sanitation of facilities, tools and equipment in the shop.
While there are many additional quality checks throughout the entire workday, the most critical part for me was the initial sanitation inspection, because I was directly responsible for this job (I was the DRI, a powerful management concept that I will address in detail in a later post). If I didn’t deliver a quality result, all workers remained idle and unproductive while problems were corrected. This not only added cost, it delayed schedules for all subsequent activities, including the final activity of the day – cleaning up. I suspect some of the software engineers reading this can already see where this analogy is headed (but please, do keep reading).
There are many different cleaning jobs in the butcher shop, but today I will focus on the one that had to be done every single day – cleaning everything involved in the final production of steaks, chops, hamburger, sausage, etc. Everything used in this production process had to be cleaned nightly: scales, sinks, knives, aprons, saws, grinder, hamburger-patty machine, sausage-link press, various containers, tables, floors and walls. These were industrial grade machines meant for commercial use on a daily basis. The most complicated of these machines (the one that made hamburger and sausage patties in various weights and shapes) would break down into many dozens of pieces for cleaning. The process consisted of tearing everything down into component parts, applying disinfectant soap, and then using a high-pressure sprayer with 140 degree water to wash everything. When this was finished you put everything back together ready for use the next day.
Now, how many of you have ever washed your car at one of those places where you drive into a bay and spray it with high pressure soap and water using one of those long wands? And how many times did you spray the car only to receive a deluge of water and/or mud in your face? So you have the general idea what the cleaning process might be like, but now imagine cleaning scores of different parts large and small, most requiring one hand to hold or rotate the part while the other hand aimed the nozzle in the direction that maximized cleaning effectiveness while trying to keep yourself dry, knowing that a direct spray at close range would literally peel the skin off your fingers. I became quite good at spatial reasoning and trajectory calculation as a result, and on a good night could remain relatively dry. However, on a hot summer night the temperature and humidity inside the building created a poor man’s steam-room that would really sweat the toxins out of your body (I have no further comment on what toxins may have been in there or how they may have gotten there). My Dad would always joke that we should be glad we had this job because we had the cleanest fingernails in town.
Now back to the software business and an attempt to tie everything together.
Quality issues in the software business can also bring an entire operation to a grinding halt. In fact, in some fields (medical devices for example), quality issues can have life of death consequences in the same way that food quality issues can have.
There is right now a significant root cause analysis under way to uncover the cause of the recent deadly E. coli outbreak in Europe, demonstrating the tragic consequences of problems with food quality (numerous examples can be found in the meat industry as well, but I’m trying to do everything I can to keep any remaining vegetarian readers in the fold by providing an occasional vegetable reference).
I have a team in my organization that is responsible for delivering a core network element in the mobile telecommunications network. If this product is not functioning properly mobile phone users cannot place calls or receive data, and an outage has the potential to affect tens of millions of subscribers at once. You can imagine the cost to the service provider in terms of lost revenue, and worse, loss of trust from their customers if this happens. This is a product where we focus on quality before any other consideration.
However, you don’t need to be writing software for a medical device or core network element for these same principles to apply. Yes, maybe product quality is a lower priority than time to market or multiple other considerations, but the same principles still apply. The earlier in the process that quality issues are addressed the more efficient and cost effective you are. Producing a quality architecture will prevent many issues from existing in the first place by simplifying the implementation. Producing quality code reduces the cost and duration of system testing, and pays long term benefits in maintainability and extensibility. Developers: take responsibility for producing quality. When I was writing software I was always disappointed when anyone other than I found a defect in my code. Take ownership of producing quality through code reviews, static and dynamic code analysis, unit testing and test automation.
In closing, when you build software make sure you focus on producing quality from the start, not on fixing problems when you are “finished” writing code. Make sure your architecture is clean. Make sure your code is clean. Make sure your tests are clean. And it never hurts anything to make sure your fingernails are clean.
What do other developers out there think (or other SOB’s for that matter)?
They say you don’t want to see sausage being made, but you are directly involved in the sausage-making process of software development every day. Is it practical for developers to produce high quality software and minimize the testing effort at the tail end of the delivery schedule?