Isotupa Consulting Inc.
473 Oakfield Crt.
Waterloo, Ontario, N2T 1X5
tel. 519 885-5187






Feb 3rd, 2004

The Importance of Testing...

 I have recently been involved in a project management project to bring a software deployment to conclusion. The project was very much overdue, about 2 years late in fact. The functionality of the product was a great match to the user needs, the user community was eager to use it, it will save the organization a bundle and the management support is and has always been there. Why was it late then? Lack of testing…

 Why it is so hard to do proper testing? Everyone agrees that testing should be done. But very often the only person that actually tests the software is the programmer checking that the new function or fix she/he just wrote runs through. Can you imagine someone building a car and only testing the individual parts one at a time under ideal circumstances in a laboratory? I wouldn’t drive a car like that - would you? But we, as clients, often do not demand nor allow time for proper testing when it comes to software. Or do not want to pay for it. It is just assumed that it has been done. Many end user organizations that I have encountered will axe the more expensive vendor who defines testing costs and effort on their proposal over a vendor that is cheaper, faster and who does not mention testing costs (as usually there are not any included).

 Depending on the source one chooses to believe, 50% -90% of all IT projects fail in one way or another. The most common reason for the failure of a software project is a mismatch between the actual needs of the organization and the features of the delivered product. But the lack of testing is high on the list as well. Testing is not as hard for the brain as thinking through the real software requirements for a process that does not exist. Testing is a simple mechanical exercise.

 I have been involved in projects where the upper management and the vendor have understood the value of testing and the efforts required. Every one of these projects has been successful, on time - more or less, and within the cost. And yes, the cost of the testing process can be quite substantial. I am convinced that the cost of testing is always less than the cost of not performing the proper testing. There are always bugs in software, but in a well tested application they are hard to find. This results in better user acceptance, quicker adoption and better time-to-value.

 The cost and time savings of not testing are lost very quickly with lost data, low performance, patches upon patches from the vendor that just cause new problems and plummeting employee morale. And this applies both to the vendor and the end user organization.

 At the vendor side situation can be quite Dilbertian. I worked once for a major software vendor as a development manager. My repeated requests for a software tester were continually turned down as we were already running too expensive an operation to support our software. We had 7 people mostly working on solving customer problems caused by our buggy software. So I had to use these expensive SW engineers and analysts to do some rudimentary software testing. These software engineers had no training nor aptitude for testing software. Add to this the fact that they were testing their own software most of the time, this work was not on their job descriptions and the pitfalls are evident. The project costs were racking up, schedules were slipping and bad software was getting out!!!

 Saving in the software testing was false savings. The fact is that a software bug is 10 to 1000 times more expensive to fix if a client finds it versus the developer or tester finding it while testing the software before the release.

 So what should an end user do to get their vendor to do proper testing? The end user should require and participate in a detailed acceptance testing, require the vendor to provide testing documentation (what, how, when and by whom) and be prepared for the extra time and cost. It is cheaper in the long run. The end user can also include penalties for failing the testing. If a bug fix requires more than one line of code, the end user should not accept the patch in 24 hours or less as the fix cannot have been tested properly in that time...

 What can a vendor do to get compensated for its testing costs? Document! Clients pay to receive things and services, so these have to be tangible. Documentation about what was tested, how it was done, when it was done and whose neck is on the line is a tangible service. This will also help the vendor to get the payments as the client has explicitly accepted the vendor’s testing and thus accepted the product delivered.

 I will be the first to admit that it is an enormous mountain to move - getting the upper management – both at the vendor organization and at the client - to accept the costs and time required to do software projects right. But we, as the IT professionals, know how to do these things right if we are just allowed to. And the benefits are there for all.

 This was clearly evident from my recent project. The system went through a thorough acceptance testing, the testing and issues were documented and prioritized. The vendor did get information on what the priorities were and what was not working. The vendor was provided with clear documentation and clear requirements for project acceptance and level of testing expected of them. In a very short time, we did get the system working and providing value to the client. It is not bug free, but thus far in normal use everything appears to work just fine.

Yours sincerely,

Peter Isotupa, M.Sc (Tech)


Isotupa Consulting, Inc.