Article first published as The Importance of Utilizing a Software QA Team on Technorati.

For anybody involved in a traditional software development process this question may look outright silly. How would you assure quality without a QA team?

There are many agile organizations however, which operate without dedicated QA people. Some do it out of necessity, i.e. lack of financial resources. Many do it out of conviction. They believe that they found a more effective way of assuring quality.

It is Not a Question of Whether You Need Quality

Of course you do. Software needs to satisfy a level of quality before it is released. Assuring quality involves running a variety of test cases emulating a broad set of usage scenarios, load factors, environment combinations, edge cases etc.

Automation is an Answer

QA can be automated to a large degree. There are many tools and frameworks on the market which help with automation of unit, integration, regression, stress and performance tests. Automated test cases guarantee repeatability of the process, can be executed faster and reduce need for manual labor.

Agile QA

Agile methodologies incorporated a number of ideas to address QA needs:

  • Test-driven development (TDD) process requires developers to write test cases before writing the actual code. When developer completes code, by definition it has to pass test cases.
  • Continuous integration (CI) involves integrating code from all developers early and often – typically many times a day. Combined with automated test cases it results in catching and resolving integration issues immediately.
  • Small releases reduce risk of running into major issues or surprises.
  • Frequent releases allow to respond to and address any issues which might be found on production rapidly.

TDD and CI make developers clearly responsible for writing and execution of test cases. With a good adherence to the process it is entirely possible to develop software without a dedicated QA team.

Independent Validation

Well designed processes incorporates a system of checks and balances which provide a feedback loop, a mechanism to keep the process on track. QA process which relies solely on the Development team lacks such mechanism. QA team can bring a different perspective, add breadth to the test cases and provide independent validation of software. Hence even with a perfect agile process there is value in having a QA team. It can be however much smaller than in a traditional software development process.

What is the Right Level of Quality?

Quality is not binary. It is not a question of whether you have it or not. The right question is what is the acceptable level of quality. Marketing application would require a different level of rigor than an air traffic control software or software powering medical devices. When thinking through the QA process and team structure, a good starting point is articulation and understanding of the acceptable QA criteria.

Measuring Quality

Many QA teams use process oriented metrics such as a number of defects found which suffer from a fairly common metric ailment. They are easy to measure but not necessarily aligned with value. It is more insightful to measure effectiveness of QA by focusing on outcomes, i.e. assessing how good the software is after it has been deployed to production. Outcome oriented metrics answer questions such as:

  • Is software uptime exceeding uptime commitment?
  • Are response times below expected thresholds?
  • Are there any unhandled exceptions reported by the software?
  • What are the numbers and severity of defects found on production and reported by customers or employees?

My Practical Perspective

I have been working with agile methodologies for a number years. At my prior startup we had a small team of great QA engineers and they made a big impact on quality of our products. We had 99.99% uptime and very few issues reported by customers. At my current startup we are religious about following Test Driven Development and Continuous Integration practices. The process is working very well. It does not come for free. We spend roughly the same amount of time developing test cases as writing the actual code. It is definitely worth it. We also have a tiny but independent QA team.