Article first published as The Importance of Utilizing a Software QA Team on Technorati.
For anybody involved in a traditional software development process this question may look outright silly. How would you assure quality without a QA team?
There are many agile organizations however, which operate without dedicated QA people. Some do it out of necessity, i.e. lack of financial resources. Many do it out of conviction. They believe that they found a more effective way of assuring quality.
It is Not a Question of Whether You Need Quality
Of course you do. Software needs to satisfy a level of quality before it is released. Assuring quality involves running a variety of test cases emulating a broad set of usage scenarios, load factors, environment combinations, edge cases etc.
Automation is an Answer
QA can be automated to a large degree. There are many tools and frameworks on the market which help with automation of unit, integration, regression, stress and performance tests. Automated test cases guarantee repeatability of the process, can be executed faster and reduce need for manual labor.
Agile QA
Agile methodologies incorporated a number of ideas to address QA needs:
- Test-driven development (TDD) process requires developers to write test cases before writing the actual code. When developer completes code, by definition it has to pass test cases.
- Continuous integration (CI) involves integrating code from all developers early and often – typically many times a day. Combined with automated test cases it results in catching and resolving integration issues immediately.
- Small releases reduce risk of running into major issues or surprises.
- Frequent releases allow to respond to and address any issues which might be found on production rapidly.
TDD and CI make developers clearly responsible for writing and execution of test cases. With a good adherence to the process it is entirely possible to develop software without a dedicated QA team.
Independent Validation
Well designed processes incorporates a system of checks and balances which provide a feedback loop, a mechanism to keep the process on track. QA process which relies solely on the Development team lacks such mechanism. QA team can bring a different perspective, add breadth to the test cases and provide independent validation of software. Hence even with a perfect agile process there is value in having a QA team. It can be however much smaller than in a traditional software development process.
What is the Right Level of Quality?
Quality is not binary. It is not a question of whether you have it or not. The right question is what is the acceptable level of quality. Marketing application would require a different level of rigor than an air traffic control software or software powering medical devices. When thinking through the QA process and team structure, a good starting point is articulation and understanding of the acceptable QA criteria.
Measuring Quality
Many QA teams use process oriented metrics such as a number of defects found which suffer from a fairly common metric ailment. They are easy to measure but not necessarily aligned with value. It is more insightful to measure effectiveness of QA by focusing on outcomes, i.e. assessing how good the software is after it has been deployed to production. Outcome oriented metrics answer questions such as:
- Is software uptime exceeding uptime commitment?
- Are response times below expected thresholds?
- Are there any unhandled exceptions reported by the software?
- What are the numbers and severity of defects found on production and reported by customers or employees?
My Practical Perspective
I have been working with agile methodologies for a number years. At my prior startup we had a small team of great QA engineers and they made a big impact on quality of our products. We had 99.99% uptime and very few issues reported by customers. At my current startup we are religious about following Test Driven Development and Continuous Integration practices. The process is working very well. It does not come for free. We spend roughly the same amount of time developing test cases as writing the actual code. It is definitely worth it. We also have a tiny but independent QA team.
Robert Tekiela
CTO Insights 
I rarely get to say this about posts about testing –> What a well-balanced and thoughtful opinion piece on QA! Thank you!
A footnote to history, I created Selenium for testing web apps originally out of a lack of financial resources to hire a QA department. (Commercial tools were more expensive than my servers, and there were no QA staff available.) Only later, did I pick up the conviction that automation’s the right way to do it. 🙂
Jason,
Thank you for the nice words. I am a big fan of Selenium 🙂
Bob
Hello Robert – good insights
you are correct in your assertion that Quality is achieved through test case driven QA. It is also a fact that quality can be measured and is not “fuzzy”.
There are qualities regarding functional adherence ( based upon what was to be built)
There are operational qualities (the amount of expected throughput, latency, availability)
There are also qualities regarding the user experience.
All of there qualities are expressions of requirements ( some explicit, some implicit) that should be meet to make the application successful. I have found that in working with clients there are over 50 qualities that can be measured to understand the overall behavioral characteristics of an application.
Too often QA is left in the dark until they are handed a code base. Instead, their independence should be combined with prepared knowledge through active participation in the beginning of the development process which will make a big difference in meeting expectations.
Sheppard,
Thank you for adding breadth to the thinking on the subject of QA. There are certainly many facets and dimensions.
Bob
Interesting article.
It’s true that TDD and CI are effective in Agile processes. But you have to be aware that without a specialized QA team your tests will be driven by people with a development background which in a sense will lead to miss some “crazy” or “out of bounds” testing that your end-user may do.
I’ve used Selenium and other automated test tool (French SalomeTMF automation modules) and despite the fact that it’s a very efficient to improve the testing process it requires days/weeks of test cases maintenance and development.
Gerard
I think every organization need QA team and it will help to progress in your business.
Developers writing test cases – Does this happen? Do they have the patience to think and write the test cases? If the product is very simple, probably yes. Eventhough, I agree that we want developers to think about quality when developing, we need QA Team