What is product risk?

As a TMMi Lead Assessor I often find myself in a fact-finding interview sitting across the table from a tester or test manager with a puzzled look on their face. The cause of their consternation is usually because I have asked them to explain how they deal with product risk and, perhaps strangely, they don’t feel they understand precisely what product risk is. Why is it that product risk is a difficult concept for many individuals involved in testing?  I hope this blog will shed a little light on what product risk actually is and how it is dealt with in a testing project.

Risk comes in many guises. When managing a project involving any element of testing there are at least three main types of risk that need to be taken into consideration:

1. Business risk
2. Project risk
3. Product risk

Project risk is probably the one that project managers are familiar with and are most comfortable in defining and mitigating.  Project risks involve such things as going over-budget, missing key milestones or deadlines, issues of resource availability, scope-creep, lack of support from higher management and so on.

Business risks associated with a project are usually more the concern of higher management (although project managers should realise that project risks have an impact on business risks) and revolve around the possible impact to the organisation as a whole if individual projects suffer some form of disaster. Loss of confidence in the company or corporate embarrassment resulting from a failed product or service may contribute to a falling market share and a smaller customer base which in turn leads to a reduced share price and loss of turnover or profit.

Project and business risks can be mitigated in a number of ways: planning well in advance, ensuring back-up resource is available, having an effective monitoring and control process in place, ensuring staff are adequately trained for their role are just some of the mitigation options. Project and business risks can be categorised and quantified in terms of the likelihood that something will happen and the impact (a monetary value if possible) if they do occur.

Product risk is the set of things that could go wrong with the service, software or whatever is being produced by the project.  In the same way that project and business risks are quantified (using likelihood and impact) product risks should also be categorised and measured. The main difference with a product risk is that it is normally, but not always, mitigated by appropriate testing (in contrast, a project risk cannot be mitigated by testing).

Product risks can be classified in many ways but they fall into one of two main categories, functional and non-functional. Functional risks relate to how the product may not achieve the activities it is designed to do, such as receiving data, performing calculations, producing reports and interfacing with other systems.  Non-functional product risks relate to such possible problems as not performing a (correct) calculation quickly enough or being unstable with a high number of concurrent users. Products from different domains will have some risks in common and some risks that are either unique to that domain or at least less likely to occur (or have a smaller overall impact).

What is product risk used for? Testing is a risk reduction exercise. We test things to find out if there is anything wrong with them in order to eliminate (we hope) or reduce the possibility that it will fail in production.  Identifying and evaluating product risks for their relative probability and impact leads,  during the earliest part of project and test planning, to a clearer picture of what needs to be tested and provides a means to help in determining the effort, skills and techniques required to achieve a product with minimal residual risk.

I hope it is obvious that the things that are more likely to go wrong (and that would cause the greatest impact if they did so) should, where possible, be tested first and most. A product risk assessment also enables risks to be prioritised, either as part of initial planning or dynamically during test execution. That means there will be the ability to choose to test first the features and functions that are more likely to go wrong and/or cause the greatest damage.  As a result there will be more time available to identify and correct faults that are most likely to occur and wreak the most havoc.

So why is it that I get so many puzzled looks when I mention product risk? I think it is partly a matter of terminology and partly a matter of practice.  Terminology because there are still some practitioners out there who haven’t heard of product risk and some who don’t seem to understand or care about risk as a topic. Their main concern is to start testing as soon as possible.  Practice, because most testers, when presented with the test basis (or to make it plainer, the requirements) can and do come up with a set of test ideas which can then lead to test conditions, scripts and so on without considering the relative risks. A lot of the time this is done from experience of previous releases or similar applications. In a way, the risks are being identified (because the tests themselves could be said to be a kind of documentation of the risks) but there is an important step missing. You may end up with a list of tests to perform but you are still left with the problem of deciding what is more important to test first and what may be left until later (or not tested at all if time runs out).

My view is that it is worth taking just a little extra time to discuss with subject matter experts and business representatives what could go wrong with the product and what are the potential impacts before starting on the generation of test ideas, conditions and scripts. The business users may even want to have some input into the decision of what to test first – but that’s a topic for another blog.