Production defects in software are some of the most costly and time consuming problems a project can face, so the more you can do to mitigate them, the better. That’s why software testing is so important.
Read on for a comprehensive guide to what production defects are and how to prevent them. We’ll cover:
- Production issues in software development
- Defining the defect priority and severity
- What can you do to reduce production defects?
- Peer code reviews – static testing
- Continuous Integration
- Test in a QA environment
- Perform regression testing
- Effective defect tracking
- Empower testers to make project recommendations
- Add a warranty period
Production issues in software development
As you probably know, a production issue is any incident that occurs in the live product which causes it to not meet requirements. In software testing this refers to a defect that has only been discovered after the product has gone live, which often makes it more expensive and difficult to fix.
Production defects are really important to track because they tell you how good your testing is. When we analyse the performance of the testing function we’ll often look at the number of defects in testing, then in production, to create a defect detection percentage.
For instance, of 100 defects, you found three of them in production, so your defect detection percentage is 97%. This is a quick and easy way to get an overall understanding of how good your test process is and can help you set targets for improvement. It also often demonstrates the cost of failure, as the later you find the defect, the more it costs you to fix.
Defining the production defects priority and severity
The severity of production defects is designated by how badly the defect affects the product – if it sets the hardware on fire, it has high severity! If it’s a typo on the page, it is less severe. That said, it is really important to consider how a defect can cause issues to the end user. A typo that causes a user to fill in a necessary field incorrectly could then be classed a severe defect.
The priority of production defects refers to how soon you need to fix the problem. This is a matter of dependencies, as some issues may in themselves be low priority, but if a high priority function is blocked by that issue, it can become a much higher priority. It’s necessary to prioritise your production defects so you know what to fix first.
When defining defect priority and severity it’s best practice to take a triage approach with a tester, developer and business representative. User input is vital, as severity can be highly subjective! For example, a few years ago we were consulting with an insurance company where users would speak to customers and enter 30 fields of information into a form. They were implementing a new system, and one of the things we noticed was that in the old system, the user moved down the columns to complete them, whereas in the new system the user moved across the rows. The testers had put it as low severity because they saw it just as a design inconsistency. But when we tested it with the actual users, it turned out they all used tab to whizz through the form. So suddenly with the new system they couldn’t tab down the page and so it was taking 10 times longer to fill in! In short, you can have all the testing expertise in the world, but you need to ask an end user too.
What can you do to reduce production defects?
There’s a lot to be done early on and during a project that can prevent production defects in your software testing. Here is a helpful list of methods you can employ to prevent them.
Peer code reviews – static testing
If you’re a tester you’ll almost certainly know about static testing. While it’s less exciting than other forms of testing, it’s one of those processes that can give you much more output for your input.
Review the documentation as early as possible and check the specification and requirements for ambiguity or errors. Once the developers start coding, employ peer code reviews to improve the quality of the code. You can also get code analysis tools to automate some of the quality control, but reviewing the code regularly will help identify and reduce errors.
Continuous Integration
Again, this is about keeping the code clean. It’s a relatively new concept but it’s very effective, and is often referred to as CI/CD. Put simply, you merge your code into a central code base regularly, meaning you always have the latest test environment ready to use.
You can read a guide to continuous integration in software development here.
Test in a QA environment
Your software testing is only as good as your testing environment. Having a Quality Assurance (QA) environment to re-test changes is a great way to catch defects before they make it live.
Often it is too expensive to maintain a QA environment that’s very close to the live environment because of the amount of data or servers needed. However, by having a pre-production testing environment, you can mitigate the risk of missing production defects that don’t show up in a less accurate testing environment, without having to run such a high-cost environment for the length of the testing phase.
If you’re struggling with managing your test environment, we can help.
Perform regression testing
Regression testing is a really helpful way to catch defects that come from updates or code changes. This form of testing happens once you’ve identified defects and implemented the changes which have fixed those issues. Great – but that fix may have broken something else! So regression testing identifies areas that could have been affected by defect fixes and allows the test cases for that area to be re-run to ensure no new defects have been introduced.
How much you choose to test depends on the risk and its priority, as well as how much of an issue it will be if a production defect is found. For instance, when we’ve worked on a project that develops medical software, they have a very intensive regression policy where pretty much every test is run once more at the end of all the defect changes, because they can’t risk having any production issues. The extent to which you perform regression testing will obviously depend on your requirements, but it’s best practice to at least cover the most high risk areas.
Effective production defect tracking
You’d be amazed how many people forget to learn from their mistakes. By effectively tracking and documenting your defects (and their priority and severity) you can prevent them from happening again and manage the project better.
Root cause analysis is crucial here. While the project is going on you might see similar defects coming up, but unless you take the time to find the source of these issues and fix that, you’re going to keep getting production issues.
This is especially relevant for agile projects. Often if the team is taking a more devops approach and not documenting extensively, a tester can easily turn round to a developer in an informal way and tell them about a bug. The developer then has a quick fix for it and the defect is considered sorted. However, so often with software development, that fix can break something else in a different area, but you won’t know how or why it’s happened because there’s no record of what was done. So even if you’re working in an agile project, it’s still important to make a quick note of defects raised. And if defects keep coming, do the root cause analysis and then actually take time to fix the cause that you find!
Empower testers to make project recommendations
Following on from your root cause analysis, you may spot a significant issue in an earlier point in the project which is causing production defects further down the line.
By empowering testers to make recommendations in other areas of the business, these issues can be mitigated and you can save time and money in the testing process. This is often a culture problem, and it can be hard for you as a tester to implement. Remember to approach situations like this with coherent data and focus on the business objectives and requirements that you will be helping to meet with your suggestion.
Add a warranty period
A common issue we see is that the systems that record test defects are often completely different to production defects – the first is test management, whereas the second is service management. Because of this, there are different mechanisms for finding issues, which makes identifying where production defects came from harder.
Another problem we’ve seen is that by the time production defects are discovered, many of the experts who were working on the project have gone to work on something else, making fixing the issues even more expensive. To prevent this, we often advise that software development projects implement a warranty period. This is where key team members agree to return to support on any production defects that arise, working with service management. This is more of a devops approach and can be really successful in reducing the expense and issues caused by production defects.
Hopefully this guide will help you reduce production defects following your software testing project. If you’re unsure where to start, or want more personalised advice, get in touch with us by clicking the link below.
Leave a Reply