Software testing is a crucial part of the software development life cycle (SDLC). It involves the identification of software issues so that you can guarantee software quality and meet customer requirements. While effective testing unlocks numerous benefits, from improved software quality, cost savings, and greater customer satisfaction, most software testers will agree that the testing process can be challenging.

So, imagine a world where every test case runs smoothly, defects are caught before they become major issues, and testing methods consistently evolve to deliver high-quality software. This vision can become a reality through the integration of Artificial Intelligence (AI) and the Test Maturity Model Integration (TMMi) framework.

The Significance of Software Testing

In this context, it’s important to note a few key factors that emphasise the significance of thorough software testing in our technology-driven era:

• Rapid Technological Advancements: In a world of constant technological progress, encompassing AI-powered applications and Internet of Things (IoT) ecosystems, ensuring software reliability and robustness becomes increasingly challenging.
• Increasing Software Complexity: Modern software systems, especially those powered by AI’s intricate algorithms and neural networks, demand rigorous testing due to their complex interactions.
• User Expectations: Users today expect intuitive, agile software experiences free from glaring flaws. Minor software hiccups or security issues can have significant consequences, highlighting the need for comprehensive testing.
• Competitive Landscape: In the competitive tech industry, the ability to release high-quality software quickly is a game-changer. Effective testing is crucial for rapid, quality-driven releases.
• Security Concerns: As our reliance on software grows, so does the risk of cybersecurity threats. Software vulnerabilities can lead to severe data breaches, underscoring the importance of meticulous testing.
• AI’s Impact: AI’s influence extends into critical sectors like autonomous transportation and healthcare, making exhaustive testing essential for operational and safety standards.
• Cost of Failure: Software glitches can harm reputation and even jeopardise safety, with investing in robust testing often outweighing the consequences of potential errors.
• Regulatory Requirements: Many industries face strict regulatory testing and validation mandates, and non-compliance can result in legal and business challenges.

In essence, the increasing relevance of AI only amplifies the need for adept software testing. It’s not only crucial for ensuring the functional excellence of AI-infused applications but also for fostering user trust and maintaining a leading position in the ever-changing tech landscape.

Understanding AI in Software Testing

At its core, AI in software testing involves utilising advanced computational strategies and algorithms to enable machines to mimic human-like decision-making. This includes leveraging machine learning (ML), natural language processing (NLP), and other AI subsets to refine and automate testing processes.

These innovations empower testing systems to extract insights from historical data, adapt to dynamic software environments, and intelligently govern test case creation, execution, and defect detection.

Now, let’s explore AI’s components in software testing in more detail:

• Machine Learning: ML facilitates training on extensive datasets, enabling the identification of software patterns and anomalies, thereby improving defect detection.
• Natural Language Processing (NLP): NLP processes and evaluates textual data related to software nuances, enhancing test case creation and analytical outcomes.
• Automation: AI-driven tools autonomously execute test cases, reducing the need for manual intervention and enabling continuous testing in agile environments.
• Predictive Analytics: AI can highlight potential glitches or software risk areas by leveraging historical data and ongoing code modifications, optimising testing focus.
• Optimisation: AI fine-tunes test suites by prioritising critical test cases and expediting the identification of significant defects.

The integration of AI into software testing improves testing methods throughout the software development lifecycle, from initial planning to execution and reporting. This empowers teams to increase test coverage, expedite testing processes, and improve overall software quality.

Understanding How AI & TMMi Come Together in Software Testing 

The TMMi framework provides a structured approach for evaluating, guiding, and improving an organisation’s software testing procedures. Its primary goal is to help organisations assess and enhance the maturity of their testing capabilities.

Many advantages come with the implementation of the TMMi framework. These include:

• Improved software quality
• Increased operational efficiency
• Increased risk management
• Alignment with industry best practices

These advantages collectively contribute to the overall success of software development projects.

In essence, TMMi provides a systematic approach for evaluating and refining testing procedures. It helps organisations determine how best to incorporate AI into their testing methodologies. By employing AI technologies, organisations can automate testing tasks, fine-tune test cases, improve defect detection, and more – all in alignment with TMMi principles. 

While TMMi doesn’t inherently possess AI functionalities, it establishes a foundation for organisations to effectively integrate AI into their testing processes, ultimately resulting in improved testing maturity levels.

You’re likely well acquainted with the time-quality-cost triangle, often referred to as the project management triangle. This framework examines the interdependent relationship between time, quality, and cost within any project.

When we introduce the factor of carbon emissions, it accentuates the significance of striking a balance between environmental sustainability and conventional project constraints. This implies that achieving high-quality results within defined time and budget parameters must also consider the project’s carbon footprint. This is becoming an increasingly important aspect businesses need to consider.

Amidst today’s rapidly changing digital landscape, businesses across different industries, from technology startups to established multinational corporations, are giving increasing attention to sustainability. Thirty of the UK’s FTSE100 companies have enrolled in the United Nations Race to Zero campaign, underscoring a collective commitment to curbing environmental impact.

As companies work on reducing their environmental impact, it’s becoming vital to integrate sustainability into all aspects of their operations, including software testing. However, the challenge facing these businesses lies in finding the balance between sustainability goals and maintaining profitability and quality.

In this article, we would like to introduce you to shift left testing. This approach not only has the potential to enhance sustainability efforts but also to bring about improvements in other aspects of your business, including process efficiency, cost-effectiveness, and product quality.

What is Shift Left Testing?

Shift left testing is a software testing approach that involves conducting tests earlier in the development process. It aims to identify and rectify defects during the early stages, reducing the time and cost required for fixing them later.

For instance, traditional practices tend to put tasks like identifying defects and testing toward the end of a project. However, the shift left approach brings these tasks closer to the planning phase.

The advantages of this approach are clear. Spotting a problem in the requirements document well ahead of the product deadline, rather than right before it, can lead to significant time and budget savings. Even though this might mean higher testing costs earlier on, it often prevents unexpected expenses down the line.

At Experimentus, we take this further with our unique ‘Shift left and compress’ methodology.

What Do We Mean by Compress in ‘Shift Left and Compress’?

‘Shift left and compress’ is how we emphasise the importance of tailoring a shift left testing approach to your specific business and product. This customisation leads to better outcomes.

The key here is consistency and dedicating time to optimise your test processes. This involves factors like:

·  Uniformity of testing processes

·  Responsibility for risk assessments

·  Prioritisation of risks

·  Wait times in test environments

·  Utilisation of your test environments

By applying the shift left and compress technique, your process becomes more streamlined, which means you require fewer testers for less time to accomplish the same work. It also has the added benefit of reducing the learning curve for new testers joining projects, as there are consistent processes in place, making your projects more scalable.

What Are The Benefits of Shift Left Testing?

This approach offers several benefits:

1. Reduced overall development costs: By catching risks and bugs early, you avoid putting too much effort and resources into flawed software later in the development lifecycle, ultimately saving money.

2.   More efficient processes: Implementing a shift left approach naturally encourages better process consistency. This leads to greater project uniformity and smoother operations.

3.   Easier stakeholder approval: Starting testing earlier helps showcase the value of testing sooner, especially when stakeholders are well-informed.

4.   Improved product quality: By preventing software from being built on top of defects, you end up with better software quality, a stronger code foundation, and fewer patches.

5.   Quicker project completion: If you’re working on a project where time is of the essence, early and efficient testing could speed up the time it takes to bring your product to market.

Given these benefits, it makes business sense to use shift left testing (and it’s likely something you should consider anyway) – it can significantly enhance your development process, improve software quality, and ultimately lead to more efficient and cost-effective project outcomes. This becomes even more relevant today due to the increasing focus on sustainability and environmentally friendly practices.

How Can Shift Left Testing Improve Sustainability?

While shift left testing does not directly improve your carbon footprint, it can indirectly contribute to environmental sustainability in several ways:

• Promoting efficiency through earlier testing and the strategic implementation of automation reduces the need for excessive manual testing, leading to resource conservation and enhanced sustainability.
• The minimisation of rework and the reduction of extensive testing cycles results in optimised efficiency, while proactive defect identification and resolution mitigate the potential for product failures and subsequent waste, thereby supporting environmental well-being.
• Involving testers and quality assurance professionals right from the outset fosters a culture of quality and accountability. This ensures that sustainable practices are woven into the development process right from the beginning.

Nevertheless, we recognise that from a green IT standpoint, quantifying the impact of shift left testing on your carbon footprint can be challenging. However, other metrics can indicate positive progress toward environmental sustainability, such as reduced defect occurrences, improved product stability, and smoother development timelines.

Conclusion

When implemented correctly, the shift left testing approach doesn’t undermine quality or require any sacrifices. On the contrary, it works to improve efficiency and overall processes.

This method works to fine-tune the development process without imposing undue strains on quality or demanding undue trade-offs from the project team. Rather than compromising the end product’s quality, or burdening the team with excessive demands, shift left testing emerges as a guiding principle that actually improves the effectiveness of the entire project journey while also contributing to sustainability goals.

Ready to embrace sustainable software testing practices and enhance your project outcomes? Explore the potential of shift left testing and its ability to improve efficiency, quality, and environmental impact.

The growth and widespread adoption of information technology systems has transformed how we live, work, and communicate. However, this digital revolution comes with a significant environmental cost. Projections suggest that by 2030, IT systems will account for a staggering 20% of the global energy demand. To put this into perspective, the transportation industry currently represents 25% of the global energy consumption. The fact that IT systems are approaching this level of energy consumption highlights the seriousness of the issue at hand.

The production of energy, predominantly from fossil fuel sources, is a leading cause of greenhouse gas emissions, including carbon dioxide (CO2), which contributes to climate change. As the IT industry expands, it becomes increasingly important to take responsibility for its environmental impact and actively work towards reducing energy consumption and associated CO2 emissions.

In this context, software assumes a crucial role in tackling the environmental impact of the IT industry, offering potential solutions to combat deforestation and mitigate emissions. However, it is essential to recognise that software alone does not consume energy or emit harmful substances. The challenge lies in how software is developed and utilised.

By optimising resource utilisation in software testing, organisations can achieve several benefits. Firstly, they can reduce the overall cost of testing by minimising the hardware and energy requirements. This can result in significant cost savings, especially for large-scale testing efforts. Secondly, the testing process becomes more efficient and streamlined, enabling faster delivery cycles and shorter time-to-market. Most importantly, by conserving resources, organisations contribute to sustainability efforts by reducing their environmental impact and carbon footprint.

Greener Testing Practices and Techniques

To ensure sustainable software testing, organisations can adopt various greener testing practices and techniques that focus on minimising the environmental impact of testing activities. These practices aim to improve testing efficiency and, if properly deployed, will help reduce energy consumption without compromising the quality of the testing carried out. Here are some key strategies (some of which you will already be utilising) to consider and look at through a carbon lens:

• Risk Based Testing: By prioritising tests using a risk-based testing approach, organisations can allocate their resources more efficiently by focusing on critical functionality, high-risk areas, and test cases that are most likely to uncover defects. This means investing more time and effort in testing scenarios that have a higher impact on the software’s functionality, stability and security.
• Shift Left: Defect prevention and early detection are key practices to implement for sustainable software testing. By prioritising these strategies, organisations can significantly reduce energy consumption, minimise waste, and enhance the overall efficiency of the software development process.  Effective static testing, root cause analysis of defects and effective lessons learnt/retrospective processes facilitate this.
• Test Case Optimisation: Reducing test case redundancy is fundamental to greener testing. Organisations can streamline the testing process by analysing and eliminating redundant or overlapping test cases, reducing resource consumption, and minimising energy usage. This optimisation ensures that only essential and unique test cases are executed, saving time and energy.
• Test Execution Sequencing: Optimising the sequence in which test cases are executed can contribute to energy efficiency. By prioritising tests that require similar resources or configurations, organisations can minimise the need for frequent setup changes and reduce energy consumption associated with system reboots or reconfigurations. This approach helps maximise the utilisation of resources and minimise unnecessary energy consumption.
• Energy-Efficient Testing Approaches: Organisations can adopt specific testing approaches to reduce energy consumption. For example, leveraging virtualised testing environments allows multiple virtual machines to share physical resources, optimising resource utilisation and minimising energy waste. Similarly, utilising cloud-based testing platforms provides scalable and on-demand infrastructure, enabling organisations to allocate resources efficiently and reduce energy consumption during testing.
• Test Environment Management: Efficient management of test environments can significantly impact energy consumption. By closely monitoring and managing the availability and utilisation of testing resources, organisations can prevent resource wastage and minimise energy consumption. This includes implementing automated provisioning and deprovisioning processes, ensuring resources are only active when required.
• Energy aware: As an industry we have introduced a number of non-functional testing levels such as performance, security and user experience testing. We now need to think about the environmental impact of the application, it’s potential energy usage over the lifetime of the application and how we can test that this is minimised.
• Continuous Testing and Test Automation: Implementing continuous testing and test automation practices can improve overall testing efficiency and reduce energy consumption. Organisations can optimise resource utilisation, minimise manual effort, and conserve energy by automating repetitive and time-consuming testing tasks. Continuous testing practices enable the early detection of defects, reducing the need for extensive retesting and its associated energy consumption.

By incorporating these greener testing practices and techniques, testing organisations can contribute to sustainable software development practices, minimising energy consumption and reducing their environmental impact.

In order to help organisations evaluate where they are we have developed a Sustainable Software Testing survey that reports at a high level on an organisation’s progress in this area and will continue to evolve it to help align the results with the impact on one’s carbon footprint.

Final thoughts

As the software industry continues to evolve, integrating sustainable IT principles into testing becomes increasingly important. Organisations that prioritise sustainable practices in software testing not only contribute to a greener planet but also gain a competitive advantage, as they can deliver high-quality software efficiently while minimising their environmental impact.

Start looking at what you do and how you do it using a carbon lens and identify some things you can change and measure. It may not be possible to make highly accurate objective measures immediately, but measures can show that you are moving in the right direction.  You can therefore make informed decisions and demonstrate progress.

As a community of people in the software testing space, if we all look at what we do and how we use IT through a carbon lens we can all make changes (big and small) that will help to improve our organisation’s carbon footprint.

At Experimentus, in collaboration with Dominic Plummer, Head of Sustainable Quality Engineering, Enterprise Engineering, and Technology at NatWest, we have been actively engaged in exploring ways to enhance the sustainability of software testing and establish effective measurement techniques.

As a result of our collaborative efforts, we have developed a free survey that enables you to quickly assess, visualise, and establish a baseline for the level of sustainability within their software testing practices.

Join us in shaping a sustainable future for software testing!

In the current IT world, there is a massive drive to adopt modern technologies to enhance the software development lifecycle mechanism, accelerate releases, accelerate Time to Value, and reduce time-to-market. Quality assurance is an essential part of the SDLC process, but unfortunately, according to recent research, companies tend to exhaust their IT budgets on inefficient routines rather than on innovation. Test environment management is one such area that is often overlooked or seen as a discretionary expense. While it’s one of the most significant factors that could contribute towards successful quality assurance measures. Software quality automation is widespread; however, in the majority of organisations, there is a gap in the process which can be enhanced by Test Environment Management (TEM).

One of the most common issues that remain unsolved, especially among distributed teams, is shared environment utilisation. Environment conflicts, complex interfaces, inadequate environment config details, and tracking of deployments/defects are a few other common challenges encountered in day-to-day delivery. These challenges introduce significant risks to the quality of projects and their timely delivery.

Test environment management can be that one-stop solution that could help timely project delivery alongside driving a quality outcome.

What is Test Environment Management?

If you have ever been engaged in the SDLC process, then you’ve probably dealt with test environments. When preparing for a software release, many concerns about test environments come forth, which are generally hidden.

You may have encountered or been part of conversations that include statements such as:

• “Please don’t deploy to staging; other projects are load testing on it,”
• “Can you refresh the test data so that the business team can regression test on it?”
• “It failed in live because we didn’t have the same version of the software in the test environment. Another program moved the production environment forward, and we didn’t know.”

These types of requests are often raised ad-hoc, and how those requests are processed through effective communication is nothing but test environment management. These communications comprise risk elements like tracking dependencies, scheduling data refreshes, building new testing environments, and dealing with conflicts between end users.

Test Environment Management (TEM) is a set of processes that helps organisations to speed up their software releases with active collaboration between all team members, capture environment demands, establish a simple and transparent environment utilisation, organise effective cooperation through planning and scheduling resources, and control their lifecycles.

It is an essential practice that helps streamline the software delivery process by providing validated, stable, and available test environments to delivery teams when they need them to execute test scenarios or reproduce incidents.

An effective and efficient test environment management process will help drive focus on the following key deliverables:

• Ensuring complete transparency of test environments, including software version history, deployment details, and availability of test environments. A clear understanding & control over your environment creates a path to optimise and innovate.
• Allocation of test environments across project teams, individual test engineers, and DevOps teams.
• Tracking of resource utilisation, defining simplified methods to acquire and release resources (environments).
• Organising simultaneous shared access to environments.
• On-demand creation of new test environments, upgrading or decommissioning existing outdated test environments to sustain the necessary resources.
• Continuous improvement, innovation, and automation to eliminate the amount of routine manual tasks and optimise infrastructure costs.
• Standardised environment usage by simplifying daily tasks and operations.

Why is Test Environment Management important?

Most organisations’ project / DevOps / Support teams typically need a number of test environments for each production application. Depending on the circumstances, this could be a single test environment or multiple environments with different purposes on a path to production.

Mostly IT architecture comprises distributed systems, various integrations, development, and testing across shared environments. All of these factors add additional complexity to managing projects, releases, and even achieving business goals.

With the rapid rise of Agile and DevOps culture, the necessity for regular and even more frequent software releases has spiked significantly, sometimes to the level of daily and hourly or on-demand. It is, therefore, important to have a TEM practice to manage this and assist in achieving desired outcomes.

A well-defined Test Environment Management practice helps answers the questions below:

• Is there a way to effectively manage & coordinate all scheduled releases?
• How to avoid utilisation conflicts between different teams (DevOps, QA & Release Management) when using a shared environment?
• Is the information regarding the availability of a shared environment up-to-date?
• Is it possible to reuse/re-purpose the same test environment again?
• How can you track all software package versions across all environments?

TEM enables companies to:

• Significantly increase team productivity. This is achieved by eliminating ad-hoc methods of communication & complicated tools interaction to help identify development and test environment availability. Also, by tracking resource usage, providing an easy mechanism to acquire and release resources, and organising simultaneous shared access with specially designed TEM tooling, such as Enov8 or Plutora. Any distributed team can easily access the schedule for shared environments, book use of the environment, and deal with a single point of contact for this.
• Accelerate software releases and time to market. When organisations comply with standardised and repeatable TEM processes, it will significantly help to speed up go-to-market time.
• Cut IT and Test Environment costs. Properly implemented Test Environment Management eliminates a number of elements that risk leading to overspending, helps to identify underutilised resources, and prevents the creation of extra test environments without real demand.

Test environment-related problems can have a huge impact on the success of project teams, as research has shown that they can consume almost a third of the project teams’ effort. Unfortunately, testers spend an average of one day per week identifying and resolving environmental issues, resulting in a 20% reduction in resource availability. Moreover, a third of environments are not adequately or efficiently utilised. These unnecessary costs have an adverse economic impact and contribute to unsustainable practices. It is essential to ensure test environment-related problems are properly managed so project teams can work effectively and maintain crucial “green” standards.

By appropriately employing good environment management processes, both the costs and time wasted as a result of these statistics and examples can be dramatically reduced.

Before and after view of TEM implementation:

What are your main frustrations with your test environments? By taking our Test Environment Survey you’ll get a personalised summary of how you measure up to best practices in your industry.

Article written by Rajiv Radha, TEM Senior Consultant, Experimentus

As businesses become evermore complex and reliant on IT solutions, the need for faster and more comprehensive solutions grows. In order to exploit these emerging technologies and meet business demand, organisations should turn their attention to internal processes as well as engage with third parties, system integrators, and specialist consultancies to successfully move to DevOps, Agile, and AI techniques.

There has always been a desire to build quality into the application rather than the more costly option of inspecting it at the end of the project. Organisations typically have specialist quality teams in place to help manage the risk.

As organisations prioritise delivering more comprehensive IT-related projects by using different development methodologies and technologies, it has become more challenging for them to step back and evaluate how efficient and effective their current software testing processes are. It has also become harder for them to find the time to analyse how efficient they are and be able to identify, create, and deploy new ways of working that will benefit the whole business.

TMMi – The Test Maturity Model integration

In recent years, a plethora of test maturity models have been developed to help organisations identify and implement software testing process improvements. The Test Maturity Model integration (TMMi) has become the de facto standard for software test process improvement activity and provides a formal independent process for organisations to be certified against if required.

First introduced in 2005, the TMMi model was based on the original TMM model from the Illinois Institute of Technology (1996), incorporating ISTQB, IEEE, CMMi and other software testing models used in the industry. This independent model has been adopted worldwide and has helped organisations improve their software quality testing processes, reduce risk, and adopt a Shift Left approach.

TMMi is a vital tool for assessing and benchmarking where an organisation is, identifying the
process improvements that need to be incorporated and helping develop a process
improvement plan to address the issues. Most organisations we have worked with aim for
TMMi Level 3, and once this has been achieved, then target TMMi level 5.

TMMi Roadmap

A typical road map for this might be an assessment to identify strengths and weaknesses
against TMMi Level 3, develop a process improvement plan, define, and implement the
process improvements and run some health checks to validate that the benefits are being
realised.

Once this has been successfully piloted, the changes can then be rolled out across other
programmes of work. This ensures that all projects benefit from the investment that’s been
made in the changes.

Over the years, we have seen many organisations using TMMi purely to drive their own
internal process improvement activities, but also a number of organisations wanting to
achieve their certification as well.

No matter the reason for utilising the TMMi model, our roadmap for using TMMi typically
starts with an informal assessment to establish where an organisation is against a specific
level of TMMi, and the output of this assessment is used as a baseline and input into the
process improvement plan (PIP).

The PIP can be reviewed and prioritised internally to drive the process improvement activity
against an agreed timeline. As improvements are implemented, health checks can be
conducted to ensure the anticipated benefits are realised, and further work can be conducted
on the process improvement activities. This cycle can be repeated to ensure the process
improvements are adopted by other projects and institutionalised across the organisation to
maximise the benefits.

In our experience, TMMI Level 3 is the most difficult to achieve as it requires the process
areas to be defined, documented, and implemented across the organisation. Level 3 also looks
for appropriate consistency of approach across the organisation. Once TMMi Level 3 has
been achieved, the organisation may look at TMMi level 4 (Measured) and TMMi level 5
(Optimised) as the next goal. However, as mentioned, organisations do not need to be
certified but can assess themselves against the TMMi model to demonstrate they are
achieving the benefits of the targeted level of maturity.

For organisations targeting or requiring TMMi certification, once the assessment has been
conducted and any necessary process improvements made, the organisation can go for formal
certification. This needs to be conducted by accredited TMMi assessors, following the
guidelines laid down by the TMMi Foundation for conducting certification assessments. The
results will be presented to the TMMI Foundation, which will formally issue the relevant
certification documentation.

The key to success is understanding the TMMi model and how it can support your process
improvement activities. Furthermore, it can be achieved reasonably easily as the model is
freely available from the TMMi Foundation. The TMMi Professional qualification can also
be obtained if required, and individuals within the organisation can become an accredited
TMMi Assessor or Lead Assessor.

Based on recent research, the TMMi Foundation published the following results showing the
main reasons organisations adopted TMMi. One of the key considerations was to use an
industry-recognised standard that supported modern development approaches.

The benefits of utilising TMMi

We have seen significant benefits achieved by organisations utilising TMMi for their process
improvement and helping to prioritise what they should focus on. The following are just
some examples of benefits achieved by various organisations across different Industries at
different levels of TMMi:

Using TMMi as the measurement part of a process improvement programme, Experimentus
has typically delivered these types of results:

● A UK bank saved 8% of their entire IT budget and improved development output by 12%
● An insurer saved £440k on a £2m project
● A retailer saves at least 12% on every project after the improvements
● A test services company has increased their efficiency by 8%
● A Bank decreased the overall testing estimates for the current release by 32%
● Cost of testing was reduced by 28%

Using the TMMi model to understand your software test maturity might not be suitable for
every organisation, but if it helps people to think about managing the risk and quality of what
they do and looking for ways of becoming more effective and efficient within their
organisation, then it might be worth looking at.

If you’re interested in finding out how mature your test process is against the TMMi
model, then please feel free to use our free TMMi eSurvey.

It will only take 15 minutes to complete, and you will receive a report identifying strengths and weaknesses against all levels of the TMMi model and a comparison of how you score against other organisations in your industry sector.