Driving Quality: Sustainable Software Testing Insights from our 9 Month Survey Analysis

Introduction

This analysis summarises the key insights from a survey of professionals on sustainability practices in software testing at their organisations. The survey has been live from May 2023 to February 2024 to understand how organisations align testing goals with sustainability objectives, manage risks and quality, and monitor testing performance.

The survey findings indicate that while many organisations recognise the importance of sustainability, there is still room for improvement in aligning testing processes with sustainability goals. A majority of respondents said their organisations do not have formal processes to measure the environmental impact of testing activities. There was also limited monitoring of resource usage and waste generation related to testing.

Some positive trends included an increased focus on automating test cases to reduce manual effort and reliance on cloud-based testing tools to make processes more efficient. A few organisations have also implemented practices such as code reviews and monitoring to catch issues early and minimise rework.

In conclusion, the key insights from the survey suggest that sustainability in software testing is an emerging area of focus for many organisations. Formalising processes to set goals, measure impact, and continuously enhance practices can help organisations strengthen their sustainability performance over time.

The Dataset

The dataset provided detailed information on participant responses to the survey. It included columns for Participant ID, Date Submitted, Country, City, Company/Organisation, followed by responses to specific questions.   Each question was related to sustainable software testing practices, covering topics like testing goals, risk-based testing approaches, monitoring progress, test script prioritisation, and more.

Given the structure and content of the dataset, we could analyse trends in responses, aggregate scores by category, and identify areas where participants are performing well or may need improvements. The analysis informs the creation of short sharp messages that can discuss common challenges, best practices, and insights into sustainable software testing based on the survey data.

The Analysis started by examining the data to understand its structure  and identify any initial insights and any cleaning that may be required i.e. test data removal, columns not relevant for the purpose of the analysis etc.

Analysis Action Plan

Given the structure of the dataset and the specific columns highlighted, we were able to perform several analyses to extract insights, trends, and observations that could be valuable.

The plan of action was to perform the following:

• Category Scores Analysis: Calculate average scores for each category (Process, Environment, Housekeeping, Measurements) and the overall score. This will help identify which areas participants are performing well in and which areas need improvement.
• Dynamic Text Response Analysis: Analyse the distribution of dynamic text responses for each category and overall. This can provide insight into common feedback themes.
• Correlation Analysis: Explore correlations between different categories to see if performance in one area relates to performance in others.

The Findings, Trends, Insights, Observations

In this section, our data analysis revealed the following key discoveries.  

What are the average scores for each category and the overall score?

These scores provide an overview of the performance in different categories based on the survey responses. These scores provide insights into the performance across different sustainability categories in software testing practices, with the Process category scoring the highest on average and the Measurements category the lowest. The overall average score indicates a moderate level of sustainability integration into software testing practices among the respondents.

Message – Our analysis of the data has provided insightful findings on the average scores for each category and the overall score in terms of sustainability practices. The percentage figures highlight areas of strength and opportunities for improvement, guiding us towards more environmentally friendly and efficient processes. This concise overview serves as a call to action for continuous improvement in our sustainability efforts.

Sustainability integration in software testing varies widely across organisations, with average scores ranging from 57-71% across different categories. This highlights both achievements and opportunities for improvement, signalling the diversity in practices and the potential for shared learning.

What are the standard deviations for the scores in each category?

• Process                      23%
• Environments           26%
• Housekeeping           29%
• Measurements         27%
• Overall Score            23%

These values represent the variability or dispersion of the scores within each category. A higher standard deviation indicates greater variability among the scores in that category.

Message– The standard deviations for the scores across different categories indicate a moderate level of variability in responses. This suggests that while there are areas of consensus, there are also diverse opinions and practices within each category. It highlights the importance of tailored strategies to address specific needs and opportunities for improvement in our sustainability efforts.

From the average scores and their standard deviations, what insights can we note in relation to the integration of sustainability practices within software testing?

These figures provide insights into the integration of sustainability practices within software testing in several ways:

• Average Scores: The average scores indicate the general level of integration of sustainability practices across different areas. Higher scores suggest better integration and adoption of sustainable practices. For instance, a higher average in the “Process” category suggests that the processes involved in software testing are more aligned with sustainability goals.
• Standard Deviations: The standard deviations indicate the variability or consistency of the scores within each category. A lower standard deviation suggests that the scores are more consistent, indicating a uniform level of integration of sustainability practices across the respondents. Conversely, a higher standard deviation indicates greater variability, suggesting differences in how sustainability practices are integrated across different respondents or areas.

Together, these metrics can help identify areas of strength and opportunities for improvement in integrating sustainability practices within software testing. They can guide targeted efforts to enhance sustainability in areas with lower average scores or higher variability, ultimately contributing to more sustainable software development practices.

Message – This reveals a clear picture: while there’s a strong effort towards integrating sustainability in software testing, the journey is still ongoing. The variation in scores across categories shows us that practices differ widely, signalling both achievements and areas needing attention. This diversity in performance underscores the opportunity for shared learning and collective improvement towards a more sustainable future in software testing.

There are achievements in integrating sustainability into software testing, there are also clear opportunities for improvement. By focusing on areas with lower average scores or higher variability, organisations can identify where they need to enhance their sustainability practices. Learning from areas or organisations with more consistent performance can guide efforts to strengthen sustainability integration, moving towards a more sustainable future in software testing.

What percentage of participants scored highest and conversely the lowest?

• Percentage of participants who scored the highest: 7.0%
• Percentage of participants who scored the lowest: 4.0%

These percentages indicate that 7% of participants scored the highest in the overall score, while 4% scored the lowest. This provides insight into the distribution of scores among participants, highlighting the range of performance in integrating sustainability practices within their software testing processes.

Key Insights:

• Consistent Performance in Process: The average score in the “Process” category is the highest among all categories at 71%, indicating that participants generally perform well in integrating sustainability practices into their processes.
• Opportunities for Improvement: The “Housekeeping” and “Measurements” categories have the lowest average scores at 57%, suggesting these are areas where participants could focus on improving their sustainability practices.
• Variability in Integration of Practices: The standard deviations indicate variability in how sustainability practices are integrated across participants, with “Housekeeping” showing the highest variability at 29%. This suggests that while some participants are excelling in this area, others may need additional support or resources.
• High Achievers and Room for Growth: With 7% of participants scoring the highest and 4% scoring the lowest, there’s a relatively small but significant range in performance. This highlights both the presence of high achievers in sustainability practices and the opportunity to elevate those at the lower end of the spectrum.

These insights can guide targeted efforts to enhance sustainability in software testing, focusing on areas with lower scores and higher variability, and leveraging the practices of high achievers to elevate overall performance.

Message – The analysis highlights a diverse competitive landscape in sustainability practices within software testing, showing a spectrum from leaders (7%) to those just beginning (4%), pointing to significant opportunities for growth and learning industry-wide.

While some organisations are making efforts to integrate sustainability into their software testing processes, there’s a significant opportunity for growth and improvement. Learning from the high performers could provide valuable insights for those at the lower end of the spectrum, indicating a path forward for enhancing sustainability practices across the board.

What organisations with high “Process” scores differ from those with lower scores in terms of sustainability integration?

The comparison between organisations with high and low Process scores in terms of sustainability integration reveals significant differences across various areas:

• Environments: Organisations with high Process scores average 74%, compared to 51% for those with lower scores, a difference of 23%.
• Housekeeping: There’s a 27% higher average score (70% vs. 43%) in Housekeeping for organisations with high Process scores.
• Measurements: In Measurements, the difference is 31%, with high Process score organisations averaging 72% compared to 41% for lower scores.
• Overall Score: Reflecting these differences, the Overall Score averages 76% for organisations with high Process scores, versus 47% for those with lower scores, marking a 29% difference.

These insights reveal that effective process integration plays a crucial role in the adoption of sustainability practices within organisations. Organisations that score higher in the Process category tend to integrate sustainability more effectively across all areas, suggesting a strong correlation between well-established processes and the comprehensive integration of sustainability practices.

Message– Effective process integration is key to adopting sustainability practices, with a strong correlation between well-established processes and comprehensive sustainability integration within organisations.

What specific practices contribute to the success of organisations with high Process scores in integrating sustainability into their software testing processes?

These practices include:

• Alignment with Business Goals: Ensuring that testing goals and objectives are closely aligned with the broader business goals, including sustainability objectives. This alignment helps in prioritising testing activities that contribute to sustainability.
• Risk-Based Approach: Establishing an approach to testing based on identified risks and quality goals. This allows organisations to focus their resources on areas with the highest impact on sustainability.
• Monitoring and Measurement: Regularly monitoring and measuring the progress and performance of testing activities against predefined goals and expectations. This includes measuring the impact of testing activities on the environment and incorporating sustainability measures.
• Clear Ownership and Responsibilities: Having clear ownership of testing and quality activities, including sustainability responsibilities. This ensures that there is accountability for integrating sustainability into testing practices.
• Organisational Processes and Frameworks: Deploying organisational test processes or frameworks that incorporate sustainability considerations. This includes defining roles and job descriptions that emphasise sustainability responsibilities.

Organisations that excel in these areas tend to have higher Process scores, indicating a more mature integration of sustainability practices within their software testing processes.

Message – Achieve sustainable software testing by aligning goals with business sustainability, prioritising risks, measuring environmental impact, assigning clear ownership, and embedding sustainability into organisational frameworks.

What are the key factors contributing to the variability in scores across the categories of Process, Environments, Housekeeping, and Measurements?

The correlation figures, provide a clearer understanding of the key factors contributing to the variability in scores across the categories of Process, Environments, Housekeeping, and Measurements:

• Environments and Measurements also show a high degree of correlation (77%), suggesting that environmental initiatives are closely tied to the metrics used to measure sustainability.
• Strong Correlation Between Categories: The correlations between all categories are indeed strong, with percentages ranging from 66% to 82%. This confirms that improvements or declines in one area are likely to be associated with similar changes in the others, indicating interconnectedness and mutual influence among the categories.
• Highest Correlation: The highest correlation is between Housekeeping and Measurements (82%), underscoring the close link between operational practices and the metrics used to measure sustainability outcomes. This suggests that enhancements in housekeeping practices can have a direct and significant impact on sustainability measurements.
• Process as a Central Factor: The Process category indeed shows strong correlations with all other categories, with a 76% correlation with Measurements and a 71% correlation with Environments. This reinforces the idea that the processes an organisation implements for sustainability are crucial for achieving high performance across all sustainability integration areas.
• Interconnectedness of Categories: The strong correlations across all categories, as highlighted by the percentages, confirm the interconnectedness of sustainability practices. This interconnectedness suggests that effective integration in one area can enhance performance in others, emphasising the need for a holistic approach to sustainability.

These findings underscore the significance of a holistic approach to sustainability, illustrating how efforts in one area not only led to improvements across the board but also highlight the interconnectedness of various categories. Supported by rounded percentages, the insights advocate for focusing on foundational process improvements, which are pivotal in driving positive effects across all areas of sustainability integration.

Message – Efforts in sustainability are deeply interconnected; focusing on foundational improvements can drive widespread positive impacts across all areas of integration.

How do low scoring organisations approach sustainability integration in testing?

Low-scoring organisations may approach the integration of sustainability measures differently due to various factors. While the data does not provide direct insights into the specific practices of low-scoring organisations, we can infer some potential approaches and challenges based on the lower scores across the categories:

• Limited Alignment with Business Goals: Organisations may not fully integrate their testing goals with sustainability objectives, often due to a lack of awareness or understanding of sustainability’s role within testing practices.
• Resource Constraints and Limited Resources: They face limitations in budget, personnel, time, and technological resources, restricting their ability to implement effective sustainability practices across all testing activities.
• Early Stages of Sustainability Practice Maturity and Lack of Expertise: Many organisations are in the initial phases of developing sustainability practices, experimenting to find suitable approaches, and often lack specialised knowledge or expertise in sustainability integration.
• Lack of Awareness and Training: A significant gap in awareness and training on sustainability in software testing exists among staff, leading to insufficient initiative or motivation to incorporate sustainable practices.
• Inadequate Measurement, Monitoring, and Reporting: Low-scoring organisations often do not have robust systems for measuring and monitoring the impact of their testing activities on sustainability, coupled with struggles in accurately measuring sustainability performance and reporting it transparently.
• Focus on Immediate Testing Needs Over Long-Term Goals and Inadequate Processes: There is a tendency to prioritise immediate testing needs and deadlines over long-term sustainability goals, often resulting from underdeveloped or inefficient processes for integrating sustainability into business operations.
• Poor Stakeholder Engagement: Insufficient engagement with stakeholders, including employees, customers, and the community, regarding sustainability goals and practices, can hinder progress and alignment with broader sustainability objectives.

To effectively boost sustainability in software testing, a strategic approach is essential, focusing on aligning business and sustainability goals, overcoming resource constraints, maturing sustainability practices, enhancing staff awareness and training, and establishing robust measurement and monitoring systems, thereby fostering increased awareness and stakeholder engagement for comprehensive integration.

Message– To enhance sustainability in software testing, organisations should align testing goals with sustainability objectives, overcome resource constraints, mature their sustainability practices, improve staff awareness and training, establish robust measurement systems, and balance immediate needs with long-term sustainability goals.

Measurement and monitoring systems are critical yet lacking, with the Measurements category scoring lowest. Developing metrics to evaluate environmental impact is essential for progress tracking and continuous enhancement.

What observations can be made about the ‘Housekeeping’ category, as to the potential factors contributing to its low scores based on common challenges organisations face in this area?

• Inadequate Training and Awareness: Employees may not be sufficiently trained or aware of the importance of sustainability practices in their daily operations, leading to less effective housekeeping practices.
• Lack of Clear Policies and Procedures: The absence of clear, documented policies and procedures for sustainability practices can result in inconsistent implementation of housekeeping measures.
• Resource Constraints: Limited resources, such as budget, personnel, or technology, can hinder the ability to implement or maintain effective sustainability practices in housekeeping.
• Operational Priorities: Housekeeping practices may be deprioritised in favour of other operational needs or objectives, leading to less attention and investment in sustainability efforts in this area.
• Measurement and Feedback: A lack of effective mechanisms for measuring the impact of housekeeping practices on sustainability, and providing feedback on performance, can limit improvement efforts.

Message– Addressing these challenges through targeted strategies such as enhancing training programs, developing clear sustainability policies, allocating resources effectively, prioritising sustainability in operational planning, and establishing robust measurement and feedback systems could help improve scores in the ‘Housekeeping’ category.

To boost scores in ‘Housekeeping’, organisations should enhance employee education, develop robust sustainability policies, optimize resource allocation, prioritise sustainability in operations, and establish metrics to track performance and provide feedback for continuous progress. Targeted efforts in these areas can help strengthen housekeeping practices.

What are the key factors contributing to the variability in scores across the categories of Process, Environments, Housekeeping, and Measurements?

• Organisational Priorities and Focus: Organisations may prioritise certain aspects of sustainability over others based on their specific business goals, industry standards, and regulatory requirements. This can lead to higher scores in some categories where there is a stronger focus and lower scores in others that are less prioritised.
• Resource Allocation: The amount of resources (time, budget, personnel) allocated to sustainability initiatives can significantly impact the scores. Organisations with more resources dedicated to sustainability efforts in software testing are likely to score higher across the categories.
• Maturity of Sustainability Practices: The maturity level of an organisation’s sustainability practices plays a crucial role. Organisations with well-established, mature practices are likely to have less variability in their scores, indicating consistent integration of sustainability across all categories.
• Awareness and Training: The level of awareness and training regarding sustainability among staff can influence the scores. Organisations that invest in educating their employees about sustainability practices and their importance are likely to perform better.
• Measurement and Monitoring Systems: The presence and effectiveness of systems to measure and monitor sustainability practices can affect the scores. Organisations with robust measurement and monitoring systems are more likely to have consistent and higher scores across categories.
• Innovation and Continuous Improvement: Organisations that actively seek innovative ways to integrate sustainability into their software testing practices and focus on continuous improvement are likely to have higher and more consistent scores.

These factors contribute to the variability in scores, highlighting the complexity of integrating sustainability into software testing practices and the need for a holistic approach to improve performance across all categories.

Message – Variability in sustainability scores across Process, Environments, Housekeeping, and Measurements is influenced by organisational priorities, resource allocation, practice maturity, employee awareness, effective measurement systems, and a commitment to innovation and continuous improvement. For better performance, focus on a holistic approach that integrates sustainability uniformly across all aspects of software testing.

Variability in scores indicates a need for tailored strategies to address specific organisational needs at different maturity levels. A one-size-fits-all approach will not be effective.

Sharing insights from high performers can inspire others, as exemplified by the 7% scoring highest. Benchmarking success stories fosters industry-wide learning and improvement.

Other Messages

Based on the insights derived from the dataset analysis here are further 10 key messages that can help other organisations act towards becoming sustainable in software testing:

1. Growing Focus on Sustainability: There’s an increasing awareness and focus among organisations on integrating sustainability into software testing, highlighting its emerging importance in the industry.
2. Automation and Cloud Adoption for Efficiency: Organisations are increasingly automating test cases and adopting cloud-based tools to enhance testing efficiency, reduce manual efforts, and minimise environmental impacts.
3. Early Issue Detection and Continuous Monitoring: Implementing practices like code reviews and continuous monitoring helps catch issues early, minimising rework and promoting sustainability through reduced resource consumption.
4. Need for Formal Processes and Strategic Integration: There’s a critical need for formalising processes for setting sustainability goals, measuring impact, and aligning testing processes with business and sustainability objectives for strategic integration.
5. Insights from the Field: Insights from a survey of 46 professionals underline the current state of sustainability practices in software testing, revealing both advancements and areas needing improvement.
6. Call to Action for Comprehensive Improvement: Immediate action is needed to address gaps in testing processes, including test environments, housekeeping, and measurement capabilities, to effectively meet sustainability goals.
7. Educating and Training Teams on Sustainability: It’s essential to educate and train software testing teams on sustainability practices to cultivate a culture of environmental responsibility.
8. Adoption of Sustainable Technologies and Methodologies: There’s an opportunity to explore and integrate sustainable technologies and methodologies throughout the software development lifecycle to enhance sustainability.
9. Measuring Impact and Sharing Best Practices: Developing metrics and KPIs to measure the environmental impact of testing activities is crucial, as is sharing best practices within and across organisations to foster collective progress towards greener software testing.
10. Commitment to Continuous Improvement: Organisations must commit to a mindset of continuous improvement, regularly reviewing and enhancing sustainability practices in software testing to achieve long-term sustainability goals.

These messages underscore the importance of integrating sustainability into software testing, offering unique insights and actionable steps across the spectrum—from raising awareness and adopting technology to formalising processes and pursuing continuous improvement—for organisations to enhance their environmental impact and testing efficiency.