The scope of the TMMi model

The intention of TMMi is to cover all testing activities (including test process improvement) for software engineering and systems engineering. While software engineering relates specifically to the creation of software systems, systems engineering does not necessarily include software – but TMMi will still be relevant and useful here.

The TMMi Model is generic enough to apply to all delivery life cycles, all types of industry (i.e. domains) – such as finance, insurance, retail, government and utilities – and any kind of organisational structure. It is also detailed enough to cover:

• All test levels – including static testing – from lower levels of unit or component test all the way through to the higher levels, such as system integration, user acceptance and operational acceptance testing.
  
• All test types – including functional and non-functional
  
• All the associated work that needs to be done to support testing – in other words, ‘structured testing’ – including life cycle, test techniques, test organisation and infrastructure.

TMMi and CMMI

We have already seen that TMMi is a complementary model to CMMI. They are of a similar structure, use much the same vocabularies for similar definitions and some goals are the same.

Because of this, TMMi can be used at the same time as the staged version of CMMI when conducting assessments on both the testing process (TMMi) and the development process (CMMI). When training is needed in both models, this may be easily conducted in parallel (and take less time overall) as the models are so similar at a high level.

Some CMMI process areas support TMMi at the corresponding or higher Level. In some limited cases there is a relationship from a TMMi Level to a higher CMMI level. In general though, when TMMi Process Areas and Practices are elaborated upon in CMMI, TMMi does not repeat or copy the elaboration, but merely makes reference to CMMI for the purpose.

An example of this is configuration management, which is applicable to all TMMi Process Areas, through the Generic Practices. The details of configuration management processes are not duplicated or explained in detail in TMMi, but the CMMI process area is referenced.

TMMi and the approach to test process improvement

TMMi can be used as a reference model when planning or carrying out test process improvement. It is not, though, in of itself an approach for test process improvement. There are several approaches that can be employed, including Six Sigma, Total Quality Management (TQM) and others, generally based on the Deming cycle of plan – do – check- act. However, the Software Engineering Institute developed the IDEAL model for use with CMMI, which has also been shown to be useful when implementing TMMi. We cover the IDEAL model more fully in a later lesson https://www.experimentus.com/lessons/the-ideal-model/.

TMMi in the context of assessments

Different organisations use TMMi in different ways, depending on what they are trying to achieve. Some just want to find out how their testing process compares with other organisations in the industry as a whole, or in their sector / domain. Others may need to prove that they are operating to a certain standard, either internally (to higher management) or externally to their customers or other organisations with which they collaborate. Still others just want to obtain a certificate.

No matter what the ultimate reason for an assessment, one of the main outputs from the exercise is a list of opportunities for improvement. TMMi, being so detailed, is a very good model to use for this purpose. For example, by assessing an organisation at the level of the TMMi Sub-practices, the outcome of an assessment can provide low-level feedback on what aspects of the test process could be introduced, enhanced, refined or upgraded.

Note that TMMi is not, in itself, an assessment method. An assessment method includes all the surrounding and supporting processes and procedures for conducting an assessment. These requirements are found in the TAMAR document, not in the TMMi framework. Apart from anything else, TAMAR helps to ensure that all assessments are conducted consistently, resulting in comparable results between organisations even if they have been assessed by different assessment providers using different accredited methods.

When using TMMi for the purposes of assessment, the model elements must be interpreted as necessary, taking into account many factors, such as:.

• The context of the organisational unit being assessed
  
  
• The delivery life cycle used
  
  
• The strategic objectives of the organisation (e.g. quality, time to market)?
  
  
• The size and structure of the organisation

All information gathered should be judged appropriately taking into account the organisational context, At the end of the day, test processes should be fit for purpose, be usable, and should produce positive results.

Assessment teams should not evaluate a test organisation based on what they would do themselves, but against what is available within the industry as a whole.

The assessment team must interpret the requirements of the model appropriately. to ensure that, while the precise way an activity is undertaken may vary widely, the activity is present and is appropriate to the needs of both the model and of the organisation.

As an example, let’s take a test plan.

• In a V-model delivery life cycle, the assessment team would look for extensive test plans (maybe based on all aspects of IEEE 829) to be defined early in the life cycle, reviewed extensively and approved before test specification, design and execution starts.
  
  
• For an Agile delivery life cycle methodology, the test plans may be much shorter or held in a tool, undertaken within a sprint, and not necessarily reviewed formally or widely. More importantly, test plans for the team may be defined in parallel with test specification and design.
  

Both would achieve the model requirements once the context is understood. In both examples the test plan is appropriate, fit for purpose and achieves the desired result.

The main point to understand is that the model is not a prescriptive “one size fits all” solution. It contains a lot of words. Don’t just read the words – understand and interpret what is being asked for.

For the philosophy behind “The Finger Pointing to the Moon” see: http://www.satrakshita.com/the_finger_and_the_moon.htm . Basically, look beyond the words to what the underlying meaning is.

For this section of the course (TMMi Maturity Levels) there are two learning objectives:

• LO 3.1 [K2] Summarise the TMMi Maturity Levels and Process Areas
• LO 3.2 [K2] Explain the TMMi Maturity Levels

In a classroom-based situation, the syllabus for this course expects this section of the training to take approximately 60 minutes.

The TMMi Foundation accredits assessors and lead assessors in conjunction with TMMi assessment methods.

Although assessors do not, initially, need to be associated with an accredited assessment method, they can only take part in formal assessments if they are.

An application for assessor accreditation must demonstrate:

• A solid background in testing
  • Demonstrated by being certified at ISTQB Foundation Level, or equivalent
    
    
• A good grounding of process improvement training
  • Demonstrated by passing the TMMi Professional exam and
  • Having received TMMi Assessor or equivalent training
    
    
• Experience in running assessments
  • For new applicants, no previous participation in an assessment is required
  • For existing assessors re-applying for accreditation, a minimum of 50 hours participation on formal or informal assessments in the 36 months prior to application

The Foundation will also accredit lead assessors who must demonstrate:

• A solid background in testing
  • proved by being certified at ISTQB Advanced Level (Test Manager), or an equivalent test management certification
    
    
• A solid background in Process Improvement
  • Having received TMMi Assessor or equivalent training
  • Having received TMMi Lead Assessor or equivalent training
    
    
• Experience in running assessments
  • For new applicants, a minimum of 75 hours participation on at least one informal or formal assessment in the previous 12 months is required
  • For those applying for re-accreditation, a minimum of 225 hours participation on formal or informal assessments in the previous 36 months is necessary
  • Evidence of participation in all assessment activities in accordance with the assessment team leader responsibilities as defined in TAMAR

To be able to publicly certify, an organisation’s assessor resources (including lead assessors):

• must be independently certified by the Foundation as having sufficient training, skills and experience to undertake assessments
  
  
• can only be certified against an assessment method which has been accredited by the Foundation and
  
  
• must operate in accordance with TAMAR and the Data Submission Requirements document (DSR)

The full assessor accreditation rules and criteria can be found on the TMMi website. The current version is 3.0a, published in July 2022.

The TMMi Foundation sets the requirements for TMMi assessment methods through TAMAR (TMMi Assessment Method Application Requirements).

The TMMi Foundation assesses and approves assessment methods against TAMAR.

A TMMi assessment method must be approved by the TMMi Foundation in order for it to be able to be used to provide TMMi Level certification.

An assessment method must abide by the following rules:

• No deviation is allowed from the TMMi Reference Model which must be used as published In order to provide TMMi certification
  
  
• No deviation or changes are allowed to the accredited assessment method
  
  
• The TMMi Foundation reserves the right to audit any certification assessment

The standard reference model is in the public domain, which means that anyone can use it as a basis for driving process improvement change. However if an assessing organisation wants to be publicly accredited, then it must abide by the set of requirements for assessment methods. These are contained in two documents:

1. The TAMAR (TMMi Assessment Method Application Requirements)
   • This document stipulates exactly how an assessment should be conducted
     
     
2. The Assessment DSR (Data Submission Requirements) document
   • This document stipulates what data is required to be collected and sent to the Foundation for them to validate formal assessments
   • It also stipulates what data is required to validate assessor performance

Organisations can submit their assessment method and associated tools for accreditation. The Foundation will decide if it complies with the TAMAR and DSR requirements.

The benefits of obtaining TMMi Foundation accreditation are that:

• An organisation will have an independent confirmation that their assessment method (and assessments|) are in line with international standards and industry best practice.
  
  
• The organisation would be using an assessment method against a reference model which works and is consistent with broader industry IT practices.
  
  
• The organisation can validly publicise the formal Maturity Level achievement. While non-accredited assessment methods can produce Maturity Level ratings, this has little credence if the assessment method is not accredited and therefore possibly non-compliant. It is better to state “Achieved Maturity Level x” when assessed against a TMMi assessment method accredited by an independent body.
  
  
• As part of the responsibilities towards assessment method accreditation, the TMMi Foundation will provide levels of governance to ensure compliance in practice. This includes surprise audits of assessment providers (including observing actual assessments)

The TAMAR and DSR are covered in greater detail in later lessons.

History and aims of the TMMi Foundation

The TMMi Foundation is a non-profit organisation based in Chester, UK. The board of directors and management executive are from several countries.

The founder members were:

• Andrew Goslin
• Mac Miller
• Fran O’Hara
• Klaus Olsen
• Geoff Thompson
• Erik van Veenendaal and
• Brian Wells

Brian Wells launched the TMMi Foundation at the ICSTest-UK conference on 28 September 2005. The conference dedicated a track to TMM, including two case studies presented by the retailer Marks and Spencer and service provider Improve QS and an afternoon tutorial on TMM. Brian Wells, then chair of the TMMi Foundation, spoke about the history of TMM, its structure and the aims of the Foundation.

Since then, members of many organisations have continued to develop the Foundation.

The current chair is Geoff Thompson and the CEO is Erik van Veenendaal

The aims of the Foundation include:

• To create a definition of an international core TMMi reference model standard
  
  
• To place it in the public domain
  
  
• The creation and management of an independent, unbiased central data repository
  
  
• The provision of industry and other analysis services
  
  
• The provision of an independent accreditation process for TMMi Assessment Methods based on the standard model.

TMMi sources

There were a number of sources used when drafting the TMMi standard reference model. No single source is more important that others and the guiding principle has been to build in the best of currently available research in the public domain and to develop a unique, complete, robust, pragmatic, usable and flexible model.

The independent TMMi Foundation initiative was established with the sole intent of developing the TMMi standard into the independent TMMi framework which is in place today.

The major sources were:

• The original TMM model developed by Ilene Burnstein at the Illinois Institute of Technology in response to behavioural characteristics of immature and mature software organisations. This TMM model was used as the framework on which the TMMi Foundation developed TMMi
  
  
• The Capability Maturity Model Integration (CMMI). The Test Maturity Model Integration (TMMi) has been developed as a staged model in line with CMMI to complement the existing CMMI framework. Capability Maturity Model Integration (CMMI) is developed by the CMMI project, which aimed to improve the usability of maturity models by integrating many different models into one framework
  • The project consisted of members of industry, government and the Carnegie Mellon Software Engineering Institute (SEI)
  • CMMI succeeds the Capability Maturity Model (CMM) or Software CMM
  • The CMM was developed from 1987 until 1997
  • In 2002, CMMI Version 1.1 was released. Version 1.2 followed in August 2006
  • CMMI is now owned by the US Department of Defense
    
    
• Beizer’s Progressive Phases of a Tester’s Mental Model [Beizer, 1990]. This indicates 5 phases to a tester’s mental life:
  • Phase 0 – there is no difference between testing and debugging
  • Phase 1 – The purpose of testing is to show that the software works
  • Phase 2 – The purpose of testing is to show that the software doesn’t work
  • Phase 3 – The purpose of testing is not to prove anything but to reduce the perceived risk of not working to an acceptable value
  • Phase 4 – Testing is not an act. It is a mental discipline that results in low risk software without much testing effort
    
    
• Gelperin and Hetzel’s Evolutionary Testing Model [Gelperin and Hetzel, 1988] – Covered in more detail later in the course
  
  
• The EU-funded Metrics Based -TMM project in 2000 attempted to enhance the TMM model and adapt it to the requirements of the IT test industry https://wwwbruegge.informatik.tu-muenchen.de/lehrstuhl_1/files/teaching/ws0708/ManagementSoftwareTesting/12-4-1-FPdef.pdf
  
  
• International testing standards – e.g. IEEE 829 Standards for Software Testing Documentation
  
  
• The TMMi uses, where appropriate, the ISTQB Standard Glossary of terms used in software testing
  
  
• Terminology may also be drawn from IEEE 829 Standard for software Test Documentation (which includes test terminology), BS 7925 -1 Glossary of Testing Terms and others
  
  
• The TMMi framework has drawn from the best practices and procedures from other assessment methods, in particular TMM, TPI and TOM which have demonstrated elements of good practice proven though experience to avoid re-inventing the wheel
  
  
• In addition, there are a growing number of surveys and research. All have contributed to the TMMi development providing an empirical foundation and required level of practicality

The TMMi Model is constantly developing, drawing on the research and experience of its members. One of the ways in which the TMMi Foundation does this is by co-ordinating a public forum of interested parties to facilitate free interchange of information, education, ideas and usage of the public standard. In order to be useful and usable by the IT test industry, it should, above all else, reflect experience of what is good practice. The model is founded on contributions based on the practicality of testing in the real world.

Introduction

The TMMi Professional syllabus states that students must have an understanding of how CMMI works to be able to understand the links with TMMi.

CMMI was largely guided by Software-CMM, a model widely used in the IT industry

The CMMI model has both a staged and continuous representation model. The differences between them are subtle but significant.

Staged representation

In the staged representation there are prescribed, ordered, stages that an organisation must go through to improve its development process (i.e. Maturity Levels). It is organisational maturity that is the focus of the staged representation. It provides a proven sequence of improvements that is a foundation for the next one and enables organisations to compare Maturity Levels by using a common rating.

Continuous representation

In the continuous representation there are no fixed levels or stages to go through and organisations can select areas for improvement from different categories according to their perceived need. Process Area capability is the focus of the continuous representation. Using the continuous representation allows the

selection of the order of improvement that best meets the organisational goals and addresses specific areas of risk. It also allows organisations to compare at Process Area level.

Selecting a CMMI model

There are several factors to be considered when deciding which one of the representations to choose for implementing CMMI. Some organisations may simply opt for the one with which they are most familiar, but most often the decision will be based on which of the potential benefits are most important to the organisation’s stated goals. Some of the benefits of each representation are listed below.

Benefits of the staged model

• It provides a proven sequence of improvements
  
  
• It starts with basic management practices and progresses by means of a tried-and-tested, pre-defined path through a succession of Levels
  
  
• Each Level provides a foundation for the next one
  
  
• It gives a single rating summarising assessment results
  
  
• It enables comparison across and among organisations by means of the Maturity Levels
  
  
• It also provides an easy migration from Software-CMM (SW-CMM) to CMMI

Benefits of the continuous model

• It allows selection of the order of improvement that best meets the business objectives of the organisation
  
  
• It allows mitigation of specific areas of the organisation’s risk
  
  
• It enables comparisons across and among organisations Process Area by Process Area or by comparing results by equivalent staging

The TMMi model, currently, follows only a staged representation, but in the future there may be the option to use a continuous approach.

Learning objectives

In this section of the course the learning objectives are as follows:

• LO 2.1 [K2] Summarise the structure of the CMMI (v1.3) process improvement model
  
  
• LO 2.2 [K2] Understand the aspects of the CMMI (v2) model with testing-specific relevance
  
  
• LO 2.3 [K2] Compare the suitability of CMMI for test process improvement to the TMMi Model developed specifically for test process improvement
  
  
• LO 2.4 [K2] Compare the continuous and staged representation including their strengths and weaknesses
  
  
• LO 2.5 [K2] Summarise the sources (e.g. TMM) used during the development of the TMMi Model
  
  
• LO 2.6 [K2] Understand the evolution of the testing process as described by Gelperin and Hetzel
  
  
• LO 2.7 [K2] Describe the scope of the TMMi Model

In a classroom-based situation, the syllabus for this course expects this section of the training to take approximately 70 minutes.

Other models are available

The testing community has created a number of test and quality-related reference models, including:

• Test Process Improvement (TPI)
  
  
• Test Organisation Maturity Model (TOM)
  
  
• Test Maturity Model (TMM)

The Test Process Improvement (TPI) model was developed based on Sogeti’s knowledge and experience of working with their clients and is one of the industry recognised standard test process improvement models. TPI is an example of a model which focuses on key Process Areas:

• Test strategy
• Life cycle model
• Moment of involvement
• Estimating and planning
• Test specification techniques
• Static test techniques
• Metrics
• Test automation
• Test environment
• Office environment
• Commitment and motivation
• Test functions and training
• Scope of methodology
• Communication
• Reporting
• Defect management
• Test-ware management
• Test process management
• Evaluation
• Low-level testing

Other models, such as TOM, are primarily focussed on the organisation rather than the Specific Practices that go to make up organisational processes.

TMM was developed by the Illinois Institute of Technology and was based on much historical research and references in the testing arena. While structured along the lines of CMM, it was lacking in flexibility and was incomplete.

Different models have different approaches. One of the benefits of TMMi is that it is highly structured and repeatable. This is one of the reasons Experimentus chose TMMi as its preferred quality model.

There a number of other possible reasons why other models have not seen good uptake within the industry, including:

• The focus of the model does not address the perceived problems for the organisation
  
  
• The models have limited focus and do not adequately address the needs of the industry either for the present or the future
  
  
• The benefits of using the model are not clear
  
  
• The model is academically driven and does not reflect real life
  
  
• The model is driven by commercial goals which are not aligned with the purposes of process improvement

Benefits achieved

TMMi Foundation World-wide survey

The TMMi Foundation World-wide survey 2020-2021 found that actual examples of benefits achieved through using TMMi included:

• Test estimation being 60% more accurate and up to an accuracy of 95% overall
  
• Test predictability increased by 70%
  
• Defect detection percentage improvement of 20% – 30%
  
• Defect leakage to production halved
  
• Greater product risk coverage
  
• Test execution lead times reduced from 19 weeks to 5 weeks
  
• Time to market reduced by 7%
  
• Test efficiency increase of 25%
  
• Test productivity improved by 25%
  
• Regression test cycle length reduced by 87%

Experimentus case studies

As has been said, there is little published information about costs and benefits from assessments, but Experimentus does have some figures based on case studies of assessments we have conducted. These include:

• 40% savings were made by a UK Government testing department after changes were introduced to ensure compliance for a Level 3 certification
  
  
• A UK bank saved 8% of their entire IT budget and improved development output by 12%
  
  
• An insurer saved £440,000 on a £2m project
  
  
• A retailer continues to save at least 12% on every project after improvements were introduced
  
  
• A test services company increased their efficiency by 8%
  
  
• A retailer reversed their decision to outsource, when recommendations (including an ROI) demonstrated to them that it would be a lot cheaper if they just improved their internal processes

These are not just apocryphal stories, but figures based on actual case studies. There are some more case studies on the Experimentus web-site if you want to see some more detail.

Case study – retail client

One of Experimentus’ clients – a retailer – saw the following benefits as a direct result of using TMMi to help in their improvement program:

• They have delivered more projects on time and to budget
  
  
• They have reduced overall fault levels by 40%
  
  
• There has been a reduction in development and test costs by up to 30%
  
  
• The total production technical support costs have been reduced
  
  
• The TMMi implementation “paid” for itself within 12 months and the company is seeing real value added to their bottom line.

Typical benefits

As with costs, benefits can be broken down into direct and indirect types. Sometimes the indirect benefits are larger, or more important, than the direct ones, so they should not be ignored. It is well worth taking into account the indirect benefits when the overall value of a TMMi improvement program is being calculated.

Because long-term investments can be allocated to improvement programs, it is also critical that the return on investment (ROI) of an improvement program is calculated accurately and consistently.

Direct benefits

Direct benefits include:

• A rise in productivity
  
  
• Early defect detection resulting in a reduced need for live fixing

Indirect benefits

Indirect benefits include:

• Improved staff experiences leading to increased morale and motivation and as a result, better quality work
  
  
• Increased customer loyalty
  
  
• Allows greater movement of employees within projects
  
  
• Improved working environments
  
  
• Recognition of testing as a profession
  
  
• Integration of testing into the development process, irrespective of the methodology being followed (for example following a traditional model like v- model or where agile is used).
  
  
• TMMi will help guide the introduction and use of a structured and controlled set of test processes.

Return on investment

Return on investment (ROI) is calculated by direct costs and benefits, since it is easier to quantify direct costs than indirect ones.

General benefits

While there are specific benefits, there are also general ones to consider that can be just as important and valuable, so they should not be ignored. General benefits include:

• Organisations are enabled to be more effective and efficient
  
  
• Product quality can be improved
  
  
• Increased testing productivity – potentially leading to shorter delivery time frames
  
  
• TMMi suits any development life cycle model
  
  
• There is a shift-left paradigm, moving from defect detection to defect prevention

This is not a full list, but just to set a scene for the kinds of improvements that may be expected.

As yet, there is little published information about costs and benefits from assessments. In fact, most organisations will want to keep that information confidential, particularly those that are not performing well.

Typical effort for assessments

It is impossible to arrive at a price for an assessment that will apply to all organisations and circumstances. There are many factors that must be taken into consideration when calculating the effort needed for an assessment, such as:

• The costs of the assessing organisation – i.e. their charge rate
  
  
• The size and scope of the assessment – is it an organisation, a sector of the business, a program or a single project?
  
  
• How many projects typically are currently running or have been conducted in the previous 12 months? TMMi assessments have to assess a proportion of live projects (covered later in how to do assessments).
  
  
• How many interviews need to be conducted (again, covered in how to do assessments)
  
  
• Locations where interviews have to be conducted
  
  
• Whether it is a formal or informal assessment

Very often, only direct costs and benefits are considered but indirect benefits can be larger than direct ones

Typical costs for assessments

It is necessary to spend money to achieve benefits. Any improvement program will involve costs. The aim is to save time and money, but, also to improve the quality of the product. This takes effort and needs to be understood and accepted.

Costs fall into two main categories – direct and indirect.

Direct costs

Direct costs include such things as:

• Money – to pay for extra work or physical resources
  
  
• Work hours – either extra hours (overtime) or normal work hours diverted to improvement activities
  
  
• Training and education in new or updated processes – including updating training documentation
  
  
• External consultancy – to help with planning and implementing process improvement

Indirect costs

Indirect costs are not quite so easy to quantify for a particular improvement program, but they will include such items as:

• Time spent in training
  
  
• Productivity lost due to opposition to process change

Aspects of testing that can be improved

Now we know why it is important to carry out test process improvement, what are the features of testing that can be made better?

There are many of aspects of testing that can be improved as part of a TMMi assessment. These include (but certainly not limited to):

• Documentation of any type, including strategies, plans, test case definitions, test scripts and reports
  
  
• All types of reviews – informal, formal, walkthroughs and inspections
  
  
• Monitoring and control processes
  
  
• The application of test design techniques – both functional and non-functional
  
  
• Definition and management of test environments
  
  
• Organisational test measurement definition, collection and reporting
  
  
• And root cause analysis as part of defect prevention

All aspects of testing are covered, to a greater or lesser extent, as part of assessments up to Maturity Level 3. At Levels 4 and 5, more formality and depth is added.

Typical reasons for conducting test process improvement

There are several reasons why test process improvement is needed. For a start, many organisations have tough business objectives, including:

• Reducing the time it takes to get software products from the drawing board to the customer and
  
  
• Minimising production costs in order to maintain profit margins while competing in a market where reduced prices are being paid, particularly at the on-line checkout
  
  

Apart from these direct business objectives, other factors make test process improvement essential, including that:

• There is an expectation from customers of more functionality, better performance and increased reliability
  
  
• And this from the number of end-users that is increasing quickly – so there is a larger crowd liable to uncover problems in actual use
  
  
• As a consequence of adding increased functionality and complexity to software and hardware, the chance of introducing defects is ever greater
  
  
• Testing needs to keep up with the new methods, techniques and tools that are becoming available to support development and maintenance
  
  
• And many applications are co-developed, or use completely outsourced development organisations, making both the development process and testing process more complicated and difficult to manage

TMMi World-wide survey 2021

The 2020-2021 TMMi World-wide user survey cited enhancing software quality, increasing testing productivity, achieving TMMi certification and reducing product risk as the top four reasons that organisations started a TMMi-based test improvement project, as shown in the graph below.

Main reasons for adopting TMMi for test process improvement 2020-2021

The benefits that organisations are trying to achieve do vary, based on their business objectives.

It is interesting to note that about a third of TMMi adopters are not Interested, at least primarily, in becoming TMMi certified, but that most typically look for the other benefits that test process improvement can bring.

The result of all this is…

For all these reasons, it is becoming ever-more challenging to create and build applications and testing needs to be proportionately more effective and efficient. Test process improvement activities help this to be achieved.

It is an increasingly competitive world, and organisations need to make money and grow in a tough climate. It is certain that there are many more reasons for conducting test process improvement, but it is clear there is a demand to improve.

Systems and applications are playing an ever-increasing part of every-day life (both personally and in business). There is a greater need for development to focus on quality aspects. Customers and users are becoming more demanding – software failures are much more visible now and can have major impact on day-to-day activities – economically, socially and internationally.

Testing is becoming (even more of) a key activity for helping to improve both product quality and the performance of the development process.

Industry is undergoing a fast pace of change!

What is the purpose of TMMi?

We have seen that the TMMi Model encompasses all aspects of test and test-related quality activities within the software development life cycle and the integration between development processes and test processes.

TMMi is designed to be applicable, in its standard form, to all circumstances. It is intended to be detailed enough to cover all test activities but generic enough to apply to all delivery life cycles (including waterfall, v-model, iterative and all flavours of Agile).  

In assessing organisational test processes against the model, it should be effective at being able to identify (and share within the organisation) strengths, weaknesses, and best practices, enabling the opportunity to improve test effectiveness while at the same time promoting the integration of test processes.

Part of the assessment exercise is to facilitate the identification and management of risk in the test-related quality processes and to promote organisational risk management.

Assessment using the TMMi Model should not just identify the strengths and weaknesses of the organisation’s test processes but should also be useful as an input to structuring a realistic plan of process improvements based on the maturity level required by the organisation.

By identifying weaknesses against the TMMi Model, it is a relatively easy exercise to structure required improvements (and to prioritise them based on current impact and strategic requirements and benefits).

By adopting a standard, public domain reference model to evaluate the capability and maturity of an organisation, it is possible to compare against similar organisations and across industry sectors.

With an assessment method and assessor accreditation available through the TMMi Foundation, there is independent validation and governance of the assessment results to ensure that they are fully comparable.

In summary, the purposes of TMMi are:

• To develop and provide a standard reference model applicable for all industries which will
  
  
  • aid in identifying testing strengths and best practices
    
    
  • help improve integrated test effectiveness and efficiency
    
    
  • enable identification of issues and risks in an organisation and  
    
    
  • enable creation of a programme of test process improvements
    
    

Other purposes are:

• To provide a detailed model for test process improvement that is positioned as complementary to CMMI
  
  
• To use Maturity Levels for process evaluation and improvement
  
  
• for the TMMi Foundation to provide public governance, measurement and accreditation

A standard test reference model

A framework for test process improvement must be able to encompass all the activities related to testing that we saw in the previous lesson.  By covering them in detail, the model becomes a guideline and reference framework for test process improvement.

With this in mind, it was clear there was nothing in the public domain that:

• Was flexible enough to cater for all the existing development methods,  such as the waterfall model or Agile.
  
  
• Was able to cater for future development needs without major re-work
  
  
• Could add value and support common standards such as BS7925 and ISO 9000
  
  
• Was able to be used in stand-alone mode but would also seamlessly support software engineering models such as CMMI and ITIL

So a number of leading thinkers in the IT testing industry identified a fast-emerging strategic requirement to provide a standard reference model for the industry as a whole and to place it in the public domain. The TMMi Foundation identified the need for a public-domain standard test reference model that:

• Incorporates good elements from existing models
  
  
• Addresses current needs and trends
  
  
• Draws on professional test expertise
  
  
• Covers all test-related quality activities
  
  
• Applies to all software development life cycles
  
  
• Is usable by the international community
  
  
• Provides a common standard
  
  
• Supports industry leading software engineering capability models
  
  
• and is a process that in its broadest sense would encompass all software product quality-related activities

As a result, the TMMi Model was designed to:

• Utilise appropriate elements of existing models
  
  
• Draw on existing expertise within the industry and incorporate best practices
  
  
• Cover all aspects of test and test-related quality activities
  
  
• Be generic enough to be applied to all software development life-cycles
  
  
• Be easily understood and usable globally
  
  
• Provide a common standard to assess in detail the capability and maturity of an organisation
  
  
• Support industry leading software engineering models

TMMi is complementary to CMMI. CMMI covers testing and quality only lightly, whereas TMMi covers quality in great detail and supports other models such ITIL, TPI and ISO 9000.

We need to understand why there is a need for a standard reference model to evaluate the capabilities of test processes.

Since the 1970s, there have been continuing efforts to measure and compare overall software engineering processes which have included CMM and CMMI. This has happened because, as IT expanded, it became more complex and common, both in the business context and in our everyday lives. Thus there was a need to make it more efficient and effective.

Since the 1990s, testing as a discipline and profession has been evolving and maturing. It is now considered a necessary quality element for any IT organisation.  In parallel, the test industry needs to be able to measure how efficient and effective it is to help justify the work it performs (as if more justification is needed). The test reference model is used as a predefined structure to help verify the test processes.

We know there are substantial efforts being made to improve the quality of products. Technology is always moving forward in its ability to handle more lines of code.  New technologies mean that there are new areas for development which are not as easily understood as the existing ones. As technologies become more diverse, more lines of code are created and the more complex the test process becomes. As a result, the harder it becomes to maintain quality. Look at how the industry has changed over the last 5, 10 and 15 years and the increased demands that have been placed on testing.

Despite various quality improvement approaches, the software industry is still seen as far from as effective and efficient as it could be. Why is this?

Software testing has not traditionally been taken as seriously as the development process. CMMI and other development methods such as ITIL have been around for more than 30 years, but how much more efficient today are development and test teams? CMMI is widely represented by organisations who claim to have robust processes and procedures but does that necessarily mean that their processes are also efficient and effective?

Given that testing accounts for a large proportion of a project’s budget (testing accounts for between 30% and 50% of total project costs) there is a need to use the resources wisely. Testing as a proportion is growing now. Think how long testing used to be in proportion to development in mainframe applications. It was historically about 10%, because it took a much longer time to develop an application in relation to the time spent in testing it.

But modern applications can be developed much more quickly and are much more customer-facing. So there is a greater need to carry out extensive testing around such aspects as performance, usability and security. As a result the proportion of testing is higher. With modern web-based and safety systems testing is nearer 80% of the total project cost. Development time can be dramatically reduced due to the technologies used, but testing is actually a larger aspect now.

Introduction

As with all successful projects, there is primarily a need to understand the requirements. It is always helpful to go back to the start of what we are trying to do.

Any test process standard reference model must ensure it embraces the entire test life cycle, from project inception all the way through to production support encompassing the entire lifetime of the product. To achieve this, any model needs to be broad in its subject matter to ensure it covers – and enables evaluation of – all test and test-related quality activities. It also needs to do this comprehensively and and in a structured way.

It starts with a definition of testing

Testing is defined as:

“The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.”

[ISTQB Standard Glossary of Terms]

Testing covers many activities which stretch, integrate closely with (and are often dependent on) software development activities such as:

And testing types such as:

Any framework for test process improvement must be able to encompass all these activities.

For this section of the course (the context of test process improvement) there are three learning objectives:

• LO 1.1 [K2] Provide examples of the typical business reasons for test process improvement.
  
  
• LO 1.2 [K2] Understand the different aspects of testing that can be improved.
  
  
• LO 1.3 [K2] Summarise the typical costs and benefits of the TMMi.
  
  

In a classroom-based situation, the syllabus for this course expects this section of the training to take approximately 60 minutes.

Overview

The exam consists of 40 multiple choice questions. Each question is worth 1 point. To pass, you must obtain at least 65% correct answers (26 or more points).

The exam lasts for 60 minutes. If English is not your native language, 75 minutes is allowed (but you need to request this in advance).

The distribution of subjects will be as follows:

• Context of test process improvement – 3 questions
• Introduction to TMMi Model – 4 questions
• TMMi Maturity Levels – 3 questions
• Structure of the TMMi – 4 questions
• TMMi Model – 14 questions
• TMMi in an Agile context – 4 questions
• TMMi assessments – 4 questions
• Implementing TMMi – 4 questions

From this distribution you can see the model itself is by far the most important area on which to focus. 

If you are attending a course run by a TMMi-recognised training provider, you may be able to take the exam on-site on the last day of the course. In this case there will be an invigilator present. If you are self-studying, you can arrange to take the exam independently – and on-line if you wish via PearsonVue or ISQI. Details can be found on the TMMi website. or https://isqi.org/en/.