• TMMi does not offer a standard approach to a change program in an organisation.
  
• There are many approaches available, such as Edward Deming’s ‘Plan – Do – Check – Act’ cycle and the IMPROVE method.
  
• CMMI uses the IDEAL model, which shows what needs to be done when implementing TMMi improvements and comprises 5 key phases:
  • Initiating – Establish the infrastructure for improvement to promote a a successful process
    
    
  • Diagnosing – Determine the organisation’s current state and what it wants to achieve
    
    
  • Establishing – Plan and specify how the chosen solution will be established
    
    
  • Acting – Execute the plan
    
    
  • Learning – Learn by experience and improve the skills needed to implement change
    
    

The material in this lesson is no longer examined and has been included just for your information. If you do not wish to spend time on something that is no longer part of the official syllabus you can safely set this lesson to ‘complete’. 

Once improvements have been implemented, shown to be viable, and working, it is important to get them into the organisational plan so they can be managed in concert with other work.

There should be regular feedback to the sponsor at appropriate intervals (say, weekly or fortnightly) with progress reports in the standard organisational format (for example, RAG statuses and cost / effort used).

All stakeholders should continue to be informed of milestone achievement and provided with updates regarding the new implementations, so that everyone understands the process.

Some provision of support and guidance should be made available to help in answering user questions and for dealing with issues. It very important that, when problems are reported, they are addressed, and fixed quickly, to maintain confidence in the new processes.

After a suitable time (for example, six months after implementation) the new processes should be re-evaluated, to confirm they remain working and the expected benefits are still being delivered.

The material in this lesson is no longer examined and has been included just for your information. If you do not wish to spend time on something that is no longer part of the official syllabus you can safely set this lesson to ‘complete’. 

The curve of the Gartner hype cycle is used in many areas of new process or new technology uptake, not just for the IDEAL model.

It shows that, initially, there is enthusiasm for change, which peaks before moving to a downward trend as new processes bed-in and people realise they need to change.

If the decision is taken that people do not accept it, and revert to their old ways, then the improvement is finished.

If, however, stakeholders keep working at it (and, if need be, slightly amend processes so they fit) then practical benefits will eventually be realised.

The material in this lesson is no longer examined and has been included just for your information. If you do not wish to spend time on something that is no longer part of the official syllabus you can safely set this lesson to ‘complete’. 

There are numerous ways of working out the potential, and actual, financial benefits of implementing TMMi. Here we show one way, based on the method Experimentus uses.

Below is an example of a spreadsheet used to help calculate and document the costs and benefits for a project improvement plan (PIP) for a real client:

• For each area of improvement, we estimate how many person-days we believe it will take to make the improvement.
  
  
• We then agree with the client what the estimated average cost of a person day should be
  • This will include both the client’s resource costs and Experimentus’ resource costs – dependent on our level of involvement.
    
    
• These are then used to calculate the estimated cost of implementation. Here we have estimated £1,000 per day.
  
  
• We then calculate the anticipated benefit, and, by subtracting the cost, we can estimate the net potential benefit for each of the activities.

In the table above, for PIP ID 5, you can see that we have an improvement with a potential benefit of zero against it. Often we find that there are some improvements for which there are no financial benefit. The benefit they do provide is that they are foundation stones for the other changes. If these are not implemented, the benefits of the other changes would not be delivered.

These items still need to be in the costings, and as long as the overall calculation still demonstrates a sufficient benefit, there is not an issue. The advantage is that by including these zero-benefit items with the positive-amount benefits, the overall cost of the implementation is fully understood.

The above is an example of a potential benefits summary where there is positive value-add to the business.

We have included a confidence factor (in other words, how confident we are that the benefits will be realised) which can be adjusted to each situation. The implementation factor (how much could implementation costs vary from the amount originally estimated) can be positive or negative.

For this example, from experience with this particular client, we have estimated an extra 50% could be needed. The final net potential benefit is calculated by subtracting the adjusted cost from the adjusted benefit.

For this section of the course (Implementing TMMi) there are eight learning objectives:

• LO 8.1 [K2] Summarise the activities of the initiating phase of the improvement framework
  
  
• LO 8.2 [K2] Summarise the key elements of a test policy
  
  
• LO 8.3 [K2] Summarise the activities of the diagnosing phase of the improvement framework
  
  
• LO 8.4 [K2] Summarise the activities of the establishing phase of the improvement framework
  
  
• LO 8.5 [K2] Summarise the activities of the acting phase of the improvement framework
  
  
• LO 8.6 [K2] Summarise the activities of the learning phase of the improvement framework
  
  
• In a classroom-based situation, the syllabus expects this section of the training to take approximately 70 minutes.

The TMMi Reference Model is intended to be generic enough to apply to all test life cycle activities and delivery models but detailed enough to cover all anticipated activities and deliverables. However, the model needs to be interpreted by the assessment team depending on the organisational context

Assessments uncover the maturity of the test process and will determine if the organisation has met a specific level of maturity. The result of the assessment will help prescribe what improvements are required and an action plan for the improvements can be defined.

Assessment Criteria Quick Comparison

Informal assessments can be led by an experienced assessor (who need not be accredited by the TMMi Foundation)

A team can consist of just one person if necessary

Only one type of evidence is required  – in other words, no corroboration of evidence is required

The informal assessment does not lead to a formal TMMi rating or certification, but is used for a relatively quick and inexpensive understanding of the maturity of a test process and what the areas for improvement are.

A formal assessment must be led by a TMMi accredited lead assessor.

The assessment team must consist of at least two individuals – the lead assessor and at least one other TMMi Foundation Accredited assessor

The evidence gathered must include interviews with relevant staff and an examination of documentary artefacts, all of which must be corroborated

The formal assessment leads to a rating of the maturity of the organisation in terms of the TMMi Level achieved, and, if appropriate, the organisation can be certified.

A formal assessment also allows testing strengths and weaknesses to be identified in detail, providing a full analysis of the gaps that must be filled before an organisation can be said to be operating at a specific level of Maturity against the TMMi Model.

Rating Comparison

The table below shows the relationship between the ordinal scale and nominal ratings and to which process components they are applicable

Remember that Sub-practices are not required to be scored, since they are informative components.

There may be many cases where scoring is not clear-cut. Rating Goals is not a simple mathematical exercise, but all circumstances have to be taken into consideration to arrive at a fair score that truly represents the position.

When rating Process Areas, the element of consensus within the assessment team is no longer applied. There is a strict rule for identifying the score, which is that the Process Area is only as strong as the weakest link. The weakest Specific or Generic Goal becomes the highest score for the Process Area.  

Scoring individual Maturity Levels is relatively simple – the Maturity Level is based on the weakest rating of the relevant Process Areas.

To achieve a Maturity level, both of the following must apply:

All Process Areas at the particular Maturity Level must be scored as Fully Achieved or Largely Achieved.

Assessment process

The diagram below illustrates a typical high-level structure of an assessment process. The main documentary outputs are shown on the right of the related processes.

For this section of the course (TMMi assessments) there are three learning objectives:

• LO 7.1 [K2] Explain the role of assessments in the overall improvement process
  
  
• LO 7.2 [K2] Compare informal assessments to formal assessment
  
  
• LO 7.3 [K2] Summarise the generic assessment process

In a classroom-based situation, the syllabus for this course expects this section of the training to take approximately 60 minutes, but I recommend at least 90 minutes of study

• Testing covers many activities which stretch and integrate closely with (and are often dependent on) software development activities
  
  
• A framework for test process improvement must be able to encompass all the activities related to testing
  
  
• The purposes of TMMi are to:
  • Develop and provide a standard reference model applicable for all industries
    
    
  • Identify testing strengths and best practices
    
    
  • Improve integrated test effectiveness and efficiency
    
    
  • Identify issues and risks in an organisation
    
    
  • Identify a programme of test process improvements
    
    
  • Provide public governance, measurement and accreditation
    
    
  • Provide a detailed model for test process improvement that is positioned as complementary to CMMI
    
    
  • Use Maturity Levels for process evaluation and improvement
    
    
• Typical reasons for conducting test process improvement
  • A need for a reduced time to market
    
    
  • Software and hardware is tending towards Increased functionality and complexity
    
    
  • An expectation of Increased reliability
    
    
  • An expectation of ever-better performance
    
    
  • The number of users is increasing quickly
    
    
  • Products and applications need to be cheaper to produce to maintain profit margins
    
    
  • New methods, techniques and tools are becoming available to to support development and maintenance
    
    
  • Many applications are co-developed or use completely outsourced development methods
    
    
• Aspects of testing that can be improved
  • Documentation
    
    
  • All types of reviews
    
    
  • Monitoring and control
    
    
  • Test design techniques – both functional and non-functional
    
    
  • Managing test environments
    
    
  • Organisational test measurement
    
    
  • Root cause analysis
    
    
• Typical costs for assessments
  • Direct costs
    • Money – to pay for extra work or physical resources
    • Work hours
    • Training and education costs
    • External consultancy
      
  • Indirect costs
    • Time spent in training
    • Productivity lost due to opposition to process change
      
      
• Typical benefits
  • Direct benefits
    • A rise in productivity
    • Early defect detection
      
      
  • Indirect benefits
    • Improved staff experiences
    • Increased customer loyalty
    • Allows greater movement of employees within projects
    • Improved working environments
    • Recognition of testing as a profession
    • Integration of testing into the development process
      
      
• Return on investment (ROI) is calculated using direct costs and benefits
  
  
• Other test reference models are available, including:
  • Test Process Improvement (TPI)
    
    
  • Test Organisation Maturity Model (TOM)
    
    
  • Test Maturity Model (TMM)

• The structure of CMMI
  • Staged representation
    • Provides a proven sequence of improvements.
    • Starts with basic management practices and progresses through a proven, pre-defined path of a succession of Levels
    • Each Level provides a foundation for the next one
    • Gives a single rating summarising assessment results
    • Enables comparison across and among organisations by means of Maturity Levels
    • Provides an easy migration from SW-CMM to CMMI
  • Continuous representation
    • Allows selection of the order of improvement that best meets the business objectives of the organisation
    • Allows mitigation of specific areas of the organisation’s risk
    • Enables comparisons across and among organisations Process Area by Process Area or by comparing results by equivalent staging
• TMMi currently follows only a staged representation

Below is a diagram showing the structure of the staged CMMI model

Compare the staged representation diagram with the continuous representation diagram below:

The table below shows the links between the staged and continuous representation.

• TMMi Levels and the equivalent CMMI Levels

• CMMI supports TMMi
•  The sources for TMMi include:
  • TMM
  • CMMI
  • Beizer’s Progressive Phases of a Tester’s Mental Model
  • Gelperin and Hetzel’s Evolutionary Testing Model
  • International testing standards – e.g. IEEE 829
  • ISTQB Standard Glossary of testing terms
  • Best practices and procedures from other assessment methods
  • Industry surveys and research
• TAMAR
  • The TMMi Foundation sets requirements for TMMi assessment methods through TAMAR (TMMi Assessment Method Application Requirements)
  • The TMMi Foundation assesses and approves assessment methods against TAMAR
  • A TMMi assessment method must be approved by the TMMi Foundation in order for it to be able to be used to provide TMMi Level certification

• Level 1 – Initial – A chaotic and undefined process often seen as part of the development process
  
  
• Level 2 – Managed – A Managed Process clearly separated from development. Existing practices are retained from project to project.
  
  
• Level 3 – Defined – Testing is no longer a phase which simply follows coding – it is fully integrated into the project life cycle with testing involvement starting as early as requirements definition
  
  
• Level 4 – Measured – The testing organisation is thoroughly designed, well founded and measured, establishing quantitative objectives for product quality.
  
  
• Level 5 – Optimisation – A fully-defined and continually self-improving process, clearly fulfilling its improvement objectives, capable of controlling the cost and effectiveness of testing.
  
  
• Each level is empirically validated.
  
  
• TMMi is a staged model. This means to achieve higher levels, lower levels must also be achieved.
  
  
• Apart from Level 1, each Maturity Level contains a number of Process Areas. These are specific groups of activities which can be evaluated together.