A Review Of SEI CMM & ISO 9001
Standards are important - whether international or company specific - because of what Dr. W. Edward Deming of Total Quality Management fame said, "Quality isn't about people doing their best."
May 20, 2002
Standards are important - whether international or company specific - because of what Dr. W. Edward Deming of Total Quality Management fame said, "Quality isn't about people doing their best." And that probably would scare the dickens out of most managers who view quality as all about personal best and personally doing the right thing. And Deming said, "No, it's about having the right process in place such as when people understand them, then they can do their best within the system. "--Scott Duncan, Standards Chair for ASQ/Software Division.
Dozens of assessment and credentialing methods clamor for the attention of software developers, project managers, and the IT powers that be, but only two have gained enough recognition and widespread use to become the de facto prize fighters in the arena of certifications and standards - ISO 9001:2000 and the Capability Maturity Model (CMM).Bottom line, CMM is a method for measuring and improving capability of software development processes, and ISO 9001 is registration of a quality management system based on established good practices, said Ed Shaffer, a client manager for the British Standards Institution.
"Accredited organizations such as the BSI are authorized to issue ISO 9001 certificates certifying that a registered company meets software standards including design, replication, installation, and maintenance," he said. "With CMM, the Software Engineering Institute (SEI) defines and controls specific criteria for each maturity level: initial, repeatable, defined, managed, and optimized. As an IT firm moves from level 1 through level 5 of software development capability maturity, the organization must satisfy a larger set of criteria."
The intent of ISO 9001 - originally released in 1994 and updated in 2000 - is to minimize variations in quality, rather than to increase quality (Arora & Asundi, 1999). In fact, the International Organization for Standards has developed de facto "check box" standards for quality in a variety of industries worldwide. ISO 9001 is used predominantly among European and Asian software development companies. More generally, Indian IT firms see ISO certification as a marketing tool and a way of breaking away from the pack (Arora & Asundi, 1999).
"ISO certification enhances the ability of firms to grow, but with only a modest impact at best on the rates it can get," concluded Arora & Asundi in their 1999 study of the Indian software industry. "ISO certification is an important means of signaling to potential customers and does help in enabling a firm to provide more sophisticated and higher value-added services to get such contracts, and hence earn higher price per unit of effort."
Developed in 1993, the CMM model was originally used by the U.S. Department of Defense to access the capability of their contractors by prescribing standards in different stages that they must pass at any given level of maturity (Arora & Asundi, 1999).
"ISO 9001 gives customers an ability to access the minimal level of quality that they should expect from a vendor," said Scott Duncan, a process improvement specialist. "It is a 'de jure' [meaning 'of the law'] standard because 50 countries have adopted and recognize the standard worldwide," he said. "CMM is a 'de facto' standard, not a standard in fact. It is a standard that exists because so many people believe that it's good and they use it. Both documents were created to give customers a consistent way of knowing, 'When you do X, what minimum standards do you meet?'"
In general, U.S. organizations view standards as too constricting, said Ron Radice, CMM assessor and president of Software Technology Transition. In CMM and ISO 9001, U.S. organizations approach those standards and guidelines from two perspectives:
To get ISO 9001 certified, companies must satisfy all 20 of its clauses. One criticism aimed at ISO 9001 is that it lacks a continuous process improvement (CPI) mechanism whereas CMM has a CPI component. The newer version - ISO 9001:2000 - is more customer oriented and emphasizes the need for quality improvement. To compare the two versions of ISO 9001, go to http://praxiom.com/iso-new.htm
"In ISO 9001, people used to argue that the process improvement was not even an important aspect of it," Duncan said. "Now everybody who I spoke to and helped write it said that CPI is really in there. And if you really look, it really was, but it wasn't written with enough clarity." Although ISO standards are generic and non-prescriptive, they assist IT firms to define, standardize, and document their software processes, and help them to focus on variables that are important to quality (Arora & Asundi, 1999). Documenting current software processes and conducting internal audits and improvement workshops contributes to acquiring relevant knowledge (Stelzer & Mellis, 1999).
Guidelines, checklists, and training schemes help to transfer important information and communicate process innovations to all relevant members of the organization (Stelzer & Mellis, 1999). "Both CMM and ISO say, 'Document your process. Say what you do and go out there and behave according to the way you say that you're going to behave,'" Duncan said. "And from an auditor's perspective, be prepared to show that you actually do it."
For some IT firms, ISO certification provides the first step in an ongoing commitment to quality. A high percentage of Indian software firms that obtained ISO 9001 certification also received TickIT or CMM certification (Arora & Asundi, 1999).
"Both CMM and ISO 9001 are variable. If you look at the average, organizations seem to attain ISO 9001 within two years if they have good intent of supporting it," Radice said. "CMM has five levels. The SEI maintains that it takes about 18 months to move from level to level. I suppose what that suggests is that CMM has more in it than ISO 9001."
Look at SEI's demographics and you'll see that some organizations move from level to level in nine months and others take much longer than two to three years, he said. In CMM-I, the SEI adds a new key process area or component to level 2, which forces organizations to think about data both in measurement and analysis, he said. In the new version, the SEI encourages people to think about data more deliberately - how it's defined, how it's captured, and how it relates to business, he said.
In CMM level 3, one process - called the engineering process - contains the entire software development lifecycle and includes design, development, and testing, Duncan said. Only with CMM-I did SEI give testing its own section called verification and validation. But for years, CMM level 2 was all about project management. Level 3 is mostly about defining processes accurately, communicating them to people, and training people so they use them properly, he said.
"Unfortunately, when some organizations move through level 2 and level 3, they don't always have their data in the best of waves," Radice said. "In some cases, it even lacks integrity and in other cases, they don't understand what it means yet. The model doesn't get them to think about the data until level 4."
Level 4 is about using the data that you collect through management systems to more accurately represent what's going on in your project and trying to manage through data rather than through whoever has the highest job title and shouts loudest at the meeting, Duncan said. And CMM level 5 is all about well defined processes and reporting data, when you go to change your process and optimize what you're doing, you believe your data, he said. And if your data shows that what you wanted to change in your process is sending your metrics in a bad direction, you believe that, he said.
"For an organization to satisfy all of CMM level 5 is a journey," Radice said. "It's not something that can be done in two years. There are some Indian organizations that come out of nowhere and appear at level 5. How could they have gotten there that fast when it's taking everyone else so much longer? There's a lot of suspicion about some of those [companies]. There's no proof that they're at level 5.
"What may be working for them is that many of the newer Indian companies have come from older Indian companies that had a good quality philosophy and way of life. So if they're able to bring that over to the new company, maybe they can move a little faster to five o'clock. The key is - do they have a foundation? If there's a reasonable foundation to build upon such as ISO 9001 or CMM, then they will all move faster."
What a lot of people don't realize about ISO 9001 and SEI CMM is that they're all about management including senior management organizing projects and resources, training people, and coordinating communication between departments, Duncan said.
In a 1999 report that studied 24 European software organizations that had implemented an ISO 9001 quality system and 31 European and U.S. software organizations that had conducted a CMM-based improvement effort, 100% of the organizations that reported marked success of process improvement also reported that their managers actively monitored the progress of effort and that management commitment is considerably less in organizations with less successful improvement efforts.
"Sponsorship becomes very, very clear through management and senior management," Ron Radice said. "When senior management understands and believes, and sends a convincing message to the organization, the organization follows. If senior management says we need quality, we need quality for one of two reasons: (1) competition is growing, or (2) we just want to be better."
Mere conformance to a standard, attaining certification, or reaching a CMM level is not a relevant goal for staff members; implementing measures just for the sake of the CMM or the ISO 9001, and not for the sake of quality and productivity, imposes extra burdens on project teams (Stelzer & Mellis, 1999). The real test of improvement objectives is the degree to which everyone can make the translation from top management goals to the goals of each person being asked to achieve; objectives must be decomposed to specific measures for project managers and programmers (Stelzer & Mellis, 1999).
The most significant factors of successful improvement efforts include: senior management monitoring the improvement initiative; clear, compensated assignment of responsibilities for process improvement; involvement of respected people in process improvement; involvement of technical staff in the improvement effort; adequate amount of staff time and resources dedicated to process improvement; and clearly stated and well understood process improvement goals (Stelzer & Mellis, 1999).
"Whatever that vision is, that leadership is essential," Radice said. "Then, you can't just ask people to do things. You need to provide funding and resources. There's different ways of doing that. Basically, [you need] to create an infrastructure that would facilitate the improvement."
For example, in CMM terms, software engineering process groups or quality groups could act as "change agents," he said. In an ISO 9001 context, quality managers regularly play the role (Stelzer & Mellis, 1999). These groups and individuals need to be objective and act independently from the organization or from the people actually doing the work, and give them feedback in a non-threatening way so that people can understand and put the results together with a vision and learn how to improve, he said.
Management should always give the simplest possible process that meets the needs of the people in the organization, he said. They should not be getting processes that define every step for every thing that they do because: (1) those processes are never perfect even though the process writers think that they're perfect and (2) if you give anybody - whether in software or any phase of life - a process that is totally defined for them, they no longer have to think.
"And that's totally unacceptable because problems always occur in life and in business," Radice said. "And you want people to think through the problem. You want them to build upon the process, but not get caught up in using the process.
"Too many people say, 'I'm following the process. What do you want me to do?' That's the wrong position to put people in. When the process is not sufficient, you rely on the people to learn how to improve the process. The process needs to be simple because overly onerous or detailed processes don't work and people don't use them."
Works Cited
1. Arora, Ashish, and Asundi, Jai. "Quality Certification and the Economics of Contract Software Development: A Study of the Indian Software Industry," April 1999: www.heinz.cmu.edu/swic/workpapers.htm
2. Stelzer, Dirk, and Mellis, Werner. "Success Factors of Organizational Change in Software Process Improvement," May 1999: http://sern.ucalgary.ca/~ruhe/Offerings/SENG511/Student_Papers/PP2.pdf
Dozens of assessment and credentialing methods clamor for the attention of software developers, project managers, and the IT powers that be, but only two have gained enough recognition and widespread use to become the de facto prize fighters in the arena of certifications and standards - ISO 9001:2000 and the Capability Maturity Model (CMM).Bottom line, CMM is a method for measuring and improving capability of software development processes, and ISO 9001 is registration of a quality management system based on established good practices, said Ed Shaffer, a client manager for the British Standards Institution.
"Accredited organizations such as the BSI are authorized to issue ISO 9001 certificates certifying that a registered company meets software standards including design, replication, installation, and maintenance," he said. "With CMM, the Software Engineering Institute (SEI) defines and controls specific criteria for each maturity level: initial, repeatable, defined, managed, and optimized. As an IT firm moves from level 1 through level 5 of software development capability maturity, the organization must satisfy a larger set of criteria."
The intent of ISO 9001 - originally released in 1994 and updated in 2000 - is to minimize variations in quality, rather than to increase quality (Arora & Asundi, 1999). In fact, the International Organization for Standards has developed de facto "check box" standards for quality in a variety of industries worldwide. ISO 9001 is used predominantly among European and Asian software development companies. More generally, Indian IT firms see ISO certification as a marketing tool and a way of breaking away from the pack (Arora & Asundi, 1999).
"ISO certification enhances the ability of firms to grow, but with only a modest impact at best on the rates it can get," concluded Arora & Asundi in their 1999 study of the Indian software industry. "ISO certification is an important means of signaling to potential customers and does help in enabling a firm to provide more sophisticated and higher value-added services to get such contracts, and hence earn higher price per unit of effort."
Developed in 1993, the CMM model was originally used by the U.S. Department of Defense to access the capability of their contractors by prescribing standards in different stages that they must pass at any given level of maturity (Arora & Asundi, 1999).
"ISO 9001 gives customers an ability to access the minimal level of quality that they should expect from a vendor," said Scott Duncan, a process improvement specialist. "It is a 'de jure' [meaning 'of the law'] standard because 50 countries have adopted and recognize the standard worldwide," he said. "CMM is a 'de facto' standard, not a standard in fact. It is a standard that exists because so many people believe that it's good and they use it. Both documents were created to give customers a consistent way of knowing, 'When you do X, what minimum standards do you meet?'"
In general, U.S. organizations view standards as too constricting, said Ron Radice, CMM assessor and president of Software Technology Transition. In CMM and ISO 9001, U.S. organizations approach those standards and guidelines from two perspectives:
- When they are forced to do it.
- When they see the merits of improving and acknowledge that both CMM and ISO 9001 can afford them a path towards improvement.
To get ISO 9001 certified, companies must satisfy all 20 of its clauses. One criticism aimed at ISO 9001 is that it lacks a continuous process improvement (CPI) mechanism whereas CMM has a CPI component. The newer version - ISO 9001:2000 - is more customer oriented and emphasizes the need for quality improvement. To compare the two versions of ISO 9001, go to http://praxiom.com/iso-new.htm
"In ISO 9001, people used to argue that the process improvement was not even an important aspect of it," Duncan said. "Now everybody who I spoke to and helped write it said that CPI is really in there. And if you really look, it really was, but it wasn't written with enough clarity." Although ISO standards are generic and non-prescriptive, they assist IT firms to define, standardize, and document their software processes, and help them to focus on variables that are important to quality (Arora & Asundi, 1999). Documenting current software processes and conducting internal audits and improvement workshops contributes to acquiring relevant knowledge (Stelzer & Mellis, 1999).
Guidelines, checklists, and training schemes help to transfer important information and communicate process innovations to all relevant members of the organization (Stelzer & Mellis, 1999). "Both CMM and ISO say, 'Document your process. Say what you do and go out there and behave according to the way you say that you're going to behave,'" Duncan said. "And from an auditor's perspective, be prepared to show that you actually do it."
For some IT firms, ISO certification provides the first step in an ongoing commitment to quality. A high percentage of Indian software firms that obtained ISO 9001 certification also received TickIT or CMM certification (Arora & Asundi, 1999).
"Both CMM and ISO 9001 are variable. If you look at the average, organizations seem to attain ISO 9001 within two years if they have good intent of supporting it," Radice said. "CMM has five levels. The SEI maintains that it takes about 18 months to move from level to level. I suppose what that suggests is that CMM has more in it than ISO 9001."
Look at SEI's demographics and you'll see that some organizations move from level to level in nine months and others take much longer than two to three years, he said. In CMM-I, the SEI adds a new key process area or component to level 2, which forces organizations to think about data both in measurement and analysis, he said. In the new version, the SEI encourages people to think about data more deliberately - how it's defined, how it's captured, and how it relates to business, he said.
In CMM level 3, one process - called the engineering process - contains the entire software development lifecycle and includes design, development, and testing, Duncan said. Only with CMM-I did SEI give testing its own section called verification and validation. But for years, CMM level 2 was all about project management. Level 3 is mostly about defining processes accurately, communicating them to people, and training people so they use them properly, he said.
"Unfortunately, when some organizations move through level 2 and level 3, they don't always have their data in the best of waves," Radice said. "In some cases, it even lacks integrity and in other cases, they don't understand what it means yet. The model doesn't get them to think about the data until level 4."
Level 4 is about using the data that you collect through management systems to more accurately represent what's going on in your project and trying to manage through data rather than through whoever has the highest job title and shouts loudest at the meeting, Duncan said. And CMM level 5 is all about well defined processes and reporting data, when you go to change your process and optimize what you're doing, you believe your data, he said. And if your data shows that what you wanted to change in your process is sending your metrics in a bad direction, you believe that, he said.
"For an organization to satisfy all of CMM level 5 is a journey," Radice said. "It's not something that can be done in two years. There are some Indian organizations that come out of nowhere and appear at level 5. How could they have gotten there that fast when it's taking everyone else so much longer? There's a lot of suspicion about some of those [companies]. There's no proof that they're at level 5.
"What may be working for them is that many of the newer Indian companies have come from older Indian companies that had a good quality philosophy and way of life. So if they're able to bring that over to the new company, maybe they can move a little faster to five o'clock. The key is - do they have a foundation? If there's a reasonable foundation to build upon such as ISO 9001 or CMM, then they will all move faster."
What a lot of people don't realize about ISO 9001 and SEI CMM is that they're all about management including senior management organizing projects and resources, training people, and coordinating communication between departments, Duncan said.
In a 1999 report that studied 24 European software organizations that had implemented an ISO 9001 quality system and 31 European and U.S. software organizations that had conducted a CMM-based improvement effort, 100% of the organizations that reported marked success of process improvement also reported that their managers actively monitored the progress of effort and that management commitment is considerably less in organizations with less successful improvement efforts.
"Sponsorship becomes very, very clear through management and senior management," Ron Radice said. "When senior management understands and believes, and sends a convincing message to the organization, the organization follows. If senior management says we need quality, we need quality for one of two reasons: (1) competition is growing, or (2) we just want to be better."
Mere conformance to a standard, attaining certification, or reaching a CMM level is not a relevant goal for staff members; implementing measures just for the sake of the CMM or the ISO 9001, and not for the sake of quality and productivity, imposes extra burdens on project teams (Stelzer & Mellis, 1999). The real test of improvement objectives is the degree to which everyone can make the translation from top management goals to the goals of each person being asked to achieve; objectives must be decomposed to specific measures for project managers and programmers (Stelzer & Mellis, 1999).
The most significant factors of successful improvement efforts include: senior management monitoring the improvement initiative; clear, compensated assignment of responsibilities for process improvement; involvement of respected people in process improvement; involvement of technical staff in the improvement effort; adequate amount of staff time and resources dedicated to process improvement; and clearly stated and well understood process improvement goals (Stelzer & Mellis, 1999).
"Whatever that vision is, that leadership is essential," Radice said. "Then, you can't just ask people to do things. You need to provide funding and resources. There's different ways of doing that. Basically, [you need] to create an infrastructure that would facilitate the improvement."
For example, in CMM terms, software engineering process groups or quality groups could act as "change agents," he said. In an ISO 9001 context, quality managers regularly play the role (Stelzer & Mellis, 1999). These groups and individuals need to be objective and act independently from the organization or from the people actually doing the work, and give them feedback in a non-threatening way so that people can understand and put the results together with a vision and learn how to improve, he said.
Management should always give the simplest possible process that meets the needs of the people in the organization, he said. They should not be getting processes that define every step for every thing that they do because: (1) those processes are never perfect even though the process writers think that they're perfect and (2) if you give anybody - whether in software or any phase of life - a process that is totally defined for them, they no longer have to think.
"And that's totally unacceptable because problems always occur in life and in business," Radice said. "And you want people to think through the problem. You want them to build upon the process, but not get caught up in using the process.
"Too many people say, 'I'm following the process. What do you want me to do?' That's the wrong position to put people in. When the process is not sufficient, you rely on the people to learn how to improve the process. The process needs to be simple because overly onerous or detailed processes don't work and people don't use them."
Works Cited
1. Arora, Ashish, and Asundi, Jai. "Quality Certification and the Economics of Contract Software Development: A Study of the Indian Software Industry," April 1999: www.heinz.cmu.edu/swic/workpapers.htm
2. Stelzer, Dirk, and Mellis, Werner. "Success Factors of Organizational Change in Software Process Improvement," May 1999: http://sern.ucalgary.ca/~ruhe/Offerings/SENG511/Student_Papers/PP2.pdf