Software Process Improvement Works!
1 Background 1.1 History Advanced Information Services Inc. (AIS) is an independent software contracting organization that operates a main office in Peoria, Illinois, a satellite office in the Chicago area, and a subsidiary in Chennai, India. The business segments of AIS include software development, consulting, software process training, and Internet services. AIS began developing commercial systems in 1986. Embedded system development effort began in 1997. The AIS Development Group maintains a staff of between 20 and 35 people. The customers with whom AIS works look to the organization to develop state-of-the-art systems. The majority of commercial system development is an unprecedented effort as engineers utilize cutting-edge technology in their development work. The systems are from many different application domains. Platforms vary from the Internet to client-server to mainframe. Programming languages used include C, C++, COBOL, Java, and Visual BASIC. The development projects for which AIS provides resources can be classified as either custom commercial or embedded systems. These projects are typically staffed with teams ranging in size from one to six people. 1.2 Business Needs and Software Process Improvement The continuing commitment of AIS to process improvement is motivated by the business needs to: · Deliver defect-free software products and satisfy customers increasingly demanding time-to-market goals. · Achieve a sustained competitive advantage in the consulting business by becoming a provider of teams of process trained engineers. · Minimize the impact of staff turnover.The software process improvement initiative at AIS started in January 1992. The goals were to: · Improve profitability of development projects by meeting cost estimates and schedule commitments with reasonable consistency. · Provide a continuing management focus on the progress and visibility of each project from initial commitment to orderly progression through the development life-cycle phases and customer acceptance. · Enable continuous improvement of the development process through a changed organizational culture biased towards rapid implementation of many small incremental improvements as opposed to a few large changes. 1.3 Process Improvement Strategy AIS named its process improvement initiative Continuous Process Improvement (CPI). The company chose the SEI Capability Maturity Model (CMM)1 as the process maturity framework to improve organizational process capability and the Institute of Electrical and Electronics Engineers (IEEE) standards as the guidelines for software engineering. The Personal Software Process (PSP)2 was adopted as the enabling technology to improve individual engineer performance and productivity. Process definition gradually evolved and matured across maturity levels and key process areas (KPAs) through a simple and effective mechanism of the Process Improvement Proposal (PIP). To evaluate and implement the PIPs, a Software Engineering Process Group (SEPG), utilizing the skills and experience of many engineers, was established with the allocation of these resources being on a part-time basis. Additionally, Watts Humphreys Managing the Software Process [Humphrey 89] was used as a guide. The approach was to: · Conduct self-assessments with a focus on action. · Utilize the skill and experience of many people to implement the findings and improvements. · Utilize a PIP process to enable and guide the gradual evolution of AIS processes. · Maintain organization awareness of improvement efforts through quarterly status reviews. 1 The Capability Maturity Model (CMM) is a model developed through the efforts of the SEI, which provides software organizations with the guidance necessary to gain control of their processes for developing high-quality software. Based on the software process maturity framework, the CMM is designed to assist software organizations in the selection of process improvement activities by determining their current process maturity.
2 The PSP is a structured and disciplined framework that helps engineers plan, track, and control their individual work. It assists software engineers in estimating the size of the work product and effort needed to complete the task, and it provides a framework to produce a high-quality product. The PSP utilizes disciplined practices such as personal design reviews and code reviews to enable software engineers to remove defects early in the life cycle of a work product.
[Title Page] [Abstract] [Preface] [Chapter 1] [Chapter 2] [Chapter 3] [Chapter 4] [Appendix A] [References] [DTIC page] [PDF file] The Software Engineering Institute (SEI) is a federally funded research and development center sponsored by the U.S. Department of Defense and operated by Carnegie Mellon University. Copyright 2001 by Carnegie Mellon UniversityURL: http://www.sei.cmu.edu/publications/documents/99.reports/99tr027/99tr027chap01.html Last Modified: 21 March 2001
Software Process Improvement Works!
2 Continuous Process Improvement Mechanisms 2.1 Organization Infrastructure The unshaded boxes in Figure 1 represent the entities that comprise the Development Group. These resources may also have some additional corporate-wide responsibilities. The AIS President provides resources, approvals, and policy direction to the Development Manager. The AIS Development Manager provides approvals and direction to the Development Group entities and to the software development process. The Development Manager also reviews status and issues with the President. Computing Support and Internet Services manage the development environment. The Project Managers and software engineers enact the software development process. Project Managers review plans, issues, and status with the Development Manager. The Subcontract Manager is responsible for administering the terms of the master subcontract agreement with the AIS subsidiary in Chennai, India. The Software Quality Assurance (SQA) Group is responsible for audit and review of the development process and for sharing its findings with the Development Manager. SQA notifies the President of issues. The SEPG performs Interim ProfileSM assessments of all current projects. The group also reviews PIPs to assess what impact implementation may have on the organizations process. If the determination is made that the proposal meets the companys criteria for implementation, work is done to make it a part of the company process. The Software Configuration Management (SCM) Group provides configuration management and change control support. Project Managers and software engineers staff the SQA, SCM, and SEPG functions. Figure 1: Organization Infrastructure Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
2.2 Project Processes and Capability Maturity Model In 1992, the AIS CPI-documented processes were incorporated into a one-volume manual. This was organized with a set of policies in the introductory section. Management and software engineering processes and standards were added in sequential order as they were developed. Version 1.0 of the CPI Manual adopted existing IEEE standards for life cycle, requirements specification, quality assurance, configuration management, verification and validation, and testing. The policies regarding those processes acted as pointers to the IEEE standards that were elsewhere in the library. A set of detailed Work Breakdown Structures (WBS 1.0), an integration of best practices from several successful projects, were added during this time period. Estimating spreadsheets that mirrored the activities in the WBS were developed to help with planning size and effort for each project phase. (The SEPG collects data from these spreadsheets to update the standard productivity rates that are part of the spreadsheets.) New processes were added and existing processes were modified as a result of PIPs and self-assessment findings. In 1995, it was decided that process improvement efforts would be more effective if the processes were organized in the same manner as the improvement model that was being used. The CPI was restructured by dividing the process manual into three volumes to match the three process categories in the key practices of the CMM, Version 1.1. The Management, Organizational, and Engineering Manuals were developed by organizing each volume by key process area and sequenced by maturity level. Each AIS process was then mapped to a key process area. Processes that did not appear to pertain to a KPA were analyzed and moved to other company administrative documentation. The processes were then organized with all relevant documentation located in the same subject. By 1995, over 50 PIPs had been written to improve the WBS and it had evolved to Version 1.4. The WBS then consisted of 414 tasks. Indications were that an analysis should be conducted to determine which WBS tasks were actually being used by projects. The unplanned activities being added to projects were also analyzed. This information, along with recent PIPs and feedback from Project Managers, was used to remove redundancies and to consolidate similar small effort tasks. Modifications encompassed the inclusion of small effort tasks into checklists, the identification of tasks that should be pointers to scripts, and the creation of a process for tailoring. As a result of the introduction of PSP, it was necessary to integrate PSP tasks into the WBS and associated processes. Project Managers presented draft iterations of WBS 2.0 with the final draft consisting of 207 tasks. The architecture of the WBS was changed to be consistent with the new architecture of the CPI Manual. The project management tasks were moved to one template separate from the software engineering templates. The software engineering templates were revised for each life-cycle phase. The project management template consisted of tasks that were grouped by KPA. Each WBS task contained a reference to a CMM key process area goal or key practice. The Development Groups standard process is documented in hard copy. The SEPG is the owner of the process documentation and is responsible for updates. New versions are targeted for release to coincide with Quarterly Status Reviews. In January of 1999, the CPI processes were made available online through the CPI on the Web (CPIW) application. 2.2.1 Project Management Project management has evolved to be a well-defined, highly integrated set of processes that are used throughout the life cycle of a project. (See Figure 2.) Figure 2: Project Planning Process (1) Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
The Pre-Commitment Process is executed when an initial written or verbal Request for Proposal is received. This process is also initiated prior to the beginning of each phase of a project. It is the mechanism to obtain a commitment from senior management, the customer, development team, and other affected groups such as Software Quality Assurance and Software Configuration Management. During the Pre-Commitment Process, the Risk Identify, Analyze, and Plan phases of the Risk Management Process are completed. The output from the process (Risk Identify, Analyze, and Plan form) is used to communicate with the Senior Executive and to gain an understanding of how to proceed with the project. Assumptions from this process are integrated into the Statement of Work Process, and mitigating activities or tasks are added to the project plan. The Statement of Work Process is executed in stages. Figure 3 shows a relationship between the stages. Determinations as to the actual work to be performed, identification of the customer, dates for completion, and the criteria for determining the success are established during the Goals and Objectives stage. During the Work Breakdown Structure stage, the team tailors the AIS standard WBS to fit the project and agrees on the activities and tasks to be estimated. Standard estimating spreadsheets and historical data are used to estimate the size, effort, schedule, and cost for the activities and tasks in the WBS. Each software engineer uses the PSP and his or her own historical data to estimate the size, effort, and schedule when there are components to be built. These estimates are used in planning with some adjustment for dependencies, such as team inspections. The Statement of Work (SOW) is then written (assembled) and reviewed with the customer. Figure 3: Project Planning Process (2) Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
After a commitment to proceed with the project is received from the customer, the Project Plan Process is executed. This process involves refining the plan with input from the customer review of the SOW. Project startup administrative tasks such as the Time Accounting Process, Project History Book, and Computing Support requests are also initiated. Figure 4 shows how the Development Tracking Process is utilized during each phase as the mechanism for project tracking and oversight and for determining corrective actions. Each week the Development Manager meets with each Project Manager to assess quality by reviewing the defects found and fixed in each phase of the process. They also assess the Schedule Plan vs. Actual and Effort Plan vs. Actual by reviewing the earned value tracking charts. Unplanned activities and intergroup coordination activities are discussed. The risks that were identified during the Risk Identify, Analyze, Plan Process are reviewed on a periodic and event-driven basis, and new mitigation strategies are determined. Figure 4: Project Tracking & Control Processes Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Software Configuration Management, Software Quality Assurance, and Requirements Management are included as activities in the WBS for each life-cycle phase. At the Phase Review, which is held at the end of each life-cycle phase, the quality, effort, and schedule results are presented to the customer along with the final deliverables as stated in the Statement of Work. The customer is given a Customer Feedback form for rating and commenting on the project phase. During the Quarterly Status Review (QSR), the Project Manager from each project presents quality, effort, and schedule status to the Development Group. 2.2.2 Software Engineering Software Engineering Work Breakdown Structures exist for the software engineering tasks of Requirements Engineering, Preliminary Design, Design/Code/Test, Installation/Checkout, and Operational Support phases. A Work Breakdown Structure for objected-oriented analysis and design has been piloted in projects, and will be incorporated into the AIS process in the future. Deliverables for each of the software engineering tasks are defined along with processes and standards. All phases of the life cycle require formal inspections, and these are referenced in the Work Breakdown Structures. Planning, Overview, Preparation, Meeting, Rework, and follow-up stages organize the formal inspection process. Guidelines, scripts, and forms are incorporated into the process. AIS software engineers use PSP during the Design/Code/Test phase of our software life cycle. (See Figure 5.) Activities include: · Engineers estimate their own modules using their historical data on size estimates, productivity rates, and effort estimates. · Engineers create a plan for each module. · Engineers submit their plans to the Project Manager. · Project Manager produces the project plan. Figure 5: Software Engineering Processes Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
During the development of the product, the following occur: · Engineers follow PSP life-cycle phases: Design, Personal Design Review, Team Design Inspection, Code, Personal Code Review, Team Code Inspection, Compile, and Unit Test. · Personal checklists are used for design and code reviews. · Defects are recorded during all the phases of the work product. · Checklists are updated based on defect analysis. · Engineers track their progress using Planned Value vs. Earned Value1 and Planned Effort vs. Actual Effort. · Engineers report their progress to the Project Manager every week. · Project Manager updates the project plan and reports the status of the project to the Development Manager. · Upon completion, engineers record the actual size of the work product, and actual time taken to complete the product, and analyze size, effort, and defect data to make necessary improvements. (See Figure 6 and Figure 7.) · Subsequent to engineers reporting their data, the Project Manager compiles all the information to reflect Planned vs. Actual Earned Value and Planned vs. Actual Hours from a total project perspective as shown in Figure 8 and Figure 9. Figure 6: Individual Planned vs. Actual Earned Value Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 7: Individual Planned vs. Actual Hours Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 8: Project Planned vs. Actual Earned Value Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 9: Project Planned vs. Actual Hours Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
2.3 Organization Processes 2.3.1 Process Improvement Proposal The PIP process complements the evolutionary process maturity framework inherent in the CMM by ensuring that each one of many small changes introduced is based on practitioners experience. In July 1992, AIS adopted PIP forms in order to empower engineers to evolve AIS processes by means of suggesting small incremental improvements. The PIP form provides a direct correlation between the CMM KPA and the improvement benefit to the AIS process. Where possible, these benefits are quantified. The SEPG was made responsible for evaluating PIPs, identifying resources to implement improvements, providing feedback to the authors, maintaining PIP records, and reporting PIP status in the QSR. 2.3.2 Training Program The commitment to required software engineering training institutionalizes the software process to existing staff, as well as newly hired staff. It is the key mechanism that enables AIS to supply the industry with teams of process-trained engineers. The AIS Training Program consists of three tracks: Core Processes, Job Requirements, and Career Path. The training in the job-related and career path tracks is delivered to meet project needs or an individuals career path goals. These courses are generally outsourced. AIS staff teaches classes in object-oriented analysis and design and Java. The following software engineering courses are required for all positions related to software development and are taught at AIS: · Managing the AIS Software Process - Introduces software quality issues and the processes necessary for continuous improvement. Software process maturity is defined within the framework of the SEI CMM. AIS processes and functions such as SQA, SCM, and SEPG are introduced in relationship to the CMM key process areas. Assignments include creation of a project plan. Text: Managing the Software Process by Watts Humphrey [Humphrey 89] · Personal Software Process - Defines the software engineering process in a series of seven personal processes beginning with a basic process (PSP0) and building to a process capable of supporting larger scale development (PSP3). The engineers learn to plan, track, and improve their own processes and how to improve quality by using defect management. Assignments consist of 8 to 10 programming assignments and 5 report exercises. Text: A Discipline for Software Engineering by Watts Humphrey [Humphrey 95] · Requirements Engineering - Defines requirements engineering as the process of elicitation, analysis, specification, and verification, and as the productthe Software Requirements Specification based on IEEE standards. Methods, tools, and issues are discussed. Resources: Industry expert and AIS workshop materials · Inspection Process - Explains the concepts and principles behind software inspections, introduces the AIS inspection process, and provides practice with AIS inspection process. Text: Software Inspection Process [Ebenau 94] 2.4 Process Maturity Assessment 2.4.1 Self-Assessment A formal assessment did not seem financially feasible at the onset of the companys process improvement efforts. In lieu of a formal assessment, the decision was made to utilize the SEI-technical report, A Method for Assessing the Software Engineering Capability of Contractors [Humphrey 87]. All members of the Development Group completed the questionnaire associated with this report. Project teams were interviewed to identify and prioritize the major assessment findings. At the conclusion of these efforts, the determination was made that the organization was operating at the Ad Hoc level as an organization. The major findings were mapped to Humphreys suggestions for an organization to improve its processes with the objective of moving from the Ad Hoc level to the SEI CMM Level 2 key process areas. The Project Management System was the focus for advancing from the Ad Hoc level. The areas identified as target items on which to focus were: · Commitment Review · Quarterly Status Review · Phase ReviewThe organization elected to have resources expend effort in three KPAs from CMM Level 2Project Planning, Project Tracking, and Software Configuration Management. The process areas were: · Project Process · Project Tracking Process · Environment Documentation The findings were followed up with specific aforementioned actions of: · Establishment of the CPI Initiative · Creation of the CPI Manual · Organization of the Software Engineering Process Group (SEPG)The first QSR was held in March 1992. The same model was used for improvement through 1993, using the questionnaire to provide internal process assessments every six months, to report on the results, and to plan improvements from the findings. In 1994, the SEI Interim Profile process was tailored for use by AIS. The current process is for individual project team members to complete the Software Project Maturity Questionnaire semiannually. From this questionnaire, the SEPG constructs a draft project profile, meets with the team to review results, and works to resolve discrepancies. A final project profile is then prepared. The final project profiles are rolled up into the organization profile. Project Managers present project profiles in the QSR, and the SEPG presents the organization profile. 2.4.2 CMM-Based Assessment for Internal Process Improvement In April 1996, AIS participated in a CMM-Based Assessment for Internal Process Improvement (CBA IPI) with an external, independent SEI-licensed lead assessor. The objectives of the appraisal were to identify the strengths and weaknesses of the AIS software process, to identify the highest priority issues for software process improvement, and to provide a framework for process improvement actions. The scope was to assess five projects where AIS had primary responsibility for project and process management and to evaluate the AIS software process in all 13 of the key process areas of the CMM for Repeatable and Defined levels. The following global strengths were noted during the assessment: · Processes at AIS have undergone evaluation and have evolved since 1992. There is strong evidence of the correlation between project profitability and process fidelity. With the organizational culture and process in place, it is expected that the benefits of process improvement effort will continue into the future. · Process improvement efforts are included in performance evaluations (first item in position accountability statements). · The SQA function has gained the confidence of the projects as a value-added role. · Software engineers use the PSP in most aspects of their jobs.Improvement opportunities were identified in the KPAs of Software Configuration Management, Software Quality Assurance, Subcontract Management, and Organization Process Definition. The remainder of the KPAs for Levels 2 and 3 were fully satisfied. The PSP addresses a number of these satisfied KPAs including Software Project Planning and Software Project Tracking and Oversight at Level 2, as well as Organization Process Focus, Integrated Software Management, Software Product Engineering, and Peer Reviews from Level 3. There seems to be a correlation between the KPAs that PSP addresses and the satisfied KPAs. An action plan based on the improvement opportunities identified in the assessment was presented to the Development Group at the June 1996 QSR. The action plan identified each task, planned schedule for implementation, and resources assigned. In addition to improvement opportunities identified at the Repeatable and Defined levels, the action plan contained tasks related to moving the organization to the Managed (Quantitative) level. The status of the plan was reviewed at the beginning of every subsequent QSR until all tasks were implemented. Adjustments in resources were made to keep the plan on schedule. The Interim Profile process is to be conducted semiannually. A reassessment (CBA IPI) was planned for April 1999. The follow-up assessment was conducted in April 1999. The primary purpose of the appraisal was to provide recommendations on which to focus AIS future process improvement actions. The primary goal of the assessment was not to provide a CMM level rating, although one of the objectives was to ascertain this information. The scope again was limited to AIS projects where AIS had the primary responsibility for project and process management. Four current projects were included in the assessment and were evaluated in all 15 of the KPAs for the CMM levels Repeatable, Defined, and Managed levels as well as the Defect Prevention KPA from the Optimizing Level. As a result of the assessment, the following global strengths were identified: · Quality and process culture supports personnel versatility and movement within the organization. · Process culture also provides organizational stability while moving into new application domains. · Training infrastructure provides effective knowledge and skills necessary for personal growth and organizational success. · People "make" the process.Evidence gathered during the assessment verified that all of the KPAs for Levels 2 and 3 were fully satisfied. Improvement opportunities were identified for the following KPAs: Quantitative Process Management, Software Quality Management, and Defect Prevention. The findings from the assessment determined that the AIS Development Group had implemented the CMM key practices and had satisfied the goals found in Level 2 and Level 3 of the CMM. An action plan based on the improvement opportunities is being formulated. The action plan will identify an activity, resource(s), and schedule for each improvement opportunity or recommendation that was an outcome of the assessment. 1 Earned Value tracking is a method for evaluating a project's progress by establishing relative value for each task to be completed. By using the Earned Value system, a common measure can be used to judge the progress of a project. A common value scale is assigned to each task, regardless of the type of work. Total hours for the entire project are estimated, and each task is assigned an earned value based on its individual estimated percentage of the total hours. Earned Value is credited only when a task is fully completed. If a task is too big, it can be broken down into subtasks. By doing this, the project will be able to show some Earned Value for the completion of the subtask.
[Title Page] [Abstract] [Preface] [Chapter 1] [Chapter 2] [Chapter 3] [Chapter 4] [Appendix A] [References] [DTIC page] [PDF file] The Software Engineering Institute (SEI) is a federally funded research and development center sponsored by the U.S. Department of Defense and operated by Carnegie Mellon University. Copyright 2001 by Carnegie Mellon UniversityURL: http://www.sei.cmu.edu/publications/documents/99.reports/99tr027/99tr027chap02.html Last Modified: 21 March 2001
Software Process Improvement Works!
3 Results The changes in AIS process capability can be delineated into three different eras. The first era was the time period before 1992, at which time AIS did not use a model for software process improvement. In the second era, from 1992 to 1995, the company implemented the CMM. Data indicate that project schedules and effort predictability improved with the introduction of the CMM maturity framework. The third era began in 1995 when AIS piloted the PSP. The PSP was integrated into the defined process in 1996. Data gathered reflected an increase in product quality, as well as improvements in productivity levels. The consequent reduction in rework led to lower life-cycle cost and greater profitability. The data shown are for commercial systems that AIS has been developing since 1986. The data for the embedded system development are not included with the commercial system data because the application domain is quite different. The embedded system work was a series of components each developed by an individual through the entire life cycle. Each component was delivered to the customer as a separate entity. The results for embedded systems will be discussed in the PSP section. 3.1 Schedule and Effort Predictability Figure 10 graphically depicts the impact of the CPI initiative and activities on projects schedule commitments. Before 1992, successful completion of projects was dependent upon the individual ability and Herculean efforts of a few Project Managers and engineers. The AIS Development Group has progressed from an ad hoc environment with unpredictable and frequent over-budget efforts to an environment of disciplined commitments. The process limits in Figure 10 show the change in capability to predict schedule from 1988 through 1998. Although the company was established in 1986, data were not available for projects started earlier than July 1988. The process limits were calculated using a moving range. The date when the project development phase started was used as the horizontal coordinate because experience indicates that the process maturity of the project phase is influenced by the maturity of the organization when the phase started. Each data point represents a new development phase. Figure 10: Schedule Deviation Individual Value Control Chart - Commercial Systems Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
After 1992, when use of the CMM as the improvement model was instituted, schedules became more predictable. At that time, project schedules were based on the AIS defined pre-commitment process with formal executive review and involvement of Project Managers and engineers in project planning. CPI activities such as Quarterly Status Review, Phase Review, and Software Quality Assurance ensured project visibility. Executive oversight of the development process provided a significant positive impact on the projects ability to meet schedule commitments. The data indicate that the average schedule deviation improved from 112% in the pre-model era to 41% in the CMM era to 5% in the CMM+PSP era. Schedule predictability again improved when each software engineer used PSP to plan the schedule for his or her component and made that commitment to the Project Manager and overall project plan. Figure 11 graphically depicts the impact of the CPI initiative and activities on a projects effort commitments. While meeting the schedule met the customers needs, much of the work was contracted on a fixed-cost bid and, if effort was not predictable, the project phase was not profitable. When AIS began using the PSP, in addition to continuing CMM-based improvement, effort became more predictable. Some of the effort predictability was the result of the PSP-trained software engineers planning their own work and utilizing their own historical data to estimate; however, the most significant improvement in effort predictability occurred because of the quality practices the PSP-trained software engineers applied. The data reflect the average effort deviation improved from 87% in the pre-model era to 37% in the CMM era to -4% in the CMM + PSP era. Figure 11: Effort Deviation Individual Value Control Chart - Commercial Systems Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
3.2 Quality & Productivity In June 1995, the PSP was piloted on a project and was later incorporated into the AIS defined process. Figure 12 shows that projects using the PSP have significant reductions in System Test duration. PSP-trained engineers have few or no acceptance test and usage defects. Similar results were not achieved prior to PSP adoption. The consequent near elimination of rework time and effort flows directly to the company bottom line. Since adopting the PSP, acceptance tests that are executed by the customer have also been of short duration because of little or no rework. On the pilot project where AIS introduced the PSP in 1995, some of the software engineers had been trained in PSP and some had not. The experience and education of the engineers in both groups were similar. Acceptance test defects were tracked to the components developed by each engineer, and results showed that the PSP-trained software engineers produced a higher quality product. Figure 13 reflects these data. Figure 12: System Test Data Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 13: Acceptance Test Defects: PSP vs. Non-PSP Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 14 illustrates how the use of a disciplined process enabled the PSP-trained engineers to be more productive. The time used for the LOC/hour calculation is the total time for the production of components. For each component, the time was tracked for all phases of that component including design, code, and test activities and all planning, reviews, and inspections. Figure 14: Productivity: PSP vs. Non-PSP Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 15 contains defect removal data from projects that have completed the entire development life cycle since the integration of tracking defects by phase into the process. Individual PSP reviews are used along with formal inspections throughout the life cycle of a product and have proven to be an effective defect-containment strategy. Figure 15: Defects Removed by Phase Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
3.3 Individual Software Engineer When AIS ventured into embedded software domain, the process framework and PSP assisted in improving development practices. Figure 16, Figure 17, Figure 18, and Figure 19, the data on embedded software development done by an individual software engineer. This was an experienced individual who had knowledge of PSP; therefore, there was a higher degree of quality in the work from the beginning of development. These charts clearly demonstrate the improvement in estimating accuracy of size and effort. Unit/component test defects were around one defect/KLOC. During the development of these components, the engineer showed a slight gain in productivity because of no rework tail. Figure 16: Size Estimating Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 17: Time Estimating Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 18: Productivity Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
Figure 19: Test Defects Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
3.4 Process Culture Continuous improvement of the development process occurs because of an organizational culture that understands continuous improvement and its relationship to the CMM, and participates in proposing and implementing small incremental improvements. Figure 20 shows the number of PIPs implemented by CMM key process area since 1992. Of the current group, 95% of all engineers who have been with the group more than 6 months have submitted at least one PIP. Figure 20: Implemented PIPS by KPA Figures in this file are displayed in a separate browser window. This window will remain open to display figures in this file, although it might be hidden behind other browser windows.
From the period July 1992 to October 1998, engineers submitted 635 individual PIPs. The SEPG implemented 412 PIPs in an orderly and institutionalized manner. Sixty-three individual engineers have submitted PIPs indicating that CPI is broad based within the AIS Development Group. Data also provide evidence that participation in process improvement efforts benefits individual career enhancement prospects. Engineers submitting the most PIPs have been promoted to President, Development Manager, Process Manager, and Training Director. The impact of CPI has been most evidenced in the change of attitude within the organization. It is no longer necessary to explain the value of process improvement to employeesit is a way of life. Evidence of this is found in the comments from members of the Development Group in response to the question "What convinced you that process improvement works?" 3.4.1 Individual Anecdotes One software engineer, who has been with AIS about six months, described her experience as compared to previous employment. "About two years back I was hired for incorporating changes in the existing information system of a major life insurance company. The system was so huge that none of us (including the on-site managers) had an in-depth understanding of the entire system as there was no written Software Requirements Specification. We were told what modules to make the changes in. But it would take hours to make a single change because we had to look through undocumented programs, each of them having between 13 KLOC - 15 KLOC. The thought "how I wish the code was documented" crossed my mind not just once, but every single time I started to make a change! And we were aware that these changes probably had ripple effects in other modules that needed to be taken care of. But we had no clue as to which modules were getting affected, and how they were getting affected. Moreover fresh requirements were being gathered on-site as we were busy enhancing the modules. The result - chaos!" She continued, "When I browsed through the AIS web site and read that they gave high priority to quality and software processes, I knew where I had to go! I know now that I made a right decision because at AIS every task is done in a structured and organized manner using processes. What I have learned from my experience is that processes are not only important in the work environment, but in every walk of life!" Another software engineer who has been with us for several years told her story. "I was working on a large project at a Fortune-100 customer when our customer supervisor came hurriedly to my desk and said that she wanted some information about our project urgently. At that time my AIS Project Manager was not present, but my team member and I could provide all the information she wanted about the project. The customer supervisor wanted to know: Is it on schedule? How much effort was spent? What was the cost at this point? How many hours were spent on enhancements? How much did we spend on Software Change Requests? We gave her the information.... She was thoroughly impressed! This incident convinced me that our process of planning and tracking projects is very useful, not only to us but to our customers also! This made me realize the importance of process." A new Project Manager shared her account of what convinced her that process improvement works. "In recently assuming the role of Project Manager of a new development project, with little experience managing a project using software processes from the beginning of a project to its close, I have been able to make the transition in responsibilities with a relatively small amount of effort. I firmly believe the process training I have received throughout my career here at AIS, along with the highly qualified process-trained staff, which have not only been my supervisors but my mentors, are the two major contributing factors for this smooth transition. Although I have worked with the process information contained in the Continuous Process Improvement on the Web (CPIW) application in my role as a Technical Writer for the company, I see it as a whole new tool from this role. The process information, templates, forms, etc., contained within the AIS CPIW have not only provided a wealth of information to me, but I can truly see how these documented processes, standards, guidelines, and policies, when used properly, can result in significant cost savings to the company and their customers." A manager at AIS who started as a software engineer in 1990 shared his experience. "During my first two years at AIS, there had been many announcements in staff meetings about how we were going to be more competitive, deliver quality products, and make deliveries on time (through CASE tools, C++, OO, etc.). So far I had participated on two projects that had disappointed key customers by their lateness and problems. In January 1992, our AIS Chief Executive Officer (CEO) went to a conference on software process improvement. When he returned, he immediately called a staff meeting, and gave each employee a copy of Managing the Software Process. We were all asked to read the first six chapters. Our "Continuous Process Improvement" initiative was born and became the foundation for all other software development initiatives at AIS. Using the principles of process improvement, we did a better job - not by trying harder or working fasterbut by continuously learning from our past experiences and applying this to future work." Other comments from software engineers include: "Everything is under control if we follow the process." "Software process improvement helped me realize that I had to spend more time in design, and thus I reduced the time spent in test." [Title Page] [Abstract] [Preface] [Chapter 1] [Chapter 2] [Chapter 3] [Chapter 4] [Appendix A] [References] [DTIC page] [PDF file] The Software Engineering Institute (SEI) is a federally funded research and development center sponsored by the U.S. Department of Defense and operated by Carnegie Mellon University. Copyright 2001 by Carnegie Mellon UniversityURL: http://www.sei.cmu.edu/publications/documents/99.reports/99tr027/99tr027chap03.html Last Modified: 21 March 2001
Software Process Improvement Works!
Appendix A The AIS Approach to the Balanced Scorecard AIS has adapted the Balanced Scorecard approach for its organizations objectives and needs. There are cause-and-effect relationships throughout the Balanced Scorecard. When "Engineers use the Personal Software Process," which is a Learning and Growth performance driver, that also enables "Components with target percent of defects removed before compile and test." This leads to "Components, modules, programs with zero integration test defects" to meet the Internal Business Process strategic objective of "Engineers achieve the highest possible quality in the design, code phases of a component, module or program." Satisfying that objective drives "Defect-free deliveries" and "Customer responses indicating value achieved" to satisfy the strategic objective from the Customer perspective to "Consistently meet or exceed customer expectations for defect-free and on-time delivery." The Balanced Scorecard approach ensures that CPI benefits will continue in the future by demonstrating management commitment to CPI, formulating strategy based on CPI, communicating the strategy to the entire organization, and aligning employee career goals with organization goals. The Balanced Scorecard demonstrates the company managements commitment to continuous process improvement. This method communicates strategy to the entire organization, links individual accountabilities to strategic objectives, and provides a method against which to measure progress. The following matrix illustrates how this approach is used by AIS as a guideline. Strategic Objectives Strategic Measurements Core Outcomes Strategic Measurements Performance Drivers
Financial Consistently meet or exceed shareholder expectations for - revenue growth - profitability - return on investment Employee target ratio of gross revenue to base salary Projects profitability target Increase in shareholder equity Designated expenses target reduction in expense to revenue ratio
Customer Consistently meet or exceed customer expectations for - defect free and on-time delivery - value for products and services - achieving time-to-market goals Customer responses indicating value "achieved" Statements of Work lost due to not meeting customer time to market goals Defect-free deliveries On-time or ahead-of-schedule deliveries
Employee Consistently meet or exceed employee expectations for - training - compensation - communication - work environment - performance management - career development Employee responses and assessment indicating P-CMM Repeatable level key process areas fully satisfied Disciplined, repeatable, and stable work force practices documented
Internal Business Process Projects achieve predictable results for effort, schedule, and defects within known range of AIS organization defined process capability Engineers achieve the highest possible quality in the design, code phases of a component, module or program AIS organization defined process is continuously improved Projects with actual effort and schedule less than committed effort and schedule Components, modules, programs with zero integration test defects New products or product enhancements with documented quality better than its predecessor Projects planned and managed according to their defined process which is an approved tailoring of the AIS organization defined process Components with target percent of defects removed before compile and test Process Improvement Proposals submitted and implemented
Learning and Growth Investment in people, process, and technology enables achievement of customer, employee, and shareholder satisfaction goals Engineers achieving training goals Engineers align their career goals with company goals Engineers improve productivity continuously Engineers acquire new skills Engineers achieving career plans Engineers use the Personal Software Process
[Title Page] [Abstract] [Preface] [Chapter 1] [Chapter 2] [Chapter 3] [Chapter 4] [Appendix A] [References] [DTIC page] [PDF file] The Software Engineering Institute (SEI) is a federally funded research and development center sponsored by the U.S. Department of Defense and operated by Carnegie Mellon University. Copyright 2001 by Carnegie Mellon UniversityURL: http://www.sei.cmu.edu/publications/documents/99.reports/99tr027/99tr027app-a.html Last Modified: 21 March 2001