United States General Accounting Office GAO Report to Congressional Committees May 2003 DEFENSE ACQUISITIONS Assessments of Major Weapon Programs GAO-03-476 a May 2003 DEFENSE ACQUISITIONS Assessments of Major Weapon Programs Highlights of GAO-03-476, a report to Congressional Committees The weapons the Department of GAO assessed 26 defense programs ranging from the Marine Corps’ Defense (DOD) develops have no Advanced Amphibious Assault Vehicle to the Missile Defense Agency’s rival in superiority. How they are Theater High Altitude Area Defense system. GAO’s assessments are developed can be improved, anchored in a knowledge-based approach to product development that without sacrificing the superiority reflects best practices of successful programs. This approach centers on of the outcome. GAO’s reviews over the past 20 years have found attaining high levels of knowledge in three elements of a new product or consistent problems with weapon weapon—technology, design, and production. If a program is not attaining investments—cost increases, this level of knowledge, it incurs increased risk of technical problems, schedule delays and performance accompanied by cost and schedule growth (see figure). If a program is shortfalls—along with underlying falling short in one element, like technology maturity, it is harder to attain causes, such as pressure on knowledge in succeeding elements. managers to promise more than they can deliver. The best practices of successful product developments offer a knowledge- based approach DOD can use to improve the way it develops new weapons. This report is new for GAO, and draws on its work in best practices for product development. GAO’s goal for this report is to provide congressional and DOD decision makers with an independent, knowledge-based assessment of defense programs that identifies potential risks, and offers an opportunity for action when a program’s projected attainment of knowledge diverges from the best practice. It can also highlight those programs that employ practices All of the programs GAO assessed proceeded with less knowledge at critical worthy of emulation by other junctures than suggested by best practices, although several came close to programs. GAO plans to update and meeting best practice standards. GAO also found that programs generally issue this report annually to the did not track statistical process control data, a key indicator for production congressional defense committees. maturity. Program stakeholders can use these assessments to recognize the gaps in knowledge early and to take advantage of opportunities for constructive intervention—such as adjustments to schedule, trade-offs in requirements, and additional funding. GAO makes no recommendations. Program office comments are included in the assessments of GAO has summarized the results of its assessments in an easy to read two- each individual program. page format. Each two-page assessment contains a profile of the product www.gao.gov/cgi-bin/getrpt?GAO-03-476. that includes a description; a timeline of development; a baseline comparison of cost, schedule, and quantity changes to the program; and a To view the full report, including the scope graphical and narrative depiction of how the product development and methodology, click on the link above. For more information, contact Paul Francis at knowledge of an individual program compared to best practices. Each (202) 512-4841 or firstname.lastname@example.org. program office submitted comments and they are included with each individual assessment as appropriate. Contents Foreword iii Letter 1 A Knowledge-Based Approach Can Lead to Better Acquisition Outcomes 2 Observations 5 Agency Comments 7 Scope of Our Review 7 Appendixes Appendix I: Assessments of Individual Programs 9 Introduction 9 Advanced Amphibious Assault Vehicle (AAAV) 15 Airborne Laser (ABL) 17 Advanced Extremely High Frequency (AEHF) Communications Satellite 19 AN/APG-79 Active Electronically Scanned Array (AESA) Radar 21 AIM-9X Short-Range Air-to-Air Missile 23 Advanced Threat Infrared Countermeasures/Common Missile Warning System (ATIRCM/CMWS) 25 Advanced Wideband Satellite (AWS) 27 Cooperative Engagement Capability (CEC) 29 CH-47F Improved Cargo Helicopter 31 RAH-66 Comanche 33 EX-171 Extended Range Guided Munition (ERGM) 35 Excalibur Artillery Round 37 F/A-18E/F Super Hornet 39 F/A-22 Raptor 41 Joint Air-to-Surface Standoff Missile (JASSM) 43 Joint Common Missile 45 Joint Primary Aircraft Training System (JPATS) 47 F-35 Joint Strike Fighter (JSF) 49 Joint Standoff Weapon (JSOW) 51 National Polar-orbiting Operational Environmental Satellite System (NPOESS) 53 Page i GAO-03-476 Acquisition Trends and Risks Contents Patriot Advanced Capability 3 (PAC-3) Program 55 Space Based Infrared System (SBIRS) High 57 Theater High Altitude Area Defense (THAAD) 59 Tactical Tomahawk Missile 61 V-22 Osprey 63 Wideband Gapfiller Satellite (WGS) Communications System 65 Appendix II: Methodology 67 System Profile Assessment 67 Product Knowledge Assessment 68 Appendix III: GAO Contact and Acknowledgments 71 Figures Figure 1: Knowledge Build at Key Points in Product Development Reduces the Risk of Unknowns 4 Figure 2: Graphic Depiction of Best Practices for Technology, Design, and Production Knowledge 10 Figure 3: Depiction of a Notional Weapon System Program’s Knowledge as Compared with Best Practices 12 Abbreviations DOD Department of Defense GAO General Accounting Office USAF United States Air Force USN United States Navy USMC United States Marine Corps This is a work of the U.S. Government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. It may contain copyrighted graphics, images or other materials. Permission from the copyright holder may be necessary should you wish to reproduce copyrighted materials separately from GAO’s product. Page ii GAO-03-476 Acquisition Trends and Risks A United States General Accounting Office Washington, D.C. 20548 Comptroller General of the United States May 15, 2003 Foreword Congressional Committees Recent military operations in Iraq have soundly demonstrated the superiority of United States military capabilities. The Department of Defense (DOD) develops weaponry that is unmatched in levels of technological sophistication and lethality. Despite their superiority, weapon systems routinely take much longer to field, cost more to buy, and require more support than investment plans provide for. In a constrained funding environment, unforeseen cost growth in weapon systems forecloses other investment choices for the government, both within and outside of DOD. DOD’s investment in major weapon systems is expected to grow considerably in the future as DOD works to keep legacy systems while investing in future capabilities such as unmanned aircraft, satellite networks, and information communication systems. For example, the investment in weapons from fiscal years 2003 through 2009 will exceed $1 trillion. Such an investment clearly requires DOD to be as efficient and effective as possible in the development and acquisition of weapon systems. In the last several years, we have undertaken a body of work that examines weapon acquisition issues from a different, more cross-cutting perspective—one that draws lessons learned from best product development practices to see if they apply to weapon system development. We found that programs managed with a knowledge-based approach— where product knowledge is demonstrated at critical points in a development cycle—place themselves on a low-risk path to production. These programs are more likely to be executed within cost and schedule estimates. We believe that by employing this approach, DOD can still field superior weapons without attendant cost and schedule growth. This report is a new product for GAO. It provides decision makers with a snapshot of program performance and risk and is focused on each system’s developmental progress vis a vis best practices. Each assessment is summarized in an easy to read, visually descriptive 2-page format that provides a fact-based analysis of each program’s cost, schedule, and development status. We plan to issue this report annually in early spring, and we intend to increase the number of systems reviewed each year. We have briefed numerous committee staff on the product and received positive feedback regarding the report’s utility and breath of coverage. Page iii GAO-03-476 Acquisition Trends and Risks The continuing war on terrorism, regional instability, the challenge of transforming the military, as well as the federal government’s short-and long-term budget pressures have created a challenging environment for DOD. It faces a number of difficult missions that will put its strategies and resources under enormous strain. Consequently, it is important that weapon systems be acquired using a knowledge-based approach to ensure that their development is within cost, schedule, and performance parameters. We believe that this report can provide useful insights on key risks in development, allow decision makers to take corrective actions, and thereby place programs in a better position to succeed. David M. Walker Comptroller General of the United States Page iv GAO-03-476 Acquisition Trends and Risks A United States General Accounting Office Washington, D.C. 20548 May 15, 2003 Leter Congressional Committees The Department of Defense (DOD) is on the threshold of several major investments in programs that are likely to dominate budget and doctrinal debates well into the next decade. These programs include, among others, the Missile Defense Agency’s suite of land, sea, air, and space defense systems; the Army’s Future Combat System; and the Air Force and Navy’s Joint Strike Fighter. In fiscal year 2003, the Congress appropriated $127 billion to DOD for the research, development, and procurement of weapon systems. Funding for weapon systems is projected to continue growing to $182 billion in fiscal year 2009—an increase of over 43 percent. In total, the investment in weapons from fiscal years 2003 through 2009 will exceed $1 trillion. Thus, it is essential that sound foundations for these and other weapon system investments be laid now so that the resulting programs can be executed within estimates of available resources. The challenge of putting new programs on a better footing than their predecessors is a daunting one. Clearly, the acquisition process produces superior weapons. But it does so at a high price. Weapon systems routinely take much longer to field, cost more to buy, and require more support than investment plans provide for. These consequences reduce the buying power of the defense dollar, delay capabilities for the warfighter, and force unplanned—and possibly unnecessary—trade-offs among programs. DOD has undertaken a number of acquisition reforms over the preceding two decades in response to those problems, but while there have been individual successes, these reforms have not yet yielded consistent improvements in program outcomes. More recently, DOD leadership has embraced an evolutionary acquisition approach, coupled with time-phased requirements. This approach supports developing weapons in smaller, more predictable iterations of increasing capabilities, rather than the past approach of attempting to achieve a weapon’s maximum capability in one design leap. DOD is also striving to give programs, such as missile defense, more flexibility to make trade-offs between cost, schedule, and performance that can lead to better investment decisions. It is also currently looking at how to revise its planning, programming, and budgeting process that has been in place for over 40 years. Key to any effort to improve weapon system outcomes is using the lessons that can be learned from the best practices of successful commercial and Page 1 GAO-03-476 Acquisition Trends and Risks defense product development programs. We have found that these practices can be collectively described as a knowledge-based approach whose success depends on the timely attainment and use of a product’s technology, design, and production maturity. In this report, we compare the knowledge gained on 26 DOD weapon system programs with best practices. Our objective is to provide decision makers a means to quickly gauge the progress and potential risks—based on demonstrated knowledge—of the individual weapon system programs. A Knowledge-Based All product development efforts, whether for a car, a plane, a missile, or a satellite, go through a process of building knowledge. Ultimately, this Approach Can Lead to process brings together and integrates all of the technologies, components, Better Acquisition and subsystems needed for the product to work and to be reliably manufactured. The product development process can be characterized as Outcomes the reduction of risk and the resolution of unknowns through the attainment of knowledge. About 7 years ago, at the request of the Senate Committee on Armed Services, we began an extensive body of work identifying best practices in product development, both in DOD and in the commercial sector. Of particular interest were cases in which increasingly sophisticated products were being developed in significantly less time and at lower cost than their predecessors. A major reason for these successes was the use of a product development process that was anchored in knowledge. Product developers employed specific practices to ensure that a high level of knowledge regarding critical facets of the product was achieved at key junctures in development. We have characterized these junctures as three knowledge points. We have also identified key indicators that can be used to assess the attainment of each knowledge point. When tied to major events on a program’s schedule, they can disclose whether gaps or shortfalls exist in demonstrated knowledge, which can presage future cost, schedule, and performance problems. These knowledge points and associated indicators are defined as follows. • Knowledge point 1: Resources and needs are matched. This level of knowledge is attained when a match is made between a customer’s needs and the developer’s technical, financial, and other resources. Technology maturity is a particularly important indicator of resource availability. A best practice is to achieve a high level of technology maturity at the start of product development. This means that the Page 2 GAO-03-476 Acquisition Trends and Risks technologies needed to meet essential product requirements have been demonstrated to work in their intended environment. • Knowledge point 2: The product design is stable. This level of knowledge is attained when the product’s design demonstrates its ability to meet the customer’s requirements. A best practice is to achieve design stability at the system-level critical design review, usually held midway through development. Completion of engineering drawings at the system design review provides tangible evidence that the design is stable. • Knowledge point 3: Production processes are mature. This level of knowledge is attained when it is demonstrated that the product can be manufactured within cost, schedule and quality targets. A best practice is to achieve production maturity at the start of production. This means that all key manufacturing processes produce output within statistically acceptable limits for quality. As illustrated in figure 1, the process is building block in nature as the attainment of each successive knowledge point builds on the proceeding one. While the knowledge itself builds continuously without clear lines of demarcation, the attainment of knowledge points is sequential. In other words, production maturity cannot be attained if the design is not mature, and design maturity cannot be attained if the key technologies are not mature. Page 3 GAO-03-476 Acquisition Trends and Risks Figure 1: Knowledge Build at Key Points in Product Development Reduces the Risk of Unknowns For the most part, all three knowledge points are eventually attained on a completed product. The difference between highly successful product developments—those that deliver superior products within cost and schedule projections—and problematic product developments is how this knowledge is built and how early in the development cycle each knowledge point is attained. When knowledge is built more slowly than these points suggest, less knowledge is on hand at key decisions or events, such as the decisions to start a development program, hold the critical design review, and start production. This invites greater cost, schedule, and performance risks because (1) problems are more likely to be discovered late in the process and will therefore be more difficult and costly to correct and (2) a variety of pressures encourage program managers to underestimate the difficulties. It is important to note that successful product developers treat technology development as a different and separate effort that precedes product development. This treatment of technology development is key to reaching the first knowledge point at the start of product development, as it is a prerequisite for capturing design and production knowledge early in product development. This approach to attaining knowledge puts program managers—and programs—in a better position to succeed. Page 4 GAO-03-476 Acquisition Trends and Risks Observations When programs proceed with less knowledge than suggested by best practices, cost, schedule, and performance problems often result. To varying degrees, all the programs we assessed proceeded with lower levels of knowledge at critical junctures and thus attained key elements of product knowledge later in development. In some programs, the consequences of proceeding with early knowledge deficits have already been felt. For example: • The F-22 Fighter began product development with key technologies immature—deferring knowledge point 1—and subsequently had only a quarter of the desired amount of engineering drawings completed at the critical design review—deferring knowledge point 2. The program has experienced substantial cost increases and schedule delays in the latter stages of development. • The Patriot Advanced Capability missile also reached knowledge points 1 and 2 later than best practices. The seeker technology did not demonstrate maturity until close to the production decision and the design remains unstable. Each seeker still needs to be reworked about 3 times on average before it passes quality inspections. The cost of the seeker has increased by 76 percent and contributed to a 2-year delay in the program’s schedule. • The Extended Range Guided Munition program began with only one of its 20 critical technologies mature—deferring knowledge point 1. While progress has been made, program officials do not expect to achieve maturity on all technologies until after the design review. The lack of mature technologies contributed to subsequent test failures, cost increases, and schedule delays. If programs attain more knowledge as suggested by best practices, they are in a better position to succeed in meeting cost, schedule, and performance expectations. We found some programs that did attain key product knowledge earlier than most. For example: • The National Polar-orbiting Operational Environmental Satellite System program ensured that its pacing technologies were demonstrated before committing to product development. The program plans to demonstrate three critical sensors on a demonstrator satellite prior to their inclusion on the new satellite. Page 5 GAO-03-476 Acquisition Trends and Risks • The Theater High Altitude Area Defense System made significant strides in product development, following a problematic preliminary development phase. In 2000, we reported that the program’s delayed demonstration of technologies and components and reliance on full- system testing to discover problems, was a very costly method to mature the system’s design and nearly caused the cancellation of the program. The program has since structured a product development phase that places a much greater emphasis on early demonstration of components, a testing program that incorporates sufficient time between tests for learning, and a plan to achieve design stability by releasing 90 percent of engineering drawings by the time of the critical design review—knowledge point 2. In general, we found that the greatest absence of knowledge was in the area of production. Almost no programs collected statistical process control data, the indicator for production maturity. Unlike technology readiness levels, which can be applied at any time, and engineering drawing release data, which is captured on all programs, few programs collected statistical process control data. While the absence of this data does not necessarily mean that production processes were immature, attained knowledge could not be assessed against an objective standard. Other indicators of production maturity, such as scrap and rework rates, can indicate positive trends, but are not prospective—that is, they are not useful in guiding preparations for production. To some extent, statistical process control data is not being collected because DOD has been delegating more responsibility to prime contractors and reducing the amount of data requested. The lack of such data may put program offices in a disadvantaged position to gain insights about a contractor’s production progress. We have recently issued a report that recommends that DOD collect statistical process control data on its weapon system programs and DOD has agreed with this recommendation.1 We conducted our review from September 2002 through May 2003 in accordance with generally accepted government auditing standards. 1 U.S. General Accounting Office, Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: July 15, 2002). Page 6 GAO-03-476 Acquisition Trends and Risks Agency Comments DOD did not provide general comments on a draft of this report, but did provide technical comments on individual assessments. These comments, along with program office comments, are included with each individual assessment as appropriate. Scope of Our Review We selected programs for the assessments based on several factors, including (1) high dollar value, (2) stage in acquisition, and (3) congressional interest. The majority of the 26 programs covered in this report are considered major defense acquisition programs by DOD. A program is defined as major if its estimated research and development costs exceed $365 million or its procurement exceeds $2.19 billion in fiscal year 2000 constant dollars. We plan to include more programs in subsequent years, with a greater focus on programs early enough in development that the assessments can be used to improve the program’s prospects for success, and issue this report annually to the congressional defense committees. The individual assessment of each program can be found in appendix I. Appendix II contains detailed information on our methodology. We are sending copies of this report to interested congressional committees; the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force; and the Director, Office of Management and Budget. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you have any questions on this report, please contact me at (202) 512-4841 or Paul Francis at (202) 512-2811. Major contributors to this report are listed in appendix III. Jack L. Brock, Managing Director Acquisition and Sourcing Management Page 7 GAO-03-476 Acquisition Trends and Risks List of Congressional Committees The Honorable John W. Warner Chairman The Honorable Carl Levin Ranking Member Committee on Armed Services United States Senate The Honorable Ted Stevens Chairman The Honorable Daniel K. Inouye Ranking Member Subcommittee on Defense Committee on Appropriations United States Senate The Honorable Duncan Hunter Chairman The Honorable Ike Skelton Ranking Minority Member Committee on Armed Services House of Representatives The Honorable Jerry Lewis Chairman The Honorable John P. Murtha Ranking Minority Member Subcommittee on Defense Committee on Appropriations House of Representatives Page 8 GAO-03-476 Acquisition Trends and Risks Appendix I Assessments of Individual Programs AA ppp ep ned n x id e x Iis Introduction For the 26 programs, each assessment provides the historical and current program status and offers the opportunity to take early corrective action when a program’s projected attainment of knowledge diverges significantly from the best practices. The assessments also identify programs that are employing practices worthy of emulation by other programs. If a program is attaining the desired levels of knowledge, it has less risk—but not zero risk—of future problems. Likewise, if a program shows a gap between demonstrated knowledge and best practices, it indicates an increased risk—not a guarantee—of future problems. The real value of the assessments is recognizing gaps early, which provides opportunities for constructive intervention—such as adjustments to schedule, trade-offs in requirements, and additional funding—before cost and schedule consequences mount. Our assessment of each program is summarized in two components—(1) a system profile and (2) a product knowledge assessment. The system profile presents a general description of the product in development; a picture of the product or a key element of the product; a schedule timeline identifying key dates in the program; a table identifying the prime contractor; the program office location, and the fiscal year 2004 requested funding if available; and a table summarizing the cost, schedule and quantity changes to the program. The rest of the assessment analyzes the extent to which product knowledge at the three key knowledge points has been attained. We depict the extent of knowledge in a stacked bar graph and provide a narrative summary at the bottom of the first page. The second page is devoted to a narrative assessment of technology, design and production maturity, as well as other program issues identified and comments from the program office. The product knowledge figure is based on the three knowledge points and the key indicators for the attainment of knowledge. A “best practice” line is drawn based on the ideal attainment of the three types of knowledge at the three knowledge points (see fig. 2). Page 9 GAO-03-476 Acquisition Trends and Risks Appendix I Assessments of Individual Programs Figure 2: Graphic Depiction of Best Practices for Technology, Design, and Production Knowledge The first major point on the best practice line represents two facts: a commitment to a new product development has been made and the key technologies needed for the new product are mature. When all critical technologies have reached a technology readiness level 7, technology maturity—and thus knowledge point 1—has been attained. In our assessment, the technologies that have reached technology readiness level 7, a prototype demonstrated in an operational environment, are considered mature and those that reach technology readiness level 6, a prototype demonstrated in a relevant environment, are assessed as attaining 50 percent of the desired level of knowledge. Satellite technologies that achieved technology readiness level 6 were assessed as fully mature due to the difficulty of demonstrating maturity in an operational environment—space. (Technology readiness levels are more fully explained in appendix II.) The second major point on the best practice line captures technology maturity plus design maturity—knowledge point 2. A design is considered mature when 90 percent of the engineering drawings have been released or deemed releasable to manufacturing. In the Page 10 GAO-03-476 Acquisition Trends and Risks Appendix I Assessments of Individual Programs successful programs we have studied, design maturity is attained about halfway through the product development phase. The third major point on the best practice line captures the sum of technology maturity, design maturity, and production maturity. Production is considered mature when all key production processes are in statistical control. Ideally, this occurs before the first products for delivery to the customer are manufactured. As can be seen, knowledge about the technology, design, and production of a new product builds over time. While the knowledge itself builds continuously without clear lines of demarcation, the attainment of knowledge points is sequential. In other words, production maturity cannot be attained if the design is not mature, and design maturity cannot be attained if the key technologies are not mature. Data for a given weapon system program is then plotted against the best practices line. In the assessments that follow, a brown bar indicates the technology knowledge attained by a weapon system program. The actual technology readiness levels attained for a program’s key technologies are measured at the start of development—normally milestone II or milestone B in the Department of Defense’s (DOD) acquisition process. The closer a program’s attained knowledge is to the best practice line, the more likely the weapon will be delivered within its estimated cost and schedule. A knowledge deficit at this point—indicated by a gap between the technology knowledge attained by the weapon system and the best practices line— means the program proceeded with immature technologies and may face a greater likelihood of cost and schedule increases as technology risks are discovered and resolved. A green bar indicates the design knowledge attained by a weapon system program. This is calculated by measuring the percent of engineering drawings released to manufacturing. The green bar is stacked on top of the brown bar to indicate whether any cumulative gap—considering both technology and design—exists at the halfway point of product development. A blue bar indicates the production knowledge attained by a weapon system program. This is calculated by measuring the percentage of key production processes in statistical control. The blue bar is stacked on top of the brown and green bars to indicate whether any cumulative technology, design, and production gaps exist at the time production begins. In some cases, we obtained projections from the program office of future knowledge attainment. These projections are depicted as dashed bars. Figure 3 depicts an example of an assessment for a notional weapon system. Page 11 GAO-03-476 Acquisition Trends and Risks Appendix I Assessments of Individual Programs Figure 3: Depiction of a Notional Weapon System Program’s Knowledge as Compared with Best Practices An interpretation of this notional example would be that the product development began with key technologies immature, thereby missing knowledge point 1. Knowledge point 2 was not attained at the design review as some technologies were still not mature and only a small percentage of engineering drawings had been released. Projections for the production decision show that the program is expected to achieve a greater level of maturity, but will still fall short. It is likely that this program would have had significant cost and schedule increases. We also found three situations in which programs were unable to provide key knowledge indicators. We used three types of labels in the knowledge figures to depict those situations. Programs with these labels are distinguished from those that have elected not to collect data that can be used to assess progress against best practices. First, a few programs are planning to collect the relevant knowledge indicator, but they have not yet begun collecting it. In these situations, we annotate the graph with the phrase “Data unavailable.” Second, a few programs have not followed the traditional acquisition model. For example, one program combined the Page 12 GAO-03-476 Acquisition Trends and Risks Appendix I Assessments of Individual Programs development start decision with the production decision. Another program used commercial off-the-shelf components, which negated the need to monitor production processes. In these situations, we annotate the graph with the phrase “Not applicable.” Finally, some programs were unable to provide or reconstruct the relevant knowledge indicator because the event happened too many years ago. In these situations, we annotate the graph with the phrase “Not assessed.” Our assessments of the 26 systems follow. Page 13 GAO-03-476 Acquisition Trends and Risks Appendix I Assessments of Individual Programs Page 14 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: AAAV Advanced Amphibious Assault Vehicle (AAAV) The Marine Corps’ AAAV is designed to transport troops from ships to shore at higher speeds and from farther distances than the existing AAV-7. It is designed to be more mobile, lethal, reliable, and effective in all weather conditions. AAAV will have two variants—a troop carrier for 17 Marines and a command vehicle to manage combat operations in the field. Prime contractor: General Dynamics Program office: Woodbridge, Va. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $240.7 million 12/00 12/01 change Procurement $97.9 million Research & development cost $1,395.7 $1,616.4 15.8 Quantity 0 vehicles Procurement cost $6,256.7 $6,741.7 7.8 Total program cost $7,732.7 $8,440.9 9.2 Program unit cost $7.544 $8.235 9.2 Total quantities 1,025 1,025 0.0 Acquisition cycle time (months) 138 150 8.7 AAAV demonstrated most technology and design knowledge at critical junctures in the program. At the start of the program, all but one of the critical technologies were mature. The design was close to meeting best practice standards at the design review, signifying the design was stable. Early development of fully functional prototypes and other design practices facilitated design stability. However, late maturation of the remaining technology may lead to some redesign. Also, the demonstration of production maturity remains a concern because the program is currently uncertain about requiring the contractor to use statistical process controls to achieve quality objectives. The AAAV production decision is not scheduled until September 2005. Remaining efforts include developmental, operational, live fire, and reliability testing. GAO-03-476 Acquisition Trends and Risks Page 15 Common Name: AAAV Appendix I AAAV Program Other The Marine Corps has recently restructured the AAAV Technology Maturity program to add 12 additional months of testing before Four of the five critical technologies had the September 2005 production decision. This change demonstrated an acceptable level of maturity at the more than doubles the number of vehicle test months start of product development. The remaining previously planned. The change also moves the initial technology, moving map navigation, is not expected to operational capability date from September 2007 to achieve maturity until the spring of 2003. Program September 2008. The program estimates a $480 million officials stated that maturing this technology is increase in acquisition costs—$101 million for added contingent on developing and testing system testing, $75 million for development, and $304 million hardware. As a backup, program officials said they for recurring production. could carry out the AAAV mission using existing technology, but it would not provide full vehicle-to- Program Office Comments vehicle situational awareness. AAAV program officials concurred with our assessment. Design Maturity The AAAV design is essentially complete. However, late maturation of the new mapping system may lead to some redesign, if testing identifies any problems. At the critical design review, AAAV had completed 77 percent of the drawings—not up to the best practice standard of 90 percent, but higher than many DOD programs. Early engineering prototypes—fully integrated and functional—allowed the program to demonstrate that the design worked as required. These early prototypes have completed over 4,000 hours of testing that resulted in design improvements for subsequent prototypes. To complete development, program plans call for building and testing nine development prototypes and one live fire test vehicle. These prototypes will be production representative vehicles for developmental, operational, live fire and reliability testing. The first prototype is scheduled to be available by May 2003. Production Maturity Program officials are developing a production readiness plan to ensure vehicles will meet cost, schedule, and quality objectives. At this time, they are uncertain whether this plan will require the contractor to use statistical process controls, the best practice standard. As the prime contractor currently produces the nine developmental prototype vehicles, it is not tracking statistical process control data. Instead, it is using postproduction inspections, considered less efficient and effective than statistical process controls to achieve quality. Page 16 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: ABL Airborne Laser (ABL) The Missile Defense Agency’s ABL is designed to destroy enemy ballistic missiles almost immediately after their launch. The system, carried aboard a highly modified Boeing 747 aircraft, uses a high- energy chemical laser to rupture the skin of enemy missiles; a beam control/fire control subsystem to guide the laser beam through the aircraft, focus the beam on the target, and maintain the beam’s quality as it travels through the atmosphere; and a battle management subsystem to plan and execute the engagement. We assessed all components. Prime contractor: Boeing Program office: Albuquerque, NM FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $610.0 million 1/97 12/02 change Procurement $0 million Research & development cost $2,400.5 $4,415.7 84 Quantity 0 systems Procurement cost $3,170.6 NA NA Total program cost $5,571.1 NA NA Program unit cost $795.9 NA NA Total quantities 7 NA NA Acquisition cycle time (months) 118 NA NA Note: Latest costs only through FY 2007. Procurement funding, quantities, and the initial capability date have yet to be determined. NA = not applicable. Only one of ABL’s critical subsystems has demonstrated acceptable levels of maturity. The Missile Defense Agency is developing an initial ABL system to demonstrate technology critical to the system’s design and plans to begin development of a second improved demonstration aircraft in 2003. Either of these aircraft, or later improved configurations, could be given to the Air Force for operational testing and production if system-level tests show that any one of them is capable of destroying a threat missile at an operational range. Although the agency’s development strategy incorporates some knowledge-based practices, it is difficult to see how the discipline of a knowledge- based approach can be achieved when uncertainty exists about whether the effort is a technology development or a product development. GAO-03-476 Acquisition Trends and Risks Page 17 Common Name: ABL Appendix I ABL Program The Missile Defense Agency has made changes that are expected to improve its ability to evolve ABL’s Technology Maturity critical technologies, including adopting a flexible requirements setting process, providing additional Only one of ABL’s five critical subsystems, the time and facilities to develop and test these aircraft itself, represents mature technology. A second technologies, and attaining the knowledge to match subsystem, which directs the laser energy through the the warfighters’ needs with demonstrated technology. aircraft, consists of several technologies that have On the other hand, it is not clear whether the start of a been tested in a simulated environment. However, block represents a technology development or a three other subsystems consist of low-fidelity product development. This uncertainty may hamper prototype technologies that have only been tested in a the application of knowledge standards and forfeit the laboratory environment. They include the laser, the discipline necessary to ensure successful product battle management subsystem, and the ground development. support subsystem. Problems associated with maturing technology have Program Office Comments consistently been a source of cost and schedule In commenting on a draft of this assessment, program growth throughout the life of the program. DOD officials reemphasized their commitment to spiral analysts attribute this growth to the increased development and capabilities-based acquisition. They complexity of designing laser subsystems, substantial plan to use this strategy to improve the critical aspects increases in engineering analysis and design, and of the system by allowing the pace of technological greater than anticipated aircraft engineering development to dictate the introduction of improved complexity. capabilities into the system. They believe this strategy The program is managed under the Missile Defense is not inconsistent with knowledge-based acquisition. Agency’s new capabilities-based acquisition strategy. They also mentioned that laser power depends not This approach develops an operational system only on the number of laser modules but also on through a series of block upgrades. The agency plans module efficiency, optics, and pointing precision. They to use the first two blocks, block 2004 and block 2008, admit that the laser subsystem should be operated in to demonstrate critical technologies, but if tests show flight before any production decision is made. either configuration has any battlefield utility, that Program officials are conducting emergency configuration could be deployed in the event of an operational capability planning to support a possible emergency. emergency ABL deployment. This decision will be The 2004 configuration will have a 6-module laser, based on the potential threat and an assessment of the rather than the 14 modules planned for the production capabilities ABL may provide. system. The optical components can withstand the The program office indicated that all but one of the heat produced by a 6-module laser, but the agency battle management components have been tested in would have to redesign optical components for the an operational environment. This component is the system to withstand the heat associated with an active ranger system, which provides crucial angle increase in laser power. In addition, the 2004 measurements and range data for engaging ballistic configuration is far too heavy to allow the addition of missiles. laser modules that will likely be needed in an operational ABL system. To accommodate more modules, a weight reduction program has begun that includes redesigning many components and the increased use of composite materials. The program is considering whether to use a different aircraft configuration that would allow the system’s weight to be moved forward to relieve stress on the airframe. However, its use would require additional design changes. Page 18 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: AEHF Advanced Extremely High Frequency (AEHF) Communications Satellite The Air Force’s AEHF is a satellite system intended to replace the existing Milstar system with improved, survivable, jam-resistant, worldwide, secure communication capabilities at lower launch costs. First launch of an AEHF satellite is expected in 2006. The system also includes a mission control segment with service-specific terminals to process satellite information. DOD is negotiating international partner participation in the program. We assessed the satellites and mission control segments. Prime contractor: Lockheed Martin Program office: El Segundo, Calif. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $778.1 million 10/01 12/02 change Procurement $0 million Research & development cost $4,339.3 $4,608.4 6.2 Quantity 0 satellites Procurement cost $1,286.2 $509.1 -60.4 Total program cost $5,625.5 $5,117.5 -9.0 Program unit cost $1,125.1 $1,705.8 51.6 Total quantities 5 3 -40.0 Acquisition cycle time (months) 111 118 6.3 The AEHF satellite program demonstrated most technology knowledge at development start. Eleven of the 12 critical technologies were mature, according to best practice standards. The remaining technology is not projected to be mature prior to the critical design review, nor does it have a backup technology. However, some elements of this technology are mature. The program expects to complete 90 percent of its drawings by the critical design review. The manufacture of the communications and transmission security subsystem is a major challenge facing the program as upgrades are being added into the new cryptological equipment. If production of this subsystem slips, first launch could slip correspondingly as no backup exists. GAO-03-476 Acquisition Trends and Risks Page 19 Common Name: AEHF Appendix I AEHF Program security subsystem’s functions as it is integrated into the AEHF satellite's communications subsystem. Technology Maturity However, continued complications in fabrication could potentially place the entire program at cost, Eleven of the 12 critical technologies have reached schedule, and performance risk. maturity according to best practice standards. The program does not project achieving maturity on the Other Program Issues remaining technology—the phased array antenna—by the design review in June of 2004, nor does it have a In December 2002, the Deputy Secretary of Defense backup capability. However, some elements of this decided to change the acquisition strategy of AEHF technology have been demonstrated in an operational from a five-satellite program to a three-satellite environment. program. Under the revised strategy, full capability may no longer be satisfied by an AEHF-only Design Maturity constellation. The program has completed 150 or more of the 6,000 Program Office Comments total drawings for release to manufacturing. Program officials project completing 90 percent of drawings by In commenting on a draft of this assessment, program the system critical design review in June 2004. The officials stated that the program is executing very well program has completed key segment level preliminary since contract definitization in August 2002, with cost design reviews and is expected to complete all design and schedule variance at less than 1 percent. reviews by the second quarter of fiscal year 2004. Currently, at approximately 33 percent complete Program officials consider the design and toward first launch, the total program is on track and development of the satellite subcomponents low risk estimated to finish on time and on budget. The system because those components have been used on other preliminary design review has been completed. space systems. However, the integration of these Critical design reviews are on track for completion by subcomponents into a subsystem, such as the phased- Spring 2004. Funding cuts have, in the past, caused array antenna, has yet to be successfully schedule slips and cost increases. Given the focus on demonstrated at the AEHF satellite frequencies. the critical design review, the impacts of changing requirement will have increasing deleterious effects. Program officials assessed the software development The program remains focused on addressing critical for the mission control system as moderate risk and risks that threaten cost, schedule, and performance. have developed a risk mitigation strategy. This New system security requirements recently received strategy includes consulting with the National from the National Security Agency for the space, Software Engineering Institute and the Aerospace mission control, and terminal segments are being Corporation and conducting a software development evaluated. After aggressive risk management, the most capability evaluation. Also, the program office has likely impacts include additional testing, verification, incorporated spiral development and the use of and program documentation. The program has also software emulators so users and developers can see begun developing engineering models for all of the how the software will look and work. Until these critical subsystems. These efforts are on track and actions are completed, software may be at risk for proceeding well. unplanned cost and schedule growth. Production Maturity Any future problems with the fabrication of the communications and transmission security microprocessor, a component designed to limit access to satellite transmissions to authorized users, could delay the production schedule and the launch of the first satellite planned for December 2006. Program officials have started a number of risk reduction efforts, including a chip emulator whose purpose is to simulate the communications and transmission Page 20 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: AESA AN/APG-79 Active Electronically Scanned Array (AESA) Radar The Navy’s AESA radar is one of the top upgrades for the F/A-18E/F aircraft. It is to be the aircraft’s primary search/track and weapon control radar and is designed to correct deficiencies in the current radar. According to the Navy, the AESA radar is key to maintaining the Navy’s air-to-air fighting advantage and will improve the effectiveness of the air-to-ground weapons. When completed, the radar will be inserted in new production aircraft and retrofitted into existing aircraft. Prime contractor: Raytheon Program office: Patuxent River, Md. FY 2003 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $107.1 million 6/01 12/01 change Procurement $24.06 million Research & development cost $518.9 $494.2 -4.7 Quantity 0 radars Procurement cost NA NA NA Total program cost NA NA NA Program unit cost NA NA NA Total quantities NA NA NA Acquisition cycle time (months) 69 68 -1.4 Note: NA = not applicable. The AESA radar’s demonstrated knowledge is difficult to characterize. The fact that almost all of the engineering drawings have been completed suggests design stability. However, until the technologies are demonstrated, the potential for design changes remains. The AESA radar is also dependent on other programs that could pose significant risk to the radar’s cost, schedule, and technical performance. The technology and design risks are significant given that the AESA radar is only a few months from a production decision. The Navy is currently reassessing the radar’s technology maturity. Although many of the F/A-18E/F aircraft will be retrofitted with the AESA radar, full funding for the retrofitting has not been budgeted. If the radar is not ready for production as scheduled, more aircraft will have to be retrofitted. GAO-03-476 Acquisition Trends and Risks Page 21 Common Name: AESA Appendix I AESA Radar Program planned total buy of 548 F/A-18E/F aircraft will not receive the radar as they are being produced. Plans are Technology Maturity to retrofit the radar onto 136 aircraft at a projected cost of $3.14 million each. This cost does not include The AESA program’s four critical technologies were the cost of new APG-73 mechanical scanned radars not mature at the start of development in February that will be installed in the aircraft until AESA radars 2001, and they were not mature as of May 2002. The are available for retrofit. If delays occur in the AESA Navy is currently reassessing the maturity of these radar deliveries, retrofit costs will increase. technologies. At the time of its last assessment, two of the technologies had been tested using simulation and The AESA radar is projected to weigh about two had been tested in the laboratory. Program 270 pounds more than the current radar and will officials indicated that they have several options for require a more capable cooling system than the one dealing with immature technologies, including currently on the aircraft. The Navy expects some utilizing backup technologies. Initial flight tests of the minor degradation in aircraft performance, such as radar in an aircraft are scheduled for June 2003— slightly decreased range, as a result of the increased concurrent with the production decision. All four weight and new cooling system. technologies are not expected to be mature until The AESA program is linked to a number of other late 2004. corporate and Navy programs. For example, the radar will use a 32-port fiber channel fabric module Design Maturity developed by Boeing as a commercial venture. At the design review, 67 percent of the currently Technical difficulties with the module have caused projected total drawings were completed. In the schedule delays and may impact cost and performance period between June 2002 and December 2002, the of the radar. Also, Raytheon is developing some number of total expected drawings increased by 21 hardware and software for the radar with company percent. Program officials stated that the increase was funds or in coordination with other programs. due to new or modified drawings for systems Disruptions in these efforts could adversely impact the supporting the radar such as the radome, shield, and AESA program. aircraft airframe. Program officials indicated that they currently have 98 percent of the drawings complete; Program Office Comments however, the technology maturation process may lead The AESA program did not provide a general to more design changes. statement in response to our review but did provide Production Maturity technical comments that were incorporated where appropriate. We could not assess the AESA program’s production maturity against best practices, as statistical control data was not available. Other Program Issues Program officials estimate that the first low-rate production unit will exceed its cost target by 27 percent. Subcontractor development cost was considered to be the biggest contributor to this increase. The effects of the cost increase may be minimized in low-rate production lots 1-3 because of firm fixed price contract options. Program officials stated that cost reduction initiatives were underway to reduce the cost overruns by half by full-rate production. Delivery of the first production AESA radars for insertion into F/A-18E/F aircraft on the production line is scheduled for fiscal year 2005. As a result, 254 of the Page 22 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: AIM-9X AIM-9X Short-Range Air-to-Air Missile The AIM-9X is a follow-on version of the existing AIM-9M short-range missile for Air Force and Navy fighters. The AIM-9X is designed to be a highly maneuverable, launch-and-leave missile; capable of engaging targets using passive infrared guidance to provide full day/night operations and improved resistance to countermeasures and expanded target acquisition. The full capabilities of the AIM-9X will not be achieved without completing development of the helmet mounted cueing system—a separate development program that we did not assess. Prime contractor: Raytheon Missile System Company Program office: Patuxent River, Md. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $2.7 million 1/97 12/01 change Procurement $104.9 million Research & development cost $577.0 $594.6 3.0 Quantity 531 missiles Procurement cost $2,116.1 $2,055.1 -2.9 Total program cost $2,693.0 $2,649.6 -1.6 Program unit cost $0.268 $0.261 -2.5 Total quantities 10,049 10,142 0.9 Acquisition cycle time (months) 92 105 14.1 The AIM-9X program entered production in September 2000 without assuring that the manufacturing processes were mature. However, because the missile is a follow-on to the AIM-9M missile, program officials believe that they have significant production knowledge. The program ensured, prior to entering low-rate initial production, that the missile design was stable. The program did attain knowledge early in development by using proven technologies from predecessor systems and other programs, as well as testing numerous prototype versions of the missile. As a result, the program released the majority of its engineering drawings at the design review. GAO-03-476 Acquisition Trends and Risks Page 23 Common Name: AIM-9X Appendix I AIM-9X Program rework data. The AIM-9X acquisition cost and schedule history shows the program has been able to Technology Maturity meet its goals. All of the AIM-9X critical technologies are mature Program Office Comments because they have been demonstrated in developmental tests using actual hardware in realistic In commenting on a draft of this assessment, program conditions. Specifically, the program used prototypes officials acknowledged they did not contractually to test new technologies and existing missile require collection of statistical process control data on components that are being employed in a new critical manufacturing processes. Program officials operational environment. stated their strategy for demonstrating manufacturing process maturity includes building, testing, and Design Maturity evaluating production representative missiles; conducting multiple readiness reviews; utilizing low- The design of the AIM-9X is complete, and 100 percent rate initial production to test production processes; of the drawings have been released to manufacturing. and maturing production processes before full-rate The AIM-9X program built and tested 43 prototypes of production. various configurations during development to help mature the missile's design. Hardware and software performance was assessed at subsystem and system levels, and design changes were incorporated into the prototypes until a mature and stable missile configuration was demonstrated. The AIM-9X program held design reviews for the 11 subsystems between October 1997 and March 1998. The early design reviews, prototypes, and early testing allowed the program to achieve a stable design at the system design review in March 1998. At that time, the contractor had released 94 percent of its engineering drawings to manufacturing. Production Maturity The AIM-9X program does not contractually require collection of statistical process control data on critical manufacturing process, but it has undertaken an acquisition strategy to incentivize the contractor to reach cost and quality goals. However, the contractor and program officials believe that they have significant knowledge about producing the missile. The AIM-9X is a variant of the AIM-9M missile and uses components produced for other weapon systems, providing the program with significant production knowledge. In addition, to improve the production capabilities, the contractor built developmental units on production equipment. Program officials believe this practice has allowed them to mature the manufacturing processes. According to program officials, most of the critical processes on the AIM-9X are at the subcontractor level and a process exists to attain cost and quality goals. This is accomplished primarily by postproduction inspections to track production yield, scrap, and Page 24 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: ATIRCM/CMWS Advanced Threat Infrared Countermeasures/Common Missile Warning System The Army’s and Special Operations’ ATIRCM/CMWS is a component of the Suite of Integrated Infrared Countermeasures planned to defend U.S. aircraft from advanced infrared-guided missiles. The system will be employed on Army and Special Operations aircraft. ATIRCM/CMWS includes an active infrared jammer, a missile warning system, and a countermeasure dispenser capable of loading and employing expendables, such as flares, chaff, and smoke. Prime contractor: BAE Systems Program office: Huntsville, Ala. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $7.2 million 3/96 12/01 change Procurement $75.7 million Research & development cost $568.0 $540.1 -4.9 Quantity 2 units Procurement cost $2,338.9 $1,971.9 -15.7 Total program cost $2,906.9 $2,512.0 -13.6 Program unit cost $0.940 $2.330 148.0 Total quantities 3,094 1,078 -65.2 Acquisition cycle time (months) Classified Classified NA Note: NA = not available. The ATIRCM/CMWS is scheduled to enter production in May 2003 with no assurance that production processes are in control. The CMWS portion of the ATIRCM/CMWS program entered limited production in February 2002 to meet an urgent need. Full-rate production for ATIRCM/ CMWS was delayed because of reliability problems, which may indicate that production processes were not in control. These problems are, at least in part, a consequence of design proceeding with known shortfalls in knowledge: key technologies were demonstrated late in development and only a small number of design drawings were completed by design review. Resolving these knowledge shortfalls has led to cost and schedule increases. While the key technologies appear mature, reliability and producibility issues could necessitate design changes. GAO-03-476 Acquisition Trends and Risks Page 25 Common Name: ATIRCM/CMWS Appendix I ATIRCM/CMWS Program The program delayed the production decision for the combined system an additional year to the currently Technology Maturity scheduled May 2003 date primarily due to reliability issues. Reliability testing was halted because of The five critical technologies for the system are numerous failures with the ATIRCM subsystem. mature, but they did not mature until after the system Reliability failure can be an indicator of producibility design review. Most of the early technology and process control problems. The program plans to development effort was focused on the application to build and develop six additional subsystems during rotary wing aircraft. However, when product 2002 and 2003. The full-rate production decision for development began in 1995, the requirements were the complete system is now scheduled for 2005. expanded by Office of the Secretary of Defense direction to include Navy and Air Force fixed wing Other Program Issues aircraft. According to program officials, they did not fully anticipate the additional technology needed to The Army procured an initial 32 systems in fiscal year meet these much more demanding requirements. This 2002 that only included the CMWS. The Army plans to change caused problems that largely contributed to procure a total of 99 ATIRCM/CMWS systems to outfit cost increases of more than 150 percent to the special operations aircraft between fiscal year 2002 development contract. The Navy and the Air Force and 2009. subsequently dropped out of the program, rendering the extra effort needless. Program Office Comments In commenting on a draft of this assessment, program Design Maturity officials stated that the Army eliminated the program's The basic design of the system is complete, with funding for fiscal years 2002 and 2003. In fiscal year 100 percent of the drawings released to 2003, the Special Operations Command funded the manufacturing. However, reliability and producibility urgent procurement of 32 CMWSs. Subsequently, the issues could require design changes. The design was Army reinstated the program for fiscal years 2004– particularly immature at the critical design review, 2009. The program office stated that the loss of with only 22 percent of the drawings complete. A funding in fiscal year 2003 slowed the program major cause was that the technology requirements markedly. The program's acquisition strategy remains were not well understood until the system design to equip Special Operations forces before equipping review, leading to the discovery that a major redesign the remainder of the Army. was needed to meet requirements. It was not until The system was modified in 2002 to address ATIRCM 2 years after the design review that 90 percent of the reliability, producibility, and built-in-test issues. Six drawings were released and the design was ATIRCM systems are being manufactured and tested considered stable. According to program officials, the to demonstrate and verify the enhancements. ATIRCM immature design caused inefficient manufacturing, is scheduled to begin low-rate initial production in rework, and testing and contributed to the 3-year May 2003, and CMWS is scheduled to begin low-rate schedule delay. initial production in January 2004. The program office stated that low-rate production is required to maintain Production Maturity a production base. The system's operational testing is The ATIRCM/CMWS program does not collect planned for March 2005. According to the program statistical control data on its critical manufacturing office, the prime contractor indicated that statistical processes. Program officials have identified the process control is not within its corporate philosophy, absence of statistical process control data as a particularly for a program with such low production weakness and believe it should be instituted. Despite rates and quantities. this shortfall in knowledge, the Army entered limited CMWS subsystem production in February 2002 to meet an urgent need of the U.S. Special Operations Command. Page 26 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: AWS Advanced Wideband Satellite (AWS) The Advanced Wideband Satellite system is designed to provide improved, survivable, jam- resistant, worldwide, secure and general purpose communications to support the National Aeronautics and Space Administration, DOD and the intelligence community. It will replace the current Milstar satellite system and supplement the AEHF satellite system, reviewed elsewhere in this report. It will be the cornerstone of a DOD architecture that includes the multiple satellite systems. Prime contractor: In competition Program office: El Segundo, Calif. FY 2004 funding request: FY 2003 dollars in millions President’s budget Latest Percent R&D $439.3 million 02/03 00/00 change Procurement $0.0 million Research & development cost $5,814.5 $0.0 0.0 Quantity 0 satellites Procurement cost $2,343.9 $0.0 0.0 Total program cost $8,158.4 $0.0 0.0 Program unit cost $2,039.6 $0.0 0.0 Total quantities within budget 4 0 0 Acquisition cycle time (months) 75 0 0 Note: Program costs and schedule have yet to be formally approved by DOD. Costs only through FY 2009. The AWS/TSAT program is scheduled to enter product development with only one of its five critical technologies mature, according to best practices. The initial product development period will likely require concurrent technology and product development activities to maintain schedule. Although the new draft space acquisition guidance allows this approach, it is contrary to the best practice of separating technology development from product development. GAO-03-476 Acquisition Trends and Risks Page 27 Common Name: AWS Appendix I AWS/TSAT Program Technology Maturity Of the five AWS/TSAT key space segment technologies, one is mature while the other four are scheduled to reach maturity by January 2006, more than 2 years after development start. Three of the four immature technologies have a backup technology available in case of development difficulties. However, use of these technologies would degrade system overall performance. The Single Access Laser Communications technology has no backup and, according to program officials, any delay in maturing this technology would result in a slip in the expected launch date. Other Program Issues The program plans a development cycle that is, according to DOD documentation, aggressive. The satellite development cycle is planned to be 75 months: 27 months for technology development; 15 months for product development; and 33 months for satellite build, test and launch. This period of time is substantially shorter than the development cycle for the AEHF satellite (118 months vs. 75 months), though the AWS/TSAT system is expected to provide a transformational leap in satellite communications capability. The program is managed under the new National Security Space Acquisition process, which makes no clear distinction between the end of technology development and the start of product development. Therefore, the AWS/TSAT acquisition strategy may allow the system's technology development and product development to be conducted concurrently prior to the production decision. DOD’s acquisition system policy states that one of the entrance criteria for the system development and demonstration phase is technology maturity. The AWS/TSAT acquisition strategy does not ensure that technology maturity will be achieved prior to the start of product development consistent with best practices. Program Office Comments In commenting on a draft of this assessment, program officials stated that the National Security Space Acquisition Policy was developed to streamline the decision-making framework and to tailor it for space systems, in order to more efficiently field systems that incorporate rapidly changing technology advances. Page 28 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: CEC Cooperative Engagement Capability (CEC) The Navy’s CEC is designed to connect radar systems to enhance detection and engagement of air targets. Ships and planes equipped with their version of CEC hardware and software will share real-time data to create composite radar tracks, essentially allowing the battle group to see the same radar picture. A CEC-equipped ship will then be able to detect and launch missiles against targets its radar cannot see. We assessed block 1 of the CEC. The Navy is developing a more advanced block 2 CEC. Prime contractor: Raytheon Systems Corporation Program office: Washington, D.C. FY 2004 funding request FY 2003 dollars in millions Approved Latest Percent R&D $72.5 million 5/95 4/02 change Procurement $128.6 million Research & development cost $1,154.0 $2,052.8 77.9 Quantity 21 systems Procurement cost $1,292.5 $2,127.6 64.6 Total program cost $2,493.9 $4,180.4 67.6 Program unit cost $13.6 $15.4 12.8 Total quantities 183 272 48.6 Acquisition cycle time (months) 16 16 0.0 The technologies and design of the CEC program block 1 are fully mature. The program’s production maturity could not be assessed. The program lacks the necessary data primarily because the government does not collect it on the commercially available portions. However, program and contractor officials consider the processes to be capable of producing a quality product on time and on cost. Block 1 of the CEC program was approved in April 2002 for full-rate production for the shipboard version and continued low-rate initial production for the airborne version. GAO-03-476 Acquisition Trends and Risks Page 29 Common Name: CEC Appendix I CEC Program Some solutions for interoperability and integration issues will also be assessed in follow-on testing. Technology Maturity However, many of these issues are expected to be resolved through the introduction of block 2. The plan In January 2002, the Office of Naval Research assessed was approved in April 2002. Block 2 is expected to CEC’s six critical technologies. Five of the provide cost, performance, and functional technologies assessed as mature were incorporated improvements over the current system, though its into the shipboard version when it successfully details are yet to be defined. Among the anticipated completed the operational evaluation in May 2001. The characteristics of block 2 is interoperability with sixth technology, a data processor, was not assessed legacy combat systems. as part of the operational evaluation but was determined to be mature. Program Office Comments Design Maturity In commenting on a draft of this assessment, program officials stated that a production readiness review CEC’s basic design appears complete, as all of the conducted in October 2001 found CEC production to drawings needed to build the shipboard version have be mature. They evaluated all areas of production, been released to manufacturing. including quality, configuration management, CEC program officials noted that new drawings processes and procedures, drawings, and testing. They continue to be released. They explained that as stated that the contractor is delivering systems on commercially available technologies, which comprise schedule and within cost. To date, 29 systems over 5 approximately 60 percent of CEC’s hardware, become years have been successfully delivered, installed, more advanced, portions of the system will need to be tested, and many have been deployed. Following and redesigned to incorporate those advances. operational testing and evaluation, the Navy found CEC to be operationally suitable and effective and the Production Maturity DOD Director for Operational Test and Evaluation found CEC demonstrated the highest reliability of any We could not assess the CEC program’s production system tested so far of comparable complexity. maturity against the best practice as data were not According to program officials, CEC’s use of available. According to program officials, the commercial off-the-shelf components enables the noncommercially available portions of CEC do not program to select mature cost-effective components involve any critical manufacturing processes. Officials from industry, instead of manufacturing them in- indicated that they do not have insight into house. In recognition of the above, DOD approved the manufacturing processes for the commercially program for full-rate production in April 2002. available portions, including whether these processes are critical and whether the contractor has them under statistical control. The program officials and the contractor are confident that a quality product can be delivered on time and within cost based on the contractor’s adherence to industry standards and past performance on the low- rate initial production contracts for the shipboard version. Other Program Issues Battle group-level interoperability, integration, and built-in-test false alarm rates were identified as areas needing improvement following the operational evaluation. Program officials expect a solution for the alarm rates to be in place for a follow-on operational test and evaluation planned for 2004. Page 30 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: CH-47F CH-47F Improved Cargo Helicopter The Army’s CH-47F heavy lift helicopter is intended to provide transportation for tactical vehicles, artillery, engineer equipment, personnel, and logistical support equipment. It is expected to operate in both day and night. The purpose of the CH-47F program is to improve the performance and extend the useful life of the CH-47. This effort includes installing a digitized cockpit, rebuilding the airframe, and reducing aircraft vibration through airframe stiffening. Prime contractor: Boeing Program office: Huntsville, Ala. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $0 million 5/98 10/02 change Procurement $516.0 million Research & development cost $148.0 $169.6 14.6 Quantity 16 aircraft Procurement cost $2,613.9 $5,927.7 126.8 Total program cost $2,761.9 $6,097.3 120.7 Program unit cost $9.145 $17.986 96.7 Total quantities 302 339 12.3 Acquisition cycle time (months) 82 99 20.7 The CH-47F helicopter began low-rate production in December 2002, although key production processes were not in control. Program officials believe that CH-47F production is low risk because no new technology is being inserted into the aircraft, two prototypes have been produced, and the production process has been demonstrated during the development phase. The CH-47F technologies and design appear mature, although a low percentage of engineering drawings were released at the design review. Production unit costs have more than doubled due to contractor rate increases, increases in system capabilities, and initial underestimation of program cost. GAO-03-476 Acquisition Trends and Risks Page 31 Common Name: CH-47F Appendix I CH-47F Program Technology Maturity and (4) initial underestimation of program costs. According to the program manager, the Army has fully Although we did not assess technology maturity in funded the program’s cost growth of about $2.5 billion detail, the CH-47F is a modification of the existing (then-year dollars). This increase in program cost CH-47D helicopter. Program officials believe that all necessitated rebaselining the CH-47F program. The critical technologies are mature and have been Army approved the CH-47F acquisition program demonstrated prior to integration into the CH-47F baseline. development program. Program Office Comments Design Maturity The CH-47F program office generally concurred with The CH-47F design is complete, with 100 percent of this assessment. the drawings released to manufacturing. However, at the design review only 37 percent of the system’s engineering drawings were complete. Since that time, the number of drawings completed has increased substantially. The majority of the new drawings were instituted to correct wire routing and installation on the aircraft, changes program officials believed could not be determined until after the first prototype was developed. Production Maturity CH-47F production maturity could not be determined because the program does not use statistical process control to ensure that production processes are stable. Program officials believe the production is low risk because two prototypes have been produced and the production processes have been demonstrated during the development phase. The Army plans to conduct operational testing in fiscal year 2004 to demonstrate its readiness to proceed into full-rate production. Prior to that decision, the Army plans to complete a risk assessment for the CH-47F to eliminate any production risk that remains. Other Program Issues Both the total cost and the program unit cost for the CH-47F production program have more than doubled. This growth triggered a Nunn-McCurdy breach (see 10 U.S.C. 2433) in December 2001, requiring a review by the Secretary of Defense and a report to Congress. As a result, the Secretary of Defense has certified to Congress that the CH-47F is essential for national security, there are no alternatives, the new cost estimates are reasonable, and the management structure is in place to continue to keep costs under control. According to the program office, the cost increases were due to (1) prime contractor labor rate increases and material cost growth, (2) additional system capabilities required by the Army, (3) recapitalization of 36 Special Operations aircraft, Page 32 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: Comanche RAH-66 Comanche The Army’s Comanche is a multi-mission helicopter intended to perform tactical armed reconnaissance. It is designed to operate in adverse weather across a wide spectrum of threat environments and provide improved speed, agility, reliability, maintainability, and low observability over existing helicopters. It is also expected to lower operating costs through the use of integrated diagnostics, a composite airframe, and a bearingless rotor system. It will replace the AH-1, OH-6, and OH-58 helicopters. Prime contractor: Boeing-Sikorsky Program office: Huntsville, Ala. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $1.1 billion 7/00 10/02 change Procurement $0 million Research & development cost $8,886.4 $12,556.3 41.3 Quantity 0 aircraft Procurement cost $30,550.9 $21,939.5 -28.2 Total program cost $39,824.0 $34,545.0 -13.2 Program unit cost $32.831 $53.146 61.9 Total quantities 1,213 650 -46.4 Acquisition cycle time (months) 222 250 14.9 Most of the Comanche’s critical technologies have demonstrated acceptable levels of maturity, and the program appears very close to meeting the best practice standard for a stable design. This level of maturity follows many years of difficult development. Since the program’s first cost estimate was originally approved in 1985, the research and development cost has almost quadrupled and the time to obtain an initial capability has increased from 9 years to over 21 years. The program has recently undergone another major restructuring to incorporate an evolutionary acquisition approach and reduce concurrency and lower overall risk. This restructuring shows promise of being a knowledge- based program that matches program resources with user requirements. GAO-03-476 Acquisition Trends and Risks Page 33 Common Name: Comanche Appendix I RAH-66 Comanche program requirements with force requirements and program risks. Weight issues were addressed through Technology Maturity increased engine performance. Initial operational capability was moved from December 2008 to Seven of the Comanche’s eight critical technologies September 2009 to reduce risk and significantly are considered mature. Only one critical technology, increase the amount of testing conducted. the radar cross-section needed for low observability, requires additional development. The Army expects Program Office Comments that this technology will reach maturity in fiscal year 2005, a year before the production decision. In commenting on a draft of this assessment, program officials generally concurred with our assessment. Design Maturity They added that in October 2002, the Office of the Secretary of Defense approved the Comanche The Comanche program has released 73 percent of the program as an evolutionary acquisition approach. The engineering drawings to manufacturing. The program Comanche quantity was reduced from 1213 to 650 has improved its ability to reach design maturity by based on emerging results of the Comanche’s role in rescheduling the design review from July 2002 to April the Objective Force. This reduction in quantities, 2003. The program estimates that it will complete combined with the research and development cost 90 percent of the drawings by the design review under growth, resulted in a program acquisition unit cost the proposed plan, instead of the former 59 percent increase of approximately 62 percent. Excluding under the previous program. impacts of the quantity reduction, the average Critical technologies have not yet been integrated and procurement unit cost increased 18 percent and the demonstrated on the Comanche airframe. Prior to the program acquisition unit cost increased 23 percent. proposed program restructure, integration of critical technologies was considered high risk, even though most of the technologies had reached maturity on other platforms. Program officials believe that the restructured program reduces integration risks and that the longer development schedule will allow for reduced concurrent development and additional integration time and facilities, thereby reducing critical risks. The longer schedule also provides additional time for near-term development testing, use of a production representative aircraft for initial operational testing, and full qualification testing. Additionally, the phasing of development and operational tests was revised and expanded to reduce overall program risk. Other Program Issues Continuing cost and schedule issues have led to the most recent restructuring of the program. In October 2002, the Office of the Secretary of Defense approved the Comanche program to continue under an evolutionary acquisition approach. However, because of uncertainties with future funding and capabilities, quantities were reduced from 1213 to 650 aircraft. This reduction in quantities, combined with the research and development cost growth, resulted in a unit cost increase of approximately 62 percent. Program officials stated that the restructuring added a more robust internal review process and balanced Page 34 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: ERGM EX-171 Extended Range Guided Munition (ERGM) The Navy’s ERGM is a rocket-assisted projectile that is fired from a gun aboard ships. It can be guided to land targets at ranges of between about 10 and 50 nautical miles to provide fire support for ground troops. ERGM is expected to offer increased range and accuracy compared to the Navy’s current gun range of 13 nautical miles. ERGM requires modifications to existing 5-inch guns, a new munitions-handling system (magazine), and a new fire control system. We assessed the projectile only. Prime contractor: Raytheon Program office: Washington, D.C. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $28.6 million 4/97 12/02 change Procurement $3.8 million Research & development cost $77.6 $326.1 320.5 Quantity 0 rounds Procurement cost $308.4 $159.4 -48.3 Total program cost $386.0 $485.5 25.8 Program unit cost $0.045 $0.150 233.8 Total quantities 8,570 3230 -62.3 Acquisition cycle time (months) 50 121 146.9 The ERGM program began product development with very few of its critical technologies mature according to the best practices standards. While significant progress has been made in the past 7 years, program officials do not expect to achieve maturity on all critical technologies until after the design review. No production representative engineering drawings had been released at the time of our assessment, and none are projected by the system design review. The program office currently expects to release these 1 year later. In June 2002, the program conducted a successful test of a guided tactical round under realistic launch conditions. This test did not evaluate the performance of a new warhead design. GAO-03-476 Acquisition Trends and Risks Page 35 Common Name: ERGM Appendix I ERGM Program fiscal years 2002 and 2003, stretched out program milestones and will delay deployment of ERGM until Technology Maturity 2006. Fourteen of ERGM’s 20 critical technologies have Other Program Issues demonstrated technological maturity. The remaining 6 technologies are approaching maturity, and program Future program costs are not accurately reflected in officials expect that all 20 critical technologies will be the latest program cost estimate and the fiscal year demonstrated in an operational environment by the 2004/2005 budget request. The cost estimate is based end of 2003, approximately 7 months after the design on a much lower production quantity than is contained review. Three of the technologies yet to reach maturity in either the approved or the current draft revision of are part of the new unitary warhead design, and a the ERGM acquisition program baseline. The budget fourth is related to this change. Program officials request does not fully fund the 80 operational test recently identified the unitary warhead’s safe/arm rounds currently required. device and fuze as a critical technology, after a Navy Two testing issues could affect the program. The safety review concluded that it needed to be Director of Operational Test and Evaluation has raised redesigned to meet applicable DOD standards. a concern about test range restrictions that could limit The ERGM program began development with only one realistic operational testing. Finally, the project of its critical technologies mature. Having only one manager stated that the availability of a fully capable critical technology mature at the start of product ship to support development testing could be an issue development has caused cost and schedule problems. due to funding shortfalls for magazine modifications For example, when the program began, none of the on these ships. components of the rocket motor had been integrated Some of the cost increases and schedule slippages to into an ERGM representative design. Subsequent date may be attributed to the fact that the contractor problems with the performance reliability of the motor relocated the program in 1998, resulting in a loss of resulted in cost growth of more than $13 million. trained personnel and development inertia. Design Maturity Program Office Comments None of ERGM’s approximately 127 production In commenting on a draft of this assessment, program representative engineering drawings have been officials stated that although production released to manufacturing. The program office plans representative drawings will not be available at the for all of these drawings to be released in June 2004, design review, the entire ERGM design would be about one year after the design review. In the under configuration control. Design maturity at that meantime, the design review will be used to validate time will be sufficient to produce all-up rounds for the design of the development test rounds. The June land and ship-based development testing. Based on 2004 drawing release, which will reflect knowledge data obtained from flight and qualification tests in gained from 8 of 18 flight tests and some qualification fiscal year 2004, minor revisions to the ERGM tests, will be used to build the 80 production technical data package may be made. Production representative operational test rounds. Program representative drawings will be finalized by June 2004. officials pointed out that progress has been made in Program officials stated that they are highly maturing the design. For example, the main elements encouraged by the significant progress in ERGM of the design were validated during the guided gunfire development activities over the last 18 months. They test in June 2002. further stated that they have a high degree of In January 2002, in order to meet lethality and safety confidence that ERGM will meet all performance requirements, the Navy decided to make a significant requirements, while meeting the production cost goals change to the warhead design, moving from a specified in the acquisition program baseline. multiple-submunition design to a single explosive—or unitary—warhead. This decision, coupled with the decision to stay within planned funding levels for Page 36 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: Excalibur Excalibur Artillery Round The Army’s Excalibur is a family of extended range, precision, 155-mm artillery projectiles. It is designed to increase soldier survivability by allowing the Future Combat Systems’ nonline of sight cannon to fire from farther away and defeat threats more quickly, while reducing logistic support. It also is intended to be more effective when fired at urban targets, through a combination of altered trajectory and global positioning system accuracy. Prime contractor: Raytheon Missile Systems Program office: Picatinny Arsenal, NJ. FY 2004 request: FY 2003 dollars in millions Approved Latest Percent R&D $134 million 4/97 2/03 change Procurement $0 million Research & development cost $59.8 $662.8 1,008.7 Quantity 0 projectiles Procurement cost $676.2 $4,135.9 511.6 Total program cost $736.0 $4,798.7 552.0 Program unit cost $0.004 $0.062 1,578.8 Total quantities 200,000 77,677 -61.2 Acquisition cycle time (months) 160 136 -15.0 The Excalibur program’s three critical technologies are not fully mature, even though product development began over 5 years ago. The technologies appear to be approaching maturity, and program officials project demonstrating technology and design maturity before the design review in 2005. Currently, 13 percent of the drawings are at the level that could be released to manufacturing. Program officials expect to have a stable design by the design review. The program has undergone a major restructuring effort. It has encountered a number of challenges since development began, including a substantial decrease in planned quantities, a relocation of the contractor’s plant, limited early funding, technical problems, changes in program direction, and a merger with another program. GAO-03-476 Acquisition Trends and Risks Page 37 Common Name: Excalibur Appendix I Excalibur Program Program Office Comments In commenting on a draft of this assessment, program Technology Maturity officials generally agreed with the information in this None of the Excalibur’s three critical technologies are report. However, they provided the following fully mature according to best practice standards. clarifying comments. According to program officials, all three have been Concerning the Excalibur design maturity, program demonstrated in a relevant environment and are officials stated that approximately 600 drawings are expected to reach maturity before the design review in anticipated at the subsystem level. But because the March 2005. The Excalibur’s design and requisite program is still in research and development, no technologies have changed since product drawings have been officially released to development was started. The three critical manufacturing. The program is fabricating hardware technologies for the current design are the guidance in a research and development environment. control system, the airframe, and the warhead. The warhead was not considered a critical technology in 1997 because the Excalibur design called for a warhead that was under production for other munitions. Based on Army direction, the program has undertaken development of a different warhead that is currently undergoing testing. Design Maturity About 13 percent of the Excalibur’s engineering drawings are at a level that could be released to manufacturing. The program office plans to have all of its drawings complete and released to manufacturing by the design review in March 2005. However, program officials could not estimate the total number of drawings expected. Other Program Issues The program has gone through many changes since the beginning of product development in May 1997. It was almost immediately restructured due to limited funding, and it was restructured again in 2001. In response to congressional direction, the program was restructured to merge with the joint Swedish/U.S. program known as Trajectory Correctable Munitions. The merger should help the program deal with design challenges, including issues related to its folding fin design. Also, in May 2002, the Office of the Secretary of Defense directed the program to develop the Excalibur for the Future Combat Systems nonline of sight cannon and to field it in fiscal year 2008. Although program officials have not yet released the new cost and schedule estimates, the net effect of these changes has been to increase the program’s schedule and to substantially decrease planned procurement quantities. As a result, the program’s overall costs and unit costs have dramatically increased. Page 38 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: F/A-18E/F F/A-18E/F Super Hornet The Navy’s F/A-18E/F is a multi-mission tactical aircraft designed to meet fighter escort, interdiction, fleet air defense, and close air support mission requirements. The program was approved as a major modification to earlier F/A-18 aircraft in 1992. It is intended to complement and replace the Navy's F/A-18C/D and F-14 aircraft. Prime contractor: Boeing Program office: Patuxent River, Md. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $179.0 million 6/92 12/01 change Procurement $3.0 billion Research & development cost $6,174.4 $6,181.6 0.1 Quantity 42 aircraft Procurement cost $61,215.3 $41,368.2 -32.4 Total program cost $67,389.7 $47,549.8 -29.4 Program unit cost $67.390 $86.770 28.8 Total quantities 1,000 548 -45.2 Acquisition cycle time (months) 102 112 9.8 The F/A-18E/F went into full-rate production in June 2000. Although the program proceeded without obtaining full product knowledge at key decision points, it embraced the concepts of attaining design and manufacturing knowledge early in development. The program released just over half of its engineering drawings by its design review. When low-rate production began, nearly all of the drawings were released and about 75 percent of the manufacturing processes were in control. The Navy reduced some program risk because aviation electronics from an earlier version of the F/A-18 were incorporated into the baseline F/A-18E/F. Furthermore, focus was placed on commonality between the F/A-18 C/D and the F/A-18 E/F, which further reduced risk. GAO-03-476 Acquisition Trends and Risks Page 39 Common Name: F/A-18E/F Appendix I F/A-18E/F Program The level of effort and timing to incorporate some of the sensors—the AESA radar and the Advanced Technology Maturity Targeting sensor—may prove to be a challenge. We have assessed the AESA radar elsewhere in this We did not assess the technology maturity of the report. F/A-18E/F program because it is already in full-rate production. Nevertheless, we did not identify any Program Office Comments technical challenges during the development of this aircraft in our previous reviews. In commenting on a draft of this assessment, program officials stated initial schedule delays were due to a Design Maturity procurement reduction of 10 aircraft in a 1998 Program Objective Memorandum. Since that time, the The F/A-18E/F design appears complete. The program contractor has consistently delivered aircraft ahead of has released 100 percent of the design drawings to schedule. Program officials also noted that the aircraft manufacturing. At the time of the critical design are demonstrating two to three times the quality of the review in July 1994, 56 percent of the engineering F/A-18C/D and have provided measurable drawings were completed and released to improvements to squadron readiness. In addition, all manufacturing for aircraft structure and systems. F/A-18E/F preplanned upgrades continue to track to According to program officials, they decided to their program schedules. The Joint Mounted Helmet proceed despite the low level of completed drawings Cueing System has completed operational evaluation, because the knowledge gathered from earlier and the system has been incorporated into lot 24 of the F/A-18C/D models gave them confidence that the aircraft (deliveries of which began in September design was stable. By the time of the low-rate initial 2001). Deliveries of the Advanced Targeting Forward production decision, 99 percent of the drawings had Looking Infrared Sensor production units began in been released. April 2002, and the units were deployed in January 2003. Finally, program officials stated that the AESA Production Maturity radar program continues to execute as planned, and According to program officials, they currently have the program has received the first engineering and 100 percent of their critical manufacturing processes manufacturing development unit. under control, according to the best practice standard. Therefore, they are no longer tracking processes using statistical process control. However, defects are still monitored through inspections, failures, and age exploration testing, and during maintenance. If production problems are identified, the program would resume statistical process control analysis where necessary. Program officials estimate that about 75 percent of key manufacturing processes were in control at the low-rate production decision in March 1997. Program officials stated that they concentrated on maturing their manufacturing processes before starting production. As a result of these efforts, labor efficiency rates have steadily improved. Other Program Issues The F/A-18E/F will not reach its full potential until after the incorporation of several preplanned upgrades—the Active Electronically Scanned Array (AESA) radar, the Joint Mounted Helmet Cueing System coupled with the AIM-9X missile, and the Advanced Targeting Forward Looking Infrared sensor. Page 40 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: F/A-22 F/A-22 Raptor The Air Force’s F/A-22, originally planned to be an air superiority fighter, will also have air-to-ground attack capability. It is being designed with advanced features, such as stealth characteristics, to make it less detectable to adversaries and capable of high speeds for long ranges. It also has integrated aviation electronics (avionics) designed to greatly improve pilots' awareness of the situation surrounding them. It is designed to replace the Air Force’s F-15 aircraft. Prime contractor: Lockheed Martin Program office: Dayton, Ohio FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $936.5 million 2/92 12/02 change Procurement $4.2 billion Research & development cost $20,938.2 $30,836.1 47.3 Quantity 22 aircraft Procurement cost $54,272.2 $39,049.2 -28.0 Total program cost $75,461.2 $70,469.4 -6.6 Program unit cost $116.5 $253.5 117.7 Total quantities 648 278 -57.1 Acquisition cycle time (months) 203 230 13.3 Because the F/A-22 Program Office stopped collecting process control data in 2000, the program began production in 2001 with no proof that processes were in control, as defined by best practice standards. Technology appears mature and the design appears stable; however, problems with the vertical tail and the avionics have been discovered recently, which require design modifications. Delays in capturing technology, design, and production knowledge and these latest problems contributed to cost increases and schedule delays. The potential exists for further cost increases and schedule delays as a significant amount of the test program remains, including operational tests. Also, the latest production cost estimate is likely to increase because of several factors, and the estimate assumes over $25 billion in offsets from cost reduction plans. GAO-03-476 Acquisition Trends and Risks Page 41 Common Name: F/A-22 Appendix I F/A-22 Program practice standards. In September 2001, the Air Force awarded a contract for 10 aircraft to begin F/A-22 Technology Maturity production. Although we did not assess the F/A-22 key Other Program Issues technologies using technology readiness levels, the three critical technologies (supercruise, stealth, and In September 2001, the Air Force acknowledged an integrated avionics) appear mature. Two of these estimated production cost increase of $5.4 billion technologies, integrated avionics and stealth, were (then-year dollars) over the congressional cost limit. late to mature. It was not until September 2000, or We believe conditions exist that makes it likely over 9 years into product development, that the production costs will increase even further. In integrated avionics reached maturity. During addition, the Air Force is counting on over $25 billion development, the integrated avionics was a source of in cost reduction plans to offset estimated cost growth schedule delays and cost growth. Since 1997, avionics and enable the program to meet the production cost software development and flight-testing have been estimate. If these cost reduction initiatives are not delayed, and the cost of avionics development has achieved as planned, production costs could increase. increased by over $980 million. Moreover, the Air Further, the contractor has yet to demonstrate it can Force did not complete an evaluation of stealth efficiently build the development aircraft, and technology on a full-scale version of the aircraft until estimates of the cost to build the production aircraft several years into product development. continue to increase. In December 2002, DOD estimated development costs Design Maturity would increase by $876 million and that the funding The basic design of the F/A-22 is essentially complete, necessary to cover this cost increase would be as engineering drawings are complete. However, transferred from production funding. Avionics design changes have been necessary as a result of problems discovered in flight-testing are the primary flight tests and structural tests. For example, problems contributor to a six-month extension to the with excessive movement of the vertical tails and development program. avionics failures in flight tests were discovered, and they will require costly design modifications. The Air Program Office Comments Force still has to complete a significant amount of development testing and operational testing. Until In commenting on a draft of this assessment, program initial operational testing is completed as planned in officials stated that the report implies that had the F/A- June 2004, the possibility of additional design changes 22 deferred product development until engineering remains. and testing were accomplished, at a level providing higher product knowledge, substantial cost increases Design knowledge for the F/A-22 was built slowly. and schedule delays would have been prevented. The Only 26 percent of the total drawings were released at issues cited as examples do not pose a substantial risk the 1995 design review. The program released to either cost or schedule and have either been fixed 90 percent of the drawings over 3 years later, after the through minor design change or are anticipated to be first two development aircraft had been delivered. resolved without major impact to continued testing Late drawing release contributed to parts shortages and production. A program of this nature is expected and work performed out of sequence during assembly, to have both design and technological maturities to which drove up costs and contributed to delaying overcome and there will be some element of risk flight tests by 83 months. throughout its development and into the production process. Production Maturity The program office stopped collecting process control information in November 2000. The contractor estimated that nearly half of the key processes had reached a marginal level of control, but not up to best Page 42 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: JASSM Joint Air-to-Surface Standoff Missile (JASSM) JASSM is a joint Air Force and Navy program designed to attack surface targets outside of the range of area defenses. JASSM will be delivered by a variety of aircraft, including the F-16 C/D, the B-52H, the F/A-18E/F, the B-2, and the B-1B. The system includes the missile, software, and software interfaces with the host aircraft and mission planning system. Prime contractor: Lockheed Martin Program office: Eglin, Fla. FY 2004 funding request FY 2003 dollars in millions Approved Latest Percent R&D $56.3 million 11/98 12/02 change Procurement $102.5 million Research & development cost $863.6 $1,209.5 40.1 Quantity 250 missiles Procurement cost $1,078.7 $2,568.4 138.1 Total program cost $1,962.9 $3,777.8 92.5 Program unit cost $0.795 $0.852 7.2 Total quantities 2,469 4,434 79.6 Acquisition cycle time (months) 75 87 16.0 The JASSM program entered production in December 2001 without ensuring that production processes were in control, according to best practice standards. However, program officials indicated that they have demonstrated the production processes and that they sample statistical data at the subsystem level. The program ensured that the technology was mature and that the design was stable at critical points in development, closely tracking best practice standards. Redesign remains one area of concern because recent test failures have led to the delay of operational tests. The program has identified fixes to the problems, and a retrofit plan is in progress. The contractor’s ability to attain a higher production rate is another area of concern. GAO-03-476 Acquisition Trends and Risks Page 43 Common Name: JASSM Appendix I JASSM capacity and expanding facilities to support full-rate production plus anticipated foreign military sales. Technology Maturity Program officials believe that none of the manufacturing processes that affect critical system The JASSM program used existing technologies and so characteristics are problematic, although there are its level of technology maturity is high. Although none key production processes that have cost implications, of the subsystems involve new technologies, three such as the bonding for the low observable materials critical technologies are new applications of existing and the painting/coating application. technologies. These three technologies are the global positioning system anti-spoofing receiver module, the Program Office Comments low observable technology, and the composite materials. The program office reports these In commenting on a draft of this assessment, program technologies to be mature. officials stated that JASSM has established a new benchmark for missile development by ensuring Design Maturity weapon system design maturity and production capability were demonstrated during development The contractor has released 100 percent of the prior to entering low-rate initial production. JASSM’s drawings to manufacturing. The two remaining acquisition strategy incorporated existing technology concerns are the software for the missile and the to reduce program risk and speed up delivery of the status of integration with aircraft, although program weapon to the warfighter. The officials further stated officials believe the risks are low. that JASSM’s development cycle is 33 percent faster Recent failures in development and operational tests than comparable weapon systems, with production have led to the delay of the remaining JASSM unit prices 50 percent less than weapon systems with operational tests. During an operational test on less capability. The contractor was contracted to October 10, 2002, the missile flew its planned route produce 82 all-up production prove-out test rounds and penetrated the target, but it failed to detonate. during development on the production line prior to According to program officials, this failure occurred low-rate initial production missile delivery. Program because the requested test methodology was officials noted that establishment of production experimental and exceeded original design representative hardware during development was key requirements for the fuze. On October 24, 2002, during to the contractor’s ability to prove out all production the last of 11 developmental tests, the missile went out processes. The contractor has a capitalization plan to of control and crashed at the test site. According to meet full-rate production quantities. program officials, this failure was due to a failed actuator. Program officials believe they have identified the problems in both cases and have a retrofit plan. Retrofits will be tested in spring 2003. However, if additional problems occur, they will have to be corrected while JASSM is in production, which may require additional retrofitting of missiles already produced. Production Maturity Program officials do not collect production process control data at the system level. However, they stated that all production processes had been demonstrated and that statistical data is collected at the subsystem level and is sampled as required. Program officials indicated that the contractor will produce at the rates required for the first production lot and 76 missiles will be delivered. A contract for the second lot, 100 missiles, has been signed. Production concerns remaining include achieving full-rate production Page 44 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: Joint Common Missile Joint Common Missile The Joint Common Missile is an air-launched and potentially a ground-launched missile designed to target tanks; light armored vehicles; missile launchers; command, control, and communications vehicles; bunkers; and buildings. It is designed to provide line of sight and beyond line-of-sight capabilities. It can be employed in a fire-and-forget mode—providing maximum survivability—or a precision attack mode, providing the greatest accuracy. The Joint Common Missile will be a joint Army and Navy program with USMC participation. Prime contractor: In competition Program office: Huntsville, Ala. FY 2004 funding request: FY 2003 dollars in millions Latest Percent R&D $183.8 million Approved 11/02 change Procurement $0 million Research & development cost NA $563.92 0 Quantity 0 missiles Procurement cost NA $1,597.3 0 Total program cost NA $2,161.2 0 Program unit cost NA TBD 0 Total quantities NA 8,425 0 Acquisition cycle time (months) NA 60 0 Note: Funding from FY 2004 President’s Budget. Total Army and Navy Acquisition Objective is 77,400. Official cost position to be finalized by 6/2003. NA = not applicable. The Joint Common Missile is scheduled to enter product development before any of its critical technologies are fully mature, according to best practices. Furthermore, program officials currently project that the critical technologies will not reach maturity until a year after the design review. The Army will initially focus development on an air- launched version. GAO-03-476 Acquisition Trends and Risks Page 45 Common Name: Joint Common Missile Appendix I Joint Common Missile System risk. Prototype testing of a multi-mode seeker (tower and captive flight), a multi-purpose warhead (heavy Technology Maturity armor and building structures), and a rocket motor (high maximum to minimum thrust profiles over None of the Joint Common Missile’s three critical operational temperatures) is currently being technologies have demonstrated full maturity. These conducted with results to be available in sufficient critical technologies include a multi-mode seeker for time to support the milestone decision to begin the increased countermeasure resistance, a boost-sustain development phase. propulsion for increased standoff range, and a multi- purpose warhead for increased lethality capability. Program officials noted that many of the components of these technologies are currently in production on other missile systems, but that they have not been fully integrated. While backup technologies exist for each of the critical technologies, substituting any of them would result in degraded performance or increased costs. Design Maturity Program officials project that full integration of the subsystems into the Joint Common Missile will be mature one year after the system design review, which is scheduled for July 2004. Other Program Issues The current cost estimates are from the fiscal year 2004 President’s budget. This cost estimate will be updated at the conclusion of the Army's formal estimating process. The formal estimating process began in January 2003 for presentation at the milestone decision review in September 2003. According to program officials the Army's acquisition objective is 54,400 missiles and the Navy's acquisition objective is 23,000. Program officials also indicated that the modular design will reduce life-cycle costs, including demilitarization, and will enable continuous technology insertion to ensure improvements against advancing threats. Program Office Comments In commenting on a draft of this assessment, program officials stated that they plan to demonstrate the technological maturity required by DOD acquisition system policy before beginning the development phase in September 2003. Program officials further stated that the technological maturity projected represents a major achievement in the technology's demonstrated readiness in a relevant environment and provides the critical technologies the maturity necessary to accomplish system integration of demonstrated subsystems, thereby reducing program Page 46 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: JPATS Joint Primary Aircraft Training System (JPATS) JPATS is a joint acquisition by the Air Force and the Navy to replace the aging primary trainer aircraft fleet. JPATS is a variant of the Beech Pilatus PC-9 commercial aircraft, but it has been modified significantly to incorporate military unique requirements. The JPATS program includes the aircraft; the ground-based training system (simulators, course materials), and an integrated training management system. We assessed the aircraft. Prime contractor: Raytheon Program office: Wright Patterson AFB, Ohio FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $0 million 8/95 12/01 change Procurement $283.0 million Research & development cost $349.1 $294.2 -15.7 Quantity 52 aircraft Procurement cost $2,720.3 $4,316.8 58.7 Total program cost $3,138.6 $4,674.8 49.0 Program unit cost $4.408 $5.970 35.4 Total quantities 712 783 10.0 Acquisition cycle time (months) 97 113 16.5 The JPATS aircraft entered full production in December 2001 without ensuring that the manufacturing processes were mature. The aircraft entered limited production in 1995 before achieving design stability. DOD considered the aircraft a mature commercial product that did not require extensive product development. However, program officials underestimated the number of design changes needed to accommodate the military unique requirements. The design has subsequently changed about 70 percent from the commercial baseline. The JPATS initial operating capability occurred in 2002, 2 years later than originally planned. GAO-03-476 Acquisition Trends and Risks Page 47 Common Name: JPATS Appendix I JPATS Program Program Office Comments In commenting on a draft of this assessment, program Technology Maturity officials disagreed with our analysis of production Although we did not assess the JPATS aircraft key maturity. They stated that statistical process control is technologies, the aircraft is a derivative of a not the only determinant of maturity. The production commercial aircraft and the technologies appear line was certified by the International Organization for mature. Standardization in 1994 and by the Federal Aviation Administration in 1999, and is currently producing Design Maturity aircraft according to these guidelines. Assembly labor The basic design of the aircraft is currently complete. hours per aircraft are on a 78 percent learning curve, and they have decreased 65 percent since the first However, the military unique design was only about operational aircraft was delivered. The production line 5 percent complete shortly after the program was approved to enter limited production in 1995. The rate increased to five aircraft per month by the end of 2002, and remains there still, even as design changes design has changed about 70 percent from its are incorporated into the production line. After initial commercial baseline. Testing has revealed tangible examples of design immaturity. Several subsystems, production difficulties, over the past year the contractor has been delivering aircraft ahead of including the engine, the UHF radio, and the schedule while incorporating engineering changes to environmental control system, have required extensive modification or redesign. These and other increase the suitability of the system. Program officials also stated that the cycle time should be problems have delayed both aircraft testing and the reduced by 6 months because the JPATS program was production decision. unable to award a contract or proceed with contract In November 2001, operational testers concluded that performance pending the disposition of several bid JPATS was operationally effective but not protests. operationally suitable. They cited concerns about the aircraft's reliability, availability, and maintainability. GAO Comments They also reported that the full JPATS had not yet been tested due to uncorrected deficiencies in the Our prior work has shown that leading commercial aircraft and the immaturity of the software-intensive firms rely on statistical control data as the best training information management system. The indicator of production readiness. Despite its contractor is incorporating changes to the aircraft as a commercial origins, the JPATS program entered result of operational test issues. Operational testers limited and full production without this information. expressed concern that some changes may adversely Subsequent testing has uncovered numerous problems impact other critical subsystems. Despite these issues, that require modification and retrofit. Although the the Air Force proceeded into full-rate production the aircraft has been production certified by the Federal following month. Aviation Administration, its regulations merely require the contractor to maintain a generic quality control Production Maturity system and do not provide assurance that the components can be built within cost and on schedule. Production maturity remains at issue because We used DOD official documents to determine information about the contractor's manufacturing acquisition cycle time. process controls is not available. The Air Force did not require this information because the aircraft was considered a commercial derivative. Other factors could affect production maturity. In 2002, two key modifications—the environmental control system and the UHF radio—began to be incorporated on the aircraft. The program office has also identified additional retrofit requirements and is evaluating a replacement for the collision warning system. The rework associated with these changes may affect aircraft production efficiencies. Page 48 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: JSF F-35 Joint Strike Fighter (JSF) The JSF program goals are to develop and field a family of stealthy, strike fighter aircraft for the Navy, Air Force, Marine Corps, and U.S. allies, with maximum commonality to minimize life-cycle costs. The carrier suitable version will complement the Navy F/A-18 E/F. The Air Force version will primarily be an air-to-ground replacement for the F-16 and the A-10 and complement the F/A-22. The short take-off and vertical landing version will replace the Marine Corps F/A-18 and AV-8B. Significant foreign military purchases are expected. Prime contractor: Lockheed Martin Program office: Arlington, Va. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $4.4 billion 10/01 12/01 change Procurement $0 million Research & development cost $32,788.6 $32,880.8 0.3 Quantity 0 aircraft Procurement cost $145,733.8 $147,604.7 1.3 Total program cost $180,047.0 $180,485.5 0.2 Program unit cost $62.8 $63.0 0.2 Total quantities (U.S. only) 2,866 2,866 0.0 Acquisition cycle time (months) 185 185 0.0 The JSF program entered the development phase without demonstrating that its eight critical technologies had reached maturity according to best practice standards. Two technologies, propulsion and critical fabrication techniques, were very close to maturity. DOD conducted an independent review in 2001 and concluded that the technology maturity was sufficient to proceed into product development. The JSF program no longer focuses on the previous 8 technology areas, instead it uses a different method of integration and risk management that currently tracks 23 program level risks. We were unable to assess the new risk areas, but program data indicates that the majority are moderate risk. The program expects to have 80 to 90 percent of its critical build-to-packages completed by the final design review in 2005. GAO-03-476 Acquisition Trends and Risks Page 49 Common Name: JSF Appendix I JSF Program JSF preliminary design review in late March 2003. We were unable to review the results of those meetings Technology Maturity prior to the release of this report, but program office data indicates the discovery of higher risk levels for During its concept development phase, the Joint the propulsion system and overall aircraft weight. Strike Fighter had eight critical technologies: short take-off vertical landing/integrated flight propulsion Other Program Issues control, prognostic and health management, integrated support systems, subsystems technology, Due to the highly complex nature of the JSF design, integrated core processor, radar, mission systems the Director, Operational Test and Evaluation, expects integration, and manufacturing. We reported in numerous test challenges for the program. These May 2000 and again in October 2001 that low levels of challenges include the integration of highly advanced maturity in these technologies could increase the sensors with the avionics systems, vertical thrust likelihood of program cost and schedule growth. capability for the Marine Corps version, and performance and maintenance requirements of the The program experienced cost growth and schedule low observable capabilities. The program has received concerns during the concept demonstration phase, authority for its low-rate production quantity to reach prior to starting product development in October 2001. 15 percent—427 aircraft—of the total production run. This included manufacturing delays for hardware used on the propulsion system for the Marine Corps Program Office Comments version. To reduce cost and schedule delays, the program eliminated planned risk-reduction efforts and In commenting on a draft of this assessment, program delayed other technology demonstrations until after officials stated that, prior to the start of the product development began. development phase, JSF’s key technologies had gone through an extensive series of tests and An independent review performed by DOD in 2001, demonstrations, culminating in four experimental using a different method than technology readiness aircraft proving flight capabilities for each service levels, concluded that the overall technology maturity variant in over 200 hours of flight. An independent of the JSF program was sufficient to enter into DOD review concluded that JSF had demonstrated product development. Today, the program no longer sufficient technical maturity for low risk entry into the monitors the eight specific technologies from the development phase. For this phase, the program previous phase. Instead, the program is using officials stated that JSF has adapted the contractor's Lockheed Martin’s Key System Development risk mitigation approach. Risk mitigation assessments Integration approach to monitor overall technology in February 2003 indicated that most program level development and design integration. Further, the risks were rated moderate using the contractor’s program tracks 23 program level risk areas and has approach. Cost and schedule planning for the assessed 19 as moderate and 2 as high. Five of eight development phase has evolved as the services critical technologies from the concept development iterated system operational requirements with life phase are contained within elements of these program cycle cost. The JSF air system preliminary design level risks. We have not evaluated the current JSF review is scheduled in March 2003, and the first of technique for assessing risks. three critical design reviews is to occur in April 2004. Finally, program officials stated that the program is Design Maturity being executed in accordance with its cost, schedule, The program has committed time and funding to the and technical baselines. system development and demonstration phase that should improve its chances for success. Specifically, the new program structure will now include additional test aircraft, increased software on the aircraft, and a greater number of flight test hours. Program documents indicate that the 1996 estimated cost and schedule for JSF’s development phase have increased by 56 percent and 40 percent, respectively, due to changes in program scope. Meetings were held for the Page 50 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: JSOW Joint Standoff Weapon (JSOW) JSOW is a joint Air Force and Navy guided bomb to attack targets from outside of the range of most enemy air defenses. There are three JSOW variants that use a common air vehicle. Two variants (JSOW A and B) carry submunitions to attack soft targets or armored vehicles. The unitary variant (JSOW C) uses a seeker, autonomous targeting acquisition software, and a single warhead to attack targets. We assessed the unitary variant and the common air vehicle. Prime contractor: Raytheon Systems Company Program office: Patuxent River, MD FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $0.8 million 6/92 12/01 change Procurement $65.89 million Research & development cost $325.2 $311.5 -4.2 Quantity 175 missiles Procurement cost $3,871.4 $803.8 -79.2 Total program cost $4,196.6 $1,115.3 -73.4 Program unit cost $0.538 $0.372 -30.9 Total quantities 7,800 3,000 -61.5 Acquisition cycle time (months) 89 112 25.8 The JSOW program is scheduled to begin low-rate production in March 2003 without knowing that production processes are in control, according to best practice standards. The program instead relies on an after-production process of inspection to discover defects. Immature technology at the start of development at least partially delayed design maturity, and developmental testing of the seeker is not complete. GAO-03-476 Acquisition Trends and Risks Page 51 Common Name: JSOW Appendix I JSOW Program Program Office Comments In commenting on a draft of this assessment, program Technology Maturity officials stated that the contractor has completed The JSOW unitary’s technology appears mature. The 17 consecutive months of on-schedule deliveries, program office identified the imaging infrared seeker increasing the inventory to over 850 combat ready with the autonomous acquisition software as the only assets. In addition, program officials noted that the Air critical technology for the system. The seeker was not Force has upgraded its JSOW inventory to mission mature at the start of development, but it did ready as a result of a successful resolution of demonstrate maturity in October 2001—over three- remaining manufacturing, navigation, and vibration fourths through development—when it was flown tolerance issues. The JSOW unitary continues aboard an aircraft in a captive flight test. Program development and its performance is being monitored officials stated that in three free-flight tests, the by the program office. seeker's performance substantially exceeded requirements. Design Maturity The JSOW unitary variant’s basic design appears complete. At the system design review in May 2002, the program office had completed 99 percent of the drawings. The Navy included nine developmental tests in its development program—three sled tests with the warhead, three free flights with the seeker, and three combined warhead/seeker tests. The Navy has completed two of the warhead sled tests and the seeker free-flight tests. Production Maturity JSOW production maturity could not be determined because the contractor does not use statistical process controls to ensure that production processes are stable and units are produced with few, if any, defects. Rather, the contractor uses a process of post- production inspection to control production quality. The contractor collects this postproduction data on a factorywide basis that includes JSOW production but is not specific to it. According to the program office, the contractor delivered end items in the past that included manufacturing defects. The program office attributes these defects at least partially to suppliers and to reorganization and relocation of the prime contractor to Tucson, Arizona. To mitigate the risk of further manufacturing problems, the Navy has instituted a series of reviews of major suppliers. The Navy will conduct an additional production readiness review after the low-rate production is approved. Program officials report that the contractor is meeting its revised production schedule and that the scrap and rework rates remain low. Page 52 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: NPOESS National Polar-orbiting Operational Environmental Satellite System (NPOESS) The NPOESS is a joint National Oceanic and Atmospheric Administration (NOAA), DOD, and National Aeronautics and Space Administration satellite program to monitor the weather and environment. Current NOAA and DOD satellites will be merged into a single national system (NPOESS), with projected savings of at least $1.3 billion. The program consists of five segments: space; command, control, and communications; interface data processing; launch; and field terminals. Prime contractor: Northrop Grumman Space Technology Program office: Silver Spring, Md. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $544.4 million 8/02 9/02 change Procurement $0 million Research & development cost $4,029.2 $4,431.6 10.0 Quantity 0 satellites Procurement cost $1,155.6 $1,264.5 9.4 Total program cost $5,628.2 $6,183.4 9.9 Program unit cost $938.0 $1,030.6 9.9 Total quantities 6 6 0.0 Acquisition cycle time (months) 172 174 1.2 The NPOESS program entered product development in August 2002 with most of its technologies mature. The program also completed a significant portion of the engineering drawings well in advance of the design review; however, the total number has yet to be determined. Over 5 years ago, program officials considered the program to have several high-risk areas. Since then, officials have implemented several efforts, which are expected to reduce all program areas to low risk by the first NPOESS launch, currently scheduled for the 2008- 2009 time frame. Perhaps the most significant step taken to reduce risk was to put the pacing space sensor technologies into full development in advance of the satellite system itself. GAO-03-476 Acquisition Trends and Risks Page 53 Common Name: NPOESS Appendix I NPOESS Program radiometer suite, the cross-tracked infrared sounder, the advanced technology microwave sounder and the Technology Maturity ozone mapper/profiler suite. This satellite will provide the program office and the data processing centers The NPOESS spacecraft and the sensors under with an early opportunity to work with the sensors, development consist of 14 key technologies; twelve ground control, and data processing systems, thus were mature at the start of development in August allowing lessons learned to be incorporated into the 2002. NPOESS satellites. In 1997, the program office determined that the space segment had high cost and technical risks and that the Program Office Comments interface data processing segment and overall system The NPOESS integrated program office concurred integration effort had high cost, schedule, and with this assessment. technical risks. To reduce the risk to the data processing segment, two contractors selected for program definition and risk reduction each conducted four ground-based demonstrations of the data processing hardware and software components. Therefore, the program office expects the data processing segment to be relatively mature before product development. Program officials indicated that they achieved maturity by concentrating on the early development of key individual sensors. The acquisition strategy focused on maturing key sensor technologies using individual development contracts structured to demonstrate the maturity of each sensor through a component-level design review prior to the system- level design review. The two technologies that are not mature are needed for two key sensors—the cross- track infrared sounder and the conical microwave imager/sounder. However, program officials project that those two technologies will be mature by the system design review in 2005. Design Maturity Although the total number of engineering drawings has yet to be determined, program officials indicated that at least 52 percent of the 6,829 currently identified drawings were completed and released to manufacturing by the end of January 2003. Program officials further project that all of the currently identifiable drawings will be complete by the system design review in 2005. The program is taking advantage of a unique opportunity to demonstrate design maturity. The NPOESS Preparatory Project, a planned demonstration satellite, is to be launched in 2006, about 2 to 3 years before the first NPOESS satellite launch. The demonstration satellite is scheduled to carry four critical sensors—the visible/infrared imager Page 54 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: PAC-3 Patriot Advanced Capability 3 (PAC-3) Program The Army’s Patriot system is a long-range, high- medium altitude air and missile defense system. PAC-3 is designed to enhance the Patriot’s ability to detect and identify missiles and other targets, increase system computer capabilities and the number of missiles in each launcher, improve communications, and incorporate a new hit-to-kill missile. The PAC-3 system has two primary components, the fire unit and the missile. We assessed both components. Prime contractor: Raytheon (prime) Lockheed Martin (missile segment) Program office: Huntsville, Ala. FY 2003 dollars in millions Approved Latest Percent FY 2004 funding request: 2/95 12/02 change R&D $174.5 million Research & development cost $2,760.7 $4,476.2 62.1 Procurement $561.6 million Procurement cost $3,721.9 $7904.9 112.4 Quantity 108 missiles Total program cost $6,482.6 $12,381.2 91.0 Program unit cost $5.170 $10.326 99.8 Total quantities 1,254 1,199 -4.4 Acquisition cycle time 66 136 106.1 The PAC-3 program currently has only about one- fourth of its critical production processes under statistical control using best practice standards. Continuing problems with producing and testing the missiles are partially explained by the absence of process control and partially a consequence of maturing PAC-3’s design late in development. Technical and design challenges disrupted the early part of product development, causing cost and schedule increases and delays in attaining production knowledge. PAC-3’s basic design is now complete and the technology appears mature. However, the contractor must increase production earlier than planned because DOD decided to accelerate deliveries. This decision may present new production challenges because the contractor must find and train additional personnel. GAO-03-476 Acquisition Trends and Risks Page 55 Common Name: PAC-3 Appendix I Patriot PAC-3 Program performance was adversely affected by PAC-3 missile reliability and launch failures. According to program Technology Maturity officials, there were several anomalies caused by manufacturing practices, software, and test hardware. Although we did not assess the PAC-3 technologies However, they believe there are no systemic issues using technology readiness levels, the system's critical and the anomalies have been corrected. A flight test to technologies appear mature. However, a key validate these corrections is scheduled for the spring technology, the Ka band seeker, was particularly late of 2003. to mature. The seeker did not mature until 1999, close to the low-rate production decision. Problems The program has adopted an evolutionary acquisition experienced during development increased the approach, with production decisions every 2 to 3 seeker's cost by 76 percent and delayed the contractor years. In October 2002, DOD decided to buy 208 in attaining design and production knowledge. missiles covering the next 2 years. DOD plans to accelerate the production rate immediately by adding Design Maturity a second manufacturing shift and test equipment. Because production was not expected to be PAC-3’s basic design is complete, with 100 percent of accelerated to this level this early in production, the the drawings released to manufacturing. Only contractor must expeditiously find and train qualified 21 percent of the drawings were complete when the personnel. The accelerated plan requires additional program held its design review, which led to a number funding of $239 million for fiscal years 2003 and 2004. of problems. For example, the contractor attributed a $101 million cost increase to first-time manufacturing Program Office Comments problems, such as some subsystems not fitting together properly and some not passing ground or In commenting on a draft of this assessment, program environmental tests. These problems were a major officials stated that they believe production processes contributor to a 2-year schedule delay. To reduce are in control. Program officials stated that they have missile costs, the contractor has identified several meticulously and methodically examined every critical major design changes, which will be incorporated into process from a labor and inspection standpoint to help the design in 2004. ensure a consistent and quality product. Despite the less than fully successful operational tests, they also Production Maturity believe that they have the most successful development flight test program in the history of The program has 23 percent of the key manufacturing missile development. They provided technical processes used to assemble the missile and the seeker comments, which were incorporated as appropriate. under control. Production maturity has deteriorated from the 35 percent that was in control at the October 1999 low-rate production decision. A switch in the manufacturing facilities may have played a role. According to program officials, the program entered production before process control was emphasized to the contractor. The contractor is still having difficulties building the missile. For example, each seeker still needs to be reworked about three times on average before it passes quality inspections. Program officials have added quality tests of components, which have improved the situation, but the contractor has not yet demonstrated that these tests will eliminate the need for seeker rework in the future. Other Program Issues The Army conducted four operational tests in 2002; none were completely successful. The PAC-3 system defeated half of the targets in flight-testing. System Page 56 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: SBIRS High Space Based Infrared System (SBIRS) High SBIRS High will consist of a constellation of four satellites plus one spare, two sensors on a non- SBIRS satellite, and associated ground stations. SBIRS High is to provide missile warning and missile defense information and will be used to support the technical intelligence and battlespace characterization missions. The first launch of SBIRS High is scheduled for fiscal year 2007. Prime contractor: Lockheed Martin Program office: El Segundo, Cal. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D: $617.2 million 3/98 6/02 change Procurement: $95.4 million Research & development cost $3,378.4 $6,077.3 79.9 Quantity: 1 satellite Procurement cost $558.1 $1,417.4 154.0 Total program cost $4,127.0 $8,241.2 99.7 Program unit cost $825.4 $1,648.2 99.7 Total quantities 5 5 0.0 Acquisition cycle time (months) NA NA NA Note: NA = not available. The SBIRS High program’s critical technologies have demonstrated acceptable levels of maturity. This level of maturity follows many years of difficult development. The level of design stability is unknown since the contractor was unable to provide information on the total number of releasable drawings at specific milestones. Similarly, production maturity could not be determined because the contractor does not collect statistical control data. The SBIRS High program is building the first two satellites using research and development funding with a first launch expected in fiscal year 2007. The program also recently underwent a major restructuring to reduce program risk. GAO-03-476 Acquisition Trends and Risks Page 57 Common Name: SBIRS High Appendix I SBIRS High Program 10 U.S.C. 2433) occurred on December 31, 2001, requiring a review by the Secretary of Defense and a Technology Maturity report to Congress. As a result, DOD certified to Congress in May 2002 that the SBIRS High program is The SBIRS High program’s three critical essential for national security, there are no technologies—the infrared sensor, thermal alternatives that provide equal or greater capability at management, and the on-board processor—are now less cost, cost estimates are reasonable, and the mature. Program officials indicated that the hardware management structure is in place to continue to keep was built and tested in a thermal vacuum chamber costs under control. under expected flight conditions. When the program began product development in 1996, none of its Program Office Comments critical technologies were mature, according to best practice standards. Program officials generally concurred with our assessment and provided technical comments, which Design Maturity we have incorporated where appropriate. Program officials added that the fiscal year 2004 budget fully Program officials do not know how many total funds their restructured program and directs the drawings are expected for SBIRS High, and thus do satellite procurement to begin in fiscal year 2006. not track the number of releasable drawings. As a result, we could not assess design stability relative to best practices. Program officials did state that the current number of releasable drawings is 2,342, about twice the number at the time of the design review. This means that at most, no more than half of the drawings could have been releasable at the design review. Design stability has been an issue for this program. During development, the satellite was redesigned to maintain key performance parameters. Redesign efforts resulted in a 6-month slip to the spacecraft and increased the requirement for ground processing. On the other hand, the two sensors that will be aboard non-SBIRS satellites are considered stable with subsystem qualification nearing completion, and integration and delivery of the flight payload are expected within the year. The first of these sensors is scheduled for delivery in May 2003—three months behind schedule. This delay is attributed to problems with radio waves emitted by the sensor's electronics that interfere with the host satellite. Despite these integration difficulties, data shows that the sensors will perform much better than expected. Production Maturity We could not assess the SBIRS High production maturity relative to best practice standards because the contractor does not use statistical process control to ensure that production processes are stable. Other Program Issues The total unit cost of the SBIRS High program rose more than 25 percent in 1 year. The notification to Congress of the Nunn-McCurdy breach (see Page 58 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: THAAD Theater High Altitude Area Defense (THAAD) THAAD is an element of the terminal defense segment of the Ballistic Missile Defense System. Its mission is to defend against short and medium range ballistic missiles. THAAD’s ability to intercept inside and outside the atmosphere makes effective countermeasures more difficult and allows multiple intercept opportunities. The system includes missiles, launchers, radars, command and control/ battle management (C2/BM), and THAAD support equipment. Prime contractor: Lockheed Martin Project office: Huntsville, Ala. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $730.6 million 1/92 2/03 change Procurement $0 million Research & development cost $4,382.7 $10,548.0 138.0 Quantity 0 missiles Procurement cost NA NA NA Total program cost NA NA NA Program unit cost NA NA NA Total quantities (U.S. only) NA NA NA Acquisition cycle time (months) 114 NA NA Note: Procurement schedule, funding, and quantities have yet to be determined. The THAAD schedule no longer includes production milestones. NA = not applicable. Most of THAAD’s critical technologies have demonstrated acceptable levels of maturity and the program appears close to meeting the best practice standard for a stable design. The program’s launcher and radar have essentially attained technological maturity, but the missile and the command and control/battle management components are somewhat less mature. This level of maturity follows many years of difficult development. It appears that the THAAD program has mostly recovered from initial problems driven by an early fielding requirement and poor quality control. The current THAAD acquisition strategy shows a much greater emphasis on attaining knowledge. The program expects to reach technological maturity and design stability by February 2004. GAO-03-476 Acquisition Trends and Risks Page 59 Common Name: THAAD Appendix I THAAD Program missile in the more stressing flight environment inside the atmosphere, and (3) prepares the system for initial Technology Maturity operational test and evaluation. THAAD program officials assessed 47 technologies in Program Office Comments four major elements—command and control/battle management; missile interceptor; launcher; and radar. In commenting on a draft of this assessment, program Of the four elements, the radar is currently the most officials stated that, to ensure the highest probability mature, followed by the launcher, command and of success in flight-testing, a substantial amount of control/battle management, and the missile. The ground testing is being conducted in the next year and program has made progress on technology maturity a half. This testing includes exhaustive engineering since it began development despite early failures in and qualification level testing on all flight components. intercept attempts. Early flight-test failures were Program officials further stated that the extensive caused by a combination of the compressed test design, fabrication, and test preparation activity has schedule and quality control problems. The program been very successful to date, and the program remains was restructured twice, before the first successful healthy with a slightly ahead-of-schedule and under- intercept occurred in 1999. The research and cost status. development cost grew from $4.4 to $10.5 billion prior to the program's transfer to the Missile Defense Agency, partially as a result of these problems. The current program strategy appears geared to obtaining the necessary knowledge by providing more time for maturing the technology before flight tests and placing greater emphasis on risk reduction efforts. This strategy includes utilizing technology readiness levels to assess technological maturity. Design Maturity The program has released about 82 percent of total drawings. Program officials expect to release about 91 percent of the drawings by the system-level design review in February 2004. The Missile Defense Agency is redesigning the missile to be more reliable and testable, with significantly fewer parts than the previous version. The first flight test of the redesigned missile is not scheduled to occur until at least 6 months after the system design review. Depending on the outcome, flight tests could require more design changes and delay achieving design stability. Other Program Issues THAAD was recently transferred from the Army to the Missile Defense Agency, which has restructured and modified the contract to a block upgrade approach. Therefore, limited information is currently available on the total projected costs of this program. In response to the prior program setbacks, the THAAD project office is accelerating some risk reduction activities, and it has planned a series of flight tests that (1) tests the missile in a less stressing intercept environment outside the atmosphere, (2) tests the Page 60 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: Tactical Tomahawk Tactical Tomahawk Missile The Navy’s Tactical Tomahawk (block IV) is a major upgrade to the Tomahawk Land Attack Missile (block III). The Tactical Tomahawk missile will provide ships and submarines with enhanced capability to attack targets on land. New features include improved antijamming global positioning system, in-flight retargeting, and ability to transmit battle damage imagery. The system includes the missile, the weapon control system, and the mission planning system. We assessed only the missile. Prime contractor: Raytheon Systems Company Program office: Patuxent River, Md. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $71.4 million 9/99 12/01 change Procurement $277.6 million Research & development cost $559.5 $584.1 4.4 Quantity 267 missiles Procurement cost $1,236.4 $1,546.0 25.0 Total program cost $1,795.9 $2,130.0 18.6 Program unit cost $1.316 $1.235 -6.1 Total quantities 1,365 1,725 26.4 Acquisition cycle time (months) 58 69 19.0 The Tactical Tomahawk missile entered low-rate production without ensuring that production processes were in control. Program officials indicated that they plan to collect production process control data over the next year, prior to award of the full-rate production contract in fiscal year 2004. At that time, program officials expect over 80 percent of the low-rate production missiles to be in various stages of assembly. The technology and design have reached acceptable levels of maturity. While engineering drawings have improved to 96 percent, the program only had about half of its drawings released at the design review. Program plans call for a full-rate production decision in May 2004. GAO-03-476 Acquisition Trends and Risks Page 61 Common Name: Tactical Tomahawk Appendix I Tactical Tomahawk Program awarded in mid- January 2003 for 167 units. Program officials stated that total quantities have increased to Technology Maturity 2,396. We did not assess the technology readiness levels of Program Office Comments the key technologies for the Tactical Tomahawk missile. At the time of our review, critical technologies In commenting on a draft of this assessment, program were mature. According to the program office, the officials stated that two development test flights, critical technologies for the key subsystems—antijam conducted prior to low-rate production awards, global positioning system, digital scene matching area demonstrated that the Tactical Tomahawk missile correlator, and cruise engine—were modified design met or exceeded technical and key derivatives from other programs or upgrades to performance parameters. They also noted that, due to existing Tomahawk subsystems and consequently the stability of the design and successful completion already mature. To date, subsystem and the majority of all component and flight qualification testing, the of missile-level qualification testing has been Navy’s operational test agency issued a favorable completed successfully. operational assessment, stating that the Tactical Tomahawk missile is potentially suitable and Design Maturity potentially operationally effective. The basic design of the Tactical Tomahawk missile is essentially complete. The critical design review occurred in June 2000. At that time, approximately 47 percent of the drawings had been released to manufacturing. In October 2002, at the first low-rate initial production award, 723 of 750 total drawings, or about 96 percent, had been released. Production Maturity Officials plan to collect statistical control data at the start of the manufacturing process but do not expect to have meaningful statistical data until sometime in 2004. Manufacture of the Tactical Tomahawk missile is scheduled to begin at the subcontractor's facility in 2003 and missile assembly in 2004. Although two low- rate production contracts have been awarded, program officials stated that data regarding manufacturing process controls currently is very limited. Program officials told us that it is too soon to know what percentage of critical manufacturing processes will be under statistical control when the full-rate production contract is awarded in mid-2004, but that they plan to start collecting production process control data over the next year. Other Program Issues The Tactical Tomahawk missile successfully completed its first developmental flight test in August 2002, and the first low-rate production contract for 25 units was awarded in October 2002. A second and final low-rate production contract was Page 62 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: V-22 V-22 Osprey The V-22 Osprey is a tilt-rotor, vertical takeoff and landing aircraft designed to meet the amphibious/ vertical assault needs of the Marine Corps, long- range missions of Special Operations forces, and combat search and rescue needs of the Navy. The V-22 will replace the CH-46E and the CH-53A/D in the Marine Corps; the H-53 and H-60 will augment the C-130 in the Air Force and the Special Operations Command; and supplement the H-60 in the Navy. We assessed the block A version. Prime contractor: Bell-Boeing Program office: Patuxent River, Md. FY 2004 funding request: FY 2003 dollars in millions Approved Latest Percent R&D $543.9 million 2/87 12/01 change Procurement $1.11 billion Research & development cost $3,568.5 $10,253.5 187.3 Quantity 11 aircraft Procurement cost $29,499.2 $32,312.7 9.5 Total program cost $33,264.7 $42,617.5 28.1 Program unit cost $36.434 $93.051 155.4 Total quantities 913 458 -49.8 Acquisition cycle time (months) 117 261 123.1 The V-22 program plans to enter full-rate production without ensuring that the manufacturing processes are mature. Redesign of the aircraft’s hydraulic and electric system, and software changes have been made to address safety, reliability, maintainability, and logistics supportability. These design changes and others are undergoing developmental testing to ready the aircraft for an operational test and evaluation test period in late 2004 through early 2005 to determine if the V-22 is operationally suitable and effective. The design changes, however, have not been incorporated into the low- rate production aircraft currently being produced. The value of contract modifications needed to address the cost of these design changes is not yet known. Also, parts shortages and quality issues are currently effecting low-rate production costs. Some key performance requirements have been eliminated. GAO-03-476 Acquisition Trends and Risks Page 63 Common Name: V-22 Appendix I V-22 Program and incorporated into production aircraft. Delivery of block A aircraft is expected to start in the fourth Technology Maturity quarter of fiscal year 2003. However, the cost of contract modifications needed to reconfigure already Although we did not specifically assess the V-22’s produced aircraft and aircraft still on the assembly technology maturity, the program office believes key line to the bock A configuration has not been technologies to be mature. An operational test report, negotiated. dated November 2000, determined that the V-22 was not operationally suitable because of poor reliability, Program Office Comments maintainability, availability, human factors, and interoperability problems. Immature technology, in In commenting on draft of this assessment program part, contributed to this assessment. officials stated that they have restructured the program to gather more technical knowledge through Design Maturity a more rigorous “event-driven” flight test program. Program officials strongly disagreed that the program As a result of a crash in December 2000, the V-22 has plans to enter full-rate production without ensuring undergone several design changes. Specifically, the aircraft’s hydraulic and electrical lines were that manufacturing process are mature. V-22s are currently being manufactured at a minimum redesigned to improve safety, reliability, sustaining rate (11 aircraft per year). A May 20th maintainability, and logistics supportability. The V-22 flight control system software was also redesigned. defense acquisition board review is scheduled to consider increasing this rate. Manufacturing processes The program office estimates that redesign of the V-22 and tooling are in place and being continually analyzed resulted in 1,755 additional drawings, increasing the and improved. Both companies utilize statistical total number of drawings to 7,490. To date, all of these drawings are complete. process control techniques and numerous metrics to assess program performance. They do not use the The success of these design changes will be process capability index, the only metric that GAO determined as the aircraft undergoes additional uses as a basis for their assessment. Program officials developmental testing through 2005. Testing will are also undertaking an affordability review to reduce address many issues, including high rate of descent, the aircraft unit cost to $58 million by 2010. High unit handling qualities, austere environment operations, costs are driven by the current low production and ship operations. The operational assessment of quantities and will remain the norm until production these characteristics will not occur until late 2004 or quantities increase. early 2005. Recent decisions to defer some V-22 operational requirements previously considered GAO Comments critical until later blocks will void the need for some design changes in the block A. Our prior work has shown that leading commercial firms rely on statistical control data, specifically, the Production Maturity process capability index, as the best indicator of production readiness. The V-22 program entered low- Neither V-22 contractor collects statistical process rate production without this information and has control data on its critical manufacturing processes. A experienced production quality problems. recent program management assessment rated V-22 production as cautionary. Part shortages and quality problems caused inefficiencies in shop and assembly operations, as well as scrap, rework, repair, and schedule delays. Other Program Issues Low-rate production of the V-22 continues. V-22s are being fabricated and partially assembled, but not delivered until the first set of upgrades—referred to as block A—needed to bring the V-22 to a safe operational and suitable configuration are approved Page 64 GAO-03-476 Acquisition Trends and Risks Appendix I Common Name: WGS Wideband Gapfiller Satellite (WGS) Communications System The Wideband Gapfiller Satellite system is a joint Air Force and Army program intended to provide communications to the U.S. warfighters, allies, and Coalition Partners during all levels of conflict short of nuclear war. It is the next generation wideband component in the DOD’s future Military Satellite Communications architecture. Prime contractor: Boeing Satellite Systems (BSS) Program office: El Segundo, Calif. FY 2004 funding request: FY 2003 dollars in millions Estimate Latest Percent R&D $36.7 million 12/01 12/02 change Procurement $34.6 million Research & development cost $181.2 $244.8 35.1 Quantity 0 satellites Procurement cost $831.5 $1,389.8 67.2 Total program cost $1,012.7 $1,634.6 61.4 Program unit cost $337.6 $326.9 -3.2 Total quantities 3 5 66.7 Acquisition cycle time (months) 50 55 10.0 The WGS program’s critical technologies and design are mature, while its production processes are nearly mature. DOD plans to rely on commercial technologies that will not require extensive product development. However, two of these processes use statistical control rates that are below the level prescribed by best practice standards. The program recently added two satellites to better support intelligence, surveillance, and reconnaissance missions in the future. GAO-03-476 Acquisition Trends and Risks Page 65 Common Name: WGS Appendix I WGS Program The WGS program’s two critical technologies—the digital channelizer and the phased array antenna—are mature. Most of these technologies are commercial derivatives. For this reason, many of the satellite technologies selected were already at high levels of maturity. In fact, the program is leveraging commercial technology and practices by modifying commercial satellites to better support unique military requirements. Design Maturity The WGS design is essentially complete, as the program has released approximately 95 percent of the expected drawings. Production Maturity The contractor has six of its eight key manufacturing processes under control, according to the best practice standards. Program officials indicated that they are bringing the remaining processes under statistical control. Program Office Comments In commenting on a draft of this assessment, program officials stated that while critical technology areas being applied to WGS are fairly mature, the manufacturing of the systems using these technologies is relatively new for the contractor. Risk of production problems was to be reduced due to other commercial satellite system developments and production ahead of WGS in the development and production schedule. However, due to the drastic loss of commercial satellite orders, only one commercial satellite with similar technologies as WGS is now leading WGS in the manufacturing schedule. Recently identified problems found on the “leader” program will impact the WGS manufacturing schedule, and a first launch schedule delay of 4 to 6 months can be expected due to time needed to resolve the “leader” program manufacturing problems. Satellites four and five have been directed by DOD to be launched in fiscal year 2009 and fiscal year 2010, respectively. These dates are outside the allowable dates of the WGS contract option clauses and will require renegotiation to finalize their cost. The cost is expected to increase to compensate for loss of learning curve from over a 3- year break in production, parts obsolescence, and inflation. Page 66 GAO-03-476 Acquisition Trends and Risks Appendix II Methodology Appendx Ii In conducting our work, we evaluated performance and risk data from each of the programs included in this report. We summarized our assessments of each individual program in two components—a system profile and a product knowledge assessment. We did not validate or verify the data provided by DOD. However, we took several steps to address data quality. Specifically, we reviewed the data and performed various quality checks, which revealed some discrepancies in the data. We discussed these discrepancies with program officials and adjusted the data accordingly. System Profile In the past 3 years, DOD revised its policies governing weapon system acquisitions and changed the terminology used for major acquisition Assessment events. In order to make DOD’s acquisition terminology more consistent across the 26 program assessments, we standardized the terminology for key program events. In the individual program assessments, program start refers to the initiation of a program; DOD usually refers to program start as milestone I or milestone A, which begins the concept and technology development phase. Similarly, development start refers to the commitment to product development that coincides with either milestone II or milestone B, which begins DOD’s system development and demonstration phase. The production decision generally refers to the decision to enter the production and deployment phase, typically with low-rate initial production. Initial capability refers to the initial operational capability, sometimes also called first unit equipped or required asset availability. The funding request information presented refers to the President’s fiscal year 2004 budget request, except where noted. The program cost comparisons are the latest estimates provided by the individual programs. The quantities listed refer to total quantities, including both procurement and development quantities. To assess the cost, schedule, and quantity changes of each program, we reviewed DOD’s selected acquisition reports or obtained data directly from the program offices. In general, we compared the latest available selected acquisition report information with a baseline for each program. For systems that have started product development—those that are beyond milestone II or B—we compared the latest available Selected Acquisition Report to the development estimate from the first Selected Acquisition Report issued after the program was approved to enter development. For systems that have not yet started product development, we compared the latest available data to the planning estimate issued after milestone I or A. For systems not included in selected acquisition reports, we attempted to Page 67 GAO-03-476 Acquisition Trends and Risks Appendix II Methodology obtain comparable baseline and current data from the individual program offices. All cost information is presented in base year 2003 dollars, unless otherwise noted, using Office of the Secretary of Defense approved deflators to eliminate the effects of inflation. We have depicted only the programs’ main elements of acquisition cost—research and development, and procurement, however the total program costs displayed also include military construction and acquisition operation and maintenance costs. Because of rounding and these additional costs, in some situations the total cost may not match the exact sum of the research and development and procurement costs. The program unit costs are calculated by dividing the total program cost by the total quantities planned. These costs are often referred to as program acquisition unit costs. The schedule assessment is based on acquisition cycle time, defined as the number of months between the program start, usually milestone I or A, and the achievement of initial operational capability or an equivalent fielding date. The intent of these comparisons is to provide an aggregate or overall picture of a program’s history. These assessments represent the sum total of the federal government’s actions on a program, not just those of the program manager and the contractor. DOD does a number of detailed analyses of changes that attempt to link specific changes with triggering events or causes. Our analysis does not attempt to make such detailed distinctions. Product Knowledge To assess the product development knowledge of each program at key points in development, we submitted a data collection instrument to each Assessment program office. The results are graphically depicted in each two-page assessment. The methodology used to generate each graph is discussed at the beginning of appendix I. We also reviewed pertinent program documentation, such as the operational requirements document, the acquisition program baseline, test reports, and major program reviews. To assess technology maturity, we asked program officials to apply a tool, referred to as technology readiness levels, for our analysis. The National Aeronautics and Space Administration originally developed technology readiness levels, and the Army and Air Force Science and Technology research organizations use them to determine when technologies are ready Page 68 GAO-03-476 Acquisition Trends and Risks Appendix II Methodology to be handed off from science and technology managers to product developers. Technology readiness levels are measured on a scale of one to nine, beginning with paper studies of a technology’s feasibility and culminating with a technology fully integrated into a completed product. Our best practices work has shown that a technology readiness level of 7— demonstration of a technology in an operational environment—is the level of technology maturity that constitutes a low risk for starting a product development program. In most cases, we did not validate the program offices’ selection of critical technologies or the determination of the demonstrated level of maturity. We sought to clarify the technology readiness levels in those cases where information existed that raised concerns. If we were to conduct a detailed review, we might adjust the critical technologies assessed, the readiness level demonstrated or both. It was not always possible to reconstruct the technological maturity of a weapon system at key decision points after the passage of many years. To assess design maturity, we asked program officials to provide the percentage of engineering drawings completed or projected for completion by the design review, the production decision, and as of our current assessment. Completed engineering drawings were defined as the number of drawings released or deemed releasable to manufacturing that can be considered the “build to” drawings. To assess production maturity, we asked program officials to identify the number of critical manufacturing processes and, where available, to quantify the extent of statistical control achieved for those processes. We used a standard called the Process Capability Index, which is a process performance measurement that quantifies how closely a process is running to its specification limits.1 The index can be translated into an expected product defect rate and we have previously found it to be a best practice. We sought other data, such as scrap and rework trends in those cases where quantifiable statistical control data was unavailable. 1 Process Capability Index provides assurance that production processes are under 100 percent statistical control. A high index value equates to fewer defects per part based on statistical process control data. The general rule of thumb used by the manufacturing industry states that if the index value for a process is less than 1.33, then the process is not capable of producing a part with acceptable consistency. Page 69 GAO-03-476 Acquisition Trends and Risks Appendix II Methodology Although the knowledge points provide excellent indicators of potential risks, by themselves, they do not cover all elements of risk that a program encounters during development, such as funding instability. Our detailed reviews on individual systems normally provide for a fuller treatment of risk elements. Page 70 GAO-03-476 Acquisition Trends and Risks Appendix III GAO Contact and Acknowledgments Appendx iI GAO Contact Paul Francis (202) 512-2811 Acknowledgments David B. Best, James A. Elgas, and James L. Morrison made key contributions to this report and were largely supported by GAO’s Acquisition and Sourcing Management staff. The following staff were responsible for individual programs. Staff System Michael Hazard Advanced Amphibious Assault Vehicle (AAAV) Steven Martinez Advanced Extremely High Frequency (AEHF) Communications Satellite David Hubble Advanced Wideband Satellite/Transformational Communications Satellite (AWS/TSAT) Gaines Hensley AN/APG-79 Active Electronically Scanned Array (AESA) Radar Marvin Bonner AIM-9X Short-Range Air-to-Air Missile Thomas Gordon Airborne Laser (ABL) Dana Solomon/Carrie Advanced Threat Infrared Wilson/Danny Owens Countermeasures/Common Missile Warning System (ATIRCM/CMWS) Johanna Ayers Cooperative Engagement Capability (CEC) Leon Gill CH-47F Improved Cargo Helicopter Wendy Smythe RAH-66 Comanche Marti Dey/Ron Schwenn EX-171 Extended Range Guided Munition (ERGM) Larry Gaston Excalibur Artillery Round Cheryl Andrew F/A-18E/F Super Hornet Donald Springman F-22 Raptor Beverly Breen/Lynn Joint Air-to-Surface Standoff Missile (JASSM) Lavalle Danny Owens Joint Common Missile Rae Ann Sapp/Art Cobb Joint Primary Aircraft Training System (JPATS) Brian Mullins/Ron F-35 Joint Strike Fighter (JSF) Schwenn Page 71 GAO-03-476 Acquisition Trends and Risks Appendix III GAO Contact and Acknowledgments Staff System Carol Mebane Joint Standoff Weapon (JSOW) Bruce Thomas National Polar-orbiting Operational Environmental Satellite System (NPOESS) Matthew Lea Patriot Advanced Capability 3 Program (PAC-3) Tana Davis Tactical Tomahawk Missile William Lipscomb/Tana Theater High Altitude Area Defense (THAAD) Davis Maricela Cherveny/Nancy Space Based Infrared Satellite-High (SBIRS- Rothlisberger High) Jerry Clark V-22 Osprey Art Gallegos/Tony Wideband Gapfiller Satellite Communication Beckham System (120185) Page 72 GAO-03-476 Acquisition Trends and Risks GAO’s Mission The General Accounting Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- GAO Reports and text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents Testimony using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO Mailing Lists” under “Order GAO Products” heading. Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548 To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 512-6061 To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in E-mail: email@example.com Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548 United States Presorted Standard General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. GI00 Official Business Penalty for Private Use $300 Address Service Requested
Defense Acquisitions: Assessments of Major Weapon Programs
Published by the Government Accountability Office on 2003-05-15.
Below is a raw (and likely hideous) rendition of the original report. (PDF)