United States Government Accountability Office GAO Report to the Committee on Armed Services, U.S. Senate WEAPONS December 2012 ACQUISITION REFORM Reform Act Is Helping DOD Acquisition Programs Reduce Risk, but Implementation Challenges Remain GAO-13-103 December 2012 WEAPONS ACQUISITION REFORM Reform Act Is Helping DOD Acquisition Programs Reduce Risk, but Implementation Challenges Remain Highlights of GAO-13-103, a report to Committee on Armed Services, U.S. Senate Why GAO Did This Study What GAO Found For the past 3 years, DOD has been The Department of Defense (DOD) has taken steps to implement fundamental implementing the Reform Act Weapon Systems Acquisition Reform Act of 2009 (Reform Act) provisions, requirements which are aimed at including those for approving acquisition strategies and better monitoring helping weapon acquisition programs weapon acquisition programs. DOD is also continuing to take additional steps to establish a solid foundation from the strengthen policies and capabilities. Some provisions, such as issuing guidance start. This helps to prevent cost for estimating operating and support costs, are being implemented. growth, thus helping the Defense dollar go further. This is the third in a series GAO’s analysis of 11 weapon acquisition programs showed the Reform Act has of GAO reports on the Reform Act. reinforced early attention to requirements, cost and schedule estimates, testing, and reliability. For example, prior to starting development, an independent review GAO was asked to determine team raised concerns about the Ground Combat Vehicle program’s many (1) DOD’s progress in implementing Reform Act provisions; (2) the impact requirements and the risks associated with its 7-year schedule. Subsequently, the Reform Act has had on specific the Army reduced the number of requirements by about 25 percent and acquisition programs; and prioritized them, giving contractors more flexibility in designing solutions. In (3) challenges remaining. To do this, addition, the developmental test and evaluation office—resulting from the Reform GAO analyzed documents and Act—used test results to help the Joint Light Tactical Vehicle program develop a interviewed officials from the four OSD more realistic reliability goal and a better approach to reach it. Shown below are offices created as a result of the areas where the Reform Act influenced several programs in GAO’s review. Reform Act, other DOD offices, the military services, and 11 weapon Reform Act Influence on Select Case Study Programs acquisition programs we chose as Program Requirements Cost and schedule Testing Reliability case studies. Case study programs Ohio Class Replacement were selected based on their Ground Combat Vehicle development status and interaction Joint Light Tactical Vehicle with the four OSD offices. Results Ship to Shore Connector cannot be generalized to all DOD KC-46 Tanker weapon acquisition programs. Source: GAO analysis of DOD data. What GAO Recommends While DOD has taken steps to implement most of the fundamental Reform Act GAO recommends DOD develop provisions, some key efforts to date have been primarily focused on DOD’s additional cost estimating and Reform largest weapon acquisition programs. DOD faces five challenges—organizational Act implementation guidance; make capability constraints, the need for additional guidance on cost estimating and lessons learned available to the acquisition community; and assess the Reform Act implementation, the uncertainty about the sufficiency of systems adequacy of the military services’ engineering and developmental testing resources, limited dissemination of systems engineering and lessons learned, and cultural barriers between the Office of the Secretary of developmental testing workforce. DOD Defense (OSD) and the military services—that limit its ability to broaden the generally concurred with the Reform Act’s influence to more programs. Service officials believe additional recommendations. GAO clarified one guidance is needed to improve their cost estimates and other implementation recommendation to make it clear that efforts. They also believe that lessons learned from programs that experience DOD needs to designate an office(s) significant cost and schedule increases should be shared more broadly within the within the Acquisition, Technology and acquisition community. These challenges seem straightforward to address, but Logistics organization to provide they may require more resources, which have been difficult to obtain. Ensuring practical Reform Act implementation the services have key leaders and staff dedicated to systems engineering and guidance to program offices. developmental testing activities, such as chief engineers at the service level and technical leads on programs, as well as breaking down cultural barriers are more View GAO-13-103. For more information, difficult to address. They will require continued monitoring and attention by the contact Michael J. Sullivan at (202) 512-4841 or firstname.lastname@example.org. Under Secretary for Acquisition, Technology and Logistics, service acquisition executives, and offices established as a result of the Reform Act to address. United States Government Accountability Office Contents Letter 1 Background 4 DOD Has Implemented Most of the Fundamental Reform Act Provisions and Is Strengthening Acquisition Activities 8 The Reform Act Is Helping Programs Identify and Mitigate Risks Earlier in the Acquisition Process 14 Challenges Exist That Could Limit the Reform Act’s Ability to Influence Systemic Change 22 Conclusions 28 Recommendations for Executive Action 30 Agency Comments and Our Evaluation 30 Appendix I Objectives, Scope, and Methodology 33 Appendix II Progress of Reform Act Offices in Implementing Weapon System Acquisition Reform Act Provisions 36 Appendix III Comments from the Department of Defense 38 Tables Table 1: Programs Selected for Case Study Review 3 Table 2: Key Responsibilities of Offices Established as a Result of the Reform Act 5 Table 3: Reform Act Influence on Case Study Programs 15 Table 4: List of Programs Selected for Case Study Review 34 Table 5: Implementation of Select Reform Act Provisions- Systems Engineering 36 Table 6: Implementation of Select Reform Act Provisions- Developmental Test and Evaluation 36 Table 7: Implementation of Select Reform Act Provisions- Cost Assessment and Program Evaluation 37 Table 8: Implementation of Select Reform Act Provisions- Performance Assessments and Root Cause Analyses 37 Page i GAO-13-103 Weapons Acquisition Reform Figures Figure 1: DOD’s Progress in Implementing Four Fundamental Reform Act Provisions 10 Figure 2: Revised Review Process for Major Defense Acquisition Programs 12 Abbreviations CAPE Cost Assessment and Program Evaluation DOD Department of Defense DT&E Developmental Test and Evaluation HMS Handheld, Manpack, and Small Form Fit OSD Office of the Secretary of Defense PARCA Performance Assessments and Root Cause Analyses SE Systems Engineering This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-13-103 Weapons Acquisition Reform United States Government Accountability Office Washington, DC 20548 December 14, 2012 The Honorable Carl Levin Chairman The Honorable John McCain Ranking Member Committee on Armed Services United States Senate For several decades, GAO has reported on poor outcomes encompassing cost and schedule growth on the Department of Defense’s (DOD) major weapon acquisition programs. Many problems can be traced to a culture where the military services begin programs with inflexible requirements, immature technologies, and overly optimistic cost and schedule estimates. Given pressures to reduce spending across the government, including DOD, finding ways to prevent or mitigate cost growth is crucial to U.S. national security. A solid program foundation using good developmental testing and systems engineering, and reliable cost estimates is needed in order to help avoid cost overruns and promote better acquisition outcomes. There have been numerous attempts in the past to improve DOD acquisition outcomes, including the Packard Commission and the Goldwater-Nichols Act in the 1980s and Federal Acquisition Streamlining Act of 1994.1 More recently, Congress passed the Weapon Systems Acquisition Reform Act of 2009 (Reform Act)2 to improve the way weapon systems are acquired and avoid cost and schedule overruns. In 2009, the Senate Armed Services Committee asked us to begin reporting on DOD’s implementation of Reform Act provisions and the impact the Reform Act has had on weapon acquisition programs. This is our third report addressing these topics. The first report focused on DOD’s initial efforts to implement Reform Act provisions for systems engineering and developmental testing, including the placement of new 1 Final Report to the President by the President’s Blue Ribbon Commission on Defense Management (June 1986); Goldwater-Nichols Department of Defense Reorganization Act of 1986, Pub. L. No. 99-433; and Pub. L. No. 103-355 (FASA). 2 Pub. L. No. 111-23, as amended by the Ike Skelton National Defense Authorization Act for Fiscal Year 2011, Pub. L. No. 111-383 §§ 813 and 1075, and the National Defense Authorization Act for Fiscal Year 2012, Pub. L. No. 112-81 §§ 819 and 837. Page 1 GAO-13-103 Weapons Acquisition Reform offices for these activities within the Office of the Secretary of Defense (OSD).3 Our second report examined the challenges the services face as they try to strengthen systems engineering and developmental testing activities on weapon acquisition programs.4 This report examines (1) DOD’s progress in implementing Reform Act provisions; (2) the impact the Reform Act has had on specific acquisition programs; and (3) challenges remaining in improving the weapons acquisition process. To determine DOD’s progress in implementing Reform Act provisions we interviewed officials and analyzed documentation from the four offices whose leadership was established within OSD as a result of the Reform Act—Systems Engineering (SE), Developmental Test and Evaluation (DT&E), Cost Assessment and Program Evaluation (CAPE), and Performance Assessments and Root Cause Analyses (PARCA).5 We focused our review on the offices’ implementation of four fundamental Reform Act provisions: developing policy and guidance; approving acquisition documents; monitoring programs and conducting program assessments; and developing performance measures. We also analyzed documentation we collected on selected weapon acquisitions. To determine the Reform Act’s impact on specific defense acquisition programs, we chose as case studies, 11 weapon systems at various points in the development process based on recommendations by OSD officials and GAO’s previous evaluations of these programs. At the time of our case study selection, all of the weapon systems in our review had been given Milestone A approval. This approval allows the programs to start the technology development phase of the acquisition process. Four of the 11 weapon systems had not yet received Milestone B approval. This approval signifies the start of engineering and manufacturing development activities. We believe these four programs offer the best insight into how the OSD offices and Reform Act policies are influencing 3 GAO, Defense Acquisitions: DOD Needs to Develop Performance Criteria to Gauge Impact of Reform Act Changes and Address Workforce Issues, GAO-10-774 (Washington, D.C.: July 29, 2010). 4 GAO, Weapons Acquisition Reform: Actions Needed to Address Systems Engineering and Developmental Testing Challenges, GAO-11-806 (Washington, D.C.: Sept. 19, 2011). 5 The Director of Systems Engineering and the Director of Developmental Test and Evaluation offices were originally established as a result of the Reform Act. These offices have since been renamed as the offices of the Deputy Assistant Secretary of Defense for Systems Engineering and the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation. Page 2 GAO-13-103 Weapons Acquisition Reform acquisition strategies. The other seven programs were conducting engineering and manufacturing development activities or had been given Milestone C approval to begin production and deployment phase activities. Each of these programs provided different insights in their interactions with the OSD offices, but some Reform Act provisions, such as conducting preliminary design reviews prior to Milestone B, may not apply to these programs since they were beyond development start. A complete list of programs we reviewed is provided below. Table 1: Programs Selected for Case Study Review Programs that had not started development at the time of our case study selection System development start date Ground Combat Vehicle December 2013 (estimated) a Joint Light Tactical Vehicle August 2012 Ohio Class Replacement submarine August 2016 (estimated) a Ship to Shore Connector July 2012 Programs that were in development at the time of our case study selection Global Hawk unmanned aerial vehicle March 2001 Gray Eagle unmanned aerial vehicle April 2005 Joint Strike Fighter October 2001 KC-46 Tanker aircraft February 2011 Littoral Combat Ship Seaframe February 2011 Remote Minehunting System December 1999 Small Diameter Bomb Increment II July 2010 Source: GAO analysis. a During the course of our review, the Joint Light Tactical Vehicle and Ship to Shore Connector programs held a Milestone B review and received approval to start development. To determine challenges remaining for implementing the Reform Act, we conducted our own assessment and interviewed the leaders of the OSD offices established as a result of the Reform Act, military service acquisition chiefs, and program managers. These officials are responsible for advising and overseeing weapon acquisition program development, funding and developing new weapon acquisition programs, and executing the day-to-day development plans, respectively. We also solicited the opinions of the Under Secretary of Defense for Acquisition, Technology and Logistics and other senior OSD leaders. Additional information about our scope and methodology can be found in appendix I. We conducted this performance audit from January 2012 to December 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to Page 3 GAO-13-103 Weapons Acquisition Reform obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In May 2009, Congress passed the Weapon Systems Acquisition Reform Background Act of 2009 (Reform Act) in an effort to improve the way weapon systems are acquired and avoid further cost overruns on such programs. When signing the Reform Act into law, the President stated that its purpose was to limit weapon system cost overruns and that it would strengthen oversight and accountability by appointing officials who will closely monitor the weapons systems acquisition process to ensure that costs are controlled. Four offices were established as a result of the Reform Act: SE, DT&E, CAPE and PARCA. The SE and CAPE offices existed under other organizational titles prior to the Reform Act. Staffing levels, following the Reform Act, remained relatively stable for both of these offices, but some reorganization was necessary to reflect new Reform Act responsibilities. The DT&E and PARCA offices were newly established. The key roles and responsibilities of these four offices as outlined in the Reform Act are explained below: Page 4 GAO-13-103 Weapons Acquisition Reform Table 2: Key Responsibilities of Offices Established as a Result of the Reform Act Office Primary responsibilities Systems Engineering • serves as principal advisor to the Secretary of Defense and the Under Secretary of Defense for Acquisition, Technology and Logistics on systems engineering activities in the department • develops systems engineering and development planning guidance for DOD • reviews and approves major defense acquisition program systems engineering plans • monitors major defense acquisition program systems engineering activities • reports to Congress annually on systems engineering organization, capabilities, and activities Developmental Test and • serves as principal advisor to the Secretary of Defense and the Under Secretary of Defense Evaluation for Acquisition, Technology and Logistics on developmental test and evaluation activities • develops developmental test and evaluation guidance for DOD • reviews and approves major defense acquisition program developmental test and evaluation plans • monitors and reviews acquisition program developmental test and evaluation activities of major defense acquisition programs • reports to Congress annually on developmental test and evaluation organization, capabilities, and activities Cost Assessment and Program • serves as principal advisor to the Secretary of Defense and other senior officials on matters Evaluation related to cost analysis and the planning and programming phases of the planning, programming, budgeting, and execution system • develops independent cost estimates for major defense acquisition programs prior to major milestone decisions and updates independent cost estimates when programs have exceeded critical cost thresholds, known as Nunn McCurdy breaches • reviews existing systems and methods for tracking and assessing operation and support costs on major defense acquisition programs • develops analysis of alternative study guidance for major defense acquisition programs • approves the analysis of alternatives study plan for each major defense acquisition program Performance Assessments and • assesses major acquisition program performance through independent analyses and through Root Cause Analyses the Defense Acquisition Executive Summary process • identifies the root causes of cost growth and other problems on programs that experience a critical Nunn McCurdy cost breach Source: GAO Analysis of Weapon Systems Acquisition Reform Act of 2009. Each of the offices has varying levels of interaction with defense acquisition programs. For example, the SE and DT&E offices have ongoing interaction with acquisition programs throughout development in the form of program reviews, working group activities, and other program meetings. They also coordinate program documents in preparation for major milestone reviews. CAPE issues guidance to programs on how to conduct an analysis of alternatives at the beginning of the acquisition process. The office approves the analysis of alternative study plan that is Page 5 GAO-13-103 Weapons Acquisition Reform developed based on its guidance. It then develops independent cost estimates for major milestone reviews and in the event that an acquisition program experiences a Nunn-McCurdy breach.6 According to PARCA, it assesses all major defense acquisition programs at least once per quarter or when requested by the Under Secretary of Defense for Acquisition, Technology and Logistics, and disseminates this information to senior leaders. The office also interacts with specific programs if they experience a Nunn-McCurdy breach. In these cases, the office assesses program performance not less than semi-annually until 1 year after it receives a new milestone approval. In addition to the new organizational requirements, the Reform Act requires DOD to ensure that the acquisition strategy for major defense acquisition programs includes measures to ensure competition or the option of competition throughout the program life cycle. This could include strategies such as maintaining two sources for a system (dual-sourcing) and breaking requirements for supplies or services previously provided or performed under a single contract into separate smaller contracts (unbundling of contracts).7 Major defense acquisition programs are also required to provide for competitive prototyping—where two or more competing teams produce prototypes before a design is selected for further development—prior to Milestone B unless a waiver is properly 6 10 U.S.C. § 2433 establishes the requirement for the DOD to prepare unit cost reports on major defense acquisition programs or designated subprograms. If a program exceeds cost growth thresholds specified in the law, this is commonly referred to as a Nunn- McCurdy breach, which DOD is required to report to Congress and, if applicable, submit a certification to Congress in order to continue the program, in accordance with 10 U.S.C. § 2433a. 7 Pub. L. No. 111-23 § 202. Page 6 GAO-13-103 Weapons Acquisition Reform granted by the milestone decision authority,8 and to meet the following Milestone B certification requirements, including:9 • Appropriate trade-offs among cost, schedule, and performance objectives have been made to ensure the program is affordable; • A preliminary design review and formal post-preliminary design review assessment have been conducted and on the basis of such assessment the program demonstrates a high likelihood of accomplishing its intended mission; • Technology has been demonstrated in a relevant environment on the basis of an independent review and assessment by the Assistant Secretary of Defense for Research and Engineering; • Reasonable cost and schedule estimates have been developed to execute, with concurrence of the Director of CAPE, the program’s product development and production plan; • Funding is available to execute the program’s product development and production plan; • DOD has completed an analysis of alternatives with respect to the program; and • The Joint Requirements Oversight Council10 has approved program requirements, including an analysis of the operational requirements. 8 Pub. L. No. 111-23 § 203(a). Specifically, the Reform Act required DOD to modify its guidance relating to the operation of its acquisition system to incorporate these competitive prototyping provisions. DOD did so through Directive-Type Memorandum (DTM) 09-027, “Implementation of Weapon System Acquisition Reform Act of 2009 (Dec. 4, 2009, incorporating Change 3, Dec. 9, 2011). Major defense acquisition programs are those estimated by DOD to require an eventual total expenditure for research, development, test, and evaluation, including all planned increments, of more than $365 million, or for procurement, including all planned increments, of more than $2.19 billion in fiscal year 2000 constant dollars or designated by the Under Secretary of Defense for Acquisition, Technology, and Logistics. The Milestone Decision Authority for major defense acquisition programs is the Under Secretary of Defense for Acquisition, Technology and Logistics, the head of a DOD component, or if delegated the component acquisition executive. 9 Pub. L. No. 111-23; various sections, codified at 10 U.S.C. § 2366b. The Reform Act revised the Milestone B certification requirements for trade-offs, preliminary design, technology demonstration, and reasonable cost and schedule estimates. The remaining Milestone B certification requirements bulleted here were unrevised by the Reform Act. 10 The Joint Requirements Oversight Council is an advisory council to the Chairman of the Joint Chiefs of Staff with the responsibility to: (1) identify, assess, and approve joint military requirements; (2) assist acquisition officials in identifying alternatives to acquisition programs; and (3) assist the Chairman of the Joint Chiefs of Staff in assigning priority for joint military requirements. Page 7 GAO-13-103 Weapons Acquisition Reform The Reform Act also requires the Joint Requirements Oversight Council to ensure trade-offs among cost, schedule, and performance objectives are considered for joint military requirements.11 GAO previously reported that the Council considered trade-offs made by the military services before validating requirements, but the military services did not consistently provide high-quality resource estimates to the Council for proposed programs in fiscal year 2010. We also found that the Council did not prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner.12 DOD has implemented most of the fundamental Reform Act provisions as DOD Has required and is taking additional steps to strengthen acquisition reviews, Implemented Most of policies, and capabilities. Offices established as a result of the Reform Act are continuing to issue policies, review and approve relevant the Fundamental acquisition documents, monitor weapon acquisition program activities, Reform Act and develop performance measures. In addition, all four of the major defense acquisition programs we reviewed that had not started Provisions and Is development when we selected our case studies plan to implement Strengthening Reform Act provisions regarding preliminary design reviews, competitive prototyping, and competition. Also, some provisions, such as issuance of Acquisition Activities guidance on estimating operating and support costs, by the CAPE, are still in the process of being completed. Finally, we found that the Under Secretary of Defense for Acquisition, Technology and Logistics has revised the defense acquisition review process to consider additional knowledge collected on programs earlier and efforts are being made to strengthen acquisition policies and capabilities. DOD Is Continuing to The offices established as a result of the Reform Act—SE, DT&E, CAPE, Implement Reform Act and PARCA—are continuing to make progress in implementing four Provisions fundamental Reform Act provisions aimed at strengthening acquisition outcomes and oversight of weapon acquisition programs. Specifically, the offices are (1) developing policy and guidance to the military services for conducting work in their respective areas, (2) approving acquisition 11 Pub. L. No. 111-23 § 201. 12 GAO, DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews, GAO-11-502, (Washington, D.C.: June 16, 2011). Page 8 GAO-13-103 Weapons Acquisition Reform documents prior to milestone reviews, (3) monitoring and assessing weapon acquisition program activities on a consistent basis, and (4) developing performance measures to assess acquisition program activities. Figure 1 provides the status of DOD efforts to implement the four fundamental provisions. Some offices are still in the process of completing a few of these provisions. For example, CAPE and PARCA are in the process of developing policies and guidance for their respective areas and DT&E is in the process of establishing performance measures that can be used to assess weapon acquisition program activities. The office piloted the performance measures on two major defense acquisition programs and reported that they are currently applying them to over 40 programs. Note that some activities related to approving documents and monitoring or assessing programs require on-going efforts on the part of some of the offices. Page 9 GAO-13-103 Weapons Acquisition Reform Figure 1: DOD’s Progress in Implementing Four Fundamental Reform Act Provisions Interactive Graphic Click on status to show Reform Act provisions. Click on the “” to clear. See appendix II for the non-interactive, printer-friendly version. Monitor Develop programs/ Develop policy and Approve conduct performance Reform Act provisions Office guidancea documents assessments measures status Completing Completing Systems Completed on annual on annual Completed Engineering basis basis Developmental Completing Completing Test and Completed on annual on annual In process Evaluation basis basis Cost Assessment Completing Not Not and Program In process on annual applicable applicable Evaluation basis Performance Completing Assessment Not In process on annual In process and Root Cause applicable basis Analyses Source: GAO analysis of DOD data. a The Reform Act does not specify a date of completion for developing policy and guidance. b DOD Directive-Type Memorandum (DTM) 11-003, “Reliability Analysis, Planning, Tracking, and Reporting,” (Mar. 21, 2011); DOD DTM 10-017, “Development Planning to Inform Materiel Development Decision Reviews and Support of Analysis of Alternatives,” (Sept. 13, 2010, Incorporating Change 2, Dec. 9, 2011); DOD Instruction 5134.16, “Deputy Assistant Secretary of Defense for Systems Engineering (DASD(SE)),” (Aug. 19, 2011). Page 10 GAO-13-103 Weapons Acquisition Reform We also found evidence that major defense acquisition programs are integrating Reform Act provisions in their acquisition strategies. The four weapon acquisition programs we reviewed that had not started development activities when we began our review plan to implement Reform Act provisions related to preliminary design reviews, competitive prototyping, and competition. For example, the Ground Combat Vehicle has two contractors developing competitive prototypes of two key subsystems to support technology development. The program intends to conduct preliminary design reviews on both contractors’ designs prior to Milestone B and to conduct full and open competition through Milestone C. Similarly, according to program officials, the Joint Light Tactical Vehicle program had three contractors develop full-system prototypes during the technology development phase and held preliminary design reviews on each contractor’s design prior to Milestone B. The program plans to continue competition throughout engineering and manufacturing development. None of the four programs in our review received a waiver from Reform Act provisions. Additional Emphasis Is OSD is taking additional steps to strengthen the department’s oversight of Being Placed on Improving weapon acquisition programs and guidance for developing the programs. Milestone Review Process, In June 2011, for example, the Under Secretary for Acquisition, Technology and Logistics revised the weapons acquisition review process Policies, and Service to consider acquired knowledge on weapon acquisition programs earlier Capabilities than before. The revised review process includes two new review points. The first new review—the pre-engineering and manufacturing development review—occurs before the release of a final request for proposal for the engineering and manufacturing development phase. The purpose of this new review is to assess each program’s acquisition strategy, request for proposal, and key related planning documents earlier in the process, and to determine whether program plans are affordable, executable, and reflect sound business arrangements. The second new review—the acquisition strategy and request for proposals review and approval—occurs prior to Milestone C, the production decision. The review provides the milestone decision authority an opportunity to review the acquisition strategy and request for proposals for the production and deployment phase prior to Milestone C. Figure 2 illustrates the revised review process. Page 11 GAO-13-103 Weapons Acquisition Reform Figure 2: Revised Review Process for Major Defense Acquisition Programs According to the Under Secretary of Defense for Acquisition, Technology and Logistics, who is the authority for making milestone decisions for most major weapon acquisitions, the prior review process did not provide an adequate opportunity for review of program plans prior to release of the final request for proposals—the point at which DOD’s requirements, schedule, planned program content, and available funding should be firm and available for review. Further, the Under Secretary stated that making changes to acquisition strategies and program plans after all bidding activities, proposal evaluation, and source selection are complete is difficult and highly disruptive.13 DOD is also rewriting the DOD Instruction 5000.0214 to include an extensive restructure of acquisition policies according to the Under Secretary of Defense for Acquisition, Technology and Logistics. This update will implement Section 832 of the National Defense Authorization Act for Fiscal Year 2012,15 which requires DOD to issue guidance on actions to be taken to improve its processes for estimating, managing, and reducing operation and support cost, as well as ensure competition in maintenance and sustainment of subsystems of major weapon systems, among other things. In addition to current policies implementing the Reform Act, officials stated that key provisions from the Reform Act will also be included in the updated instruction. 13 DOD Principal Deputy Under Secretary of Defense for Acquisition, Technology and Logistics, “Improving Milestone Process Effectiveness,” (June 23, 2011). 14 DOD Instruction 5000.02, Operation of the Defense Acquisition System, Dec. 2, 2008. 15 Pub. L. No. 112-81 § 832. Page 12 GAO-13-103 Weapons Acquisition Reform In addition to implementing the provisions of the Reform Act, DOD offices have taken other steps to strengthen acquisition capabilities throughout the department. For example: • The SE office, according to DOD officials, led efforts to establish working groups to help the services address systemic reliability issues across the unmanned aircraft and rotary wing portfolios earlier in the process. The office also led several workforce development initiatives to attract and retain a qualified engineering workforce and supported the implementation of legislation requiring each acquisition program office to name a key technical advisor who is responsible for all engineering activities. • The DT&E office, according to DOD officials, championed updates to DOD Instructions that will require weapon acquisition programs to consider using DOD test capabilities before paying contractors to develop similar capabilities. In addition, the office supported legislation requiring major defense acquisition program offices to have a government test agency serving as the lead developmental test and evaluation organization for the program and a chief developmental tester. The chief developmental tester position, as included in the National Defense Authorization Act for Fiscal Year 2012, is tasked with coordinating the planning, management, and oversight of all developmental testing activities, among other things.16 • The CAPE office, according to DOD officials, established an operating and support cost directorate to build its expertise and place more emphasis on developing better operating and support cost estimates throughout the acquisition life cycle. This directorate will coordinate the development of an operating and support cost estimating guidebook. • The PARCA office, according to DOD officials, is providing additional insights to the Under Secretary of Defense for Acquisition, Technology and Logistics on systemic acquisition problems. Specifically, the office is examining a wide range of acquisition-related information from the past 40 years, such as contract type, stability of key performance parameters, and program manager tenure to 16 Pub. L. No. 112-81 § 835. Page 13 GAO-13-103 Weapons Acquisition Reform determine if there is any statistical correlation between these factors and good or poor acquisition outcomes. We identified four key areas where the Reform Act had a significant The Reform Act Is influence on programs in the 3 years since it was enacted: Helping Programs (1) requirements, (2) cost and schedule, (3) testing, and (4) reliability. These four areas have been common sources of problems in the past. Identify and Mitigate For example, the services typically started new weapon acquisition Risks Earlier in the programs with requirements that were both demanding and inflexible and planned to use relatively unproven technologies to meet the Acquisition Process requirements—all of which increased program risks. In addition, cost and schedule estimates were frequently too optimistic based on the proposed requirements and technologies. Design problems stemming from rigid requirements and the use of immature technologies to meet them were often discovered during testing and fixed late in the development cycle and resulted in cost increases, performance shortfalls, and schedule delays. Finally, DOD’s inattention to reliability has resulted in a dramatic increase in the number of systems that have not met suitability requirements during operational testing. Deficiencies—such as high failure rates and disappointing improvements in the reliability, availability, and maintainability of weapon systems—have limited program performance and increased operation and support costs.17 We examined 11 programs at various stages of the acquisition process to determine how the offices and policies established as a result of the Reform Act impacted their acquisition strategies and decision-making process. Four programs had not yet passed Milestone B, development start, at the time we began our review. Of the remaining seven programs, three had breached Nunn-McCurdy cost thresholds since the act was passed and have had to satisfy the Reform Act’s new requirements with regards to certification. The other programs had significant interaction with one or more of the OSD offices established by the Reform Act. Table 3 indicates in which of the four areas each program has been affected by the Reform Act. 17 DOD Report of the Defense Science Board Task Force on Developmental Test and Evaluation (May 1, 2008). Page 14 GAO-13-103 Weapons Acquisition Reform Table 3: Reform Act Influence on Case Study Programs Program Requirements Cost and schedule Testing Reliability Before Milestone B Ground Combat Vehicle a Joint Light Tactical Vehicle Ohio Class Replacement a Ship to Shore Connector After Milestone B Joint Strike Fighter Global Hawk Gray Eagle KC-46 Tanker Littoral Combat Ship Seaframe Remote Minehunting System Small Diameter Bomb II Source: GAO analysis of DOD data. a During the course of our review, the Joint Light Tactical Vehicle and Ship to Shore Connector programs held a Milestone B review. In some cases, programs have made changes based on input from OSD offices like systems engineering or developmental test and evaluation; in other cases, programs have integrated Reform Act policies, such as preliminary design reviews and competitive prototyping, into their acquisition planning. Programs that were already in development or production when the Reform Act was passed were less likely to have interactions with the OSD offices on requirements trades because these discussions typically occur prior to Milestone B. A discussion of how individual programs have been affected in the areas of requirements, cost and schedule realism, testing, and reliability follows. Greater Emphasis Is Being The Reform Act places significant emphasis on early problem solving and Placed on Requirements, requires programs to put much more effort toward considering trade-offs but Challenges Remain among cost, schedule, and performance requirements prior to Milestone B. As part of this effort, it requires the Secretary of Defense to ensure that acquisition, budget, and cost estimating officials have the opportunity to raise cost and schedule matters before performance objectives are established. The Reform Act also charges the Joint Requirements Oversight Council with the responsibility to ensure that cost, schedule, and performance trade-offs for joint military requirements are considered, Page 15 GAO-13-103 Weapons Acquisition Reform and to include combatant commanders in the process to ensure the user’s needs are adequately satisfied. The offices established as a result of the Reform Act have helped programs, such as the Joint Light Tactical Vehicle and Ground Combat Vehicle, make trade-offs among cost, schedule, and technical performance requirements. As a result, these programs have developed a more realistic acquisition strategy from a cost, schedule, and technical standpoint. • Joint Light Tactical Vehicle: The program held several reviews prior to Milestone B to identify, modify, or eliminate requirements that were unachievable or unaffordable, thus establishing a more technically realistic program. Officials from DT&E and SE participated in these reviews. By involving both the requirements and acquisition communities in the reviews, the Army was able to reduce the required capability to cut costs while ensuring that trade-off decisions would not impair the system’s ability to meet the warfighter’s operational needs. Examples of requirements changes that helped to cut costs as well as reduce risk include: • allowing the active suspension system, crew displays, and integrated starter-generator to be tradable design features. These changes resulted in a 30 percent reduction in the average unit manufacturing cost from the initial target of $475,000 to $331,000, while at the same time reducing technical and weight risk. According to program officials, this makes the $250,000 unit manufacturing goal more achievable. • reducing the reliability requirement and changing the Army helicopter lift requirement based on the results of technology development prototype testing. This mitigated technical risks going into development. The program recently moved into engineering and manufacturing development, but not all requirements issues have been resolved and future trade-offs may be necessary. For example, early testing showed that none of the three prototype variants met the program’s soft soil or sand slope requirement. This requirement has not been changed. Program and OSD officials are monitoring this issue closely and plan to actively manage it during engineering and manufacturing development. • Ground Combat Vehicle: The Ground Combat Vehicle program exhibited some of the same problems experienced by previous DOD programs prior to Milestone B—demanding and inflexible requirements. The SE office and the Under Secretary of Defense for Page 16 GAO-13-103 Weapons Acquisition Reform Acquisition, Technology and Logistics are helping the program set achievable requirements. Following its materiel development decision in February 2010, the program issued a request for proposals that contained nearly 1,000 requirements and a challenging 7-year schedule for the delivery of the first production vehicle. At the request of the Under Secretary for Acquisition, Technology and Logistics, the Army established an independent review team to assess the risks associated with the program’s schedule. The team, which included an SE official, raised concerns about the program’s high number of mandatory requirements and the risks associated with the 7-year schedule. To mitigate program risks, the Army reduced the number of performance requirements by about 25 percent and prioritized the others, giving competing contractors flexibility in addressing some performance requirements. The Army issued a revised request for proposal in November 2010. In August 2011, the Under Secretary of Acquisition, Technology and Logistics approved the program’s entry into technology development, but expressed concern about the cost and schedule risks associated with delivering a production vehicle in 7 years. Because of these concerns, the Under Secretary directed the Army to consider other alternatives, such as existing vehicles, that could meet warfighter needs. The analysis is currently planned to be completed in March 2013 to inform the Milestone B decision. Program Cost and By establishing a new cost assessment and program evaluation office Schedule Estimates Are and requiring this office to scrutinize program cost and schedule Becoming More Realistic estimates beginning at Milestone A, CAPE officials believe that the Reform Act has helped infuse more realism in cost estimates and promote earlier discussions about affordability. CAPE officials also believe that because their independent cost and schedule estimates have become more visible within DOD and Congress, the military services are developing more realistic estimates. We saw evidence of these benefits in the programs we reviewed, including the Ohio Class Replacement, Littoral Combat Ship Seaframe, and F-35 Joint Strike Fighter programs. • Ohio Class Replacement: The CAPE office was involved in the decision-making process to ensure program affordability. The office prepared an independent cost estimate and reviewed the program’s affordability goals prior to Milestone A. The service and independent estimates were within 2 percent of each other. However, the Under Secretary of Defense for Acquisition, Technology and Logistics directed the Navy to do a rigorous cost comparison of a 16 missile tube design versus a 20 missile tube design. The Navy determined that a 16 missile tube configuration would meet warfighter Page 17 GAO-13-103 Weapons Acquisition Reform requirements and users’ needs while reducing program costs by about $200 million per submarine, or approximately $3 billion for the total program. It would also simplify the ships’ design and integration effort. The CAPE office validated the savings associated with the 16 missile tube design. As a result, the Navy incorporated the 16 missile configuration as the program baseline. • Littoral Combat Ship Seaframe: The CAPE office helped make program costs more visible. Prior to the program’s Milestone B decision, CAPE completed an independent cost estimate of the seaframe program and found that the resources in the future years’ defense plan budget were lower than the projected program costs for the same time period. Navy officials attributed this problem to the overlap between the timing of the milestone decision and the president’s budget submission. The office further noted that the resources in the Navy’s budget did not include the additional development activities required to support two full ships. Without this information, decision makers would not have had visibility into the expected costs of the seaframe program or be able to make more fully informed decisions. As a result, the Navy re-phased its funding in the budget, adding approximately $397 million to fully fund the development program. Joint Strike Fighter: The SE and CAPE offices helped the program develop more realistic cost and schedule estimates. CAPE officials have been involved in reviews of the F-35 Joint Strike Fighter program even prior to the passage of the Reform Act and have continued to be heavily involved in subsequent programs reviews. For example, the cost analysis improvement group, which was the predecessor to CAPE, led a multi-functional joint estimating team review of the program in 2008. This review found problems with the program’s funding and schedule. In 2010, the Joint Strike Fighter program notified Congress that its estimated unit costs had increased by more than 80 percent since the original Milestone B baseline in 2001. This increase triggered a Nunn-McCurdy unit cost breach and later prompted the program executive officer to commission a technical baseline review of the program to help determine the resources needed to complete development. Officials from the SE office participated in this technical review. The CAPE office also did an independent cost estimate of the program as part of the Nunn- McCurdy certification process. Based on information from these efforts, DOD developed a more realistic program plan by adding $4.6 billion to the development program, reducing near-term procurement quantities by 125 aircraft, and extending the development test period Page 18 GAO-13-103 Weapons Acquisition Reform by 4 years to accommodate developmental testing, address the increased program scope, and fix software issues. Programs Are Adding More The Reform Act significantly strengthened the role of developmental Tests to Reduce testing in the department. In the 2 decades prior to the Reform Act, the Development Risks prominence of developmental testing had declined within OSD. In the early 1990’s, developmental testing was part of an all-encompassing test organization that reported directly to the Under Secretary of Defense for Acquisition. According to a former senior developmental testing official, by 2004, two people worked on developmental testing activities within the systems engineering organization. In establishing a separate office for DT&E, the Reform Act reinforced the need for robust developmental testing early in the acquisition process. The Reform Act gave the Deputy Assistant Secretary for Developmental Test and Evaluation formal approval authority for the test and evaluation master plans of major defense acquisition programs. This authority enables the Deputy Assistant Secretary to help ensure that programs have robust test and evaluation plans. Our case study reviews illustrate the efforts that the DT&E office has made to help programs such as the Small Diameter Bomb II and KC-46 Tanker obtain more design and performance knowledge early in a program’s acquisition life cycle. • Small Diameter Bomb II: Air Force program officials acknowledged that developmental and operational testing officials worked closely with them as they prepared the test and evaluation master plan for the Milestone B decision. After reviewing the plan, developmental and operational test officials concluded that the program would benefit from adding a 28-shot test program prior to entering operational testing. According to the program office, the purpose of the additional testing is to further establish the performance of the weapon in realistic scenarios and to increase the likelihood of completing operational testing without a failure. DT&E officials stated this testing would provide more complete knowledge about the bomb’s functionality and help reduce risk of a major redesign moving forward. Program officials stated that they allocated an additional $41 million to its developmental test program to conduct the 28 additional flight tests prior to operational testing. • KC-46 Tanker: The program office acknowledged that the DT&E office, as part of an integrated test team comprised of government and industry officials, helped identify options that could add time to the test plan for important testing if unexpected delays are encountered. In its fiscal year 2011 annual report, the operational test and Page 19 GAO-13-103 Weapons Acquisition Reform evaluation office reported that the KC-46’s planned flight test program was not executable, determining that more time would be needed for military flight-testing. It based this conclusion on the historical flight test experience of similar programs. Program officials stated that they were initially reluctant to change the test plan because they had awarded a fixed price contract and any changes could result in reopening the contract, leading to potential cost increases. However, the integrated test team identified a recovery period that may be applied to the KC-46 aerial refueling certification if delays are encountered. The contractor now has a plan that could allocate an additional 1.5 months for two test aircraft to complete this testing, if deemed necessary. This testing would provide more knowledge about the program’s aerial refueling performance prior to operational testing. DT&E officials stated that they plan to continue working with the program to address overall flight test challenges. While testing remains one of the program’s risk areas, this change may lessen that risk. Greater Emphasis Is Being The Reform Act emphasizes the need for designing more reliable weapon Placed on Weapon System systems. It charges the Deputy Assistant Secretary for Systems Reliability Engineering with the responsibility to ensure the systems engineering approach used by major acquisition programs includes a robust plan for improving reliability. The DT&E office reviews programs’ reliability growth test plans. This testing provides visibility over how reliability is improving and uncovers design problems so fixes can be incorporated before production begins. A reliability growth curve is used to track projected and actual improvements in reliability over time. The Reform Act further requires that the Deputy Assistant Secretary for Systems Engineering develop policies and guidance for the inclusion of provisions relating to systems engineering and reliability growth in requests for proposals. We observed evidence of this increased emphasis in the Joint Light Tactical Vehicle, Remote Minehunting System, Gray Eagle, and Global Hawk programs. • Joint Light Tactical Vehicle: The DT&E office helped this program develop a more realistic reliability growth plan prior to Milestone B. Based on the performance of prototype vehicles, developmental test officials determined that the program’s reliability growth curve was unrealistic. For example, officials reported that the program’s initial reliability growth plan assumed a starting reliability that was almost 60 percent higher than what had actually been demonstrated during technology development. It also assumed commonality between the two vehicle variants, a large reliability increase in a short test time, Page 20 GAO-13-103 Weapons Acquisition Reform and two corrective action periods. The DT&E office recommended that the program eliminate the vehicle commonality assumption, add more test miles, and add another corrective action period to its test plan. It also recommended that the program consider lowering the vehicle’s reliability requirement. Based on this input, the Army revised its plan by adding two vehicles and 40,000 more test miles to ensure reliability is adequately addressed for both variants. With approval of the user, it also reduced the reliability requirement from 3,600 to 2,400 miles mean time between operational mission failures. • Remote Minehunting System: The SE office worked with program officials to improve reliability growth planning, which was found to be one of the key factors leading to the program’s Nunn McCurdy unit cost breach in 2009. Before the breach, program officials had not funded a reliability growth program or established a design for reliability process. The program had a reliability goal of 150 hours mean time between failures, but program officials stated that testing demonstrated a reliability of only about 45 hours. Since the breach, the program has worked closely with the SE office to establish a reliability program plan and a growth curve to track reliability improvements. During the Nunn-McCurdy review, the program developed a three phase reliability growth program to improve the program’s subsystems, components, and manufacturing processes that contributed to poor reliability. According to program officials, phase one of the reliability growth program was completed in 2011, and reliability has improved by 40 percent, going from 45 hours mean time between operational mission failures to 63 hours. Although this improvement is still below the minimum requirement of 75 hours, program officials stated that phase two of the reliability growth program is scheduled to be completed in April 2013 and is projected to achieve the program’s 75 hour minimum requirement. • Gray Eagle and Global Hawk: The SE office has worked to improve reliability across the unmanned aircraft portfolio, including the Gray Eagle and Global Hawk. Prior to the Gray Eagle’s second low rate initial production decision in 2011,18 SE officials raised concerns about the system’s poor reliability. As a result, the Army was directed to undertake a reliability improvement program. The Under Secretary of 18 Low-rate initial production is production of the system in the minimum quantity necessary to provide articles for operational tests, establish an initial production base, and permit an orderly increase in the production rate for the system. Page 21 GAO-13-103 Weapons Acquisition Reform Defense for Acquisition, Technology and Logistics approved the program for low rate initial production, but stressed the need to improve the operational reliability as quickly as possible. SE officials worked with program officials to establish a reliability working group, develop reliability growth curves, and develop a reliability enhancement management plan. According to SE officials, the Gray Eagle initially improved the reliability of the aircraft by 15 percent and the ground control station by 30 percent. According to program officials, the initial reliability goals were overstated and not needed to meet the program’s overall operational availability requirement. Based on initial operational test results in August 2012, the program office is working with the user to redefine the reliability goals without impacting the system’s ability to meet its overall operational availability requirement. According to the PARCA office, these efforts have been informed by a detailed reliability model that they built in consultation with the Army. This model showed the relationship between the aircraft’s reliability and its availability to perform operational missions. SE officials also found similar reliability problems on the Global Hawk program and worked with program officials to establish a reliability growth and improvement plan and reliability growth curves. According to SE officials, the time between unscheduled maintenance on the Global Hawk has improved on the order of 50 to 80 percent. While DOD has taken steps to implement most of the fundamental Challenges Exist That Reform Act provisions, some key efforts to date have been primarily Could Limit the focused on DOD’s largest major defense acquisition programs. Expanding the reach of the Reform Act to bring about systemic change to Reform Act’s Ability DOD’s weapons acquisition process so that it influences all programs, to Influence Systemic however, still has challenges. Although senior leaders were receptive to the Reform Act principles, they identified several challenges that currently Change limit DOD’s ability to broaden the Reform Act’s influence. We grouped these challenges into five general categories: (1) organizational capability constraints; (2) need for additional guidance on cost estimating and Reform Act implementation; (3) uncertainty about the sufficiency of service level systems engineering and developmental testing resources; (4) limited dissemination of lessons learned; and (5) cultural barriers. Organizational Capability Leaders of two of the offices established as a result of the Reform Act told Constraints Make It us that even though they have implemented most of the fundamental Difficult to Expand Reform Reform Act provisions, they have had to limit their activities to a portion of acquisition programs in their portfolios due to resource constraints. Thus, Act’s Impact it is doubtful that they could expand the scope of their activities to include Page 22 GAO-13-103 Weapons Acquisition Reform more weapon acquisition programs at current staffing levels. For example, the DT&E office has had to be selective in its level of oversight of acquisition programs because the current staff of around 70 government and contractor personnel cannot adequately cover a portfolio of over 200 acquisition programs, according to its Deputy Assistant Secretary. The office has dropped virtually all but the largest programs from its oversight list and eliminated oversight of some major automated information systems. CAPE officials estimated that its cost assessment division would need to double in size in order to meet the Reform Act’s requirements. However, soon after the Reform Act was enacted, budgetary constraints limited the expansion of the cost estimating workforce to about 25 percent of the necessary growth. According to CAPE officials, its current cost analysis staffing is not adequate to meet its mission of improving the analytical skills of the defense cost estimating workforce, issuing policy, and providing sound and unbiased cost and schedule estimates. The office has delegated its independent cost estimating responsibility for most major automated information systems to the military services and some guidance has yet to be issued. The SE and PARCA offices are also struggling in some regards. For example, according to its Deputy Assistant Secretary, the SE office is continuously challenged to maintain the high caliber, qualified personnel required to provide assistance to and oversight of its portfolio of over 200 acquisition programs. Further, PARCA officials stated that the availability of government positions, particularly at the senior executive service level, continues to be a critical issue for the office. The two divisions within the PARCA office, the performance assessments division and the root cause division, do not currently have permanent government personnel at the senior executive level. Officials also stated current proprietary information rules limit the ability of PARCA contractor personnel to handle and maintain some weapon system information, severely impeding operations. Services Would Like More Offices within OSD have not yet issued more detailed guidance that could Cost Estimating and help institutionalize better cost estimating practices and steer program Reform Act decisions related to competitive prototyping and preliminary design reviews. The CAPE office has not issued guidance for operating and Implementation Guidance support costs estimates, such as fuel and maintenance costs, that have been estimated to account for two-thirds or more of a system’s total life Page 23 GAO-13-103 Weapons Acquisition Reform cycle cost.19 In addition, although not specifically required by the Reform Act, the CAPE office has not issued guidance for the services to use when developing Milestone A program cost estimates. As a result, senior leaders may not have access to realistic cost estimates prior to Milestone B for decision making purposes. Military service officials told us they are particularly interested in getting guidance on data that should be included in the cost analysis requirements description, which forms the basis for their cost estimates at Milestones B and C. CAPE officials recognize that, while some progress has been made, they need to complete the guidance, but have not been able to dedicate resources to do so. Some officials also told us that they find the competitive prototyping and preliminary design review requirements confusing and would like guidance on how to implement these requirements. DOD policy requires the technology development strategy for major defense acquisition programs to provide for prototypes of the system or, if a system prototype is not feasible, for prototypes of critical subsystems before Milestone B approval.20 However, officials from the Ground Combat Vehicle program were unclear as to when and what type of prototype to use. From a broader perspective, other military officials questioned the value of competitive prototyping as a blanket requirement for all programs, especially for programs that are using mature technologies, given the cost. For example, senior acquisition officials questioned the necessity of spending $400 million on competitive prototyping for the Small Diameter Bomb II program since the program was aware of problems with one contractor’s design. However, program officials indicated that competitive prototyping enabled them to identify design issues early in development and realize a savings of $1 billion. Officials from the Ground Combat Vehicle program we spoke with also indicated that they struggled with the timing of when to hold the program’s preliminary design review and what type of knowledge was required, since better guidance is needed. The program plans to hold multiple design reviews prior to Milestone B to consider contractor and government designs of the weapon system and then hold another review after Milestone B in order to resolve differences between the government’s and selected contractor’s preliminary designs. 19 The Reform Act required CAPE to issue guidance relating to full consideration of life cycle management and sustainability costs in major defense acquisition programs. 20 DOD Directive-Type Memorandum 09-027, Implementation of the Weapon Systems Acquisition Reform Act of 2009 (Dec. 4, 2009, incorporating Change 3, Dec. 9, 2011). Page 24 GAO-13-103 Weapons Acquisition Reform We spoke with OSD officials to determine which office should be providing guidance or assistance to program managers on competitive prototyping and preliminary design review issues. None of the offices have official responsibility for these efforts. OSD officials stated that these are program decisions and should be discussed with their respective military service level acquisition officials. Services May Not Have OSD officials believe that the services may lack resources in key Sufficient Resources to positions that could help strengthen systems engineering and Oversee and Conduct developmental testing activities on weapon acquisition programs. For example, according to the Deputy Assistant Secretary of Defense for Systems Engineering and Systems Engineering, the Navy and Air Force have reassigned the duties Developmental Testing and responsibilities of their service-level chief engineers, thereby de- Activities emphasizing the importance of systems engineering. The Deputy Assistant Secretary believes maintaining strong systems engineering leadership at the service level is essential for tying the systems engineering community together and promoting good systems engineering practices throughout each respective service. According to the DT&E and SE office’s March 2012 joint annual report to the Congress, the Navy abolished its chief engineer position and while the Air Force recently began to take steps to relocate the systems engineering function to the headquarters level, the impact of a recent reorganization on systems engineering activities is not yet known. In addition, the Deputy Assistant Secretary for Developmental Test and Evaluation expressed concern that the military services may not be implementing new legislation that requires each major defense acquisition program be supported by a chief developmental tester that oversees developmental test and evaluation activities. He stated that in some cases one person is serving as the chief developmental tester across multiple programs instead of having one person dedicated specifically to each program.21 The Deputy Assistant Secretary is trying to determine the extent to which this practice is occurring and then plans to work with the services to get more focused leadership for each program. 21 The requirement contained in the National Defense Authorization Act for Fiscal Year 2012 required “each major defense acquisition program to be supported by a chief developmental tester.” However, it did not specifically prohibit one person serving as the chief developmental tester across multiple programs. Page 25 GAO-13-103 Weapons Acquisition Reform It is also unclear whether the services have a sufficient number of qualified personnel to conduct systems engineering and test and evaluation activities. The services planned to grow these workforces by over a combined 5,000 people between fiscal years 2009 and 2015 and had made progress in growing each of these workforces through fiscal year 2010. However, budget cuts have resulted in DOD canceling some of its weapon acquisition programs and reassessing its decision to increase the acquisition workforce. Last year, we recommended that the Secretary of Defense report the impact budget cuts were having on the military service workforce and their ability to meet weapon acquisition program needs in the areas of developmental testing and systems engineering.22 In the DT&E and SE offices’ March 2012 joint annual report to the Congress, the Deputy Assistant Secretary for Systems Engineering reported that the Army has reduced its systems engineering workforce growth plan as compared to the plan reported in March 2011 joint annual report, and that contractor-to-civilian conversions have been suspended. In addition, the Deputy Assistant Secretary believes a prolonged hiring freeze in the Air Force could potentially create new experience gaps in the workforce. The Deputy Assistant Secretary for Developmental Test and Evaluation did not discuss the impact of budget cuts on the services’ test and evaluation workforce growth plan in the March 2012 joint annual report to the Congress and neither office reported on whether the services had an adequate workforce to meet the needs of the current portfolio of weapon acquisition programs. Expanded Use of Lessons DOD has not taken full advantage of sharing lessons learned obtained Learned Would Help through root cause analyses of programs that experience Nunn-McCurdy Expand Impact across the cost and schedule breaches with the acquisition workforce, particularly program managers. According to the Defense Acquisition Guidebook, Acquisition Portfolio which provides best practices the acquisition workforce can use on programs, lessons learned are a tool that the program manager may use to help identify potential areas of risk associated with a weapon acquisition system by reviewing the experiences encountered in past programs. Lessons learned databases document what worked and what did not work in past programs, in the hopes that future programs can avoid the same pitfalls. Further, if the right best practices are applied, they help to avoid common problems and improve quality, cost, or both. 22 GAO-11-806. Page 26 GAO-13-103 Weapons Acquisition Reform The PARCA office has made some effort to educate program managers on how to avoid acquisition problems through classes taught at the Defense Acquisition University. However, these courses are geared towards educating new program managers and may not be reaching a wide range of program officials. Nevertheless, some officials indicated that this information would be helpful for program officials to understand and avoid problems that have affected weapon acquisition programs in the past. Other officials also stated that it would be helpful if root cause analysis assessments contained more detailed information so acquisition officials could better understand problems and apply lessons learned. For example, when cost estimating was determined to be a root cause of a problem, officials stated they would have found it more beneficial to know if immature technologies or unrealistic requirements were the basis for the poor cost estimate. Systemic Changes Will Be Perhaps the most difficult challenge the department faces in making Difficult Until Cultural systemic changes to the acquisition process is changing the cultural Barriers Are Addressed relationship between the military services, which fund and develop new weapon acquisition programs, and OSD offices, which provide advice to and oversee the programs. Senior military service officials have told us they believe they understand and can manage the risks of specific weapon acquisition programs without much assistance from OSD. On the other hand, OSD officials believe more assistance is needed, as evidenced by the high number of programs that have experienced Nunn- McCurdy breaches and poor operational testing results. For example, since it was established in 2009, the DT&E office has assessed whether 15 programs were ready to begin operational testing. The office recommended that 5 of the programs—Global Hawk Block 20/30, Standard Missile 6, Joint Tactical Radio System Handheld Manpack Small Form (HMS) Rifleman Radio, Joint Tactical Radio System HMS Manpack, and MQ-1C Gray Eagle—not proceed into operational testing. However, military service acquisition chiefs decided to allow all 5 of these programs to proceed anyway. Four of the programs—Global Hawk Block 20/30, Standard Missile 6, Joint Tactical Radio System HMS Rifleman Radio and Joint Tactical Radio System HMS Manpack—demonstrated poor performance in operational testing, in areas such as reliability, effectiveness, or suitability. Operational testing results for the MQ-1C Gray Eagle have not yet been reported. On the other hand, a few service officials we met with were reluctant to accept some recommendations made by OSD offices because they believed the recommendations were overly burdensome and could Page 27 GAO-13-103 Weapons Acquisition Reform significantly impact weapon acquisition programs’ cost and schedule outcomes without a lot of benefit. This was the case for the KC-46 Tanker program, where program officials were concerned that additional testing recommended by the developmental test and evaluation, and operational testing offices, as part of the integrated test team, could have significant contractual implications during development. In this case, officials identified additional flight test opportunities without having to renegotiate the fixed price contract. However, the additional allotted test time is not equivalent to the 6 to 8 months the developmental testing office felt should be added. A similar situation occurred on the Ship to Shore Connector program. The Navy disagreed with a DT&E office recommendation to conduct full system testing prior to procuring additional craft during initial production. The DT&E office believed the program was high risk because the Ship to Shore Connector was a complete redesign of a previous system with no reuse of any major component (engines, gearboxes, hydraulics, command and control software). Navy officials, however, believe the program is low risk since it is an evolutionary program and has one critical technology, a fire suppression system, which has already been sufficiently demonstrated and qualified through test and evaluation. In addition, the Navy estimated that it would cost $15 million to revise the existing production schedule to accommodate the full system testing as recommended by the DT&E office. The DT&E office and the Navy reached a compromise whereby OSD would review available system test results before more craft are authorized. Current fiscal pressures, along with the threat of more to come, have Conclusions DOD officials looking for ways to increase buying power by controlling cost and schedule overruns on weapon acquisition programs. The offices established as a result of the Reform Act, as well as policy provisions have helped DOD make inroads towards putting weapon acquisition programs on more solid footing. Together, the offices and policy provisions place more attention on requirements, costs, testing, and reliability as early as Milestone A. The provisions of the act, when specifically focused on newer programs, are having a positive impact on the programs and the acquisition process. They show that expert attention to the cost and achievability of capability requirements, the assumptions made for cost and funding of programs, and the amount of systems engineering knowledge that is brought to bear early make programs more executable. Page 28 GAO-13-103 Weapons Acquisition Reform Although senior officials we spoke with throughout the department are receptive to the broad principles of the Reform Act, it is too early to tell if the Reform Act is going to result in systemic change to DOD’s weapon acquisition process. DOD faces several challenges that must be addressed to get lasting change—organizational capability constraints, the need for additional cost estimating and implementation guidance, the possibility of insufficient systems engineering and developmental testing resources, limited dissemination of lessons learned, and cultural barriers between OSD and the services. Some challenges appear to be straight forward to address, such as providing guidance for estimating operating and support costs, providing additional guidance for conducting preliminary design reviews and competitive prototyping activities, and disseminating lessons learned to the broader acquisition community. However, they may require more resources, which have been difficult to obtain. For Reform Act policies and practices to have a systemic effect across the entire portfolio of weapon system acquisition programs, the department must also address challenges related to systems engineering and developmental testing resources and cultural barriers between OSD and the services. This begins with the services identifying key leaders at the headquarters level and within program offices to guide systems engineering and developmental testing efforts and then ensuring that there are enough trained staff to carry out these activities. OSD will need to continue monitoring the services’ efforts. It will also require an environment where the services stop proposing new weapon systems with inflexible requirements, immature technologies, and cost, schedule, and funding assumptions that are too optimistic at the start of a program. Breaking down cultural resistance to change will take more cooperation between the Under Secretary for Acquisition, Technology and Logistics and other OSD offices, and service acquisition executives to address, as well as continuity of leadership. Efforts by the PARCA office to identify factors that correlate to good or poor acquisition outcomes, particularly as it relates to program manager tenure, will be beneficial. The services’ ability to demonstrate that the Reform Act is influencing all weapon acquisition programs, not just the biggest, will be a key indicator for determining whether the Reform Act has had a positive effect on DOD’s culture. Page 29 GAO-13-103 Weapons Acquisition Reform We recommend that the Secretary of Defense take the following four Recommendations for actions to enable systemic change across the entire portfolio of weapon Executive Action acquisition programs: • direct the Director of Cost Assessment and Program Evaluation to issue guidance for estimating weapon acquisition program costs at Milestone A and operating and support costs throughout the acquisition life cycle by the end of fiscal year 2013 and ensure that the office prioritizes its resources accordingly to accomplish this task; • designate responsibility for providing advice and guidance to program offices on competitive prototyping and preliminary design reviews to the appropriate organization within the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics and ensure that the guidance is developed. The office(s) designated would be the focal point for addressing program office issues related to the practical implementation of these Reform Act provisions, such as the type of competitive prototyping to use, the timing and benefits of holding preliminary design reviews prior to milestone B, and if a preliminary design review should be held after milestone B; • direct the Deputy Assistant Secretaries of Defense for Systems Engineering and Developmental Test and Evaluation to assess and include in their annual report to the Congress beginning with the report on fiscal year 2012 activities: • the extent to which the office can perform their required activities with allocated resources; • the impact budget cuts are having on the military services total workforce (civilians, military, and contractors) and ability to meet program office needs; and • progress the services have made filling leadership positions, such as chief engineers at the service level and technical leads for systems engineering and developmental testing at the program office level; • direct the Director of Performance Assessments and Root Cause Analyses to make lessons learned collected during its root cause analysis evaluations available to the acquisition workforce and ensure that the office prioritizes its resources accordingly. DOD provided us written comments on a draft of this report. DOD Agency Comments concurred with two recommendations and partially concurred with two and Our Evaluation others. DOD’s comments appear in appendix III. DOD also provided technical comments, which we incorporated as appropriate in the report. Page 30 GAO-13-103 Weapons Acquisition Reform DOD agreed with the intent of our first recommendation, but noted that due to resource constraints, the Cost Assessment and Program Evaluation office could not guarantee that it would be able to issue guidance for estimating major defense acquisition program costs at Milestone A and operating and support costs throughout the acquisition lifecycle by the end of fiscal year 2013. We continue to believe that the Cost Assessment and Program Evaluation office should issue the guidance by the end of fiscal year 2013. However, if that is not possible from a resource standpoint, the office should commit to a date and devote the resources to meeting that date. We will continue to monitor DOD’s efforts to develop the guidance. Although DOD concurred with our second recommendation, we revised this recommendation based upon discussions with DOD officials during the agency comment period. Our revision clarified the intent of this recommendation, which is to have the Secretary of Defense designate a specific organization within the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics to provide advice and guidance on competitive prototyping and preliminary design reviews. We understand that the department has issued Reform Act implementation guidance and has incorporated aspects of competitive prototyping and preliminary design reviews in the Defense Acquisition Guidebook. Further, we recognize that program offices we visited are taking steps to implement the guidance that has already been issued. However, based on our discussions with senior level officials, we believe one or more offices need to be designated with the responsibility of developing additional guidance and answering program specific questions related to the practical implementation of the requirements. As noted earlier in our report, some officials questioned when to use prototyping or what type of prototyping should be used. In addition, there were questions about the timing of the preliminary design reviews. DOD partially concurred with our third recommendation. DOD noted that the type of information we recommended be assessed and reported on should be included as part of DOD’s human capital strategic planning process and as such, be reported in DOD’s annual Acquisition Workforce Strategic Plan. We agree that the impact of budget cuts on the workforce and the status of leadership positions could be addressed in the annual strategic plan. However, we continue to believe that the Deputy Assistant Secretaries for Systems Engineering and Developmental Test and Evaluation should include an assessment in their joint annual report to the Congress on the respective offices’ ability to perform activities specified in the Reform Act with available resources. Page 31 GAO-13-103 Weapons Acquisition Reform DOD concurred with our fourth recommendation, which would make lessons learned from root cause analyses available to the acquisition workforce. We are sending copies of this report to the Secretary of Defense and appropriate Congressional Committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you have any questions about this report or need additional information, please contact me at (202) 512-4841 or email@example.com. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were Cheryl Andrew, Assistant Director; Laura Greifner, Julie Hadley, Megan Porter, Rae Ann Sapp, and Marie Ahearn. Michael J. Sullivan Director Acquisition and Sourcing Management Page 32 GAO-13-103 Weapons Acquisition Reform Appendix I: Objectives, Scope, and Appendix I: Objectives, Scope, and Methodology Methodology This report examines DOD’s continued implementation of the Weapon Systems Acquisition Reform Act of 2009 (Reform Act). Specifically, we examined (1) DOD’s progress in implementing Reform Act provisions; (2) the impact the Reform Act has had on specific acquisition programs; and (3) challenges remaining in improving the weapons acquisition process. To assess DOD’s progress in implementing Reform Act provisions, we interviewed officials and analyzed documents, such as reports to the Congress and guidance issued from the Office of the Secretary of Defense (OSD) offices of the (1) Deputy Assistant Secretary of Defense for Systems Engineering (SE), (2) Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DT&E), (3) Cost Assessment and Program Evaluation (CAPE), and (4) Performance Assessments and Root Cause Analyses (PARCA) to determine the extent to which provisions have been implemented. We focused our review on the offices’ implementation of four fundamental Reform Act provisions: developing policy and guidance; approving acquisition documents; monitoring programs and conducting program assessments; and developing performance measures. In cases where provisions had not been implemented, we asked officials about the reasons for the delay and the expected time frame for completion. We also interviewed officials and analyzed documents from the office of the Assistant Secretary of Defense for Research and Engineering and the Defense Procurement and Acquisition Policy office, as well as four weapon acquisition programs, which had not yet started development to determine the progress DOD has made implementing Reform Act provisions related to preliminary design reviews, competitive prototyping, and competition. We believe these programs offer the best glimpse at how the OSD offices and Reform Act policies are influencing acquisition strategies. The weapon acquisition programs we chose for this analysis were part of a larger case study review that is described below. To determine the impact the Reform Act has had on specific weapon acquisition programs, we selected 11 weapon system programs to use as case studies. For each program we reviewed relevant program documentation such as the test and evaluation master plans, assessments of operational test readiness, systems engineering plans, program support reviews, root cause analyses, analysis of alternatives reports and cost estimates as applicable. We also interviewed appropriate program officials and officials from the OSD offices for SE, DT&E, and CAPE to obtain their perspectives about (1) the level of interaction between the programs and OSD offices; (2) changes made to program Page 33 GAO-13-103 Weapons Acquisition Reform Appendix I: Objectives, Scope, and Methodology acquisition strategies as a result of interactions with the OSD offices; and (3) benefits and challenges with implementing Reform Act provisions on each of the programs. We also reviewed the performance assessments and root cause analyses office’s root cause analysis documentation for programs that incurred Nunn-McCurdy cost or schedule breaches. We selected our case studies based on input from the officials in the OSD offices for SE, DT&E, CAPE, PARCA, and operational test and evaluation. We also discussed possible case studies with GAO employees who monitor and report on weapon acquisition programs on an annual basis. The programs we selected for review represent a variety of platforms, including sea vessels, manned and unmanned aircraft, and land systems. Specifically, we examined 11 programs at various stages of the acquisition process. Four programs had not yet passed Milestone B, development start, at the time we began our review. The remaining seven programs had completed their Milestone B review and were in development at the time of our case study selection. Of the seven programs, three have breached Nunn-McCurdy cost thresholds since the act was passed and have had to satisfy the act’s new requirements with regards to certification. The other programs had significant interaction with one or more of the OSD offices established by the Reform Act. A complete list of programs is provided below. Table 4: List of Programs Selected for Case Study Review Programs before Milestone B Service Platform Ground Combat Vehicle Army Ground a Joint Light Tactical Vehicle Army/ Marines Ground Ohio Class Replacement Navy Sea a Ship to Shore Connector Navy Sea Programs after Milestone B Global Hawk Air Force Air Gray Eagle Army Air Joint Strike Fighter Joint Air KC-46 Tanker Air Force Air Littoral Combat Ship Seaframe Navy Sea Remote Minehunting System Navy Sea Small Diameter Bomb Increment II Air Force Air Source: GAO analysis. a During the course of our review, the Joint Light Tactical Vehicle and Ship to Shore Connector programs held a Milestone B. Page 34 GAO-13-103 Weapons Acquisition Reform Appendix I: Objectives, Scope, and Methodology While our sample of 11 case studies allowed us to learn about the impact the Reform Act offices have had on DOD acquisitions, it was designed to provide anecdotal information, not findings that would be representative of all the department’s weapon acquisition programs. To determine challenges remaining in improving defense acquisitions we relied on information we collected and analyzed during our case study review of 11 weapon acquisition programs. We also solicited the opinions of the Under Secretary of Defense for Acquisition, Technology and Logistics; other senior level officials in the Office of the Secretary of Defense including the leaders of each of the offices created as a result of the Reform Act; as well as the military services’ Senior Acquisition Executives. We conducted this performance audit from January 2012 to December 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 35 GAO-13-103 Weapons Acquisition Reform Appendix II: Progress of Reform Act Offices Appendix II: Progress of Reform Act Offices in Implementing Weapon System Acquisition Reform Act Provisions in Implementing Weapon System Acquisition Reform Act Provisions Table 5: Implementation of Select Reform Act Provisions- Systems Engineering Reform Act Provisions Status Develop policy and guidance Complete: Issued a reliability, availability, and maintainability Directive-Type Memorandum, a development planning Directive-Type Memorandum, and a DOD Instruction for DASD (Systems Engineering), participated in JCIDS revisions, developed guidance for incorporating systems engineering into development contracts, streamlined the Systems Engineering Plan and Program Protection Plan, and released an update to a the Defense Acquisition Guidebook, chapter 4. Continue to refine policies and guidance as necessary. Approve documents Completing on annual basis: Approved 52 Systems Engineering Plans since 2009, including 15 in fiscal year 2011. Monitor programs / conduct assessments Completing on annual basis: Review portfolio of 234 programs. In fiscal year 2011, participated in 73 overarching integrated product team meetings, 6 peer reviews of acquisition contracts, and conducted 15 Program Support Reviews. Develop performance measures Complete: Developed a set of time-based metrics to assess each program’s ability to execute its system engineering plans and address risks the office had identified in prior reviews. The metrics measure program cost, schedule, staffing, reliability, availability and maintainability, software, integration, performance and manufacturing, and are to be incorporated into each program’s systems engineering plan and evaluated at various points in the development process. Source: GAO analysis of DOD data. a DOD Directive-Type Memorandum (DTM) 11-003,”Reliability Analysis, Planning, Tracking, and Reporting,” (Mar. 21, 2011); DOD DTM 10-017, “Development Planning to Inform Materiel Development Decision Reviews and Support of Analysis of Alternatives,” (Sept. 13, 2010, Incorporating Change 2, Dec. 9, 2011); DOD Instruction 5134.16, “Deputy Assistant Secretary of Defense for Systems Engineering (DASD(SE)),” (Aug. 19, 2011). Table 6: Implementation of Select Reform Act Provisions- Developmental Test and Evaluation Reform Act Provisions Status Develop policy and guidance Complete: Updated guide for incorporating test and evaluation into acquisition contracts. Championed updates to DOD Instruction assigning responsibilities and authorities to Developmental Test and Evaluation office, which is in the process of being updated. Updated guidance to include reliability factors in the Test and Evaluation Master Plan. Continue to refine policies and guidance as necessary. Approve documents Completing on annual basis: Reviewed and approved 186 Test and Evaluation Master Plans since 2009, including 44 in fiscal year 2011. Monitor programs / conduct assessments Completing on annual basis: Review portfolio of nearly 250 programs. In fiscal year 2011, participated in 22 defense acquisition board meetings and 59 overarching integrated product team meetings. The office has also conducted 16 Assessment of Operational Test Readiness reviews since 2009. Develop performance measures In process: Piloted performance measures on two programs. The measures were then updated and are being applied to over 40 programs that were selected for reporting in the fiscal year 2012 joint annual report. The assessments are being used to support the write- up of the program engagement section. Source: GAO analysis of DOD data. Page 36 GAO-13-103 Weapons Acquisition Reform Appendix II: Progress of Reform Act Offices in Implementing Weapon System Acquisition Reform Act Provisions Table 7: Implementation of Select Reform Act Provisions- Cost Assessment and Program Evaluation Reform Act Provisions Status Develop policy and guidance In process: Issued first policy document in May 2012, which is the basis for additional policy documents. Updating its Operating and Support Cost Estimating Guidebook, which will address the Reform Act requirement for DOD to issue guidance related to full consideration of life cycle management and sustainability costs in major defense acquisition programs. Approve documents Not applicable: The Reform Act does not require that the office approve acquisition documents. Monitor programs / conduct assessments Completing on annual basis: Conducted independent cost assessments for Milestone A and B certification on 30 future and current major defense acquisition programs, since 2009, including 8 in fiscal year 2011. The office conducted 3 Milestone C and 3 Nunn- McCurdy certification reviews in fiscal year 2011. Develop performance measures Not applicable: The Reform Act does not require that the office develop performance measures. Source: GAO analysis of DOD data. Table 8: Implementation of Select Reform Act Provisions- Performance Assessments and Root Cause Analyses Reform Act Provisions Status Develop policy and guidance In process: Developing guidance to assist offices in conducting root cause analyses. The guidance for conducting performance assessments is expected to be released in early fiscal year 2013. Approve documents Not applicable: The Reform Act does not require that the office approve acquisition documents. Monitor programs / conduct assessments Completing on annual basis: Completed 14 Root Cause Analyses for programs which have undergone a Nunn-McCurdy breach or were requested by OSD and has completed 26 semi-annual follow-up reports on these programs. Providing OSD with the execution status of DOD’s portfolio of acquisition programs through the Defense Acquisition Executive Summary process. Develop performance measures In process: Utilize Defense Acquisition Executive Summary information to identify cost performance, schedule, funding, and technical performance issues on Major Defense Acquisition Programs. Continuing to develop performance measures. Source: GAO analysis of DOD data. Page 37 GAO-13-103 Weapons Acquisition Reform Appendix III: Comments from the Appendix III: Comments from the Department of Defense Department of Defense Page 38 GAO-13-103 Weapons Acquisition Reform Appendix III: Comments from the Department of Defense Page 39 GAO-13-103 Weapons Acquisition Reform Appendix III: Comments from the Department of Defense Page 40 GAO-13-103 Weapons Acquisition Reform Appendix III: Comments from the Department of Defense (121024) Page 41 GAO-13-103 Weapons Acquisition Reform The Government Accountability Office, the audit, evaluation, and GAO’s Mission investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no Obtaining Copies of cost is through GAO’s website (http://www.gao.gov). Each weekday GAO Reports and afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted Testimony products, go to http://www.gao.gov and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov. Contact: To Report Fraud, Website: http://www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in E-mail: firstname.lastname@example.org Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, email@example.com, (202) 512- Congressional 4400, U.S. Government Accountability Office, 441 G Street NW, Room Relations 7125, Washington, DC 20548 Chuck Young, Managing Director, firstname.lastname@example.org, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Please Print on Recycled Paper.
Weapons Acquisition Reform: Reform Act Is Helping DOD Acquisition Programs Reduce Risk, but Implementation Challenges Remain
Published by the Government Accountability Office on 2012-12-14.
Below is a raw (and likely hideous) rendition of the original report. (PDF)