oversight

Weapons Acquisition Reform: Reform Act Is Helping DOD Acquisition Programs Reduce Risk, but Implementation Challenges Remain

Published by the Government Accountability Office on 2012-12-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to the Committee on Armed
                Services, U.S. Senate




                WEAPONS
December 2012



                ACQUISITION
                REFORM

                Reform Act Is Helping
                DOD Acquisition
                Programs Reduce
                Risk, but
                Implementation
                Challenges Remain




GAO-13-103
                                                December 2012

                                                WEAPONS ACQUISITION REFORM
                                                Reform Act Is Helping DOD Acquisition Programs
                                                Reduce Risk, but Implementation Challenges Remain
Highlights of GAO-13-103, a report to
Committee on Armed Services, U.S. Senate




Why GAO Did This Study                          What GAO Found
For the past 3 years, DOD has been              The Department of Defense (DOD) has taken steps to implement fundamental
implementing the Reform Act                     Weapon Systems Acquisition Reform Act of 2009 (Reform Act) provisions,
requirements which are aimed at                 including those for approving acquisition strategies and better monitoring
helping weapon acquisition programs             weapon acquisition programs. DOD is also continuing to take additional steps to
establish a solid foundation from the           strengthen policies and capabilities. Some provisions, such as issuing guidance
start. This helps to prevent cost               for estimating operating and support costs, are being implemented.
growth, thus helping the Defense dollar
go further. This is the third in a series       GAO’s analysis of 11 weapon acquisition programs showed the Reform Act has
of GAO reports on the Reform Act.               reinforced early attention to requirements, cost and schedule estimates, testing,
                                                and reliability. For example, prior to starting development, an independent review
GAO was asked to determine
                                                team raised concerns about the Ground Combat Vehicle program’s many
(1) DOD’s progress in implementing
Reform Act provisions; (2) the impact
                                                requirements and the risks associated with its 7-year schedule. Subsequently,
the Reform Act has had on specific              the Army reduced the number of requirements by about 25 percent and
acquisition programs; and                       prioritized them, giving contractors more flexibility in designing solutions. In
(3) challenges remaining. To do this,           addition, the developmental test and evaluation office—resulting from the Reform
GAO analyzed documents and                      Act—used test results to help the Joint Light Tactical Vehicle program develop a
interviewed officials from the four OSD         more realistic reliability goal and a better approach to reach it. Shown below are
offices created as a result of the              areas where the Reform Act influenced several programs in GAO’s review.
Reform Act, other DOD offices, the
military services, and 11 weapon                Reform Act Influence on Select Case Study Programs
acquisition programs we chose as                  Program                           Requirements     Cost and schedule    Testing     Reliability
case studies. Case study programs                 Ohio Class Replacement                                                               
were selected based on their                      Ground Combat Vehicle                                                                
development status and interaction                Joint Light Tactical Vehicle                                                         
with the four OSD offices. Results                Ship to Shore Connector                                                                
cannot be generalized to all DOD                  KC-46 Tanker                                                                           
weapon acquisition programs.
                                                Source: GAO analysis of DOD data.

What GAO Recommends
                                                While DOD has taken steps to implement most of the fundamental Reform Act
GAO recommends DOD develop                      provisions, some key efforts to date have been primarily focused on DOD’s
additional cost estimating and Reform
                                                largest weapon acquisition programs. DOD faces five challenges—organizational
Act implementation guidance; make
                                                capability constraints, the need for additional guidance on cost estimating and
lessons learned available to the
acquisition community; and assess the           Reform Act implementation, the uncertainty about the sufficiency of systems
adequacy of the military services’              engineering and developmental testing resources, limited dissemination of
systems engineering and                         lessons learned, and cultural barriers between the Office of the Secretary of
developmental testing workforce. DOD            Defense (OSD) and the military services—that limit its ability to broaden the
generally concurred with the                    Reform Act’s influence to more programs. Service officials believe additional
recommendations. GAO clarified one              guidance is needed to improve their cost estimates and other implementation
recommendation to make it clear that            efforts. They also believe that lessons learned from programs that experience
DOD needs to designate an office(s)             significant cost and schedule increases should be shared more broadly within the
within the Acquisition, Technology and          acquisition community. These challenges seem straightforward to address, but
Logistics organization to provide               they may require more resources, which have been difficult to obtain. Ensuring
practical Reform Act implementation             the services have key leaders and staff dedicated to systems engineering and
guidance to program offices.                    developmental testing activities, such as chief engineers at the service level and
                                                technical leads on programs, as well as breaking down cultural barriers are more
View GAO-13-103. For more information,          difficult to address. They will require continued monitoring and attention by the
contact Michael J. Sullivan at (202) 512-4841
or sullivanm@gao.gov.
                                                Under Secretary for Acquisition, Technology and Logistics, service acquisition
                                                executives, and offices established as a result of the Reform Act to address.
                                                                                                   United States Government Accountability Office
Contents


Letter                                                                                     1
               Background                                                                  4
               DOD Has Implemented Most of the Fundamental Reform Act
                 Provisions and Is Strengthening Acquisition Activities                    8
               The Reform Act Is Helping Programs Identify and Mitigate Risks
                 Earlier in the Acquisition Process                                      14
               Challenges Exist That Could Limit the Reform Act’s Ability to
                 Influence Systemic Change                                               22
               Conclusions                                                               28
               Recommendations for Executive Action                                      30
               Agency Comments and Our Evaluation                                        30

Appendix I     Objectives, Scope, and Methodology                                        33



Appendix II    Progress of Reform Act Offices in Implementing Weapon
               System Acquisition Reform Act Provisions                                  36



Appendix III   Comments from the Department of Defense                                   38



Tables
               Table 1: Programs Selected for Case Study Review                            3
               Table 2: Key Responsibilities of Offices Established as a Result of
                        the Reform Act                                                    5
               Table 3: Reform Act Influence on Case Study Programs                      15
               Table 4: List of Programs Selected for Case Study Review                  34
               Table 5: Implementation of Select Reform Act Provisions- Systems
                        Engineering                                                      36
               Table 6: Implementation of Select Reform Act Provisions-
                        Developmental Test and Evaluation                                36
               Table 7: Implementation of Select Reform Act Provisions- Cost
                        Assessment and Program Evaluation                                37
               Table 8: Implementation of Select Reform Act Provisions-
                        Performance Assessments and Root Cause Analyses                  37




               Page i                                   GAO-13-103 Weapons Acquisition Reform
Figures
          Figure 1: DOD’s Progress in Implementing Four Fundamental
                   Reform Act Provisions                                                            10
          Figure 2: Revised Review Process for Major Defense Acquisition
                   Programs                                                                         12




          Abbreviations

          CAPE              Cost Assessment and Program Evaluation
          DOD               Department of Defense
          DT&E              Developmental Test and Evaluation
          HMS               Handheld, Manpack, and Small Form Fit
          OSD               Office of the Secretary of Defense
          PARCA             Performance Assessments and Root Cause Analyses
          SE                Systems Engineering



          This is a work of the U.S. government and is not subject to copyright protection in the
          United States. The published product may be reproduced and distributed in its entirety
          without further permission from GAO. However, because this work may contain
          copyrighted images or other material, permission from the copyright holder may be
          necessary if you wish to reproduce this material separately.




          Page ii                                           GAO-13-103 Weapons Acquisition Reform
United States Government Accountability Office
Washington, DC 20548




                                   December 14, 2012

                                   The Honorable Carl Levin
                                   Chairman
                                   The Honorable John McCain
                                   Ranking Member
                                   Committee on Armed Services
                                   United States Senate

                                   For several decades, GAO has reported on poor outcomes
                                   encompassing cost and schedule growth on the Department of Defense’s
                                   (DOD) major weapon acquisition programs. Many problems can be traced
                                   to a culture where the military services begin programs with inflexible
                                   requirements, immature technologies, and overly optimistic cost and
                                   schedule estimates. Given pressures to reduce spending across the
                                   government, including DOD, finding ways to prevent or mitigate cost
                                   growth is crucial to U.S. national security. A solid program foundation
                                   using good developmental testing and systems engineering, and reliable
                                   cost estimates is needed in order to help avoid cost overruns and
                                   promote better acquisition outcomes. There have been numerous
                                   attempts in the past to improve DOD acquisition outcomes, including the
                                   Packard Commission and the Goldwater-Nichols Act in the 1980s and
                                   Federal Acquisition Streamlining Act of 1994.1 More recently, Congress
                                   passed the Weapon Systems Acquisition Reform Act of 2009 (Reform
                                   Act)2 to improve the way weapon systems are acquired and avoid cost
                                   and schedule overruns.

                                   In 2009, the Senate Armed Services Committee asked us to begin
                                   reporting on DOD’s implementation of Reform Act provisions and the
                                   impact the Reform Act has had on weapon acquisition programs. This is
                                   our third report addressing these topics. The first report focused on
                                   DOD’s initial efforts to implement Reform Act provisions for systems
                                   engineering and developmental testing, including the placement of new


                                   1
                                     Final Report to the President by the President’s Blue Ribbon Commission on Defense
                                   Management (June 1986); Goldwater-Nichols Department of Defense Reorganization Act
                                   of 1986, Pub. L. No. 99-433; and Pub. L. No. 103-355 (FASA).
                                   2
                                     Pub. L. No. 111-23, as amended by the Ike Skelton National Defense Authorization Act
                                   for Fiscal Year 2011, Pub. L. No. 111-383 §§ 813 and 1075, and the National Defense
                                   Authorization Act for Fiscal Year 2012, Pub. L. No. 112-81 §§ 819 and 837.




                                   Page 1                                          GAO-13-103 Weapons Acquisition Reform
offices for these activities within the Office of the Secretary of Defense
(OSD).3 Our second report examined the challenges the services face as
they try to strengthen systems engineering and developmental testing
activities on weapon acquisition programs.4 This report examines
(1) DOD’s progress in implementing Reform Act provisions; (2) the impact
the Reform Act has had on specific acquisition programs; and
(3) challenges remaining in improving the weapons acquisition process.

To determine DOD’s progress in implementing Reform Act provisions we
interviewed officials and analyzed documentation from the four offices
whose leadership was established within OSD as a result of the Reform
Act—Systems Engineering (SE), Developmental Test and Evaluation
(DT&E), Cost Assessment and Program Evaluation (CAPE), and
Performance Assessments and Root Cause Analyses (PARCA).5 We
focused our review on the offices’ implementation of four fundamental
Reform Act provisions: developing policy and guidance; approving
acquisition documents; monitoring programs and conducting program
assessments; and developing performance measures. We also analyzed
documentation we collected on selected weapon acquisitions. To
determine the Reform Act’s impact on specific defense acquisition
programs, we chose as case studies, 11 weapon systems at various
points in the development process based on recommendations by OSD
officials and GAO’s previous evaluations of these programs. At the time of
our case study selection, all of the weapon systems in our review had
been given Milestone A approval. This approval allows the programs to
start the technology development phase of the acquisition process. Four
of the 11 weapon systems had not yet received Milestone B approval.
This approval signifies the start of engineering and manufacturing
development activities. We believe these four programs offer the best
insight into how the OSD offices and Reform Act policies are influencing


3
  GAO, Defense Acquisitions: DOD Needs to Develop Performance Criteria to Gauge
Impact of Reform Act Changes and Address Workforce Issues, GAO-10-774 (Washington,
D.C.: July 29, 2010).
4
  GAO, Weapons Acquisition Reform: Actions Needed to Address Systems Engineering
and Developmental Testing Challenges, GAO-11-806 (Washington, D.C.: Sept. 19, 2011).
5
  The Director of Systems Engineering and the Director of Developmental Test and
Evaluation offices were originally established as a result of the Reform Act. These offices
have since been renamed as the offices of the Deputy Assistant Secretary of Defense for
Systems Engineering and the Deputy Assistant Secretary of Defense for Developmental
Test and Evaluation.




Page 2                                            GAO-13-103 Weapons Acquisition Reform
                                        acquisition strategies. The other seven programs were conducting
                                        engineering and manufacturing development activities or had been given
                                        Milestone C approval to begin production and deployment phase
                                        activities. Each of these programs provided different insights in their
                                        interactions with the OSD offices, but some Reform Act provisions, such
                                        as conducting preliminary design reviews prior to Milestone B, may not
                                        apply to these programs since they were beyond development start. A
                                        complete list of programs we reviewed is provided below.

Table 1: Programs Selected for Case Study Review

Programs that had not started development at the time of our case study selection                      System development start date
Ground Combat Vehicle                                                                                        December 2013 (estimated)
                               a
Joint Light Tactical Vehicle                                                                                                  August 2012
Ohio Class Replacement submarine                                                                                 August 2016 (estimated)
                         a
Ship to Shore Connector                                                                                                           July 2012
Programs that were in development at the time of our case study selection
Global Hawk unmanned aerial vehicle                                                                                            March 2001
Gray Eagle unmanned aerial vehicle                                                                                               April 2005
Joint Strike Fighter                                                                                                         October 2001
KC-46 Tanker aircraft                                                                                                       February 2011
Littoral Combat Ship Seaframe                                                                                               February 2011
Remote Minehunting System                                                                                                 December 1999
Small Diameter Bomb Increment II                                                                                                  July 2010
                                        Source: GAO analysis.
                                        a
                                         During the course of our review, the Joint Light Tactical Vehicle and Ship to Shore Connector
                                        programs held a Milestone B review and received approval to start development.


                                        To determine challenges remaining for implementing the Reform Act, we
                                        conducted our own assessment and interviewed the leaders of the OSD
                                        offices established as a result of the Reform Act, military service
                                        acquisition chiefs, and program managers. These officials are responsible
                                        for advising and overseeing weapon acquisition program development,
                                        funding and developing new weapon acquisition programs, and executing
                                        the day-to-day development plans, respectively. We also solicited the
                                        opinions of the Under Secretary of Defense for Acquisition, Technology
                                        and Logistics and other senior OSD leaders. Additional information about
                                        our scope and methodology can be found in appendix I.

                                        We conducted this performance audit from January 2012 to December
                                        2012 in accordance with generally accepted government auditing
                                        standards. Those standards require that we plan and perform the audit to


                                        Page 3                                                  GAO-13-103 Weapons Acquisition Reform
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objectives. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objectives.


             In May 2009, Congress passed the Weapon Systems Acquisition Reform
Background   Act of 2009 (Reform Act) in an effort to improve the way weapon systems
             are acquired and avoid further cost overruns on such programs. When
             signing the Reform Act into law, the President stated that its purpose was
             to limit weapon system cost overruns and that it would strengthen
             oversight and accountability by appointing officials who will closely
             monitor the weapons systems acquisition process to ensure that costs
             are controlled.

             Four offices were established as a result of the Reform Act: SE, DT&E,
             CAPE and PARCA. The SE and CAPE offices existed under other
             organizational titles prior to the Reform Act. Staffing levels, following the
             Reform Act, remained relatively stable for both of these offices, but some
             reorganization was necessary to reflect new Reform Act responsibilities.
             The DT&E and PARCA offices were newly established. The key roles and
             responsibilities of these four offices as outlined in the Reform Act are
             explained below:




             Page 4                                    GAO-13-103 Weapons Acquisition Reform
Table 2: Key Responsibilities of Offices Established as a Result of the Reform Act

Office                            Primary responsibilities
Systems Engineering               •   serves as principal advisor to the Secretary of Defense and the Under Secretary of Defense
                                      for Acquisition, Technology and Logistics on systems engineering activities in the department
                                  •   develops systems engineering and development planning guidance for DOD
                                  •   reviews and approves major defense acquisition program systems engineering plans
                                  •   monitors major defense acquisition program systems engineering activities
                                  •   reports to Congress annually on systems engineering organization, capabilities, and activities
Developmental Test and            •   serves as principal advisor to the Secretary of Defense and the Under Secretary of Defense
Evaluation                            for Acquisition, Technology and Logistics on developmental test and evaluation activities
                                  •   develops developmental test and evaluation guidance for DOD
                                  •   reviews and approves major defense acquisition program developmental test and evaluation
                                      plans
                                  •   monitors and reviews acquisition program developmental test and evaluation activities of
                                      major defense acquisition programs
                                  •   reports to Congress annually on developmental test and evaluation organization, capabilities,
                                      and activities
Cost Assessment and Program       •   serves as principal advisor to the Secretary of Defense and other senior officials on matters
Evaluation                            related to cost analysis and the planning and programming phases of the planning,
                                      programming, budgeting, and execution system
                                  •   develops independent cost estimates for major defense acquisition programs prior to major
                                      milestone decisions and updates independent cost estimates when programs have exceeded
                                      critical cost thresholds, known as Nunn McCurdy breaches
                                  •   reviews existing systems and methods for tracking and assessing operation and support
                                      costs on major defense acquisition programs
                                  •   develops analysis of alternative study guidance for major defense acquisition programs
                                  •   approves the analysis of alternatives study plan for each major defense acquisition program
Performance Assessments and       •   assesses major acquisition program performance through independent analyses and through
Root Cause Analyses                   the Defense Acquisition Executive Summary process
                                  •   identifies the root causes of cost growth and other problems on programs that experience a
                                      critical Nunn McCurdy cost breach
                                          Source: GAO Analysis of Weapon Systems Acquisition Reform Act of 2009.


                                          Each of the offices has varying levels of interaction with defense
                                          acquisition programs. For example, the SE and DT&E offices have
                                          ongoing interaction with acquisition programs throughout development in
                                          the form of program reviews, working group activities, and other program
                                          meetings. They also coordinate program documents in preparation for
                                          major milestone reviews. CAPE issues guidance to programs on how to
                                          conduct an analysis of alternatives at the beginning of the acquisition
                                          process. The office approves the analysis of alternative study plan that is




                                          Page 5                                                               GAO-13-103 Weapons Acquisition Reform
developed based on its guidance. It then develops independent cost
estimates for major milestone reviews and in the event that an acquisition
program experiences a Nunn-McCurdy breach.6 According to PARCA, it
assesses all major defense acquisition programs at least once per quarter
or when requested by the Under Secretary of Defense for Acquisition,
Technology and Logistics, and disseminates this information to senior
leaders. The office also interacts with specific programs if they experience
a Nunn-McCurdy breach. In these cases, the office assesses program
performance not less than semi-annually until 1 year after it receives a
new milestone approval.

In addition to the new organizational requirements, the Reform Act
requires DOD to ensure that the acquisition strategy for major defense
acquisition programs includes measures to ensure competition or the
option of competition throughout the program life cycle. This could include
strategies such as maintaining two sources for a system (dual-sourcing)
and breaking requirements for supplies or services previously provided or
performed under a single contract into separate smaller contracts
(unbundling of contracts).7 Major defense acquisition programs are also
required to provide for competitive prototyping—where two or more
competing teams produce prototypes before a design is selected for
further development—prior to Milestone B unless a waiver is properly




6
  10 U.S.C. § 2433 establishes the requirement for the DOD to prepare unit cost reports
on major defense acquisition programs or designated subprograms. If a program exceeds
cost growth thresholds specified in the law, this is commonly referred to as a Nunn-
McCurdy breach, which DOD is required to report to Congress and, if applicable, submit a
certification to Congress in order to continue the program, in accordance with 10 U.S.C.
§ 2433a.
7
    Pub. L. No. 111-23 § 202.




Page 6                                          GAO-13-103 Weapons Acquisition Reform
granted by the milestone decision authority,8 and to meet the following
Milestone B certification requirements, including:9

•    Appropriate trade-offs among cost, schedule, and performance
     objectives have been made to ensure the program is affordable;
•    A preliminary design review and formal post-preliminary design review
     assessment have been conducted and on the basis of such
     assessment the program demonstrates a high likelihood of
     accomplishing its intended mission;
•    Technology has been demonstrated in a relevant environment on the
     basis of an independent review and assessment by the Assistant
     Secretary of Defense for Research and Engineering;
•    Reasonable cost and schedule estimates have been developed to
     execute, with concurrence of the Director of CAPE, the program’s
     product development and production plan;
•    Funding is available to execute the program’s product development
     and production plan;
•    DOD has completed an analysis of alternatives with respect to the
     program; and
•    The Joint Requirements Oversight Council10 has approved program
     requirements, including an analysis of the operational requirements.



8
  Pub. L. No. 111-23 § 203(a). Specifically, the Reform Act required DOD to modify its
guidance relating to the operation of its acquisition system to incorporate these
competitive prototyping provisions. DOD did so through Directive-Type Memorandum
(DTM) 09-027, “Implementation of Weapon System Acquisition Reform Act of 2009 (Dec.
4, 2009, incorporating Change 3, Dec. 9, 2011). Major defense acquisition programs are
those estimated by DOD to require an eventual total expenditure for research,
development, test, and evaluation, including all planned increments, of more than $365
million, or for procurement, including all planned increments, of more than $2.19 billion in
fiscal year 2000 constant dollars or designated by the Under Secretary of Defense for
Acquisition, Technology, and Logistics. The Milestone Decision Authority for major
defense acquisition programs is the Under Secretary of Defense for Acquisition,
Technology and Logistics, the head of a DOD component, or if delegated the component
acquisition executive.
9
  Pub. L. No. 111-23; various sections, codified at 10 U.S.C. § 2366b. The Reform Act
revised the Milestone B certification requirements for trade-offs, preliminary design,
technology demonstration, and reasonable cost and schedule estimates. The remaining
Milestone B certification requirements bulleted here were unrevised by the Reform Act.
10
   The Joint Requirements Oversight Council is an advisory council to the Chairman of the
Joint Chiefs of Staff with the responsibility to: (1) identify, assess, and approve joint
military requirements; (2) assist acquisition officials in identifying alternatives to acquisition
programs; and (3) assist the Chairman of the Joint Chiefs of Staff in assigning priority for
joint military requirements.




Page 7                                                GAO-13-103 Weapons Acquisition Reform
                         The Reform Act also requires the Joint Requirements Oversight Council
                         to ensure trade-offs among cost, schedule, and performance objectives
                         are considered for joint military requirements.11 GAO previously reported
                         that the Council considered trade-offs made by the military services
                         before validating requirements, but the military services did not
                         consistently provide high-quality resource estimates to the Council for
                         proposed programs in fiscal year 2010. We also found that the Council
                         did not prioritize requirements, consider redundancies across proposed
                         programs, or prioritize and analyze capability gaps in a consistent
                         manner.12


                         DOD has implemented most of the fundamental Reform Act provisions as
DOD Has                  required and is taking additional steps to strengthen acquisition reviews,
Implemented Most of      policies, and capabilities. Offices established as a result of the Reform
                         Act are continuing to issue policies, review and approve relevant
the Fundamental          acquisition documents, monitor weapon acquisition program activities,
Reform Act               and develop performance measures. In addition, all four of the major
                         defense acquisition programs we reviewed that had not started
Provisions and Is        development when we selected our case studies plan to implement
Strengthening            Reform Act provisions regarding preliminary design reviews, competitive
                         prototyping, and competition. Also, some provisions, such as issuance of
Acquisition Activities   guidance on estimating operating and support costs, by the CAPE, are
                         still in the process of being completed. Finally, we found that the Under
                         Secretary of Defense for Acquisition, Technology and Logistics has
                         revised the defense acquisition review process to consider additional
                         knowledge collected on programs earlier and efforts are being made to
                         strengthen acquisition policies and capabilities.


DOD Is Continuing to     The offices established as a result of the Reform Act—SE, DT&E, CAPE,
Implement Reform Act     and PARCA—are continuing to make progress in implementing four
Provisions               fundamental Reform Act provisions aimed at strengthening acquisition
                         outcomes and oversight of weapon acquisition programs. Specifically, the
                         offices are (1) developing policy and guidance to the military services for
                         conducting work in their respective areas, (2) approving acquisition


                         11
                              Pub. L. No. 111-23 § 201.
                         12
                           GAO, DOD Weapon Systems: Missed Trade-off Opportunities During Requirements
                         Reviews, GAO-11-502, (Washington, D.C.: June 16, 2011).




                         Page 8                                      GAO-13-103 Weapons Acquisition Reform
documents prior to milestone reviews, (3) monitoring and assessing
weapon acquisition program activities on a consistent basis, and
(4) developing performance measures to assess acquisition program
activities. Figure 1 provides the status of DOD efforts to implement the
four fundamental provisions. Some offices are still in the process of
completing a few of these provisions. For example, CAPE and PARCA
are in the process of developing policies and guidance for their respective
areas and DT&E is in the process of establishing performance measures
that can be used to assess weapon acquisition program activities. The
office piloted the performance measures on two major defense acquisition
programs and reported that they are currently applying them to over 40
programs. Note that some activities related to approving documents and
monitoring or assessing programs require on-going efforts on the part of
some of the offices.




Page 9                                   GAO-13-103 Weapons Acquisition Reform
Figure 1: DOD’s Progress in Implementing Four Fundamental Reform Act Provisions

 Interactive Graphic   Click on status to show Reform Act provisions. Click on the “” to clear. See appendix II for the non-interactive, printer-friendly version.


                                                                    Monitor
                          Develop                                  programs/    Develop
                         policy and           Approve               conduct   performance                                 Reform Act provisions
      Office             guidancea           documents            assessments  measures                                          status


                                            Completing             Completing
 Systems
                        Completed             on annual             on annual             Completed
 Engineering
                                                 basis                 basis


 Developmental                              Completing             Completing
 Test and               Completed             on annual             on annual             In process
 Evaluation                                      basis                 basis


 Cost Assessment                                                   Completing
                                                     Not                                     Not
 and Program             In process                                 on annual
                                             applicable                                   applicable
 Evaluation                                                            basis



 Performance
                                                                   Completing
 Assessment                                          Not
                         In process                                 on annual             In process
 and Root Cause                              applicable
                                                                       basis
 Analyses


                                                      Source: GAO analysis of DOD data.
                                                      a
                                                          The Reform Act does not specify a date of completion for developing policy and guidance.
                                                      b
                                                          DOD Directive-Type Memorandum (DTM) 11-003, “Reliability Analysis, Planning, Tracking,
                                                          and Reporting,” (Mar. 21, 2011); DOD DTM 10-017, “Development Planning to Inform Materiel
                                                          Development Decision Reviews and Support of Analysis of Alternatives,” (Sept. 13, 2010,
                                                          Incorporating Change 2, Dec. 9, 2011); DOD Instruction 5134.16, “Deputy Assistant Secretary
                                                          of Defense for Systems Engineering (DASD(SE)),” (Aug. 19, 2011).




                                                 
                                                     Page 10                                                        GAO-13-103 Weapons Acquisition Reform
                            We also found evidence that major defense acquisition programs are
                            integrating Reform Act provisions in their acquisition strategies. The four
                            weapon acquisition programs we reviewed that had not started
                            development activities when we began our review plan to implement
                            Reform Act provisions related to preliminary design reviews, competitive
                            prototyping, and competition. For example, the Ground Combat Vehicle
                            has two contractors developing competitive prototypes of two key
                            subsystems to support technology development. The program intends to
                            conduct preliminary design reviews on both contractors’ designs prior to
                            Milestone B and to conduct full and open competition through Milestone
                            C. Similarly, according to program officials, the Joint Light Tactical
                            Vehicle program had three contractors develop full-system prototypes
                            during the technology development phase and held preliminary design
                            reviews on each contractor’s design prior to Milestone B. The program
                            plans to continue competition throughout engineering and manufacturing
                            development. None of the four programs in our review received a waiver
                            from Reform Act provisions.


Additional Emphasis Is      OSD is taking additional steps to strengthen the department’s oversight of
Being Placed on Improving   weapon acquisition programs and guidance for developing the programs.
Milestone Review Process,   In June 2011, for example, the Under Secretary for Acquisition,
                            Technology and Logistics revised the weapons acquisition review process
Policies, and Service       to consider acquired knowledge on weapon acquisition programs earlier
Capabilities                than before. The revised review process includes two new review points.
                            The first new review—the pre-engineering and manufacturing
                            development review—occurs before the release of a final request for
                            proposal for the engineering and manufacturing development phase. The
                            purpose of this new review is to assess each program’s acquisition
                            strategy, request for proposal, and key related planning documents earlier
                            in the process, and to determine whether program plans are affordable,
                            executable, and reflect sound business arrangements. The second new
                            review—the acquisition strategy and request for proposals review and
                            approval—occurs prior to Milestone C, the production decision. The
                            review provides the milestone decision authority an opportunity to review
                            the acquisition strategy and request for proposals for the production and
                            deployment phase prior to Milestone C. Figure 2 illustrates the revised
                            review process.




                            Page 11                                   GAO-13-103 Weapons Acquisition Reform
Figure 2: Revised Review Process for Major Defense Acquisition Programs




                                       According to the Under Secretary of Defense for Acquisition, Technology
                                       and Logistics, who is the authority for making milestone decisions for
                                       most major weapon acquisitions, the prior review process did not provide
                                       an adequate opportunity for review of program plans prior to release of
                                       the final request for proposals—the point at which DOD’s requirements,
                                       schedule, planned program content, and available funding should be firm
                                       and available for review. Further, the Under Secretary stated that making
                                       changes to acquisition strategies and program plans after all bidding
                                       activities, proposal evaluation, and source selection are complete is
                                       difficult and highly disruptive.13

                                       DOD is also rewriting the DOD Instruction 5000.0214 to include an
                                       extensive restructure of acquisition policies according to the Under
                                       Secretary of Defense for Acquisition, Technology and Logistics. This
                                       update will implement Section 832 of the National Defense Authorization
                                       Act for Fiscal Year 2012,15 which requires DOD to issue guidance on
                                       actions to be taken to improve its processes for estimating, managing,
                                       and reducing operation and support cost, as well as ensure competition in
                                       maintenance and sustainment of subsystems of major weapon systems,
                                       among other things. In addition to current policies implementing the
                                       Reform Act, officials stated that key provisions from the Reform Act will
                                       also be included in the updated instruction.


                                       13
                                         DOD Principal Deputy Under Secretary of Defense for Acquisition, Technology and
                                       Logistics, “Improving Milestone Process Effectiveness,” (June 23, 2011).
                                       14
                                            DOD Instruction 5000.02, Operation of the Defense Acquisition System, Dec. 2, 2008.
                                       15
                                            Pub. L. No. 112-81 § 832.




                                       Page 12                                           GAO-13-103 Weapons Acquisition Reform
In addition to implementing the provisions of the Reform Act, DOD offices
have taken other steps to strengthen acquisition capabilities throughout
the department. For example:

•     The SE office, according to DOD officials, led efforts to establish
      working groups to help the services address systemic reliability issues
      across the unmanned aircraft and rotary wing portfolios earlier in the
      process. The office also led several workforce development initiatives
      to attract and retain a qualified engineering workforce and supported
      the implementation of legislation requiring each acquisition program
      office to name a key technical advisor who is responsible for all
      engineering activities.

•     The DT&E office, according to DOD officials, championed updates to
      DOD Instructions that will require weapon acquisition programs to
      consider using DOD test capabilities before paying contractors to
      develop similar capabilities. In addition, the office supported
      legislation requiring major defense acquisition program offices to have
      a government test agency serving as the lead developmental test and
      evaluation organization for the program and a chief developmental
      tester. The chief developmental tester position, as included in the
      National Defense Authorization Act for Fiscal Year 2012, is tasked
      with coordinating the planning, management, and oversight of all
      developmental testing activities, among other things.16

•     The CAPE office, according to DOD officials, established an operating
      and support cost directorate to build its expertise and place more
      emphasis on developing better operating and support cost estimates
      throughout the acquisition life cycle. This directorate will coordinate
      the development of an operating and support cost estimating
      guidebook.

•     The PARCA office, according to DOD officials, is providing additional
      insights to the Under Secretary of Defense for Acquisition,
      Technology and Logistics on systemic acquisition problems.
      Specifically, the office is examining a wide range of acquisition-related
      information from the past 40 years, such as contract type, stability of
      key performance parameters, and program manager tenure to




16
     Pub. L. No. 112-81 § 835.




Page 13                                     GAO-13-103 Weapons Acquisition Reform
                             determine if there is any statistical correlation between these factors
                             and good or poor acquisition outcomes.

                        We identified four key areas where the Reform Act had a significant
The Reform Act Is       influence on programs in the 3 years since it was enacted:
Helping Programs        (1) requirements, (2) cost and schedule, (3) testing, and (4) reliability.
                        These four areas have been common sources of problems in the past.
Identify and Mitigate   For example, the services typically started new weapon acquisition
Risks Earlier in the    programs with requirements that were both demanding and inflexible and
                        planned to use relatively unproven technologies to meet the
Acquisition Process     requirements—all of which increased program risks. In addition, cost and
                        schedule estimates were frequently too optimistic based on the proposed
                        requirements and technologies. Design problems stemming from rigid
                        requirements and the use of immature technologies to meet them were
                        often discovered during testing and fixed late in the development cycle
                        and resulted in cost increases, performance shortfalls, and schedule
                        delays. Finally, DOD’s inattention to reliability has resulted in a dramatic
                        increase in the number of systems that have not met suitability
                        requirements during operational testing. Deficiencies—such as high
                        failure rates and disappointing improvements in the reliability, availability,
                        and maintainability of weapon systems—have limited program
                        performance and increased operation and support costs.17 We examined
                        11 programs at various stages of the acquisition process to determine
                        how the offices and policies established as a result of the Reform Act
                        impacted their acquisition strategies and decision-making process. Four
                        programs had not yet passed Milestone B, development start, at the time
                        we began our review. Of the remaining seven programs, three had
                        breached Nunn-McCurdy cost thresholds since the act was passed and
                        have had to satisfy the Reform Act’s new requirements with regards to
                        certification. The other programs had significant interaction with one or
                        more of the OSD offices established by the Reform Act. Table 3 indicates
                        in which of the four areas each program has been affected by the Reform
                        Act.




                        17
                          DOD Report of the Defense Science Board Task Force on Developmental Test and
                        Evaluation (May 1, 2008).




                        Page 14                                      GAO-13-103 Weapons Acquisition Reform
Table 3: Reform Act Influence on Case Study Programs

Program                                Requirements                Cost and schedule            Testing                  Reliability
                                                       Before Milestone B
Ground Combat Vehicle                                                                                                       
                               a
Joint Light Tactical Vehicle                                                                                                
Ohio Class Replacement                                                                                                      
                         a
Ship to Shore Connector                                                                                                       
                                                        After Milestone B
Joint Strike Fighter                                                       
Global Hawk                                                                                                                  
Gray Eagle                                                                                                                  
KC-46 Tanker                                                                                                                  
Littoral Combat Ship Seaframe                                              
Remote Minehunting System                                                                                                    
Small Diameter Bomb II                                                                                                       
                                       Source: GAO analysis of DOD data.
                                       a
                                        During the course of our review, the Joint Light Tactical Vehicle and Ship to Shore Connector
                                       programs held a Milestone B review.


                                       In some cases, programs have made changes based on input from OSD
                                       offices like systems engineering or developmental test and evaluation; in
                                       other cases, programs have integrated Reform Act policies, such as
                                       preliminary design reviews and competitive prototyping, into their
                                       acquisition planning. Programs that were already in development or
                                       production when the Reform Act was passed were less likely to have
                                       interactions with the OSD offices on requirements trades because these
                                       discussions typically occur prior to Milestone B. A discussion of how
                                       individual programs have been affected in the areas of requirements, cost
                                       and schedule realism, testing, and reliability follows.


Greater Emphasis Is Being              The Reform Act places significant emphasis on early problem solving and
Placed on Requirements,                requires programs to put much more effort toward considering trade-offs
but Challenges Remain                  among cost, schedule, and performance requirements prior to Milestone
                                       B. As part of this effort, it requires the Secretary of Defense to ensure that
                                       acquisition, budget, and cost estimating officials have the opportunity to
                                       raise cost and schedule matters before performance objectives are
                                       established. The Reform Act also charges the Joint Requirements
                                       Oversight Council with the responsibility to ensure that cost, schedule,
                                       and performance trade-offs for joint military requirements are considered,



                                       Page 15                                                 GAO-13-103 Weapons Acquisition Reform
and to include combatant commanders in the process to ensure the
user’s needs are adequately satisfied. The offices established as a result
of the Reform Act have helped programs, such as the Joint Light Tactical
Vehicle and Ground Combat Vehicle, make trade-offs among cost,
schedule, and technical performance requirements. As a result, these
programs have developed a more realistic acquisition strategy from a
cost, schedule, and technical standpoint.

•   Joint Light Tactical Vehicle: The program held several reviews prior
    to Milestone B to identify, modify, or eliminate requirements that were
    unachievable or unaffordable, thus establishing a more technically
    realistic program. Officials from DT&E and SE participated in these
    reviews. By involving both the requirements and acquisition
    communities in the reviews, the Army was able to reduce the required
    capability to cut costs while ensuring that trade-off decisions would
    not impair the system’s ability to meet the warfighter’s operational
    needs. Examples of requirements changes that helped to cut costs as
    well as reduce risk include:
    •   allowing the active suspension system, crew displays, and
        integrated starter-generator to be tradable design features. These
        changes resulted in a 30 percent reduction in the average unit
        manufacturing cost from the initial target of $475,000 to $331,000,
        while at the same time reducing technical and weight risk.
        According to program officials, this makes the $250,000 unit
        manufacturing goal more achievable.
    •   reducing the reliability requirement and changing the Army
        helicopter lift requirement based on the results of technology
        development prototype testing. This mitigated technical risks
        going into development.

    The program recently moved into engineering and manufacturing
    development, but not all requirements issues have been resolved and
    future trade-offs may be necessary. For example, early testing
    showed that none of the three prototype variants met the program’s
    soft soil or sand slope requirement. This requirement has not been
    changed. Program and OSD officials are monitoring this issue closely
    and plan to actively manage it during engineering and manufacturing
    development.

•   Ground Combat Vehicle: The Ground Combat Vehicle program
    exhibited some of the same problems experienced by previous DOD
    programs prior to Milestone B—demanding and inflexible
    requirements. The SE office and the Under Secretary of Defense for



Page 16                                  GAO-13-103 Weapons Acquisition Reform
                              Acquisition, Technology and Logistics are helping the program set
                              achievable requirements. Following its materiel development decision
                              in February 2010, the program issued a request for proposals that
                              contained nearly 1,000 requirements and a challenging 7-year
                              schedule for the delivery of the first production vehicle. At the request
                              of the Under Secretary for Acquisition, Technology and Logistics, the
                              Army established an independent review team to assess the risks
                              associated with the program’s schedule. The team, which included an
                              SE official, raised concerns about the program’s high number of
                              mandatory requirements and the risks associated with the 7-year
                              schedule. To mitigate program risks, the Army reduced the number of
                              performance requirements by about 25 percent and prioritized the
                              others, giving competing contractors flexibility in addressing some
                              performance requirements. The Army issued a revised request for
                              proposal in November 2010. In August 2011, the Under Secretary of
                              Acquisition, Technology and Logistics approved the program’s entry
                              into technology development, but expressed concern about the cost
                              and schedule risks associated with delivering a production vehicle in 7
                              years. Because of these concerns, the Under Secretary directed the
                              Army to consider other alternatives, such as existing vehicles, that
                              could meet warfighter needs. The analysis is currently planned to be
                              completed in March 2013 to inform the Milestone B decision.

Program Cost and          By establishing a new cost assessment and program evaluation office
Schedule Estimates Are    and requiring this office to scrutinize program cost and schedule
Becoming More Realistic   estimates beginning at Milestone A, CAPE officials believe that the
                          Reform Act has helped infuse more realism in cost estimates and
                          promote earlier discussions about affordability. CAPE officials also
                          believe that because their independent cost and schedule estimates have
                          become more visible within DOD and Congress, the military services are
                          developing more realistic estimates. We saw evidence of these benefits in
                          the programs we reviewed, including the Ohio Class Replacement,
                          Littoral Combat Ship Seaframe, and F-35 Joint Strike Fighter programs.

                          •   Ohio Class Replacement: The CAPE office was involved in the
                              decision-making process to ensure program affordability. The office
                              prepared an independent cost estimate and reviewed the program’s
                              affordability goals prior to Milestone A. The service and independent
                              estimates were within 2 percent of each other. However, the Under
                              Secretary of Defense for Acquisition, Technology and Logistics
                              directed the Navy to do a rigorous cost comparison of a 16 missile
                              tube design versus a 20 missile tube design. The Navy determined
                              that a 16 missile tube configuration would meet warfighter



                          Page 17                                   GAO-13-103 Weapons Acquisition Reform
    requirements and users’ needs while reducing program costs by
    about $200 million per submarine, or approximately $3 billion for the
    total program. It would also simplify the ships’ design and integration
    effort. The CAPE office validated the savings associated with the 16
    missile tube design. As a result, the Navy incorporated the 16 missile
    configuration as the program baseline.

•   Littoral Combat Ship Seaframe: The CAPE office helped make
    program costs more visible. Prior to the program’s Milestone B
    decision, CAPE completed an independent cost estimate of the
    seaframe program and found that the resources in the future years’
    defense plan budget were lower than the projected program costs for
    the same time period. Navy officials attributed this problem to the
    overlap between the timing of the milestone decision and the
    president’s budget submission. The office further noted that the
    resources in the Navy’s budget did not include the additional
    development activities required to support two full ships. Without this
    information, decision makers would not have had visibility into the
    expected costs of the seaframe program or be able to make more fully
    informed decisions. As a result, the Navy re-phased its funding in the
    budget, adding approximately $397 million to fully fund the
    development program.

    Joint Strike Fighter: The SE and CAPE offices helped the program
    develop more realistic cost and schedule estimates. CAPE officials
    have been involved in reviews of the F-35 Joint Strike Fighter program
    even prior to the passage of the Reform Act and have continued to be
    heavily involved in subsequent programs reviews. For example, the
    cost analysis improvement group, which was the predecessor to
    CAPE, led a multi-functional joint estimating team review of the
    program in 2008. This review found problems with the program’s
    funding and schedule. In 2010, the Joint Strike Fighter program
    notified Congress that its estimated unit costs had increased by more
    than 80 percent since the original Milestone B baseline in 2001. This
    increase triggered a Nunn-McCurdy unit cost breach and later
    prompted the program executive officer to commission a technical
    baseline review of the program to help determine the resources
    needed to complete development. Officials from the SE office
    participated in this technical review. The CAPE office also did an
    independent cost estimate of the program as part of the Nunn-
    McCurdy certification process. Based on information from these
    efforts, DOD developed a more realistic program plan by adding $4.6
    billion to the development program, reducing near-term procurement
    quantities by 125 aircraft, and extending the development test period


Page 18                                   GAO-13-103 Weapons Acquisition Reform
                               by 4 years to accommodate developmental testing, address the
                               increased program scope, and fix software issues.

Programs Are Adding More   The Reform Act significantly strengthened the role of developmental
Tests to Reduce            testing in the department. In the 2 decades prior to the Reform Act, the
Development Risks          prominence of developmental testing had declined within OSD. In the
                           early 1990’s, developmental testing was part of an all-encompassing test
                           organization that reported directly to the Under Secretary of Defense for
                           Acquisition. According to a former senior developmental testing official, by
                           2004, two people worked on developmental testing activities within the
                           systems engineering organization. In establishing a separate office for
                           DT&E, the Reform Act reinforced the need for robust developmental
                           testing early in the acquisition process. The Reform Act gave the Deputy
                           Assistant Secretary for Developmental Test and Evaluation formal
                           approval authority for the test and evaluation master plans of major
                           defense acquisition programs. This authority enables the Deputy
                           Assistant Secretary to help ensure that programs have robust test and
                           evaluation plans. Our case study reviews illustrate the efforts that the
                           DT&E office has made to help programs such as the Small Diameter
                           Bomb II and KC-46 Tanker obtain more design and performance
                           knowledge early in a program’s acquisition life cycle.

                           •   Small Diameter Bomb II: Air Force program officials acknowledged
                               that developmental and operational testing officials worked closely
                               with them as they prepared the test and evaluation master plan for the
                               Milestone B decision. After reviewing the plan, developmental and
                               operational test officials concluded that the program would benefit
                               from adding a 28-shot test program prior to entering operational
                               testing. According to the program office, the purpose of the additional
                               testing is to further establish the performance of the weapon in
                               realistic scenarios and to increase the likelihood of completing
                               operational testing without a failure. DT&E officials stated this testing
                               would provide more complete knowledge about the bomb’s
                               functionality and help reduce risk of a major redesign moving forward.
                               Program officials stated that they allocated an additional $41 million to
                               its developmental test program to conduct the 28 additional flight tests
                               prior to operational testing.

                           •   KC-46 Tanker: The program office acknowledged that the DT&E
                               office, as part of an integrated test team comprised of government
                               and industry officials, helped identify options that could add time to the
                               test plan for important testing if unexpected delays are encountered.
                               In its fiscal year 2011 annual report, the operational test and


                           Page 19                                    GAO-13-103 Weapons Acquisition Reform
                                evaluation office reported that the KC-46’s planned flight test program
                                was not executable, determining that more time would be needed for
                                military flight-testing. It based this conclusion on the historical flight
                                test experience of similar programs. Program officials stated that they
                                were initially reluctant to change the test plan because they had
                                awarded a fixed price contract and any changes could result in
                                reopening the contract, leading to potential cost increases. However,
                                the integrated test team identified a recovery period that may be
                                applied to the KC-46 aerial refueling certification if delays are
                                encountered. The contractor now has a plan that could allocate an
                                additional 1.5 months for two test aircraft to complete this testing, if
                                deemed necessary. This testing would provide more knowledge about
                                the program’s aerial refueling performance prior to operational testing.
                                DT&E officials stated that they plan to continue working with the
                                program to address overall flight test challenges. While testing
                                remains one of the program’s risk areas, this change may lessen that
                                risk.

Greater Emphasis Is Being   The Reform Act emphasizes the need for designing more reliable weapon
Placed on Weapon System     systems. It charges the Deputy Assistant Secretary for Systems
Reliability                 Engineering with the responsibility to ensure the systems engineering
                            approach used by major acquisition programs includes a robust plan for
                            improving reliability. The DT&E office reviews programs’ reliability growth
                            test plans. This testing provides visibility over how reliability is improving
                            and uncovers design problems so fixes can be incorporated before
                            production begins. A reliability growth curve is used to track projected and
                            actual improvements in reliability over time. The Reform Act further
                            requires that the Deputy Assistant Secretary for Systems Engineering
                            develop policies and guidance for the inclusion of provisions relating to
                            systems engineering and reliability growth in requests for proposals. We
                            observed evidence of this increased emphasis in the Joint Light Tactical
                            Vehicle, Remote Minehunting System, Gray Eagle, and Global Hawk
                            programs.

                            •   Joint Light Tactical Vehicle: The DT&E office helped this program
                                develop a more realistic reliability growth plan prior to Milestone B.
                                Based on the performance of prototype vehicles, developmental test
                                officials determined that the program’s reliability growth curve was
                                unrealistic. For example, officials reported that the program’s initial
                                reliability growth plan assumed a starting reliability that was almost 60
                                percent higher than what had actually been demonstrated during
                                technology development. It also assumed commonality between the
                                two vehicle variants, a large reliability increase in a short test time,



                            Page 20                                    GAO-13-103 Weapons Acquisition Reform
     and two corrective action periods. The DT&E office recommended
     that the program eliminate the vehicle commonality assumption, add
     more test miles, and add another corrective action period to its test
     plan. It also recommended that the program consider lowering the
     vehicle’s reliability requirement. Based on this input, the Army revised
     its plan by adding two vehicles and 40,000 more test miles to ensure
     reliability is adequately addressed for both variants. With approval of
     the user, it also reduced the reliability requirement from 3,600 to 2,400
     miles mean time between operational mission failures.

•    Remote Minehunting System: The SE office worked with program
     officials to improve reliability growth planning, which was found to be
     one of the key factors leading to the program’s Nunn McCurdy unit
     cost breach in 2009. Before the breach, program officials had not
     funded a reliability growth program or established a design for
     reliability process. The program had a reliability goal of 150 hours
     mean time between failures, but program officials stated that testing
     demonstrated a reliability of only about 45 hours. Since the breach,
     the program has worked closely with the SE office to establish a
     reliability program plan and a growth curve to track reliability
     improvements. During the Nunn-McCurdy review, the program
     developed a three phase reliability growth program to improve the
     program’s subsystems, components, and manufacturing processes
     that contributed to poor reliability. According to program officials,
     phase one of the reliability growth program was completed in 2011,
     and reliability has improved by 40 percent, going from 45 hours mean
     time between operational mission failures to 63 hours. Although this
     improvement is still below the minimum requirement of 75 hours,
     program officials stated that phase two of the reliability growth
     program is scheduled to be completed in April 2013 and is projected
     to achieve the program’s 75 hour minimum requirement.

•    Gray Eagle and Global Hawk: The SE office has worked to improve
     reliability across the unmanned aircraft portfolio, including the Gray
     Eagle and Global Hawk. Prior to the Gray Eagle’s second low rate
     initial production decision in 2011,18 SE officials raised concerns about
     the system’s poor reliability. As a result, the Army was directed to
     undertake a reliability improvement program. The Under Secretary of


18
  Low-rate initial production is production of the system in the minimum quantity
necessary to provide articles for operational tests, establish an initial production base, and
permit an orderly increase in the production rate for the system.




Page 21                                             GAO-13-103 Weapons Acquisition Reform
                                 Defense for Acquisition, Technology and Logistics approved the
                                 program for low rate initial production, but stressed the need to
                                 improve the operational reliability as quickly as possible. SE officials
                                 worked with program officials to establish a reliability working group,
                                 develop reliability growth curves, and develop a reliability
                                 enhancement management plan. According to SE officials, the Gray
                                 Eagle initially improved the reliability of the aircraft by 15 percent and
                                 the ground control station by 30 percent. According to program
                                 officials, the initial reliability goals were overstated and not needed to
                                 meet the program’s overall operational availability requirement. Based
                                 on initial operational test results in August 2012, the program office is
                                 working with the user to redefine the reliability goals without impacting
                                 the system’s ability to meet its overall operational availability
                                 requirement. According to the PARCA office, these efforts have been
                                 informed by a detailed reliability model that they built in consultation
                                 with the Army. This model showed the relationship between the
                                 aircraft’s reliability and its availability to perform operational missions.
                                 SE officials also found similar reliability problems on the Global Hawk
                                 program and worked with program officials to establish a reliability
                                 growth and improvement plan and reliability growth curves. According
                                 to SE officials, the time between unscheduled maintenance on the
                                 Global Hawk has improved on the order of 50 to 80 percent.

                             While DOD has taken steps to implement most of the fundamental
Challenges Exist That        Reform Act provisions, some key efforts to date have been primarily
Could Limit the              focused on DOD’s largest major defense acquisition programs.
                             Expanding the reach of the Reform Act to bring about systemic change to
Reform Act’s Ability         DOD’s weapons acquisition process so that it influences all programs,
to Influence Systemic        however, still has challenges. Although senior leaders were receptive to
                             the Reform Act principles, they identified several challenges that currently
Change                       limit DOD’s ability to broaden the Reform Act’s influence. We grouped
                             these challenges into five general categories: (1) organizational capability
                             constraints; (2) need for additional guidance on cost estimating and
                             Reform Act implementation; (3) uncertainty about the sufficiency of
                             service level systems engineering and developmental testing resources;
                             (4) limited dissemination of lessons learned; and (5) cultural barriers.


Organizational Capability    Leaders of two of the offices established as a result of the Reform Act told
Constraints Make It          us that even though they have implemented most of the fundamental
Difficult to Expand Reform   Reform Act provisions, they have had to limit their activities to a portion of
                             acquisition programs in their portfolios due to resource constraints. Thus,
Act’s Impact                 it is doubtful that they could expand the scope of their activities to include


                             Page 22                                     GAO-13-103 Weapons Acquisition Reform
                           more weapon acquisition programs at current staffing levels. For
                           example, the DT&E office has had to be selective in its level of oversight
                           of acquisition programs because the current staff of around 70
                           government and contractor personnel cannot adequately cover a portfolio
                           of over 200 acquisition programs, according to its Deputy Assistant
                           Secretary. The office has dropped virtually all but the largest programs
                           from its oversight list and eliminated oversight of some major automated
                           information systems. CAPE officials estimated that its cost assessment
                           division would need to double in size in order to meet the Reform Act’s
                           requirements. However, soon after the Reform Act was enacted,
                           budgetary constraints limited the expansion of the cost estimating
                           workforce to about 25 percent of the necessary growth. According to
                           CAPE officials, its current cost analysis staffing is not adequate to meet
                           its mission of improving the analytical skills of the defense cost estimating
                           workforce, issuing policy, and providing sound and unbiased cost and
                           schedule estimates. The office has delegated its independent cost
                           estimating responsibility for most major automated information systems to
                           the military services and some guidance has yet to be issued.

                           The SE and PARCA offices are also struggling in some regards. For
                           example, according to its Deputy Assistant Secretary, the SE office is
                           continuously challenged to maintain the high caliber, qualified personnel
                           required to provide assistance to and oversight of its portfolio of over 200
                           acquisition programs. Further, PARCA officials stated that the availability
                           of government positions, particularly at the senior executive service level,
                           continues to be a critical issue for the office. The two divisions within the
                           PARCA office, the performance assessments division and the root cause
                           division, do not currently have permanent government personnel at the
                           senior executive level. Officials also stated current proprietary information
                           rules limit the ability of PARCA contractor personnel to handle and
                           maintain some weapon system information, severely impeding
                           operations.


Services Would Like More   Offices within OSD have not yet issued more detailed guidance that could
Cost Estimating and        help institutionalize better cost estimating practices and steer program
Reform Act                 decisions related to competitive prototyping and preliminary design
                           reviews. The CAPE office has not issued guidance for operating and
Implementation Guidance    support costs estimates, such as fuel and maintenance costs, that have
                           been estimated to account for two-thirds or more of a system’s total life




                           Page 23                                   GAO-13-103 Weapons Acquisition Reform
cycle cost.19 In addition, although not specifically required by the Reform
Act, the CAPE office has not issued guidance for the services to use
when developing Milestone A program cost estimates. As a result, senior
leaders may not have access to realistic cost estimates prior to Milestone
B for decision making purposes. Military service officials told us they are
particularly interested in getting guidance on data that should be included
in the cost analysis requirements description, which forms the basis for
their cost estimates at Milestones B and C. CAPE officials recognize that,
while some progress has been made, they need to complete the
guidance, but have not been able to dedicate resources to do so.

Some officials also told us that they find the competitive prototyping and
preliminary design review requirements confusing and would like
guidance on how to implement these requirements. DOD policy requires
the technology development strategy for major defense acquisition
programs to provide for prototypes of the system or, if a system prototype
is not feasible, for prototypes of critical subsystems before Milestone B
approval.20 However, officials from the Ground Combat Vehicle program
were unclear as to when and what type of prototype to use. From a
broader perspective, other military officials questioned the value of
competitive prototyping as a blanket requirement for all programs,
especially for programs that are using mature technologies, given the
cost. For example, senior acquisition officials questioned the necessity of
spending $400 million on competitive prototyping for the Small Diameter
Bomb II program since the program was aware of problems with one
contractor’s design. However, program officials indicated that competitive
prototyping enabled them to identify design issues early in development
and realize a savings of $1 billion. Officials from the Ground Combat
Vehicle program we spoke with also indicated that they struggled with the
timing of when to hold the program’s preliminary design review and what
type of knowledge was required, since better guidance is needed. The
program plans to hold multiple design reviews prior to Milestone B to
consider contractor and government designs of the weapon system and
then hold another review after Milestone B in order to resolve differences
between the government’s and selected contractor’s preliminary designs.



19
  The Reform Act required CAPE to issue guidance relating to full consideration of life
cycle management and sustainability costs in major defense acquisition programs.
20
  DOD Directive-Type Memorandum 09-027, Implementation of the Weapon Systems
Acquisition Reform Act of 2009 (Dec. 4, 2009, incorporating Change 3, Dec. 9, 2011).




Page 24                                           GAO-13-103 Weapons Acquisition Reform
                          We spoke with OSD officials to determine which office should be
                          providing guidance or assistance to program managers on competitive
                          prototyping and preliminary design review issues. None of the offices
                          have official responsibility for these efforts. OSD officials stated that these
                          are program decisions and should be discussed with their respective
                          military service level acquisition officials.


Services May Not Have     OSD officials believe that the services may lack resources in key
Sufficient Resources to   positions that could help strengthen systems engineering and
Oversee and Conduct       developmental testing activities on weapon acquisition programs. For
                          example, according to the Deputy Assistant Secretary of Defense for
Systems Engineering and   Systems Engineering, the Navy and Air Force have reassigned the duties
Developmental Testing     and responsibilities of their service-level chief engineers, thereby de-
Activities                emphasizing the importance of systems engineering. The Deputy
                          Assistant Secretary believes maintaining strong systems engineering
                          leadership at the service level is essential for tying the systems
                          engineering community together and promoting good systems
                          engineering practices throughout each respective service. According to
                          the DT&E and SE office’s March 2012 joint annual report to the
                          Congress, the Navy abolished its chief engineer position and while the Air
                          Force recently began to take steps to relocate the systems engineering
                          function to the headquarters level, the impact of a recent reorganization
                          on systems engineering activities is not yet known. In addition, the Deputy
                          Assistant Secretary for Developmental Test and Evaluation expressed
                          concern that the military services may not be implementing new
                          legislation that requires each major defense acquisition program be
                          supported by a chief developmental tester that oversees developmental
                          test and evaluation activities. He stated that in some cases one person is
                          serving as the chief developmental tester across multiple programs
                          instead of having one person dedicated specifically to each program.21
                          The Deputy Assistant Secretary is trying to determine the extent to which
                          this practice is occurring and then plans to work with the services to get
                          more focused leadership for each program.




                          21
                            The requirement contained in the National Defense Authorization Act for Fiscal Year
                          2012 required “each major defense acquisition program to be supported by a chief
                          developmental tester.” However, it did not specifically prohibit one person serving as the
                          chief developmental tester across multiple programs.




                          Page 25                                           GAO-13-103 Weapons Acquisition Reform
                           It is also unclear whether the services have a sufficient number of
                           qualified personnel to conduct systems engineering and test and
                           evaluation activities. The services planned to grow these workforces by
                           over a combined 5,000 people between fiscal years 2009 and 2015 and
                           had made progress in growing each of these workforces through fiscal
                           year 2010. However, budget cuts have resulted in DOD canceling some
                           of its weapon acquisition programs and reassessing its decision to
                           increase the acquisition workforce. Last year, we recommended that the
                           Secretary of Defense report the impact budget cuts were having on the
                           military service workforce and their ability to meet weapon acquisition
                           program needs in the areas of developmental testing and systems
                           engineering.22 In the DT&E and SE offices’ March 2012 joint annual report
                           to the Congress, the Deputy Assistant Secretary for Systems Engineering
                           reported that the Army has reduced its systems engineering workforce
                           growth plan as compared to the plan reported in March 2011 joint annual
                           report, and that contractor-to-civilian conversions have been suspended.
                           In addition, the Deputy Assistant Secretary believes a prolonged hiring
                           freeze in the Air Force could potentially create new experience gaps in
                           the workforce. The Deputy Assistant Secretary for Developmental Test
                           and Evaluation did not discuss the impact of budget cuts on the services’
                           test and evaluation workforce growth plan in the March 2012 joint annual
                           report to the Congress and neither office reported on whether the
                           services had an adequate workforce to meet the needs of the current
                           portfolio of weapon acquisition programs.


Expanded Use of Lessons    DOD has not taken full advantage of sharing lessons learned obtained
Learned Would Help         through root cause analyses of programs that experience Nunn-McCurdy
Expand Impact across the   cost and schedule breaches with the acquisition workforce, particularly
                           program managers. According to the Defense Acquisition Guidebook,
Acquisition Portfolio      which provides best practices the acquisition workforce can use on
                           programs, lessons learned are a tool that the program manager may use
                           to help identify potential areas of risk associated with a weapon
                           acquisition system by reviewing the experiences encountered in past
                           programs. Lessons learned databases document what worked and what
                           did not work in past programs, in the hopes that future programs can
                           avoid the same pitfalls. Further, if the right best practices are applied,
                           they help to avoid common problems and improve quality, cost, or both.



                           22
                                GAO-11-806.




                           Page 26                                  GAO-13-103 Weapons Acquisition Reform
                           The PARCA office has made some effort to educate program managers
                           on how to avoid acquisition problems through classes taught at the
                           Defense Acquisition University. However, these courses are geared
                           towards educating new program managers and may not be reaching a
                           wide range of program officials. Nevertheless, some officials indicated
                           that this information would be helpful for program officials to understand
                           and avoid problems that have affected weapon acquisition programs in
                           the past. Other officials also stated that it would be helpful if root cause
                           analysis assessments contained more detailed information so acquisition
                           officials could better understand problems and apply lessons learned. For
                           example, when cost estimating was determined to be a root cause of a
                           problem, officials stated they would have found it more beneficial to know
                           if immature technologies or unrealistic requirements were the basis for
                           the poor cost estimate.


Systemic Changes Will Be   Perhaps the most difficult challenge the department faces in making
Difficult Until Cultural   systemic changes to the acquisition process is changing the cultural
Barriers Are Addressed     relationship between the military services, which fund and develop new
                           weapon acquisition programs, and OSD offices, which provide advice to
                           and oversee the programs. Senior military service officials have told us
                           they believe they understand and can manage the risks of specific
                           weapon acquisition programs without much assistance from OSD. On the
                           other hand, OSD officials believe more assistance is needed, as
                           evidenced by the high number of programs that have experienced Nunn-
                           McCurdy breaches and poor operational testing results. For example,
                           since it was established in 2009, the DT&E office has assessed whether
                           15 programs were ready to begin operational testing. The office
                           recommended that 5 of the programs—Global Hawk Block 20/30,
                           Standard Missile 6, Joint Tactical Radio System Handheld Manpack
                           Small Form (HMS) Rifleman Radio, Joint Tactical Radio System HMS
                           Manpack, and MQ-1C Gray Eagle—not proceed into operational testing.
                           However, military service acquisition chiefs decided to allow all 5 of these
                           programs to proceed anyway. Four of the programs—Global Hawk Block
                           20/30, Standard Missile 6, Joint Tactical Radio System HMS Rifleman
                           Radio and Joint Tactical Radio System HMS Manpack—demonstrated
                           poor performance in operational testing, in areas such as reliability,
                           effectiveness, or suitability. Operational testing results for the MQ-1C
                           Gray Eagle have not yet been reported.

                           On the other hand, a few service officials we met with were reluctant to
                           accept some recommendations made by OSD offices because they
                           believed the recommendations were overly burdensome and could


                           Page 27                                   GAO-13-103 Weapons Acquisition Reform
              significantly impact weapon acquisition programs’ cost and schedule
              outcomes without a lot of benefit. This was the case for the KC-46 Tanker
              program, where program officials were concerned that additional testing
              recommended by the developmental test and evaluation, and operational
              testing offices, as part of the integrated test team, could have significant
              contractual implications during development. In this case, officials
              identified additional flight test opportunities without having to renegotiate
              the fixed price contract. However, the additional allotted test time is not
              equivalent to the 6 to 8 months the developmental testing office felt
              should be added. A similar situation occurred on the Ship to Shore
              Connector program. The Navy disagreed with a DT&E office
              recommendation to conduct full system testing prior to procuring
              additional craft during initial production. The DT&E office believed the
              program was high risk because the Ship to Shore Connector was a
              complete redesign of a previous system with no reuse of any major
              component (engines, gearboxes, hydraulics, command and control
              software). Navy officials, however, believe the program is low risk since it
              is an evolutionary program and has one critical technology, a fire
              suppression system, which has already been sufficiently demonstrated
              and qualified through test and evaluation. In addition, the Navy estimated
              that it would cost $15 million to revise the existing production schedule to
              accommodate the full system testing as recommended by the DT&E
              office. The DT&E office and the Navy reached a compromise whereby
              OSD would review available system test results before more craft are
              authorized.


              Current fiscal pressures, along with the threat of more to come, have
Conclusions   DOD officials looking for ways to increase buying power by controlling
              cost and schedule overruns on weapon acquisition programs. The offices
              established as a result of the Reform Act, as well as policy provisions
              have helped DOD make inroads towards putting weapon acquisition
              programs on more solid footing. Together, the offices and policy
              provisions place more attention on requirements, costs, testing, and
              reliability as early as Milestone A. The provisions of the act, when
              specifically focused on newer programs, are having a positive impact on
              the programs and the acquisition process. They show that expert
              attention to the cost and achievability of capability requirements, the
              assumptions made for cost and funding of programs, and the amount of
              systems engineering knowledge that is brought to bear early make
              programs more executable.




              Page 28                                   GAO-13-103 Weapons Acquisition Reform
Although senior officials we spoke with throughout the department are
receptive to the broad principles of the Reform Act, it is too early to tell if
the Reform Act is going to result in systemic change to DOD’s weapon
acquisition process. DOD faces several challenges that must be
addressed to get lasting change—organizational capability constraints,
the need for additional cost estimating and implementation guidance, the
possibility of insufficient systems engineering and developmental testing
resources, limited dissemination of lessons learned, and cultural barriers
between OSD and the services. Some challenges appear to be straight
forward to address, such as providing guidance for estimating operating
and support costs, providing additional guidance for conducting
preliminary design reviews and competitive prototyping activities, and
disseminating lessons learned to the broader acquisition community.
However, they may require more resources, which have been difficult to
obtain.

For Reform Act policies and practices to have a systemic effect across
the entire portfolio of weapon system acquisition programs, the
department must also address challenges related to systems engineering
and developmental testing resources and cultural barriers between OSD
and the services. This begins with the services identifying key leaders at
the headquarters level and within program offices to guide systems
engineering and developmental testing efforts and then ensuring that
there are enough trained staff to carry out these activities. OSD will need
to continue monitoring the services’ efforts. It will also require an
environment where the services stop proposing new weapon systems
with inflexible requirements, immature technologies, and cost, schedule,
and funding assumptions that are too optimistic at the start of a program.
Breaking down cultural resistance to change will take more cooperation
between the Under Secretary for Acquisition, Technology and Logistics
and other OSD offices, and service acquisition executives to address, as
well as continuity of leadership. Efforts by the PARCA office to identify
factors that correlate to good or poor acquisition outcomes, particularly as
it relates to program manager tenure, will be beneficial. The services’
ability to demonstrate that the Reform Act is influencing all weapon
acquisition programs, not just the biggest, will be a key indicator for
determining whether the Reform Act has had a positive effect on DOD’s
culture.




Page 29                                     GAO-13-103 Weapons Acquisition Reform
                      We recommend that the Secretary of Defense take the following four
Recommendations for   actions to enable systemic change across the entire portfolio of weapon
Executive Action      acquisition programs:

                      •   direct the Director of Cost Assessment and Program Evaluation to
                          issue guidance for estimating weapon acquisition program costs at
                          Milestone A and operating and support costs throughout the
                          acquisition life cycle by the end of fiscal year 2013 and ensure that the
                          office prioritizes its resources accordingly to accomplish this task;
                      •   designate responsibility for providing advice and guidance to program
                          offices on competitive prototyping and preliminary design reviews to
                          the appropriate organization within the Office of the Under Secretary
                          of Defense for Acquisition, Technology and Logistics and ensure that
                          the guidance is developed. The office(s) designated would be the
                          focal point for addressing program office issues related to the practical
                          implementation of these Reform Act provisions, such as the type of
                          competitive prototyping to use, the timing and benefits of holding
                          preliminary design reviews prior to milestone B, and if a preliminary
                          design review should be held after milestone B;
                      •   direct the Deputy Assistant Secretaries of Defense for Systems
                          Engineering and Developmental Test and Evaluation to assess and
                          include in their annual report to the Congress beginning with the
                          report on fiscal year 2012 activities:
                          •    the extent to which the office can perform their required activities
                               with allocated resources;
                          •    the impact budget cuts are having on the military services total
                               workforce (civilians, military, and contractors) and ability to meet
                               program office needs; and
                          •    progress the services have made filling leadership positions, such
                               as chief engineers at the service level and technical leads for
                               systems engineering and developmental testing at the program
                               office level;
                      •   direct the Director of Performance Assessments and Root Cause
                          Analyses to make lessons learned collected during its root cause
                          analysis evaluations available to the acquisition workforce and ensure
                          that the office prioritizes its resources accordingly.


                      DOD provided us written comments on a draft of this report. DOD
Agency Comments       concurred with two recommendations and partially concurred with two
and Our Evaluation    others. DOD’s comments appear in appendix III. DOD also provided
                      technical comments, which we incorporated as appropriate in the report.




                      Page 30                                   GAO-13-103 Weapons Acquisition Reform
DOD agreed with the intent of our first recommendation, but noted that
due to resource constraints, the Cost Assessment and Program
Evaluation office could not guarantee that it would be able to issue
guidance for estimating major defense acquisition program costs at
Milestone A and operating and support costs throughout the acquisition
lifecycle by the end of fiscal year 2013. We continue to believe that the
Cost Assessment and Program Evaluation office should issue the
guidance by the end of fiscal year 2013. However, if that is not possible
from a resource standpoint, the office should commit to a date and devote
the resources to meeting that date. We will continue to monitor DOD’s
efforts to develop the guidance.

Although DOD concurred with our second recommendation, we revised
this recommendation based upon discussions with DOD officials during
the agency comment period. Our revision clarified the intent of this
recommendation, which is to have the Secretary of Defense designate a
specific organization within the Office of the Under Secretary of Defense
for Acquisition, Technology and Logistics to provide advice and guidance
on competitive prototyping and preliminary design reviews. We
understand that the department has issued Reform Act implementation
guidance and has incorporated aspects of competitive prototyping and
preliminary design reviews in the Defense Acquisition Guidebook.
Further, we recognize that program offices we visited are taking steps to
implement the guidance that has already been issued. However, based
on our discussions with senior level officials, we believe one or more
offices need to be designated with the responsibility of developing
additional guidance and answering program specific questions related to
the practical implementation of the requirements. As noted earlier in our
report, some officials questioned when to use prototyping or what type of
prototyping should be used. In addition, there were questions about the
timing of the preliminary design reviews.

DOD partially concurred with our third recommendation. DOD noted that
the type of information we recommended be assessed and reported on
should be included as part of DOD’s human capital strategic planning
process and as such, be reported in DOD’s annual Acquisition Workforce
Strategic Plan. We agree that the impact of budget cuts on the workforce
and the status of leadership positions could be addressed in the annual
strategic plan. However, we continue to believe that the Deputy Assistant
Secretaries for Systems Engineering and Developmental Test and
Evaluation should include an assessment in their joint annual report to the
Congress on the respective offices’ ability to perform activities specified in
the Reform Act with available resources.


Page 31                                   GAO-13-103 Weapons Acquisition Reform
DOD concurred with our fourth recommendation, which would make
lessons learned from root cause analyses available to the acquisition
workforce.


We are sending copies of this report to the Secretary of Defense and
appropriate Congressional Committees. In addition, the report is available
at no charge on the GAO website at http://www.gao.gov.

If you have any questions about this report or need additional information,
please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact
points for our Office of Congressional Relations and Public Affairs may be
found on the last page of this report. Key contributors to this report were
Cheryl Andrew, Assistant Director; Laura Greifner, Julie Hadley, Megan
Porter, Rae Ann Sapp, and Marie Ahearn.




Michael J. Sullivan
Director
Acquisition and Sourcing Management




Page 32                                  GAO-13-103 Weapons Acquisition Reform
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              This report examines DOD’s continued implementation of the Weapon
              Systems Acquisition Reform Act of 2009 (Reform Act). Specifically, we
              examined (1) DOD’s progress in implementing Reform Act provisions;
              (2) the impact the Reform Act has had on specific acquisition programs;
              and (3) challenges remaining in improving the weapons acquisition
              process.

              To assess DOD’s progress in implementing Reform Act provisions, we
              interviewed officials and analyzed documents, such as reports to the
              Congress and guidance issued from the Office of the Secretary of
              Defense (OSD) offices of the (1) Deputy Assistant Secretary of Defense
              for Systems Engineering (SE), (2) Deputy Assistant Secretary of Defense
              for Developmental Test and Evaluation (DT&E), (3) Cost Assessment and
              Program Evaluation (CAPE), and (4) Performance Assessments and
              Root Cause Analyses (PARCA) to determine the extent to which
              provisions have been implemented. We focused our review on the offices’
              implementation of four fundamental Reform Act provisions: developing
              policy and guidance; approving acquisition documents; monitoring
              programs and conducting program assessments; and developing
              performance measures. In cases where provisions had not been
              implemented, we asked officials about the reasons for the delay and the
              expected time frame for completion. We also interviewed officials and
              analyzed documents from the office of the Assistant Secretary of Defense
              for Research and Engineering and the Defense Procurement and
              Acquisition Policy office, as well as four weapon acquisition programs,
              which had not yet started development to determine the progress DOD
              has made implementing Reform Act provisions related to preliminary
              design reviews, competitive prototyping, and competition. We believe
              these programs offer the best glimpse at how the OSD offices and
              Reform Act policies are influencing acquisition strategies. The weapon
              acquisition programs we chose for this analysis were part of a larger case
              study review that is described below.

              To determine the impact the Reform Act has had on specific weapon
              acquisition programs, we selected 11 weapon system programs to use as
              case studies. For each program we reviewed relevant program
              documentation such as the test and evaluation master plans,
              assessments of operational test readiness, systems engineering plans,
              program support reviews, root cause analyses, analysis of alternatives
              reports and cost estimates as applicable. We also interviewed appropriate
              program officials and officials from the OSD offices for SE, DT&E, and
              CAPE to obtain their perspectives about (1) the level of interaction
              between the programs and OSD offices; (2) changes made to program


              Page 33                                 GAO-13-103 Weapons Acquisition Reform
                                        Appendix I: Objectives, Scope, and
                                        Methodology




                                        acquisition strategies as a result of interactions with the OSD offices; and
                                        (3) benefits and challenges with implementing Reform Act provisions on
                                        each of the programs. We also reviewed the performance assessments
                                        and root cause analyses office’s root cause analysis documentation for
                                        programs that incurred Nunn-McCurdy cost or schedule breaches.

                                        We selected our case studies based on input from the officials in the OSD
                                        offices for SE, DT&E, CAPE, PARCA, and operational test and
                                        evaluation. We also discussed possible case studies with GAO
                                        employees who monitor and report on weapon acquisition programs on
                                        an annual basis. The programs we selected for review represent a variety
                                        of platforms, including sea vessels, manned and unmanned aircraft, and
                                        land systems. Specifically, we examined 11 programs at various stages of
                                        the acquisition process. Four programs had not yet passed Milestone B,
                                        development start, at the time we began our review. The remaining seven
                                        programs had completed their Milestone B review and were in
                                        development at the time of our case study selection. Of the seven
                                        programs, three have breached Nunn-McCurdy cost thresholds since the
                                        act was passed and have had to satisfy the act’s new requirements with
                                        regards to certification. The other programs had significant interaction
                                        with one or more of the OSD offices established by the Reform Act. A
                                        complete list of programs is provided below.

Table 4: List of Programs Selected for Case Study Review

Programs before Milestone B                                       Service                                          Platform
Ground Combat Vehicle                                             Army                                             Ground
                               a
Joint Light Tactical Vehicle                                      Army/ Marines                                    Ground
Ohio Class Replacement                                            Navy                                             Sea
                         a
Ship to Shore Connector                                           Navy                                             Sea
Programs after Milestone B
Global Hawk                                                       Air Force                                        Air
Gray Eagle                                                        Army                                             Air
Joint Strike Fighter                                              Joint                                            Air
KC-46 Tanker                                                      Air Force                                        Air
Littoral Combat Ship Seaframe                                     Navy                                             Sea
Remote Minehunting System                                         Navy                                             Sea
Small Diameter Bomb Increment II                                  Air Force                                        Air
                                        Source: GAO analysis.
                                        a
                                         During the course of our review, the Joint Light Tactical Vehicle and Ship to Shore Connector
                                        programs held a Milestone B.




                                        Page 34                                                 GAO-13-103 Weapons Acquisition Reform
Appendix I: Objectives, Scope, and
Methodology




While our sample of 11 case studies allowed us to learn about the impact
the Reform Act offices have had on DOD acquisitions, it was designed to
provide anecdotal information, not findings that would be representative
of all the department’s weapon acquisition programs.

To determine challenges remaining in improving defense acquisitions we
relied on information we collected and analyzed during our case study
review of 11 weapon acquisition programs. We also solicited the opinions
of the Under Secretary of Defense for Acquisition, Technology and
Logistics; other senior level officials in the Office of the Secretary of
Defense including the leaders of each of the offices created as a result of
the Reform Act; as well as the military services’ Senior Acquisition
Executives.

We conducted this performance audit from January 2012 to December
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 35                                  GAO-13-103 Weapons Acquisition Reform
Appendix II: Progress of Reform Act Offices
                                         Appendix II: Progress of Reform Act Offices in
                                         Implementing Weapon System Acquisition
                                         Reform Act Provisions


in Implementing Weapon System Acquisition
Reform Act Provisions
Table 5: Implementation of Select Reform Act Provisions- Systems Engineering

Reform Act Provisions                        Status
Develop policy and guidance                  Complete: Issued a reliability, availability, and maintainability Directive-Type
                                             Memorandum, a development planning Directive-Type Memorandum, and a DOD
                                             Instruction for DASD (Systems Engineering), participated in JCIDS revisions, developed
                                             guidance for incorporating systems engineering into development contracts, streamlined
                                             the Systems Engineering Plan and Program Protection Plan, and released an update to
                                                                                               a
                                             the Defense Acquisition Guidebook, chapter 4. Continue to refine policies and
                                             guidance as necessary.
Approve documents                            Completing on annual basis: Approved 52 Systems Engineering Plans since 2009,
                                             including 15 in fiscal year 2011.
Monitor programs / conduct assessments       Completing on annual basis: Review portfolio of 234 programs. In fiscal year 2011,
                                             participated in 73 overarching integrated product team meetings, 6 peer reviews of
                                             acquisition contracts, and conducted 15 Program Support Reviews.
Develop performance measures                 Complete: Developed a set of time-based metrics to assess each program’s ability to
                                             execute its system engineering plans and address risks the office had identified in prior
                                             reviews. The metrics measure program cost, schedule, staffing, reliability, availability
                                             and maintainability, software, integration, performance and manufacturing, and are to be
                                             incorporated into each program’s systems engineering plan and evaluated at various
                                             points in the development process.
                                         Source: GAO analysis of DOD data.
                                         a
                                          DOD Directive-Type Memorandum (DTM) 11-003,”Reliability Analysis, Planning, Tracking, and Reporting,” (Mar. 21, 2011); DOD
                                         DTM 10-017, “Development Planning to Inform Materiel Development Decision Reviews and Support of Analysis of Alternatives,”
                                         (Sept. 13, 2010, Incorporating Change 2, Dec. 9, 2011); DOD Instruction 5134.16, “Deputy Assistant Secretary of Defense for Systems
                                         Engineering (DASD(SE)),” (Aug. 19, 2011).


Table 6: Implementation of Select Reform Act Provisions- Developmental Test and Evaluation

Reform Act Provisions                    Status
Develop policy and guidance              Complete: Updated guide for incorporating test and evaluation into acquisition contracts.
                                         Championed updates to DOD Instruction assigning responsibilities and authorities to
                                         Developmental Test and Evaluation office, which is in the process of being updated.
                                         Updated guidance to include reliability factors in the Test and Evaluation Master Plan.
                                         Continue to refine policies and guidance as necessary.
Approve documents                        Completing on annual basis: Reviewed and approved 186 Test and Evaluation Master
                                         Plans since 2009, including 44 in fiscal year 2011.
Monitor programs / conduct assessments   Completing on annual basis: Review portfolio of nearly 250 programs. In fiscal year 2011,
                                         participated in 22 defense acquisition board meetings and 59 overarching integrated
                                         product team meetings. The office has also conducted 16 Assessment of Operational
                                         Test Readiness reviews since 2009.
Develop performance measures             In process: Piloted performance measures on two programs. The measures were then
                                         updated and are being applied to over 40 programs that were selected for reporting in the
                                         fiscal year 2012 joint annual report. The assessments are being used to support the write-
                                         up of the program engagement section.
                                         Source: GAO analysis of DOD data.




                                         Page 36                                                                 GAO-13-103 Weapons Acquisition Reform
                                         Appendix II: Progress of Reform Act Offices in
                                         Implementing Weapon System Acquisition
                                         Reform Act Provisions




Table 7: Implementation of Select Reform Act Provisions- Cost Assessment and Program Evaluation

Reform Act Provisions                    Status
Develop policy and guidance              In process: Issued first policy document in May 2012, which is the basis for additional
                                         policy documents. Updating its Operating and Support Cost Estimating Guidebook, which
                                         will address the Reform Act requirement for DOD to issue guidance related to full
                                         consideration of life cycle management and sustainability costs in major defense
                                         acquisition programs.
Approve documents                        Not applicable: The Reform Act does not require that the office approve acquisition
                                         documents.
Monitor programs / conduct assessments   Completing on annual basis: Conducted independent cost assessments for Milestone A
                                         and B certification on 30 future and current major defense acquisition programs, since
                                         2009, including 8 in fiscal year 2011. The office conducted 3 Milestone C and 3 Nunn-
                                         McCurdy certification reviews in fiscal year 2011.
Develop performance measures             Not applicable: The Reform Act does not require that the office develop performance
                                         measures.
                                         Source: GAO analysis of DOD data.



Table 8: Implementation of Select Reform Act Provisions- Performance Assessments and Root Cause Analyses

Reform Act Provisions                    Status
Develop policy and guidance              In process: Developing guidance to assist offices in conducting root cause analyses. The
                                         guidance for conducting performance assessments is expected to be released in early
                                         fiscal year 2013.
Approve documents                        Not applicable: The Reform Act does not require that the office approve acquisition
                                         documents.
Monitor programs / conduct assessments   Completing on annual basis: Completed 14 Root Cause Analyses for programs which
                                         have undergone a Nunn-McCurdy breach or were requested by OSD and has completed
                                         26 semi-annual follow-up reports on these programs. Providing OSD with the execution
                                         status of DOD’s portfolio of acquisition programs through the Defense Acquisition
                                         Executive Summary process.
Develop performance measures             In process: Utilize Defense Acquisition Executive Summary information to identify cost
                                         performance, schedule, funding, and technical performance issues on Major Defense
                                         Acquisition Programs. Continuing to develop performance measures.
                                         Source: GAO analysis of DOD data.




                                         Page 37                                           GAO-13-103 Weapons Acquisition Reform
Appendix III: Comments from the
              Appendix III: Comments from the Department
              of Defense



Department of Defense




              Page 38                                      GAO-13-103 Weapons Acquisition Reform
Appendix III: Comments from the Department
of Defense




Page 39                                      GAO-13-103 Weapons Acquisition Reform
Appendix III: Comments from the Department
of Defense




Page 40                                      GAO-13-103 Weapons Acquisition Reform
           Appendix III: Comments from the Department
           of Defense




(121024)
           Page 41                                      GAO-13-103 Weapons Acquisition Reform
                      The Government Accountability Office, the audit, evaluation, and
GAO’s Mission         investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
                      Website: http://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in   E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.