oversight

Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed

Published by the Government Accountability Office on 2003-11-10.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to the Senate and House
                Committees on Armed Services



November 2003
                DEFENSE
                ACQUISITIONS
                DOD’s Revised Policy
                Emphasizes Best
                Practices, but More
                Controls Are Needed




GAO-04-53

                                                November 2003


                                                DEFENSE ACQUISITIONS

                                                DOD’s Revised Policy Emphasizes Best
Highlights of GAO-04-53, a report to the        Practices, but More Controls Are Needed
Senate and House Committees on Armed
Services




The Department of Defense’s                     DOD’s new policy supports knowledge-based, evolutionary acquisitions by
(DOD) investment in new weapon                  adopting lessons learned from successful commercial companies. One of
systems is expected to exceed                   those lessons is a knowledge-based approach, which requires program
$1 trillion from fiscal years 2003 to           managers to attain the right knowledge at critical junctures—also known
2009. To reduce the risk of cost and            as knowledge points—so they can make informed investment decisions
schedule overruns, DOD revamped
its acquisition policy in May 2003.
                                                throughout the acquisition process. The policy also embraces an
The policy provides detailed                    evolutionary or phased development approach, which sets up a more
guidance on how weapon systems                  manageable environment for attaining knowledge. The customer may not
acquisitions should be managed.                 get the ultimate capability right away, but the product is available sooner
                                                and at a lower cost. Leading firms have used these approaches—which
The Senate report accompanying                  form the backbone of what GAO calls the best practices model—to
the National Defense Authorization              determine whether a project can be accomplished with the time and money
Act for Fiscal Year 2004 required               available and to reduce risks before moving a product to the next stage
GAO to determine whether DOD’s                  of development.
policy supports knowledge-based,
evolutionary acquisitions and                   By adopting best practices in the acquisition policy, DOD’s leadership has
whether the policy provides the
necessary controls for DOD to
                                                taken a significant step forward. The next step is to provide the necessary
ensure successful outcomes, such                controls to ensure a knowledge-based, evolutionary approach. Implementing
as meeting cost and schedule goals.             the necessary controls at all three knowledge points along the acquisition
                                                process helps decision makers ensure a knowledge-based approach is
The report also required GAO to                 followed. Without controls in the form of measurable criteria that decision
assess whether the policy is                    makers must consider, DOD runs the risk of making decisions based on
responsive to certain requirements              overly optimistic assumptions. Each successive knowledge point builds on
in the Bob Stump National Defense               the preceding one, and having clearly established controls helps decision
Authorization Act for Fiscal Year               makers gauge progress in meeting goals and ensuring successful outcomes.
2003 concerning DOD’s
management of the acquisition
                                                DOD Policy Incorporates Best Practices but Does Not Have Sufficient Controls
process.
                                                                                  Best practices’ intent in  Sufficient controls in
                                                 Knowledge point                  DOD policy?                DOD policy?
                                                 Knowledge point 1 (at program Yes                           No 

                                                 launch): Technologies, time, 

GAO recommends that the                          funding, and other resources 

Secretary of Defense strengthen                  match customer needs 

DOD’s acquisition policy by                      Knowledge point 2 

requiring additional controls to                 (between integration and         Yes                        No 

ensure decision makers will follow               demonstration): Design 

                                                 performs as expected

a knowledge-based, evolutionary
                                                 Knowledge point 3                Yes                        No 

approach. DOD partially concurred                (at production commitment):

with GAO’s recommendations.                      Production meets cost, schedule,
                                                 and quality targets
                                                Sources: DOD (data), GAO (analysis).


                                                DOD was responsive to the requirements in the Bob Stump National Defense
                                                Authorization Act for Fiscal Year 2003. DOD’s responses reflected the
www.gao.gov/cgi-bin/getrpt?GAO-04-53.
                                                committee’s specific concerns about the application of certain statutory and
To view the full product, including the scope   regulatory requirements to the new evolutionary acquisition process, for
and methodology, click on the link above.       more guidance for implementing spiral development, and about technology
For more information, contact Katherine V.
Schinasi at (202) 512-4841 or                   readiness at program initiations.
schinasik@gao.gov.
Contents 



Letter                                                                                            1
                       Results in Brief 
                                                         2
                       Background
                                                                3
                       DOD’s Revised Policy Provides a Framework for 

                         Knowledge-Based, Evolutionary Acquisitions                               5
                       Controls Are Needed to Ensure Knowledge-Based Approach
                         Is Followed                                                              8
                       DOD Is Responsive to Congressional Requirements about New
                         Acquisition Process                                                    10
                       Conclusions                                                              14
                       Recommendations for Executive Action                                     14
                       Agency Comments and Our Evaluation                                       15
                       Scope and Methodology                                                    16

Appendix I             Section 802, Evolutionary Acquisition                                    19



Appendix II            Section 803, Spiral Development                                          24



Appendix III 	         Section 804, Technology Maturity, and Section 822,
                       Independent Technology Readiness Assessments                             26



Appendix IV            Comments from the Department of Defense                                  28



Related GAO Products                                                                            31



Tables
                       Table 1: Excerpts of Best Practices Contained in DOD’s New Policy          6
                       Table 2: Comparison of Controls Used in Best Practices Model and
                                DOD Policy                                                        8
                       Table 3: How DOD Policy Addresses Section 802’s Statutory and
                                Regulatory Requirements                                         19




                       Page i                                    GAO-04-53 DOD’s Acquisition Policy
          Table 4: How DOD Responded to Section 802’s Requirements
                   Regarding Specific Matters for Each Increment of an
                   Evolutionary Acquisition Process                                                 21
          Table 5: How DOD Policy Addresses Section 803’s Requirements                              24
          Table 6: How DOD Responded to Section 804 and Section 822
                   Requirements                                                                     26


Figures
          Figure 1: Distribution of Development Funds for Fiscal Year 2003                          5
          Figure 2: Comparison of DOD’s Framework and Commercial Best
                   Practices Model                                                                  7




          Abbreviations

          DOD               Department of Defense 

          DODD              DOD Directive 

          DODI              DOD Instruction 

          MDA               Milestone Decision Authority 

          TEMP              Test and Evaluation Master Plan 





          This is a work of the U.S. government and is not subject to copyright protection in the
          United States. It may be reproduced and distributed in its entirety without further
          permission from GAO. However, because this work may contain copyrighted images or
          other material, permission from the copyright holder may be necessary if you wish to
          reproduce this material separately.




          Page ii                                              GAO-04-53 DOD’s Acquisition Policy
United States General Accounting Office
Washington, DC 20548




                                   November 10, 2003 


                                   The Honorable John W. Warner 

                                   Chairman 

                                   The Honorable Carl Levin 

                                   Ranking Minority Member 

                                   Committee on Armed Services 

                                   United States Senate 


                                   The Honorable Duncan Hunter 

                                   Chairman 

                                   The Honorable Ike Skelton 

                                   Ranking Minority Member 

                                   Committee on Armed Services 

                                   House of Representatives 


                                   The Department of Defense (DOD) plans to spend more than $1 trillion 

                                   from fiscal years 2003 to 2009 for developing and procuring weapon 

                                   systems. To get the most out of its investment, DOD has set goals to 

                                   develop and procure weapons faster and at less cost. Guiding this effort is 

                                   DOD’s newly revised acquisition policy. Issued in May 2003, the policy 

                                   seeks to foster greater efficiency while building flexibility in the 

                                   acquisition process. The policy embraces a knowledge-based, evolutionary

                                   framework that emphasizes shorter development times. 


                                   DOD intends to use its policy to improve its record for meeting cost and 

                                   schedule estimates and for delivering capabilities as promised. Congress 

                                   has expressed support for DOD’s efforts to revamp its acquisition process 

                                   and has established requirements that DOD must take to ensure a 

                                   disciplined approach. Recognizing that a consistent and disciplined 

                                   application of policies will be key to achieving desired outcomes, the 

                                   Senate Armed Services Committee directed us to assess DOD’s 

                                   current acquisition policies to determine whether they (1) support 

                                   knowledge-based, evolutionary acquisitions, (2) provide the necessary 

                                   controls to ensure DOD’s policy intent is followed, and (3) respond to 

                                   specific requirements in the Bob Stump National Defense Authorization 





                                   Page 1                                       GAO-04-53 DOD’s Acquisition Policy
                   Act for Fiscal Year 2003 (Public Law 107-314) regarding DOD plans to
                   manage its new acquisition process.1


                   DOD has made major improvements to its acquisition policy by adopting
Results in Brief   knowledge-based, evolutionary practices used by successful commercial
                   companies. If properly applied, these best practices can put DOD’s
                   decision makers in a better position to deliver high-quality products on
                   time and within budget. The policy requires decision makers to have the
                   knowledge they need before moving to the next phase of development.
                   To ensure that the acquisition environment is conducive to a knowledge-
                   based approach, the policy embraces evolutionary development, which
                   allows managers to develop a product in increments rather than trying to
                   incorporate all the desired capabilities in the first version that comes off
                   the production line. Leading companies who use the knowledge-based
                   evolutionary approach have shown that these best practices help reduce
                   risks at critical junctures during a product’s development and help ensure
                   that decision makers get the most out of their investments.

                   DOD’s recent policy changes are a significant step forward. The next step
                   is for DOD to provide the necessary controls to ensure a knowledge-based,
                   evolutionary approach is followed. For example, the policy does not
                   establish measures to gauge design and manufacturing knowledge at
                   critical junctures in the product development process. Without specific
                   requirements to demonstrate knowledge at key points, the policy allows
                   significant unknowns to be judged as acceptable risks, leaving an opening
                   for decision makers to make uninformed decisions about continuing
                   product development.

                   DOD was responsive to the requirements in the Defense Authorization Act
                   for Fiscal Year 2003. DOD’s responses reflected the committee’s specific
                   concerns about the application of certain statutory and regulatory
                   requirements2 to the new evolutionary acquisition process, for more
                   guidance for implementing spiral development, and about technology
                   readiness at program initiations.




                   1
                    S. Rept. No. 108-46, Report of the Senate Armed Services Committee accompanying S.
                   1050, National Defense Authorization Act for Fiscal Year 2004, at page 346 (May 13, 2003).
                   2
                    As used here, the term “regulatory requirements” refers to policies governing DOD’s
                   acquisition system.




                   Page 2                                                GAO-04-53 DOD’s Acquisition Policy
                  This report makes recommendations that the Secretary of Defense
                  strengthen DOD’s acquisition policy by requiring additional controls to
                  ensure decision makers will follow a knowledge-based, evolutionary
                  approach. DOD partially concurred with our recommendations. DOD
                  believes the current acquisition framework includes the controls
                  necessary to achieve effective results, but department officials will
                  continue to monitor the process to determine whether other controls are
                  needed to achieve the best possible outcomes. DOD agreed it should
                  record and justify program decisions for moving from one stage of
                  development to next but did not agree with the need to issue a report
                  outside of the department.


                  Traditionally, DOD’s strategy for acquiring major weapon systems has
Background        been to plan programs that would achieve a big leap forward in capability.
                  However, because the needed technologies often are not yet mature,
                  programs stay in development for years until the technologies are
                  demonstrated. As a result, weapon systems have frequently been
                  characterized by poor cost, schedule, and performance outcomes. This has
                  slowed modernization efforts, reduced the buying power of the defense
                  dollar, delayed capabilities for the warfighter, and forced unplanned—and
                  possibly unnecessary—trade-offs among programs.

                  Our extensive body of work shows that leading companies use a product
                  development model that helps reduce risks and increase knowledge when
                  developing new products. This best practices model enables decision
                  makers to be reasonably certain about their products at critical junctures
                  during development and helps them make informed investment decisions.
                  This knowledge-based process can be broken down into three cumulative
                  knowledge points.

             •	  Knowledge point 1: A match must be made between the customer’s
                 needs and the developer’s available resources—technology, engineering
                 knowledge, time, and funding—before a program starts.
             • 	 Knowledge point 2: The product’s design must be stable and must
                 meet performance requirements before initial manufacturing begins.
             • 	 Knowledge point 3: The product must be producible within cost,
                 schedule, and quality targets and demonstrated to be reliable before
                 production begins.

                  To bolster the knowledge-based process, leading companies use
                  evolutionary product development, an incremental approach that enables
                  developers to rely more on available resources rather than making



                  Page 3                                      GAO-04-53 DOD’s Acquisition Policy
promises about unproven technologies. While the user may not initially
receive the ultimate capability under this approach, the initial product is
available sooner and at a lower, more predictable cost. Also, leading
companies know that invention cannot be scheduled and its cost is
difficult to estimate. They do not bring technology into new product
development unless that technology has been demonstrated to meet the
user’s requirements. Allowing technology development to spill over into
product development puts an extra burden on decision makers and
provides a weak foundation for making product development estimates.

DOD understands that it must improve acquisition process outcomes if
it is to modernize its forces within currently projected resources. To
help achieve this goal, DOD has revised its acquisition policy, called the
5000 series, to reflect best practices from successful commercial and DOD
programs.3 The policy covers most—but not all—major acquisitions. The
Secretary of Defense has delegated authority to the Missile Defense
Agency and to the National Security Space Team to develop separate
guidance for missile defense and space systems, respectively.4
Approximately 35 percent of DOD’s development funds in 2003 went to
these systems. (Figure 1 shows how $43.1 billion in development funds
were distributed across space, missile defense, and systems covered by
the 5000 series.) This report addresses policy for the defense programs
covered exclusively under the 5000 series.




3
 DOD Directive 5000.1, The Defense Acquisition System, describes the management
principles for DOD’s acquisition programs. DOD Instruction 5000.2, The Operation of the
Defense Acquisition System, outlines a framework for managing acquisition programs.
Collectively, these are known as the 5000 series.
4
  Program elements of the ballistic missile defense program enter the formal DOD
acquisition cycle at milestone C (production commitment) and are subject to the 5000
series from that point on. To use the streamlined process of the National Security Space
Acquisition Policy, a Space System Program Director/Program Manager must request that
the DOD Space Milestone Decision Authority (the Under Secretary of the Air Force) grant a
waiver and an exemption to the processes and procedures described in DOD Instruction
5000.2 (the waiver authority does not include DOD Directive 5000.1). Notwithstanding
these policy exceptions, statutory requirements for major defense acquisition programs
continue to apply to missile defense and space programs.




Page 4                                              GAO-04-53 DOD’s Acquisition Policy
                    Figure 1: Distribution of Development Funds for Fiscal Year 2003




                    Note: Includes research, development, test, and evaluation budget activities for advanced component
                    development and prototypes; system development and demonstration; and operational systems
                    development.




                    DOD’s leaders have made significant improvements to DOD’s acquisition
DOD’s Revised       policy by adopting the knowledge-based, evolutionary approach used by
Policy Provides a   leading commercial companies. The revised policy has the potential to
                    transform DOD’s acquisition process by reducing risks and increasing the
Framework for       chances for successful outcomes. The policy provides a framework for
Knowledge-Based,    developers to ask themselves at key decision points whether they have
                    the knowledge they need to move to the next phase of acquisition. If
Evolutionary        rigorously applied, this knowledge-based framework can help managers
Acquisitions        gain the confidence they need to make significant and sound investment
                    decisions for major weapon systems. In placing greater emphasis on
                    evolutionary product development, the policy sets up a more manageable
                    environment for achieving knowledge. Another best practice reflected in
                    the policy’s framework is separating technology development from
                    product development, which reduces technological risk at the start of
                    a program.




                    Page 5                                                    GAO-04-53 DOD’s Acquisition Policy
                                          As shown in table 1, DOD’s policy emphasizes best practices used by
                                          leading companies.

Table 1: Excerpts of Best Practices Contained in DOD’s New Policy

 Best practices                           DOD policy
 Knowledge-based acquisition	             DOD Directive 5000.1 (enclosure 1, p. 5): Program managers “shall provide knowledge
                                          about key aspects of a system at key points in the acquisition process. … shall reduce
                                          technology risk, demonstrate technologies in a relevant environment … prior to program
                                          initiation. They shall reduce integration risk and demonstrate product design prior to the
                                          design readiness review. They shall reduce manufacturing risk and demonstrate
                                          producibility prior to full-rate production.”
 Evolutionary acquisition 	               DOD Instruction 5000.2 (p. 3): “Evolutionary acquisition is the preferred DOD strategy for
                                          rapid acquisition of mature technology for the user. … The objective is to balance needs
                                          and available capability with resources, and to put capability into the hands of the user
                                          quickly. The success of the strategy depends on consistent and continuous definition of
                                          requirements, and the maturation of technologies … that provide increasing capability
                                          towards a materiel concept.”
 Separating technology development from   DOD Instruction 5000.2 (p. 6): “The project shall exit Technology Development when an
 product development                      affordable increment of militarily-useful capability has been identified, the technology for
                                          that increment has been demonstrated in a relevant environment, and a system can be
                                          developed for production within a short timeframe (normally less than five years).”
Source: DOD.




                                          Similar to the best practices model, DOD’s policy divides its acquisition
                                          process into phases, as shown in figure 2. Key decisions are aligned with
                                          the three critical junctures of a product’s development, or knowledge
                                          points.




                                          Page 6                                                  GAO-04-53 DOD’s Acquisition Policy
Figure 2: Comparison of DOD’s Framework and Commercial Best Practices Model




                                      In other similarities, DOD’s framework pinpoints program start at
                                      milestone B, about the same point as program start on the best practices
                                      model. At the midway point on both approaches, a stable product design
                                      should be demonstrated. With DOD’s framework, managers are required to
                                      know—by the time full-rate production decision review occurs—whether
                                      the product can be produced within cost, schedule, and quality targets.
                                      This requirement occurs earlier in the best practices model, before
                                      production begins, or at knowledge point 3. Leading companies have
                                      used this approach to reduce risks and to make costs and delivery dates
                                      more predictable.




                                      Page 7                                    GAO-04-53 DOD’s Acquisition Policy
                                            While DOD has strengthened its acquisition policy with a knowledge-
Controls Are 
                              based, evolutionary framework, the policy does not include many of the
Needed to Ensure 
                          same controls that leading companies rely on to attain a high level of
                                            knowledge before making additional significant investments. Controls are
Knowledge-Based 
                           considered effective if they are backed by measurable criteria and if
Approach Is Followed 
                      decision makers are required to consider them before deciding to advance
                                            a program to the next level. Controls used by leading companies help
                                            decision makers gauge progress in meeting cost, schedule, and
                                            performance goals and ensure that managers will (1) conduct activities to
                                            capture relevant product development knowledge, (2) provide evidence
                                            that knowledge was captured, and (3) hold decision reviews to determine
                                            that appropriate knowledge was captured to move to the next phase.

                                            To determine if DOD has the necessary controls, we compared controls in
                                            DOD’s policy with those used in the best practices model at three critical
                                            junctures. Table 2 shows the presence or absence of controls for various
                                            versions of DOD policy since 1996, including the May 2003 revision.

Table 2: Comparison of Controls Used in Best Practices Model and DOD Policy

                                                                              Controls used in DOD policy
                                                                March 1996   October 2000      October 2002       May 2003
Controls used in commercial best practices model                  policy        policy         interim policy      policy
Knowledge point 1: Occurs at program launch. Match exists between requirements and resources. Technologies needed to meet
essential product requirements have been demonstrated to work in their intended environment and the producer has completed a
preliminary design of the product.
   Demonstrate technologies to high readiness levels                               X                 X                 X
   Ensure that requirements for the product are informed by 

   the systems engineering process                                                                                     X
   Establish cost and schedule estimates for product based
   on knowledge from preliminary design using systems
   engineering tools
   Conduct decision review for program launch                       X              X                 X                 X
Knowledge point 2: Occurs between integration and demonstration. Design is stable and has been demonstrated through prototype
testing. Ninety percent of engineering drawings are releasable to manufacturing organizations.
   Complete 90 percent of design drawings
   Complete subsystem and system design reviews
   Demonstrate with prototype that design meets 

   requirements                                                                    X                 X                 X
   Obtain stakeholders concurrence that drawings are
   complete and producible
   Complete failure modes and effects analysis
   Identify key system characteristics




                                            Page 8                                          GAO-04-53 DOD’s Acquisition Policy
                                                                                 Controls used in DOD policy
                                                                March 1996      October 2000      October 2002        May 2003
 Controls used in commercial best practices model                 policy           policy         interim policy       policy
    Identify critical manufacturing processes
    Establish reliability targets and growth plan based on
    demonstrated reliability rates of components and
    subsystems
    Conduct decision review to enter system demonstration                                               X                 X
 Knowledge point 3: Occurs at production commitment. Product is ready to be manufactured within cost, schedule, and quality
 targets. All key manufacturing processes have come under statistical control and product reliability has been demonstrated.
    Demonstrate manufacturing processes
    Build production representative prototypes
    Test production representative prototypes to achieve
    reliability goal
    Test production representative prototypes to demonstrate
    product in operational environment
    Collect statistical process control data
    Demonstrate critical processes capable and in
    statistical control
    Conduct decision review to begin production                      X                X                 X                 X
Sources: GAO and DOD.




                                                At all three knowledge points, DOD’s policy does not provide all the
                                                necessary controls used by commercial companies. For example, at
                                                program launch (milestone B) or when knowledge point 1 should be
                                                reached, the policy requires decision makers to identify and validate a
                                                weapon system’s key performance requirements and to have a technical
                                                solution for the system before program start. This information is then used
                                                to form cost and schedule estimates for the product’s development.
                                                However, the policy does not emphasize the use of a disciplined systems
                                                engineering process for balancing a customer’s needs with resources to
                                                deliver a preliminary design. The lack of effective controls at knowledge
                                                point 1 could result in gaps between requirements and resources being
                                                discovered later in development.

                                                At the design readiness review or when knowledge point 2 should be
                                                reached, DOD’s policy does not require specific controls to document that
                                                a product is ready for initial manufacturing and demonstration. DOD’s
                                                policy suggests appropriate criteria, such as number of subsystem and
                                                system design reviews completed, percentage of drawings completed,
                                                planned corrective actions to hardware and software deficiencies,
                                                adequate development testing, completed failure modes and effects


                                                Page 9                                         GAO-04-53 DOD’s Acquisition Policy
                     analysis, identification of key system characteristics and critical
                     manufacturing processes, and availability of reliability targets and growth
                     plans. However, these criteria are not required. For example, we found
                     that a key indicator of a product’s design stability is the completion of
                     90 percent of the engineering drawings supported by design reviews.
                     DOD’s policy does not require that a certain percentage of drawings or
                     design reviews be completed to ensure the design is mature enough to
                     enter the system demonstration phase. As a result, a decision maker has
                     no benchmark to consider when deciding to advance a program to the
                     next level of development.

                     Finally, at production commitment or when knowledge point 3 should be
                     reached, DOD’s policy does not require specific controls to document that
                     a product can be manufactured to meet cost, schedule, and quality targets
                     before moving into production. For example, the policy states there
                     should be “no significant manufacturing risks” at the start of low-rate
                     production but does not define what this means or how it is to be
                     measured. DOD’s policy does not require the demonstrated control of
                     manufacturing processes and the collection of statistical process control
                     data until full-rate production begins but even then fails to specify a
                     measurable control. Given that low-rate production can last several years,
                     a significant number of products can be manufactured before processes
                     are brought under control, creating a higher probability of poor cost and
                     schedule outcomes.


                     While supporting efforts to build more flexibility into the DOD acquisition
DOD Is Responsive    process and to develop weapon systems using an evolutionary approach,
to Congressional     Congress asked DOD to be more disciplined in its approach. The Defense
                     Authorization Act for Fiscal Year 2003 required DOD to address (1) the
Requirements about   way it plans to meet certain statutory and regulatory requirements for
New Acquisition      managing its major acquisition programs, (2) needed guidance for
                     implementing spiral developments, and (3) technology readiness (at
Process              acquisition program initiation). DOD was responsive to all three
                     requirements. With regard to the second requirement, a description of the
                     process that would be used to independently validate that measurable exit




                     Page 10                                     GAO-04-53 DOD’s Acquisition Policy
                            criteria for applying a spiral development process5 have been met was
                            unclear. DOD stated that the milestone decision authority6 provides that
                            independent validation as part of DOD’s milestone approval process.
                            DOD’s responses to the relevant sections of the act are summarized below.
                            More detailed comparisons are provided in appendixes I, II, and III.


Section 802: Evolutionary   Requirements: This section directed DOD to report on its plan to meet
Acquisition                 certain statutory and regulatory requirements for managing its major
                            acquisition programs applying an evolutionary acquisition process. These
                            include establishing and approving operational requirements and cost and
                            schedule goals for each increment, meeting requirements for operational
                            and live fire testing for each increment, and optimizing total system
                            performance and minimizing total ownership costs.

                            DOD response: In April 2003, DOD submitted its report reflecting how
                            these requirements are addressed in its acquisition policy. According to
                            the report, the policy addresses the statutory and other requirements
                            applicable to all major defense acquisition programs, including each
                            increment of evolutionary acquisition programs. For example, the policy
                            requires that each program or increment of an evolutionary acquisition
                            have a milestone B decision to approve program initiation and to permit
                            entry into system development and demonstration. The policy specifies
                            the statutory and regulatory information necessary to support the
                            decision.


Section 803: Spiral         Requirements: This section authorizes DOD to conduct a research and
Development                 development program for a major defense acquisition program using spiral
                            development only if approved by the Secretary of Defense or authorized




                            5
                             A spiral development program is defined in section 803 of the Defense Authorization Act
                            for Fiscal Year 2003 as a research and development program conducted in discrete phases
                            or blocks, each of which will result in the development of fieldable prototypes and will not
                            proceed into acquisition until specific performance parameters, including measurable exit
                            criteria, have been met.
                            6
                             The milestone decision authority is typically the DOD component acquisition executive or
                            designee, or for certain large programs, the head of the component or the Under Secretary
                            of Defense for Acquisition, Technology, and Logistics.




                            Page 11                                               GAO-04-53 DOD’s Acquisition Policy
high-level designee.7 A program cannot be conducted as a spiral
development unless the Secretary of Defense or designee approves a plan
that describes such things as the program strategy, test plans, performance
parameters, and measurable exit criteria. The section also requires the
Secretary of Defense to issue guidance addressing the appropriate
processes for an independent validation that exit criteria have been met,
the operational assessment of fieldable prototypes, and the management
of these types of programs. It further requires the Secretary to report to
Congress on the status of each program applying spiral development by
September 30 of each year from 2003 to 2008.

DOD response: DOD established a technology development strategy in
the new policy to address this requirement. The strategy must be
completed before a program can enter the technology development phase.
The strategy also documents the cost and schedule goals, the test plans,
the number of prototypes, and a program strategy for the total research
and development program. The strategy requires a test plan to ensure the
goals and exit criteria for the first technology spiral demonstration are
met, and the policy requires an independent operational assessment for
the release of each product increment to the user. What is unclear in
DOD’s guidance is the process that will be used for independently
validating whether measurable cost, schedule, and performance exit
criteria have been met. However, DOD stated that the milestone decision
authority provides independent validation that exit criteria have been met
as part of DOD’s milestone approval process. As of October 23, 2003,
DOD’s report on the status of each program applying spiral development
was still in draft and not yet submitted. DOD’s current draft report states
that there are no research and development programs that have been
approved as spiral development programs as of September 30, 2003.
Section 803 requirements were implemented in DOD Instruction 5000.2,
which was effective in May 2003. DOD anticipates that there will be
approved spiral development programs to report in 2004.




7
 Section 803 authorized the Secretary of Defense to delegate authority to approve a spiral
development plan to the Under Secretary of Defense for Acquisition, Technology, and
Logistics or to the senior acquisition executive of the military department or defense
agency concerned. The authority may not be delegated further.




Page 12                                               GAO-04-53 DOD’s Acquisition Policy
Section 822: Independent   Requirements: This section added a requirement to section 804
Technology Readiness       of the National Defense Authorization Act for Fiscal Year 2002
Assessments                (Public Law 107-107) that directed DOD to report by March of each year
                           between 2003 and 2006 on the maturity of technology at the initiation of
                           major defense acquisition programs. Each report is required to (1) identify
                           any major acquisition program that entered system development and
                           demonstration during the preceding calendar year with immature key
                           technology that was not demonstrated in, at minimum, a relevant
                           environment, as required by the new policy; (2) justify the incorporation
                           of any key technology on an acquisition program that does not meet that
                           requirement; (3) identify any instances that the Deputy Under Secretary
                           of Defense for Science and Technology did not concur with the technology
                           assessment and explain how the issue has been or will be resolved;
                           (4) identify each case in which a decision was made not to conduct an
                           independent technology readiness assessment for a critical technology on
                           a major defense acquisition; and (5) explain the reasons for the decision
                           each year through 2006.

                           DOD response: In March 2003, DOD reported that two programs entered
                           system development and demonstration in 2002 with critical technologies
                           that did not meet demonstration requirements and provided justification
                           for them.8 DOD did not identify or report any cases where an independent
                           technology readiness assessment was not conducted or where the
                           Under Secretary disagreed with assessment findings.




                           8
                            The two programs were the Composite Health Care System II and the Joint Tactical Radio
                           System, Cluster 1.




                           Page 13                                            GAO-04-53 DOD’s Acquisition Policy
                       DOD can maximize its $1 trillion investment in new weapons over the next
Conclusions            6 years by ensuring effective implementation of the new acquisition policy.
                       DOD’s leaders have taken noteworthy steps by incorporating into the
                       policy a framework that supports a knowledge-based, evolutionary
                       acquisition process, similar to one used by leading commercial companies
                       to get successful outcomes. A framework is an important and significant
                       step. DOD must now turn its attention to establishing controls. As leading
                       companies have found, having clearly established controls to capture and
                       use appropriate knowledge to make decisions at critical junctures is
                       crucial for delivering affordable products as planned. DOD’s policy
                       addresses specific congressional requirements and includes some controls
                       that leading companies use to capture knowledge at the start of a program.
                       However, additional controls are needed to ensure that decisions made
                       throughout product development are informed by demonstrated
                       knowledge.


                       DOD must design and implement necessary controls to ensure that
Recommendations for    appropriate knowledge is captured and used at critical junctures to make
Executive Action       decisions about moving a program forward and investing more money.
                       We recommend that the Secretary of Defense require additional controls
                       for capturing knowledge at three key points—program launch, design
                       readiness review for transitioning from system integration to system
                       demonstration, and production commitment. The additional controls for
                       program launch (milestone B) should ensure the capture of knowledge
                       about the following:

                  •	   Cost and schedule estimates based on knowledge from a preliminary
                       design using systems engineering tools.

                       The additional controls for transitioning from system integration to system
                       demonstration (design readiness review) should ensure the capture of
                       knowledge about the following:

                  •   Completion of 90 percent of engineering drawings.
                  •   Completion of subsystem and system design reviews.
                  •	  Agreement from all stakeholders that drawings are complete and the
                      design is producible.
                  • Completion of failure modes and effects analysis.
                  • Identification of key system characteristics.
                  • Identification of critical manufacturing processes.
                  • 	 Reliability targets and a reliability growth plan based on demonstrated
                      reliability rates of components and subsystems.



                       Page 14                                     GAO-04-53 DOD’s Acquisition Policy
                            The additional controls for the production commitment (milestone C)
                            should ensure the capture of knowledge about the following:

                       •   Completion of production representative prototypes.
                       •	  Availability of production representative prototypes to achieve reliability
                           goal and demonstrate the product in an operational environment.
                       • Collection of statistical process control data.
                       • 	 Demonstration that critical manufacturing processes are capable and in
                           statistical control.

                            Because knowledge about technology, design, and manufacturing at
                            critical junctures can lower DOD’s investment risk, decisions that do not
                            satisfy knowledge-based criteria should be visible and justified. Therefore,
                            we also recommend that the Secretary of Defense document the rationale
                            for any decision to move a program to the next stage of development
                            without meeting the knowledge-based criteria, including those listed in the
                            first recommendation. The responsible milestone decision authority
                            should justify the decision in the program’s acquisition decision
                            memorandum and in a report to Congress.


                            DOD provided us with written comments on a draft of this report. The
Agency Comments 
           comments appear in appendix IV.
and Our Evaluation 

                            DOD partially concurred with our recommendation that the Secretary
                            require additional controls for capturing knowledge at three key points:
                            program launch, design readiness review for transitioning from system
                            integration to system demonstration, and production. DOD stated that it
                            agrees in principle with the advantages of using knowledge-based controls
                            at key points in the acquisition process to assess risk and ensure readiness
                            to proceed into the next phase of the acquisition process. DOD believes
                            the current acquisition framework includes the controls necessary to
                            achieve effective results, but it will continue to monitor the process to
                            determine whether others are necessary to achieve the best possible
                            outcomes.

                            While we believe DOD’s effort to establish a solid framework for
                            evolutionary acquisitions is a giant step forward, our work has shown that
                            a disciplined application of controls in the process is needed to implement
                            the framework if better acquisition outcomes are to be achieved. DOD’s
                            policy does not include all the necessary controls to ensure a high level of
                            product knowledge is attained and used for making decisions to move a



                            Page 15                                      GAO-04-53 DOD’s Acquisition Policy
              program forward in the product development process. Leading product
              developers use additional controls, as listed in our first recommendation,
              to achieve the knowledge necessary to reduce risk to reasonable levels at
              critical junctures before making additional significant investments in
              product development. Simply monitoring the process may not be enough
              for DOD to achieve the best outcomes. Therefore, we are retaining our
              recommendation that the Secretary require additional controls at three
              critical points in the acquisition process.

              DOD also partially concurred with our recommendation that the Secretary
              document in each program’s acquisition decision memorandum and in a
              report to Congress the rationale for any decision to move a program to the
              next stage of development without meeting the knowledge-based criteria,
              including those described in the first recommendation. DOD agreed that it
              should record and be accountable for program decisions. Decision makers
              will continue to use the acquisition decision memorandum to document
              program decisions and the rationale for them. DOD did not concur with
              the need for a report outside the department. Because we believe strongly
              that knowledge-based criteria used to gauge a product’s development
              progress at critical junctures can lower DOD’s investment risks, we think
              it is important that decisions made without satisfying knowledge-based
              criteria be justified in a visible and transparent way to hold managers
              accountable for moving a program forward absent this knowledge.
              Therefore, we are retaining our recommendation for reporting the basis
              for decisions to move forward in a report to Congress.


              We reviewed DOD’s revised and past acquisition policies, DOD Directive
Scope and     5000.1, DOD Instruction 5000.2, and DOD 5000.2-R,9 which provide
Methodology   management principles and mandatory policies and procedures for
              managing acquisitions programs. We contacted an official in the Office of
              the Under Secretary of Defense for Acquisition, Technology, and Logistics
              who is responsible for the development of the policy to better understand
              its content. We also reviewed information from the Defense Acquisition
              University that provided educational material on the policies.

              We reviewed the relevant sections of the Bob Stump National Defense
              Authorization Act for Fiscal Year 2003 and the accompanying Senate



              9
               DOD 5000.2-R was canceled and replaced with the Interim Defense Acquisition
              Guidebook.




              Page 16                                            GAO-04-53 DOD’s Acquisition Policy
Armed Services Committee report to identify the requirements applicable
to DOD’s acquisition policy. We compared these requirements with DOD’s
responses to determine whether they have been addressed.

Finally, we used information from more than 10 GAO products that
examine how commercial best practices can improve outcomes for
various DOD programs. During the past 6 years, we have gathered
information based on discussions and visits with the following companies:

•     3M                         •   Boeing Commercial Airplane Group
•     Chrysler                   •   Bombardier Aerospace
•     Ford Motor                 •   Hughes Space and Communication
•     Motorola                   •   Xerox
•     Hewlett-Packard            •   Caterpillar
•     Cummins                    •   General Electric Aircraft Engines
•     Toyota                     •   Harris Semiconductor
•     Honda                      •   Texas Instruments
•     John Deere                 •   Varian Oncology Systems
•	    Ethicon-Endo Surgery
     (division of Johnson &
     Johnson)


Although the approaches varied, these companies consistently applied the
basic processes and standards in use. We compared this information with
the acquisition framework and controls established by DOD’s policy. We
concentrated on whether the policy provides a framework for a
knowledge-based, evolutionary process and the controls necessary to
carry out this intent.

We conducted our review from April 2003 to September 2003 in
accordance with generally accepted government auditing standards.

We are sending copies of this report to the Secretary of Defense; the
Secretaries of the Air Force, Army, and Navy; and the Director of the
Office of Management and Budget. We will also provide copies to others
on request. In addition, the report will be available at no charge on the
GAO Web site at http://www.gao.gov.




Page 17                                     GAO-04-53 DOD’s Acquisition Policy
Please contact me at (202) 512-4841 if you have any questions

concerning this report. Other key contributors to this report were 

Lily Chin, Chris DePerro, Matt Lea, Mike Sullivan, and Adam Vodraska. 





Katherine V. Schinasi 

Director, Acquisition and Sourcing Management 





Page 18                                    GAO-04-53 DOD’s Acquisition Policy
Appendix I: Section 802, Evolutionary
Acquisition

                                             Section 802 of the Defense Authorization Act for Fiscal Year 2003 required
                                             the Secretary of Defense to submit a report to Congress explaining how
                                             the Department of Defense (DOD) plans to meet certain statutory and
                                             regulatory requirements for acquisition programs following an
                                             evolutionary approach. In April 2003, the Secretary reported how these
                                             requirements were addressed in DOD’s policy (such as in tables of
                                             statutory and regulatory information requirements contained in enclosure
                                             3 of Instruction 5000.2). According to the report, DOD’s policy requires
                                             that each program—including an increment of an evolutionary
                                             acquisition—have a milestone B decision to approve program initiation
                                             and to permit entry into systems development and demonstration. DOD’s
                                             policy specifies the statutory and regulatory information necessary to
                                             support the decision. We examined the policy to ensure the statutes and
                                             regulations identified in section 802 were addressed. Table 3 provides a
                                             list of the statutory and regulatory requirements identified in section 802, a
                                             corresponding document and page number where the requirement appears
                                             in DOD’s policy, and a description of the requirement from the policy.

Table 3: How DOD Policy Addresses Section 802’s Statutory and Regulatory Requirements

Section 802 requirements                   DOD policy reference examples     Brief description from policy
Requirements of chapter 144 of title 10,
United States Code
   Sec. 2430, major defense acquisition Department of Defense Instruction    Table E.2.T1 provides a description (criteria) and the
   program defined                      (DODI) 5000.2, enclosure 2, p. 16    decision authority by acquisition program category.
                                                            a
   Sec. 2431, weapons development          Not referenced
   and procurement schedules
   Sec. 2432, Selected Acquisition         DODI 5000.2, enclosure 3, p.19    Required at milestone B or program initiation and
   Reports                                                                   annually thereafter; end of quarter following
                                                                             milestone C decision; full-rate production decision.
                                                                             Also required when there is a baseline breach.
   Sec. 2433, unit cost reports            DODI 5000.2, enclosure 3, p. 19   Required on quarterly basis.
   Sec. 2434, independent cost             DODI 5000.2, enclosure 6, p. 30   Required at program initiation for ships (cost
   estimates; operational manpower         DODI 5000.2, enclosure 3, p. 19   assessment only). Required at milestones B and C
   requirements                                                              and full-rate production decision.
   Sec. 2435, baseline description         DODI 5000.2, enclosure 3, p. 19   Required at program initiation for ships. Required at
                                           DODI 5000.2, p. 8                 milestones B and C and full-rate production decision.
                                                                             Program deviation report required immediately upon
                                                                             program deviation.
   Sec 2440, technology and industrial     DODI 5000.2, enclosure 3, p. 19   Required at milestones B and C (part of acquisition
   base plans                                                                strategy).




                                             Page 19                                          GAO-04-53 DOD’s Acquisition Policy
                                                                Appendix I: Section 802, Evolutionary
                                                                Acquisition




 Section 802 requirements                                   DOD policy reference examples                Brief description from policy
     Sec. 139, Director of Operational                      Department of Defense Directive              Cites Director of Operational Test and Evaluation as
     Test and Evaluation                                    (DODD) 5000.1, p. 3 
                        key official of defense acquisition system. Director
                                                            DODI 5000.2, enclosure 3 
                   responsible for assessing adequacy of operational
                                                            DODI 5000.2, enclosure 5, pp. 27-29
         tests and live fire tests and evaluating operational
                                                                                                         effectiveness, suitability, and survivability of systems.
     Sec. 181, Joint Requirements                           DODD 5000.1, p. 3                            Chairman of the Joint Chiefs of Staff, with assistance
     Oversight Council                                      DODI 5000.2, p. 4 	                          of Joint Requirements Oversight Council, responsible
                                                                                                         for assessing and providing advice regarding
                                                                                                         capability needs for defense acquisition programs.
                                                                                                         Chairman also responsible for validating and
                                                                                                         approving capabilities documents.
     Sec. 2366, major systems and                           DODI 5000.2, enclosure 3, p. 19              Live fire testing and reporting required for all covered
     munitions programs: survivability                      DODI 5000.2, enclosure 5, p. 29              systems. Strategy or live fire waiver and alternate
     testing and lethality testing required                 DODI 5000.2, p. 10                           plan required at milestone B. Live fire test and
     before full-scale production                                                                        evaluation report required for full-rate production
                                                                                                         decision.
     Sec. 2399, operational test and                        DODI 5000.2, enclosure 3, p. 18     Director of Operational Test and Evaluation shall
     evaluation of defense acquisition                      DODI 5000.2, pp. 9-10               determine operational effectiveness and suitability of
     programs                                               DODI 5000.2, enclosure 5, pp. 27-28 system under realistic conditions. Beyond Low Rate
                                                                                                Initial Production Report required at full-rate
                                                                                                production decision. DOD may not conduct
                                                                                                operational testing until Director of Operational Test
                                                                                                and Evaluation approves test plan.
     Sec. 2400, low-rate initial production                 DODI 5000.2, enclosure 3, p. 19              Low-rate initial production quantities will be
     of new systems                                         DODI 5000.2, pp. 9-10                        determined by milestone B.
 DODD 5000.1                                                DODD 5000.1, pp. 1, 2                        Policies in directive apply to all acquisition programs.
                                                                                                         Evolutionary acquisition strategies are preferred
                                                                                                         approach to satisfying operational needs.
 DODI 5000.2                                                DODI 5000.2, p. 1                            Instruction applies to all defense technology projects
                                                                                                         and acquisition projects.
 Chairman of the Joint Chiefs of Staff                      DODI 5000.2, p. 4                            Chairman of Joint Chiefs of Staff provides advice
 Instruction 3170.01Bb                                                                                   through Instruction 3170.01. Capability documents
                                                                                                         required at concept decision and at
                                                                                                         milestones A, B, and C.
 Other provisions of law and regulations                    DODI 5000.2, enclosure 3, pp. 18-22 Several other statutory, regulatory, and contract
 (including successor documents) that                                                           requirements are addressed in the policy.
 are applicable to such programs
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.
                                                                a
                                                                 Although the statutory requirement of 10 U.S.C., section 2431 is not specifically cited in the policy,
                                                                the same summarized information is submitted to Congress in the Selected Acquisition Reports for
                                                                the first quarter of a fiscal year as required by 10 U.S.C., section 2432. The requirement for Selected
                                                                Acquisition Reports is addressed by DOD’s acquisition policy. The statute concerning weapons
                                                                development and procurement schedules requires the Secretary of Defense to submit budget
                                                                justification documents regarding this information for each weapon system for which fund
                                                                authorization is required (and for which procurement funds are requested in the budget) not later than
                                                                45 days after the President submits the budget to Congress.
                                                                b
                                                                 Chairman of the Joint Chiefs of Staff Instruction 3170.01B was revised and reissued June 24, 2003,
                                                                as Joint Chiefs of Staff Instruction 3170.01C with an accompanying manual.




                                                                Page 20                                                     GAO-04-53 DOD’s Acquisition Policy
                                           Appendix I: Section 802, Evolutionary
                                           Acquisition




                                           Section 802 also required DOD to report on its plans for addressing certain
                                           acquisition process issues regarding each increment of an evolutionary
                                           process. DOD reported on how it plans to establish and approve
                                           operational requirements and cost and schedule goals; meet requirements
                                           for operational and live fire testing; monitor cost and schedule
                                           performance; achieve interoperability; and consider total system
                                           performance and total ownership costs. We compared DOD’s response
                                           with section 802’s reporting requirements. As shown in table 4, DOD was
                                           responsive to the 802 requirements.

Table 4: How DOD Responded to Section 802’s Requirements Regarding Specific Matters for Each Increment of an
Evolutionary Acquisition Process

Section 802 requirements                    DOD response                                  Policy reference
“The manner in which the Secretary plans
to establish and approve, for each
increment of an evolutionary acquisition
process–”
   “operational requirements; and” 	        “Operational Requirements: Each program       Capability development document,
                                            is required to have documented, approved      including key performance parameters,
                                            operational requirements in accordance        required at milestone B or program
                                            with authorized Joint Staff procedures. For   initiation (DODI 5000.2, p. 7, and enclosure
                                            evolutionary acquisition programs, the        3, p. 20).
                                            requirements documents are typically
                                            time-phased and specify the capability
                                            expected of each increment.”
   “cost and schedule goals.” 	             “Cost and Schedule Goals: At program          Acquisition program baseline required at
                                            initiation, each program and program          milestone B or program initiation (DODI
                                            increment is required to have an              5000.2, p. 8, and enclosure 3, p. 19).
                                            Acquisition Program Baseline approved by
                                            the Milestone Decision Authority (MDA).
                                            The Acquisition Program Baseline includes
                                            cost and schedule goals.”




                                           Page 21                                              GAO-04-53 DOD’s Acquisition Policy
                                              Appendix I: Section 802, Evolutionary
                                              Acquisition




Section 802 requirements                       DOD response                                 Policy reference
“The manner in which the Secretary plans,
for each increment of an evolutionary
acquisition process–”
  “to meet requirements for operational        “At program initiation, each program or      Test and Evaluation Master Plan required
  testing and live fire testing;”              program increment must have a Test and       at milestone B or program initiation (DODI
                                               Evaluation Master Plan (TEMP) approved       5000.2, enclosure 3, p. 21, and enclosure
                                               by the Director, Operational Test and        5, p. 25).
                                               Evaluation. The TEMP includes
                                               requirements for operational and live fire
                                               testing.”
  “to monitor cost and schedule                “Progress against cost and schedule goals    DOD’s policy contains an enclosure on
  performance; and”                            [for each increment] is addressed via        resource estimation (DODI 5000.2,
                                               automated reporting systems at both the      enclosure 6, pp. 30-31). Also, several
                                               Office of Secretary of Defense staff level   reporting requirements address monitoring
                                               and at the Service staff level, and by       cost and schedule performance such as
                                               reviews conducted in the context of the      acquisition program baselines and selected
                                               acquisition oversight model.”                acquisition reports (DODI 5000.2,
                                                                                            enclosure 3, pp. 18-21).
  “to comply with laws requiring reports to    “DoD acquisition policy requires             The service shall perform an independent
  Congress on results testing and on cost      independent assessment by the operational    operational assessment prior to release of
  and schedule performance.”                   test authority prior to release of each      each successive increment to the user
                                               successive increment to the user. All        (DODI 5000.2, enclosure 5, p. 29). Also,
                                               reports by statute will be submitted.”       several statutory reports are listed in the
                                                                                            policy addressing testing and cost and
                                                                                            schedule performance (DODI 5000.2,
                                                                                            enclosure 3, pp. 18-21).




                                              Page 22                                             GAO-04-53 DOD’s Acquisition Policy
                                                                Appendix I: Section 802, Evolutionary
                                                                Acquisition




 Section 802 requirements                                        DOD response                                 Policy reference
 “The manner in which the Secretary plans
 to ensure that each increment of an
 evolutionary acquisition process is
 designed–”
     “to achieve interoperability within and                     “Each increment of an evolutionary          Acquisition program baseline required at
     among United States forces and                              acquisition program is required to have an milestone B or program initiation (DODI
     United States coalition partners; and”                      Acquisition Program Baseline, approved by 5000.2, p. 8, and enclosure 3, p. 19).
                                                                 the MDA, which includes an Interoperability
                                                                 key performance parameter.”
     “to optimize total system performance                       “Each increment of an evolutionary           Acquisition strategy required at milestone B
     and minimize total ownership costs by                       acquisition program is required to have an or program initiation (DODI 5000.2, p. 7,
     giving appropriate consideration to–”                       acquisition strategy, approved by the MDA, and enclosure 3, p. 20).
     • “logistics planning;”                                     that addresses [where applicable]: logistics
                                                                 planning; manpower, personnel and
     • “manpower, personnel, and training;”
                                                                 training; human, environmental, safety,
     • 	 “human, environmental, safety,                          occupational health; accessibility (human),
         occupational health, accessibility,                     survivability, operational continuity (as
         survivability, operational continuity                   required by requirements document),
         and security factors;”                                  security factors; critical program
     • 	 “protection of critical program                         information; and spectrum management.
         information; and”                                       These factors are taken into account as the
                                                                 Department of Defense considers total
     • “spectrum management”
                                                                 system performance and total ownership
                                                                 costs.”
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.




                                                                Page 23                                              GAO-04-53 DOD’s Acquisition Policy
Appendix II: Section 803, Spiral Development 



                                                Section 803 of the Defense Authorization Act for Fiscal Year 2003
                                                authorized the Secretary of Defense to conduct major defense acquisition
                                                programs as spiral development programs. However, the section placed a
                                                limitation on these programs. It stated that a research and development
                                                program for a major acquisition may not be conducted as a spiral
                                                development program unless the Secretary of Defense or authorized
                                                high-level designee gives approval. The section requires the Secretary of
                                                Defense to issue guidance for the implementation of such programs to
                                                address appropriate processes for ensuring the independent validation of
                                                exit criteria being met, the operational assessment of fieldable prototypes,
                                                and the management of these types of programs.

                                                DOD responded to these requirements principally by incorporating into
                                                the acquisition policy the requirement for a technology development
                                                strategy. This strategy is a prerequisite for a project to enter the
                                                technology development phase of the acquisition process, or milestone A.
                                                Table 5 compares the spiral development plan requirements in the act with
                                                the technology development strategy requirements in DOD’s May 2003
                                                acquisition policy.

Table 5: How DOD Policy Addresses Section 803’s Requirements

Section 803 requirements                                                  DOD policy
“A spiral development plan for research and development                   Section 3.5.4 of DOD Instruction 5000.2 states that the technology
program for a major defense acquisition program shall, at a               development strategy document the following:
minimum, include the following matters:”
   “A rationale for dividing the research and development program           The rationale for adopting an evolutionary strategy. For an
   into separate spirals, together with a preliminary identification of     evolutionary acquisition, either spiral or incremental, the
   the spirals to be included.”                                             strategy shall include a preliminary description of how the
                                                                            program will be divided into technology spirals and
                                                                            development increments.
   “A program strategy, including overall cost, schedule and                A program strategy, including overall cost, schedule, and
   performance goals for the total research and development.”               performance goals for the total research and development
                                                                            program.
   “Specific cost, schedule, and performance parameters,                    Specific cost, schedule, and performance goals, including exit
   including measurable exit criteria, for the first spiral to be           criteria, for the first technology spiral demonstration.
   conducted.”
   “A testing plan to ensure that performance goals, parameters,            A test plan to ensure that the goals and exit criteria for the first
   and exit criteria are met.”                                              technology spiral demonstration are met.
   “An appropriate limitation on the number of prototype units that         An appropriate limitation on the number of prototypes units that 

   may be produced under the research and development                       may be produced and deployed during technology 

   program.”                                                                development. 





                                                Page 24                                                 GAO-04-53 DOD’s Acquisition Policy
                                                                Appendix II: Section 803, Spiral Development




 Section 803 requirements                                                              DOD policy
     “Specific performance parameters, including measurable exit                          Specific performance goals and exit criteria that must be met
     criteria, that must be met before the major defense acquisition                      before exceeding the number of prototypes that may be
     program proceeds into production units in excess of the                              produced under the research and development program.
     limitation on the number of prototype units.”
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.




                                                                As shown in the table, DOD’s policy generally responded to the
                                                                requirements in the act concerning guidance for implementation of spiral
                                                                development programs. While the policy includes a technology
                                                                development strategy that requires a test plan to ensure the goals and exit
                                                                criteria for the first technology spiral demonstration are met and an
                                                                independent operational assessment for the release of each product
                                                                increment to the user, it is unclear what the process is for independently
                                                                validating that cost, schedule, and performance exit criteria have been.
                                                                However, DOD stated that the milestone decision authority provides
                                                                independent validation that exit criteria have been met as part of DOD’s
                                                                milestone approval process. Section 803 also requires that a spiral
                                                                development plan include “[s]pecific cost, schedule, and performance
                                                                parameters, including measurable exit criteria, for the first spiral to be
                                                                conducted.” DOD’s policy substituted “parameters” for “goals” and did not
                                                                use the term “measurable” in describing the required exit criteria.

                                                                Finally, section 803 requires the Secretary of Defense to submit to
                                                                Congress by September 30 yearly from 2003 through 2008 a status report
                                                                on each spiral development program. The report is to include information
                                                                on unit costs for the projected prototypes. As of October 23, 2003, DOD’s
                                                                report on the status of each program applying spiral development was still
                                                                in draft and not yet submitted. DOD’s current draft report states that there
                                                                are no research and development programs that have been approved as
                                                                spiral development programs as of September 30, 2003. Section 803
                                                                requirements were implemented in DOD Instruction 5000.2, which was
                                                                effective in May 2003. DOD anticipates that there will be approved spiral
                                                                development programs to report in 2004.




                                                                Page 25                                             GAO-04-53 DOD’s Acquisition Policy
Appendix III: Section 804, Technology
Maturity, and Section 822, Independent
Technology Readiness Assessments
                                            Section 804 of the Defense Authorization Act for Fiscal Year 2002 required
                                            DOD to report on the maturity of technology at the initiation of major
                                            defense acquisition programs. The act directed DOD to report by March 1
                                            of each year between 2003 and 2006 on a requirement in DOD’s policy that
                                            technology must have been demonstrated in a relevant environment (or,
                                            preferably, in an operational environment) to be considered mature
                                            enough to use for product development in systems integration. Each
                                            report is required to (1) identify any major acquisition program that
                                            entered system development and demonstration during the preceding
                                            calendar year with immature key technology that was not demonstrated
                                            in, at minimum, a relevant environment, as required by the new policy;
                                            (2) justify the incorporation of any key technology on an acquisition
                                            program that does not meet that requirement; (3) and identify any
                                            instances that the Deputy Under Secretary of Defense for Science and
                                            Technology did not concur and explain how the issue has been or will be
                                            resolved, including information on the use of independent readiness
                                            assessments. Section 822 of the Defense Authorization Act for Fiscal Year
                                            2003 amended section 804 by adding a requirement that the Secretary of
                                            Defense identify each case in which an authoritative decision has been
                                            made within DOD not to conduct an independent technology readiness
                                            assessment for a critical technology on a major defense acquisition
                                            program and explain the reasons for the decision. On March 18, 2003, DOD
                                            submitted its first report. Table 6 shows the specific requirements for the
                                            report and DOD’s response.

Table 6: How DOD Responded to Section 804 and Section 822 Requirements

Reporting requirements                                               DOD’s response
Section 804: “identify each case in which a major defense            DOD reported two programs (Joint Tactical Radio System Cluster
acquisition program entered system development and                   1 and Composite Health Care System II) that entered system
demonstration during the preceding calendar year and into which      development and demonstration with key technologies that did not
key technology has been incorporated that does not meet the          meet the technological maturity requirement. While DOD did not
technological maturity requirement … and provide a justification     specifically identify all the technologies for these programs that did
for why such key technology was incorporated.”a                      not meet the requirements, it did provide its justification for why
                                                                     the technologies were incorporated.
Section 804: “identify any determination of technological maturity   DOD reported that, in all cases, the Deputy Under Secretary of
with which the Deputy Under Secretary of Defense for Science         Defense for Science and Technology concurred with the
and Technology did not concur and explain how the issue has          technology readiness assessment of the program manager and
been or will be resolved.”                                           the milestone decision authority.




                                            Page 26                                                GAO-04-53 DOD’s Acquisition Policy
                                                               Appendix III: Section 804, Technology
                                                               Maturity, and Section 822, Independent
                                                               Technology Readiness Assessments




 Reporting requirements                                                                      DOD’s response
 Section 822: “identify each case in which an authoritative decision DOD did not report on or identify any cases. 

 has been made within the Department of Defense not to conduct 

 an independent technology readiness assessment for a critical 

 technology on a major defense acquisition program and explain 

 the reasons for the decision.” 

Sources: National Defense Authorization Acts for Fiscal Years 2002 and 2003, DOD, and GAO.
                                                               a
                                                                Technology must have been demonstrated in a relevant environment (or, preferably in an operational
                                                               environment) to be considered mature enough to use for product development in systems integration.




                                                               Page 27                                                   GAO-04-53 DOD’s Acquisition Policy
Appendix IV: Comments from the
Department of Defense




             Page 28         GAO-04-53 DOD’s Acquisition Policy
                        Appendix IV: Comments from the Department of Defense




Now on pp. 14 and 15.




                        Page 29                                          GAO-04-53 DOD’s Acquisition Policy
                Appendix IV: Comments from the Department of Defense




Now on p. 15.




                Page 30                                          GAO-04-53 DOD’s Acquisition Policy
Related GAO Products 



              Best Practices: Setting Requirements Differently Could Reduce 

              Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: 

              February 11, 2003. 


              Best Practices: Capturing Design and Manufacturing Knowledge 

              Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: 

              July 15, 2002.


              Defense Acquisitions: DOD Faces Challenges in Implementing Best 

              Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.


              Best Practices: Better Matching of Needs and Resources Will Lead 

              to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: 

              March 8, 2001. 


              Best Practices: A More Constructive Test Approach Is Key to 

              Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: 

              July 31, 2000.


              Defense Acquisition: Employing Best Practices Can Shape Better 

              Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: 

              April 26, 2000. 


              Best Practices: DOD Training Can Do More to Help Weapon 

              System Program Implement Best Practices. GAO/NSIAD-99-206. 

              Washington, D.C.: August 16, 1999. 


              Best Practices: Better Management of Technology Development 

              Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. 

              Washington, D.C.: July 30, 1999. 


              Defense Acquisitions: Best Commercial Practices Can Improve Program 

              Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. 


              Defense Acquisition: Improved Program Outcomes Are Possible. 

              GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998. 


              Best Practices: DOD Can Help Suppliers Contribute More to Weapon 

              System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.





              Page 31                                   GAO-04-53 DOD’s Acquisition Policy
                     Related GAO Products




           Best Practices: Successful Application to Weapon Acquisition Requires
           Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:
           February 24, 1998.

           Best Practices: Commercial Quality Assurance Practices Offer
           Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.:
           August 26, 1996.




(120214)

           Page 32                                  GAO-04-53 DOD’s Acquisition Policy
                           The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission              Congress, exists to support Congress in meeting its constitutional responsibilities
                           and to help improve the performance and accountability of the federal
                           government for the American people. GAO examines the use of public funds;
                           evaluates federal programs and policies; and provides analyses,
                           recommendations, and other assistance to help Congress make informed
                           oversight, policy, and funding decisions. GAO’s commitment to good government
                           is reflected in its core values of accountability, integrity, and reliability.


                           The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of        through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and            text files of current reports and testimony and an expanding archive of older
                           products. The Web site features a search engine to help you locate documents
Testimony                  using key words and phrases. You can print these documents in their entirety,
                           including charts and other graphics.
                           Each day, GAO issues a list of newly released reports, testimony, and
                           correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                           daily. The list contains links to the full-text document files. To have GAO e-mail
                           this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail
                           alerts” under the “Order GAO Products” heading.


Order by Mail or Phone 	   The first copy of each printed report is free. Additional copies are $2 each. A
                           check or money order should be made out to the Superintendent of Documents.
                           GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                           single address are discounted 25 percent. Orders should be sent to:
                           U.S. General Accounting Office
                           441 G Street NW, Room LM
                           Washington, D.C. 20548
                           To order by Phone: 	 Voice:      (202) 512-6000
                                                TDD:        (202) 512-2537
                                                Fax:        (202) 512-6061


                           Contact:
To Report Fraud,
                           Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in        E-mail: fraudnet@gao.gov
Federal Programs           Automated answering system: (800) 424-5454 or (202) 512-7470


                           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs 	           U.S. General Accounting Office, 441 G Street NW, Room 7149
                           Washington, D.C. 20548