oversight

Missile Defense: Knowledge-Based Practices Are Being Adopted, but Risks Remain

Published by the Government Accountability Office on 2003-04-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to Congressional Requesters




April 2003
             MISSILE DEFENSE

             Knowledge-Based
             Practices Are Being
             Adopted, but Risks
             Remain




GAO-03-441
                                                April 2003


                                                MISSILE DEFENSE

                                                Knowledge-Based Practices Are Being
Highlights of GAO-03-441, a report to
Congressional Requesters                        Adopted, but Risks Remain



The Department of Defense (DOD)                 Our work has shown that programs are most successful when they evolve
would like to build a capable                   products over time rather than try to make big leaps in capability and when
missile defense system that paces               the programs adopt knowledge-based acquisition processes. Similarly, MDA
an ever-evolving threat. This is an             is taking an evolutionary approach to developing the missile defense system
expensive and risky endeavor                    by developing capabilities in spirals or “blocks” rather than attempting to
because it requires a diverse set of
technologies to be quickly
                                                deliver all desired capabilities at one time. The agency intends to facilitate
developed, integrated, and                      this approach by keeping requirements flexible before beginning activities to
deployed across an array of                     integrate technologies into a planned block, following a knowledge-based
platforms. DOD estimates that it                development plan, and demonstrating that technologies work as intended
will need $50 billion for missile               before beginning system integration of a block. In addition, the agency is
defense research and development                seeking to involve stakeholders—such as the military services and
over the next 6 years and likely                operational testers—early in the development effort.
additional funds in subsequent
years. GAO was asked to review                  However, MDA has not adopted some knowledge-based practices regarding
the Missile Defense Agency’s                    long-term investment decision making and, as a result, the missile defense
(MDA) strategy for this investment              program’s success could be hampered. First, MDA is not making an early
and determine what knowledge-
based practices characteristic of
                                                determination of the full cost of a capability. Such an estimate would help
successful programs are being                   decision makers more effectively evaluate which technologies to include
adopted by MDA; what significant                because they offer the best capability for the funds invested. Second, DOD is
practices are not being adopted;                not allocating a “wedge” of funds in its Future Years Defense Plan for system
and whether MDA is following the                production and operations. Without this wedge, DOD may not have the funds
practices that it has adopted.                  needed to procure and maintain the missile defense system.

                                                In addition, the President’s directive to begin fielding a missile defense
                                                capability by 2004 places MDA in danger of getting off track early and
GAO is recommending that DOD
prepare life cycle cost estimates for           impairing the effort over the long term. This danger is highlighted by MDA’s
missile defense elements before                 decision to not follow some of its knowledge-based practices as it develops
beginning integration activities and            the first block of the system. For example, MDA is beginning system
explore the option of setting aside             integration of its first block with immature technology and limited testing.
funds to produce and operate the                While doing so may help MDA meet the President’s deadline, it also
missile defense system over the                 increases the potential that some elements may not work as intended.
long term.
                                                Examples of Missile Defense Elements




www.gao.gov/cgi-bin/getrpt?GAO-03-441.

To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Robert Levin at
(202) 512-4841 or levinr@gao.gov.
Contents


Letter                                                                                                 1
             Results in Brief                                                                           3
             Background                                                                                 4
             Acquisition Strategy Adopts Many Knowledge-Based Practices                                 8
             Two Knowledge-Based Practices Have Not Been Adopted                                       15
             MDA Is Not Following Some Knowledge-Based Practices in
               Developing 2004 Capability                                                              17
             Conclusion                                                                                20
             Recommendations for Executive Action                                                      21
             Agency Comments and Our Evaluation                                                        22
             Scope and Methodology                                                                     22

Appendix I   Comments from the Department of Defense                                                   24



Table
             Table 1: Events and Accomplishments within MDA’s Integrated
                      Master Plan                                                                      12


Figures
             Figure 1: Examples of Missile Defense Programs Transferred from
                      Services into MDA                                                                 5
             Figure 2: Notional Architecture of Future Ballistic Missile Defense
                      System                                                                            7


             Abbreviations

             BMDS              Ballistic Missile Defense System
             DOD               Department of Defense
             MDA               Missile Defense Agency


             This is a work of the U.S. Government and is not subject to copyright protection in the
             United States. It may be reproduced and distributed in its entirety without further
             permission from GAO. It may contain copyrighted graphics, images or other materials.
             Permission from the copyright holder may be necessary should you wish to reproduce
             copyrighted materials separately from GAO’s product.




             Page i                                                        GAO-03-441 Missile Defense
United States General Accounting Office
Washington, DC 20548




                                   April 30, 2003

                                   The Honorable Carl Levin
                                   Ranking Member
                                   Committee on Armed Services
                                   United States Senate

                                   The Honorable Jack Reed
                                   United States Senate

                                   The threat from foreign ballistic missiles has grown steadily since the end
                                   of the Cold War. At least 25 countries now have or are in the process of
                                   acquiring missiles capable of delivering nuclear, biological, or chemical
                                   weapons. Countering this threat demands not only that the Department of
                                   Defense (DOD) develop cutting-edge technology, but also that it acquire
                                   and deploy complex systems more rapidly and effectively.

                                   DOD faces significant technical challenges in building a missile defense
                                   program. So far, DOD has had mixed results in achieving a “hit-to-kill”
                                   capability to destroy enemy warheads reliably, and DOD is still completing
                                   development of the components needed to detect and track a missile in all
                                   phases of its flight—from the boost phase, through the midcourse, and
                                   into the terminal phase. As it works to develop and prove these
                                   capabilities, DOD must also make sure that all missile defense elements
                                   can work together as an integrated system—a complex task in itself, given
                                   the number and diversity of elements involved in missile defense. Once
                                   these basic hurdles have been overcome, DOD still faces a far greater
                                   technical challenge in achieving target discrimination—that is, the
                                   capability to distinguish real warheads from decoys—to defeat more
                                   sophisticated threats.

                                   DOD also faces the challenge of delivering a weapon system with the
                                   capability promised and within the time and cost promised. For example,
                                   previous efforts to develop an airborne laser system and a space-based
                                   tracking and surveillance system encountered substantial delays, cost
                                   overruns, and other difficulties because DOD undertook these efforts




                                   Page 1                                             GAO-03-441 Missile Defense
without fully understanding the effort that would be needed to mature
technologies critical to developing the systems’ required capabilities.1

Lastly, the development of missile defense weapon systems will also
be costly. DOD estimates that it will need about $50 billion for missile
defense research and development between fiscal years 2004 and 2009,
or an average of over $8 billion per year. This investment does not include
funding after 2009 to complete development of those systems, let alone the
funding needed before and after 2009 to produce and operate the systems.

In January 2002, facing these challenges, the Secretary of Defense directed
the Missile Defense Agency (MDA) to orchestrate the integration of DOD’s
diverse missile defense elements into one layered “system of systems” and
to change its acquisition strategy as needed in doing so. In response, MDA
has sought to implement practices that have proven successful in other
DOD and commercial-sector programs. Taken together, these practices
demand a high level of knowledge about a product at key junctures during
development and that this knowledge be used to make prudent investment
decisions. For example, developers should know that a technology has
been demonstrated to work as intended before it is integrated into
a product.

The importance of making the best decisions possible is underscored by
the time-critical nature of MDA’s efforts. In December 2002, the President
directed DOD to begin fielding an initial capability in 2004 to protect the
United States against missile attacks. To help ensure that MDA is using
its resources wisely to develop the best possible system, you asked
us to determine the extent to which the agency’s acquisition strategy
incorporates the knowledge-based practices characteristic of successful
development programs. To better answer your question, we separated it
into three parts: (1) What knowledge-based practices are being adopted by
MDA? (2) What significant practices are not being adopted? and (3) Is
MDA following the knowledge-based practices that it has adopted?




1
 U.S. General Accounting Office, Missile Defense: Knowledge-Based Decision Making
Needed to Reduce Risks in Developing Airborne Laser, GAO-02-631 (Washington, D.C.:
July 12, 2002). U.S. General Accounting Office, Defense Acquisitions: Space-Based
Infrared System-low at Risk of Missing Initial Deployment Date, GAO-01-6
(Washington, D.C.: Feb. 28, 2001).




Page 2                                                    GAO-03-441 Missile Defense
                   MDA has adopted practices that offer the best opportunity to develop
Results in Brief   a complex weapon system successfully. Similar to the successful product
                   development practices of leading commercial organizations, MDA is
                   developing the missile defense system in “blocks,” rather than trying to
                   make a big leap in capability. The agency intends to facilitate this
                   evolutionary approach by keeping the system’s requirements flexible
                   before beginning activities to integrate technologies into a planned
                   block of the missile defense system, following a knowledge-based
                   development plan, and maturing technology before beginning system
                   integration of a block. In addition, the agency is seeking to involve
                   stakeholders—such as the military services and the operational testers—
                   early in the development effort.

                   However, MDA has not adopted two significant practices regarding
                   long-term investment decision making, and, as a result, the program’s
                   success could be hampered. First, MDA is not making an early
                   determination of the full cost of a capability. Specifically, before
                   beginning system integration, MDA does not estimate the total costs for
                   development, production, operations, and sustainment of that block. Such
                   an estimate would help decision makers in evaluating which technologies
                   to include because they offer the best capability for the funds invested.
                   MDA officials told us that they are considering steps to provide such
                   estimates earlier. Second, DOD is not allocating a “wedge” of funds in its
                   Future Years Defense Plan (fiscal years 2004 through 2009) for system
                   production and operations. MDA officials told us that such a wedge has
                   not been set aside because MDA’s acquisition strategy does not presume
                   that a decision will be made to produce and operate the system. DOD
                   risks, however, that when it is ready to procure and maintain the missile
                   defense system, it will not have the funds to do so unless it reduces or
                   eliminates its investment in other important weapon systems. This
                   approach brings little transparency to future investment choices and may
                   constrain options for decision makers.

                   The President’s directive to begin fielding an initial defensive capability in
                   2004 also places MDA in danger of getting off track early and introducing
                   more risk into the missile defense effort over the long term. This danger is
                   highlighted by MDA’s decision to not follow some of its knowledge-based
                   practices as it develops the first block of the missile defense system,
                   which will provide the initial capability. Because of time pressures, MDA
                   must include components that have not been demonstrated as mature and
                   ready for system integration into a particular element, let alone the block
                   overall. For example, MDA has encountered considerable difficulty in
                   developing a new three-stage booster and has yet to flight test interceptor


                   Page 3                                               GAO-03-441 Missile Defense
             boosters in configurations planned for fielding in September 2004. Also,
             MDA’s test program has been limited to date and is under considerable
             schedule pressures. A knowledge-based approach to testing validates
             whether components (1) work individually, (2) work together as a system
             in a controlled setting, and (3) work together as a full system in a realistic
             setting. MDA’s integrated flight tests to date have used surrogate and
             prototype components and have been executed under non-stressing
             conditions. As a result, testing to date has provided only limited data for
             determining whether the system will work as intended in 2004. Also, MDA
             has no plans to demonstrate through flight testing the upgraded primary
             radar in Alaska that will be used to detect and track enemy missiles.

             We are making recommendations to DOD for providing decision makers
             with more timely information on the cost and funding needs of missile
             defense. In commenting on a draft of this report, DOD concurred with
             our recommendations.


             In January 2002, the Secretary of Defense refocused the ballistic missile
Background   defense program. The Secretary delegated to MDA authority to manage all
             ballistic missile defense systems under development and shifted such
             programs controlled by the military services—such as the Army’s Theater
             High Altitude Area Defense program—to the agency. Other programs
             moved to MDA are highlighted in figure 1. These programs, previously
             recognized by DOD as major defense acquisition programs, are now
             considered “elements” and have been consolidated into one overall major
             program called the Ballistic Missile Defense System (BMDS). Future
             architectures of the BMDS may also include “stand-alone” components
             (primarily sensors) that would operate in concert with the various missile
             defense elements.




             Page 4                                                GAO-03-441 Missile Defense
Figure 1: Examples of Missile Defense Programs Transferred from Services
into MDA




Source: DOD.



In December 2002, President Bush directed the Department of Defense to
begin fielding the first block of the ballistic missile defense system for
operational use in 2004. That is, in addition to focusing resources on the
development of a testbed for developmental testing of missile defense
elements, he instructed MDA to build in an operational capability that
would protect the United States against missile attacks. The fielding of
such capabilities is referred to as an “initial defensive operations”
capability, and, in a statement by the Secretary of Defense, “…would be a
very preliminary, modest capability.” The initial capability will be based on
the testbed and augmented with additional developmental assets.

When fully deployed, the BMDS will include (1) space- and ground-based
sensors to provide early warning and tracking of missile launches;
(2) ground-based radars to identify and refine the tracks of threatening
reentry vehicles and associated objects; (3) ground- and sea-based
interceptors to destroy enemy missiles through “hit-to-kill” impacts; and
(4) fire control nodes for battle management and execution of the
ballistic missile defense mission. A notional architecture of future BMDS
blocks is illustrated in figure 2. For example, the initial capability for
defense of the United States against long-range missiles would come




Page 5                                                GAO-03-441 Missile Defense
    from the Ground-Based Midcourse Defense element and BMDS sensors,
    as follows:

•   Ground-Based Midcourse Defense Element. The principal components
    of the Ground-Based Midcourse Defense element for defensive operations
    include interceptors sited at Fort Greely, Alaska, and Vandenberg Air
    Force Base, California; a fire control node for battle management and
    execution located at Schriever Air Force Base, Colorado, with a backup
    node at Fort Greely; an upgraded Cobra Dane radar at Eareckson Air
    Station in Shemya, Alaska; and an upgraded early warning radar at Beale
    Air Force Base, California.
•   BMDS Sensors. Sensors external to the Ground-Based Midcourse
    Defense element and available for defensive operations include Defense
    Support Program satellites for missile warning and forward-deployed
    Aegis AN/SPY-1 radars on existing Navy cruisers.

    The above assets comprise the initial configuration, which is scheduled for
    fielding at the end of September 2004. The agency’s near-term intention is
    to expand this capability by adding more interceptors at Fort Greely,
    Alaska; a sea-based X-band radar deployed in the Pacific for use in flight
    testing; and an upgraded early warning radar at Fylingdales, England, by
    the end of 2005.




    Page 6                                             GAO-03-441 Missile Defense
Figure 2: Notional Architecture of Future Ballistic Missile Defense System




                                          Page 7                             GAO-03-441 Missile Defense
                       To ensure the delivery of high-quality products on time and within budget,
Acquisition Strategy   successful developers have adopted acquisition strategies that are
Adopts Many            anchored in knowledge. Specifically, they establish decision points for
                       moving forward from technology development to product development
Knowledge-Based        and on to production.2 At each decision point, decision makers ask
Practices              themselves whether they have gained the knowledge they need to proceed
                       into the next acquisition phase. For example, they determine whether the
                       work can be completed with the money and time available and whether
                       the product will be worth the required investment. If any of these
                       questions are answered negatively, the program does not go forward.
                       Other practices that facilitate successful outcomes include developing
                       systems in stages rather than attempting to deliver all desired capabilities
                       at one time, keeping requirements flexible so that the system can be
                       produced within available resources, making sure technology is proven
                       before incorporating it into a development program, and involving the
                       right people at the right time in decision making. Commercial and DOD
                       programs that have successfully implemented these practices have found
                       that they help curb the incentive to rely on immature technologies and to
                       over-promise the capability that can be delivered. Moreover, these
                       programs found that keeping stakeholders involved in decision making
                       ensured that the developed product better met the customers’ needs.

                       MDA realizes the value of these practices and is seeking to incorporate
                       them into its acquisition strategy. Specifically, as discussed below, MDA
                       plans to evolve the missile defense system over time, rather than trying to
                       make a big leap in its capability. MDA is also planning to keep the system’s
                       requirements flexible before beginning system integration and to follow a
                       knowledge-based development plan. In addition, MDA is seeking to
                       involve stakeholders—such as the military services and the operational
                       testers—early in the development effort.




                       2
                        The three acquisition phases are distinguished by the activities that occur during each
                       of the three phases. During technology development, scientists apply scientific knowledge
                       to a practical engineering problem and demonstrate that components with desired
                       capabilities can be developed. Product development includes integrating those components
                       into a stable system design and demonstrating that the design will result in a product that
                       meets the customer’s needs and can be produced with the time and money available.
                       Production is the manufacturing of the product.




                       Page 8                                                        GAO-03-441 Missile Defense
Evolutionary Development   Historically, many new development programs in DOD have sought to
                           achieve a great leap ahead in capabilities. Because the technology was
                           often not available to make such leaps, programs were often in
                           development for years while engineers tried to develop and mature the
                           needed technologies. As the time required to develop a system increased,
                           so did the cost of the system.

                           In contrast, development programs are most successful when they take an
                           evolutionary, or phased, approach. In doing so, they establish time-phased
                           plans to develop new products in increments. The first increment often
                           has a limited capability because it incorporates technology that is already
                           mature or can be matured quickly. As new technology is developed, it is
                           incorporated into subsequent increments so that the product’s capability
                           evolves over time. This approach reduces risks by introducing less new
                           content and technology into a program’s design and development effort.
                           An evolutionary strategy also enables developers to deliver a series of
                           interim capabilities to the customer more quickly. Recognizing the benefits
                           of evolving systems, DOD recently revised its acquisition system policy to
                           encourage evolutionary development.

                           The Missile Defense Agency’s new strategy for developing the ballistic
                           missile defense system adopts evolutionary development. At the beginning
                           of each block, the agency intends to predict the capability that can be
                           developed given a “tool box”3 of currently available technology and then to
                           design that system. As additional technology matures, the agency can
                           incorporate it into the next block being developed.

                           Two key practices adopted by MDA can be expected to help the agency in
                           taking an evolutionary approach. They include (1) keeping requirements
                           flexible and (2) following a knowledge-based development plan with
                           specific decision points and criteria for moving forward.

Flexible requirements      Customers generally want new products that are high performance at
                           low cost, delivered as soon as possible. But developing and producing
                           such a product may exceed the developer’s technology or engineering
                           expertise, or may be too costly and time-consuming for the customer to
                           accept. Therefore, what a customer needs in a product and what a
                           developer can produce given available resources must be matched to



                           3
                            All possible elements/components and interfaces that could be used in a ballistic missile
                           defense system.




                           Page 9                                                        GAO-03-441 Missile Defense
                   form an achievable set of product requirements before development
                   begins. To make this match, both the customer and the developer must be
                   flexible so that potential gaps between needs and resources can be closed.
                   Flexibility represents the customer’s ability and willingness to lower
                   product expectations, coupled with the product developer’s willingness
                   and ability to invest more resources to reduce technical risks before
                   beginning system integration. Without flexibility, resources and needs can
                   still be matched, but the options for closing the gaps between the two are
                   limited to additional investments on the developer’s part. In fact, our past
                   reviews have found that many traditional DOD acquisition programs
                   incurred substantial cost increases and schedule delays because
                   requirements by the military services were unrealistic and inflexible.

                   Because of its flexibility, MDA’s new “capabilities-based approach” for the
                   development of missile defense elements provides a greater opportunity
                   to resolve this potential gap between resources and needs.4 Instead of
                   proceeding with the development of a solution based on firm operational
                   requirements set by the services, the agency, in coordination with the
                   warfighter, considers a number of system architectural options that can
                   be developed given the mature technologies that are available. Decision
                   makers evaluate the alternative architectures according to the potential
                   military utility that each offers, given the constraints of cost and schedule.
                   For example, decision makers would compare the regions of the world
                   that each architecture could defend from missile launches (defended area)
                   or each architecture’s probability of engagement success. This acquisition
                   approach eliminates any possible gap between resources and needs; the
                   customer (warfighter) accepts the best capability that the developer can
                   deliver given available resources and assumes enhanced capabilities will
                   be built into future blocks.

Knowledge-based    In the early stages of a major defense acquisition program, DOD
development plan   establishes a master schedule for moving through development and into
                   production. Historically, once these milestones have been established,
                   they have often been an impetus for moving forward even if requisite
                   knowledge had not been attained—an action that invariably caused
                   schedules to slip and costs to rise. By contrast, we have found that
                   successful developers place more importance on capturing specific



                   4
                    At the beginning of systems integration, MDA plans to establish system capability
                   specifications. As in any organization, these specifications may change if testing shows
                   that they are unattainable or that meeting them will be too costly or take too much time.




                   Page 10                                                        GAO-03-441 Missile Defense
technology, design, and manufacturing knowledge than meeting
milestones and they use this knowledge to make investment decisions.
Moreover, these developers identify and use specific markers or criteria—
such as technology readiness levels, percentage of engineering drawings
released to the manufacturer, or the percentage of manufacturing
processes under statistical control—to ensure that the program has
sufficient knowledge to move forward.

MDA has similarly adopted a structured plan, called the Integrated Master
Plan, for moving forward with requisite knowledge. Every block would
move through eight formal “events,” each of which would include an
identified set of accomplishments that should be completed before the
program moves on to the next event. (See table 1 for a list of events and
their associated accomplishments.) As a block moves through the events,
MDA plans to use quantitative criteria whenever possible to enhance
decisions on whether to continue developing the block as it is or to make
changes. At the end of a block’s development, MDA expects to recommend
one of four alternative actions to decision makers. Officials could
recommend that the elements be (1) transferred to the services to be
produced and fielded in its current configuration, (2) further developed
in a subsequent block, (3) retained as a test asset in the missile defense
testbed, with some capability available for operational use, or
(4) terminated.

One such quantitative criterion adopted by MDA is technology readiness
levels. Our reviews have found that successful developers often use
technology readiness levels as an analytical tool to assess the maturity of
technology being considered for inclusion in a product. There are nine
levels of maturity. The level increases as the technology becomes closer in
form, fit, and function to the actual system and is demonstrated in more
realistic environments. For example, technology is least mature, or least
ready for inclusion in a product, when it is an idea being explored in paper
studies. Conversely, technology is most mature when it has been
incorporated into the intended product and that product has been
demonstrated in its intended environment. The lower the level of
technology readiness, the more ground that must be covered to bring the
technology to the point at which it can meet the intended product’s cost,
schedule, and performance requirements with little risk. We found that
most successful developers insert new technology into a product only
when the technology has been incorporated into prototype hardware and
that hardware has been demonstrated to work in the environment in
which it is expected to be used.



Page 11                                             GAO-03-441 Missile Defense
MDA’s knowledge-base development plan incorporates the use of
technology readiness levels at Event 1 to assess the maturity of technology
proposed for a block configuration. The strategy calls for including new
technology at system integration (Event 4) if that technology has been
proven in prototype hardware that works in the environment in which it
is expected to be used. While the incorporation of mature technology at
system integration is MDA’s preferred approach, the strategy retains the
flexibility to include less mature technology if it offers a significant benefit
and the risk of including it is acceptable. In such instances, MDA expects
to develop a plan for reducing the risk of moving forward with immature
technology and to remove the technology from the block if the risk has not
been reduced at subsequent decision points.

Table 1: Events and Accomplishments within MDA’s Integrated Master Plan

Event 0–Block Capability Alternatives
  Block planning process completed
  Long lead targets, tests, and exercises identified
  Affordability Analysis completed
  Preliminary block plan approved
Event 1–Preliminary Configuration Definition
  Preliminary block description approved
  Technology readiness levels assessed
  Performance assessments updated
  Preliminary concept of operations and operational architecture drafted
  Risks assessed and mitigation programs established
  Detailed cost estimates for elements/components available
  Cost/benefit analysis updated
  Integration/test objectives defined
  Element/component preliminary design reviews completed
  Required funding identified
  Integrated master schedule completed
Event 2–Configuration Definition
  Critical design reviews for all element/component/targets programs completed
  Performance/cost assessments updated
  Risks assessed and mitigation programs updated
  Military utility characterized and concept of operations refined
  Preliminary integration test plan available
  Funding available and resources allocated
  Block definition updated
  Integrated master schedule updated
Event 3–First Development Article
  First development article built and initial tests completed
  Targets built and initial tests completed




Page 12                                                   GAO-03-441 Missile Defense
                    Test range and support planning completed
                    Concept of operations defined and operational architecture available
                    Funding and Estimate at Completion assessed
                  Event 4–Integrated Test Readiness Review
                    Block integration test planning completed
                    Element/component test and checkout completed
                    Target test and checkout completed
                    Ballistic missile defense system tactics, techniques, and procedures for designated
                    user defined
                    Funding and Estimate at Completion updated
                    Operational characterization of each element completed
                    Operational certification of element completed
                  Event 5–Interim Test and Progress Review
                    50 percent of system test objectives accomplished
                    Support systems defined
                    Training systems defined
                    Funding updated and Estimate at Completion verified
                    Initial transition plans completed
                    Initial operational characterization completed
                  Event 6–Element/component transition decision points
                    System/element/component testing completed
                    Operational characterization completed
                    Support systems planned and budgeted
                    Training systems planned and budgeted
                    Transition plans completed and funded
                    Production plans available
                    Updated block definition available
                    Element/component certification of military utility completed
                    Service total obligation authority available
                  Event 7–Block Certification of military utility
                    Military utility assessed and system element/component offered for transition
                    Ballistic missile defense system capability demonstrated
                    Life cycle cost estimate indicates long-term affordability
                    Reliability, maintainability, and availability, and support requirements characterized
                    Block certification of military utility completed
                    Integration of declared block capability of ballistic missile defense system
                 Source: DOD.

                 Note: The events and accomplishments in MDA’s Integrated Master Plan are being revised. This set
                 was in effect as of February 2003.


Involvement of   Developers have found that if they are to be successful, all groups that
Stakeholders     have a stake in the product should be involved at all appropriate stages
                 in the development effort. For example, cost analysts are needed to
                 accurately estimate the cost of the product, experts in test and evaluation
                 are needed to objectively assess the performance of product prototypes,



                 Page 13                                                           GAO-03-441 Missile Defense
and others are needed to enhance understanding of the customer’s
needs. By involving these groups from the time a product design is
created and keeping them involved throughout the product integration
and demonstration phases, a program can ensure that it has a complete
perspective.

A key forum for stakeholder input is the Missile Defense Support Group,
which includes representatives from the Joint Air and Missile Defense
Organization; the Comptroller’s Office; the Director, Operational Test and
Evaluation; and other units across DOD. The support group provides
advice on such subjects as policy, operations, testing, acquisition, and
resources to the Director of the Missile Defense Agency and the Under
Secretary of Defense for Acquisition, Technology and Logistics, and
supports the Senior Executive Council5 in decision making. In addition,
an analytical working group remains in close contact with MDA
management so that it can collect information for the Missile Defense
Support Group as well as conduct independent analyses of the missile
defense program’s work.

Initially, not all members of the Missile Defense Support Group and its
working group believed that MDA’s approach to stakeholder involvement
would be successful. Soon after the support group was established,
members voiced concern that they were not getting sufficient access to
agency information. Members said that communication with MDA was
poor and that access to knowledgeable MDA individuals was limited, all
of which made it difficult to provide timely advice. For example, in
April 2002, the agency presented options to the support group to address
capability shortfalls in sea-based terminal defense caused by the loss of
the Navy Area missile defense program.6 Members of the group questioned
why a successor to the Navy Area program was needed. MDA planned
additional briefings pertaining to this issue; however, the group was never
briefed and MDA, without additional group input, went directly to the
Under Secretary of Defense for Acquisition, Technology and Logistics to
obtain approval for a particular approach to sea-based terminal missile


5
  The Senior Executive Council is led by the Deputy Secretary of Defense, and its members
are the Under Secretary of Defense for Acquisition, Technology and Logistics, and the
Secretaries of the Army, Navy, and Air Force. The Council provides oversight of MDA’s
activities and is responsible for making program adjustments and deciding to transition or
transfer a capability to the services. Furthermore, the Council approves MDA’s investment
strategy and decisions.
6
    DOD cancelled the Navy Area program in 2001 due to cost overruns.




Page 14                                                       GAO-03-441 Missile Defense
                       defense. DOD officials told us that they did not seek further input from the
                       group because there was insufficient time for it to fully understand and
                       evaluate the issue before a decision had to be made.

                       Shortly after this, support group members provided comments and
                       suggestions to the Under Secretary and the MDA director on increasing
                       communication, obtaining access to MDA personnel, and receiving timely
                       information. Subsequently, MDA made progress in addressing the support
                       group’s suggestions and concerns relating to these issues. As a result,
                       according to support group members, the level of interaction,
                       communication, and involvement has improved. In particular, support
                       group members attend weekly system and element review meetings and
                       have regular interaction with agency personnel outside of the Missile
                       Defense Support Group forum.


                       While MDA has adopted many of the practices of successful acquisition
Two Knowledge-         programs, it has not incorporated two particularly significant ones.
Based Practices Have   Specifically, before beginning system integration, successful developers
                       reduce their investment risk by estimating total costs and determining that
Not Been Adopted       funding is available for developing, producing, and operating the system.
                       (We recently reported on the significance of DOD’s costs for operating its
                       weapon systems and keeping them ready for action over many years.7)

                       In MDA’s case, decision makers would benefit from having this knowledge
                       available before MDA begins system integration because decision makers
                       would be better positioned to consider whether to delay until subsequent
                       blocks those elements that currently have unaffordable production,
                       operation, or maintenance costs and whether costs might be lowered in
                       the future by inserting new technology or implementing better engineering
                       solutions. The information would also help decision makers to compare
                       all elements’ costs and decide which elements should be included in a
                       planned block of the missile defense system because they offer the best
                       capability for the funds invested.

                       As of February 2003, MDA’s draft Integrated Master Plan did not call for
                       an element’s life cycle cost to be estimated at the beginning of system



                       7
                        U.S. General Accounting Office, Best Practices: Setting Requirements Differently
                       Could Reduce Weapon Systems’ Total Ownership Costs, GAO-03-57 (Washington, D.C.:
                       Feb. 11, 2003).




                       Page 15                                                   GAO-03-441 Missile Defense
integration (Event 4) but rather at the point when that element is
considered for transfer to a military service for production, operation, and
maintenance (Event 7). Moreover, MDA may never estimate the full cost of
some elements because, according to officials, some elements may never
be transferred to a military service for production and operation. For
example, MDA plans to continually upgrade elements such as the
Ground-Based Midcourse Defense element in the missile defense test bed
even though it would be available for combat use. In such cases, MDA
plans to estimate only the element’s development costs—not its
production, operation, and maintenance costs.

Also, DOD has not allocated funds in its Future Years Defense Plan
(fiscal years 2004 through 2009) for the production, operation, and
maintenance of any elements that might be transferred in the future to the
military services. MDA has established optional decision points called
“off ramps” where elements such as Theater High Altitude Area Defense
could be transferred to the military services, but DOD has not set aside a
“wedge” of funding for the element’s production and operating costs.

MDA officials told us that the agency is considering revisions to its
Integrated Master Plan so that it can provide decision makers with
complete life cycle cost information on each element prior to beginning
system integration activities for each block of the missile defense system.
For example, MDA anticipates defining each element’s training and
support systems before it begins system integration activities. The officials
said that that they are still, however, in the process of determining the cost
information that needs to be collected and the timing of its collection. In
terms of setting aside a wedge of funding in the Future Years Defense Plan
for production and operations costs, MDA officials told us that no such
action is planned at this time because MDA’s acquisition strategy does
not presume that a decision will be made to transfer the element to the
military service. They told us that they expect funding would be
made available.

However, we believe that unplanned operation and maintenance costs
could be a particular problem for DOD because its budget for these
expenses is already stressed by the rising cost to operate and maintain
many aging weapon systems. Also, when DOD is at the point of deciding
whether to transfer elements of the missile defense system to a military
service for production and operation, DOD could find that it does not have
the funds available for missile defense without reducing or eliminating
funding for other important weapon systems.



Page 16                                              GAO-03-441 Missile Defense
                           Because the President directed that a missile defense capability be fielded
MDA Is Not                 beginning in 2004, MDA will not be able to follow some knowledge-based
Following Some             practices in developing the initial capability in this brief time. As noted
                           earlier, MDA’s draft Integrated Master Plan recommends that when a
Knowledge-Based            block enters system integration it include mature technology but the plan
Practices in               allows for the inclusion of less mature technology if the benefits are
                           significant and if risks can be mitigated. Given the Presidential direction,
Developing                 MDA must include components in the block that have not been
2004 Capability            demonstrated as mature and ready for integration into a particular
                           element, let alone the block overall. MDA’s plan also calls for rigorous
                           testing before the agency recommends that the system or its elements be
                           available for fielding. However, MDA’s test program has been limited to
                           date and is under considerable time pressures.


System Integration         Our past reviews of DOD and commercial product development programs
of Block 2004 Begins       have shown that programs are in a much better position to succeed if
with Some Immature         components that incorporate new technologies are matured to a high level
                           before being integrated into a product. Conversely, developers that
Technologies               initiated product developments with immature technology increased the
                           risk that their products would fail tests and that some aspects of the
                           products’ design would have to be reworked because components did not
                           perform as predicted. The overall impact of such problems was often that
                           products did not deliver the promised capability or the developers had to
                           spend additional time and money to develop that capability.

                           While its draft Integrated Master Plan recommends that system integration
                           begin with mature technologies (Event 4), MDA has begun including
                           components into the Block 2004 configuration that are not yet mature.
                           Two examples are the Cobra Dane radar and the boosters for the
                           Ground-Based Midcourse Defense interceptors.

                       •   The Cobra Dane radar is located at Eareckson Air Station in the western
                           end of the Aleutian Islands chain in Alaska. Planned hardware and
                           software upgrades intended to provide the radar with real-time acquisition
                           and tracking capabilities are expected to be completed in fiscal year 2004.
                           MDA has no plans, however, to demonstrate the expected functionality of
                           the radar through integrated flight tests.
                       •   MDA has encountered considerable difficulty in developing a three-stage
                           booster for the Ground-Based Midcourse Defense element and has yet to
                           flight test interceptor boosters in configurations planned for fielding in
                           September 2004. By the time the new booster was flight tested in August
                           2001, it was already about 18 months behind schedule. The first booster



                           Page 17                                             GAO-03-441 Missile Defense
                            flight test was successful, but the second booster drifted off course and
                            had to be destroyed 30 seconds after launch. Subsequently, the agency
                            authorized two new contracts for developing boosters for use in the Block
                            2004 capability. While this strategy should reduce risk in the program, the
                            first demonstrations of these boosters will occur in the flight tests
                            scheduled later this year.

                            MDA officials told us that they could not deploy an initial capability in the
                            timeframe directed by the President if they did not continue to develop the
                            technology while designing the system. MDA officials told us that they
                            expect to follow their knowledge-based development plan as they develop
                            the next block (Block 2006) of the missile defense system.


Testing under Pressure      The fundamental purpose of testing is to gauge the progress being made
and Limited to Date         when an idea or concept is translated into an actual product and,
                            ultimately, to make sure the product works as intended. Leading
                            commercial firms conduct testing to discover potential developmental
                            problems early. The firms focus on validating that their products have
                            reached increasing levels of product maturity at given points in time. The
                            firms’ products have three maturity levels in common: components work
                            individually, components work together as a system in a controlled
                            setting, and components work together as a full system in a realistic
                            setting. Testing in this systematic manner helps ensure that problems
                            are identified and corrected early, when the cost of solving problems is
                            lower and more options are available. Over time, disciplined testing
                            helps confirm that the product eventually produced will meet the
                            customer’s needs.

                            In the past, when DOD programs have been schedule—rather than event—
                            driven, program managers have found it difficult to slow the program if
                            problems were identified during testing. MDA has been placed in a similar,
                            pressured position as it prepares to field an initial capability by September
                            2004. Also, only limited test data is available for determining whether a
                            credible capability will be available at that date.

One system-level test is    The capability that MDA expects to deploy is essentially a collection of
planned prior to fielding   elements that are connected by battle management software. Initially, the
                            mission of the software will be to hand off data from the radars that detect
                            and track enemy missiles to the shooters that launch interceptors to kill
                            the missiles. For example, the battle management software could
                            communicate to the Ground-Based Midcourse element data on the
                            position of an intercontinental ballistic missile being tracked by the Aegis



                            Page 18                                             GAO-03-441 Missile Defense
                                   Ballistic Missile Defense radar. MDA has begun the development of battle
                                   management hardware/software and has completed some ground tests of
                                   its capability. However, MDA plans to deploy the block although the battle
                                   management software’s ability to interoperate with the elements as an
                                   integrated missile defense system will not be flight tested until the Spring
                                   of 2004.

Element flight-test data           Integrated flight tests to date have demonstrated that the Ground-Based
is limited                         Midcourse Defense and Aegis Ballistic Missile Defense elements can
                                   defeat a mock warhead in a test environment. However, the tested
                                   elements did not include all of the same components that will be part of
                                   the elements deployed in 2004. Instead these elements were tested using
                                   some surrogate and prototype components. For example, all tests of the
                                   Ground-Based Midcourse element have included a surrogate booster and
                                   a prototype kill vehicle. In addition, tests of this element have not included
                                   the Cobra Dane radar that will be used in September 2004 to detect
                                   and track intercontinental ballistic missiles. The Cobra Dane radar will
                                   not actively participate in integrated flight tests at least through
                                   September 2007.

                                   Element flight tests have also been executed under nonstressing
                                   conditions that are not fully representative of the environments that the
                                   elements would experience in combat. All flight tests completed to date
                                   have been limited to a single corridor and intercept region, that is all
                                   targets have been launched from Vandenberg Air Force Base, California,
                                   and interceptors have been launched from the Reagan Test Site in the
                                   Marshall Islands. As a result, flight-test engagement conditions are limited
                                   to those with slower closing velocities and shorter intercept ranges.
                                   Testing under conditions such as these significantly limit the data MDA
                                   can collect on system effectiveness and readiness.

Operational test data is limited   An operational test assesses the effectiveness of the system against the
                                   known threat and its suitability for combat use. U.S. law requires that such
                                   tests be carried out on major defense acquisition programs and assessed
                                   by DOD’s Director, Operational Test and Evaluation, before a full-rate
                                   production decision is made. The purpose of the Director’s assessment is
                                   to advise the Secretary of Defense and Congress on the effectiveness of
                                   the system against the known threat and its suitability for combat use.

                                   MDA does not plan to operationally test the Block 2004 Ground-Based
                                   Missile Defense element before it is available for initial defensive
                                   operations. The September 2004 fielding is not connected with a full-rate
                                   production decision that would clearly trigger statutory operational testing


                                   Page 19                                              GAO-03-441 Missile Defense
             requirements. Nonetheless, according to DOD officials, MDA plans to
             incorporate both developmental and some operational test requirements
             in integrated flight tests.

             The Director, Operational Test and Evaluation, will provide comments on
             an element’s operational effectiveness and suitability as demonstrated in
             these tests.8 However, Operational Test and Evaluation officials said that
             because developmental tests are scripted, planned events, they do not
             provide the opportunity to assess how the equipment and its operators will
             function under unforeseen conditions.


             MDA is attempting to build a ballistic missile defense capability that paces
Conclusion   an ever-evolving threat. This is an expensive and risky endeavor, because
             it requires a diverse set of technologies that must be quickly developed,
             integrated, and deployed across an array of land-, air-, sea-, and space-
             based platforms. Whether MDA can successfully meet the challenge of
             quickly developing an effective and suitable missile defense system
             depends in large part on its willingness to adopt practices that have made
             other developers successful and to implement those practices as it
             develops each block.

             Certainly, the presidential directive has already caused MDA to not follow
             some of the knowledge-based practices that it had adopted as it develops
             Block 2004. Giving up this approach opens the door to greater cost and
             performance risks. Beginning system integration of Block 2004 with
             immature technology increases the potential that some element may not
             work as intended. If this happens, MDA will be faced either with fielding a
             less than credible system or likely spending more money in an attempt to
             develop the desired capability within the time allowed. In addition to the
             challenge it faces in Block 2004, MDA faces the challenge of getting its
             acquisition program back on track. Because the ballistic missile threat is


             8
              The National Defense Authorization Act for Fiscal Year 2002 (P.L. 107-107) requires the
             Director of Operational Test and Evaluation to (1) annually assess, and report to Congress
             on, the adequacy and sufficiency of MDA’s test program during the preceding fiscal year,
             (2) monitor the development of MDA’s plan for ensuring that each critical technology for a
             missile defense program is successfully demonstrated in an appropriate environment
             before that technology enters into operational service (and provide the Director of MDA
             with appropriate comments), and (3) review, on an ongoing basis, the development of
             MDA’s annual program goals (including testing goals) and annual program plan (including
             schedules for flight tests and other significant testing activities) and provide any resulting
             comments on the plans to the Secretary of Defense and the Director of MDA.




             Page 20                                                         GAO-03-441 Missile Defense
                      rapidly increasing, MDA could always believe it is operating in an
                      emergency environment. Yet, it has never been proven that it takes longer
                      to acquire a weapon system if a knowledge-based acquisition plan is
                      followed. Instead, the opposite should be true, because such a plan
                      decreases the likelihood that deadlines will be missed because critical
                      elements do not work as intended.

                      MDA and DOD also need to address the long-term implications of their
                      investment strategy. Both are assuming increased investment risk by not
                      having the right information available for decision makers at the right
                      time. The level of anticipated spending magnifies this risk. MDA officials
                      told us they are considering changes to MDA’s Integrated Master Plan to
                      identify life cycle costs at the beginning of system integration activities so
                      that tradeoff decisions can be made in a more timely manner. However,
                      because DOD has not yet set aside funds to cover its long-term costs, the
                      department could find that it cannot afford to procure and maintain that
                      system unless it reduces or eliminates its investment in other important
                      weapon systems. By setting aside funds in the Future Years Defense Plan,
                      we believe DOD would bring needed visibility to the impending trade-offs
                      between missile defense and other weapon system spending for
                      procurement and operations.


                      To assist MDA and DOD decision makers in determining which elements
Recommendations for   or components should be included in each new block of the Ballistic
Executive Action      Missile Defense System, we recommend that the Missile Defense Agency,
                      before beginning integration activities, prepare a life cycle cost estimate
                      for configuring the element or component that the agency is considering
                      including in the block.

                      To help ensure that funds are available to produce and operate the
                      elements of the missile defense system when a decision is made to transfer
                      elements to the military services, we recommend that the Secretary of
                      Defense explore the option of requiring the services to set aside funds for
                      this purpose in the Future Years Defense Plan.




                      Page 21                                               GAO-03-441 Missile Defense
                     In commenting on a draft of this report, the DOD concurred with
Agency Comments      our recommendations.
and Our Evaluation
                     Regarding our recommendation that MDA prepare life cycle cost
                     estimates before beginning integration activities, DOD said that MDA
                     will prepare its best estimate of life cycle costs based upon projected
                     hardware life, operational cost drivers, and initial capability quantities
                     prior to integration activities for each block.

                     Regarding our recommendation that DOD set aside funds in its Future
                     Years Defense Plan in anticipation of the transfer of missile defense
                     system elements to the military services, DOD said that there is benefit in
                     budgeting funds when such a transfer is anticipated. Doing so would
                     promote budget stability and improve the likelihood that an element or
                     component would actually be fielded.

                     DOD also suggested technical changes, which we incorporated as
                     appropriate. DOD’s comments are reprinted in appendix I.


                     To address our objectives, we analyzed documents that detailed the
Scope and            Missile Defense Agency’s new acquisition practices and compared the
Methodology          practices to those of successful development programs. We also obtained
                     detailed briefings from Missile Defense Agency officials regarding the
                     agency’s plan for the implementation of these practices and contrasted
                     that plan to the implementation plan of successful programs. In addition,
                     we discussed the challenges and risks that the agency faces as it
                     implements its new plan with the Institute of Defense Analyses,
                     Alexandria, Virginia. We also discussed these issues with all members of
                     the Missile Defense Support Group, including the Office of the
                     Undersecretary for Acquisition, Technology and Logistics; Office of the
                     Undersecretary for Policy; Office of the Undersecretary (Comptroller);
                     General Counsel; Office of the Assistant Secretary (Command, Control,
                     Communications, and Intelligence); Office of the Director, Operational
                     Test and Evaluation; Office of the Director, Program Analysis and
                     Evaluation; Office of the Director, Cost Analysis Improvement Group,
                     in Washington, D.C.; and the Joint Staff; Department of the Army;
                     Department of the Air Force; Department of the Navy; and the Missile
                     Defense Agency in Arlington, Virginia.




                     Page 22                                              GAO-03-441 Missile Defense
We conducted our review from March 2002 to March 2003 in accordance
with generally accepted government auditing standards.


As arranged with your staff, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from its
issue date. At that time, we plan to provide copies of this report to the
Chairmen and Ranking Minority Members of the Senate Committee on
Armed Services; the Senate Committee on Appropriations, Subcommittee
on Defense; the House Committee on Armed Services; the House
Committee on Appropriations, Subcommittee on Defense; the Secretary of
Defense; and the Director, Missile Defense Agency. We will make copies
available to others upon request. In addition, the report will be available at
no charge on the GAO Web site at http://www.gao.gov/.

If you or your staff have any questions concerning this report, please
contact me on (202) 512-4841. Major contributors to this report were
Katherine Schinasi, Barbara Haynes, Cristina Chaplain, David Hand,
Alan Frazier, Randy Zounes, Adam Vodraska, Jose Ramos, and
Greg Lagana.




R. E. Levin
Director
Acquisition and Sourcing Management




Page 23                                              GAO-03-441 Missile Defense
             Appendix I: Comments from the Department
Appendix I: Comments from the Department
             of Defense



of Defense




             Page 24                                    GAO-03-441 Missile Defense
           Appendix I: Comments from the Department
           of Defense




(120135)
           Page 25                                    GAO-03-441 Missile Defense
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548