oversight

Strategic Defense System: Stable Design and Adequate Testing Must Precede Decision to Deploy

Published by the Government Accountability Office on 1990-07-06.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                                                   _ ..“- ,.... -..


                  Unit&   Stqtes   General   Accounting   Office
                   Report to the Chairman, Legislation
GAO               ‘axd Fktimal Security Subcommittee,
                   Cgmx&tee on Govement Operations,
                   House af Representatives




                  Stable Design and
                  Adequate Testing Must
                  Precede Decision to
                  Deploy




GAO/IMTEC-SO-61
        ‘::                .I ‘::
    .         .
I                          . .




                                         .. ..
                                                                 :.




                                             .                   I
                                             ,.            . .
                                         .             .‘:‘.
                                                                     .:
                                                         .’            :
                                                         . .
                                                 .‘i      : : ;
                                             .               ‘.
                                                         ..
                                                 i.
                                                                ‘1,
                                                ‘.            .,    ..
                                             ; ’
                                             . .                 .’




                                                        .        ’
                                                              . ..



                                                               .




                                                 ..


                  ..                ..
                       .       ’
GAO
      United States
      General Accounting  Office
      Washington, D.C. 20548

      Information    Management    and
      Technology    Division

      B-239476

      July 6, 1990

      The Honorable John Conyers, Jr.
      Chairman, Legislation and
        National Security Subcommittee
      Committee on Government Operations
      House of Representatives

      Dear Mr. Chairman:

      This report discusses why the Strategic Defense Initiative Organization will not be able to
      support currently scheduled full-scale development or deployment decisions on any part of
      Phase I of the Strategic Defense System. The organization will not be able to support these
      decisions until it has solidified the system architecture, conducted integrated system-level
      testing in real time involving actual pieces of hardware and software, and until the
      Department of Defense adheres to technical and programmatic oversight, commensurate
      with such a system.

      As arranged with your office, unless you publicly announce the contents earlier, we plan no
      further distribution of this report until 30 days from the date of this letter. We will then send
      copies to other appropriate congressional committees; the Director, Strategic Defense
      Initiative Organization; the Director, Office of Management and Budget; the Secretary of
      Defense; and other interested parties.

      This report was prepared under the direction of Samuel W. Bowlin, Director for Defense and
      Security Information Systems, who may be reached at (202) 275-4649. Other major
      contributors are listed in appendix IV.

      Sincerely yours,




      Ralph V. Carlone
      Assistant Comptroller General
Executive Summary


Purpose          to develop and deploy the first of a multiple phase Strategic Defense
                 System designed to shield the United States from a massive Soviet
                 nuclear ballistic missile attack. At the request of the Chairman, Legisla-
                 tion and National Security Subcommittee, House Committee on Govern-
                 ment Operations, GAO reviewed the Strategic Defense Initiative
                 Organization’s (SDIo's) test and evaluation plans. These plans are to sup-
                 port a 1993 executive decision by the President on deploying Phase I
                 and a 1994-1995 acquisition decision by the Department of Defense to
                 enter full-scale development. Specifically, GAO was asked to determine

             l   what test and evaluation requirements were established for making
                 development and deployment decisions for the first phase, and

             . whether SDIO'S approach will meet those requirements and support a
                 decision on deployment.


                 The Strategic Defense System is being designed to destroy thousands of
Background       incoming ballistic missiles and warheads. This will require detecting the
                 missiles and warheads, discriminating them from hundreds of thousands
                 of decoys, precisely tracking the missiles and warheads, and then
                 destroying them-all within about 35 minutes or less, and despite the
                 enemy’s concerted effort to defeat the system. The Strategic Defense
                 System is, by far, the most complex, technologically challenging system
                 ever attempted. As currently envisioned, a sophisticated communica-
                 tions system will link together hundreds, or possibly thousands, of com-
                 puter-operated components in space and on the ground. The system will
                 consist of several subsystems (elements), which include surveillance
                 satellites, space- and ground-based weapons, and ground-based subsys-
                 tems to command and control the satellites and weapons.

                 The Strategic Defense System is being developed in several phases.
                 Phase I will be built using existing technologies; later phases would use
                 more exotic technologies, such as lasers and particle beams. On Sep-
                 tember 17, 1987, the Secretary of Defense directed that Phase I enter
                 the concept demonstration and validation stage. This stage is important
                 because enough information must be developed to show that the system
                 is feasible before a decision is made to enter full-scale development.
                 Because Phase I cannot be demonstrated outside of an actual ballistic
                 missile attack, much of this information can be generated only through
                 computer simulation and modeling.



                 Page 2                 GAO/IMTEG9OH)Il   SD1 Needs Stable Design and Adequate   Testing
                   Executive   Summary




                   In 1988, SD10 formally adopted a design (architecture) for Phase I. How-
                   ever, in January 1990 SD10 changed the design dramatically by incorpo-
                   rating a new space-based weapon, called Brilliant Pebbles. The idea
                   behind Brilliant Pebbles is that thousands of small, relatively autono-
                   mous weapons would be deployed to intercept Soviet intercontinental
                   ballistic missiles.


                   The President is scheduled to make a decision in 1993 on deploying
Results in Brief   Phase I. His options include deploying, delaying, or cancelling Phase I.
                   Public Law 99-145 states that a Strategic Defense System may not be
                   deployed in whole or in part unless the President certifies to the Con-
                   gress that the system can fulfill its mission. SD10 may be able to justify a
                   decision to delay or cancel Phase I; however, SDIO will not be able to
                   support an informed decision to deploy it. To do so requires a stable
                   design, sufficient testing and evaluation, and according to SDIO, minimum
                   funding levels. However, it is highly unlikely any of these conditions
                   will be met.

                   The addition of Brilliant Pebbles in January 1990 has reduced, changed,
                   or eliminated the need for some of the space-based elements in the 1988
                   design. At this point, it is uncertain exactly what Phase I will consist of
                   in terms of elements or what functions they will perform. Additionally,
                   while SDIO initially required real-time integrated system-level testing
                   prior to a presidential deployment decision and a full-scale development
                   decision, SD10 officials now state that such detailed tests will not be con-
                   ducted prior to either decision. Furthermore, according to SDIO, because
                   of fiscal year 1990 funding cuts, the full-scale development decision has
                   been delayed until 1994-1995, and research efforts have been scaled
                   back, further reducing the amount of information that will be available
                   for a presidential deployment decision. SD10 officials believe they will be
                   able to support a presidential deployment decision with less information
                   than was originally desired, but state that this will be done at increased
                   risk.

                   Phase I has not received the scrutiny and oversight that Defense ini-
                   tially envisioned. The Defense Acquisition Board did not conduct its
                   scheduled 1989 review, nor has it assessed how Brilliant Pebbles affects
                   the design and test and evaluation requirements. The lack of effective
                   agency oversight has contributed to the failure of other automated
                   weapons systems, none of which matches the scale and technological
                   complexity of Phase I. Consequently, any executive decision in 1993 to
                   deploy Phase I would be premature and fraught with high risk.


                   Page 3                 GAO/IMTEG90-61   SD1 Needs Stable Design and Adequate   Testing
                           Executive   Summary




Principal Findings

SD1Design and Test Plans   Several problems are hampering SDIO’S design and test plans. First, Bril-
                           liant Pebbles radically changed the design and sent the program into a
in State of Flux           state of flux. SDIO has not yet solidified the role of Brilliant Pebbles or
                           what elements will be in the final design. This causes problems because
                           without a stable design detailed Strategic Defense System requirements
                           cannot be determined. Setting requirements is especially important for
                           Phase I where weapon, sensor, and communication systems are interde-
                           pendent. Changing a requirement for a weapon system could signifi-
                           cantly affect sensor and communications systems as well. Further, an
                           unstable design increases the probability that system requirements will
                           not be adequately determined and sufficient testing will not be per-
                           formed to ensure that the system works. Nevertheless, Defense has
                           requested $265 million for full-scale development of one of the sensor
                           elements, the Boost Surveillance and Tracking System, for fiscal year
                           1991 even though Brilliant Pebbles may eliminate or reduce the require-
                           ments for this system.

                           Second, SD10 does not plan to conduct integrated system-level testing by
                           the scheduled 1993 presidential decision. SDro-funded studies and test
                           plans have cited the importance of running real-time integrated system-
                           level tests, which combine computer simulation and actual hardware
                           and software prototypes with human intervention. Furthermore, most
                           of the system tests and evaluations so far have been based on the pre-
                           Brilliant Pebbles design. Consequently, much of the testing and analyses
                           may no longer be relevant and will have to be repeated.

                           Also, according to SDIO, its ability to support an informed deployment
                           decision depended on a minimum funding level of $4.6 billion for fiscal
                           year 1990. However, funding was cut by 20 percent According to SDIO,
                           such a cut will reduce the amount of research performed, delay SDIO’S
                           full-scale development decision by 2 years, and reduce the amount of
                           information available t,o the President. Nevertheless, SDIO officials have
                           stated that they will be able to support a presidential decision on the
                           program even though less information will be available.




                           Page 4                GAO/IMTEGSOBl   SD1 Needs Stable Design and Adequate   Testing
                    Executive   Summary




Original Program    Because of the complexity, uniqueness, and cost of Phase I, the Defense
                    Acquisition Board initially was to follow a highly structured oversight
Oversight Reduced   system of annual reviews looking at cost growth, changing require-
                    ments, and test and evaluation results. However, the October 1989 pro-
                    gram review scheduled by the Defense Acquisition Board did not occur.
                    According to Defense officials, the review was cancelled because the
                    fiscal year 1990 budget had not yet been finalized and because Defense
                    was reassessing the amount of oversight needed for the Strategic
                    Defense System. Even though a Defense Acquisition Board review is
                    scheduled for June 15, 1990, major changes to the Phase I design and
                    significant reductions to test and evaluation requirements have already
                    occurred without the Board’s review and evaluation.




the Secretary of    has stabilized the design and has demonstrated the effectiveness of the
Defense             system through integrated system-level tests in real time, using system
                    hardware and software prototypes with human intervention. Further,
                    the Secretary should ensure that required oversight by the Defense
                    Acquisition Board be followed. The Board should more closely monitor
                    system design, development, testing, and evaluation. GAO is making
                    other recommendations in chapter 4.


                         recommends that the Congress not fund full-scale development for
Recommendation to   GAO
                    any element of Phase I, including $266 million for the Soost Surveillance
the Congress        and Tracking System in fiscal year 1991, until SDIO has stabilized the
                    design and has demonstrated the effectiveness of the system through
                    integrated system-level tests in real time, using system hardware and
                    software prototypes with human intervention.


                    As requested by the Chairman’s office, GAO did not obtain official
Agency Comments     agency comments on a draft of this report. However, GAO discussed the
                    information contained in this report with SD10 officials and has incorpo-
                    rated their comments where appropriate.




                    Page 5                GAO/IlHTEC908I   SD1 Needs Stable Design and Adequate   Testing
contents


Executive Summary                                                                                             2

Chapter 1                                                                                                 8
Introduction            Genesis/Evolution of the SD1 Program                                              8
                        Objective of a Strategic Defense System                                           9
                        Description of Phase I                                                           11
                        Importance of Concept Demonstration and Validation and                           13
                            Defense Acquisition Oversight
                        Constraints Affecting SDIO’s Approach to Concept                                  14
                            Demonstration and Validation
                        Objectives, Scope, and Methodology                                               15

Chapter 2                                                                                                18
Unstable Architecture   Brilliant Pebbles Sends Phase I and Strategic Defense                            18
                              System Into State of Flux
and Test Plans Will     SD10 Will Not Perform Integrated System-Level Testing                            22
Not Support a                 Prior to 1993
                        SD10 Asserts Funding Cuts Will Reduce Information                                24
Presidential                  Needed for an Informed Decision in 1993
Deployment Decision
in 1993
Chapter 3                                                                                                25
Program Management      Oversight for Phase I More Stringent Than Typical                                25
                            Acquisition
Oversight Has Been      1989 Annual DAB Review Did Not Occur; Decisions Have                             26
Reduced                     Been Made Without DAB Review and Approval
                        Congress Cuts Funding for Operational Testing and                                27
                            Evaluation Until Architecture Is Solidified

Chapter 4                                                                                                29
Conclusions and         Recommendations to the Secretary of Defense                                      31
                        Recommendation to the Congress                                                   31
Recommendations
Appendixes              Appendix I: Department of Defense Major Systems                                  32
                            Acquisition Stages
                        Appendix II: Strategic Defense System Elements of                                34
                            Phase I



                        Page 6               GAO/IMTEGBOSl   SD1 Needa Stable Design and Adequate   Testing
          Contents




          Appendix III: Test Beds Supporting Demonstration and                               35
              Validation of the Strategic Defense System
          Appendix IV: Major Contributors to This Report                                     36

Figures   Figure 1.1: Phases of a Ballistic Missile Attack                                   10
          Figure 1.2: Phase I Architecture Incorporating Brilliant                           12
               Pebbles




          Abbreviations

          DAB        Defense Acquisition Board
          GAO        General Accounting Office
          IMTEC      Information Management And Technology Division
          SD1        Strategic Defense Initiative
          SD10       Strategic Defense Initiative Organization


          Page 7                 GAO/IMTEZGgoBl   SD1 Neede Stable Dee&n and Adequate   Testing
Chapter 1

Introduction


                       The Strategic Defense Initiative (SDI) is a program to develop and deploy
                       a Strategic Defense System to shield the United States against a Soviet
                       nuclear ballistic missile attack. Research and development for the SDI
                       program began in 1984, and in 1987 a Strategic Defense System was
                       approved for acquisition. This system is to be developed in multiple
                       phases. Phase I, the subject of this report, is in the concept demonstra-
                       tion and validation stage of the Department of Defense’s acquisition
                       process.’

                       The Strategic Defense System will be an expensive undertaking. The
                       Strategic Defense Initiative Organization (SDIO) cost estimates range
                       from $69.6 to $87 billion for completion of Phase I development and
                       deployment-with     an additional $10.1 billion for Phase I operation and
                       support until deployment is completed* While the cost of follow-on
                       phases is unknown, SDIO estimates that, in addition to the Phase I costs,
                       approximately $26.4 billion will be needed for fiscal years 1990 through
                       1994 to pursue non-Phase 1 research and development. GAO has recently
                       reported to the Congress that the above figures are, at best, optimistic.’


                       On March 23, 1983, President Reagan called for a comprehensive scien-
Genesis/Evolution of   tific research effort to develop a system that would render nuclear bal-
the SD1 Program        listic missiles impotent and obsolete. In January 1984, SD1 was
                       established as a research and technology development program, and in
                       April 1984, Defense formally chartered the Strategic Defense Initiative
                       Organization as the agency responsible for managing Defense’s efforts.

                       After several years of research, SDIO decided in 1987 to develop and
                       deploy the system in phases. SDIO felt that working in phases would
                       allow SDIO to be prepared for an early deployment, if such a decision
                       were made, and to respond to changing threats. Phase I of the system is
                       to be based on currently available technologies, while later phases are to
                       incorporate technologies that are expected to be available in the future,
                       such as neutral particle beams and lasers. Phase I is not intended to be a
                       “total defense”, but is being designed to destroy a certain percentage of
                       some intercontinental and sea-launched ballistic missiles (the exact per-
                       centages are classified). Creating a full strategic defense capability will

                       ‘The Department of Defense’s mdor system acquisition process is supposed to provide a single
                       approach to designing, developing, implementing, and maintaining major weapons systems. (The five
                       stages of the acquisition process are discussed in appendix 1.)




                       Page 9                       GAO/MTFE-SO-61      SD1 Needs Stable Design and Adequate     Testing
                    Chapter 1
                    Introduction




                    require the deployment of the follow-on phases. Therefore, a critical
                    aspect of the first phase is the capability to evolve and support future
                    phases. Further, the system is not being designed to destroy sea-and air-
                    launched cruise missiles.:’ Other costly systems independent of the SDI
                    program will be needed to defend against these weapons.

                    From the beginning, SDIOhas had one overall goal-to conduct a vig-
                    orous research and technology program that would provide the basis for
                    an informed decision regarding the deployment of a Strategic Defense
                    System. The current Phase I program is intended to support an execu-
                    tive decision on deployment by the President in 1993 and an acquisition
                    decision on full-scale development by Defense in 1994 or 1995.
                    According to SDIO test and evaluation officials, the President will have a
                    range of options including deploying, delaying, or cancelling Phase I.
                    However, if the President decides to deploy the system, Phase I develop-
                    ment will not be consistent with Defense’s prudent acquisition policies
                    specifying that deployment decisions be made after full-scale
                    development.


                    The Strategic Defense System is based on a layered defense concept; the
Objective of a      system is supposed to intercept a missile or a deployed warhead, as it
Strategic Defense   travels toward its target. The basic concept is that separate sensor,
System              weapon, and command and control systems (SDIO refers to these sepa-
                    rate systems as elements) would be in space and on the ground. The
                    system would be tied together by a complex communications network
                    and a sophisticated computer-based command and control element.
                    During an attack, the system would have to function in an extremely
                    hostile environment, including nuclear explosions and a concerted effort
                    by the enemy to destroy the Strategic Defense System.

                    The threat the system would be facing, along with the environment in
                    which it would be operating, creates a unique and demanding challenge.
                    First, the space-based elements of the Strategic Defense System would
                    have to be able to detect and begin tracking thousands of missiles almost
                    immediately after launch, and then intercept and destroy some of them,
                    Those missiles that are not destroyed would release warheads, along
                    with decoys intended to confuse the system, forcing the system to track
                    hundreds of thousands of objects. Also at this point, space-based ele-
                    ments of the system would have to distinguish the warheads from the

                    %kuise missiles are guided missiles that have terrain-seeking radar and fly at moderate speed in low
                    altitude,



                    Page 9                        GAO/IMTEG904il       SD1 Needs Stable Design and Adequate      Testing
                                              Chapter 1
                                              Introduction




                                              decoys, and intercept and destroy some of the warheads. The system
                                              would need to continue discriminating and tracking the surviving war-
                                              heads so that the ground-based elements could intercept more warheads
                                              before they hit the United States. These functions would have to be
                                              tightly coordinated and performed in less than 35 minutes-all this with
                                              nuclear warheads exploding, anti-satellite weapons attacking the
                                              system, and the enemy trying to disrupt communications and computer
                                              operations. Figure 1.1 describes the phases of a ballistic missile attack.

Figure 1.1: Phases of a Ballistic Missile Attack


                                                                  Warheads and
                                                                     Decoys
                                                                                         0
                                                                             V
                                        Midcourse                             QVOL
                                     About 20 minutes                      0p 8
                                                                           e@V                        Post-Boost
                                                                                                    IO’s of seconds
      Terminal                                                        CD                          to 10’s of minutes
      30 to 60
      seconds                                                                                                                         Boost
                                                                                                                   @a             Several 1O’s
                                                                                                                             7     to loo’sof
                                                                                                                                    seconds




                                              Source. Adapted from SDI Technology Survivability   and Software, Offlce of Technology Assessment.
                                              May 1988




                                              Page 10                        GAO/KMTEG9041        SD1 Needs Stable Design and Adequate       Testing
                         Chapter 1
                         Introduction




                         In 1988, a Phase I design, made up of seven elements, was approved by
Description of Phase I   the Defense Acquisition Board. The design consisted of two space-based
                         sensors (Boost Surveillance and Tracking System and Space Surveillance
                         and Tracking System); a space-based weapon (Space-Based Interceptor);
                         two ground-based sensors (Ground Surveillance and Tracking System
                         and Ground-Based Radar); and a ground-based weapon (Ground-Based
                         Interceptor), all managed by a command and control system (Command
                         Center Element). (Appendix II describes each element.) The six surveil-
                         lance and weapon elements would be highly interdependent and rely
                         heavily on the command and control element, along with a complex com-
                         munications system. The individual elements would work together as an
                         integrated system to detect, track, discriminate, and destroy ballistic
                         missiles and their warheads. At the heart of the system is a large, dis-
                         tributed, real-time computer software system which, by some estimates,
                         could have 40-100 million lines of code. The complexity and risk of this
                         undertaking cannot be overstated. We have reported repeatedly on
                         Defense systems that were far less complex, contained far fewer lines of
                         code, but have yet to perform as intended.’

                         In January 1990, SDIO decided to include a new weapon concept, called
                         Brilliant Pebbles, in Phase I. Brilliant Pebbles involves several thousand
                         individual interceptors orbiting the earth in order to detect and destroy
                         a target by smashing into it at high speed. Brilliant Pebbles is supposed
                         to improve system survivability and reduce costs by dispersing
                         thousands of space-based interceptors that are smaller and more auton-
                         omous than the Space-Based Interceptor. The Space-Based Interceptor
                         would house a number of interceptors, but unlike Brilliant Pebbles, it
                         would rely on other satellites for tracking, targeting, and communica-
                         tions. By making interceptors autonomous, they would no longer need to
                         rely on other satellites to perform these functions. Further, because each
                         interceptor would work independently, the system’s survivability would
                         be increased, in principle, because the loss of any one interceptor would
                         not greatly affect the system’s overall effectiveness. Figure 1.2 shows
                         Phase I with the six elements and Brilliant Pebbles.




                         Page 11               GAO/IMTEG9O-61   SD1 Needs Stable Desigu and Adequate   Testing
                                             Chapter 1
                                             Introduction




Fiaure 1.2: Phase I Architecture Incorporating Brilliant Pebbles




                                              BSTS is Boost Surveillance     and Tracking System
                                              GBI is Ground-Based     Interceptor
                                              GBR is Ground-Based      Radar
                                              GSTS is Ground $urveillance       and Tracking System
                                              SSTS is Space Surveillance      Tracking System
                                              RVs are Reentry Vehicles (nuclear warheads)




                                             SD10  funded three Brilliant Pebbles research and development studies
                                             during 1939 and 1990. Studies completed by the Defense Science Board
                                             and the JASONs-a group of scientists who periodically provide teeh-
                                             nical support to the Department of Defense-determined      that the Bril-
                                             liant Pebbles concept was technically feasible. Along with these two
                                             studies, Defense issued a Space-Based Architecture Study which
                                             reviewed the four space-based elements of Phase I-Space Surveillance
                                             and Tracking System, Boost Surveillance and Tracking System, the
                                             Space-Based Interceptor, and Brilliant Pebbles-and defined and justi-
                                             fied a recommended space-based architecture. The Space-Based Archi-
                                             tecture Study recommended that research continue on both Brilliant


                                             Page 12                       GAO/IMTEC-90-61     SD1 Needs Stable Design and Adequate   Testing
                        Chapter 1
                        Introduction




                        Pebbles and the Space-Based Interceptor and that modified versions of
                        the Boost Surveillance and Tracking System and the Space Surveillance
                        and Tracking System sensors be included.

                        SDIO has decided to eliminate the Space-Based Interceptor from Phase I,
                        claiming that Brilliant Pebbles will cut costs by 20 percent, that is, from
                        $87 to $69.6 billion. However, it is unclear whether the costs for all
                        Phase I requirements have been incorporated into the $69.6 billion, Fur-
                        ther, neither figure includes the costs for operating and maintaining the
                        system.


                        On September 17, 1987, the Secretary of Defense directed that Phase I
Importance of Concept   of the Strategic Defense System enter the concept demonstration and
Demonstration and       validation stage of Defense’s major systems acquisition process. SDIO’S
Validation and          basic acquisition strategy is to complete Phase I while researching later
                        phases of a Strategic Defense System. After successfully completing con-
Defense Acquisition     cept demonstration and validation, Phase I could enter full-scale
Oversight               development.

                        According to the Defense Systems Management College, concept demon-
                        stration and validation may be the most critical of all acquisition stages.
                        The primary purpose of this stage is to validate the feasibility of Stra-
                        tegic Defense System concepts and technologies, and show that the risk
                        is low enough to enter full-scale development. According to the College,
                        the Secretary of Defense’s decision to place a major system in full-scale
                        development is extremely important. During full-scale development, the
                        system is built, tested, and ready for full-rate production. Not only will
                        development consume enormous resources, but major systems, at this
                        point, take on a life of their own and are seldom cancelled.

                        To help Defense gain sufficient design and development information,
                        major system acquisitions are reviewed by the Defense Acquisition
                        Board (DAB). The purpose of the DAB, which is chaired by the Under Sec-
                        retary of Defense for Acquisition, is to oversee major Defense acquisi-
                        tions, The DAB reviews each acquisition stage to ensure that every
                        program is ready to proceed into more advanced stages of development
                        or production. Each stage of the acquisition process, as well as each
                        milestone decision, is to be supported by test and evaluation. The pur-
                        pose of test and evaluation is to help ensure the timely development,
                        production, and fielding of systems that meet users’ requirements and




                        Page 13               GAO/IMTEC-9041   SD1 Needs Stable Deslgu and Adequate   Testing
                        Chapter 1
                        Introduction




                        perform as intended. The DABrelies on information from test and evalu-
                        ation to determine whether a system is ready to enter full-scale develop-
                        ment. The DAB reviews also ensure that plans for later stages follow
                        sound acquisition management practices.

                        The complexity, cost, and uniqueness of the Strategic Defense System
                        has prompted the need for a more detailed program review process. For
                        example, the DAB developed a guidance document called the Imple-
                        mentor, which provides a framework for additional oversight. Although
                        formal DAB reviews are usually only required at major milestones, the
                        Implementor recommends yearly DAB reviews. Further, SDIO must submit
                        additional documentation to the DAB to ensure program goals are being
                        met.


                        Constraints exist which affect how the Strategic Defense System can be
Constraints Affecting   demonstrated. Although the 1972 Anti-Ballistic Missile Treaty permits
SDIO’s Approach to      research, it limits the development, testing, and deployment of different
Concept                 types of strategic defense systems and components. According to SDIO'S
                        1989 report to the Congress on SDI compliance with the treaty, research
Demonstration and       includes conceptual design and laboratory testing. Development occurs
Validation              after research but precedes full-scale testing of systems and components
                        designed for actual deployment. The development, testing, and deploy-
                        ment of launchers, interceptors, and radars are restricted. Also, the
                        extent to which the system can demonstrate its ability to interact in a
                        nuclear environment is constrained by limits on nuclear testing.

                        Because the Strategic Defense System cannot be tested in its operational
                        environment, many system capabilities must be demonstrated through
                        computer modeling and simulation. For example, in simulations,
                        software models would mimic the behavior of sensors and weapons and
                        be used in place of the actual elements to evaluate system performance.
                        A ballistic missile attack from launch to impact must also be simulated
                        in software to prompt the element models to respond. Eventually, some
                        of these tests would involve actual prototypes5 of weapons and sensor
                        hardware and software interacting in the simulation. This type of inte-
                        grated system-level testing would be used to evaluate the performance
                        of elements within the context of the entire Strategic Defense System.
                        Thus, the ability to simulate the interaction of the systems’ hundreds of

                        “A prototype is an original or model on which a later item is formed or based. A prototype is usually
                        built during the concept demonstration and validation stage and tested prior to the full-scale develop-
                        ment decision.



                        Page 14                        GAO/lMTEX-9061       SD1 Needs Stable Design    and   Adequate   Testing
                         Chapter 1
                         Introduction




                         computers with their many millions of lines of software code becomes of
                         paramount importance in demonstrating whether the system can per-
                         form its mission.

                         Early in the SDI program, SD10 recognized the need for integrated system-
                         level testing prior to a full-scale development decision. Defense policies
                         state that models or simulations can support a full-scale development
                         decision and that prototyping is one of the most powerful tools available
                         for determining system feasibility and capability. Consequently, SDIO ini-
                         tially planned to build a simulation framework to conduct tests using
                         software and hardware prototypes. SD10 studies and contractor docu-
                         ments have supported the need for such test capabilities before a full-
                         scale development decision. However, because SDIO believed building
                         such a simulation framework would be very complex, it explored other
                         approaches for demonstrating and validating the Phase I system.

                         SDIO decided instead to use multiple test beds.l; Under this approach,
                         each test bed would address critical strategic defense issues such as
                         system performance, command and control, communications, and
                         sensing and tracking. The system performance test bed, known as the
                         System Simulator, would be at the core of this test environment. The
                         System Simulator, using computer models of individual elements, would
                         perform end-to-end, system-level tests in which all aspects of Phase I’s
                         performance would be evaluated while under a simulated ballistic mis-
                         sile attack. Although the System Simulator, or some other simulation
                         framework, might evolve to include real-time integrated hardware-,
                         software-, and human-in-the-loop capabilities, SDIO has no plans to
                         accomplish this prior to either the presidential decision on deployment
                         or the full-scale development decision.


                         On June 5, 1989, the Chairman, Legislation and National Security Sub-
Objectives, Scope, and   committee, House Committee on Government Operations, requested that
Methodology              we assess the Strategic Defense Initiative Organization’s program activi-
                         ties, test and evaluation requirements, and decisions on the development
                         and deployment of Phase I of the Strategic Defense System. Specifically,
                         the objectives of our review were to determine (1) what test and evalua-
                         tion requirements were established for making development and deploy-
                         ment decisions for the first phase of the Strategic Defense System and


                         “Test beds are sites, facilities, or activities used for testing models or prototypes. SDIO refers to its
                         network of geographically dispersed test facilities as the National Test Bed.



                         Page 15                          GAO/IMTEGSOBI         SD1 Needs Stable Design and Adequate         Testing
    Chapter 1
    Introduction




l



    (2) whether the current program approach will meet those requirements
    and support a decision on whether to deploy.

    To address our objectives, we interviewed a wide range of officials
    involved in the Strategic Defense Initiative. SDIO representatives
    included: (1) the Phase I System Engineer, as well as other Engineering
    Office officials, (2) test and evaluation officials, (3) Brilliant Pebbles
    Program Office officials, and (4) National Test Bed Program Office offi-
    cials. We discussed technical issues with members of the Phase One
    Engineering Team, which acts as a consultant to SDIO. Team members are
    experts in ballistic missile defense and include physicists, software and
    electrical engineers, and simulation analysts. We also met with officials
    from Martin Marietta, the prime contractor for the National Test Bed,
    and officials from General Electric Company, the system engineer and
    integration contractor for the Strategic Defense System. General Electric
    has the responsibility for defining Phase I demonstration and validation
    requirements, including those for test and evaluation. We met wjth
    Army and Air Force officials involved in major SD1 research and devel-
    opment as well as Defense Acquisition Board officials. Additionally, we
    consulted with ballistic missile defense experts at Sandia and Los
    Alamos National Laboratories.

    Our approach was to obtain a thorough understanding of the underlying
    engineering concepts for the Strategic Defense System by (1) obtaining
    and analyzing key system design and requirements documents, (2) inter-
    viewing and consulting with experts from the ballistic missile defense
    community, and (3) visiting and assessing research and development
    sites. We reviewed pertinent management, technical, and contract docu-
    ments provided by SDIO, SDIO contractors, the Air Force, the Army, Los
    Alamos National Laboratory, and the Defense Science Board. We ana-
    lyzed and compared system documents such as the 1988 System
    Description Document, the 1990 System Description Document, the Bril-
    liant Pebbles System Description Document, and the Space-Based Archi-
    tecture Study. Furthermore, we analyzed and compared test and
    evaluation documents including the 1987 and 1989 Test and Evaluation
    Master Plan, the Phase I Integrated Test Plan, the System Simulator
    requirements document, the technical specifications and software for
    the current System Simulator, the Brilliant Pebbles Integrated Test Plan,
    and the Brilliant Pebbles Test and Evaluation Master Plan. Through our
    analyses we were able to identify the chronology of events that led to
    SDIO's current test and evaluation approach in support of decisions on
    deployment and full-scale development. We did not, however, validate
    test and evaluation results.


    Page 16               GAO/IMTEC-90-61   SD1 Needs Stable Design and Adequate   Testing
Chapter 1
Introduction




Our work was conducted at SD10 Headquarters, Washington, D.C.; the
National Test Bed Joint Program Office, Falcon Air Force Base, Colorado
Springs, Colorado; Electronic Systems Division, Boston, Massachusetts;
the Army Strategic Defense Command, Huntsville, Alabama; Sandia
National Laboratory, Albuquerque, New Mexico; Los Alamos National
Laboratory, Los Alamos, New Mexico; the Phase One Engineering Office,
Crystal City, Virginia; the Defense Acquisition Board, Washington, D.C.;
General Electric Corporation, Blue Bell, Pennsylvania; and Martin Mari-
etta Corporation, Colorado Springs, Colorado.

We discussed the facts in this report with SDIO officials and have incor-
porated their comments where appropriate. However, in accordance
with the requester’s wishes, we did not obtain official agency comments
on a draft of the report. We conducted our work from June 1989
through May 1990, in accordance with generally accepted government
auditing standards.




Page 17               GAO/IMTEC90-61   SD1 Needs Stable Design and Adequate   Testing
Unstable Architecture and Test Plans Will Not
Support a Presidential Deployment Decision
in 1993
                          The President is scheduled to make a decision in 1993 on deploying
                          Phase I of the Strategic Defense System.’ Public Law 99-145 states that
                          a Strategic Defense System may not be deployed in whole or in part
                          unless the President certifies to the Congress that the system can sur-
                          vive, meaning that it can maintain enough effectiveness during a war to
                          fulfill its mission. However, by 1993 SD10 will not have conducted inte-
                          grated system-level tests designed to demonstrate that the entire Phase I
                          system will work as planned, so it will not be able to support an
                          informed deployment decision by the President.

                          The recent inclusion of Brilliant Pebbles has fundamentally changed the
                          1988 Phase I architecture by potentially reducing or eliminating the
                          need for two sensors and one weapon component of the system. This
                          change has significantly destabilized the architecture, resulting in a dra-
                          matic restructuring of Phase I. According to SDIO test and evaluation
                          officials, the new Phase I architecture will not be solidified until 1991
                          thus decreasing the level of system testing that can be performed by
                          1993. Furthermore, according to SDIO’S Director, because of fiscal year
                          1990 funding cuts, research will be scaled back, reducing the informa-
                          tion available on which to make a deployment decision in 1993.


                          In January 1990, Brilliant Pebbles was formally incorporated into the
Brilliant Pebbles Sends   Phase I design. However, exactly what role Brilliant Pebbles will play in
Phase I and Strategic     the Phase I architecture and even what pieces of Phase I will be
Defense System Into       deployed is uncertain and consequently, has put the design of the Stra-
                          tegic Defense System into a state of flux. The result is a destabilized
State of Flux             architecture and a major restructuring of SDIO’S program.


Stable Integrated         The SDIO Phase I System Engineer has stated that his first goal in sup-
Architecture Needed for   port of the presidential deployment decision is to create “a complete and
                          coherent system design with definition of all elements and inter-element
System Development        interfaces.“’ It is important in systems development to have a stable,
                          integrated architecture before detailed system requirements can be
                          determined. A stable architecture provides the blueprint for design and
                          development of each element. An unstable architecture causes confusion
                          and increases the probability that system requirements and integrated

                          ‘SD10 is conducting the SD1program to support a decision by the President prior to the completion of
                          his current term. Much of the specific program guidance and direction, as well as the timing of the
                          presidential decision is classified.
                          “Interfaces are the internal and external communication paths within and outside of the system.



                          Page 18                       GAO/IMTEC-90-61      SD1 Needs Stable Design and Adequate     Testing
                       Chapter 2
                       Unstable Architecture   and Test Plans Will
                       Not Support a Presidential  Deployment
                       Decision in 1993




                       test objectives will not be met, and that the system will not perform as
                       intended.

                       The Strategic Defense System is a “system-of-systems” and accordingly,
                       the elements are all interdependent. A requirement change for one ele-
                       ment may affect all the others. For example, in the 1988 Phase I archi-
                       tecture, the Boost Surveillance and Tracking System and the Space
                       Surveillance and Tracking System had a requirement to combine their
                       tracking data to provide targeting information for the Space-Based
                       Interceptor. This combined data would also be transmitted to the
                       ground-based elements to help them identify targets. The interfaces
                       must be precisely defined to support such data sharing. If the Space Sur-
                       veillance and Tracking System’s mission changes, then all the elements
                       must reflect the change as well. Accordingly, to ensure integrated
                       system development, each element developer must be aware of all the
                       other elements and the respective interfaces when designing each
                       system.


Brilliant Pebbles      Major program restructuring is going on to incorporate Brilliant Pebbles
Destabilizes Phase I   and stabilize the architecture. Brilliant Pebbles is now the focal point of
                       SDIO’S efforts. Its effect on the Phase I architecture is far-reaching.
Architecture
                       As discussed in chapter 1, the 1988 Phase I architecture was made up of
                       seven highly integrated elements. Command and control relied on rapid
                       communication and data sharing among elements. For example, during a
                       battle, the Boost Surveillance and Tracking System would begin tracking
                       boosters and would be one of several systems to alert national decision-
                       makers. The Boost Surveillance and Tracking System would then
                       transfer the information to the Space Surveillance and Tracking System,
                       which would begin tracking and pass intercept information to the Space-
                       Based Interceptor. At the same time, battle information would be sent to
                       the ground-based weapons and sensors. The Ground Surveillance and
                       Tracking System and the Ground-Based Radar would continue tracking
                       and the Ground-Based Interceptor would destroy the warheads before
                       they reenter the earth’s atmosphere. Throughout the battle, command
                       and control information would be centralized at the ground-based Com-
                       mand Center Element.

                       The addition of Brilliant Pebbles fundamentally changed this highly-
                       integrated approach to strategic defense. Brilliant Pebbles provides for
                       several thousand self-contained interceptors that can detect and destroy
                       missiles independently of the other Phase I elements. As a consequence,


                       Page 19                      GA0/1MTEG90-61   SD1 Needs Stable Design and Adequate   Testing
                       Chapter 2
                       Unstable Architecture   and Test Plans Will
                       Not Support a Presidential  Deployment
                       Dfxision in 1993




                       interfaces, sensing and tracking capabilities, communications, and the
                       numbers of elements needed-both in terms of types and constellations
                       (numbers of satellites)-will    have to be reassessed. For example, the
                       Boost Surveillance and Tracking System may not be required as a Phase
                       I element, and Brilliant Pebbles will add thousands of independent
                       interceptors to the system. Furthermore, Brilliant Pebbles will eliminate
                       the need for the Space-Based Interceptor, could reduce tracking require-
                       ments placed on the Space Surveillance and Tracking System, and could
                       expand the mission requirements for the Ground Surveillance and
                       Tracking System. Unlike the Space-Based Interceptor, whose battle man-
                       agement functions were highly dependent on ground-based command
                       and control instructions, Brilliant Pebbles, after being turned on by the
                       commander, could be autonomous.


Phase I Architecture   Defense officials have stated that the elements that will make up the
                       Phase I architecture have not yet been solidified. However, several sce-
Remains Uncertain      narios have been given. An Office of the Secretary of Defense official
                       stated that a space-based system made up of the Boost Surveillance and
                       Tracking System and BrilIiant Pebbles could be initially deployed as the
                       Phase I architecture, follo;ved by the Phase I ground-based elements.
                       Another scenario is that the Boost Surveillance and Tracking System
                       and Brilliant Pebbles would be deployed with a terminal interceptor!
                       (the High Endoatmospheric Interceptor), which is not currently included
                       in the Phase I design.

                       In April 1990, the Director of SDIO told the Congress that because of Bril-
                       liant Pebbles potential for independent operation, the Phase I architec-
                       ture may not require the Boost Surveillance and Tracking System and
                       the Space Surveillance and Tracking System or may require less capa-
                       bility from them. However, in May 1990, SDIO officials stated that the
                       Phase I architecture will include Brilliant Pebbles and all the 1988 Phase
                       I architecture elements, with the exception of the Space-Based Inter-
                       ceptor. The differences among such scenarios and the uncertain archi-
                       tecture point to the instability of basic concepts underlying the Strategic
                       Defense System.

                       Unless the architecture is stabilized and the respective elements are
                       developed, designed, and tested as an integrated system, SD10 will not


                       “Terminal interceptors are intended to destroy warheads after they have entered the earth’s
                       atmosphere.



                       Page 20                       GAO/lMTEC-9041       SD1 Needs Stable Design and Adequate       Twting
                            Chapter 2
                            Unstable Architecture   and Test Plans Will
                            Not Support a Presidential  Deployment
                            Decision in 1993




                            know whether the system will work as planned. The process of intercon-
                            necting the Phase I elements so that together they can effectively
                            accomplish the strategic defense mission is called system integration.
                            Successful system integration should result in the design and delivery of
                            complete elements that will work in concert with the other Phase I ele-
                            ments. Two of the elements, however, are preceding the rest of the
                            system. Because of presidential guidance, SDIOhas expedited the
                            research and development of Brilliant Pebbles. Consequently, Brilliant
                            Pebbles has been placed in an accelerated acquisition program. The
                            Boost Surveillance and Tracking System, which may or may not be
                            needed if Brilliant Pebbles is successful, is approaching a full-scale
                            development decision. Defense has requested $265 million for full-scale
                            development of the Boost Surveillance and Tracking System in fiscal
                            year 1991. The Boost Surveillance and Tracking System is a multi-mis-
                            sioned satellite that supports other Defense programs. Defense docu-
                            ments show that SDIOwill provide $1.3 billion (or 99 percent) of the
                            system’s funding through fiscal year 1991. The Boost Surveillance and
                            Tracking System’s specific functions for the Strategic Defense System
                            are to detect missile launches, acquire and track boosters, and assess the
                            number of boosters and warheads that have been destroyed. If Brilliant
                            Pebbles and the Boost Surveillance and Tracking System are allowed to
                            advance before the requirements of the other elements are clearly
                            defined, significant integration problems and costs could emerge.


Phase I System Test Plans   SD10 test and evaluation officials claim that including Brilliant Pebbles in
and Results May Be Moot     Phase I has destabilized the architecture and that a firm design will not
                            be available before 1991. For example, they have stated that a Prelimi-
                            nary Systems Requirements Review that was scheduled for May 1990
                            will be deferred for at least a year. Furthermore, SDIO test and evalua-
                            tion officials are rewriting test plans to incorporate Brilliant Pebbles.
                            Because most of the test and evaluation efforts have been based on the
                             1988 Phase I architecture, much of the data and analysis may no longer
                            be relevant. For example, if, as mentioned above, the Space Surveillance
                            and Tracking System’s mission is changed and it no longer provides
                            tracking information, then all of the modeling and simulation done so far
                            is useless because all the interfaces and data paths have changed. Any
                            change to one element causes this rippling effect across the system.

                            Further, system-level tests and the resulting requirements                have not
                            included Brilliant Pebbles. Accordingly, system tests must                be rewritten
                            and actual simulations rerun to include them. Because the                 other Phase I
                            elements have not included Brilliant Pebbles in their tests,              analyses, and


                            Page 21                      GAO/IMTEC-90-61   SD1 Needs Stable Design and Adequate   Testing
                         Chapter 2
                         Unstable Architecture   and Test Plans Will
                         Not Support a Presidential  Deployment
                         Decision in 1993




                         development, element test plans and simulations must also be
                         reassessed.


                         snlo-funded studies and test plans cite the importance of conducting
SD10 Will Not Perform    integrated system-level tests in real time, using actual system hardware
Integrated System-       and software with human intervention. For example, the Advanced Sim-
Level Testing Prior to   ulation Framework Study recommended that component hardware and
                         software models at various geographic locations be tested and evaluated
1993                     to see how they work in real-time operations. Moreover, SDIO'S Inte-
                         grated Test Plan notes the importance of such tests before making a
                         decision on full-scale development. According to the plan, integrated
                         system testing minimizes costs by integrating and building upon existing
                         experiment assets, data, and results, and sharing a common test bed.
                         However, an integrated system-level test in real time using hardware
                         and software and a human-in-the-loop will not be demonstrated prior to
                         the planned 1993 decision on deployment. SDIO officials feel they will be
                         able to support a presidential deployment decision with less information
                         than was originally desired, but state that this will be done at increased
                         risk.


Value of Integrated      Integrated system-level testing will help confirm whether the individual
System-Level Testing     elements and the strategic defense commander can successfully interact
                         in real time. Actual hardware and software in the test environment will
                         provide significantly more information about system performance than
                         a model. Such testing would put system components in a realistic system
                         environment for detecting problems early during design and develop-
                         ment. SD10 officials decided that such testing will be performed in the
                         System Simulator (or some other simulation framework) and will pro-
                         vide the basis for system design studies and end-to-end validation of
                         overall system performance. Simply put, the System Simulator will
                         increase SDIO'S confidence that when the independently designed and
                         developed elements become operational, they will integrate effectively
                         into a Strategic Defense System.

                         In 1988 SD10 established the Stellar Task Force to address the question
                         of how best to evolve to such an integrated system test environment.
                         While the task force fully supported the use of the System Simulator for
                         system-level tests, it concluded that trying to start off using only one
                         test environment was too complex an undertaking. It recommended




                         Page 22                       GAO/IMTEC-90451   SD1 Needs Stable Design and Adequate   Testing
                            Chapter 2
                            Unstable Architecture and Test Plans Will
                            Not Support a Presidential Deployment
                            Decision in 1993




                            breaking the problem into pieces and developing several test beds4 By
                            using several test beds, the system could be broken into units, each
                            focusing on a different part of the system. For example, a communica-
                            tions test bed would enable SDIOto conduct a series of tests on all of the
                            elements’ abilities to effectively communicate.

                            The task force’s rationale for this approach was that detailed test
                            results from each test bed could be entered into the System Simulator’s
                            data bases for more accurate system-level testing. The elements will be
                            represented at the System Simulator by software models rather than
                            actual element hardware and software. SDIO does plan to use several test
                            beds and then evolve to integrated system-level testing at the System
                            Simulator, incorporating actual element hardware and software in a
                            real-time environment. However, integrated testing will not occur before
                            the President’s scheduled 1993 decision.


System-Level Test Results   A stable architecture with defined system and element requirements is
                            required before software models can be developed to validate each of
Based on Immature           the elements and the system as a whole. However, even prior to the
Element Models              inclusion of Brilliant Pebbles, the SD10 Phase I System Engineer had
                            stated that detailed system requirements for all Phase I elements will
                            probably not be set until 1994 or 1995. Therefore, element software
                            models cannot sufficiently demonstrate how the various elements will
                            work until detailed requirements are defined. As things stand, test
                            results available in 1993 will be based on limited software representa-
                            tions of the elements. For example, Air Force officials have stated that
                            they have not been provided detailed ballistic missile defense require-
                            ments for the Boost Surveillance and Tracking System. Without such
                            requirements, system developers cannot develop an accurate software
                            representation for system-level testing. Consequently, it is unclear
                            whether detailed requirements will be determined, a model developed,
                            system testing completed, and results analyzed in sufficient time to sup-
                            port the 1993 deployment decision. Therefore, not only will system-level
                            tests not be conducted in real time or be integrated with pieces of hard-
                            ware and software, but the tests will be based on simulations involving
                            immature, possibly inaccurate element models.




                            ‘These test beds are the Surveillance and Tracking Test Bed, the Communications Network Test Red,
                            the Pilot Command Center. the Gaming Test Bed, and the System Simulator, all of which are
                            described in appendix III.



                            Page 23                      GAO/IMTEC-90-61     SD1 Needs Stable Design and Adequate    Testing
                       Chapter 2
                       Unstable Architecture   and Test Plans Will
                       Not Support a Presidential  Deployment
                       Decision in 1993




                       According to SDIO, an informed executive decision on deployment is con-
SD10 Asserts Funding   tingent, in part, on minimum funding levels. At one time, this decision
Cuts Will Reduce       was to occur in the same time frame as Defense’s formal, full-scale
Information Needed     development decision. In most major acquisitions, a deployment decision
                       is made after full-scale development is completed and the system has
for an Informed        moved into the next stage, (See app. I.) According to SDIO'S Director, an
Decision in 1993       informed decision on deployment is contingent on funding levels of $4.6
                       billion for fiscal year 1990 and $33 billion over fiscal years 1990-1994.
                       He stated that any funding reduction would reduce confidence in the
                       deployment decision and would have increasingly serious consequences
                       for the SD1 program. For example, the Director stated that a 20 percent
                       funding cut would reduce the confidence in making a decision on deploy-
                       ment due to cutbacks in research. He further stated that emerging con-
                       cepts, especially Brilliant Pebbles, would not be fully explored. Hence,
                       the space-based architecture would not be completely defined, and ini-
                       tial system development and deployment schedules would be delayed at
                       least 2 years.

                       The Congress reduced SDIO'S fiscal year 1990 budget request by 20 per-
                       cent (from $4.6 billion to $3.6 billion) because of congressional concern
                       for overall fiscal constraints and SDIO'S major uncertainties for fiscal
                       year 1990. Consequently, SDIO initiated a major replanning and restruc-
                       turing strategy to identify program priorities and impacts, and SD10
                       delayed its full-scale development decision for 2 years. Many contracts
                       were reduced, delayed, or cancelled, resulting in scaled-back demonstra-
                       tion and validation activities that, according to SDIO, will provide less
                       information than was originally envisioned to support an informed 1993
                       deployment decision. For example, the identification of system-level
                       demonstration and validation requirements in support of system-level
                       tests will not be established until fiscal year 1992; and by 1993, testing
                       on command and control functions will not be as thorough as originally
                       planned. Further, because the new Phase I architecture will not be
                       defined until 1991, the elements will be less developed and their designs
                       less detailed, therefore, less able to support system-level testing. Finally,
                       the system’s communications network will not be tested for real-time
                       operations. Nevertheless, SDIO officials have stated that they will be able
                       to support a presidential deployment decision but with increased risk.




                       Page 24                       GAO/IMTEG9@61   SD1 Needs Stable Design and Adequate   Testing
Program Management Oversight Has
ISeenReduced

                        SDI program decisions have been made without the formal review and
                        approval of the Defense Acquisition Board (DAB), increasing the risk that
                        program objectives will not be achieved. Defense, recognizing the com-
                        plexity, uniqueness, and cost of the Strategic Defense System, required
                        additional DAB oversight and directed additional monitoring of SDIO'S
                        operational test and evaluation activities. Despite the more stringent
                        oversight requirements, the 1989 annual DAB review did not occur as
                        planned. Consequently, fundamental changes to the Phase I architecture
                        and reductions in test and evaluation requirements have occurred
                        without formal DAB review and oversight.


                        As discussed in chapter 1, the DAB oversees major Defense acquisition
Oversight for Phase I   programs. As part of its oversight responsibility, the DAB conducts
More Stringent Than     formal reviews at major program milestones to ensure that programs
Typical Acquisition     are ready to proceed into more advanced stages of development and
                        that proposed program plans for later stages follow sound acquisition
                        management practices, Among the issues typically considered in a DAB
                        milestone review are cost, requirements growth, schedule delays, threat
                        assessment, acquisition strategy, and test and evaluation plans and
                        results. As part of the Defense acquisition process, DAB reviews are to
                        help reduce the risks associated with major systems acquisitions.

                        Phase I, however, is not a typical acquisition. The complexity, cost, and
                        uniqueness of the Strategic Defense System has prompted the need for a
                        more stringent acquisition process. For example, in 1987 the DAB devel-
                        oped a guidance document called the Implementor which provides a
                        framework for additional oversight. Although formal acquisition
                        reviews by the DAB are usually only required at major milestones, the
                        Implementor expanded the DAB'S oversight responsibility to include
                        yearly reviews, and lists additional documentation SDIO must submit to
                        the DAR to ensure program goals are being met.

                        In addition to the increased DAB oversight, Defense also decided in 1987
                        to establish an oversight group under its Director for Operational Test
                        and Evaluation to monitor SDIO'S test and evaluation activities. Each
                        stage of the acquisition process, as well as each milestone decision, is to
                        be supported by test and evaluation The purpose of test and evaluation
                        is to help ensure the timely development, production, and fielding of
                        systems that meet the users’ requirements and perform as intended. The
                        DAB relies on information from test and evaluation activities to deter-
                        mine whether a system is ready to enter the next stage of development.



                        Page25                 GAO/IMTEC-90-61   SD1 Needs Stable Design and Adequate   Testing
                        Chapter 3
                        Program Management   Oversight   Has
                        Been Reduced




                        The Implementor the DAR imposed on the SD1program states that the
1989 Annual DAB         Phase I program baseline must be reviewed annually by the DAB. These
Review Did Not Occur;   annual reviews were to include cost, schedule, technical performance,
Decisions Have Been     and operational assessments. The annual DAB review scheduled for
                        October 1989 was delayed until December and later cancelled, According
Made Without DAB        to Defense officials, the review was cancelled because the fiscal year
Review and Approval     1990 budget had not yet been finalized, and because Defense was reas-
                        sessing the amount of oversight needed for the Strategic Defense System
                        acquisition. The review has been rescheduled for June 15, 1990. Fur-
                        ther, significant issues have not been formally reviewed by the DAR as to
                        their impact on the SD1program. These issues include the October 1989
                        Space-Based Architecture Study’s conclusions and recommendations and
                        the dramatic effect of incorporating Brilliant Pebbles into Phase I.

                        The Space-Based Architecture Study was initiated to review the feasi-
                        bility and consequences of incorporating Brilliant Pebbles into Phase I
                        with special emphasis on the space-based elements-the Space-Based
                        Interceptor, the Boost Surveillance and Tracking System, and the Space
                        Surveillance and Tracking System, The study concluded that Brilliant
                        Pebbles was feasible, but recommended that research continue on both
                        Brilliant Pebbles and the Space-Based Interceptor. Additionally, the
                        study concluded that if Brilliant Pebbles were incorporated, major
                        changes should be made to system requirements that could significantly
                        affect program cost and schedule. For example, the study concluded
                        that (1) the Boost Surveillance and Tracking System’s ballistic missile
                        defense requirements could be reduced; (2) the Space Surveillance and
                        Tracking System is an essential asset for Phase I but may need to be
                        modified; and (3) the Ground Surveillance and Tracking System’s capa-
                        bility will have to be increased. While SD10officials have considered the
                        study’s results, the DAB has not formally reviewed the study and deter-
                        mined its consequences on the program’s design, cost, and schedule.

                        IJnder Defense’s current approach, major changes to Phase I have
                        occurred without high-level Defense oversight and management review.
                        Although the study recommended that research continue on both space-
                        based weapons, SDIO decided to include Brilliant Pebbles in the Phase I
                        architecture. This fundamental change to the program baseline was
                        made without formal DAB review and oversight. As noted in chapter 1,
                        this addition will have far-reaching impacts that will ripple across the
                        Strategic Defense System; system interfaces, sensing capabilities, com-
                        munications, and element requirements will all change significantly.




                        Page 26                      GAO/IMTEX-!#I41   SD1 Needs Stable Design and Adequate   Testing
                       Chapter 3
                       Program Management   Oversight   Has
                       Recm Redur4




                       SDIO  has also reduced the testing requirements for both the System Simu-
                       lator and the Pilot Command Center without DAB review. SD10 initially
                       established stringent System Simulator requirements that would demon-
                       strate an end-to-end, real-time system test with hardware-, software-,
                       and human-in-the loop in support of a full-scale development decision.
                       However, the System Simulator is now only required to demonstrate an
                       end-to-end simulation using software representations of the system ele-
                       ments instead of actual pieces of hardware and actual system software
                       operating in real time. Further, the Pilot Command Center, SDI’S test bed
                       to demonstrate battle management and command, control, and commu-
                       nications functions in real time, was to include actual sensor and
                        weapon hardware-in-the-loop testing. Current test plans do not include
                        actual sensor and weapon interfaces. Since these significant reductions
                        in demonstration and validation requirements have not been subject to
                        DAB review, high-level Defense Department officials have not passed
                       judgment on whether this proposed approach will provide sufficient
                        information to make an informed deployment decision.


                       The Congress has shown long-standing interest in how major weapon
Congress Cuts          systems perform and the adequacy and timeliness of operational test
Funding for            and evaluation. As early as 197 1, the Congress enacted legislation
Operational Testing    requiring Defense to provide the Congress with data on operational test
                       and evaluation results of major weapon systems before committing
and Evaluation Until   major production dollars.
Architecture Is
Solidified             In 1983, Congress enacted additional legislation creating the Office of
                       the Director of Operational Test and Evaluation. Defense directed this
                       office to provide an independent assessment of system-level testing for
                       the Strategic Defense System. This office is to provide independent over-
                       sight, coordinate the military services’ planning and execution of opera-
                       tional tests, and objectively report on test results to Defense and
                       congressional decision-makers.

                       The Congress expects independent oversight and objective reporting of
                       operational test results before it is willing to commit to production deci-
                       sions. However, the Congress, in November 1989, expressed concern
                       about the constantly changing design of the Strategic Defense System
                       and, given these changes, questioned what value the Office of Opera-
                       tional Test and Evaluation could provide to the Strategic Defense




                       Page 27                     GAO/lMTEC-9041   SD1 Needs Stable Design and Adequate   Testing
Chapter 3
Program Management      Oversight   Has
Been   Reduced




System at that time. According to the 1990 Authorization                       Act
conferees,’

...the need for a Strategic Defense Initiative operational test and evaluation activity
is premature in view of the shifting SD1architectures, plans and priorities. Accord-
ingly, the conferees agree to reduce the operational test and evaluation authoriza-
tion by $12.415 million from the requested amount and direct the Department of
Defense not to obligate any appropriated funds for such an activity until it has been
formally authorized in law.

We agree that such testing is premature given the fluid and constantly
changing nature of the Strategic Defense System architecture. However,
once the architecture is set, early system-level operational test and eval-
uation will help ensure that the acquisition process, which includes
system-level test plans and methodologies, is not only monitored but
also assessed. Furthermore, such oversight will help ensure that com-
plex system-level test results, justifying a full-scale development deci-
sion, will be understood.




‘Authorizing Appropriations for Fiscal Year 1990 for Military Activities of the Department of
Defense, House of Representatives Report Iio. 101-331,10&t Congress, 1st Session (Conference
Report).



Page 28                       GAO/IMTEC4081       SD1 Needs Stable Design and Adequate     Testing
Chapter 4

Conclusions and Recommendations


              The Strategic Defense Initiative Organization is engaged in one of the
              most complex and technologically challenging efforts ever undertaken.
              Tens of billions of dollars will be needed to develop and deploy the first
              phase of a Strategic Defense System. The first phase is designed to pro-
              tect the United States from only a portion of a massive Soviet nuclear
              ballistic missile attack.

              The Phase I architecture of the Strategic Defense System remains in a
              state of flux, primarily because Brilliant Pebbles, a new space-based
              weapon, was added to the system. The importance of a stable architec-
              ture cannot be overstated. An architecture that is not well defined runs
              the risk that system requirements will not be met and that the system
              will not perform as intended. Unless the individual Strategic Defense
              System elements are designed, developed, and tested as an integrated
              system, SDIOwill not know whether the system will work as intended+

              Brilliant Pebbles has greatly changed the Phase I architecture and high-
              lights the instability of SDIO’S design for the Strategic Defense System.
              Instead of the highly integrated, interdependent architecture envisioned
              for Phase I, Brilliant Pebbles is to provide more autonomy and require
              less interdependence among elements. As a result, element interfaces,
              surveillance and tracking capabilities, communications, and the numbers
              of elements will change. According to the Phase I System Engineer, even
              before the addition of Brilliant Pebbles to the architecture, detailed
              system requirements for all Phase I elements would probably not be
              completely defined until 1994 or 1995. Nevertheless, Defense has
              requested $265 million for full-scale development of the Boost Surveil-
              lance and Tracking System in fiscal year 1991. While the system is being
              developed to meet other Defense missions, we believe that a decision to
              enter full-scale development for the Boost Surveillance and Tracking
              System before the Phase I architecture had been solidified and its bal-
              listic missile defense requirements defined would be premature. Such
              decisions in other Defense programs have led to significant cost
              increases, schedule delays, and performance shortfalls.

              Phase I system-level test and evaluation activities will be significantly
              affected as well. System-level testing and evaluation are critical to
              making decisions on the continued development and deployment of a
              Strategic Defense System. However, the majority of system test and
              evaluation efforts so far have been based on an earlier Phase I architec-
              ture that did not include Brilliant Pebbles. Consequently, much of the
              current test data and analyses will not be applicable to the new architec-
              ture, and many of the tests will likely have to be redesigned and


              Page 29                GAO/IMTEC-90-61   SD1 Needs Stable Design and Adequate   Testing
Chapter 4
Conclusions   and Recommendations




repeated. Even though initial SDIO test plans required that real-time inte-
grated system-level tests be performed prior to a full-scale development
decision and a presidential deployment decision, SDIO officials now claim
such tests will not be conducted prior to either decision. We believe
SDIO’S initial plans could significantly reduce the risks involved in
making critical development and deployment decisions for such a costly
and technically complex system.

The President is scheduled to make a decision in 1993 on deploying
Phase I of the Strategic Defense System. The President will have a range
of options including deploying, delaying, or cancelling the Phase I
system. Public Law 99-145 states that the system may not be deployed
unless the President determines that the system is sufficiently capable
of fulfilling its mission. Given that the architecture continues to change,
the program is falling behind schedule, and the amount of test and eval-
uation data has been reduced, we do not believe that SDIO will be able to
give the President enough information to support a 1993 decision to
deploy Phase I.

We are not alone in raising concerns about the Strategic Defense
System’s changing architecture. The Congress has been troubled by the
constantly changing design of the Phase I system and has questioned the
value of operational test and evaluation oversight at this time. The Con-
gress was so concerned that for fiscal year 1990 it not only reduced SDI
program funding by 20 percent (from $4.6 billion to $3,6 billion), but
also cut funding for the Office of Operational Testing and Evaluation,
citing such activities as premature in view of the shifting Strategic
Defense System architecture, plans, and priorities.

The federal government has repeatedly found that major acquisitions
such as the Strategic Defense System are extremely difficult and com-
plex undertakings, fraught with high risks. Indeed, many have
floundered. A principal element needed to successfully design and
develop highly integrated systems is the commitment and oversight of
agency leaders. To provide that oversight they must have the informa-
tion they need-such as program cost, requirements growth, schedule
delays, acquisition strategy, and test and evaluation plans and results-
to effectively guide the acquisition effort.

High-level oversight of the SDI program, however, is not occurring.
The Defense Acquisition Board, which oversees major Defense acquisi-
tion programs, has not reviewed major changes to the Phase I architec-
ture and to the system’s test and evaluation requirements, The DAB’S


Page   30                     EAO/IMTEC-9081   SD1 Needs Stable Desigu and Adequate   Testing
                     Chapter 4
                     Conclusions   and Recommendations




                     1989 annual program review was cancelled. According to Defense offi-
                     cials, the review was cancelled because the fiscal year 1990 budget had
                     not yet been finalized, and because Defense was reassessing the amount
                     of oversight needed for the Strategic Defense System acquisition. We do
                     not believe these reasons justify cancelling scheduled oversight reviews
                     considering the dramatic, fundamental changes in the Phase I design
                     and the significant reductions in system-level test and evaluation
                     requirements. As a result, significant changes, such as the addition of
                     Brilliant Pebbles to the Phase I architecture and the results of the
                     October 1989 Space-Based Architecture Study, have not been subject to
                     formal review.


                     We recommend that the Secretary of Defense advise the President to
Recommendations to   defer a decision to deploy any element of the Phase I system until SDIO
the Secretary of     has stabilized the architecture and has demonstrated the effectiveness
Defense              of the system through integrated system-level tests in real time, using
                     system hardware and software prototypes with human intervention.
                     Additionally, the Secretary should ensure that required oversight by the
                     Defense Acquisition Board be followed. The Board should more closely
                     monitor system design, development, testing, and evaluation. Finally,
                     when the program’s architecture is stable, the Secretary should request
                     that the Congress reinstate funding for the Director of Operational Test
                     and Evaluation to provide an independent system-level assessment for
                     the Strategic Defense System.


                     We recommend that the Congress not fund full-scale development for
Recommendation to    any element of Phase I until SD10 has stabilized the architecture and has
the Congress         demonstrated the effectiveness of the system through integrated
                     system-level tests in real time, using hardware and software prototypes
                     with human intervention. This would include not providing $265 million
                     for the Boost Surveillance and Tracking System in fiscal year 1991.
                     However, if Defense needs the Boost Surveillance and Tracking System
                     for another mission, independent and separate from the Strategic
                     Defense System, it should be justified and funded to meet that mission
                     and should no longer be considered an element of the Strategic Defense
                     System.




                     Page 31                       GAO/IMTEG9OS1   SD1 Needs Stable Design and Adequate   Testing
Department of Defense Major Systems
Acquisition Stages

                         Mission area analysis and program initiation generally precede the five
                         Department of Defense acquisition stages. Defense components continu-
                         ally analyze their assigned mission areas to identify deficiencies (needs)
                         and determine if new systems or major upgrades to existing systems are
                         necessary. These analyses often result in recommendations to initiate
                         new acquisition programs through the validation of a need to correct the
                         deficiency. Once a need has been identified and validated and Defense
                         initiates an acquisition program, the program enters the concept formu-
                         lation stage.


                         In this stage, potential requirements and alternative approaches to sat-
Concept Formulation      isfy the need are identified and evaluated. Various types of analyses
Stage                    considering trade-offs among performance, life-cycle cost, and schedule
                         are conducted to select among possible concepts to satisfy the need.
                         Once a concept has been selected, it is presented to Defense for
                         approval.


                         In this stage, feasibility and desirability of the selected requirements
Demonstration and        and the system concept are further analyzed, generally using techniques
Validation Stage         like computer simulation, hardware prototyping, development test and
                         evaluation, operational test and evaluation, or a combination of test
                         methods. When the feasibility of the concept has been convincingly
                         demonstrated and validated, the program enters the full-scale engi-
                         neering and development stage.


                         In this stage, the system, including every item necessary for its logistic
Full-Scale               and operational support, is designed, fabricated, and tested. At the con-
Development Stage        clusion of this stage, the system is ready to be produced.


                         During this stage the proposed system is built and released to the user.
Full-Rate Production     At this point, the system becomes operational.
and Initial Deployment
Stage

                         This stage immediately follows deployment and extends until the
Operations Support       system is removed from Defense inventory. Two major Defense reviews
Stage                    are conducted in this stage. The first takes place 1 to 2 years after



                         Page 32               GAO/IMTEC-99-61   SD1 Needs Stable Design and Adequate   Testing
Appendix I
Department  of Defense   Major   Systems
Acquisition Stages




deployment to determine if operational readiness and support objectives                        /
are being achieved and maintained. The second, occurring 5 to 10 years                         i
after deployment, evaluates system capabilities and assesses whether
major upgrades are needed or if the system should be replaced.




Page 33                                                                                        8
                             GAO/IMTJX-9041   SD1 Needs Stable Design and Adequate   Testing   D
Appendix II

Strategic Defense System Elements of Phase I


                                               General
               System
               .--..  element                  function
                                              -I-...       Specific functions      ..--...__
               Boost Surveillance             Sensor       Detect missile launches; acquire and track
               and Tracking                                boosters, assess kills
               System                                                            ..----
               Space Surveillance             Sensor       Acquire and track warheads and satellites;
               and Tracking                                assess krlls
               System
               Ground                         Sensor       Track warheads and decoys, discriminate
               Surveillance and                            warheads from decoys; assess kills
               Tracking System         --..                                    - ..--..   ._~
               Ground-Based                   Sensor       Acquire and track warheads and decoys;
               Radar                                       discriminate warheads from decoys
               Space-Based                    Weapon       Destroy boosters and warheads
               Interceptor     ..--.
               Ground-Based                   Weapon       Destroy warheads
               Interceptor
                                                                                           -I .~
               Command Center                              Human decision-making, communications
                                                           and guidance for defense system




               Page 34                 GAO/IMTFX-90-61    SD1 Needs Stable Design and Adequate   Testing
Appendix III

Test Beds Supporting Demonstration and
Validation of the Strategic DefenseSystem

               The Surveillance and Tracking Test Bed will evaluate tracking and dis-
               crimination function performance for accuracy, computational require-
               ments, speed, robustness. It will also validate surveillance algorithms
               and integrated system performance.

               The Communications Network Test Bed will validate communications
               network operations and management and validate security of the Stra-
               tegic Defense System’s communications design.

               The Pilot Command Center will validate command structure, hardware-
               in-the-loop, and demonstrate proof of concept for mobile and fixed phys-
               ica1 configurations with selected software performance.

               The Gaming Test Bed will evaluate proposed command and control oper-
               ational concepts, build operator and user confidence in hardware-in-the-
               loop decision tools to support the deveIopment of the Pilot Command
               Center.

               The System Simulator will provide the basic vehicle for system design
               studies and end-to-end validation of overall system performance. It will
               also validate software and integration capabilities.




               Page 35               GAO/IMTEX-!W41   SD1 Needs Stable Design and Adequate   Testing
Appendix IV

Major Contributors to This Report


                       Michael T. Blair, Assistant Director
Information            Leonard J. Latham, Technical Advisor
Management and         Sally M. Obenski, Site Senior
                       Victoria L. Miller, Evaluator
Technology Division,   Teresa M. Schlee, Writer-Editor
Wa+shington, D.C.

                       Barry A. Tidwell, Evaluator-in-Charge
Denver Regional        Yvonne J. Rodriguez, Evaluator
Office




(510419)               Page 36               GAO/llWlECW61     SD1 Needs Stable Design and Adequate   Testing
                                                                 .

           .                                                           .-. -_
                                                                        . .




                             :
                                       ,-. .        .:
         ‘.:                 ..                      . . .      :.
         . .                                   ‘.
          ..   .             . .   .                         . .. ..
                       ‘.’

                                                                                                      ._ .       .- :             -_ ..l   -
                                                                                               .i   . . :.       -       . :‘.I...”
                                                                            ~                   :       Z’..            :-_ y:-
                                                                                                         ;...                             ..:
                                                                                                            ._ .           .. ..
                   .                                                    :         :.
                                                                                           .               .            : .::      ‘._. .
                                                                                . .    .
                                                                                                      .          .        -‘I.
                                                                 :                                  .          .               . .     . i




::   :
:
United States
General Accounting    Office
Washington,   D.C. 20548

Official   Business
Penalty    for Private   Use $MO