oversight

Defense Acquisition: Air Force Prematurely Recommends ADP Acquisitions

Published by the Government Accountability Office on 1990-03-29.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

Information    Management   and
Technology    Division

B-237417

March 29,199O

The Honorable John Conyers, Jr.
Chairman, Legislation
  and National Security Subcommittee
Committee on Government Operations
House of Representatives

The Honorable Frank Horton
Ranking Minority Member,
  Legislation and National Security Subcommittee
Committee on Government Operations
House of Representatives

In response to your request, this report discusses weaknesses in the Department of the Air
Force’s requirements-setting process for automated data processing system acquisitions. Air
Force regulations clearly state that system requirements should be defined and alternative
solutions evaluated before recommending the acquisition of a system. However, the Air
Force has not followed these regulations. This failure indicates a lack of commitment to the
process and a lack of appreciation for its criticality. As arranged with your office, unless you
publicly announce the contents of this report earlier, we plan no further distribution of it
until 30 days from the date of this letter. At that time, we will send copies to the Director,
Office of Management and Budget; the Secretary of Defense; and other interested parties.
This report was prepared under the direction of Samuel W. Bowlin, Director for Defense and
Security Information Systems, who may be reached at (202) 275-4649. Other major
contributors are listed in appendix III.




Ralph V. Carlone
Assistant Comptroller General
Ekeeutive Sumnwy


                   Many defense systems are too expensive, take too long to develop, and
Purpose            use obsolete technology. An increasingly important part of Department
                   of Defense procurement includes automated data processing (ADP) acqui-
                   sitions; Defense requested almost $9 billion for ADP resources in fiscal
                   year 1990. GAO reviews of Air Force actions to modernize ADP capability
                   for America’s tactical warning and attack assessment system found pro-
                   grams over budget and behind schedule, and systems that did not meet
                   performance requirements. This happened, in part, because the Air
                   Force established requirements that either could not be met or had to be
                   reduced to contain escalating program costs.

                   The Subcommittee on Legislation and National Security, House Commit-
                   tee on Government Operations, requested this report. Concerned that
                   inadequately defined requirements and a lack of evaluations of alterna-
                   tive solutions could have led to these acquisition problems, GAO evalu-
                   ated seven ongoing or proposed ADP or AuP-supported system
                   acquisitions to determine whether the Air Force (1) thoroughly defines
                   system requirements and (2) evaluates alternative solutions before rec-
                   ommending ADP system acquisitions.


                   Acquisition programs are initiated to satisfy specific mission needs or
Background         deficiencies that inhibit or prevent a military command from carrying
                   out a mission. The Air Force establishes system requirements to over-
                   come the deficiencies and satisfy the needs.

                   After the using Air Force command identifies a need, it proposes a solu-
                   tion that supplies preliminary requirements. The Air Force command
                   responsible for acquiring new systems evaluates the need and alterna-
                   tive solutions and estimates the likely cost and time necessary to acquire
                   the most feasible solution. After incorporating this information and data
                   from other commands, the using command recommends an acquisition
                   approach, which then competes for funding with other Air Force needs.
                   The information generated by this process is the basis for system
                   approval by Air Force and Defense officials, as well as decisions on the
                   system’s funding and timetable.


                   Air Force regulations clearly state that system requirements should be
Results in Brief   defined and alternative solutions evaluated before it recommends the
                   acquisition of a system and competes it for funding in the Department of
                   Defense budget process. Furthermore, major independent commissions
                   have echoed this same point since the early 1970s and have put Defense


                   Page 2                   GAO/IMTEG90-7   Air Force ADP Ftequirements-setting Process
                          Executive Summary




                          on notice that failure to do so will result in cost increases, schedule
                          delays, and performance problems. Simply put, the Air Force has not
                          followed its regulations or heeded this advice. For the seven ADP or ADP-
                          supported systems GAO reviewed, with an estimated development cost of
                          over $4.5 billion, the Air Force prematurely recommended acquisition.
                          In three cases, initial system requirements had not been adequately
                          defined or were continually changed; in four cases, alternative solutions
                          had not been evaluated.

                          The failure of the Air Force to take action to implement its regulations
                          indicates a lack of commitment to the process and a lack of appreciation
                          for its criticality. Until this is done, future ADP system acquisitions can
                          be expected to encounter cost growth and schedule delays. For example,
                          three ongoing acquisitions discussed in this report will be at least 7
                          years behind schedule and collectively, almost $900 million over their
                          original cost estimates. In this time of budget constraints, the Air Force
                          cannot afford to initiate ADP acquisitions without both solidifying sys-
                          tem requirements and evaluating alternative solutions, in order to
                          firmly establish a system’s technical approach, design, and cost.



Principal Findings

Requirements Definition   System requirements are not sufficiently defined when the Air Force
                          recommends an acquisition approach. This causes Air Force officials to
                          make assumptions when identifying a solution and estimating the cost to
                          satisfy the identified need. Such uncertainty results in performance,
                          cost, and schedule estimates that are often meaningless.

                          For example, in 1988 an Air Force task force found that the baseline
                          requirements were continually changing for its Command and Control
                          Segment program-a new satellite command and control system that
                          has been under development since 1981 and was to be completed in
                          1985. According to the task force chairman, the lack of stable baseline
                          requirements directly contributed to the system’s cost, schedule, and
                          technical problems. In October 1989, the Air Force reported that the sys-
                          tem would not be fully operational until 1993.




                          Page 3                    GAO/IMTEGSO-7 Air Force ADP Requirements-setting Process
                            Executive Summary




Evaluation of Alternative   The acquiring command does not always evaluate alternative solutions
                            to ensure that the selected approach is achievable, affordable, maintain-
Solutions                   able, and flexible enough to incorporate technological advances. For
                            example, in its Space Defense Operations Center 4 acquisition, the Air
                            Force did not evaluate alternative solutions to meeting a major system
                            requirement to control different levels of classified data, even though
                            this security requirement had not been achieved in any comparable sys-
                            tem. The contractor experienced problems in building a system to this
                            requirement because of both the software complexity and the difficulty
                            in attaining satisfactory system performance, given the extra processing
                            needed to run software with extensive security features built into it.
                            Ultimately, the contractor did not meet either the security requirement
                            or other critical system performance needs. As a result, although the
                            system was to be operational in 1988, the Air Force now estimates it will
                            not be fully operational until 1994; the current cost estimate is $576 mil-
                            lion-double the original estimate.


                            GAO recommends that the Secretary of the Air Force quickly take action
Recommendations             to implement its regulations established to assure that system require-
                            ments are adequately defined and alternative solutions are evaluated
                            before approving and recommending acquisitions. As part of this effort,
                            the Secretary should pull back and reassess currently proposed ADP
                            acquisitions competing for funding within the Department of Defense
                            where requirements and alternative solutions have not been adequately
                            defined and considered. The four proposed ADP acquisitions that GAO dis-
                            cusses in this report should be specifically pulled back and included in
                            this reassessment.


                            GAO  requested official agency comments on a draft of this report from
Agency Comments             the Department of Defense. While official written comments have not
                            been provided, GAO met with agency officials to verify data presented in
                            the report and has made revisions where appropriate.




                            Page 4                   GAO/IMTEG!M-7 Air Force ADP Requirements-setting   Process
Page 6   GAO/IM’l’EG!W-7 Air Force ADP Requirements-setting   Pnxese
Contents


Executive Summary                                                                                        2

Chapter 1                                                                                                8
Introduction            Air Force’s System Acquisition Process                                           8
                        Objectives, Scope, and Methodology                                              12

Chapter 2                                                                                               14
Air Force Prematurely   System Requirements Not Comprehensively Defined                                 14
                        Alternative Solutions Are Not Thoroughly Evaluated                              15
Recommends              Air Force Officials State Resources Are Not Available to                        17
Acquisition of ADP           Adequately Evaluate Requirements and Solutions
                        Defense Officials Recognize the Need to Conduct Early                           18
Systems                      Evaluations
                        Conclusions                                                                     20
                        Recommendations to the Secretary of the Air Force                               21
                        Agency Comments                                                                 21

Appendixes              Appendix I: Department of Defense Major Systems                                 24
                            Acquisition Phases
                        Appendix II: Cost Estimates for Developing the Seven                            26
                            ADP Systems
                        Appendix III: Major Contributors to This Report                                 27

Related GAO Products                                                                                    28




                        Abbreviations

                        ADP       Automated Data Processing
                        BMEWS     Ballistic Missile Early Warning System
                        GAO       General Accounting Office
                        IMTEC     Information Management And Technology Division
                        SON       statement of operational need


                        Page 6                   GAO/JMTEC90-7 Air Force ADP Requirements-setting   Process
Page 7   GAO/IBlTJW-?   Air Force ADP Reqniremenwtting   Procem
                                                                                                                 ,



Introduction


                      We have been consistently reporting on Defense weapon and automated
                      data processing (ADP) systems that are behind schedule, significantly
                      over budget, and often fail to perform as intended. We are not alone.
                      According to the President’s Blue Ribbon Commission on Defense Man-
                      agement,’ too many defense systems cost too much, take too long to
                      develop, and, by the time they are developed, incorporate obsolete tech-
                      nology. ADP acquisitions are an increasingly important part of Depart-
                      ment of Defense acquisitions-in fiscal year 1990 Defense requested
                      $8.7 billion for such systems. Examples cited by the Commission report
                      included acquisition problems experienced by all the military depart-
                      ments, including the Department of the Air Force, the subject of this
                      report.

                      It is commonly recognized that ADP system design and development
                      problems occur throughout the acquisition cycle. Many problems begin
                      early in the acquisition process because detailed requirements to meet
                      specified needs have not been developed. Establishing system require-
                      ments is a significant step in the acquisition process because require-
                      ments are the blueprint system developers use to design and develop
                      systems. The cost, schedule, and performance problems we identified
                      during reviews of our nation’s tactical warning and attack assessment
                      system modernization were caused, in part, because the Air Force had
                      established requirements that either could not be met or had to be
                      reduced to lower system acquisition costs.2 This review focuses on seven
                      ongoing or proposed ADP or mp-supported system acquisitions to deter-
                      mine if the problems we noted during our earlier reviews could be
                      caused by the Air Force’s procedures for setting requirements and eval-
                      uating alternative solutions.


                      The acquisition of Air Force systems is complex and involved, and gen-
Air Force’s System    erally is accomplished in five phases: (1) concept formulation, (2) dem-
Acquisition Process   onstration and validation, (3) full-scale engineering and development,
                      (4) full-rate production and initial deployment, and (5) operations sup-
                      port. (These phases are discussed in app. I.) The Air Force initiates an




                      ‘A Formula for Action: A Report to the president on Defense Acquisition, The president’s Blue Rib-
                      bon Commission on Defense Management, April 1986.

                      ‘Space Defense: Management and Technical problems Delay Operations Center Acquisition (GAO/
                      IhITlX-Sg-18, Apr. 20,1989) and Attack Warning: NORAD’s CIxnmunications System Segment
                      Replacement program Should Be Reassessed(GAO/IMTEG89-1, Nov. 30,198s).



                      Page 8                            GAO/IMTEG90-7      Air Force ADP Requirements-setting        Process
                            Chapter 1
                            Introduction




                            acquisition to satisfy a specific mission need. A need is defined as a defi-
                            ciency that inhibits or prevents the Air Force from carrying out a mis-
                            sion. Deficiencies typically result from threat changes, redefinition of
                            assigned tasks in response to shifts in national security policy, or deteri-
                            oration in operational performance of older systems. A need could also
                            result from technological advances that would enable the Air Force to
                            more effectively or efficiently carry out a mission. Requirements are the
                            overall system features and performance levels identified to satisfy a
                            need. Air Force needs, and the requirements to meet those needs, can be
                            identified by the Department of Defense, Joint Chiefs of Staff, Air Force
                            Headquarters, and individual Air Force commands.

                            The success of Air Force acquisitions can be affected by many factors,
                            including some that are out of the Air Force’s control. For example, a
                            budget cut may result in eliminating or modifying requirements after an
                            acquisition has been initiated. Also, a change in the threat could result
                            in adding requirements that increase the cost of an acquisition. How-
                            ever, successful acquisitions also depend on setting clear and attainable
                            requirements and evaluating alternatives early in the procurement pro-
                            cess. As discussed in the following section, inadequately defined
                            requirements and prematurely selected solutions contribute to cost
                            increases, lengthy system development, and systems that do not meet
                            their identified needs.


Prior Studies Cite          In 1972, a Commission on Government Procurement study group
Problems W ith Evaluating   reported3 that the requirements determination and the initial acquisition
                            planning process are accomplished prior to the significant involvement
Alternatives and            of Executive Branch top management and are accomplished in an
Establishing Requirements   unstructured manner, to the detriment of the major system acquisition
                            process. As a result, there is no way top management can effectively
                            evaluate acquisition alternatives with respect to concept, risk, and
                            schedule.

                            Further, in its summary report, the Commission* noted that establishing
                            needs and goals for a new acquisition program is one of the most vital
                            and potentially fruitful areas for improving system acquisition, and that
                            both defense and civilian programs have suffered when well-defined

                            3Final Report, Study Group #12-Major Systems Acquisition, System Requirements Determination
                            and Initial Acquisition Planning, Volume II, For the Commission on Government Procurement, Janu-
                            ary 1972.

                            4Summa.ryof the Report of The Ckxnm& ion on Govermnent Procurement, 1972.



                            Page 9                           GAO/IMTEG90-7 Air Force ADP Requirements-setting         Process
                            Chapter 1
                            Introduction




                            statements of need and goals were lacking. The report also stated that a
                            premature agency commitment to a system concept, technical approach,
                            and design often results in cost growth, performance shortfalls, and
                            schedule delays. The report stated that pressures to prematurely choose
                            a single system approach often result in limited analyses of less costly
                            alternatives. The report further found that money spent to evaluate
                            alternative approaches can be inexpensive insurance against the possi-
                            bility that a premature choice may later prove to be a poor and costly
                            one.

                            A more recent study noted similar problems with major system acquisi-
                            tions. In 1986, the President’s Blue Ribbon Commission on Defense Man-
                            agement, also known as the Packard Commission, reported that the
                            process of identifying the characteristics and specific requirements for a
                            new system generally did not adequately involve participants with a
                            detailed knowledge of the cost and schedule implications. As a result,
                            trade-offs between cost and performance did not occur to an adequate
                            degree and the system concept included requirements which may be
                            desirable but whose real cost far exceeds their value.

                            The Packard Commission study further noted that once military needs
                            are established, the next step is to “market” this system to get funding
                            authorized for its development. Such marketing takes place in a highly
                            competitive environment. This competitive environment does not
                            encourage realistic estimates of cost and schedule because system mar-
                            keters must be optimistic about how much funding and time will be
                            needed to develop the new system.

                            As a result, all too often when a system finally is approved, it has over-
                            stated requirements and understated costs. To correct this situation, the
                            Packard Commission recommended early, high-level management
                            review of requirements to assess the trade-off between cost and
                            performance.


Air Force Process for       In response to the Packard Commission’s recommendation, the Air Force
Establishing Requirements   revised its procedures5 for acquiring major systems, including ADP sys-
                            tems, in April 1987. These procedures govern how system requirements
                            are established for proposed systems. The revision was intended to


                            5Air Force Regulation 67-1, Operational Needs, Requirements, and Concepts, draft implemented on
                            April 1,1987, published in final form on October 7,1988.



                            Page 10                          GAO/lMTEC99-7      Air Force ADP Requirementssetting     Process
Chapter 1
Introduction




improve the procedures used to identify and approve military needs, ini-
tiate programs to satisfy those needs, and establish requirements for the
system to be developed.

Air Force procedures implementing the Department of Defense’s direc-
tion for establishing system requirements can be found in Directive
5000.1, Major and Non-Major Defense Acquisition Programs. The direc-
tive provides that the basis of need or requirement for each new acquisi-
tion program must be thoroughly reviewed and validated, and that a
major defense acquisition involving development of a new system must
be undertaken only after carefully assessing alternative approaches to
satisfy the need or requirements.

The Air Force’s system acquisition process begins by identifying a spe-
cific mission need. Generally, a using command-responsible for fielding
and operating systems during training or actual combat operations (e.g.,
Air Force Space Command)-identifies       a need and prepares a Statement
of Operational Need (SON). The SON'S primary purpose is to define the
need, document the validity of the need, and provide preliminary
requirements. The SON must be concise enough (five pages or less) to
facilitate processing but be sufficiently comprehensive to define the
requirements. A secondary purpose of the SON is to propose potential
solutions for the need.

A draft SON is circulated among various Air Force commands and agen-
cies to obtain their views and to avoid duplication. Under Air Force pro-
cedures, the acquiring command-a specialized command (e.g., Air
Force Systems Command) that is responsible for providing research,
development, and acquisition services to other Air Force commands-is
to review the draft SON. The acquiring command evaluates the need and
the proposed solution, identifies alternative solutions to meet the need,
and provides preliminary estimates of the cost and schedule required to
pursue the most attractive solution. It is the responsibility of the acquir-
ing command to address the possible solutions to the need described in
the SON.

After resolving any issues raised by the other commands, determining
that the expressed need is valid, and reviewing the recommended solu-
tion, the using command approves the SON, recommending the acquisi-
tion. The estimated cost to develop the solution is included with the SON
and is sent to Air Force Headquarters for use in competing for funding.
Headquarters decides which acquisition programs it will recommend



Page 11                   GAO/‘IMTEG90-7 Air Force ADP Requirements-setting   Process
                         Chapter 1
                         Introduction




                         and forwards them to the Department of Defense for consideration dur-
                         ing the budget formulation process.

                         The system acquisition cycle begins once the SON receives funding
                         approval in the Department of Defense Five-Year Defense Program and
                         the Defense Appropriation. At this point, a Program Management Direc-
                         tive is issued by Air Force Headquarters initiating the program. This
                         document provides direction to pertinent Air Force commands on their
                         roles and responsibilities for carrying out the acquisition and establishes
                         a program office to manage the acquisition. Funding levels for the sys-
                         tem and an acquisition schedule are also established at this point in the
                         acquisition process.

                         After the Program Management Directive has been approved, the using
                         command develops a System Operational Requirements Document to
                         address the detailed requirements. This document amplifies and refines
                         the SON; its needs statements are more comprehensive and quantitative
                         and are tailored to the possible solution. Air Force regulations direct
                         periodic updates and reviews of the System Operational Requirements
                         Document at subsequent acquisition phases.


                         The Subcommittee on Legislation and National Security, House Commit-
Objectives, Scope, and   tee on Government Operations, requested information on the Air Force’s
Methodology              requirements-setting process for ADP systems. We initiated this review to
                         determine if the Air Force’s procedures for setting requirements and
                         evaluating alternative solutions were causing problems in ADP acquisi-
                         tions. Our specific objective was to determine whether the Air Force
                         thoroughly defines system requirements and evaluates alternative solu-
                         tions to meet user needs before approving and recommending that an
                         ADP system be acquired. To evaluate the possible impact of the Air
                         Force’s requirements-setting procedures on the success of system acqui-
                         sitions, we (1) reviewed Air Force procedures for setting requirements
                         under Air Force Regulation 57-1, (2) reviewed how requirements were
                         established and alternative solutions evaluated for seven ongoing or
                         proposed ADP system acquisitions, and (3) evaluated whether analyses
                         of requirements and alternative solutions occur prior to recommending
                         system acquisition under the Air Force Regulation 57-l procedures.

                         During our review, we focused on seven Air Force ADP or Aup-supported
                         system acquisitions-three  ongoing programs that were approved for
                         funding and four proposed programs that were recommended for acqui-
                         sition by the using command and were competing for funding in the


                         Page 12                  GAO/IMTEG90-7 Air Force ADP Requirements-setting   Process
Y   Chapter 1
    Introduction




    D e fe n s e b u d g e t process. T h e o n g o i n g acquisitions w e r e v i e w e d w e r e th e
    C o m m a n d a n d C o n trol S e g m e n t,th e C o m m u n i c a tio n s S y s t e m S e g m e n t
    R e p l a c e m e n t,a n d th e S p a c eD e fe n s e O p e r a tio n s C e n ter 4 . T h e fo u r pro-
    p o s e d acquisition p r o g r a m s w e r e th e Ballistic M issile Early W a rning
    System, th e S p a c e - B a s e dA tm o s p h e r i c Surveillance System, th e S p a c e
    Surveillance System, a n d th e M ission P l a n n i n g System. ( A p p e n d i x II
    lists th e estimated cost for e a c h o f th e s e acquisitions.)

    W e interviewed o fficials a t A ir Force H e a d q u a r ters a n d a t th e two u s i n g
    c o m m a n d s involved in th e r e q u i r e m e n ts-setting process, a s well a s o ffi-
    cials from th e acquiring c o m m a n d involved in d e v e l o p i n g p r o g r a m cost
    estimates a n d i d e n tifying a n d evaluating alternative solutions for spe-
    cific p r o g r a m s . W e r e v i e w e d D e p a r tm e n t o f D e fe n s e a n d A ir Force
    m a n u a l s , directives, regulations, a n d g u i d a n c e ;O ffice o f M a n a g e m e n t
    a n d B u d g e t Circulars; various D e p a r tm e n t o f D e fe n s e a n d o th e r g o v e r n -
    m e n t reports a n d studies; G A O reports; a n d pertinent files m a i n ta i n e d a t
    e a c h location visited.

    W e p e r f o r m e d o u r work a t th e D e p a r tm e n t o f D e fe n s e ,Joint Chiefs o f
    S ta ff a n d A ir Force H e a d q u a r ters in W a s h i n g to n , D .C.; A ir Force S p a c e
    C o m m a n d ,P e te r s o n A ir Force B a s e ,C o l o r a d o ; M ilitary A irlift C o m -
    m a n d , S c o tt A ir Force B a s e ,Illinois; A ir Force S y s t e m s C o m m a n d ,
    A n d r e w s A ir Force B a s e ,Maryland; a s well a s its S p a c eS y s t e m s Divi-
    sion, L o s A n g e l e s ,California, a n d its E lectronic S y s t e m s Division, H a n -
    scorn A ir Force B a s e ,M a s s a c h u s e tts;A ir Force Logistics C o m m a n d ’s
    D e ta c h m e n t 2 5 , C o l o r a d o Springs, C o l o r a d o ; th e A e r o s p a c e C o r p o r a tio n
    in E l S e g u n d o ,California; a n d th e M itre C o r p o r a tio n in B e d ford, M a s s a -
    c h u s e tts. W e c o n d u c te d o u r work from S e p te m b e r 1 9 8 8 th r o u g h N o v e m -
    b e r 1 9 8 9 , in a c c o r d a n c ewith generally a c c e p te d g o v e r n m e n t a u d i tin g
    standards.




    Page 13                              G A O / l M l W X O - 7 Air F o r c e A D P Requirements-setting P r o c e s s
Air Force Prematurely RecommendsAcquisition
of ADP Systems

                      The Air Force’s acquisitions of ADP systems often result in systems
                      which do not meet users’ needs, cost more, and take longer to develop
                      than anticipated. This is due, in part, to Air Force decisions to recom-
                      mend acquiring systems before defining the system’s requirements or
                      evaluating alternative solutions. This practice has resulted in require-
                      ments which could not be met or cost more to meet than originally antic-
                      ipated. Officials within the Department of Defense and the Air Force
                      recognize that the requirements-setting process needs improvement and
                      have suggested revisions to the process.


                      Air Force regulations require that preliminary requirements be defined
System Requirements   before the statement of need is approved and the system is recom-
Not Comprehensively   mended for acquisition. In three of the ADP systems we reviewed, the Air
Defined               Force did not adequately define requirements before recommending an
                      acquisition. As a result, the Air Force developed a design based on
                      assumptions about the requirements and estimated the cost and sched-
                      ule for development based on these same assumptions. Basing acquisi-
                      tion decisions on assumptions rather than specific information resulted
                      in performance, schedule, and cost estimates, which were in some cases,
                      meaningless.

                      For example, in 1981 the Air Force estimated that the Command and
                      Control Segment (previously referred to as the Data System Moderniza-
                      tion program) for satellite command and control would cost $195 million
                      and would be operational by 1985. However, the system is laboring
                      through development problems, schedule delays, cost increases, and is
                      not yet fully operational. The Air Force currently estimates that the sys-
                      tem will be operational in 1993. As of December 1988, Air Force docu-
                      ments show that about $458 million had been spent developing the
                      system. The Air Force expects that total system costs will be at least
                      $557 million when the system achieves full operational capability.

                      In 1988, because of these problems, a task force was established to
                      review the system to determine if it was fatally flawed. The first action
                      of the task force was to document the program’s baseline requirements.
                      However, according to the task force, requirements were continually
                      being added while limited dialogue occurred between the satellite opera-
                      tors and the acquirer to define real user needs. According to the task
                      force chairman, this situation resulted in a lack of a stable requirements
                      baseline, which contributed to the program’s cost, schedule, and techni-
                      cal problems.



                      Page 14                  GAO/lMTEG90-7   Air Force ADP Requirements-setting   Process
r                       Chapter 2
                        Air Force Prematurely Recommends
                        Acquisition of ADP Systems




                        The Space Surveillance System, which is to augment existing ground-
                        based capabilities to track satellites, is another example where require-
                        ments were not comprehensively defined before recommending system
                        acquisition. Although program officials said that prelim inary require-
                        ments were established and evaluations of alternatives were done before
                        they approved the SON, data processing requirements were based on sev-
                        eral assumptions rather than comprehensively defined requirements.

                        For example, the program officials said that the data processing require-
                        ments of the Space Surveillance System (which is one of the systems
                        competing for funding in the Defense budget process) were based on the
                        assumption that m inimal on-board processing will occur on satellites.
                        However, when the Air Force recommended acquisition approval, it had
                        not decided whether data processing will be done on the satellites or at
                        ground-based stations. If the Air Force decides to process the data on
                        the satellites, it will result in increased requirements for the satellite
                        that could add to its weight, technological risk, and cost. In addition,
                        ground sensors may be sized incorrectly because on-board satellite data
                        processing will affect the amount of data processed by the ground sta-
                        tions. This is the type of requirement that should be established before
                        the Air Force recommends an acquisition approach.

                        Another example where requirements were not defined before the Air
                        Force recommended an acquisition is the M ilitary Airlift Command’s
                        M ission Planning System. This program is also competing for funding in
                        the Defense budget process. The program is to develop an automated
                        system that air crews can use to accurately and rapidly select the best
                        air route. According to the acquiring command, which is responsible for
                        providing or verifying information on alternative solutions and cost, it
                        had difficulty providing information on alternative solutions and cost
                        for the system because the SON did not define potential system require-
                        ments and its description of the need was not specific enough. As a
                        result, the acquiring command cautioned that the cost and schedule esti-
                        mate it prepared for the SON was rough and could increase significantly
                        when systems requirements are later defined.


                        Air Force regulations require that commands consider various possible
Alternative Solutions   solutions to satisfy the need before approving the statement of need and
Are Not Thoroughly      recommending system acquisition. In four systems we reviewed, thor-
Evaluated               ough evaluations of alternative system solutions were not conducted
                        before recommending an acquisition. Had the Air Force adequately eval-
                        uated alternative solutions, it would have been in a better position to


                        Page 16                      GAO/lMTEC90-7   Air Force ADP Requirementssetting   Process
                                                                                 ,

Chapter 2                                                                                Y
Air Force Prematurely Recommends
Acquisition of ADP Systems




determine whether the proposed approach was the best approach to
meet the need.

For example, in its Space Defense Operations Center 4 acquisition
effort-a system intended to monitor up to 10,000 objects in space-the
Air Force included a requirement that the system be capable of operat-
ing at a level of security, namely controlled mode security, that had not
been achieved in any comparable system. A system operating in con-
trolled mode is intended to ensure that users cleared to receive informa-
tion at the secret, confidential, or unclassified level can gain access to
only the information to which they are entitled. However, the Air Force
did not evaluate alternative solutions to determine whether there were
other approaches to achieve controlled mode security-even after it
was put on notice by two concept definition contractors (Martin Mari-
etta Corporation and Ford Aerospace Corporation) as to the risk of this
undertaking.

Martin Marietta made it clear in its initial trade-off analysis that there
had been little success in achieving controlled mode security and that
the Space Defense Operations Center 4 acquisition need not be put at
risk when other viable alternatives were available. In a subsequent
design proposal, Martin Marietta proposed that security limitations be
identified and a security analysis be undertaken. Further, Ford’s initial
design proposal identified hardware and software limitations and excep-
tions to the security requirements. The initial concerns raised by both
concept definition contractors and the limitations subsequently identi-
fied in Martin Marietta’s later design should have put the Air Force on
notice that an independent assessment of the achievability of the secur-
ity requirement was needed. However, none was performed.

It is not surprising then that the contractor experienced problems in
building a system to this requirement. The primary problems were the
software complexity and the difficulty in attaining satisfactory system
performance, given the extra processing needed to run software with
extensive security features built into it. In addition, the extra processing
load slowed system performance. As a result, the contractor could not
achieve controlled mode security or most of the system’s critical per-
formance requirements. This system was scheduled to be fully opera-
tional in 1988 at a cost of $290 million; however, the Air Force now
estimates that the system will not be completed until 1995 and will cost
$576 million.




Page 16                      GAO/IMTJZGSO-7 Air Force ADP Requirements-setting       Process
     ,

C.
                      Chapter 2
                      Air Force Prematurely Recommends
                      Acquisition of ADP Systems




                      In another instance, we found that the Air Force did not perform thor-
                      ough analyses of alternative solutions to meet system requirements to
                      upgrade the third and last Ballistic Missile Early Warning System
                      (BMEWS) site before approving the statement of need and recommending
                      system acquisition. BMEWS is a ground-based computerized radar system
                      operated by Air Force Space Command at three sites to provide ballistic
                      missile attack warning information. Air Force Space Command, the
                      using command, wrote the statement of operational need for the third
                      site based on the information developed during the upgrade to the first
                      site. However, the acquiring command stated that the requirements defi-
                      nitions were inadequate for the third site because unique hardware and
                      power requirements for that site had not been defined. As a result, the
                      acquiring command could not evaluate alternative solutions and made
                      assumptions about the user’s proposed solution when it provided infor-
                      mation on the estimated cost to upgrade the last BMEWS site. The acquir-
                      ing command qualified its cost estimate stating that the estimate was
                      based on an “oversimplified version” of data used for the first two sites
                      and not based on the unique needs of the third site. This potential acqui-
                      sition is now competing for funding based on a questionable $320 million
                      cost estimate and technological approach.


                      Air Force officials said that commands responsible for acquiring sys-
Air Force Officials   terns are not given enough time or resources to evaluate requirements
State Resources Are   and alternative solutions when commenting on statements of opera-
Not Available to      tional need. As a result, they said emphasis is placed on developing a
                      cost estimate for the solution proposed by the user and not on examining
Adequately Evaluate   the need and justification for the proposed system versus other
Requirements and      alternatives.
Solutions             Air Force officials stated that it could take years to identify and evalu-
                      ate requirements, alternative solutions, and costs for many statements
                      of operational need. However, Air Force procedures allow only 30 days
                      for the acquiring command to do these analyses. The officials added that
                      most of the 30 days is consumed by administrative handling, leaving
                      only about 1 week to work on the statement of operational need. For
                      example, acquiring command officials stated that they were allowed
                      about 1 to 2 weeks to prepare information on solutions and cost to be
                      included in the statements of operational need for three of the programs
                      we reviewed. As a result, comprehensive information about potential
                      system requirements, alternative solutions, and estimated costs is not
                      available when decisions are made to recommend acquisitions.



                      Page 17                      GAO/IMTEG90-7   Air Force ADP Requirements-setting   Process
                          Chapter 2
                          Air Force Prematurely Recommends
                          Acquisition of ADP Systems




                          In addition, acquiring command officials stated that they are not given
                          sufficient funding to provide the resources needed to analyze require-
                          ments, solutions, and costs during development of statements of opera-
                          tional needs. One acquisition official acknowledged that the amount of
                          funding would depend on the level of technological complexity and risk
                          of the systems involved; he estimated that between $500,000 and $5
                          million might be needed to develop this information for most programs.
                          Another official estimated that an average of about $1.6 million is
                          needed to develop this information. However, funding to evaluate all
                          statements of operational need at Air Force Space Systems Division has
                          declined from $1.8 million for fiscal year 1988 to $780,000 in fiscal year
                          1989. One program official told us that this funding has been reduced to
                          zero for fiscal year 1990.


                          As discussed in this report, prior GAO and Defense studies have high-
Defense Officials         lighted inadequate requirements definition and evaluations of alterna-
Recognize the Need to     tive solutions as acquisition problems. These problems continue today.
Conduct Early             Officials at all levels within the Department of Defense and the Air
                          Force have indicated an awareness that the requirements-setting pro-
Evaluations               cess needs improvement. For example, the Secretary of Defense’s July
                          1989 Defense Management Report to the President suggested further
                          revisions to the process for establishing requirements to achieve the
                          degree of improvement recommended by the Packard Commission in
                          1986. Further, a senior Joint Chiefs of Staff official has stated that the
                          requirements process needs to be closely examined and that acquisition
                          should not be initiated until all alternatives have been evaluated and a
                          system concept has been defined. Finally, Air Force Systems Command
                          proposed in March 1989 that the requirements process be changed to
                          ensure that requirements are defined and alternative solutions are eval-
                          uated before initiating an acquisition.


1989 Management Review    In July 1989, the Secretary of Defense issued a Defense Management
Report to the President   report? to the President outlining a plan to improve the defense acquisi-
                          tion process. The report acknowledges that the Department of Defense
                          has not fully implemented the Packard Commission’s recommendation
                          to conduct early high-level management reviews of requirements to
                          assess the trade-off between cost and performance. The report identified
                          further changes needed to realize improvements in the acquisition pro-
                          cess to the degree contemplated by the Commission. In particular, the

                          ‘Defense Management Report to the President, Department of Defense, July 1989.



                          Page 18                         GAO/lMTEG90-7      Air Force ADP Requirements-setting   Process
  .
                            Chapter 2
                            Air Force Prematurely Recommends
                            Acquisition of ADP Systems




                            report states that system acquisitions should not be initiated until suffi-
                            cient information is gathered on alternative solutions. The report con-
                            tains a plan to revise the requirements-setting process to ensure that
                            systems are acquired at less cost, in less time, and with greater assur-
                            ance of promised performance.

                            Under this plan, the Secretary of Defense and Chairman of the Joint
                            Chiefs of Staff will use the Joint Requirements Oversight Council to
                            review deficiencies in current capabilities and the validity of identified
                            mission needs, and develop a prioritized list of systems for review by
                            the Defense Acquisition Board. The Board is the primary advisor to the
                            Secretary of Defense on need, affordability, cost, and schedules for
                            major system acquisitions. The systems are not to be funded until infor-
                            mation on alternative solutions has been generated and a decision has
                            been made on an acquisition approach. The Air Force is revising its reg-
                            ulations to adopt the Secretary’s plan, which is to be implemented in
                            1990.


Views of the Vice           The Vice Chairman of the Joint Chiefs of Staff has made public his
Chairman, Joint Chiefs of   views on defense acquisition process deficiencies.2 The Vice Chairman
                            said the services should not be driven to make early decisions; systems
Staff                       get in trouble if minds are made up too early and commands are held to
                            prematurely established (before system concept definition) dollar
                            thresholds. He added that information on potential technologies and the
                            cost to develop them cannot reasonably be developed until system
                            requirements, alternative concepts, and alternative solutions have been
                            explored.

                            The Vice Chairman said that more attention must be given to exploring
                            alternative concepts before an acquisition enters full-scale development.
                            He added that acquisition initiation decisions should be based on the
                            results of alternative concept evaluations and the demonstration that
                            the proposed solution is valid.




                            ‘“Hems Eyes Acquisition Fixes, To Delay ‘New Starts’ Beyond Milestone Zero,” Inside the Pentagon,
                            October 14,19SS, pp. 47.



                            Page 19                          GAO/IM’IEG90-7    Air Force ADP Requirements-setting    Process
                          Chapter 2
                          Air Force Prematurely Recommends
                          Acquisition of ADP Systems




Air Force Systems         On March 13,1989, Air Force Systems Command issued a draft paper
                          which discusses deficiencies in the Air Force’s current requirements pro-
Command RecognizesTha t   cess and sets forth a proposal for improving the process. The proposal
the
- Requirements-setting
        -- _              endorses the findings of the earlier commissions3 and basically reiterates
Process Needs             the existing regulations. The proposal suggests that concept exploration,
Improvement               development activities, and trade-off analyses involving requirements
                          and cost should occur before initiating acquisitions. Air Force Systems
                          Command believes, and current regulations require, that acquisitions
                          should begin only after a thorough analysis of the need by the com-
                          mands that will acquire and use the system. The acquirer should be
                          given sufficient time to analyze available technologies and alternative
                          solutions during development of the sonr-well before initiating acquisi-
                          tion. According to Air Force System Command’s proposal, the most sig-
                          nificant change which needs to be made to the requirements process is
                          to avoid the assumption that there is only one solution to an established
                          need. Once a need is identified, the proposal states that analyses must be
                          performed involving system performance, cost, and schedule to deter-
                          mine the optimum operational capabilities,

                          Under the proposal, concept exploration and development would take
                          place before acquisitions are initiated to develop a better understanding
                          of alternative solutions; to incorporate technological considerations in
                          analyses of mission, requirements, solutions, and cost; and to identify
                          the best solution to the stated problem. Alternatives should be defined
                          in sufficient conceptual detail so that they can be considered in terms of
                          technology, support, operations, maintenance, and life-cycle costs.


                          In this and prior reviews, we identified Air Force ADP system problems
Conclusions               that began early in the acquisition cycle because requirements were not
                          well defined and alternative solutions were not thoroughly evaluated.
                          These problems occur even though Air Force regulations clearly state
                          that system requirements should be defined and alternative solutions
                          evaluated before the Air Force recommends acquisition of a system and
                          competes it for funding in the Department of Defense budget process.
                          Furthermore, since the early 1970s major independent commissions
                          have made the same point and put Defense on notice that when system
                          requirements are not comprehensively defined and alternative solutions
                          are not fully evaluated, modifications are often necessary, resulting in
                          cost increases, schedule delays, and performance problems.


                          3The Commission on Government procurement and the Packard Commission.



                          Page 20                        GAO/IMTEGSO-7 Air Force ADP Requirements-setting   Process
 .
                       Chapter 2
                       Air Force Prematurely Recommends
                       Acquisition of ADP Systems




                       In the cases we reviewed, the Air Force recommended ADP acquisitions
                       before system requirements were adequately defined or alternative solu-
                       tions to meet the need had been evaluated. Had the Air Force ade-
                       quately evaluated alternative solutions, it would have been in a better
                       position to identify the best solution in terms of achievability, cost, and
                       schedule. However, the Air Force committed to acquisitions without a
                       clear understanding of how to solve the problem, or any assurance that
                       the system it is acquiring w-ill meet the stated need. We believe this com-
                       mitment is premature. Furthermore, the failure of the Air Force to
                       resolve problems after repeated notification indicates a lack of commit-
                       ment to the process, and a lack of appreciation for its criticality.

                       Requirements need to be thoroughly defined and alternatives evaluated
                       before recommending an acquisition. Until this is done, future ADP sys-
                       tem acquisitions can be expected to encounter problems similar to those
                       discussed in this report-such as the Space Defense Operations Center
                       4’s $286 million overrun and 7-year delay. Especially in this time of
                       budget constraints, the Air Force needs to take quick action to imple-
                       ment its regulations. The Air Force cannot afford to initiate ADP acquisi-
                       tions without solidifying system requirements and evaluating
                       alternative solutions that firmly establish a system’s technical
                       approach, design, and cost.


                       To help reduce the number of costly and lengthy ADP acquisition pro-
Recommendations to     grams, we recommend that the Secretary of the Air Force quickly take
the Secretary of the   action to implement its regulations established to assure that system
Air Force              requirements are adequately defined and alternative solutions are eval-
                       uated before approving and recommending acquisitions. These evalua-
                       tions should consider technological advances and limitations;
                       requirements achievability, reasonableness, and cost-effectiveness; and
                       acquisition schedule and affordability so that the Air Force can assess
                       whether the proposed acquisition will meet its needs. As part of this
                       effort, the Secretary should pull back those programs competing for
                       funding within the Department of Defense where requirements and
                       alternative solutions have not been adequately defined and considered.
                       The four proposed ADP acquisitions discussed in this report should be
                       specifically pulled back and included in this reassessment.


                       We requested official agency comments on a draft of this report from
Agency Comments        the Department of Defense. While official written comments have not



                       Page 21                      GAO/IMTEG!W7   Air Force ADP Reqnirementa-setthg   Process
                                                                                  .
Chapter 2
Air Force Prematurely Recommends
Acquisition of ADP Systems




been provided, we met with agency officials to verify data presented in
the report and have made revisions where appropriate.




Page 22                      GAO/IMTEG9O-7 Air Force ADP Requirements-setting   Process
Page 23   GAO/MTEC?   Air Force ADP Reqnirementitting   Process
Appendix I

Department of Defense Major Systems
Acquisition Phases

                      Mission area analysis and program initiation generally precede the five
                      Department of Defense acquisition phases. Defense components continu-
                      ally analyze their assigned mission areas to identify deficiencies (needs)
                      and to determine if new systems or major upgrades to existing systems
                      are necessary. These analyses often result in recommendations to initi-
                      ate new acquisition programs through the validation of a need to correct
                      the deficiency. Once a need has been identified and validated and
                      Defense initiates an acquisition program, the program enters the concept
                      formulation phase.


                      In this phase, potential requirements and alternative approaches to sat-
Concept Formulation   isfy the need are identified and evaluated. Various types of analyses
Phase                 considering trade-offs among performance, life-cycle cost, and schedule
                      are conducted to select among possible concepts to satisfy the need.
                      Once a concept has been identified, it is presented to Defense for
                      approval.


                      In this phase, feasibility and desirability of the selected requirements
Demonstration and     and the system concept is further analyzed, generally using techniques
Validation Phase      like computer simulation, hardware prototyping, developmental test and
                      evaluation, operational test and evaluation, or a combination of test
                      methods. When the feasibility of the concept has been convincingly
                      demonstrated and validated, the program enters the full-scale engineer-
                      ing and development phase.




                      Page 24                  GAO/IMTEGso-‘I Air Force ADP Requirements-setting Process
    ,


                         Appendix I
                         Department of Defense MJor Systems
                         Acquisition Phases




                         In this phase, the system, including all of the items necessary for its
Full-scale Engineering   logistic and operational support, is designed, fabricated, and tested. At
and Development          the conclusion of this phase, the system is ready to be produced.
Phase

                         During this phase the proposed system is built and released to the user.
Full-rate Production     At this point, the system becomes operational.
and Initial Deployment
Phase

                         This phase covers that period of time immediately following deployment
Operations Support       of the system and extends until the system is removed from Defense
Phase                    inventory. Two major Defense reviews are conducted in this phase. The
                         first takes place 1 to 2 years after deployment to determine if opera-
                         tional readiness and support objectives are being achieved and main-
                         tained. The second review occurs 5 to 10 years after deployment. It
                         evaluates system capabilities and assesseswhether major upgrades are
                         needed or if the system should be replaced.




                         Page 26                      GAO/IMTEC9O-7 Air Force ADP Requirements-setting   Process
Appendix II

Cost Estimaks for Developing the Seven                                                                            *
ADP Systems

               Dollars in millions
                                                                 Recent cost         Original cost              cost
                                                                    estimate             estimate          increase
               Onaoina Svstems
               Command and Control Segment
               (prevrously called Data System
               Modernization)                                               557                 195               362
               Communications System Segment
               Replacement                                                  422a                202               220
               Space Defense Operations Center 4                            576a                290               286
               Subtotal                                                                                           866
               Proposed Systems
               Ballistic Mrssile Early Warning System                       320
               .SS;cemBased Atmospheric Surveillance
                                                                          1.131b
               Space Surveillance System                                  1,446
               Mission Plannina Svstem                                       53
               Total                                                     4.505
               aEstimates Include costs for Research, Development, Test, and Evaluation, Other Procurement, and
               Operation and Maintenance.
               bEstlmates include costs through the demonstration and validation phase. They do not include costs for
               full-scale engineering and development.




               Page 26                            GAO/IMTEG90-7 Air Force ADP Requhementxpsetting Process
      5



Appendix III

Major Contributors to This Report


                         M ichael T. Blair, Assistant Director
Inforrnation             Ronald L. Hess, Evaluator
Management and           Teresa M . Schlee, W riter-Editor
Technology Division,
Washington, D.C.

                         Frederick G. Day, Regional Management Representative
Denver Regional          Sigrid L. McGinty, Evaluator-in-Charge
Office                   Luis Maez, Evaluator
                         Peggy A. Stott, Evaluator


                         Robert J. Dziekiewicz, Evaluator
Boston Regional Office




                         Page 27                    GAO/IlMTEC90-7 Air Force ADP Reqnirement.asetting   IVocess
Related GAO Products


              Military Space Operations: Operational Problems Continue with the Sat-
              ellite Control Computer System (G~opm~G89-66, Aug. 8,1989).

              Attack Warning: Better Management Required to Resolve NORAD Inte-
              gration Deficiencies (GAO/IMTEG89-26, July 7, 1989).

              Space Defense: Management and Technical Problems Delay Operations
              Center Acquisition (GAO/IMTEC-89-18, Apr. 20, 1989).

              Attack Warning: NORAD’s Communications System Segment Replace-
              ment Program Should Be Reassessed (GAO/IMTEC89-1, Nov. 30, 1988).

              Military Space Operations: Shuttle and Satellite Computer Systems Do
              Not Meet Performance Objectives (GAO/IMW&7,       Aug. 5,1988).




(310339)      Page 28