Information Management and Technology Division B-237417 March 29,199O The Honorable John Conyers, Jr. Chairman, Legislation and National Security Subcommittee Committee on Government Operations House of Representatives The Honorable Frank Horton Ranking Minority Member, Legislation and National Security Subcommittee Committee on Government Operations House of Representatives In response to your request, this report discusses weaknesses in the Department of the Air Force’s requirements-setting process for automated data processing system acquisitions. Air Force regulations clearly state that system requirements should be defined and alternative solutions evaluated before recommending the acquisition of a system. However, the Air Force has not followed these regulations. This failure indicates a lack of commitment to the process and a lack of appreciation for its criticality. As arranged with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this letter. At that time, we will send copies to the Director, Office of Management and Budget; the Secretary of Defense; and other interested parties. This report was prepared under the direction of Samuel W. Bowlin, Director for Defense and Security Information Systems, who may be reached at (202) 275-4649. Other major contributors are listed in appendix III. Ralph V. Carlone Assistant Comptroller General Ekeeutive Sumnwy Many defense systems are too expensive, take too long to develop, and Purpose use obsolete technology. An increasingly important part of Department of Defense procurement includes automated data processing (ADP) acqui- sitions; Defense requested almost $9 billion for ADP resources in fiscal year 1990. GAO reviews of Air Force actions to modernize ADP capability for America’s tactical warning and attack assessment system found pro- grams over budget and behind schedule, and systems that did not meet performance requirements. This happened, in part, because the Air Force established requirements that either could not be met or had to be reduced to contain escalating program costs. The Subcommittee on Legislation and National Security, House Commit- tee on Government Operations, requested this report. Concerned that inadequately defined requirements and a lack of evaluations of alterna- tive solutions could have led to these acquisition problems, GAO evalu- ated seven ongoing or proposed ADP or AuP-supported system acquisitions to determine whether the Air Force (1) thoroughly defines system requirements and (2) evaluates alternative solutions before rec- ommending ADP system acquisitions. Acquisition programs are initiated to satisfy specific mission needs or Background deficiencies that inhibit or prevent a military command from carrying out a mission. The Air Force establishes system requirements to over- come the deficiencies and satisfy the needs. After the using Air Force command identifies a need, it proposes a solu- tion that supplies preliminary requirements. The Air Force command responsible for acquiring new systems evaluates the need and alterna- tive solutions and estimates the likely cost and time necessary to acquire the most feasible solution. After incorporating this information and data from other commands, the using command recommends an acquisition approach, which then competes for funding with other Air Force needs. The information generated by this process is the basis for system approval by Air Force and Defense officials, as well as decisions on the system’s funding and timetable. Air Force regulations clearly state that system requirements should be Results in Brief defined and alternative solutions evaluated before it recommends the acquisition of a system and competes it for funding in the Department of Defense budget process. Furthermore, major independent commissions have echoed this same point since the early 1970s and have put Defense Page 2 GAO/IMTEG90-7 Air Force ADP Ftequirements-setting Process Executive Summary on notice that failure to do so will result in cost increases, schedule delays, and performance problems. Simply put, the Air Force has not followed its regulations or heeded this advice. For the seven ADP or ADP- supported systems GAO reviewed, with an estimated development cost of over $4.5 billion, the Air Force prematurely recommended acquisition. In three cases, initial system requirements had not been adequately defined or were continually changed; in four cases, alternative solutions had not been evaluated. The failure of the Air Force to take action to implement its regulations indicates a lack of commitment to the process and a lack of appreciation for its criticality. Until this is done, future ADP system acquisitions can be expected to encounter cost growth and schedule delays. For example, three ongoing acquisitions discussed in this report will be at least 7 years behind schedule and collectively, almost $900 million over their original cost estimates. In this time of budget constraints, the Air Force cannot afford to initiate ADP acquisitions without both solidifying sys- tem requirements and evaluating alternative solutions, in order to firmly establish a system’s technical approach, design, and cost. Principal Findings Requirements Definition System requirements are not sufficiently defined when the Air Force recommends an acquisition approach. This causes Air Force officials to make assumptions when identifying a solution and estimating the cost to satisfy the identified need. Such uncertainty results in performance, cost, and schedule estimates that are often meaningless. For example, in 1988 an Air Force task force found that the baseline requirements were continually changing for its Command and Control Segment program-a new satellite command and control system that has been under development since 1981 and was to be completed in 1985. According to the task force chairman, the lack of stable baseline requirements directly contributed to the system’s cost, schedule, and technical problems. In October 1989, the Air Force reported that the sys- tem would not be fully operational until 1993. Page 3 GAO/IMTEGSO-7 Air Force ADP Requirements-setting Process Executive Summary Evaluation of Alternative The acquiring command does not always evaluate alternative solutions to ensure that the selected approach is achievable, affordable, maintain- Solutions able, and flexible enough to incorporate technological advances. For example, in its Space Defense Operations Center 4 acquisition, the Air Force did not evaluate alternative solutions to meeting a major system requirement to control different levels of classified data, even though this security requirement had not been achieved in any comparable sys- tem. The contractor experienced problems in building a system to this requirement because of both the software complexity and the difficulty in attaining satisfactory system performance, given the extra processing needed to run software with extensive security features built into it. Ultimately, the contractor did not meet either the security requirement or other critical system performance needs. As a result, although the system was to be operational in 1988, the Air Force now estimates it will not be fully operational until 1994; the current cost estimate is $576 mil- lion-double the original estimate. GAO recommends that the Secretary of the Air Force quickly take action Recommendations to implement its regulations established to assure that system require- ments are adequately defined and alternative solutions are evaluated before approving and recommending acquisitions. As part of this effort, the Secretary should pull back and reassess currently proposed ADP acquisitions competing for funding within the Department of Defense where requirements and alternative solutions have not been adequately defined and considered. The four proposed ADP acquisitions that GAO dis- cusses in this report should be specifically pulled back and included in this reassessment. GAO requested official agency comments on a draft of this report from Agency Comments the Department of Defense. While official written comments have not been provided, GAO met with agency officials to verify data presented in the report and has made revisions where appropriate. Page 4 GAO/IMTEG!M-7 Air Force ADP Requirements-setting Process Page 6 GAO/IM’l’EG!W-7 Air Force ADP Requirements-setting Pnxese Contents Executive Summary 2 Chapter 1 8 Introduction Air Force’s System Acquisition Process 8 Objectives, Scope, and Methodology 12 Chapter 2 14 Air Force Prematurely System Requirements Not Comprehensively Defined 14 Alternative Solutions Are Not Thoroughly Evaluated 15 Recommends Air Force Officials State Resources Are Not Available to 17 Acquisition of ADP Adequately Evaluate Requirements and Solutions Defense Officials Recognize the Need to Conduct Early 18 Systems Evaluations Conclusions 20 Recommendations to the Secretary of the Air Force 21 Agency Comments 21 Appendixes Appendix I: Department of Defense Major Systems 24 Acquisition Phases Appendix II: Cost Estimates for Developing the Seven 26 ADP Systems Appendix III: Major Contributors to This Report 27 Related GAO Products 28 Abbreviations ADP Automated Data Processing BMEWS Ballistic Missile Early Warning System GAO General Accounting Office IMTEC Information Management And Technology Division SON statement of operational need Page 6 GAO/JMTEC90-7 Air Force ADP Requirements-setting Process Page 7 GAO/IBlTJW-? Air Force ADP Reqniremenwtting Procem , Introduction We have been consistently reporting on Defense weapon and automated data processing (ADP) systems that are behind schedule, significantly over budget, and often fail to perform as intended. We are not alone. According to the President’s Blue Ribbon Commission on Defense Man- agement,’ too many defense systems cost too much, take too long to develop, and, by the time they are developed, incorporate obsolete tech- nology. ADP acquisitions are an increasingly important part of Depart- ment of Defense acquisitions-in fiscal year 1990 Defense requested $8.7 billion for such systems. Examples cited by the Commission report included acquisition problems experienced by all the military depart- ments, including the Department of the Air Force, the subject of this report. It is commonly recognized that ADP system design and development problems occur throughout the acquisition cycle. Many problems begin early in the acquisition process because detailed requirements to meet specified needs have not been developed. Establishing system require- ments is a significant step in the acquisition process because require- ments are the blueprint system developers use to design and develop systems. The cost, schedule, and performance problems we identified during reviews of our nation’s tactical warning and attack assessment system modernization were caused, in part, because the Air Force had established requirements that either could not be met or had to be reduced to lower system acquisition costs.2 This review focuses on seven ongoing or proposed ADP or mp-supported system acquisitions to deter- mine if the problems we noted during our earlier reviews could be caused by the Air Force’s procedures for setting requirements and eval- uating alternative solutions. The acquisition of Air Force systems is complex and involved, and gen- Air Force’s System erally is accomplished in five phases: (1) concept formulation, (2) dem- Acquisition Process onstration and validation, (3) full-scale engineering and development, (4) full-rate production and initial deployment, and (5) operations sup- port. (These phases are discussed in app. I.) The Air Force initiates an ‘A Formula for Action: A Report to the president on Defense Acquisition, The president’s Blue Rib- bon Commission on Defense Management, April 1986. ‘Space Defense: Management and Technical problems Delay Operations Center Acquisition (GAO/ IhITlX-Sg-18, Apr. 20,1989) and Attack Warning: NORAD’s CIxnmunications System Segment Replacement program Should Be Reassessed(GAO/IMTEG89-1, Nov. 30,198s). Page 8 GAO/IMTEG90-7 Air Force ADP Requirements-setting Process Chapter 1 Introduction acquisition to satisfy a specific mission need. A need is defined as a defi- ciency that inhibits or prevents the Air Force from carrying out a mis- sion. Deficiencies typically result from threat changes, redefinition of assigned tasks in response to shifts in national security policy, or deteri- oration in operational performance of older systems. A need could also result from technological advances that would enable the Air Force to more effectively or efficiently carry out a mission. Requirements are the overall system features and performance levels identified to satisfy a need. Air Force needs, and the requirements to meet those needs, can be identified by the Department of Defense, Joint Chiefs of Staff, Air Force Headquarters, and individual Air Force commands. The success of Air Force acquisitions can be affected by many factors, including some that are out of the Air Force’s control. For example, a budget cut may result in eliminating or modifying requirements after an acquisition has been initiated. Also, a change in the threat could result in adding requirements that increase the cost of an acquisition. How- ever, successful acquisitions also depend on setting clear and attainable requirements and evaluating alternatives early in the procurement pro- cess. As discussed in the following section, inadequately defined requirements and prematurely selected solutions contribute to cost increases, lengthy system development, and systems that do not meet their identified needs. Prior Studies Cite In 1972, a Commission on Government Procurement study group Problems W ith Evaluating reported3 that the requirements determination and the initial acquisition planning process are accomplished prior to the significant involvement Alternatives and of Executive Branch top management and are accomplished in an Establishing Requirements unstructured manner, to the detriment of the major system acquisition process. As a result, there is no way top management can effectively evaluate acquisition alternatives with respect to concept, risk, and schedule. Further, in its summary report, the Commission* noted that establishing needs and goals for a new acquisition program is one of the most vital and potentially fruitful areas for improving system acquisition, and that both defense and civilian programs have suffered when well-defined 3Final Report, Study Group #12-Major Systems Acquisition, System Requirements Determination and Initial Acquisition Planning, Volume II, For the Commission on Government Procurement, Janu- ary 1972. 4Summa.ryof the Report of The Ckxnm& ion on Govermnent Procurement, 1972. Page 9 GAO/IMTEG90-7 Air Force ADP Requirements-setting Process Chapter 1 Introduction statements of need and goals were lacking. The report also stated that a premature agency commitment to a system concept, technical approach, and design often results in cost growth, performance shortfalls, and schedule delays. The report stated that pressures to prematurely choose a single system approach often result in limited analyses of less costly alternatives. The report further found that money spent to evaluate alternative approaches can be inexpensive insurance against the possi- bility that a premature choice may later prove to be a poor and costly one. A more recent study noted similar problems with major system acquisi- tions. In 1986, the President’s Blue Ribbon Commission on Defense Man- agement, also known as the Packard Commission, reported that the process of identifying the characteristics and specific requirements for a new system generally did not adequately involve participants with a detailed knowledge of the cost and schedule implications. As a result, trade-offs between cost and performance did not occur to an adequate degree and the system concept included requirements which may be desirable but whose real cost far exceeds their value. The Packard Commission study further noted that once military needs are established, the next step is to “market” this system to get funding authorized for its development. Such marketing takes place in a highly competitive environment. This competitive environment does not encourage realistic estimates of cost and schedule because system mar- keters must be optimistic about how much funding and time will be needed to develop the new system. As a result, all too often when a system finally is approved, it has over- stated requirements and understated costs. To correct this situation, the Packard Commission recommended early, high-level management review of requirements to assess the trade-off between cost and performance. Air Force Process for In response to the Packard Commission’s recommendation, the Air Force Establishing Requirements revised its procedures5 for acquiring major systems, including ADP sys- tems, in April 1987. These procedures govern how system requirements are established for proposed systems. The revision was intended to 5Air Force Regulation 67-1, Operational Needs, Requirements, and Concepts, draft implemented on April 1,1987, published in final form on October 7,1988. Page 10 GAO/lMTEC99-7 Air Force ADP Requirementssetting Process Chapter 1 Introduction improve the procedures used to identify and approve military needs, ini- tiate programs to satisfy those needs, and establish requirements for the system to be developed. Air Force procedures implementing the Department of Defense’s direc- tion for establishing system requirements can be found in Directive 5000.1, Major and Non-Major Defense Acquisition Programs. The direc- tive provides that the basis of need or requirement for each new acquisi- tion program must be thoroughly reviewed and validated, and that a major defense acquisition involving development of a new system must be undertaken only after carefully assessing alternative approaches to satisfy the need or requirements. The Air Force’s system acquisition process begins by identifying a spe- cific mission need. Generally, a using command-responsible for fielding and operating systems during training or actual combat operations (e.g., Air Force Space Command)-identifies a need and prepares a Statement of Operational Need (SON). The SON'S primary purpose is to define the need, document the validity of the need, and provide preliminary requirements. The SON must be concise enough (five pages or less) to facilitate processing but be sufficiently comprehensive to define the requirements. A secondary purpose of the SON is to propose potential solutions for the need. A draft SON is circulated among various Air Force commands and agen- cies to obtain their views and to avoid duplication. Under Air Force pro- cedures, the acquiring command-a specialized command (e.g., Air Force Systems Command) that is responsible for providing research, development, and acquisition services to other Air Force commands-is to review the draft SON. The acquiring command evaluates the need and the proposed solution, identifies alternative solutions to meet the need, and provides preliminary estimates of the cost and schedule required to pursue the most attractive solution. It is the responsibility of the acquir- ing command to address the possible solutions to the need described in the SON. After resolving any issues raised by the other commands, determining that the expressed need is valid, and reviewing the recommended solu- tion, the using command approves the SON, recommending the acquisi- tion. The estimated cost to develop the solution is included with the SON and is sent to Air Force Headquarters for use in competing for funding. Headquarters decides which acquisition programs it will recommend Page 11 GAO/‘IMTEG90-7 Air Force ADP Requirements-setting Process Chapter 1 Introduction and forwards them to the Department of Defense for consideration dur- ing the budget formulation process. The system acquisition cycle begins once the SON receives funding approval in the Department of Defense Five-Year Defense Program and the Defense Appropriation. At this point, a Program Management Direc- tive is issued by Air Force Headquarters initiating the program. This document provides direction to pertinent Air Force commands on their roles and responsibilities for carrying out the acquisition and establishes a program office to manage the acquisition. Funding levels for the sys- tem and an acquisition schedule are also established at this point in the acquisition process. After the Program Management Directive has been approved, the using command develops a System Operational Requirements Document to address the detailed requirements. This document amplifies and refines the SON; its needs statements are more comprehensive and quantitative and are tailored to the possible solution. Air Force regulations direct periodic updates and reviews of the System Operational Requirements Document at subsequent acquisition phases. The Subcommittee on Legislation and National Security, House Commit- Objectives, Scope, and tee on Government Operations, requested information on the Air Force’s Methodology requirements-setting process for ADP systems. We initiated this review to determine if the Air Force’s procedures for setting requirements and evaluating alternative solutions were causing problems in ADP acquisi- tions. Our specific objective was to determine whether the Air Force thoroughly defines system requirements and evaluates alternative solu- tions to meet user needs before approving and recommending that an ADP system be acquired. To evaluate the possible impact of the Air Force’s requirements-setting procedures on the success of system acqui- sitions, we (1) reviewed Air Force procedures for setting requirements under Air Force Regulation 57-1, (2) reviewed how requirements were established and alternative solutions evaluated for seven ongoing or proposed ADP system acquisitions, and (3) evaluated whether analyses of requirements and alternative solutions occur prior to recommending system acquisition under the Air Force Regulation 57-l procedures. During our review, we focused on seven Air Force ADP or Aup-supported system acquisitions-three ongoing programs that were approved for funding and four proposed programs that were recommended for acqui- sition by the using command and were competing for funding in the Page 12 GAO/IMTEG90-7 Air Force ADP Requirements-setting Process Y Chapter 1 Introduction D e fe n s e b u d g e t process. T h e o n g o i n g acquisitions w e r e v i e w e d w e r e th e C o m m a n d a n d C o n trol S e g m e n t,th e C o m m u n i c a tio n s S y s t e m S e g m e n t R e p l a c e m e n t,a n d th e S p a c eD e fe n s e O p e r a tio n s C e n ter 4 . T h e fo u r pro- p o s e d acquisition p r o g r a m s w e r e th e Ballistic M issile Early W a rning System, th e S p a c e - B a s e dA tm o s p h e r i c Surveillance System, th e S p a c e Surveillance System, a n d th e M ission P l a n n i n g System. ( A p p e n d i x II lists th e estimated cost for e a c h o f th e s e acquisitions.) W e interviewed o fficials a t A ir Force H e a d q u a r ters a n d a t th e two u s i n g c o m m a n d s involved in th e r e q u i r e m e n ts-setting process, a s well a s o ffi- cials from th e acquiring c o m m a n d involved in d e v e l o p i n g p r o g r a m cost estimates a n d i d e n tifying a n d evaluating alternative solutions for spe- cific p r o g r a m s . W e r e v i e w e d D e p a r tm e n t o f D e fe n s e a n d A ir Force m a n u a l s , directives, regulations, a n d g u i d a n c e ;O ffice o f M a n a g e m e n t a n d B u d g e t Circulars; various D e p a r tm e n t o f D e fe n s e a n d o th e r g o v e r n - m e n t reports a n d studies; G A O reports; a n d pertinent files m a i n ta i n e d a t e a c h location visited. W e p e r f o r m e d o u r work a t th e D e p a r tm e n t o f D e fe n s e ,Joint Chiefs o f S ta ff a n d A ir Force H e a d q u a r ters in W a s h i n g to n , D .C.; A ir Force S p a c e C o m m a n d ,P e te r s o n A ir Force B a s e ,C o l o r a d o ; M ilitary A irlift C o m - m a n d , S c o tt A ir Force B a s e ,Illinois; A ir Force S y s t e m s C o m m a n d , A n d r e w s A ir Force B a s e ,Maryland; a s well a s its S p a c eS y s t e m s Divi- sion, L o s A n g e l e s ,California, a n d its E lectronic S y s t e m s Division, H a n - scorn A ir Force B a s e ,M a s s a c h u s e tts;A ir Force Logistics C o m m a n d ’s D e ta c h m e n t 2 5 , C o l o r a d o Springs, C o l o r a d o ; th e A e r o s p a c e C o r p o r a tio n in E l S e g u n d o ,California; a n d th e M itre C o r p o r a tio n in B e d ford, M a s s a - c h u s e tts. W e c o n d u c te d o u r work from S e p te m b e r 1 9 8 8 th r o u g h N o v e m - b e r 1 9 8 9 , in a c c o r d a n c ewith generally a c c e p te d g o v e r n m e n t a u d i tin g standards. Page 13 G A O / l M l W X O - 7 Air F o r c e A D P Requirements-setting P r o c e s s Air Force Prematurely RecommendsAcquisition of ADP Systems The Air Force’s acquisitions of ADP systems often result in systems which do not meet users’ needs, cost more, and take longer to develop than anticipated. This is due, in part, to Air Force decisions to recom- mend acquiring systems before defining the system’s requirements or evaluating alternative solutions. This practice has resulted in require- ments which could not be met or cost more to meet than originally antic- ipated. Officials within the Department of Defense and the Air Force recognize that the requirements-setting process needs improvement and have suggested revisions to the process. Air Force regulations require that preliminary requirements be defined System Requirements before the statement of need is approved and the system is recom- Not Comprehensively mended for acquisition. In three of the ADP systems we reviewed, the Air Defined Force did not adequately define requirements before recommending an acquisition. As a result, the Air Force developed a design based on assumptions about the requirements and estimated the cost and sched- ule for development based on these same assumptions. Basing acquisi- tion decisions on assumptions rather than specific information resulted in performance, schedule, and cost estimates, which were in some cases, meaningless. For example, in 1981 the Air Force estimated that the Command and Control Segment (previously referred to as the Data System Moderniza- tion program) for satellite command and control would cost $195 million and would be operational by 1985. However, the system is laboring through development problems, schedule delays, cost increases, and is not yet fully operational. The Air Force currently estimates that the sys- tem will be operational in 1993. As of December 1988, Air Force docu- ments show that about $458 million had been spent developing the system. The Air Force expects that total system costs will be at least $557 million when the system achieves full operational capability. In 1988, because of these problems, a task force was established to review the system to determine if it was fatally flawed. The first action of the task force was to document the program’s baseline requirements. However, according to the task force, requirements were continually being added while limited dialogue occurred between the satellite opera- tors and the acquirer to define real user needs. According to the task force chairman, this situation resulted in a lack of a stable requirements baseline, which contributed to the program’s cost, schedule, and techni- cal problems. Page 14 GAO/lMTEG90-7 Air Force ADP Requirements-setting Process r Chapter 2 Air Force Prematurely Recommends Acquisition of ADP Systems The Space Surveillance System, which is to augment existing ground- based capabilities to track satellites, is another example where require- ments were not comprehensively defined before recommending system acquisition. Although program officials said that prelim inary require- ments were established and evaluations of alternatives were done before they approved the SON, data processing requirements were based on sev- eral assumptions rather than comprehensively defined requirements. For example, the program officials said that the data processing require- ments of the Space Surveillance System (which is one of the systems competing for funding in the Defense budget process) were based on the assumption that m inimal on-board processing will occur on satellites. However, when the Air Force recommended acquisition approval, it had not decided whether data processing will be done on the satellites or at ground-based stations. If the Air Force decides to process the data on the satellites, it will result in increased requirements for the satellite that could add to its weight, technological risk, and cost. In addition, ground sensors may be sized incorrectly because on-board satellite data processing will affect the amount of data processed by the ground sta- tions. This is the type of requirement that should be established before the Air Force recommends an acquisition approach. Another example where requirements were not defined before the Air Force recommended an acquisition is the M ilitary Airlift Command’s M ission Planning System. This program is also competing for funding in the Defense budget process. The program is to develop an automated system that air crews can use to accurately and rapidly select the best air route. According to the acquiring command, which is responsible for providing or verifying information on alternative solutions and cost, it had difficulty providing information on alternative solutions and cost for the system because the SON did not define potential system require- ments and its description of the need was not specific enough. As a result, the acquiring command cautioned that the cost and schedule esti- mate it prepared for the SON was rough and could increase significantly when systems requirements are later defined. Air Force regulations require that commands consider various possible Alternative Solutions solutions to satisfy the need before approving the statement of need and Are Not Thoroughly recommending system acquisition. In four systems we reviewed, thor- Evaluated ough evaluations of alternative system solutions were not conducted before recommending an acquisition. Had the Air Force adequately eval- uated alternative solutions, it would have been in a better position to Page 16 GAO/lMTEC90-7 Air Force ADP Requirementssetting Process , Chapter 2 Y Air Force Prematurely Recommends Acquisition of ADP Systems determine whether the proposed approach was the best approach to meet the need. For example, in its Space Defense Operations Center 4 acquisition effort-a system intended to monitor up to 10,000 objects in space-the Air Force included a requirement that the system be capable of operat- ing at a level of security, namely controlled mode security, that had not been achieved in any comparable system. A system operating in con- trolled mode is intended to ensure that users cleared to receive informa- tion at the secret, confidential, or unclassified level can gain access to only the information to which they are entitled. However, the Air Force did not evaluate alternative solutions to determine whether there were other approaches to achieve controlled mode security-even after it was put on notice by two concept definition contractors (Martin Mari- etta Corporation and Ford Aerospace Corporation) as to the risk of this undertaking. Martin Marietta made it clear in its initial trade-off analysis that there had been little success in achieving controlled mode security and that the Space Defense Operations Center 4 acquisition need not be put at risk when other viable alternatives were available. In a subsequent design proposal, Martin Marietta proposed that security limitations be identified and a security analysis be undertaken. Further, Ford’s initial design proposal identified hardware and software limitations and excep- tions to the security requirements. The initial concerns raised by both concept definition contractors and the limitations subsequently identi- fied in Martin Marietta’s later design should have put the Air Force on notice that an independent assessment of the achievability of the secur- ity requirement was needed. However, none was performed. It is not surprising then that the contractor experienced problems in building a system to this requirement. The primary problems were the software complexity and the difficulty in attaining satisfactory system performance, given the extra processing needed to run software with extensive security features built into it. In addition, the extra processing load slowed system performance. As a result, the contractor could not achieve controlled mode security or most of the system’s critical per- formance requirements. This system was scheduled to be fully opera- tional in 1988 at a cost of $290 million; however, the Air Force now estimates that the system will not be completed until 1995 and will cost $576 million. Page 16 GAO/IMTJZGSO-7 Air Force ADP Requirements-setting Process , C. Chapter 2 Air Force Prematurely Recommends Acquisition of ADP Systems In another instance, we found that the Air Force did not perform thor- ough analyses of alternative solutions to meet system requirements to upgrade the third and last Ballistic Missile Early Warning System (BMEWS) site before approving the statement of need and recommending system acquisition. BMEWS is a ground-based computerized radar system operated by Air Force Space Command at three sites to provide ballistic missile attack warning information. Air Force Space Command, the using command, wrote the statement of operational need for the third site based on the information developed during the upgrade to the first site. However, the acquiring command stated that the requirements defi- nitions were inadequate for the third site because unique hardware and power requirements for that site had not been defined. As a result, the acquiring command could not evaluate alternative solutions and made assumptions about the user’s proposed solution when it provided infor- mation on the estimated cost to upgrade the last BMEWS site. The acquir- ing command qualified its cost estimate stating that the estimate was based on an “oversimplified version” of data used for the first two sites and not based on the unique needs of the third site. This potential acqui- sition is now competing for funding based on a questionable $320 million cost estimate and technological approach. Air Force officials said that commands responsible for acquiring sys- Air Force Officials terns are not given enough time or resources to evaluate requirements State Resources Are and alternative solutions when commenting on statements of opera- Not Available to tional need. As a result, they said emphasis is placed on developing a cost estimate for the solution proposed by the user and not on examining Adequately Evaluate the need and justification for the proposed system versus other Requirements and alternatives. Solutions Air Force officials stated that it could take years to identify and evalu- ate requirements, alternative solutions, and costs for many statements of operational need. However, Air Force procedures allow only 30 days for the acquiring command to do these analyses. The officials added that most of the 30 days is consumed by administrative handling, leaving only about 1 week to work on the statement of operational need. For example, acquiring command officials stated that they were allowed about 1 to 2 weeks to prepare information on solutions and cost to be included in the statements of operational need for three of the programs we reviewed. As a result, comprehensive information about potential system requirements, alternative solutions, and estimated costs is not available when decisions are made to recommend acquisitions. Page 17 GAO/IMTEG90-7 Air Force ADP Requirements-setting Process Chapter 2 Air Force Prematurely Recommends Acquisition of ADP Systems In addition, acquiring command officials stated that they are not given sufficient funding to provide the resources needed to analyze require- ments, solutions, and costs during development of statements of opera- tional needs. One acquisition official acknowledged that the amount of funding would depend on the level of technological complexity and risk of the systems involved; he estimated that between $500,000 and $5 million might be needed to develop this information for most programs. Another official estimated that an average of about $1.6 million is needed to develop this information. However, funding to evaluate all statements of operational need at Air Force Space Systems Division has declined from $1.8 million for fiscal year 1988 to $780,000 in fiscal year 1989. One program official told us that this funding has been reduced to zero for fiscal year 1990. As discussed in this report, prior GAO and Defense studies have high- Defense Officials lighted inadequate requirements definition and evaluations of alterna- Recognize the Need to tive solutions as acquisition problems. These problems continue today. Conduct Early Officials at all levels within the Department of Defense and the Air Force have indicated an awareness that the requirements-setting pro- Evaluations cess needs improvement. For example, the Secretary of Defense’s July 1989 Defense Management Report to the President suggested further revisions to the process for establishing requirements to achieve the degree of improvement recommended by the Packard Commission in 1986. Further, a senior Joint Chiefs of Staff official has stated that the requirements process needs to be closely examined and that acquisition should not be initiated until all alternatives have been evaluated and a system concept has been defined. Finally, Air Force Systems Command proposed in March 1989 that the requirements process be changed to ensure that requirements are defined and alternative solutions are eval- uated before initiating an acquisition. 1989 Management Review In July 1989, the Secretary of Defense issued a Defense Management Report to the President report? to the President outlining a plan to improve the defense acquisi- tion process. The report acknowledges that the Department of Defense has not fully implemented the Packard Commission’s recommendation to conduct early high-level management reviews of requirements to assess the trade-off between cost and performance. The report identified further changes needed to realize improvements in the acquisition pro- cess to the degree contemplated by the Commission. In particular, the ‘Defense Management Report to the President, Department of Defense, July 1989. Page 18 GAO/lMTEG90-7 Air Force ADP Requirements-setting Process . Chapter 2 Air Force Prematurely Recommends Acquisition of ADP Systems report states that system acquisitions should not be initiated until suffi- cient information is gathered on alternative solutions. The report con- tains a plan to revise the requirements-setting process to ensure that systems are acquired at less cost, in less time, and with greater assur- ance of promised performance. Under this plan, the Secretary of Defense and Chairman of the Joint Chiefs of Staff will use the Joint Requirements Oversight Council to review deficiencies in current capabilities and the validity of identified mission needs, and develop a prioritized list of systems for review by the Defense Acquisition Board. The Board is the primary advisor to the Secretary of Defense on need, affordability, cost, and schedules for major system acquisitions. The systems are not to be funded until infor- mation on alternative solutions has been generated and a decision has been made on an acquisition approach. The Air Force is revising its reg- ulations to adopt the Secretary’s plan, which is to be implemented in 1990. Views of the Vice The Vice Chairman of the Joint Chiefs of Staff has made public his Chairman, Joint Chiefs of views on defense acquisition process deficiencies.2 The Vice Chairman said the services should not be driven to make early decisions; systems Staff get in trouble if minds are made up too early and commands are held to prematurely established (before system concept definition) dollar thresholds. He added that information on potential technologies and the cost to develop them cannot reasonably be developed until system requirements, alternative concepts, and alternative solutions have been explored. The Vice Chairman said that more attention must be given to exploring alternative concepts before an acquisition enters full-scale development. He added that acquisition initiation decisions should be based on the results of alternative concept evaluations and the demonstration that the proposed solution is valid. ‘“Hems Eyes Acquisition Fixes, To Delay ‘New Starts’ Beyond Milestone Zero,” Inside the Pentagon, October 14,19SS, pp. 47. Page 19 GAO/IM’IEG90-7 Air Force ADP Requirements-setting Process Chapter 2 Air Force Prematurely Recommends Acquisition of ADP Systems Air Force Systems On March 13,1989, Air Force Systems Command issued a draft paper which discusses deficiencies in the Air Force’s current requirements pro- Command RecognizesTha t cess and sets forth a proposal for improving the process. The proposal the - Requirements-setting -- _ endorses the findings of the earlier commissions3 and basically reiterates Process Needs the existing regulations. The proposal suggests that concept exploration, Improvement development activities, and trade-off analyses involving requirements and cost should occur before initiating acquisitions. Air Force Systems Command believes, and current regulations require, that acquisitions should begin only after a thorough analysis of the need by the com- mands that will acquire and use the system. The acquirer should be given sufficient time to analyze available technologies and alternative solutions during development of the sonr-well before initiating acquisi- tion. According to Air Force System Command’s proposal, the most sig- nificant change which needs to be made to the requirements process is to avoid the assumption that there is only one solution to an established need. Once a need is identified, the proposal states that analyses must be performed involving system performance, cost, and schedule to deter- mine the optimum operational capabilities, Under the proposal, concept exploration and development would take place before acquisitions are initiated to develop a better understanding of alternative solutions; to incorporate technological considerations in analyses of mission, requirements, solutions, and cost; and to identify the best solution to the stated problem. Alternatives should be defined in sufficient conceptual detail so that they can be considered in terms of technology, support, operations, maintenance, and life-cycle costs. In this and prior reviews, we identified Air Force ADP system problems Conclusions that began early in the acquisition cycle because requirements were not well defined and alternative solutions were not thoroughly evaluated. These problems occur even though Air Force regulations clearly state that system requirements should be defined and alternative solutions evaluated before the Air Force recommends acquisition of a system and competes it for funding in the Department of Defense budget process. Furthermore, since the early 1970s major independent commissions have made the same point and put Defense on notice that when system requirements are not comprehensively defined and alternative solutions are not fully evaluated, modifications are often necessary, resulting in cost increases, schedule delays, and performance problems. 3The Commission on Government procurement and the Packard Commission. Page 20 GAO/IMTEGSO-7 Air Force ADP Requirements-setting Process . Chapter 2 Air Force Prematurely Recommends Acquisition of ADP Systems In the cases we reviewed, the Air Force recommended ADP acquisitions before system requirements were adequately defined or alternative solu- tions to meet the need had been evaluated. Had the Air Force ade- quately evaluated alternative solutions, it would have been in a better position to identify the best solution in terms of achievability, cost, and schedule. However, the Air Force committed to acquisitions without a clear understanding of how to solve the problem, or any assurance that the system it is acquiring w-ill meet the stated need. We believe this com- mitment is premature. Furthermore, the failure of the Air Force to resolve problems after repeated notification indicates a lack of commit- ment to the process, and a lack of appreciation for its criticality. Requirements need to be thoroughly defined and alternatives evaluated before recommending an acquisition. Until this is done, future ADP sys- tem acquisitions can be expected to encounter problems similar to those discussed in this report-such as the Space Defense Operations Center 4’s $286 million overrun and 7-year delay. Especially in this time of budget constraints, the Air Force needs to take quick action to imple- ment its regulations. The Air Force cannot afford to initiate ADP acquisi- tions without solidifying system requirements and evaluating alternative solutions that firmly establish a system’s technical approach, design, and cost. To help reduce the number of costly and lengthy ADP acquisition pro- Recommendations to grams, we recommend that the Secretary of the Air Force quickly take the Secretary of the action to implement its regulations established to assure that system Air Force requirements are adequately defined and alternative solutions are eval- uated before approving and recommending acquisitions. These evalua- tions should consider technological advances and limitations; requirements achievability, reasonableness, and cost-effectiveness; and acquisition schedule and affordability so that the Air Force can assess whether the proposed acquisition will meet its needs. As part of this effort, the Secretary should pull back those programs competing for funding within the Department of Defense where requirements and alternative solutions have not been adequately defined and considered. The four proposed ADP acquisitions discussed in this report should be specifically pulled back and included in this reassessment. We requested official agency comments on a draft of this report from Agency Comments the Department of Defense. While official written comments have not Page 21 GAO/IMTEG!W7 Air Force ADP Reqnirementa-setthg Process . Chapter 2 Air Force Prematurely Recommends Acquisition of ADP Systems been provided, we met with agency officials to verify data presented in the report and have made revisions where appropriate. Page 22 GAO/IMTEG9O-7 Air Force ADP Requirements-setting Process Page 23 GAO/MTEC? Air Force ADP Reqnirementitting Process Appendix I Department of Defense Major Systems Acquisition Phases Mission area analysis and program initiation generally precede the five Department of Defense acquisition phases. Defense components continu- ally analyze their assigned mission areas to identify deficiencies (needs) and to determine if new systems or major upgrades to existing systems are necessary. These analyses often result in recommendations to initi- ate new acquisition programs through the validation of a need to correct the deficiency. Once a need has been identified and validated and Defense initiates an acquisition program, the program enters the concept formulation phase. In this phase, potential requirements and alternative approaches to sat- Concept Formulation isfy the need are identified and evaluated. Various types of analyses Phase considering trade-offs among performance, life-cycle cost, and schedule are conducted to select among possible concepts to satisfy the need. Once a concept has been identified, it is presented to Defense for approval. In this phase, feasibility and desirability of the selected requirements Demonstration and and the system concept is further analyzed, generally using techniques Validation Phase like computer simulation, hardware prototyping, developmental test and evaluation, operational test and evaluation, or a combination of test methods. When the feasibility of the concept has been convincingly demonstrated and validated, the program enters the full-scale engineer- ing and development phase. Page 24 GAO/IMTEGso-‘I Air Force ADP Requirements-setting Process , Appendix I Department of Defense MJor Systems Acquisition Phases In this phase, the system, including all of the items necessary for its Full-scale Engineering logistic and operational support, is designed, fabricated, and tested. At and Development the conclusion of this phase, the system is ready to be produced. Phase During this phase the proposed system is built and released to the user. Full-rate Production At this point, the system becomes operational. and Initial Deployment Phase This phase covers that period of time immediately following deployment Operations Support of the system and extends until the system is removed from Defense Phase inventory. Two major Defense reviews are conducted in this phase. The first takes place 1 to 2 years after deployment to determine if opera- tional readiness and support objectives are being achieved and main- tained. The second review occurs 5 to 10 years after deployment. It evaluates system capabilities and assesseswhether major upgrades are needed or if the system should be replaced. Page 26 GAO/IMTEC9O-7 Air Force ADP Requirements-setting Process Appendix II Cost Estimaks for Developing the Seven * ADP Systems Dollars in millions Recent cost Original cost cost estimate estimate increase Onaoina Svstems Command and Control Segment (prevrously called Data System Modernization) 557 195 362 Communications System Segment Replacement 422a 202 220 Space Defense Operations Center 4 576a 290 286 Subtotal 866 Proposed Systems Ballistic Mrssile Early Warning System 320 .SS;cemBased Atmospheric Surveillance 1.131b Space Surveillance System 1,446 Mission Plannina Svstem 53 Total 4.505 aEstimates Include costs for Research, Development, Test, and Evaluation, Other Procurement, and Operation and Maintenance. bEstlmates include costs through the demonstration and validation phase. They do not include costs for full-scale engineering and development. Page 26 GAO/IMTEG90-7 Air Force ADP Requhementxpsetting Process 5 Appendix III Major Contributors to This Report M ichael T. Blair, Assistant Director Inforrnation Ronald L. Hess, Evaluator Management and Teresa M . Schlee, W riter-Editor Technology Division, Washington, D.C. Frederick G. Day, Regional Management Representative Denver Regional Sigrid L. McGinty, Evaluator-in-Charge Office Luis Maez, Evaluator Peggy A. Stott, Evaluator Robert J. Dziekiewicz, Evaluator Boston Regional Office Page 27 GAO/IlMTEC90-7 Air Force ADP Reqnirement.asetting IVocess Related GAO Products Military Space Operations: Operational Problems Continue with the Sat- ellite Control Computer System (G~opm~G89-66, Aug. 8,1989). Attack Warning: Better Management Required to Resolve NORAD Inte- gration Deficiencies (GAO/IMTEG89-26, July 7, 1989). Space Defense: Management and Technical Problems Delay Operations Center Acquisition (GAO/IMTEC-89-18, Apr. 20, 1989). Attack Warning: NORAD’s Communications System Segment Replace- ment Program Should Be Reassessed (GAO/IMTEC89-1, Nov. 30, 1988). Military Space Operations: Shuttle and Satellite Computer Systems Do Not Meet Performance Objectives (GAO/IMW&7, Aug. 5,1988). (310339) Page 28
Defense Acquisition: Air Force Prematurely Recommends ADP Acquisitions
Published by the Government Accountability Office on 1990-03-29.
Below is a raw (and likely hideous) rendition of the original report. (PDF)