United States General Accounting Office Report to Congressional Requesters - AIR TRAFFIC CONTROL Continuing Delays Anticipated for the i Advmced Automation System GAO United States General Accounting Office Washington, D.C. 20548 Information Management and Technology Division B-239567 July 18, 1990 The Honorable Frank R. Lautenberg Chairman, Subcommittee on Transportation and Related Agencies Committee on Appropriations IJnited States Senate The Honorable William Lehman Chairman, Subcommittee on Transportation and Related Agencies Committee on Appropriations House of Representatives In response to your request, we reviewed the Federal Aviation Adminis- tration’s (FAA) efforts to develop the Advanced Automation System (AAS). AAS, estimated to cost approximately $5 billion, is intended to replace aging air traffic control computer systems with new hardware, software, and controller workstations. In 1988, FAA awarded a $3.6-billion contract to International Business Machines (IBM), Inc. to complete the design and production of AAS.The first key phase of AAS. the initial sector suite system (ISSS),is to replace controller workstations at facilities that control air traffic between air- ports. As agreed with your offices, our objective was to evaluate whether FAA was effectively managing ISSSin order to minimize program delays. An explanation of our scope and methodology is contained in appendix I. Less than a year after IBM began work on the contract, I~SS implementa- Results in Brief tion had already encountered a 13-month delay because not all require- ments issues had been resolved, and FAA and IBM had underestimated the time it would take to develop and test software. However, the revised 13.month schedule does not include sufficient time to resolve require- ments issues and allocates little time to resolve problems that may arise during system testing. Consequently, it is likely that ISSSwill be delayed even longer. In commenting on a draft of this report, FAAofficials stated that they now anticipate that the ES will be delayed by a total of 19 months. Page 1 GAO/lMTJX-90.63 Air Traffic Control Automation Delays B-239567 Under FAA’S current plan, the second step of the AASis to provide addi- tional hardware and software to support terminal automation capabili- ties. This component, which is to be installed in existing en route facilities, is to allow FAA to consolidate some smaller terminal facilities into en route centers. The third step of AASis to provide additional auto- mation support in airport towers and is to allow for the consolidation of the remaining large terminal control facilities at en route facilities. FAA has about 60 headquarters employees, including 43 technical staff, dedicated to the AA.5 project. FAA also has six support contractors with over 340 personnel who provide technical and managerial guidance to FXA and monitor the efforts of IBM. IBM has over 1500 people working on the project. In July 1989, only 8 months after beginning work, an FAA and IBM task ISSSImplementation force reported a minimum of a lo-month delay in the ES software Has Been Delayed schedule. By October 1989, IBM had amended this projection to a 13- month delay. Table I shows the original schedule and the schedule with the l&month delay. Table 1: ISSS Acquisition Schedule Revised schedule Milestones Original date date Factory acceptance testing completed -~- January 2, 1991 February 2, 1992 FAA Technfcal Center acceptance tesirng completed November _____ 1, 1991 December 1, 1992 FAA Technrcal Center operatronal testing and evaluation completed ~.~~ __- August 3. 1992 September__-3, 1993 Frrst Sate operatronal readfness demonstrationa January 4, 1994 February 4, 1995 Last Sate operational readiness demonstration October 4, 1995 November 4, 1996 aA system IS deemed operatronally ready when It can perform requrred functrons In April 1990, FAA directed IBM to baseline its schedule to incorporate the 13-month delay. However, the contract has not yet been modified to reflect this delay. FAA anticipates that the contract will be modified to include all schedule delays and requirements changes in December 1990. Page 3 GAO/lMTEGSO43 Air Traffic Control Automation Delays Es239567 The announced 13-month delay probably underestimates how long I%% Further - _ -.-Schedule _ will be delayed. Additional delays are likely because requirements issues Delays Likely from the design competition phase have still not been resolved and new requirements have been identified. In addition, little time has been allo- cated for taking corrective action resulting from testing. Many Requirements Issues FAA and IBM have made some progress in addressing the unresolved requirements issues from the design competition phase. In November Are Still Not Resolved 1989, FAA and IBM had approximately 135 unresolved ISS requirements issues. By April 1990, they had reduced this number to about 45. Among the requirements issues still open were determining how to properly dis- play tabular data and aeronautical charts on controller screens. In addition, FAA has begun to identify several new requirements that threaten to further delay the schedule. For example, FAA has identified a requirement for sector-by-sector transition. This requirement would allow ISSS to be deployed at a center one sector at a time rather than a total one-time change from the old control room to the new control room. Sector-by-sector transition is expected to reduce operational risk to the ISSS by enabling some sectors to be supported by the new system while other sectors are supported by existing equipment. Currently, FAA and IBM are discussing the need for additional time to satisfy new require- ments. PAAanticipates that in December 1990 the contract will be modi- fied to include additional time to satisfy these new requirements, as well as to reflect schedule changes. Schedule Allocates 1iittle To better define the design and the requirements and thus reduce risk, FAA and IBM planned to perform early demonstrations of software capa- Time for Corrective Action bilities. However, to date, the benefits of these early demonstrations have not been fully realized because the demonstrations have been nar- rowly focused or deferred due to software delays. By failing to run early demonstrations as planned, IBM may not identify problems and resolve them until the formal testing phase, when they are more difficult and time consuming t.0 fix. Given that only 10 months is allocated for formal software qualification testing and corrective action, IBM may not have allocated enough time to fix problems revealed through testing. In addition, IBM’S planned software qualification test schedule includes many overlapping activi- ties and interdependencies, and a slip in one activity could delay the entire ISSS schedule. Page 5 GAO/IMTEG904%3 Air Traffic Control Automation Delays B-239567 However, regarding the need to explore alternatives, the Department of Transportation believed that FAA had already developed an appropriate interim solution in 1987 to meet TRACON requirements for the next 10 years. This interim solution calls for increasing the capacity of the pre- sent systems in large TRACONS by pursuing sole-source contracts to expand current system configurations to their maximum design limits. This expansion will require FAA to buy 1960s-vintage computers similar to existing processors. These antiquated processors have less processing capability than a desktop computer purchased in a local store. To meet its immediate needs for additional computer capacity, FAA may have little choice other than to upgrade the existing systems. However, because larger TRACONS may not be modernized until 2000 or beyond, the limited capabilities of this interim solution may not be able to handle long-term automation requirements and the continued growth in air traffic. The development of the ES, the first step in FAA'S plan to modernize air Conclusions traffic control computer systems, is behind schedule by at least 13 months and will likely be delayed much longer. The delay resulted because FAA and IBM failed to resolve requirements issues and underesti- mated the program’s complexity. The 13-month extension that FAA and IBM have added to the baseline schedule does not adequately consider the time required to resolve remaining requirements. Further, little time has been allotted for resolving problems that may arise from system testing. Under FAA's current plan, the ISSS delay will delay the remaining phases of AASthat are scheduled to replace current automated systems at TRACONS. This could have significant repercussions on the safety of the air traffic control system, since some large TRACON automated systems are already overloaded. The longer these aging systems are maintained, the greater the danger that additional shortfalls will occur. Given the delay in AASand the clear inadequacies of the existing computer systems in large TRACONS, FAA may not have the needed automation capabilities in time to handle the increasing air traffic of the 1990s. - We recommend that the Secretary of Transportation direct the FAA Recommendations Administrator to establish, with IBM, a new and realistic schedule for AASdevelopment and delivery. An analysis should be conducted immedi- ately that assesses remaining tasks and determines realistic timeframes B-239667 Secretary of Transportation, the FAA Administrator, and other interested parties, and will make copies available to others upon request. This report was prepared under the direction of JayEtta Hecker, Director, Resources, Community, and Economic Development Information Sys- tems, who can be reached at (202) 275-9675. Other major contributors are listed in appendix II. LQ%L Ralph V. Carlone Assistant Comptroller General Page 9 Appendix II Major Contributors to This Report .Joel C. Willemssen, Assistant Director Information Marcia C. Washington, Evaluator-in-Charge Management and Susan Maciorowski. Presidential Exchange Executive Technology Division, Andrea M. Leopold, Technical Adviser Margaret W. Price, Evaluator Washington, D.C. Lisa Pittelkau, Evaluator (610457) Page 1 I GAODMTEC90-63 Air Traffic Control Automation Delays Requests for copies of GAO reports should be sent to: U.S. General Accounting Office Post Office Box 6015 Gaithersburg, Maryland 20877 Telephone 202-275-6241 The first five copies of each report are free. Additional copies are $2.00 each. There is a 25% discount on orders for 100 or more copies mailed to a single address. Orders must be prepaid by cash or by check or money order made out to the ~BiIZH&malro!r~~~rde.- Appendix I Scopeand Methodology To accomplish our objective, we examined contract documents from the program’s design competition phase to the acquisition phase and ana- lyzed completed and remaining tasks. We examined FAA’s AASprogram management structure and the roles and responsibilities of the develop- ment and support contractors. We reviewed the acquisition phase con- tract, critical design memoranda, and system specifications. We also reviewed support contractor monthly status reports and correspondence to obtain information on m contractor performance. Further, we ana- lyzed information on the estimated development schedules versus actual completion time of systems of comparable difficulty. Finally, we inter- viewed officials from FAA’S AASprogram office, IBM, and Martin Marietta to obtain their views on program development and the causes of schedule delays. Our work was performed from August 1989 to April 1990 at FAA head- quarters, Washington, D.C., IBM, Rockville, Maryland, and Martin Mari- etta, Inc., Washington, D.C. The views of agency and contractor officials were obtained during the course of our work and have been incorpo- rated where appropriate. In addition, at the completion of our review, we discussed the report’s key facts, conclusions, and recommendations with FAA officials. Finally, we obtained formal oral comments from Department of Transportation and FAA officials on a draft of this report. These comments and our analysis are also included in this report. We conducted our review in accordance with generally accepted auditing standards. Page 10 GAO/IMTEGOO63 Air Tmfiic Control Autmnntion Delays B-239567 for IBM to complete the development and delivery of ISSSas well as the remainder of AAS.The analysis should include an appropriate safety factor, such as the time needed to conduct retesting and tuning of the system to meet performance requirements. The analysis should also explore the feasibility of revising the order of AAS implementation to expedite modernization of larger TRACONS. We obtained official oral comments from the Department of Transporta- Agency Comments and tion and FAA officials on a draft of this report. The officials agreed that Our Evaluation one reason the ISSSacquisition schedule slipped so soon after contract award was unresolved requirements issues. However, they stated that IBM'S performance on the contract was the key factor in the delay. Offi- cials added that, because of their concerns about contractor perform- ance, they requested IRM to reevaluate the ISSS schedule shortly after work commenced. Officials also commented that the contract schedule was mutually agreed upon by FAA and the contractor, and that IBM offi- cials did not express concerns at the time about the schedule being unrealistic. Officials stated that, in order to resolve open issues and to incorporate new requirements, they now intend to delay the ISSS schedule by a total of 19 months. They believe this schedule is realistic and provides sufficient time to continually test the ES throughout development. As presented in the report, we believe that unresolved requirements issues and an unrealistic schedule were key factors causing the delay. While recognizing FAA'S revised estimate of a 19-month ISSSdelay, FAA still needs to assess remaining open tasks in order to develop a basis for a realistic schedule that provides sufficient time to allow for problems that may result from software qualification testing. As arranged with your offices, unless you publicly announce the con- tents of this report earlier, we plan no further distribution of this report until 30 days after the date of this letter. We will then send copies to the Pa&?8 B-239667 One example of where additional time may be needed is in modifying the system to meet the stringent processing load and system response time requirements of IS&S.Real-time systems such as the ISSS are much more difficult to build successfully within budget and schedule than systems without severe performance requirements. At present, according to the contractor, IBM performance models indicate that the selected software design will meet system requirements. However, model results can be inaccurate; models may indicate that a design will meet requirements, but once the system is built, it may not. For example, on a complex, real- time Navy program,:] the initial version of the local communications net- work performed only one-sixth as fast as the model predicted resulting in adverse cost, schedule, and performance impacts on the Navy’s plans to improve its submarine command and control systems. According to FAA officials, FAA is anticipating that the entire AAS ISSSDelays Will Delay schedule will be delayed an amount of time comparable to the lsss delay. Other AAS Phases 18% deployment is critical to the development of other AAS phases because later phases require the new controller workstations and software. Delays in A.M may impair MA’S large TRACON facilities, since the longer that AASis delayed, the longer FAA will have to operate and maintain current systems. As we reported last year, many of the automated systems at these facili- ties were experiencing capacity shortfalls4 Indeed, almost 70 percent of the large busy TRACONS surveyed reported that they had experienced air- craft information disappearing from controllers’ screens, flickering dis- plays, or delayed computer responses to controllers’ attempts to update or request data. These overload problems threaten the ability of control- lers to track and safely handle aircraft. To address this dilemma, last year we recommended that FAA immedi- ately fix those facilities experiencing the worst shortfalls, institute a capacity management and performance program to monitor work loads and system utilization in all facilities, and investigate different alterna- tives for meeting the larger TRACONS’ air traffic control requirements until AASbecomes available. Since then, FAA has initiated some steps to remedy capacity deficiencies. 3SUBAC5 Problems May Adversely Affect Navy Attack Submarine Programs (GAO/NSIAD-86-12, Nov. 4, 1986). ‘Air Traffic Control: Computer Capacity Shortfalls May Impair Flight Safety (GAO/IMTEC-89-63, July 6, 1989). Page 6 GAO/EUl’EGVO83 Air Traffic Control Automation Delays Delay Caused by Two major reasons for the 13-month delay were (1) requirements issues scheduled to be resolved during the design competition phase were Unresolved Requirements deferred to the acquisition phase without allocating any time to resolve Issues and Inadequate the issues, and (2) FAA underestimated the time it would take to develop Schedule Estimates and test the software. During the design competition phase, IBM and Hughes were supposed to conduct a thorough analysis of requirements, including producing detailed design documentation, and conducting design reviews. However, FAA allowed the contractors to proceed with major design reviews without completing requirements analyses and with incomplete documentation. Thus, the design reviews were incom- plete. For example, IBM’S analysis did not address all specification requirements, such as those associated with the ability to electronically process and display flight plan data and aeronautical charts. Some of these specification requirements have, to date, still not been finalized. Despite these open requirements issues, FAA determined that the work completed was sufficient and awarded the acquisition contract to IBM. As a result, IBM proceeded with software development based on require- ments that were not thoroughly analyzed and finalized and based on incomplete designs and specifications. FAA did not add more time to the acquisition contract schedule to reflect the work needed to be done to analyze all requirements and to complete the design. ISSSis a complex effort that, according to FAA, requires the development of approximately one million lines of software using Ada, a relatively new programming language. Originally, IBM planned the ISSS effort to be completed in six software builds’. However, IBM subsequently deter- mined that this projection was inadequate. Now, IBM has projected that an additional build and 7 additional months will be needed to complete the software. According to IBM officials, FAA set the overall schedule and did not allow contract bidders to offer their own schedule. IBM officials admitted that they considered the acquisition schedule to be unrealistic. However, they simply accepted the challenge of trying to meet an overly ambitious schedule. In response, FAA officials stated that IBM agreed to the schedule and did not express reservations at the time about the difficulty of meeting the schedule. ‘A build is an incremental portwn of software intended to perform a specific subset of functions of the total system. Page 4 GAO/lMTEG90-63 Air Traffic Control Automation Delays E-239667 ES delays will push back later AAS phases scheduled to replace aging automated systems at terminal radar approach control (TRACON) facili- ties. This could affect air traffic safety, since large TRACON automated systems have already experienced computer capacity shortfalls resulting in data flickering and disappearing from controllers’ screens. Because of the delays in AASand the limitations of the existing TRACON computer systems, FAA may not have the needed automation capabilities in time to handle the increasing air traffic of the 1990s. MA intends to automate and modernize the nation’s air traffic control Background system through its National Airspace System Plan. AASis the centerpiece of this plan and is being acquired to increase controller productivity, reduce operating costs, save fuel and passenger time, and allow control- lers to handle anticipated traffic increases more safely and efficiently. Improvements are expected to result primarily from (1) the use of modern equipment and (2) the development of new software functions intended to automate some controller functions and allow more aircraft to fly user-preferred, fuel-efficient routes. To design AAS, FAA awarded competitive contracts to IBM and Hughes Aircraft Corporation in 1984. On July 25, 1988, after spending about $500 million on the design competition, FAA awarded a $3.6-billion con- tract to IBM to complete the design and produce the AAS. However, on August 10, 1988, a stop work order was issued to IBM as a result of a protest filed by Hughes Aircraft Corporation. On October 28, 1988, the General Services Administration’s Board of Contract Appeals issued a decision upholding FAA’Scontract award. IBM resumed work on AAS in November 1988. AAS Implementation The AAScontract calls for implementing the system in three steps. Approach Deployment of the first step is to begin in 1993, the second in 1995, and the third in 1996, According to FAA officials, the system is to be com- pleted by 2003 with contract options extending until 2010. ISSS is the primary component of the first step and constitutes the largest portion of the AAS program. It is to supply new controller work- stations at en route centers’ to replace existing controller displays, and automate some related processes that are currently done manually. ’ FAA currently maintains 20 Ar Hout~ Traffic Control Centers m the continental United States that rontrol air traffic between airport.5 Page2 GAO/lMTEG90-63 Air Traffic Control Automation Delays
Air Traffic Control: Continuing Delays Anticipated for the Advanced Automation System
Published by the Government Accountability Office on 1990-07-18.
Below is a raw (and likely hideous) rendition of the original report. (PDF)