oversight

Air Traffic Control: Continuing Delays Anticipated for the Advanced Automation System

Published by the Government Accountability Office on 1990-07-18.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

    United   States   General   Accounting   Office
    Report to Congressional Requesters                -




    AIR TRAFFIC
    CONTROL
    Continuing Delays
    Anticipated for the
i   Advmced Automation
    System
GAO
                   United States
                   General Accounting  Office
                   Washington, D.C. 20548

                   Information    Management    and
                   Technology    Division

                   B-239567

                   July 18, 1990

                   The Honorable Frank R. Lautenberg
                   Chairman, Subcommittee on Transportation
                      and Related Agencies
                   Committee on Appropriations
                   IJnited States Senate

                   The Honorable William Lehman
                   Chairman, Subcommittee on Transportation
                     and Related Agencies
                   Committee on Appropriations
                   House of Representatives

                   In response to your request, we reviewed the Federal Aviation Adminis-
                   tration’s (FAA) efforts to develop the Advanced Automation System
                   (AAS). AAS, estimated to cost approximately $5 billion, is intended to
                   replace aging air traffic control computer systems with new hardware,
                   software, and controller workstations.

                   In 1988, FAA awarded a $3.6-billion contract to International Business
                   Machines (IBM), Inc. to complete the design and production of AAS.The
                   first key phase of AAS. the initial sector suite system (ISSS),is to replace
                   controller workstations at facilities that control air traffic between air-
                   ports. As agreed with your offices, our objective was to evaluate
                   whether FAA was effectively managing ISSSin order to minimize program
                   delays. An explanation of our scope and methodology is contained in
                   appendix I.


                   Less than a year after IBM began work on the contract, I~SS implementa-
Results in Brief   tion had already encountered a 13-month delay because not all require-
                   ments issues had been resolved, and FAA and IBM had underestimated the
                   time it would take to develop and test software. However, the revised
                   13.month schedule does not include sufficient time to resolve require-
                   ments issues and allocates little time to resolve problems that may arise
                   during system testing. Consequently, it is likely that ISSSwill be delayed
                   even longer. In commenting on a draft of this report, FAAofficials stated
                   that they now anticipate that the ES will be delayed by a total of 19
                   months.




                   Page 1                             GAO/lMTJX-90.63   Air Traffic   Control   Automation   Delays
                                     B-239567




                                     Under FAA’S current plan, the second step of the AASis to provide addi-
                                     tional hardware and software to support terminal automation capabili-
                                     ties. This component, which is to be installed in existing en route
                                     facilities, is to allow FAA to consolidate some smaller terminal facilities
                                     into en route centers. The third step of AASis to provide additional auto-
                                     mation support in airport towers and is to allow for the consolidation of
                                     the remaining large terminal control facilities at en route facilities.

                                     FAA has about 60 headquarters employees, including 43 technical staff,
                                     dedicated to the AA.5 project. FAA also has six support contractors with
                                     over 340 personnel who provide technical and managerial guidance to
                                     FXA and monitor the efforts of IBM. IBM has over 1500 people working on
                                     the project.


                                     In July 1989, only 8 months after beginning work, an FAA and IBM task
ISSSImplementation                   force reported a minimum of a lo-month delay in the ES software
Has Been Delayed                     schedule. By October 1989, IBM had amended this projection to a 13-
                                     month delay. Table I shows the original schedule and the schedule with
                                     the l&month delay.

Table 1: ISSS Acquisition Schedule
                                                                                                                        Revised schedule
                                     Milestones                                                  Original date                      date
                                     Factory acceptance testing completed          -~-         January 2, 1991            February 2, 1992
                                     FAA Technfcal Center acceptance tesirng
                                     completed                                              November
                                                                                                _____ 1, 1991            December 1, 1992
                                     FAA Technrcal Center operatronal testing and
                                     evaluation completed            ~.~~ __-                   August 3. 1992          September__-3, 1993
                                     Frrst Sate operatronal readfness
                                     demonstrationa                                            January 4, 1994            February 4, 1995
                                     Last Sate operational readiness demonstration             October 4, 1995           November 4, 1996
                                     aA system IS deemed operatronally ready when It can perform requrred functrons


                                     In April 1990, FAA directed IBM to baseline its schedule to incorporate the
                                     13-month delay. However, the contract has not yet been modified to
                                     reflect this delay. FAA anticipates that the contract will be modified to
                                     include all schedule delays and requirements changes in December 1990.




                                     Page 3                                 GAO/lMTEGSO43       Air Traffic   Control   Automation   Delays
                             Es239567




                             The announced 13-month delay probably underestimates how long I%%
Further
- _ -.-Schedule
             _               will be delayed. Additional delays are likely because requirements issues
Delays Likely                from the design competition phase have still not been resolved and new
                             requirements have been identified. In addition, little time has been allo-
                             cated for taking corrective action resulting from testing.


Many Requirements Issues     FAA and IBM have made some progress in addressing the unresolved
                             requirements issues from the design competition phase. In November
Are Still Not Resolved        1989, FAA and IBM had approximately 135 unresolved ISS requirements
                             issues. By April 1990, they had reduced this number to about 45. Among
                             the requirements issues still open were determining how to properly dis-
                             play tabular data and aeronautical charts on controller screens.

                             In addition, FAA has begun to identify several new requirements that
                             threaten to further delay the schedule. For example, FAA has identified a
                             requirement for sector-by-sector transition. This requirement would
                             allow ISSS to be deployed at a center one sector at a time rather than a
                             total one-time change from the old control room to the new control room.
                             Sector-by-sector transition is expected to reduce operational risk to the
                             ISSS by enabling some sectors to be supported by the new system while
                             other sectors are supported by existing equipment. Currently, FAA and
                             IBM are discussing the need for additional time to satisfy new require-
                             ments. PAAanticipates that in December 1990 the contract will be modi-
                             fied to include additional time to satisfy these new requirements, as well
                             as to reflect schedule changes.


Schedule Allocates 1iittle   To better define the design and the requirements and thus reduce risk,
                             FAA and IBM planned to perform early demonstrations of software capa-
Time for Corrective Action   bilities. However, to date, the benefits of these early demonstrations
                             have not been fully realized because the demonstrations have been nar-
                             rowly focused or deferred due to software delays. By failing to run early
                             demonstrations as planned, IBM may not identify problems and resolve
                             them until the formal testing phase, when they are more difficult and
                             time consuming t.0 fix.

                              Given that only 10 months is allocated for formal software qualification
                              testing and corrective action, IBM may not have allocated enough time to
                              fix problems revealed through testing. In addition, IBM’S planned
                              software qualification test schedule includes many overlapping activi-
                              ties and interdependencies, and a slip in one activity could delay the
                              entire ISSS schedule.


                              Page 5                      GAO/IMTEG904%3   Air Traffic   Control   Automation   Delays
                  B-239567




                  However, regarding the need to explore alternatives, the Department of
                  Transportation believed that FAA had already developed an appropriate
                  interim solution in 1987 to meet TRACON requirements for the next 10
                  years. This interim solution calls for increasing the capacity of the pre-
                  sent systems in large TRACONS by pursuing sole-source contracts to
                  expand current system configurations to their maximum design limits.
                  This expansion will require FAA to buy 1960s-vintage computers similar
                  to existing processors. These antiquated processors have less processing
                  capability than a desktop computer purchased in a local store.

                  To meet its immediate needs for additional computer capacity, FAA may
                  have little choice other than to upgrade the existing systems. However,
                  because larger TRACONS may not be modernized until 2000 or beyond, the
                  limited capabilities of this interim solution may not be able to handle
                  long-term automation requirements and the continued growth in air
                  traffic.


                  The development of the ES, the first step in FAA'S plan to modernize air
Conclusions       traffic control computer systems, is behind schedule by at least 13
                  months and will likely be delayed much longer. The delay resulted
                  because FAA and IBM failed to resolve requirements issues and underesti-
                  mated the program’s complexity. The 13-month extension that FAA and
                  IBM have added to the baseline schedule does not adequately consider
                  the time required to resolve remaining requirements. Further, little time
                  has been allotted for resolving problems that may arise from system
                  testing.

                  Under FAA's current plan, the ISSS delay will delay the remaining phases
                  of AASthat are scheduled to replace current automated systems at
                  TRACONS. This could have significant repercussions on the safety of the
                  air traffic control system, since some large TRACON automated systems
                  are already overloaded. The longer these aging systems are maintained,
                  the greater the danger that additional shortfalls will occur. Given the
                  delay in AASand the clear inadequacies of the existing computer systems
                  in large TRACONS, FAA may not have the needed automation capabilities in
                  time to handle the increasing air traffic of the 1990s.

                                       -
                   We recommend that the Secretary of Transportation direct the FAA
Recommendations    Administrator to establish, with IBM, a new and realistic schedule for
                   AASdevelopment and delivery. An analysis should be conducted immedi-
                   ately that assesses remaining tasks and determines realistic timeframes
B-239667




Secretary of Transportation, the FAA Administrator, and other interested
parties, and will make copies available to others upon request. This
report was prepared under the direction of JayEtta Hecker, Director,
Resources, Community, and Economic Development Information Sys-
tems, who can be reached at (202) 275-9675. Other major contributors
are listed in appendix II.



LQ%L
Ralph V. Carlone
Assistant Comptroller   General




Page 9
Appendix II

Major Contributors to This Report


                       .Joel C. Willemssen, Assistant Director
Information            Marcia C. Washington, Evaluator-in-Charge
Management and         Susan Maciorowski. Presidential Exchange Executive
Technology Division,   Andrea M. Leopold, Technical Adviser
                       Margaret W. Price, Evaluator
Washington, D.C.       Lisa Pittelkau, Evaluator




(610457)               Page 1 I                   GAODMTEC90-63   Air Traffic   Control   Automation   Delays
Requests for copies of     GAO   reports   should be sent to:

U.S. General Accounting      Office
Post Office Box 6015
Gaithersburg, Maryland      20877

Telephone   202-275-6241

The first five copies of each report       are free. Additional   copies are
$2.00 each.

There is a 25% discount     on orders for 100 or more copies mailed to a
single address.

Orders must be prepaid by cash or by check or money order made
out to the ~BiIZH&malro!r~~~rde.-
Appendix I

Scopeand Methodology


             To accomplish our objective, we examined contract documents from the
             program’s design competition phase to the acquisition phase and ana-
             lyzed completed and remaining tasks. We examined FAA’s AASprogram
             management structure and the roles and responsibilities of the develop-
             ment and support contractors. We reviewed the acquisition phase con-
             tract, critical design memoranda, and system specifications. We also
             reviewed support contractor monthly status reports and correspondence
             to obtain information on m contractor performance. Further, we ana-
             lyzed information on the estimated development schedules versus actual
             completion time of systems of comparable difficulty. Finally, we inter-
             viewed officials from FAA’S AASprogram office, IBM, and Martin Marietta
             to obtain their views on program development and the causes of
             schedule delays.

             Our work was performed from August 1989 to April 1990 at FAA head-
             quarters, Washington, D.C., IBM, Rockville, Maryland, and Martin Mari-
             etta, Inc., Washington, D.C. The views of agency and contractor officials
             were obtained during the course of our work and have been incorpo-
             rated where appropriate. In addition, at the completion of our review,
             we discussed the report’s key facts, conclusions, and recommendations
             with FAA officials. Finally, we obtained formal oral comments from
             Department of Transportation and FAA officials on a draft of this report.
             These comments and our analysis are also included in this report. We
             conducted our review in accordance with generally accepted auditing
             standards.




              Page 10                     GAO/IMTEGOO63   Air Tmfiic   Control   Autmnntion   Delays
                      B-239567




                      for IBM to complete the development and delivery of ISSSas well as the
                      remainder of AAS.The analysis should include an appropriate safety
                      factor, such as the time needed to conduct retesting and tuning of the
                      system to meet performance requirements. The analysis should also
                      explore the feasibility of revising the order of AAS implementation to
                      expedite modernization of larger TRACONS.


                      We obtained official oral comments from the Department of Transporta-
Agency Comments and   tion and FAA officials on a draft of this report. The officials agreed that
Our Evaluation        one reason the ISSSacquisition schedule slipped so soon after contract
                      award was unresolved requirements issues. However, they stated that
                      IBM'S performance on the contract was the key factor in the delay. Offi-
                      cials added that, because of their concerns about contractor perform-
                      ance, they requested IRM to reevaluate the ISSS schedule shortly after
                      work commenced. Officials also commented that the contract schedule
                      was mutually agreed upon by FAA and the contractor, and that IBM offi-
                      cials did not express concerns at the time about the schedule being
                      unrealistic. Officials stated that, in order to resolve open issues and to
                      incorporate new requirements, they now intend to delay the ISSS
                       schedule by a total of 19 months. They believe this schedule is realistic
                       and provides sufficient time to continually test the ES throughout
                       development.

                       As presented in the report, we believe that unresolved requirements
                       issues and an unrealistic schedule were key factors causing the delay.
                       While recognizing FAA'S revised estimate of a 19-month ISSSdelay, FAA
                       still needs to assess remaining open tasks in order to develop a basis for
                       a realistic schedule that provides sufficient time to allow for problems
                       that may result from software qualification testing.



                       As arranged with your offices, unless you publicly announce the con-
                       tents of this report earlier, we plan no further distribution of this report
                       until 30 days after the date of this letter. We will then send copies to the




                       Pa&?8
                        B-239667




                        One example of where additional time may be needed is in modifying the
                        system to meet the stringent processing load and system response time
                        requirements of IS&S.Real-time systems such as the ISSS are much more
                        difficult to build successfully within budget and schedule than systems
                        without severe performance requirements. At present, according to the
                        contractor, IBM performance models indicate that the selected software
                        design will meet system requirements. However, model results can be
                        inaccurate; models may indicate that a design will meet requirements,
                        but once the system is built, it may not. For example, on a complex, real-
                        time Navy program,:] the initial version of the local communications net-
                        work performed only one-sixth as fast as the model predicted resulting
                        in adverse cost, schedule, and performance impacts on the Navy’s plans
                        to improve its submarine command and control systems.


                        According to FAA officials, FAA is anticipating that the entire AAS
ISSSDelays Will Delay   schedule will be delayed an amount of time comparable to the lsss delay.
Other AAS Phases        18% deployment is critical to the development of other AAS phases
                        because later phases require the new controller workstations and
                        software. Delays in A.M may impair MA’S large TRACON facilities, since
                        the longer that AASis delayed, the longer FAA will have to operate and
                        maintain current systems.

                        As we reported last year, many of the automated systems at these facili-
                        ties were experiencing capacity shortfalls4 Indeed, almost 70 percent of
                        the large busy TRACONS surveyed reported that they had experienced air-
                        craft information disappearing from controllers’ screens, flickering dis-
                        plays, or delayed computer responses to controllers’ attempts to update
                        or request data. These overload problems threaten the ability of control-
                        lers to track and safely handle aircraft.

                        To address this dilemma, last year we recommended that FAA immedi-
                        ately fix those facilities experiencing the worst shortfalls, institute a
                        capacity management and performance program to monitor work loads
                        and system utilization in all facilities, and investigate different alterna-
                        tives for meeting the larger TRACONS’ air traffic control requirements
                        until AASbecomes available. Since then, FAA has initiated some steps to
                        remedy capacity deficiencies.

                        3SUBAC5 Problems May Adversely      Affect Navy Attack Submarine Programs (GAO/NSIAD-86-12,
                        Nov. 4, 1986).

                        ‘Air Traffic Control: Computer Capacity Shortfalls   May Impair Flight Safety (GAO/IMTEC-89-63,
                        July 6, 1989).




                        Page 6                                  GAO/EUl’EGVO83       Air Traffic   Control   Automation   Delays
Delay Caused by           Two major reasons for the 13-month delay were (1) requirements issues
                          scheduled to be resolved during the design competition phase were
Unresolved Requirements   deferred to the acquisition phase without allocating any time to resolve
Issues and Inadequate     the issues, and (2) FAA underestimated the time it would take to develop
Schedule Estimates        and test the software. During the design competition phase, IBM and
                          Hughes were supposed to conduct a thorough analysis of requirements,
                          including producing detailed design documentation, and conducting
                          design reviews. However, FAA allowed the contractors to proceed with
                          major design reviews without completing requirements analyses and
                          with incomplete documentation. Thus, the design reviews were incom-
                          plete. For example, IBM’S analysis did not address all specification
                           requirements, such as those associated with the ability to electronically
                           process and display flight plan data and aeronautical charts. Some of
                           these specification requirements have, to date, still not been finalized.

                          Despite these open requirements issues, FAA determined that the work
                          completed was sufficient and awarded the acquisition contract to IBM.
                          As a result, IBM proceeded with software development based on require-
                          ments that were not thoroughly analyzed and finalized and based on
                          incomplete designs and specifications. FAA did not add more time to the
                          acquisition contract schedule to reflect the work needed to be done to
                          analyze all requirements and to complete the design.

                          ISSSis a complex effort that, according to FAA, requires the development
                          of approximately one million lines of software using Ada, a relatively
                          new programming language. Originally, IBM planned the ISSS effort to be
                          completed in six software builds’. However, IBM subsequently deter-
                          mined that this projection was inadequate. Now, IBM has projected that
                          an additional build and 7 additional months will be needed to complete
                          the software.

                          According to IBM officials, FAA set the overall schedule and did not allow
                          contract bidders to offer their own schedule. IBM officials admitted that
                          they considered the acquisition schedule to be unrealistic. However,
                          they simply accepted the challenge of trying to meet an overly ambitious
                          schedule. In response, FAA officials stated that IBM agreed to the schedule
                          and did not express reservations at the time about the difficulty of
                          meeting the schedule.



                           ‘A build is an incremental   portwn of software   intended to perform a specific subset of functions of
                           the total system.




                           Page 4                                    GAO/lMTEG90-63        Air Traffic   Control   Automation   Delays
                     E-239667




                     ES   delays will push back later AAS phases scheduled to replace aging
                     automated systems at terminal radar approach control (TRACON) facili-
                     ties. This could affect air traffic safety, since large TRACON automated
                     systems have already experienced computer capacity shortfalls
                     resulting in data flickering and disappearing from controllers’ screens.
                     Because of the delays in AASand the limitations of the existing TRACON
                     computer systems, FAA may not have the needed automation capabilities
                     in time to handle the increasing air traffic of the 1990s.


                     MA intends to automate and modernize the nation’s air traffic control
Background           system through its National Airspace System Plan. AASis the centerpiece
                     of this plan and is being acquired to increase controller productivity,
                     reduce operating costs, save fuel and passenger time, and allow control-
                     lers to handle anticipated traffic increases more safely and efficiently.
                     Improvements are expected to result primarily from (1) the use of
                     modern equipment and (2) the development of new software functions
                     intended to automate some controller functions and allow more aircraft
                     to fly user-preferred, fuel-efficient routes.

                     To design AAS, FAA awarded competitive contracts to IBM and Hughes
                     Aircraft Corporation in 1984. On July 25, 1988, after spending about
                     $500 million on the design competition, FAA awarded a $3.6-billion con-
                     tract to IBM to complete the design and produce the AAS. However, on
                     August 10, 1988, a stop work order was issued to IBM as a result of a
                     protest filed by Hughes Aircraft Corporation. On October 28, 1988, the
                     General Services Administration’s Board of Contract Appeals issued a
                     decision upholding FAA’Scontract award. IBM resumed work on AAS in
                     November 1988.


AAS Implementation    The AAScontract calls for implementing the system in three steps.
Approach              Deployment of the first step is to begin in 1993, the second in 1995, and
                      the third in 1996, According to FAA officials, the system is to be com-
                      pleted by 2003 with contract options extending until 2010.

                      ISSS is the primary component of the first step and constitutes the
                      largest portion of the AAS program. It is to supply new controller work-
                      stations at en route centers’ to replace existing controller displays, and
                      automate some related processes that are currently done manually.

                      ’ FAA currently maintains 20 Ar Hout~ Traffic   Control Centers m the continental    United States that
                      rontrol air traffic between airport.5




                      Page2                                  GAO/lMTEG90-63        Air Traffic   Control   Automation   Delays