oversight

Air Traffic Control: FAA's Modernization Investment Management Approach Could Be Strengthened

Published by the Government Accountability Office on 1999-04-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                      United States General Accounting Office

GAO                   Report to Congressional Requesters




April 1999
                      AIR TRAFFIC
                      CONTROL
                      FAA’s Modernization
                      Investment
                      Management Approach
                      Could Be Strengthened




GAO/RCED/AIMD-99-88
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Resources, Community, and
      Economic Development Division

      B-279992

      April 30, 1999

      The Honorable Slade Gorton
      Chairman
      The Honorable John D. Rockefeller IV
      Ranking Minority Member
      Subcommittee on Aviation
      Committee on Commerce, Science, and Transportation
      United States Senate

      The Honorable John J. Duncan
      Chairman
      The Honorable William O. Lipinski
      Ranking Democratic Member
      Subcommittee on Aviation
      Committee on Transportation and Infrastructure
      House of Representatives

      In response to your request, this report addresses the extent to which the Federal Aviation
      Administration’s (FAA) Acquisition Management System provides a comprehensive approach for
      managing the agency’s investments in air traffic control information technology. FAA plans to
      spend billions of dollars to replace data-processing, navigation, communications, and other
      systems under its air traffic control modernization program but has a history of poor
      performance in delivering systems on time and within budget and performance parameters. We
      found that FAA has established a structured approach for managing its modernization
      investments, but weaknesses in this approach limit its effectiveness. We are making
      recommendations to the Secretary of Transportation to strengthen FAA’s investment
      management approach.

      We are sending copies of this report to Senator Frank R. Lautenberg, Senator Richard C. Shelby,
      Representative James A. Barcia, Representative Stephen Horn, Representative Jim Turner,
      Representative Constance A. Morella, Representative Martin Olav Sabo, and Representative
      Frank R. Wolf in their capacities as Chair or Ranking Minority Member of Senate and House
      Subcommittees. We are also sending copies of this report to the Honorable Rodney E. Slater,
      Secretary of Transportation; the Honorable Jane F. Garvey, Administrator of the Federal
      Aviation Administration; and the Honorable Jacob Lew, Director, Office of Management and
      Budget. Copies will also be made available to others on request.
B-279992

Please call me at (202) 512-2834 if you have questions about the report. Major contributors to
this report are listed in appendix II.




Gerald L. Dillingham
Associate Director, Transportation
  Issues




                     Page 2                         GAO/RCED/AIMD-99-88 FAA’s Investment Management
B-279992




           Page 3   GAO/RCED/AIMD-99-88 FAA’s Investment Management
Executive Summary


                   The Federal Aviation Administration (FAA) has undertaken an ambitious
Purpose            and costly program to modernize its air traffic control system. Under this
                   program, FAA is acquiring new surveillance, data-processing, navigation,
                   and communications equipment in addition to new facilities and support
                   equipment. Totaling 126 active projects, the modernization effort is
                   estimated to cost $26.5 billion from fiscal year 1982 through fiscal 2004.1
                   Of this total, FAA estimates that it will need $12.9 billion for 59 information
                   technology projects—the software-intensive and complex information and
                   communications systems supporting the air traffic control system.

                   Given the large expenditures required to carry out FAA’s modernization
                   effort, the past problems, and the continuing concerns about key projects
                   funded under the program, the Chairmen and Ranking Members of the
                   Senate Committee on Commerce, Science, and Transportation’s
                   Subcommittee on Aviation and the House Committee on Transportation
                   and Infrastructure’s Subcommittee on Aviation asked GAO to review FAA’s
                   investment management approach as carried out through the Acquisition
                   Management System (AMS), which was implemented in April 1996. GAO
                   evaluated the processes, data, and decisions that FAA uses to select,
                   control, and evaluate its investments. This report addresses the extent to
                   which FAA, through AMS, (1) has established a structured approach for
                   selecting and controlling its investments; (2) incorporates all investments,
                   including those currently in operation, in the agency’s portfolio; and
                   (3) selects, controls, and evaluates its investments with complete and
                   reliable information.


                   FAA’s Acquisition Management System is a good first step in establishing a
Results in Brief   structured investment management approach for selecting and controlling
                   the agency’s investments. The system contains a set of policies,
                   procedures, and reporting requirements to analyze mission needs; assess
                   the affordability of proposed projects; and establish life-cycle costs,
                   schedules, benefits, and performance baselines (boundaries) to control
                   the performance of the projects that are selected. Additionally, under this
                   system, a senior management investment review group makes key
                   decisions about which investments best meet the agency’s needs and are
                   to be funded.

                   However, the system is not comprehensive in that it does not incorporate
                   all of FAA’s projects into a complete strategic investment portfolio. Key

                   1
                    The total cost of the modernization program—which includes completed, canceled, and restructured
                   projects as well as the active projects—is estimated to be $41 billion from fiscal year 1982 through
                   fiscal 2004. In this report, all dollars are expressed as current-year dollars.



                   Page 4                                  GAO/RCED/AIMD-99-88 FAA’s Investment Management
             Executive Summary




             decision-making processes and requirements of the Acquisition
             Management System are applied only to proposed projects and those
             under development but not to projects already in operation. In particular,
             agency officials have not yet developed a sound estimate of the costs to
             operate projects and these costs are not included in the agency’s financial
             plan for modernization. Because FAA does not apply the same scrutiny to
             all of its projects, senior officials are unable to fully assess and make
             trade-offs about the relative merits of spending funds to develop new
             systems, to enhance current systems, or to continue operating and
             maintaining existing systems. This report makes a recommendation
             designed to strengthen FAA’s investment management by directing the
             agency to establish and control a complete portfolio of all information
             technology investments, including those projects already in operation.

             FAA’s  Acquisition Management System currently does not provide
             complete and reliable information for selecting, controlling, and evaluating
             the agency’s investments. First, the cost data used to select projects are of
             questionable reliability because of weaknesses in FAA’s cost estimating
             practices and processes and the lack of a cost accounting system. Second,
             the information used to control projects is incomplete since FAA has not
             fully implemented an effective process for controlling the baselines for the
             costs, schedules, benefits, performance, and risks of its investments. FAA
             has approved the baseline information for only half of the required
             universe of projects, and the agency’s processes for tracking actual
             performance against estimates frequently has provided incomplete
             information. Third, FAA lacks information needed to evaluate its
             investments since the Acquisition Management System does not have a
             post-implementation evaluation process for assessing projects’ outcomes
             and feeding lessons learned back into the selection and control phases to
             help improve its management of future projects. This report makes several
             recommendations to improve FAA’s selection, control, and evaluation of its
             information technology investments.


             Over the past 17 years, FAA’s modernization projects have experienced
Background   substantial cost overruns, lengthy delays, and significant performance
             shortfalls. Because of FAA’s contention that some of its modernization
             problems were caused by federal acquisition regulations, the Congress
             enacted legislation in November 1995 that exempted the agency from most
             federal procurement laws and regulations and directed FAA to develop a
             new acquisition management system. In response, FAA implemented AMS on
             April 1, 1996. AMS provides high-level acquisition policy and guidance for



             Page 5                         GAO/RCED/AIMD-99-88 FAA’s Investment Management
                            Executive Summary




                            selecting and controlling investments throughout all phases of the
                            acquisition life cycle.

                            Funding for FAA’s modernization investments is primarily provided through
                            two of its budget accounts: (1) facilities and equipment and (2) operations.
                            The facilities and equipment account covers the costs to develop, procure,
                            and place the new equipment or facility in operation. Once the project
                            goes into full operation, it is funded by the operations account, which
                            covers the costs to support and maintain the new equipment or facility.

                            Using the methodology described in Assessing Risks and Returns: A Guide
                            for Evaluating Federal Agencies’ IT Investment Decision-Making,2 GAO
                            evaluated how FAA selects, controls, and evaluates its investments. This
                            guide incorporates GAO’s analysis of the management practices of leading
                            private and public sector organizations as well as provisions of major
                            federal legislation (e.g., the Clinger-Cohen Act of 1996) and executive
                            branch guidance that address investment decision-making. As part of its
                            evaluation, GAO examined five of FAA’s projects to determine how AMS is
                            implemented at the project level. (The five projects are described in app.
                            I.)



Principal Findings

FAA’s AMS Is Designed to    Through AMS, FAA has designed and implemented processes that provide
Provide a Disciplined,      many of the key elements leading organizations follow to select and
Structured Process for      control investments the agency funds through its facilities and equipment
                            budget account. In the selection phase, leading organizations take a
Selecting and Controlling   structured approach to determining priorities, screening and analyzing the
Investments                 relative merits of the projects, and making decisions about which projects
                            will be funded during the year. Through AMS, FAA has established two
                            processes—mission analysis and investment analysis—that together
                            constitute a set of policies, procedures, and guidance that enhances the
                            agency’s ability to screen projects submitted for funding; assess and rank
                            each project based on its relative costs, benefits, risks, and contribution to
                            FAA’s mission; and utilize a senior, corporate-level decision-making group
                            to select projects for funding. Once a project is selected, AMS requires FAA
                            officials to formally establish the life-cycle cost, schedule, benefits, and
                            performance baselines that are used to monitor the project’s status. FAA


                            2
                             GAO/AIMD-10.1.13 (Feb. 1997).



                            Page 6                           GAO/RCED/AIMD-99-88 FAA’s Investment Management
                            Executive Summary




                            has developed a number of mechanisms for monitoring projects’ estimated
                            versus actual baseline performance and reporting any variances from the
                            established baselines.


Lack of Oversight of the    FAA lacks oversight of the operations portion of its investments under AMS.
Operations Portion of       For example, FAA’s process for scoring and ranking projects prior to
Projects Prevents FAA       selection is applied to proposed projects and those under development
                            that will receive facilities and equipment funding, not to existing systems
From Managing               that are funded from the operations budget account. In contrast, leading
Investments as a Complete   organizations include all types of information technology projects (i.e.,
Portfolio                   new and existing systems) in their selection process to create a complete
                            strategic investment portfolio. FAA also has not yet developed a sound
                            estimate of the operations cost baseline for each of its projects and the
                            agency’s financial plan for the modernization program reports only costs
                            funded by the facilities and equipment budget account, omitting the
                            operations costs associated with its investments. Although FAA has
                            developed operations cost projections for 26 of the 70 projects or
                            segments of projects identified as requiring baselines, officials throughout
                            the agency told us that these estimates are not reliable. Finally, while FAA’s
                            budget provides detailed analyses of actual and projected costs for each of
                            the projects funded by the facilities and equipment budget account, it
                            provides very little project-level detail in its justification for the operations
                            budget account. FAA has two initiatives that it believes will improve the
                            data on operations costs. First, it is developing a cost accounting system,
                            although operations data from that system will not be available for at least
                            a year, given the schedule for implementing the system. Second, FAA has a
                            team that is addressing the agency’s concerns about the quality of
                            operations data and developing operations cost baselines for selected
                            projects. This team is expected to report its findings in May 1999.


Weaknesses in the           AMS  has weaknesses in all three investment management
Selection, Control, and     phases—selection, control, and evaluation—that limit FAA’s ability to
Evaluation Phases Limit     manage its investments effectively. First, the cost information used to
                            make selection decisions is of questionable reliability, and there is little
FAA’s Effectiveness in      evidence that the data or underlying analyses used in the selection process
Managing Its Investments    are validated to ensure accuracy and completeness. While the agency has
                            improvements under way, FAA’s cost estimating techniques do not yet
                            satisfy recognized standards that call for organizing and retaining projects’
                            cost information in a historical database and using cost models that are
                            calibrated and validated on the basis of actual experience. Instead, FAA’s



                            Page 7                          GAO/RCED/AIMD-99-88 FAA’s Investment Management
Executive Summary




processes allow each project to approach cost estimating in whatever
manner its estimators choose. Moreover, the data used to support FAA’s
selection decisions are not validated because FAA does not require that all
project information—such as that pertaining to costs, schedules,
performance, benefits, or risks used in making selection decisions—be
validated under AMS. Furthermore, AMS guidance does not specify what
types of validation steps should be taken nor does it require
documentation of the results.

Second, FAA has not fully implemented its procedures for controlling key
projects’ baselines. To control its projects at the agencywide level, FAA
relies on periodic reviews of each project’s acquisition program baseline, a
document that establishes a project’s cost, schedule, benefits, and
performance boundaries and that is intended to be used to monitor a
project’s status in achieving those boundaries. This baseline document is
incomplete, however, because its schedule baseline does not include any
milestones for project reviews during the operations phase of the project
and because it does not address the project’s risks. In addition, FAA has
completed about half of the baselines for its universe of projects or project
segments that require them, and agency-level processes for tracking actual
baseline performance against estimates frequently provided incomplete
information on projects’ costs, schedules, benefits, and performance. For
example, for the five projects GAO reviewed, none of the monthly baseline
status reports analyzed the projects’ estimated operations costs, assessed
estimated versus actual benefits, or contained information on the
performance requirements outlined in the projects’ baseline
documentation. Moreover, FAA’s investment control group made up of
senior managers, the Joint Resources Council, is not actively involved in
monitoring all projects after the investment decisions are made.

Third, FAA does not have a defined, documented process for conducting
post-implementation reviews of projects for the purpose of assessing
project performance as well as improving the selection and control of its
investments. FAA performs some elements of a post-implementation review
in its life-cycle management review process, including such tasks as the
independent operational test and evaluation of some projects prior to
deployment, operational performance monitoring, customer satisfaction
surveys, and periodic reviews throughout an investment’s life cycle.
However, this process is not standardized and is not required for all
projects. As a result, there is no evidence that changes, especially to the
selection and control phases, are being implemented based on lessons
learned.



Page 8                         GAO/RCED/AIMD-99-88 FAA’s Investment Management
                      Executive Summary




                      Finally, FAA’s recently implemented agencywide management information
                      system for tracking information about projects under AMS contains data
                      related to FAA’s processes for managing baselines but excludes key
                      selection data, such as mission need statements, cost-benefit analyses, risk
                      assessments, and other required reports. Informed management decisions
                      can only be made if information from all phases of the investment
                      management process is included in the decision-making process and made
                      easily available through a management information system.


                      GAO recommends that the Secretary of Transportation direct the
Recommendations       Administrator of FAA to implement a comprehensive investment
                      management approach through the Acquisition Management System.
                      Specifically, the Administrator should take the following actions:

                  •   Establish a complete portfolio of investments—including existing systems
                      funded by the operations budget account as well as projects funded by the
                      facilities and equipment account—and require the Joint Resources Council
                      to periodically review the baseline status and merits of each of these
                      investments throughout their entire life cycle. As part of this portfolio,
                      cost baselines for operating and maintaining all projects should be
                      developed, and this information should be included in the agency’s
                      financial plan for its investments and in its annual budget request to the
                      Congress.
                  •   Improve the selection process by (1) establishing clearly defined
                      procedures for validating each project’s cost, schedule, benefit,
                      performance, and risk information and (2) requiring documentation of the
                      results of the validation procedures applied to each project.
                  •   Strengthen control over investments by (1) revising the acquisition
                      program baseline requirements to include project risks and to add
                      milestones for project reviews during the operations phase and
                      (2) ensuring that project officials fully track and document estimated
                      versus actual results on all the elements (i.e., cost, schedule, benefit,
                      performance, and risk) contained in the baseline documentation.
                  •   Initiate post-implementation evaluations for projects within 3 to 12 months
                      of deployment or cancellation to compare the completed projects’ cost,
                      schedule, performance, and mission improvement outcomes with the
                      original estimates.
                  •   Incorporate key information from the selection process (e.g., mission need
                      statements, cost-benefit analyses, and risk assessments) into FAA’s
                      management information system for investments.




                      Page 9                        GAO/RCED/AIMD-99-88 FAA’s Investment Management
                     Executive Summary




                     We provided copies of a draft of this report to the Department of
Agency Comments      Transportation and FAA for their review and comment. We met with FAA
and Our Evaluation   officials, including the Associate Administrator for Research and
                     Acquisitions, who is also FAA’s Acquisition Executive. These officials
                     generally agreed with the recommendations in this report and made
                     clarifying comments, which have been incorporated as appropriate. FAA
                     officials noted that the Acquisition Management System represents a
                     substantial revision to the way that FAA contracts for large and complex
                     systems and that it is important to maintain perspective on how far the
                     agency has come in a relatively short time. These officials were concerned
                     that the use of GAO’s guide, Assessing Risks and Returns: A Guide for
                     Evaluating Federal Agencies’ IT Investment Decision-Making, unduly
                     diminished FAA’s accomplishments by comparing the agency to an ideal
                     end-state that may not exist in any single organization. They also asserted
                     that the Acquisition Management System would compare favorably with
                     acquisition systems in other federal government or private sector
                     organizations.

                     As we stated in this report, FAA’s Acquisition Management System is a good
                     first step in establishing a structured investment management approach,
                     but it has weaknesses that limit its effectiveness. The GAO guide was
                     developed to provide a structure for evaluating and assessing how well a
                     federal agency is managing its information technology resources and to
                     identify specific areas where improvements can be made. The concepts
                     and practices contained in the guide are based on the practices followed
                     by leading private and public sector organizations as well as on provisions
                     of major federal legislation and executive branch guidance that address
                     investment decision-making. While acknowledging the wide variance
                     among organizations and the complexity of the investment management
                     process, the guide focuses on the common elements that should be
                     present in any organization’s investment management process. Our review
                     evaluated the extent to which FAA’s Acquisition Management System
                     contains these elements, not how well FAA’s system compares with other
                     federal acquisition systems. Therefore, we did not make changes to the
                     report based on these comments.




                     Page 10                       GAO/RCED/AIMD-99-88 FAA’s Investment Management
Page 11   GAO/RCED/AIMD-99-88 FAA’s Investment Management
Contents



Executive Summary                                                                                 4


Chapter 1                                                                                        14
                        The ATC System Modernization Program Is Complex, Costly, and             15
Introduction              Historically Problematic
                        FAA Has Developed the Acquisition Management System to                   16
                          Manage Its Modernization Investments
                        FAA’s Acquisitions Are Funded From Two Major Budget                      18
                          Accounts
                        Guidance Provides a Framework for Assessing Federal Agencies’            19
                          Information Technology Investment Decision-Making
                        Objectives, Scope, and Methodology                                       20


Chapter 2                                                                                        22
                        FAA Has Established a Defined Process for Selecting Investments          22
FAA’s AMS Is              Funded Through the Facilities and Equipment Account
Designed to Provide a   AMS Requires That Projects’ Baselines Be Established and                 24
                          Controlled
Disciplined,            Conclusions                                                              26
Structured Process
for Selecting and
Controlling
Investments
Chapter 3                                                                                        27
                        FAA’s Selection Process Does Not Address Operations-Funded               27
Lack of Oversight of      Investment Projects
the Operations          FAA Does Not Have a Complete and Sound Operations Cost                   28
                          Baseline for Its Investments
Portion of Projects     FAA’s Operations Budget Justifies Only a Small Portion of Its            29
Prevents FAA From         Spending for Investments
Managing Investments    Conclusions                                                              30
                        Recommendation                                                           30
as a Complete           Agency Comments                                                          30
Portfolio




                        Page 12                      GAO/RCED/AIMD-99-88 FAA’s Investment Management
                      Contents




Chapter 4                                                                                      31
                      Weaknesses in Some Supporting Data Limit the Effectiveness of            31
Weaknesses in the       the Selection Process
Selection, Control,   FAA Has Not Fully Implemented Its Process for Establishing and           33
                        Tracking Key Project Baselines
and Evaluation        FAA’s AMS Lacks a Post-Implementation Review Process for                 41
Phases Limit FAA’s      Evaluating Investments
Effectiveness in      FAA’s Efforts to Implement an Agencywide Management                      42
                        Information System Do Not Include Key Selection Data
Managing Its          Conclusions                                                              44
Investments           Recommendations                                                          44
                      Agency Comments                                                          45


Appendixes            Appendix I: Background and Status of Five Projects                       46
                      Appendix II: Major Contributors to This Report                           53


Tables                Table 2.1: Information on the Baseline Elements Required in the          25
                        Acquisition Program Baseline and the Project Parameter Sheet
                      Table 4.1: Status of FAA’s Efforts to Develop AMS Baseline               37
                        Documentation on 60 Ongoing Projects and Segments for Which
                        Investment Decisions Have Been Made
                      Table 4.2: Analysis of the Missing Baseline Information Reported         39
                        in Acquisition Reviews of Five Projects


Figure                Figure 1.1: FAA’s Life-Cycle Acquisition Management Process              18




                      Abbreviations

                      AMS        Acquisition Management System
                      ATC        air traffic control
                      FAA        Federal Aviation Administration
                      GAO        General Accounting Office
                      I-BEAM     Integrated Baseline Establishment and Management
                      NAS        National Airspace System
                      SPIRE      Simplified Program Information Reporting and Evaluation


                      Page 13                      GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 1

Introduction


               The Federal Aviation Administration’s (FAA) mission is to promote the
               safe, orderly, and expeditious flow of air traffic in the United States
               through what is commonly referred to as the National Airspace System
               (NAS). FAA’s ability to fulfill its mission depends on the adequacy and
               reliability of the nation’s air traffic control (ATC) system—the principal
               component of the NAS—which comprises a vast network of radars;
               automated data-processing, navigation, and communications equipment;
               and ATC facilities. It is through the ATC system that FAA provides services
               such as controlling takeoffs and landings and managing the flow of traffic
               between airports.3 Sustained growth in air traffic and FAA’s aging
               equipment have strained the current ATC system. This growth in traffic is
               predicted to continue as the number of passengers traveling on U.S.
               airlines is expected to grow from about 674 million in 1998 to 1.055 billion
               by 2010, an increase of about 57 percent.

               To relieve the problems of aging equipment and to accommodate the
               predicted growth in air traffic, FAA initiated a multibillion-dollar
               modernization effort in December 1981. Our work over the years has
               chronicled many of FAA’s failures in meeting projects’ cost, schedule, and
               performance goals. Because of the size, complexity, cost, and
               problem-plagued past of FAA’s modernization program, we have designated
               it a high-risk information technology investment since 1995.4




               3
                FAA uses three types of facilities to manage and control traffic. Airport towers direct aircraft on the
               ground, before landing, and after takeoff when they are about 5 nautical miles from the airport and up
               to about 3,000 feet above the airport. Terminal radar approach control facilities sequence and separate
               aircraft as they approach and leave busy airports, beginning about 5 nautical miles and ending about 50
               nautical miles from the airport and generally up to 10,000 feet above the ground. Air route traffic
               control centers, called en route centers, control planes in transit and during approaches to some
               airports. Most of the en route centers’ controlled airspace extends above 18,000 feet for commercial
               aircraft. En route centers also handle aircraft at lower altitudes when dealing directly with a tower or
               when agreed upon with a terminal facility. FAA provides additional services, such as weather and pilot
               briefings, through a network of flight service stations.
               4
                FAA’s modernization program is one of four high-risk system development and modernization efforts
               in the federal government. See High-Risk Series: An Overview (GAO/HR-95-1, Feb. 1995), High-Risk
               Series: Information Management and Technology (GAO/HR-97-9, Feb. 1997), and High-Risk Series: An
               Update (GAO/HR-99-1, Jan. 1999).



               Page 14                                  GAO/RCED/AIMD-99-88 FAA’s Investment Management
                      Chapter 1
                      Introduction




                      Under its ambitious modernization program, FAA is acquiring new
The ATC System        surveillance, data-processing, navigation, and communications equipment
Modernization         in addition to new facilities and support equipment. Totaling 126 active
Program Is Complex,   projects, the modernization is estimated to cost $26.5 billion from fiscal
                      year 1982 through fiscal 2004.5 Of this total, FAA estimates that it will need
Costly, and           $12.9 billion from fiscal year 1982 through 2004 for 59 information
Historically          technology projects—the software-intensive and complex information and
                      communications systems supporting the ATC system. These projects range
Problematic           from those designed to replace equipment used by controllers to
                      communicate with aircraft and with each other to radars that provide
                      controllers with surveillance information for separating aircraft. An
                      example of an information technology project is the Display System
                      Replacement project that will modernize equipment in FAA’s en route
                      facilities by replacing 20- to 30-year-old display channels, controllers’
                      workstations, and network infrastructure.

                      Over the past 17 years, FAA’s modernization projects have experienced
                      substantial cost overruns, lengthy delays, and significant performance
                      shortfalls. To illustrate, the longtime centerpiece of the modernization
                      program—the Advanced Automation System—was restructured in 1994
                      after estimated costs to develop the system tripled from $2.5 billion to $7.6
                      billion and delays in putting significantly less-than-promised system
                      capabilities into operation were expected to run 8 years or more over the
                      original estimates.6 These problems have persisted. For example, two key
                      projects in the modernization effort—the Wide Area Augmentation System
                      and the Standard Terminal Automation Replacement System—have
                      encountered significant cost increases, delays, and changes in
                      requirements.7 In the case of the Wide Area Augmentation System,
                      between September 1997 and January 1998, total estimated costs
                      increased by $600 million, or 25 percent, from $2.4 billion to $3 billion. The
                      increased costs were attributable to FAA’s including previously overlooked
                      costs for periodically updating the project’s equipment and to
                      higher-than-expected operations and maintenance costs. In the case of the
                      Standard Terminal Automation Replacement System, although FAA has not
                      officially changed the project’s baseline approved in February 1996, the


                      5
                       The total cost of the modernization program—which includes completed, canceled, and restructured
                      projects as well as active projects—is estimated to be $41 billion from fiscal year 1982 through fiscal
                      2004. In this report, all dollars are expressed as current year dollars.
                      6
                       See Advanced Automation System: Implications of Problems and Recent Changes
                      (GAO/T-RCED-94-188, Apr. 13, 1994).
                      7
                       See Air Traffic Control: Observations on FAA’s Modernization Program (GAO/T-RCED/AIMD-98-93,
                      Feb. 26, 1998).



                      Page 15                                   GAO/RCED/AIMD-99-88 FAA’s Investment Management
                    Chapter 1
                    Introduction




                    baseline is in jeopardy of being breached because of unions’ concerns
                    surrounding human-factor and design issues,8 the refinement of
                    requirements, and the interjection of a new project phase.9 FAA estimates
                    that these issues have the potential to increase the project’s costs from
                    $294 million to $410 million above the approved baseline. FAA also
                    estimates that the project’s initial completion could be delayed by almost
                    2-1/2 years.


                    Because of FAA’s contention that some of its modernization problems were
FAA Has Developed   caused by federal acquisition regulations, the Congress enacted legislation
the Acquisition     in November 1995 that exempted the agency from most federal
Management System   procurement laws and regulations and directed FAA to develop and
                    implement a new acquisition management system that would address the
to Manage Its       unique needs of the agency.10 On April 1, 1996, in response to the
Modernization       Congress’s action, FAA implemented a new acquisition management
                    system. The system is intended to reduce the time and cost to field new
Investments         products and services by introducing (1) a new investment management
                    system that spans the entire life cycle of an acquisition, (2) a new
                    procurement system that provides flexibility in selecting and managing
                    contractors, and (3) organizational and cultural reform that supports the
                    new investment and procurement systems.

                    The Acquisition Management System (AMS) provides high-level acquisition
                    policy and guidance for selecting and controlling FAA’s investments
                    throughout all phases of the acquisition life cycle, which is organized into
                    a series of phases and decision points, including (1) mission analysis,
                    (2) investment analysis, (3) solution implementation, (4) in-service
                    management, and (5) service life extension. AMS provides guidance on the
                    documents and decisions that result from each of these phases. For
                    example, through the mission analysis process, FAA identifies critical needs
                    that the agency must meet for improving the safety, security, capacity,
                    efficiency, and effectiveness of the NAS. Approval of a mission need
                    statement by the Joint Resources Council—FAA’s corporate
                    decision-making body—signifies that the agency agrees that the need is
                    critical enough to proceed to the next phase, investment analysis. During
                    the investment analysis phase, teams of acquisition and program
                    specialists (1) identify and analyze alternatives, (2) develop baselines and


                    8
                     Concerns were raised by two unions, the National Air Traffic Controllers Association and the
                    Professional Airways Systems Specialists.
                    9
                     See Air Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-99-25, Dec. 3, 1998).
                    10
                        Department of Transportation and Related Agencies Appropriations Act of 1996 (P.L. 104-50).


                    Page 16                                    GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 1
Introduction




assesses affordability, (3) prepare an investment analysis report, and
(4) recommend a preferred solution to the mission need. FAA then scores
and ranks each proposed project based on a number of factors, including
how well, relative to other projects, it meets mission needs and has a
favorable cost-benefit ratio. Once a project is selected, life-cycle cost,
schedule, benefits, and performance baselines are established in a formal
document called the acquisition program baseline. The acquisition
program baseline, which must be approved by the Joint Resources
Council, is used to monitor a project’s status in achieving those baselines
throughout the remaining phases of the acquisition management life cycle.

During the solution implementation phase, a multidisciplinary team
develops and carries out an acquisition strategy for implementing the
project. Once the project has been implemented and is in operation (the
in-service management phase), the team monitors and assesses its
performance, costs, and support trends; proposes fixes for any defects or
other problems; incorporates product improvements; seeks new
technology to enhance the capability or reduce costs; and identifies and
prepares for decisions to correct capability shortfalls at the end of the
project’s service-life. Finally, during the service-life extension phase, a
determination is made about whether the current capability satisfies the
demand for services or whether another solution offers the potential for
improving safety or effectiveness or for significantly lowering costs. The
team initiates a process whereby the mission need would be revalidated
and the investment analysis process begun again, which could lead to a
new investment decision. See figure 1.1 for a graphic depiction of FAA’s
life-cycle acquisition management process.




Page 17                       GAO/RCED/AIMD-99-88 FAA’s Investment Management
                                              Chapter 1
                                              Introduction




Figure 1.1: FAA’s Life-Cycle Acquisition Management Process

                                                                                           INVESTM
                                                       NEED                                         E
                                                                                             DECISIONNT
                                              MISSION ION
                                                DEC IS

                                                                                ANALYSIS
                                                             IN V E ST M E NT
                                        IS
                                   ALYS
                            SION AN
                         MIS




                                                                                                                SO
                                          ION E




                                                                                                                  LU
                                   EXTE E LIF




                                                                           END




                                                                                                                    T
                                                                 FAA




                                                                                                                 ION
                                       NS




                                                                         PROGRAM
                                   SERVIC




                                                             LIFE CYCLE




                                                                                                                     IMPLEMEN
                                                            ACQUISITION
                                                         MANAGEMENT SYSTEM




                                                                                                                             TATI
                                                                                                                                 ON
                                                                     I N-S E
                                                                               RVICE MANAGEMENT                    ICE
                                                                                                               E RV ON
                                                                                                             S     I
                                                                                                          IN- ECIS
                                                                                                            D


                                              Source: FAA.



                                              Funding for FAA’s investments is primarily provided through two of its
FAA’s Acquisitions Are                        budget accounts: (1) facilities and equipment and (2) operations. The
Funded From Two                               facilities and equipment account covers the costs to develop, procure, and
Major Budget                                  place the new equipment or facility in operation. Once the project goes
                                              into full operation, it is funded by the operations account, which covers
Accounts                                      the costs to support and maintain the new equipment or facility. Costs for
                                              planned product improvements and upgrades to the technology can be
                                              funded from either the facilities and equipment or the operations
                                              accounts.

                                              Some investment funding is also provided through the research,
                                              engineering, and development account. This account is relatively small
                                              compared with the facilities and equipment and operations accounts. For
                                              example, in fiscal year 1999, $150 million was appropriated for research,
                                              engineering, and development; approximately $2.1 billion for facilities and
                                              equipment; and approximately $5.6 billion for operations.




                                              Page 18                                  GAO/RCED/AIMD-99-88 FAA’s Investment Management
                        Chapter 1
                        Introduction




                        Several recent management reforms—including the revision of the
Guidance Provides a     Paperwork Reduction Act and the passage of the Clinger-Cohen Act of
Framework for           1996, the Government Performance and Results Act of 1993, and the Chief
Assessing Federal       Financial Officers Act of 1990—have introduced requirements
                        emphasizing the need for federal agencies to improve their management
Agencies’ Information   processes for selecting and managing information technology resources.
Technology              GAO and the Office of Management and Budget have developed guidance to
                        assist federal agencies in evaluating information technology investments.
Investment              One such guide, Assessing Risks and Returns: A Guide for Evaluating
Decision-Making         Federal Agencies’ IT Investment Decision-Making, incorporates our
                        analysis of the management practices of leading private and public sector
                        organizations as well as of the provisions of major federal legislation (e.g.,
                        the Clinger-Cohen Act) and executive branch guidance that address
                        investment decision-making.11 The guide outlines three phases of a
                        successful investment management approach—selection, control, and
                        evaluation. To help ensure that real, positive change is produced as
                        agencies seek to improve their decision-making about their information
                        technology investments, agencies need to (1) institutionalize management
                        processes; (2) regularly validate the cost, benefit, and risk data used to
                        support information technology decisions; and (3) focus on measuring and
                        evaluating results. The guide provides a framework for evaluating and
                        assessing how well a federal agency is achieving these goals and identifies
                        specific areas where improvements can be made.

                        Many of the concepts in our Assessing Risks and Returns have been
                        incorporated into the Office of Management and Budget’s Capital
                        Programming Guide, which provides guidance on the planning, budgeting,
                        acquisition, and management of different kinds of capital assets, including
                        information technology.12 Our guide has also been endorsed by the federal
                        Chief Information Officers Council. A third guide we prepared, Executive
                        Guide: Leading Practices in Capital Decision-Making, summarizes 12
                        fundamental practices that have been successfully implemented by
                        organizations recognized for their outstanding capital decision-making.13
                        Since information technology investments are a form of capital asset, this
                        guide emphasizes many of the same concepts as the aforementioned
                        guides, such as evaluating alternative approaches to achieving results;



                        11
                          GAO/AIMD-10.1.13 (Feb. 1997).
                        12
                           Capital Programming Guide, Version 1.0, Supplement to Office of Management and Budget Circular
                        A-11, Executive Office of the President, Office of Management and Budget (July 1997).
                        13
                          Executive Guide: Leading Practices in Capital Decision-Making (GAO/AIMD-99-32, Dec. 1998).



                        Page 19                                 GAO/RCED/AIMD-99-88 FAA’s Investment Management
                     Chapter 1
                     Introduction




                     assessing investments as a portfolio; tracking projects’ costs, schedules,
                     and performance; and conducting post-implementation reviews.


                     Given the large expenditures required to carry out FAA’s modernization
Objectives, Scope,   program, the past problems, and the continuing concerns about key
and Methodology      projects funded under the program, the Chairmen and Ranking Minority
                     Members of the Senate Commerce, Science, and Transportation
                     Committee’s Subcommittee on Aviation and the House Transportation and
                     Infrastructure Committee’s Subcommittee on Aviation asked us to
                     evaluate the extent to which AMS provides a comprehensive approach for
                     managing FAA’s modernization investments. This report addresses the
                     extent to which FAA, through AMS, (1) has established a structured
                     approach for selecting and controlling its investments; (2) incorporates all
                     investments, including those in operation, in the agency’s portfolio; and
                     (3) selects, controls, and evaluates its investments with complete and
                     reliable information.

                     To address our objectives, we reviewed FAA’s overall approach for
                     managing investments, as carried out through AMS, including the policies,
                     procedures, and guidance for managing the life-cycle acquisition process.
                     AMS is designed to include all of FAA’s acquisition projects, including those
                     related to facilities, mission support, and information technology. Our
                     review focused primarily on information technology projects, and we
                     worked with FAA to identify the universe of such projects. Then, in concert
                     with FAA, we selected five projects for review to obtain a more detailed
                     understanding of how the investment management process is implemented
                     at the project level. The five projects we reviewed were (1) the NAS
                     Infrastructure Management System, (2) the Oceanic Automation Program,
                     (3) the Operational and Supportability Implementation System, (4) the
                     Standard Terminal Automation Replacement System, and (5) the Voice
                     Switching and Control System. We selected these projects because they
                     were in various stages of implementation when AMS was established in
                     April 1996, were key to replacing the aging NAS infrastructure or improving
                     its capacity and effectiveness, and represented significant expenditures.
                     Total estimated facilities and equipment funding for each of these projects
                     exceeds $100 million. Two of the five projects—the NAS Infrastructure
                     Management System and the Operational and Supportability
                     Implementation System—are subject to all AMS requirements because they
                     were selected after AMS went into effect in April 1996. The other three
                     projects, which were ongoing when AMS was established, are subject to AMS




                     Page 20                        GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 1
Introduction




requirements for the remaining stages in their development and
implementation.

As part of our effort, we applied the methodology prescribed in Assessing
Risks and Returns. We used the questions in this guide to determine the
extent to which FAA has decision-making and management processes and
data in place to select information technology projects and systems,
control these projects throughout their life cycles, and evaluate results and
revise the processes based on lessons learned. We provided the questions
in the guide to FAA officials and assessed their responses, along with
supporting documentation, as a basis for determining whether FAA’s
approach provided the necessary elements for managing its investments.
We also incorporated the guidance from the Executive Guide: Leading
Practices in Capital Decision-Making in determining whether FAA managed
its investments as a portfolio; tracked projects’ costs, schedules, and
performance; and conducted post-implementation reviews.

To gain an overall perspective on FAA’s investment management process,
we interviewed FAA officials responsible for implementing and managing
AMS. We also interviewed FAA officials responsible for preparing the
facilities and equipment and operations budgets and reviewed the agency’s
fiscal year 1999 budget justification documentation to understand how
data on FAA’s investments are used to support FAA’s budget request to the
Congress. Finally, we interviewed officials responsible for managing the
five projects we reviewed to obtain their views on how AMS is applied at
the project level as well as potential areas for improvement. We performed
our work from April 1998 through April 1999 in accordance with generally
accepted government auditing standards.




Page 21                        GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 2

FAA’s AMS Is Designed to Provide a
Disciplined, Structured Process for
Selecting and Controlling Investments
                        Through AMS, FAA has designed and implemented processes that provide
                        many of the key elements followed by leading organizations for selecting
                        and controlling investments. AMS’ mission analysis and investment analysis
                        processes are meant to provide FAA’s senior management with a basis for
                        screening proposed projects; evaluating their relative costs, benefits, and
                        risks; and selecting projects for funding based on their relative merits.
                        Additionally, AMS policy requires each acquisition project to have an
                        approved baseline that establishes the project’s life-cycle cost, schedule,
                        benefits, and performance boundaries and that is intended to be used to
                        monitor the project’s status in achieving those baselines.


                        During the selection phase, leading organizations take a structured
FAA Has Established     approach to determining priorities, screening and analyzing the relative
a Defined Process for   merits of projects, and making decisions about which projects will be
Selecting Investments   funded during the year. Such an approach builds on an organization’s
                        assessment of where it should invest its resources for the greatest benefit
Funded Through the      over the long term. The Clinger-Cohen Act requires federal agencies to
Facilities and          apply this sort of structured approach in deciding whether to undertake
                        particular investments in information technology systems.
Equipment Account
                        A starting point for the selection phase is the screening process in which
                        projects being submitted for funding are compared against a uniform set
                        of screening criteria and thresholds to determine whether the projects
                        meet minimal requirements and to identify at what organizational level the
                        projects should be reviewed. Next, the costs, benefits, risks, and mission
                        focus of all the projects are assessed, and the projects are compared
                        against each other and ranked or prioritized. In conducting their selection
                        processes, leading organizations require all projects to have complete and
                        accurate project proposals and justification information. Finally, a
                        decision-making body of senior managers makes decisions about which
                        projects to select for funding on the basis of mission needs and
                        organizational priorities. The selection phase helps ensure that the
                        organization selects those projects that will best support mission needs
                        and identifies and analyzes each project’s risks and proposed benefits
                        before a significant amount of funds is spent.

                        Through AMS, FAA has established two processes—mission analysis and
                        investment analysis—that together constitute a set of policies, procedures,
                        and guidance that are designed to enhance the agency’s ability to select




                        Page 22                       GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 2
FAA’s AMS Is Designed to Provide a
Disciplined, Structured Process for
Selecting and Controlling Investments




investments.14 These two processes define what should be done and who
should do it, what reports are required, who reviews and approves reports
and processes, who makes corporate-level decisions, and the roles and
responsibilities of those involved. AMS policy and guidance—which is on
an easily accessible Internet Website—contain the procedures, process
flowcharts, document templates, checklists, and other acquisition-related
information needed for FAA’s project officials and senior management to
understand and implement the selection processes.

Although AMS does not include an explicit screening step, screening
activities are part of the agency’s mission analysis process. This process
culminates in a mission need statement reflecting the Joint Resources
Council’s decision that a high-priority, critical need exists and that the
agency should go forward with a detailed investment analysis of proposed
solutions to meet that need. During mission analysis, FAA’s operating
divisions identify and quantify projected demand for and supply of
services, capability shortfalls, and technological opportunities to meet
those shortfalls, and summarize the major decision factors that the Joint
Resources Council should evaluate in considering the need. The Joint
Resources Council approves mission need statements.

Once a mission need statement is approved, AMS’ investment analysis
process provides a set of detailed steps for evaluating the costs, benefits,
and risks of alternative solutions and for selecting the best solution to
meet the need. Under AMS, multidisciplinary investment analysis
teams—comprising officials from the operating divisions and other
acquisition and engineering specialists—assess each project proposed for
funding to define the technical requirements; estimate the life-cycle costs,
benefits, schedule, and risks; and determine the project’s affordability
relative to other projects. As part of the investment analysis process, FAA
scores and ranks each proposed project on the basis of defined criteria for
how well—relative to other projects—it meets the agency’s mission
objectives, whether it is of high priority to the organization sponsoring the
project, whether it is consistent with the NAS architecture15 or provides
critical administrative capacity for the agency, and whether it has a
favorable cost-benefit ratio. AMS requires that the mission need be
revalidated during the investment analysis and that the underlying
analyses be documented through cost-benefit studies, risk assessments,
and other documents.

14
  As we discuss in chapter 3, key aspects of AMS’ selection process are limited to projects funded by
the facilities and equipment account and exclude investments funded by the operations account.
15
  The architecture is FAA’s blueprint for defining the long-range needs of the NAS.



Page 23                                   GAO/RCED/AIMD-99-88 FAA’s Investment Management
                         Chapter 2
                         FAA’s AMS Is Designed to Provide a
                         Disciplined, Structured Process for
                         Selecting and Controlling Investments




                         After considering the results of the investment analysis process, the Joint
                         Resources Council decides whether to proceed with a proposed project.
                         This step is called the investment decision, and if the project is selected,
                         the Council commits FAA to fully funding it. The project is then
                         incorporated into the agency’s financial plan for projects funded by the
                         facilities and equipment budget account.


                         Under AMS, FAA’s primary mechanism for controlling a project is the
AMS Requires That        acquisition program baseline, which establishes a project’s life-cycle cost,
Projects’ Baselines Be   schedule, benefits, and performance baselines and which is intended to be
Established and          used to monitor a project’s status in achieving those baselines. The
                         acquisition program baseline is supposed to be established when the
Controlled               investment decision is made for a project. The baseline, which is the
                         agreement between the organizations within FAA that are acquiring and
                         will use the project, sets the cost and schedule boundaries within which
                         the project is authorized to proceed, defines the performance and benefits
                         the project must achieve, and establishes the performance measurements
                         for assessing the project’s success as it advances through its life cycle.

                         FAA’s ultimate goal is to establish an acquisition program baseline
                         document for every acquisition project. When AMS was established in
                         April 1996, however, the agency decided to establish two standards for
                         baseline documentation to facilitate the preparation of baselines during
                         FAA’s transition to AMS. One standard applies to those projects initiated
                         after AMS and another, less detailed standard applies to those projects
                         ongoing at the time that AMS was established. The post-AMS projects must
                         have an acquisition program baseline that provides a full set of detailed
                         information on each baseline element as well as a document called a
                         parameter sheet that summarizes a subset of baseline information that is
                         designated for the Joint Resources Council’s control and that is
                         considered critical to assessing the project’s ability to satisfy mission
                         need, achieve needed operational capability, achieve benefits, and meet
                         the schedule requirements of interdependent programs. Projects begun
                         before AMS was established are required only to have the parameter sheet.
                         Table 2.1 summarizes the full set of baseline information contained in the
                         acquisition program baseline and the subset of information contained in
                         the parameter sheet.




                         Page 24                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
                                             Chapter 2
                                             FAA’s AMS Is Designed to Provide a
                                             Disciplined, Structured Process for
                                             Selecting and Controlling Investments




Table 2.1: Information on the Baseline Elements Required in the Acquisition Program Baseline and the Project Parameter
Sheet
Baseline          Full set of baseline information contained in the acquisition Subset of baseline information contained in
elements          program baseline                                              the parameter sheet
Cost             Life-cycle costs are broken down by (1) all sources of          Total life-cycle costs are broken down by source
                 appropriations funding, (2) all fiscal years over the project’s of appropriations funding.
                 entire life cycle, and (3) 12 detailed cost elements (program
                 management, testing and evaluation, training, data management,
                 physical integration, systems and equipment, implementation,
                 product support, operations and maintenance, in-service
                 support, in-service monitoring and assessment, and disposal of
                 replaced assets).
Schedule         Schedule includes a list of all events related to satisfying mission   Schedule includes a subset of events most
                 need, providing intended operational capability, and accruing          crucial to satisfying mission need, providing
                 benefits as well as events crucial to other related NAS programs       operational services, and accruing benefits—for
                 or systems.                                                            example, contract award, in-service decision
                                                                                        (the point at which the new system is certified
                                                                                        ready for operational use)—and the dates that
                                                                                        the first and last sites are commissioned into
                                                                                        operational use.
Benefits         Total life-cycle benefits are listed for both the government and Benefits include total life-cycle economic
                 users, with specific measures for evaluating the annual economic benefits.
                 benefits and whether these benefits have been achieved.
Performance      Performance includes all requirements for project milestones;          Performance includes the total number of
                 technical performance; acquisition of technical systems,               systems, sites, or services provided.
                 equipment, facilities, and services; coordination with outside         Additionally, it includes other performance
                 organizations; and operations support and management                   requirements deemed most critical to achieving
                 requirements.                                                          operational effectiveness, accruing benefits, or
                                                                                        meeting other dependent NAS needs and those
                                                                                        considered to pose the greatest risk for cost or
                                                                                        schedule growth.
                                             Source: FAA.



                                             At the agencywide level, FAA has a two-tiered process for monitoring
                                             projects’ estimated versus actual baselines. Under one process, known as
                                             the Integrated Baseline Establishment and Management (I-BEAM) process,
                                             the project officials and officials responsible for overseeing projects’
                                             baselines monitor the projects and prepare reports. Project officials
                                             prepare a monthly status report that is supposed to analyze estimated
                                             versus actual results for the subset of baseline information that is
                                             monitored by the Joint Resources Council and summarized in the
                                             parameter sheets prepared for every project. Project officials must also
                                             inform the Joint Resources Council of any baseline violation or “breach”
                                             through a document called a baseline management notice and receive the
                                             Council’s approval for any change in the baselines. Additionally, FAA’s
                                             acquisition oversight officials prepare a monthly report that analyzes the



                                             Page 25                               GAO/RCED/AIMD-99-88 FAA’s Investment Management
              Chapter 2
              FAA’s AMS Is Designed to Provide a
              Disciplined, Structured Process for
              Selecting and Controlling Investments




              status reports, baseline management notices, and other documents to
              identify baseline variances and report this information to the Joint
              Resources Council. Under a second process, the Acquisition Executive
              who heads the Council and other senior managers conduct acquisition
              reviews that are designed to periodically examine the status of the
              information in each project’s acquisition program baseline, along with
              assessing a project’s progress, risks, and other issues.

              In addition to these two primary control processes, FAA also has a variety
              of other reporting systems for monitoring projects’ status, including
              executive-level metrics that analyze project performance, project-level
              reviews, and numerous contractor reports on costs, schedule, and
              technical performance. However, these various reporting systems do not
              analyze a project’s status in meeting all of the detailed baseline
              requirements contained in the acquisition program baselines and
              parameter sheets.


              FAA has taken a positive step in establishing an investment management
Conclusions   approach through AMS. Under AMS, FAA has developed structured processes
              for selecting and controlling its investments. AMS provides a defined set of
              policies, procedures, and reporting requirements that are designed to
              facilitate FAA’s efforts to analyze mission needs, identify alternative
              solutions to meet those needs, assess the solutions’ affordability, and
              establish and control a project’s performance once a solution is selected.




              Page 26                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 3

Lack of Oversight of the Operations Portion
of Projects Prevents FAA From Managing
Investments as a Complete Portfolio
                      FAA’s oversight of its investments is confined to new projects and those
                      under development, limiting its ability to fully assess and make trade-offs
                      between new and existing systems and preventing the agency from
                      managing all of its investments as a complete portfolio. FAA’s oversight
                      includes the agency’s processes for scoring and ranking investments prior
                      to selection as well as its processes for establishing and monitoring
                      financial baselines after a project has been selected. But this oversight is
                      limited to projects that receive funding from the facilities and equipment
                      budget account—that is, new projects and those under development—and
                      excludes projects funded by the operations account. Moreover, the link
                      between FAA’s investment management process and its budget process is
                      limited to the facilities and equipment budget account, which has detailed
                      information on investments that does not exist for the operations budget
                      account. FAA has two ongoing initiatives that may improve its information
                      on operations costs.

                      Leading organizations determine priorities and make decisions about
                      which projects will be funded based on analyses of the relative costs,
                      benefits, and risks of all the projects in their investment portfolios,
                      including projects that are proposed, under development, and in
                      operation. Such a portfolio approach allows the organizations to evaluate
                      the relative merits of spending funds to develop new systems, enhance
                      current systems, or continue operating and maintaining existing systems.


                      FAA’s  process for scoring and ranking projects prior to selection is applied
FAA’s Selection       to proposed projects and those under development that will receive
Process Does Not      facilities and equipment funding, not to existing systems that are funded
Address               from the operations budget account. In contrast, leading organizations
                      include all types of information technology projects in their selection
Operations-Funded     process to create a complete strategic investment portfolio. By analyzing
Investment Projects   the entire portfolio, the senior managers of any organization can examine
                      the costs of maintaining existing systems versus investing in new ones;
                      comparatively rank projects based on expected costs, benefits, and risks;
                      and reach decisions based on projects’ overall contribution to the most
                      pressing organizational needs. FAA has not included operations-funded
                      projects in its scoring and ranking process because, as discussed below,
                      the agency lacks reliable data on actual operations costs for existing
                      systems and projections of operations costs for new projects.




                      Page 27                        GAO/RCED/AIMD-99-88 FAA’s Investment Management
                      Chapter 3
                      Lack of Oversight of the Operations Portion
                      of Projects Prevents FAA From Managing
                      Investments as a Complete Portfolio




                      While FAA’s policy requires baseline cost estimates for the full life cycles of
FAA Does Not Have a   all projects under AMS, in practice, FAA has not yet developed a sound
Complete and Sound    estimate of the operations costs for each of its projects. As of
Operations Cost       February 1999, the agency had developed operations cost projections for
                      only 26 of the 70 projects identified by FAA as requiring an acquisition
Baseline for Its      program baseline or parameter sheet under AMS. For existing systems
Investments           already in the operations phase when AMS was implemented in April 1996,
                      FAA also lacked information on life-cycle cost projections for operating the
                      systems in the future or actual operations costs incurred for each of these
                      systems. Moreover, FAA officials throughout the organization indicated that
                      the estimates of operations costs that have been developed for projects
                      are not reliable. Given the lack of data on operations costs, FAA’s financial
                      plan for its modernization effort reports only costs funded by the facilities
                      and equipment budget account, omitting the operations costs associated
                      with its investments.

                      FAA  has long recognized the need to project the operations costs of its
                      projects over a multiyear period. In March 1993, FAA issued guidance under
                      its predecessor acquisition system that required an annual operations plan
                      to support long-range resource allocation planning. This plan—which was
                      never prepared—was supposed to summarize the operations and support
                      funding requirements of modernization projects and other acquisitions
                      funded by the operations appropriation. The operations plan was
                      supposed to contain a financial baseline that identified detailed operations
                      requirements over a 5-year period and general requirements for an
                      additional 10 years.

                      FAA  officials told us that the agency lacks the information needed to
                      reliably estimate operations costs over a project’s life cycle or to track
                      actual operations costs against estimates because it does not have a cost
                      accounting system. In January 1997, we reported that FAA lacked reliable
                      cost estimating processes and cost accounting practices needed to
                      effectively manage its investments in information technology.16 We
                      concluded that, as a result, the Congress does not have reliable cost
                      information to use in making decisions about FAA’s billion-dollar
                      modernization investments.

                      FAA has two initiatives that it believes will improve data on operations
                      costs. In August 1998, FAA formed an operations baseline team to address
                      agencywide concerns about the quality of its estimates of the operations

                      16
                        See Air Traffic Control: Improved Cost Information Needed to Make Billion-Dollar Modernization
                      Investment Decisions (GAO/AIMD-97-20, Jan. 22, 1997).



                      Page 28                                 GAO/RCED/AIMD-99-88 FAA’s Investment Management
                         Chapter 3
                         Lack of Oversight of the Operations Portion
                         of Projects Prevents FAA From Managing
                         Investments as a Complete Portfolio




                         costs for modernization projects and the lack of integration between the
                         facilities and equipment and operations budget accounts. This team is
                         evaluating FAA’s current processes for estimating and reporting on
                         operations costs, assessing the validity of operations cost data on a sample
                         of projects, and exploring ways to improve the estimating process. The
                         team, which expects to report its findings in May 1999, plans to develop a
                         10-year operations cost baseline for selected projects and to recommend
                         revisions to the budget formulation process that will allow FAA to budget
                         for the operations costs of both new and existing systems and to better
                         address the interrelationships between the facilities and equipment and
                         operations budget accounts.

                         FAA is also developing a cost accounting system that it believes will
                         provide more reliable information on actual operations costs; however,
                         FAA has missed its initial milestones for completing the new system’s
                         design and generating improved operations cost data. According to
                         officials responsible for the new cost accounting system, the agency had
                         planned to begin accumulating data for domestic and oceanic air traffic
                         services by October 1998. FAA officials indicated that they underestimated
                         the complexity of developing the system and that they now expect to
                         accumulate data for air traffic services by April 2000 and to fully
                         implement the system by April 2001.


                         While FAA’s budget provides detailed analyses of the actual and projected
FAA’s Operations         costs for each of the projects funded by the facilities and equipment
Budget Justifies Only    budget account, it provides very little project-level detail in its justification
a Small Portion of Its   for the operations budget account. In its fiscal year 1999 President’s
                         budget request, for example, FAA justified only 5 percent, or $295 million,
Spending for             of its total $5.6 billion operations budget account. This $295 million
Investments              request—which represented the incremental increase over the prior year’s
                         appropriation—contained information on all of the activities to be funded
                         in a given year, including pay increases for the current staff, costs of hiring
                         new staff, training, accident investigations, and other activities.

                         Among the details provided for the fiscal year 1999 operations budget
                         account, FAA provided project- or system-level justifications for about 1
                         percent, or $62 million, of its total $5.6 billion operations request. This
                         information, known as the “NAS Handoff” portion of the operations
                         account, contains 1-year estimates for new equipment, systems, and
                         facilities that were initially acquired with facilities and equipment funding
                         but will now be funded under the operations budget account. The NAS



                         Page 29                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
                  Chapter 3
                  Lack of Oversight of the Operations Portion
                  of Projects Prevents FAA From Managing
                  Investments as a Complete Portfolio




                  Handoff contains details on a variety of projected operations costs,
                  including controller overtime, logistics, systems maintenance, leased
                  communications, and flight inspections and procedures.

                  One factor contributing to the different levels of detail for the two budget
                  accounts is the lack of reliable data on operations costs for individual
                  projects or systems. Also, FAA officials told us that they only justify
                  incremental increases over the prior year’s operations budget account
                  because that is traditionally all that the Congress requires of FAA in
                  preparing its budget justifications.


                  AMS’lack of oversight of the operations portion of FAA’s investments
Conclusions       impedes the agency’s ability to rigorously assess and manage all of its
                  modernization projects as a complete strategic, investment portfolio and
                  to make sound decisions about continuing, modifying, or canceling
                  projects. Excluding operations projects from its selection process
                  prevents FAA from considering the relative merits of existing systems when
                  deciding which projects to fund each year.


                  We recommend that the Secretary of Transportation direct the
Recommendation    Administrator of FAA to establish a complete portfolio of
                  investments—including existing systems funded by the operations budget
                  account as well as projects funded by the facilities and equipment
                  account—and to require the Joint Resources Council to periodically
                  review the baseline status and merits of each of these investments
                  throughout their entire life cycles. As part of this portfolio, cost baselines
                  for operating and maintaining all projects should be developed, and this
                  information should be included in the agency’s financial plan for its
                  investments and in its annual budget request to the Congress.


                  FAA   agreed with our recommendation.
Agency Comments




                  Page 30                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 4

Weaknesses in the Selection, Control, and
Evaluation Phases Limit FAA’s Effectiveness
in Managing Its Investments
                            AMS  has weaknesses in all three investment management
                            phases—selection, control, and evaluation—that limit FAA’s ability to
                            effectively manage its modernization investments. First, the information
                            used to select projects is not validated to ensure quality control, and
                            critical cost information used to support selection decisions is of
                            questionable reliability. Second, FAA has not fully implemented an effective
                            process for controlling the baselines for the cost, schedule, benefits,
                            performance, and risks of its investments. Third, FAA lacks a
                            post-implementation evaluation process for assessing projects’ outcomes
                            and feeding lessons learned back into the selection and control phases for
                            future projects. Finally, FAA has not fully implemented a standardized
                            management information system for capturing and maintaining consistent,
                            reliable, and easily accessible data on investments.


                            AMS’ processes for selecting investments contain the key elements that
Weaknesses in Some          leading organizations follow to ensure the selection of projects that
Supporting Data Limit       enhance mission performance and that are cost-effective; however,
the Effectiveness of        weaknesses in some of the data used to support the selection processes
                            limit AMS’ effectiveness. AMS’ mission and investment analysis processes
the Selection Process       provide FAA’s senior management with a basis for screening proposed
                            projects; evaluating their relative costs, benefits, and risks; and selecting
                            projects for funding based on their relative merits. But the cost
                            information used to make selection decisions is of questionable reliability,
                            and there is little evidence that the data or underlying analyses used in the
                            selection process are validated to ensure accuracy, completeness, and
                            appropriateness. As a result, FAA’s managers cannot be assured that they
                            have all of the information needed to make sound selection decisions.


Cost Data Are of            Consistently producing reliable cost estimates for projects requires
Questionable Reliability,   defining institutional processes for deriving estimates and measuring
Though Improvements Are     actual performance against these estimates. However, the cost data used
                            in FAA’s selection process are of questionable reliability. The five projects
Under Way                   we reviewed prepared their cost estimates using techniques and data that
                            we criticized in our January 1997 report.17 We found that FAA’s
                            modernization program’s cost estimating processes do not satisfy
                            recognized standards and that the agency does not have a cost accounting
                            system capable of reliably accumulating full cost information for projects.
                            FAA’s cost estimating techniques do not satisfy recognized standards that
                            call for organizing and retaining cost information on projects in a

                            17
                              GAO/AIMD-97-20 (Jan. 22, 1997).



                            Page 31                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 4
Weaknesses in the Selection, Control, and
Evaluation Phases Limit FAA’s
Effectiveness in Managing Its Investments




historical database and using cost models that are calibrated and validated
on the basis of actual experience. FAA’s processes allow each project to
approach cost estimating in whatever manner its estimators choose. The
result is inconsistency in the rigor and discipline with which cost
estimates are derived, which in turn means estimates vary in their degrees
of reliability. For example, of the six projects reviewed in our 1997 report,
two were too poorly documented to permit any comparative analysis, and
none of the remaining four satisfied all the recognized standards.
Compounding the estimating weaknesses is FAA’s practice of presenting
cost estimates as precise point estimates, thus failing to disclose the
estimates’ inherent uncertainty and risks. Moreover, the effectiveness of
FAA’s cost estimating processes also relies heavily on the quality of
projects’ actual cost information, but FAA does not have a cost accounting
system for capturing and reporting the full costs of its projects.
Consequently, FAA cannot reliably use information about actual cost
experiences to improve its future cost estimating efforts. We
recommended that FAA institutionalize defined cost estimating processes
that include, among other items, a historical database and structured
approaches and tools.

In response to our 1997 report, FAA is developing a cost estimating process
for its projects that is intended to satisfy recognized estimating standards;
drafting guidance on reporting projects’ cost estimates as ranges rather
than precise point estimates and, in fact, reporting ranges on some
systems; and developing a cost accounting system. Additionally, FAA has
developed a document, known as a standard work breakdown structure,18
which provides a good first step toward the development of a historical
database on costs. FAA officials also indicated that they are completing a
cost estimating handbook that contains a detailed discussion of cost
estimating practices. When completed, this handbook should contribute to
improving FAA’s approach to estimating projects’ costs. However, it does
not require a disciplined process for estimating costs throughout the
agency, and the draft handbook acknowledges that FAA still needs to
develop sophisticated tools and a historical database to advance its cost
estimating processes. FAA has not established firm deadlines for
completing the handbook or the other tasks related to cost estimating. As
for the cost accounting system, as we discuss earlier, FAA officials
underestimated the complexity of developing the system and found that
their implementation milestones were unrealistic. The agency plans to
fully implement the cost accounting system by April 2001.

18
 The work breakdown structure, which contains a detailed list of activities to be accomplished in
carrying out projects, is used to develop life-cycle cost estimates for projects.



Page 32                                  GAO/RCED/AIMD-99-88 FAA’s Investment Management
                         Chapter 4
                         Weaknesses in the Selection, Control, and
                         Evaluation Phases Limit FAA’s
                         Effectiveness in Managing Its Investments




Selection Data Are Not   Leading organizations validate the information and analyses submitted in a
Validated                new project proposal, which helps to ensure that all the information is
                         up-to-date, cost numbers are accurate, benefits are quantified to the extent
                         possible, alternatives are identified, underlying assumptions are
                         reasonable, and sensitivity analyses are conducted. Validation is also
                         important for ensuring that a project’s risks are identified, that the impact
                         of risks on the project’s outcomes is quantitatively or qualitatively
                         projected, and that risk mitigation strategies are explained. Explicit
                         verification and validation steps sensitize decisionmakers to important
                         factors that have a bearing on projects’ actual outcomes, enhance
                         accountability, and decrease the likelihood that project proposals will
                         contain analyses that are based on inaccurate or incomplete data or faulty
                         assumptions.

                         FAA has not completely defined the requirements for the validation process
                         under AMS, and it is not fully carrying out validation activities. First, AMS
                         does not require the validation of all data used in the selection process.
                         Under AMS guidance, the FAA organization charged with carrying out the
                         investment analysis phase is responsible for validating only the cost and
                         schedule data, not the performance, benefits, or risk analyses used as part
                         of the selection process. Second, AMS guidance does not specify what steps
                         should be taken in validating selection data, nor does it require
                         documentation of the results and resolution of the validation process.
                         Finally, our review of the two projects for which investment decisions had
                         been made under AMS indicated that FAA is not fully carrying out validation
                         activities. For one project, FAA did not provide any documentary evidence
                         of the validation efforts that were performed. For the other project, FAA
                         provided evidence of a validation review of the cost-benefit analysis, but
                         most of the results of that review were not incorporated into the final
                         version of the analysis used to support the investment decision.


                         To control its projects at the agencywide level, FAA relies on periodic
FAA Has Not Fully        reviews of each project’s acquisition program baseline, which, as noted
Implemented Its          earlier, is a document that establishes a project’s cost, schedule, benefits,
Process for              and performance boundaries and is intended to be used to monitor a
                         project’s status in achieving those baselines. This document is incomplete,
Establishing and         however, because its schedule baseline does not include any milestones
Tracking Key Project     for project reviews during the operations phase of a project and because it
                         does not address a project’s risks. Moreover, FAA has not completed about
Baselines                half of the baselines for the projects or project segments that require
                         baselines. Additionally, the agency-level processes for tracking actual



                         Page 33                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
                             Chapter 4
                             Weaknesses in the Selection, Control, and
                             Evaluation Phases Limit FAA’s
                             Effectiveness in Managing Its Investments




                             baseline performance against estimates frequently provided incomplete
                             information on projects’ costs, schedules, benefits, and performance.
                             Finally, the investment control group of senior managers, the Joint
                             Resources Council, is not actively involved in monitoring all projects after
                             the investment decisions are made. As a result, FAA’s senior managers lack
                             key information needed to make sound decisions about the future of each
                             project.

                             Leading organizations maintain a cycle of continual control and
                             monitoring after a project has been selected. Senior executives review a
                             project at specific milestones as the project moves through its life cycle
                             and as the dollars spent on the project increase. At these milestones, the
                             executives compare the expected costs, risks, and benefits of earlier
                             phases with the actual costs incurred, risks encountered, and benefits
                             realized to date. During the control phase, senior executives determine
                             whether projects should be modified, continued, accelerated, delayed, or
                             terminated. The Clinger-Cohen Act also stresses the importance of
                             consistently monitoring the progress of federal investments in information
                             technology in meeting cost, schedule, and performance objectives.


FAA’s Requirements for       Our review indicated that FAA’s acquisition program baseline is designed to
Baseline Estimates Include   capture sufficient information on most of a project’s key baseline
Most, but Not All, Key       elements, except for two limitations in the areas of schedule and risk. One
                             limitation is that the schedule baseline does not address the operations
Project Parameters           phase of the project. At leading organizations, even after a project has
                             been implemented, senior managers regularly review how well the
                             acquired system meets organizational needs, including whether it needs
                             unexpected modifications or premature replacement to meet emerging
                             needs. These reviews are used to make decisions pertaining to the
                             retirement or replacement of systems. In addition, because operations
                             activities—such as hardware upgrades, system software changes, ongoing
                             training, and maintenance costs—can consume a significant level of
                             resources, a plan for the review of each project should be developed and
                             periodically reevaluated. Given that AMS is designed to track projects
                             during their entire life cycles, we expected to see one or more milestones
                             in the schedule baseline for a project review during the operations phase
                             of FAA’s information technology investments, which can last as long as 15
                             years. Such milestones would allow the Joint Resources Council to review
                             projects periodically during the operations phase to determine whether
                             expected performance requirements and benefits are actually being




                             Page 34                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 4
Weaknesses in the Selection, Control, and
Evaluation Phases Limit FAA’s
Effectiveness in Managing Its Investments




achieved in a cost-effective manner and to evaluate the need for upgrading
or enhancing the technology.

The second limitation is that the acquisition program baseline does not
include a project’s expected risks. The Office of Management and Budget’s
guidance requires that federal agencies estimate a project’s risks and
develop a strategy for mitigating those risks.19 While FAA performs a risk
analysis as a part of the investment analysis phase, the agency does not
systematically monitor each project’s risks during the control phase. Given
that the acquisition program baseline is FAA’s primary mechanism for
controlling projects, we expected it to include a baseline assessment of
risks that could be used to monitor mitigation and resolution of actual
risks that occur during a project’s life cycle. Although all five projects we
reviewed had discussions of risk issues during their project reviews, none
of them presented any systematic assessment of estimated versus actual
risks because risk is not a required element of the acquisition program
baseline.

Requiring an assessment of a project’s risks as an element of the
acquisition program baseline and systematically monitoring those risks
would allow FAA to identify “red flag” issues that may have an impact on a
project’s cost, schedule, and performance. For example, two of the
projects we reviewed—the Standard Terminal Automation Replacement
System and the NAS Infrastructure Management System—have experienced
difficulties with acquiring or developing software. FAA promoted both of
these as projects that would use commercial off-the-shelf software that
would require very little software development. For both projects, FAA
underestimated the lines of software code that needed to be developed or
modified, and as a result, the costs for the projects have increased and the
schedules have been delayed. FAA has historically had difficulty acquiring
software—the most costly and complex component of information
technology systems—and the agency has initiated some efforts to improve
its software acquisition processes.20 Improvements in these processes,
coupled with an identification of the risks associated with software
acquisition for new projects, would help FAA better manage such risks.




19
 See Office of Management and Budget Circular No. A-11, Part 3, Planning, Budgeting, and Acquisition
of Capital Assets, Executive Office of the President (June 1997), and Capital Programming Guide,
Version 1.0, Supplement to Office of Management and Budget Circular A-11 (July 1997).
20
 See Air Traffic Control: Immature Software Acquisition Processes Increase FAA System Acquisition
Risks (GAO/AIMD-97-47, Mar. 21, 1997).



Page 35                                 GAO/RCED/AIMD-99-88 FAA’s Investment Management
                              Chapter 4
                              Weaknesses in the Selection, Control, and
                              Evaluation Phases Limit FAA’s
                              Effectiveness in Managing Its Investments




FAA Has Approved Half of      FAA has identified 70 modernization projects or project segments that
Its Required Baselines, and   require baseline documentation.21 These 70 projects and segments account
Some Baseline                 for $1.266 billion, or 55 percent, of the $2.319 billion appropriation FAA
                              requested for fiscal year 2000 for all modernization activities funded by the
Documentation Is              facilities and equipment budget account. Most of the modernization
Incomplete                    activities that do not have baselines involve improving or sustaining ATC
                              facilities and other buildings; providing technical support services; or
                              funding personnel, compensation, benefits, and travel costs.

                              Of the 70 projects and segments, 10 involve projects that are currently in
                              the investment analysis phase of AMS and, hence, do not yet require an
                              approved baseline. Thus, 60 projects and segments that account for $1.205
                              billion in fiscal year 2000 estimated costs, currently require a baseline. Of
                              the 60, half have approved baseline documentation (either acquisition
                              program baselines or parameter sheets).22 These 30 projects and segments
                              account for $619.4 million in estimated costs for fiscal year 2000. Table 4.1
                              shows the status of FAA’s efforts to develop AMS baseline documentation on
                              the 60 ongoing projects and segments.




                              21
                                Some of the 70 baselines cover entire projects, while others cover segments of projects, meaning that
                              some projects have more than one baseline. For example, 4 of the 70 baselines are related to the
                              Automated Surface Observing System, 2 are related to the Oceanic Automation Program, 2 are related
                              to the NAS Infrastructure Management System, and 2 are related to the Air Traffic Management
                              (Infrastructure) Program.
                              22
                               As noted in chapter 2, parameter sheets summarize a subset of baseline information that is
                              designated for control by the Joint Resources Council and that is considered critical to assessing a
                              project’s ability to satisfy mission need, to achieve needed operational capability and benefits, and to
                              meet the schedule requirements of interdependent programs.



                              Page 36                                   GAO/RCED/AIMD-99-88 FAA’s Investment Management
                                        Chapter 4
                                        Weaknesses in the Selection, Control, and
                                        Evaluation Phases Limit FAA’s
                                        Effectiveness in Managing Its Investments




Table 4.1: Status of FAA’s Efforts to
Develop AMS Baseline Documentation      Dollars in millions
on 60 Ongoing Projects and Segments                                                                       Fiscal year
for Which Investment Decisions Have     FAA’s efforts to                                              2000 estimated Percentage of
Been Made                               develop                   Number of           Percentage of           cost of total fiscal year
                                        baseline                projects and           total projects   projects and 2000 estimated
                                        documentation             segments            and segments         segments              costs
                                        Acquisition
                                        program
                                        baselines and
                                        parameter sheets
                                        approved by the
                                        Joint Resources
                                        Council                             30                    50           $619.4               51
                                        Parameter sheets
                                        undergoing
                                        review for
                                        validation of
                                        operations costs
                                        and other funding
                                        issues                                  4                  7            292.1               24
                                        Acquisition
                                        program
                                        baselines and
                                        parameter sheets
                                        being planned                       26                    43            293.9               24
                                        Total - all
                                        acquisition
                                        program
                                        baselines and
                                        parameter
                                        sheets                              60                  100          $1,205.4               99a
                                        a
                                        Percentages do not add to 100 because of rounding.

                                        Source: GAO’s analysis of FAA’s data.



                                        Our review of five projects found that some of the baseline documentation
                                        (both acquisition program baselines and parameter sheets) is incomplete.
                                        Of the five projects, two had acquisition program baselines (plus
                                        parameter sheets), and three had parameter sheets only. Only the
                                        performance element was fully documented on all of the projects, while
                                        the cost, benefits, and schedule elements were missing some of the
                                        information required by AMS. For the cost baseline, for example, two of the
                                        five projects did not estimate operations costs, and one of the two projects
                                        with acquisition program baselines did not provide information on the
                                        detailed cost elements required by AMS guidance. For the benefits baseline,
                                        one of the two projects with acquisition program baselines did not identify




                                        Page 37                                     GAO/RCED/AIMD-99-88 FAA’s Investment Management
                          Chapter 4
                          Weaknesses in the Selection, Control, and
                          Evaluation Phases Limit FAA’s
                          Effectiveness in Managing Its Investments




                          the measurements that would be used to determine whether a benefit had
                          been achieved. For the schedule baseline, none of the projects estimated
                          the in-service decision, a milestone required by AMS that specifies when the
                          newly acquired system or equipment is certified ready for operational use.


FAA Is Not Completely     Our review and FAA’s internal evaluation of acquisition reform23 found that
Monitoring Projects’      the I-BEAM and acquisition review processes frequently provide incomplete
Actual Performance        reporting on projects’ estimated versus actual performance in the areas of
                          cost, schedule, benefits, and performance. We reviewed the monthly status
Against Their Baselines   reports on the five projects and found information was missing on all of
                          the baseline elements. For example, none of the monthly status reports on
                          the five projects analyzed operations costs; assessed estimated versus
                          actual benefits, even though benefits were projected for fiscal years 1997
                          or 1998 on four of the five projects; or contained information on the
                          performance requirements outlined in the acquisition program baseline or
                          parameter sheet to confirm whether the original baseline requirements
                          still applied. Our results are consistent with those of FAA’s internal
                          evaluation, which found that the cost, schedule, and performance data in
                          the monthly status reports were generally inconsistent with the estimates
                          in the acquisition program baselines and parameter sheets. With regard to
                          FAA’s tracking of baseline information through the acquisition review
                          process, our review of the five projects showed that information was
                          missing for all of the baseline parameters, as shown in table 4.2.




                          23
                             Evaluation of Acquisition Reform—The First Two Years: April 1996-March 1998, FAA Program
                          Evaluation Branch, Office of Systems Architecture and Investment Analysis (May 29, 1998).



                          Page 38                                GAO/RCED/AIMD-99-88 FAA’s Investment Management
                                       Chapter 4
                                       Weaknesses in the Selection, Control, and
                                       Evaluation Phases Limit FAA’s
                                       Effectiveness in Managing Its Investments




Table 4.2: Analysis of the Missing
Baseline Information Reported in       Baseline element                  Missing information
Acquisition Reviews of Five Projects   Cost                              Four projects did not analyze operations costs.

                                                                         Three projects did not compare the original (or revised)
                                                                         baseline cost estimates with actual results achieved to
                                                                         date.

                                                                         Of the two projects with acquisition program baselines,
                                                                         neither provided information on the detailed cost
                                                                         elements required by AMS guidance.
                                       Schedule                          Three projects did not compare the original (or revised)
                                                                         baseline milestones with actual results achieved to date.

                                                                         One project did not address two of the Joint Resources
                                                                         Council-controlled milestones: contract award and
                                                                         in-service decision.

                                                                         One project only showed milestones for 3 years of the
                                                                         9-year acquisition cycle identified in the parameter sheet.
                                       Benefits                          The acquisition review did not address the baseline
                                                                         benefits in the acquisition program baselines or
                                                                         parameter sheets for any of the five projects, even
                                                                         though, for four of them, benefits were projected for fiscal
                                                                         years 1997 or 1998.
                                       Performance                       Four projects did not address all of the performance
                                                                         parameters contained in the acquisition program baseline
                                                                         or parameter sheet, and one of those projects had no
                                                                         information on performance at all.
                                       Source: GAO’s analysis of FAA’s data.



                                       FAA  officials managing the five projects we reviewed cited several factors
                                       to explain the lack of complete baseline data on their projects. First,
                                       project officials told us that frequent budget reductions—imposed either
                                       by the Congress or by FAA—make it very difficult to establish a stable
                                       baseline and to monitor a project’s performance against that baseline.
                                       Second, AMS guidance on the acquisition program baseline requirements
                                       has evolved over time, and project officials told us that detailed guidance
                                       was not available to projects that established their baselines during the
                                       first year of AMS’ implementation. Finally, the two mechanisms used by FAA
                                       to monitor projects’ status—the monthly status report and the acquisition
                                       reviews—were developed prior to AMS and thus do not always address the
                                       specific baseline elements that are supposed to be contained in the
                                       acquisition program baselines, report the current status of all baseline
                                       estimates, or report the actual deviations from those estimates.




                                       Page 39                                 GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 4
Weaknesses in the Selection, Control, and
Evaluation Phases Limit FAA’s
Effectiveness in Managing Its Investments




We recognize that variances in funding levels exist and, in some cases,
have had an impact on FAA’s ability to manage its projects. However, we
and others have found that FAA’s problems with baseline parameters have
resulted from factors other than funding. Although AMS’ detailed guidance
and reporting requirements are evolving, establishing complete acquisition
program baseline documentation that has been approved by the Joint
Resources Council helps to ensure that all projects are being held
accountable to the cost, schedule, benefits, and performance baselines
established when the Council made the initial investment decision.
Furthermore, the absence of complete, up-to-date data on estimated
versus actual results means that FAA has little assurance that its estimates
of projects’ costs, schedule, benefits, performance, or risks are sound and
accurate or that projects will be managed so that they meet the agency’s
expectations. This, in turn, restricts the ability of FAA’s senior management
to make sound decisions about continuing the agency’s investments.

FAA’s long-standing problems with implementing projects that meet their
cost, schedule, and performance objectives illustrate the need for the
agency to better manage its baselines. As noted earlier, over the past 17
years, FAA’s modernization projects have experienced substantial cost
overruns, lengthy delays, and significant performance shortfalls, problems
that have persisted since the implementation of AMS in April 1996. Two of
the projects in our review provide examples of continuing cost and
schedule problems. In the case of the Standard Terminal Automation
Replacement System, although FAA has not officially changed the baseline
approved in February 1996, the baseline is in jeopardy of being breached
because of unions’ concerns about human-factor and design issues,24 the
refinement of the project’s requirements, and the interjection of a new
project phase.25 FAA estimates that these issues have the potential to
increase the project’s costs from $294 million to $410 million above the
approved baseline. FAA also estimates that the project’s initial completion
date could be delayed by almost 2-1/2 years. Similarly, in the case of the
NAS Infrastructure Management System, the agency has not officially
changed the baseline that was approved in March 1997. However, the
project’s leader expects significant baseline breaches to occur, including a
58-percent increase in the costs from $100.8 million to $159.5 million, and
a 123-percent increase in the schedule from 48 to 107 months for the
project’s total duration.


24
 Concerns were raised by two unions, the National Air Traffic Controllers Association and the
Professional Airways Systems Specialists.
25
  GAO/RCED-99-25, Dec. 3, 1998.



Page 40                                 GAO/RCED/AIMD-99-88 FAA’s Investment Management
                              Chapter 4
                              Weaknesses in the Selection, Control, and
                              Evaluation Phases Limit FAA’s
                              Effectiveness in Managing Its Investments




Joint Resources Council       Under AMS, the Joint Resources Council conducts reviews and makes
Has Limited Involvement       decisions on each project funded from the facilities and equipment
in Reviewing Projects After   account, from the determination of mission needs to the point at which an
                              investment decision is made. After the investment decision, the Joint
the Investment Decisions      Resources Council generally only becomes directly involved in monitoring
                              a project when there is a potential or actual breach of the established
                              baseline. As a result, the Joint Resources Council is not proactively
                              involved in making decisions about the future of most projects when they
                              are being developed, deployed, operated, and maintained; when decisions
                              are being made about technology upgrades or other enhancements; and
                              when assessments are being made about whether the system or product is
                              achieving the intended benefits and meeting the expected performance
                              requirements.

                              The Acquisition Executive, who chairs the Joint Resources Council, told
                              us that he highlights projects whose baseline status is problematic
                              according to data in the monthly status report and that he provides
                              additional oversight for those projects. However, as we have stated, the
                              data contained in the monthly status reports are incomplete and generally
                              inconsistent with the estimated baseline elements in the acquisition
                              program baselines and parameter sheets.


                              FAA  does not have a defined, documented process for conducting
FAA’s AMS Lacks a             post-implementation reviews of projects to assess their performance and
Post-Implementation           to improve the selection and control of its other investments. FAA performs
Review Process for            some elements of a post-implementation review in its life-cycle
                              management review process, including such tasks as the independent
Evaluating                    operational test and evaluation of some projects prior to their deployment,
Investments                   operational performance monitoring, customer satisfaction surveys, and
                              periodic reviews throughout an investment’s life cycle. However, this
                              process is not standardized and is not required for all projects. As a result,
                              there is no evidence that changes, especially to the selection and control
                              phases, are being implemented based on lessons learned. Although FAA has
                              not yet designed or implemented a post-implementation review process
                              for individual projects, its fiscal year 1999 performance plan for the
                              Research and Acquisitions operating division includes a new requirement
                              for a post-deployment assessment of NAS modernization systems.

                              The evaluation phase “closes the loop” on the investment management
                              process by comparing actual results against baseline estimates to assess
                              performance and identify areas where future decision-making can be



                              Page 41                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
                     Chapter 4
                     Weaknesses in the Selection, Control, and
                     Evaluation Phases Limit FAA’s
                     Effectiveness in Managing Its Investments




                     improved. Lessons that are learned during the evaluation phase can be
                     used to improve future decisions about selecting and controlling projects.
                     Central to this process is the post-implementation review, with its
                     evaluation of the historical record of a project and its comparison of
                     actual versus expected costs and benefits. Recognizing the importance of
                     this phase of investment management, the Clinger-Cohen Act requires
                     federal agencies to evaluate the performance of information technology
                     projects and to use that information to decide whether to continue,
                     modify, or terminate projects.

                     At leading organizations, this review generally occurs about 3 to 12
                     months after a project has reached its end point (i.e., the point at which
                     the project has been fully implemented or canceled) and is generally
                     conducted by a group other than the project team to ensure that the
                     review is independent and objective. In conducting post-implementation
                     reviews, an organization can survey customers to determine users’
                     satisfaction with the completed product and how well the project supports
                     business processes; assess whether the investment has had its intended
                     impacts on mission goals, cost savings, compliance with the system’s
                     architecture, and other issues involving information accuracy, timeliness,
                     adequacy, and appropriateness; and evaluate current and future technical
                     issues associated with the investment.


                     Until recently, FAA lacked a centralized, standardized management
FAA’s Efforts to     information system or historical database for capturing and maintaining
Implement an         project information. While FAA had a number of stand-alone databases
Agencywide           within different groups, none provided a complete picture of estimates,
                     the assumptions that made up the estimates, revisions, and actual
Management           performance on projects. As a result, agency officials had no assurance
Information System   that the project data from these stand-alone systems were complete,
                     accurate, and up-to-date.
Do Not Include Key
Selection Data       In March 1999, FAA officials began implementing a centralized repository
                     and management information system under AMS—called the Simplified
                     Program Information Reporting and Evaluation (SPIRE)—to provide access
                     to projects’ baselines and other information. This system consolidates
                     project information related to FAA’s processes for managing key baselines,
                     including data from the acquisition program baselines and parameter
                     sheets, monthly status reports, baseline management notices, meeting
                     minutes from the acquisition reviews, Joint Resources Council decisions,
                     and other baseline information. The system does not, however, contain



                     Page 42                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
Chapter 4
Weaknesses in the Selection, Control, and
Evaluation Phases Limit FAA’s
Effectiveness in Managing Its Investments




key information from the selection process, such as mission need
statements, cost-benefit analyses, risk assessments, or other required
reports. FAA plans to implement SPIRE in three phases. Phase I, which
commenced in March 1999, provides the capability to store and display the
status and variance reports that have been input by project leaders.
Subsequent phases, which do not yet have implementation dates, will
focus on automatically generating various reports on projects’ baseline
status and variances.

Informed management decisions can be made only if accurate, reliable,
and up-to-date information from all phases of the investment management
process is included in the decision-making process. To do this requires
that agencies have a uniform mechanism—that is, a management
information system with uniform data standards and entry
procedures—for collecting, automating, and processing data on projects’
expected versus actual outcomes. Data in this system should include the
initial cost, schedule, benefits, performance, and risk estimates that were
developed during the selection process. Various analyses that were
conducted to initially justify the project, along with revised estimates,
reasons for revisions, and actual performance measured against the
estimates, should also be included. These data need to be continually
updated as each project’s implementation continues and as expenditures
increase. The data also need to be easily accessible to both the project
team and senior managers.

A management information system, if kept accurate and up-to-date, can
make data verification and validation easier by allowing an organization to
track costs, risks, and other factors over time. It is also essential from the
standpoint of establishing an organizational memory throughout the
selection, control, and evaluation phases of the investment management
process. As such, it can be used to help assess whether projects are still
aligned with mission needs and organizational objectives, determine
whether projects are meeting planned performance goals, and identify
possible revisions to the overall investment management process based on
previous experiences and lessons learned.

FAA’s efforts to consolidate project baseline information in SPIRE are a
positive step toward improving the agency’s mechanisms for tracking
projects and helping to ensure the consistency of information and
reporting on all the projects in its investment portfolio. However, SPIRE
provides an incomplete record of project information since it does not
include key information from the selection process.



Page 43                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
                      Chapter 4
                      Weaknesses in the Selection, Control, and
                      Evaluation Phases Limit FAA’s
                      Effectiveness in Managing Its Investments




                      AMS  has weaknesses in its selection, control, and evaluation phases that
Conclusions           impede FAA’s ability to manage its investments effectively and to make
                      sound decisions about continuing, modifying, or canceling projects. First,
                      using data that have not been validated and unreliable cost information
                      reduces the likelihood that FAA’s managers can make informed decisions
                      about the relative merits of competing investments. Second, requiring
                      FAA’s senior managers to stay actively involved in the process of
                      controlling project baselines and providing them with complete
                      information on all projects is critical to monitoring how well projects are
                      achieving their intended results. Third, establishing post-implementation
                      reviews is essential for evaluating projects’ performance and identifying
                      areas for which future decision-making could be improved during the
                      selection and control phases. Finally, establishing a standardized
                      management information system that includes complete information from
                      all three investment management phases—selection, control, and
                      evaluation—will help facilitate FAA’s efforts to track projects’ costs, risks,
                      and other factors over time, providing senior managers with uniform,
                      accurate, reliable, and easily accessible data on all the projects in the
                      agency’s portfolio. Taking steps to correct these weaknesses increases the
                      likelihood that FAA’s projects will meet established cost and schedule
                      objectives and contribute to measurable improvements in the agency’s
                      mission performance.


                      We recommend that the Secretary of Transportation direct the
Recommendations       Administrator of FAA to take the following actions:

                  •   Improve the selection process by (1) establishing clearly defined
                      procedures for validating projects’ cost, schedule, benefit, performance,
                      and risk information and (2) requiring documentation of the results of the
                      validation procedures applied to each project.
                  •   Strengthen control over investments by (1) revising the acquisition
                      program baseline requirements to include project risks and to add
                      milestones for project reviews during the operations phase and
                      (2) ensuring that project officials fully track and document estimated
                      versus actual results on all the elements (i.e., costs, schedule, benefits,
                      performance, and risks) contained in the baseline documentation.
                  •   Initiate post-implementation evaluations for projects within 3 to 12 months
                      of deployment or cancellation to compare the completed projects’ costs,
                      schedule, performance, and mission improvement outcomes with the
                      original estimates.




                      Page 44                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
                      Chapter 4
                      Weaknesses in the Selection, Control, and
                      Evaluation Phases Limit FAA’s
                      Effectiveness in Managing Its Investments




                  •   Incorporate key information from the selection process (e.g., mission need
                      statements, cost-benefit analyses, and risk assessments) into FAA’s
                      management information system for investments.


                      FAA   agreed with our recommendations.
Agency Comments




                      Page 45                             GAO/RCED/AIMD-99-88 FAA’s Investment Management
Appendix I

Background and Status of Five Projects


                     This appendix provides detailed information on the purpose, scope, and
                     status of the five information technology projects we reviewed. The
                     Federal Aviation Administration (FAA) considers each of these projects key
                     to replacing the aging National Airspace System’s (NAS) infrastructure or to
                     improving its capacity and effectiveness. Each project is estimated to
                     require the expenditure of more than $100 million from the agency’s
                     facilities and equipment budget account before it becomes operational.



NAS Infrastructure
Management System

Background           The NAS Infrastructure Management System will provide the next
                     generation of tools, services, and operational philosophies that govern the
                     management, operation, and maintenance of the NAS infrastructure.
                     Currently, the air traffic control system relies on a nationwide
                     infrastructure of facilities and equipment to provide communications,
                     navigation, surveillance, and automation system capabilities. To offset the
                     continual loss of experienced technical staff devoted to the maintenance
                     of the NAS systems, FAA has dispersed its maintenance workforce into work
                     centers that, in some cases, are colocated with “high-impact” facilities
                     such as terminal radar approach control facilities and en route centers.
                     The agency is also using a remote maintenance monitoring system to
                     recognize and address problems in a more timely manner. However, this
                     monitoring system simply collects performance data from individual
                     systems and logs historical maintenance actions. The NAS Infrastructure
                     Management System will build on the investment made in this monitoring
                     system and will focus on workforce and event management—two separate
                     but related components of system management. Through this project, FAA
                     will be able to track and monitor the actual costs of providing NAS services.

                     The mission need statement for this project was submitted in December
                     1995 and revalidated in February 1996, and a cost-benefit analysis was
                     completed in December 1995. FAA approved the investment decision and
                     acquisition program baseline for Phase I of the project in March 1997.

                     According to project officials, the NAS Infrastructure Management System
                     will be implemented in three phases. Only Phase I, which covers fiscal
                     years 1998 through 2002, is fully defined. In this phase, FAA plans to create
                     the initial infrastructure needed to manage, operate, and maintain overall




                     Page 46                        GAO/RCED/AIMD-99-88 FAA’s Investment Management
                     Appendix I
                     Background and Status of Five Projects




                     NAS operations. This infrastructure will consist of a National Operations
                     Control Center in Herndon, Va., and three regional operational control
                     centers in Hampton, Ga.; Olathe, Kans.; and San Diego, Calif. Below these
                     three regional centers will be up to 50 service operations centers modeled
                     after the facilities that are currently colocated with high-impact facilities.
                     Under Phase II, FAA plans to provide centralized asset management,
                     develop customer and user interaction tools, and analyze technical and
                     cost trends. Under Phase III, the agency plans to provide intelligent fault
                     correlation, information sharing, and additional functionality
                     commensurate with NAS technological improvements.


Status               According to FAA, the National Operations Control Center became
                     operational in January 1999. The three regional centers are scheduled to
                     begin operations in October 2000. Phase I of the NAS Infrastructure
                     Management System will support FAA’s Free Flight Phase I in fiscal years
                     2000-2002.26 Phases II and III are currently undergoing an investment
                     analysis to determine if further investment is warranted.

                     Although FAA has not officially changed the project’s baseline that was
                     approved in 1997, the project’s leader estimated significant breaches in
                     December 1998, as follows: (1) the cost baseline is expected to increase by
                     58 percent, from $100.8 million to $159.5 million, and (2) the schedule is
                     expected to increase by 123 percent, from 48 to 107 months’ total duration
                     for Phase I.



Oceanic Automation
Program

Background           The Oceanic Automation Program is designed to provide a platform for
                     improved air traffic control over the oceans. It evolved as a series of
                     projects—each advancing on the technology of its predecessor—from the
                     Oceanic Display and Planning System into the Oceanic Automation
                     System, and now, into the Advanced Oceanic Automation System. In the
                     late 1980s, the Oceanic Display and Planning System improved oceanic
                     traffic control by providing flight data processing and a situational display
                     of estimated aircraft positions. This system also alerted controllers when

                     26
                       Free flight is a new system of air traffic management that will provide controllers and pilots with new
                     technologies and procedures that will allow them to increase the safety, capacity, and efficiency of air
                     traffic operations throughout the NAS.



                     Page 47                                   GAO/RCED/AIMD-99-88 FAA’s Investment Management
         Appendix I
         Background and Status of Five Projects




         any flight plan or any route change requested by a pilot (referred to as
         conflict probe capability) violated appropriate separation standards. In the
         early 1990s, FAA introduced the Oceanic Automation System, which
         improved data display and communications. This system is now being
         upgraded to the Advanced Oceanic Automation System, which is designed
         to provide such features as a new flight data processor, automatic
         dependent surveillance position reporting, an advanced conflict probe,
         and data link. FAA awarded a contract to the Raytheon Systems
         Corporation in September 1995 for the Advanced Oceanic Automation
         System. The contract is composed of flexible segments, which will allow
         for incremental functional development and delivery of benefits. Oceanic
         air traffic control systems are installed at the en route centers at Oakland
         and New York and in Anchorage, Alaska.

         The mission need statement was approved in May 1992. In October 1992,
         the acquisition plan was approved to consolidate and integrate the primary
         oceanic improvement projects into a single Oceanic Automation Program.
         A revised mission need statement and acquisition plan was approved in
         January 1994.


Status   Since FAA awarded the Advanced Oceanic Automation System’s contract in
         September 1995, the scope of the project has been gradually cut back from
         an original plan of five segments (that is, five incremental deliveries of
         capabilities) to only a portion of the first segment. In July 1996, 10 months
         after the contract’s award, FAA canceled segments 3, 4, and 5 of the project
         because the agency recognized that the cost of executing these segments
         was beyond the funding that had been allocated for this project. As a
         result, FAA abandoned many controller productivity tools needed to
         increase the system’s capacity. Then, in December 1996, funding concerns
         forced FAA to revise the Segment 2 of the project, which was designed to
         replace infrastructure hardware and software that supports controller
         equipment. Eventually, in September 1997, FAA canceled the entire
         Segment 2 because the agency needed to use the project’s funds to correct
         Year 2000 problems in existing oceanic automation software and because
         it needed to transfer funds to the Host replacement program.27

         Meanwhile, FAA’s contractor was reporting performance problems with the
         Segment 1 of the project, which adds data link and automatic dependent
         surveillance in the oceanic environment. To avoid a potential $45 million

         27
           The Host replacement project replaces en route center and oceanic automation hardware that has
         reached the end of its commercial support life and may have problems with Year 2000 date
         requirements.



         Page 48                                 GAO/RCED/AIMD-99-88 FAA’s Investment Management
                  Appendix I
                  Background and Status of Five Projects




                  cost overrun for this segment, FAA reduced its scope in September 1998 by
                  eliminating the capability for automatic dependent surveillance. According
                  to project officials, the remaining elements of Segment 1 (the air-to-ground
                  data link, the ground-to-ground data link, and controller tools) have
                  successfully completed the operational test and evaluation and are
                  expected to be delivered on schedule. The last site implementation is
                  estimated for October 1999.



Operational and
Supportability
Implementation
System

Background        The Operational and Supportability Implementation System project
                  (1) replaces the Flight Service Automation System’s hardware and
                  software with a leased commercial, off-the-shelf-based service;
                  (2) provides an improved graphic weather display capability; and
                  (3) incorporates direct user access functionality that is currently being
                  obtained through two direct user access terminal contracts. The
                  integration of these three capabilities and functions into a single system
                  will enable flight service specialists to more efficiently provide weather
                  and flight-planning information for pilots.

                  The mission need statement was approved in October 1993 and revalidated
                  in December 1996. The acquisition program baseline was approved in
                  April 1997, and in August 1997, FAA awarded a contract to Harris
                  Corporation for the project. The contract requires Harris to provide up to
                  61 operational systems and 3 support systems.


Status            Since FAA awarded the contract, the project’s schedule has slipped because
                  the development effort has been larger than planned. FAA’s January 1998
                  review of the Harris system’s architecture for the project revealed that the
                  contractor’s commercial, off-the-shelf solution was not as mature as FAA
                  had envisioned when the contract was awarded and that many of the
                  contractor’s commercial products did not fully satisfy FAA’s requirements.
                  In May 1998, the agency decided to replace workstation consoles in
                  response to human-factor concerns raised by the unions that represent its




                  Page 49                            GAO/RCED/AIMD-99-88 FAA’s Investment Management
                     Appendix I
                     Background and Status of Five Projects




                     controllers and the technicians. This caused the project’s costs to increase
                     by $15.8 million. FAA also delayed first-site implementation from July 1998
                     to January 1999—a 6-month slip. The protracted development effort is not
                     expected to delay the completion of the project, with the last site to
                     receive the system still scheduled to be operational in August 2001.



Standard Terminal
Automation
Replacement System

Background           The Standard Terminal Automation Replacement System is designed to
                     replace FAA’s automated radar terminal system, which comprises 15- to
                     25-year-old air traffic controller workstations and the supporting computer
                     systems that allow controllers at terminal radar approach control facilities
                     to separate and sequence aircraft. According to FAA, the old system is
                     prone to failures and requires extensive maintenance. The old system also
                     has capacity constraints that restrict the agency from making required
                     safety and efficiency enhancements. Besides remedying those problems,
                     the new equipment is also expected to allow the system to increase the
                     level of air traffic control automation and to improve surveillance,
                     communications, and weather display. This system replaced a segment of
                     another project (the Advanced Automation System) that was terminated
                     because of serious cost and schedule problems.

                     The mission need statement for the Standard Terminal Automation
                     Replacement System was submitted in July 1993 and revised in June 1995.
                     The acquisition plan was approved in March 1996, and in September 1996,
                     FAA signed a contract with Raytheon Corporation to acquire this system. In
                     producing it, Raytheon originally intended to rely exclusively on
                     commercially available hardware and, to a large extent, on commercially
                     available software.

                     The initial strategy for replacing and enhancing the system is divided into
                     two stages. Stage 1 is expected to provide the same functions as the
                     current automated radar terminal systems. Stage 2 is expected to
                     implement new functions to help controllers move aircraft more safely and
                     efficiently. In 1997, FAA created another stage, known as early display
                     configuration, because of concerns about operational problems at Ronald
                     Reagan National Airport in Washington, D.C. This new stage will be



                     Page 50                            GAO/RCED/AIMD-99-88 FAA’s Investment Management
                      Appendix I
                      Background and Status of Five Projects




                      implemented prior to Stages 1 and 2. The new stage replaces the current
                      controller displays and monitoring equipment but uses the existing
                      computer system and software. It also provides an emergency backup
                      system.


Status                Although FAA has not officially changed the project’s baseline that was
                      approved in 1996, the baseline is in jeopardy of being breached because of
                      the unions’ concerns about human-factor and design issues, the
                      refinement of the requirements, and the interjection of a new project
                      phase. FAA estimates that these issues have the potential to increase the
                      project’s costs anywhere from $294 million to $410 million over the
                      approved baseline. FAA also estimates that the project’s initial completion
                      date could be delayed by almost 2-1/2 years. In addition, the project has
                      experienced other challenges mainly involving software testing. While
                      project officials stated that they have been able to absorb the cost
                      increases associated with this issue within the existing baseline, additional
                      problems could cause further cost increases and schedule delays. The last
                      site implementation is estimated for February 2005.



Voice Switching and
Control System

Background            The Voice Switching and Control System replaces existing communication
                      systems at en route centers with an expandable, highly reliable system for
                      both ground-to-ground and air-to-ground communication. This system will
                      also provide communication capability for new en route center controller
                      workstations that are being installed. FAA is also installing the Voice
                      Switching and Communications System Training and Backup Switch—an
                      emergency backup communications system—at all en route centers.

                      This system was designed to provide the communication capabilities for
                      the new Initial Sector Suite System workstations under the Advanced
                      Automation System program. By the time the contract was awarded in
                      December 1991 to the Harris Corporation, FAA had spent 5 years
                      developing prototypes and had incurred cost growth of about $1 billion.28
                      The contract required Harris to deliver 23 systems—21 for en route


                      28
                       According to project officials, the primary reason for this growth was the inability of commercially
                      available products to effectively and accurately manage air traffic control communications functions.



                      Page 51                                  GAO/RCED/AIMD-99-88 FAA’s Investment Management
         Appendix I
         Background and Status of Five Projects




         centers and 2 support systems. FAA’s plans called for the Voice Switching
         and Communications System to be installed with both the current
         equipment and with the new controller workstations. During the initial
         development, the cost of the project increased by $53.1 million to
         approximately $1.45 billion—primarily because of FAA’s decision in 1994 to
         cancel the Initial Sector Suite System component of the Advanced
         Automation System and replace it with the Display System Replacement
         project. The restructuring resulted in the need for additional equipment
         and testing and in the retention of contractor and project personnel longer
         than planned to field the communications equipment with new controller
         workstation equipment. FAA has also added new functionality requirements
         to the project.

         The original concept for this project was defined in 1980. In 1984,
         operational requirements were finalized, and in 1985, the project was
         approved for development. A revised draft mission need statement was
         completed in January 1994.


         Harris developed and installed the system in the existing en route
Status   controller work stations in February 1997—5 months ahead of the
         schedule established at the time the contract was awarded. Harris is
         currently reinstalling the controller interface equipment into the en route
         Display System Replacement controller workstations. Harris has
         completed the software development for the primary system to be fielded
         with the new Display System Replacement controller workstations.
         According to the project manager, the project has not encountered any
         technical problems and is not expected to incur any major delays. The last
         site implementation is estimated for May 2000.

         FAA is in the process of installing the emergency backup system at en route
         centers, and by November 1999, FAA expects to have completed
         installation at all en route centers, the FAA Technical Center, and the FAA
         Academy.




         Page 52                            GAO/RCED/AIMD-99-88 FAA’s Investment Management
Appendix II

Major Contributors to This Report


                        John H. Anderson, Jr.
Resources,              Rita A. Grieco
Community, and          Tina M. Kinney
Economic                Belva M. Martin
                        Thomas F. Noone
Development
Division, Washington,
D.C.
                        David L. McClure
Accounting and          Colleen M. Phillips
Information             Robert C. Reining
Management Division,
Washington, D.C.




(348085)                Page 53                 GAO/RCED/AIMD-99-88 FAA’s Investment Management
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested