oversight

Air Traffic Control: Improved Cost Information Needed to Make Billion Dollar Modernization Investment Decisions

Published by the Government Accountability Office on 1997-01-22.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to the Secretary of
                 Transportation



January 1997
                 AIR TRAFFIC
                 CONTROL
                 Improved Cost
                 Information Needed to
                 Make Billion Dollar
                 Modernization
                 Investment Decisions




GAO/AIMD-97-20
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Accounting and Information
      Management Division

      B-271530

      January 22, 1997

      The Honorable Federico Peña
      Secretary of Transportation

      Dear Mr. Secretary:

      This report addresses the reliability of the cost information critical to capital investment
      decision-making on air traffic control projects. Specifically, we evaluated the Federal Aviation
      Administration’s processes for estimating what projects will cost and the related accounting for
      actual project costs.

      This report contains recommendations to you. The head of a federal agency is required by 31
      U.S.C. 720 to submit a written statement on actions taken on these recommendations. You
      should send your statement to the Senate Committee on Governmental Affairs and the House
      Committee on Government Reform and Oversight within 60 days after the date of this report.
      You must also send the written statement to the House and Senate Committees on
      Appropriations with the agency’s first request for appropriations made over 60 days after the
      date of this report.

      We are providing copies of this report to the Subcommittees on Transportation of the House
      and Senate Committees on Appropriations, the House and Senate Committees on the Budget,
      the Director of the Office of Management and Budget, the Administrator of the Federal Aviation
      Administration, and other interested parties. Copies will also be made available to others upon
      request. Please call me at (202) 512-6412 if you have any questions concerning the report. Other
      contributors to this report are listed in appendix V.

      Sincerely yours,




      Dr. Rona B. Stillman
      Chief Scientist for Computers
        and Telecommunications
Executive Summary


             Wisely managing new investments in capital assets, whether commercial
Purpose      real estate, factory plant and equipment, or information technology,
             requires complete and reliable information about alternative investment
             options, such as what the investment is expected to cost and whether
             actual costs are in line with expectations. The Federal Aviation
             Administration (FAA) is in the midst of a multibillion dollar, mission-critical
             capital investment program aimed at modernizing its aging air traffic
             control (ATC) infrastructure. The vast majority of these ATC capital
             investment projects, both in terms of money and number, involve
             software-intensive information acquisition, processing, and display
             systems. Past GAO reports have noted that ATC modernization projects
             routinely cost considerably more than envisioned, in part because of poor
             cost estimates.

             Because of the importance of complete and reliable estimated and actual
             cost information to ATC investment management decisions, GAO examined
             FAA’s ATC project cost estimating and cost accounting practices. Our
             objectives were to determine whether (1) ATC project cost estimates are
             based on good estimating processes and (2) actual ATC project costs are
             being properly accumulated.


             Faced with growing air traffic volume and a deteriorating ATC
Background   infrastructure, in 1981 FAA began an ambitious ATC modernization program.
             This program, which includes investments in both new ATC facilities and
             related support as well as new and upgraded software-intensive computer
             systems, totals over 200 separate projects that FAA estimates will cost
             more than $34 billion between 1982 and 2003. Of this total, 169 projects,
             estimated by FAA to cost about $21 billion, are ATC information systems. In
             fact, FAA expects the cost to develop and deploy ATC information systems
             between now and the year 2003 will be $6.9 billion, or about one-third, of
             the information systems component of the modernization.

             Over the past 15 years, FAA’s ATC modernization projects have experienced
             substantial cost overruns, lengthy schedule delays, and significant
             performance shortfalls. To illustrate, the long-time centerpiece of this
             modernization program—the Advanced Automation System—was
             restructured in 1994 after estimated costs tripled from $2.5 billion to
             $7.6 billion and delays in putting significantly less-than-promised system
             capabilities into operation were expected to run 8 years or more. Because
             of the size, importance, complexity, and poor track record of the




             Page 2                                          GAO/AIMD-97-20 Air Traffic Control
    Executive Summary




    modernization program, GAO designated it a high-risk information
    technology initiative in 1995.1

    In 1994, GAO reported on how leading organizations improved mission
    performance through information technology. Among other matters, GAO
    reported that successful organizations manage information system
    projects as investments, continually assessing the quality of projects’
    estimated costs and carefully monitoring projects’ actual costs against
    these estimates.2 This “best practice” relative to cost estimates has since
    been embodied in the Clinger-Cohen Act of 1996, which requires the
    selection of information technology investments on the basis of competing
    projects’ estimated costs, benefits, and risks. In addition, the Chief
    Financial Officers (CFO) Act of 1990 requires federal agencies to maintain
    integrated accounting and financial management systems that permit the
    development and reporting of cost information and the systematic
    measurement of performance.

    According to Carnegie Mellon University’s Software Engineering Institute
    (SEI), deriving credible estimates of software-based systems’ costs is a
    function of how thorough and disciplined an organization’s estimating
    processes are. Accordingly, SEI has published six institutional process
    requisites that organizations in the business of building or acquiring
    software-intensive systems must possess if they are to consistently
    produce reliable cost estimates.3 These are

•   a corporate memory, or historical database(s), for cataloging cost
    estimates, revisions, reasons for revisions, actuals, and other descriptive
    information, such as any constraints or trends that affect the project;
•   structured processes for estimating software size and the amount and
    complexity of existing software that can be reused;
•   cost models calibrated/tuned to reflect demonstrated accomplishments on
    similar past projects;
•   audit trails that record and explain the values used as cost model inputs;
•   processes for dealing with externally imposed cost or schedule constraints
    in order to ensure the integrity of the estimating process; and
•   data collection and feedback processes that foster capturing and correctly
    interpreting data from work performed.

    1
     High-Risk Series: An Overview (GAO/HR-95-1, Feb. 1995).
    2
     Executive Guide: Improving Mission Performance Through Strategic Information Management and
    Technology (GAO/AIMD-94-115, May 1994).
    3
    Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software
    Organizations (CMU/SEI-95-SR-005, Jan. 1995).



    Page 3                                                       GAO/AIMD-97-20 Air Traffic Control
                   Executive Summary




                   Additionally, SEI, along with other experts in systems and software
                   engineering and project management, advocates the acknowledgement
                   and full disclosure of the inherent imprecision of these estimates.
                   Accordingly, these experts recommend qualifying early project estimates
                   by disclosing the level of uncertainty associated with them, and making
                   the estimate more precise as the project is completed and the
                   uncertainties are eliminated.

                   Although a recent development, the Statement of Federal Financial
                   Accounting Standards no. 4 (SFFAS 4), Managerial Cost Accounting
                   Concepts and Standards for the Federal Government,4 which took effect
                   October 1, 1996, requires full cost accounting by federal agencies’
                   reporting entities to support management decision-making. These
                   standards build on the CFO Act of 1990, which requires that agencies
                   develop and maintain integrated accounting and financial management
                   systems in order to develop and report cost information and systematically
                   measure performance, among other functions. In addition, the Federal
                   Managers’ Financial Integrity Act of 1982 (FMFIA) requires that agency
                   systems of internal control comply with prescribed standards and provide
                   reasonable assurances that among other things, obligations and costs
                   comply with applicable laws and that revenues and expenditures
                   applicable to agency operations are recorded and accounted for properly.



                   FAA’s ATC modernization program’s cost estimating processes do not satisfy
Results in Brief   recognized estimating requisites, and its cost accounting practices do not
                   provide for proper accumulation of actual project costs. The result is an
                   absence of reliable project cost and financial information that the
                   Congress has legislatively specified and that leading public-sector and
                   private-sector organizations point to as essential to making fully informed
                   investment decisions among competing ATC projects. Without this
                   information, the likelihood of poor ATC investment decisions is increased,
                   not only when a project is initiated but also throughout its life cycle. It also
                   means that the Congress does not have reliable cost information to use in
                   making funding decisions about FAA. Such a situation is unacceptable
                   when making small investments, but it is especially egregious when
                   making multimillion or billion dollar investments in mission-critical ATC
                   systems.


                   4
                    The Federal Accounting Standards Advisory Board (FASAB) recommends federal accounting
                   standards to the Director of OMB, the Secretary of the Treasury, and the Comptroller General. Once
                   approved, the standards are issued by OMB and GAO.



                   Page 4                                                       GAO/AIMD-97-20 Air Traffic Control
Executive Summary




With respect to cost estimating, FAA fails to meet five of the six process
requisites that SEI’s Checklists and Criteria for Evaluating the Cost and
Schedule Estimating Capabilities of Software Organizations says should be
institutionally entrenched and consistently used for information
technology projects. In the absence of such institutional policies to guide
ATC project cost estimating, FAA has adopted a cost estimating process that
allows each ATC project to approach cost estimating in whatever manner
its estimators choose. The result is inconsistency in the rigor and
discipline with which ATC project cost estimates are derived, which in turn
means estimates of varying degrees of reliability. In fact, when comparing
the approaches that six ATC projects used to derive their current official
life cycle cost estimates to SEI’s project-specific criteria entitled, A
Manager’s Checklist for Validating Software Cost and Schedule Estimates,5
 GAO found that two were too poorly documented to permit any
comparative analysis, while none of the remaining four satisfied all of the
criteria SEI associates with highly credible estimates.6 Compounding these
estimating process weaknesses is FAA’s practice of presenting cost
estimates as precise, point estimates. By doing so, FAA fails to disclose the
estimates’ inherent uncertainty and risks, thus further limiting the
estimates’ decision-making value and credibility.

With respect to cost accounting, FAA is not accumulating all ATC project
costs. In fact, FAA does not have a cost accounting system for capturing
and reporting the full cost of its ATC projects. Instead, FAA decisionmakers
use accounting and financial management systems that omit relevant
project costs, such as those associated with FAA project management. The
result is that FAA cannot reliably measure the ATC projects’ actual cost
performance against established baselines, and cannot reliably use
information relating to actual cost experiences to improve future cost
estimating efforts.




5
 CMU/SEI-95-SR-004, Jan. 1995.
6
  Since the focus of this effort is to assess FAA’s cost estimating processes and not to validate the
accuracy or completeness of the estimates, we did not evaluate the quality of the estimates.



Page 5                                                            GAO/AIMD-97-20 Air Traffic Control
                                      Executive Summary




Principal Findings

Weak Cost Estimating                  The effectiveness of FAA’s investment management process is in part a
Processes and Practices               function of the quality of the cost estimates used in the process.
Undermine FAA’s Ability to            Accordingly, GAO assessed FAA’s cost estimating processes and practices at
                                      two levels. First, on an institutional level, FAA’s cost estimating processes
Make Informed Investment              for ATC systems partially meet one and do not meet the remaining five
Decisions                             institutional process requisites that experts say are embedded in leading
                                      information technology development and acquisition organizations.7 These
                                      six requisites, which are listed in table 1, are the product of SEI’s research
                                      of leading government and private-sector estimating practices. Table 1 also
                                      provides a summary assessment of the degree to which FAA satisfies each
                                      requisite. According to SEI, an organization must address all six requisites
                                      in order to produce reliable estimates. The Clinger-Cohen Act emphasizes
                                      the need for reliable cost estimates to allow effective information
                                      technology investment decision-making.

Table 1: Summary of FAA’s
Satisfaction of SEI’s Institutional   SEI requisites                                      FAA institutional policies and practices
Requisites                            Corporate memory                                    No
                                      Sizing and reuse structure                          No
                                      Extrapolation using actual performance              No
                                      Audit trails                                        Partial
                                      Integrity in the face of dictated limits            No
                                      Data collection and feedback on actual              No
                                      performance

                                      Second, on a project level, GAO reviewed six ATC projects (five of the
                                      largest ATC projects and one small project) as case studies to determine
                                      how these projects’ current official life cycle cost estimates were derived
                                      in the absence of any institutional processes for estimating ATC projects’
                                      costs. To do so, GAO compared the cost estimating approaches used on
                                      each project to SEI criteria developed expressly for project managers to
                                      use in deciding whether or not to rely on an estimate.8 GAO found that two
                                      of the six projects had little or no documentation supporting the derivation
                                      of their estimates. The four remaining projects’ estimating approaches
                                      were documented, but none satisfied all the SEI criteria. Of these four, one

                                      7
                                       This evaluation was performed using SEI’s Checklists and Criteria for Evaluating the Cost and
                                      Schedule Estimating Capabilities of Software Organizations.
                                      8
                                       This evaluation was performed using SEI’s A Manager’s Checklist for Validating Software Cost and
                                      Schedule Estimates.



                                      Page 6                                                        GAO/AIMD-97-20 Air Traffic Control
                               Executive Summary




                               satisfied most of the criteria SEI associates with a credible cost estimate.
                               The other three failed to satisfy most of the SEI criteria. To FAA’s credit,
                               however, estimators on two of the projects that failed to satisfy most of
                               SEI’s criteria are revising life cycle cost estimates using more rigorous
                               approaches.

                               FAA’s estimating process weaknesses are exacerbated by its portrayal of
                               ATC projects’ cost estimates as firm, point estimates. According to software
                               engineering experts, cost estimates are by definition imprecise,
                               particularly early in a system’s development cycle, and should be qualified
                               to fully disclose this uncertainty. By choosing to present its ATC projects’
                               cost estimates as discrete points, FAA is improperly implying a level of
                               precision that cannot be supported, thereby further limiting the value of
                               the estimates to congressional and agency investment decisionmakers and
                               potentially misleading them.

                               During the course of GAO’s review, FAA organizations initiated several
                               efforts to improve ATC cost estimating processes. Specifically, FAA’s
                               program assessment organization is exploring the possibility of integrating
                               cost estimating functions across FAA, and is advocating FAA adoption of a
                               standard tool for recording cost model data. Another group is testing a
                               system for tracking a project’s actual operations and maintenance (O&M)
                               costs. The group’s goal is to eventually use this system’s historical data to
                               help estimate future projects’ O&M costs. However, these separate
                               initiatives are not coordinated, relatively new, and not endorsed or
                               institutionalized by FAA.


Shortcomings in Project        The effectiveness of FAA’s investment management process, including its
Cost Accounting Impede         cost estimating process, also relies heavily on the quality of actual project
FAA’s Ability to Effectively   cost information. However, FAA does not adequately accumulate all project
                               cost information needed by federal decisionmakers to make fully informed
Manage ATC Investments         decisions about the billions of dollars being spent on ATC projects. Reliable
and Improve Cost               financial information includes the full cost of a project, including direct
Estimating                     and indirect costs. Without this information, an organization does not
                               know how much is being spent on various ATC projects and, therefore,
                               does not have a basis for managing current costs or reliably estimating
                               future costs. Accountability for such costs is required by FMFIA and the CFO
                               Act.

                               Because FAA does not have a cost accounting system for ATC modernization
                               projects, project and corporate management as well as FAA oversight



                               Page 7                                         GAO/AIMD-97-20 Air Traffic Control
                      Executive Summary




                      agencies and institutions, such as OMB and the Congress, rely on
                      information produced by an assortment of financial systems. None of
                      these financial systems provide for the systematic accumulation and
                      reporting of all relevant project costs.

                      In August 1996, FAA set up an organization and subsequently initiated an
                      effort to acquire a cost accounting system. FAA has thus far specified
                      general functional requirements and plans to procure an off-the-shelf
                      system that can be tailored to FAA’s needs. FAA plans to have this cost
                      accounting system in place by October 1, 1997.


                      Because the success of FAA’s investment analysis and decision-making
Recommendations       process depends in large measure on the reliability of ATC project cost
                      information, GAO recommends that the Secretary of Transportation direct
                      the FAA Administrator to institutionalize defined processes for estimating
                      ATC projects’ costs. At a minimum, these processes should include the
                      following SEI requisites, each of which is described in more detail in this
                      report:

                  •   a corporate memory (or historical database), which includes cost and
                      schedule estimates, revisions, reasons for revisions, actuals, and relevant
                      contextual information;
                  •   structured approaches for estimating software size and the amount and
                      complexity of existing software that can be reused;
                  •   cost models calibrated/tuned to reflect demonstrated accomplishments on
                      past projects;
                  •   audit trails that record and explain all values used as cost model inputs;
                  •   processes for dealing with externally imposed cost or schedule constraints
                      in order to ensure the integrity of the estimating process; and
                  •   data collection and feedback processes that foster capturing and correctly
                      interpreting data from work performed.

                      GAO also recommends that the Secretary direct the Administrator to
                      immediately begin disclosing the inherent uncertainty and range of
                      imprecision in all ATC projects’ official cost estimates presented to
                      executive oversight agencies or the Congress.

                      Additionally, GAO recommends that the Secretary direct the Administrator
                      to acquire or develop and implement a managerial cost accounting
                      capability that will satisfy the requirements of SFFAS 4, Managerial Cost
                      Accounting Concepts and Standards for the Federal Government. This



                      Page 8                                        GAO/AIMD-97-20 Air Traffic Control
                       Executive Summary




                       system capability should provide the cost accounting and financial
                       management information needed by FAA management and those who make
                       investment decisions. Such information should include full life cycle costs,
                       which include the costs of resources consumed by a project that directly
                       or indirectly contribute to the output and the costs of identifiable
                       supporting services provided by other organizations within the reporting
                       entity.

                       GAO further recommends that the Secretary report FAA’s lack of a cost
                       accounting capability for its ATC modernization as a material internal
                       control weakness in the Department’s fiscal year 1996 FMFIA report and in
                       subsequent annual FMFIA reports until the problem is corrected.

                       Also, GAO recommends that the Secretary direct the Administrator to
                       report to the Secretary and FAA’s authorizing and appropriation
                       committees on progress being made on these recommendations as part of
                       the agency’s fiscal year 1999 budget submission.


                       GAO  received oral comments on a draft of this report from senior
Agency Comments        Department of Transportation (DOT) and FAA officials, including
and GAO’s Evaluation   representatives from the Office of the Secretary of Transportation, the
                       Executive Assistant to the FAA Chief Financial Officer, the FAA Manager of
                       the Cost Accounting System Division, and the FAA Program Director for
                       Investment Analysis and Operations Research. These officials agreed with
                       most of GAO’s findings, conclusions, and recommendations. However, they
                       stated that because they plan to have a cost accounting system in place by
                       October 1, 1997, they do not believe that our recommendations for
                       implementing a cost accounting capability and listing the lack of a cost
                       accounting system as a material internal control weakness in the agency’s
                       FMFIA report are necessary. While we are encouraged by FAA’s recent
                       actions to procure a cost accounting system, we nevertheless believe that
                       our recommendations are still warranted to ensure that FAA’s efforts are
                       complete and timely. Until FAA’s cost accounting system—which is still
                       very early in its acquisition life cycle—is developed, installed, and
                       operational, the lack of cost accounting information will continue to pose
                       a risk to sound project management and investment decision-making. As
                       such, implementation of this capability warrants aggressive action and
                       disclosure as a material internal control weakness in the agency’s FMFIA
                       report until the new system is in place and producing accurate cost
                       information.




                       Page 9                                        GAO/AIMD-97-20 Air Traffic Control
Contents



Executive Summary                                                                                      2


Chapter 1                                                                                             14
                          ATC at a Glance                                                             14
Introduction              Overview of the ATC Modernization Program                                   17
                          Sound Investment Decisions Require Reliable Cost Estimating                 24
                            and Accounting Practices and Data
                          Objectives, Scope, and Methodology                                          27

Chapter 2                                                                                             29
                          Policies and Practices for Estimating Costs Do Not Satisfy “Best            29
ATC Modernization           Practices” Requisites
Lacks the Institutional   Individual ATC Projects’ Cost Estimating Approaches Are                     32
                            Inconsistent and Sometimes Unreliable
Capacity to Produce       FAA’s Portrayal of ATC Projects’ Cost Estimates Implies an                  35
Reliable Cost               Unjustified Level of Precision
Estimates                 FAA Organizations Have Initiated Efforts to Improve Cost                    36
                            Estimating
                          Conclusions                                                                 37
                          Recommendations                                                             37
                          Agency Comments                                                             38

Chapter 3                                                                                             39
                          Agencies Are Required to Maintain Adequate Accounting Systems               39
FAA Is Not                ATC Project Costs Are Not Being Properly Accumulated                        40
Adequately                FAA Lacks a Cost Accounting System for Its ATC Modernization                41
                            Program
Accounting for ATC        Conclusions                                                                 43
Project Costs             Recommendations                                                             43
                          Agency Comments and Our Evaluation                                          44

Appendixes                Appendix I: Summary Comparison of SEI’s Requisites for                      46
                            Reliable Estimating Processes and FAA’s Institutional Policies
                            and Practices
                          Appendix II: Detailed Comparison of SEI’s Checklist for Each                47
                            Requisite and FAA Practices for ATC Projects
                          Appendix III: SEI Checklist for Validating the Reliability of a             55
                            Project’s Cost Estimate
                          Appendix IV: Six ATC Projects’ Cost Estimates’ Satisfaction of              59
                            SEI’s Checklist




                          Page 10                                      GAO/AIMD-97-20 Air Traffic Control
          Contents




          Appendix V: Major Contributors to This Report                              60


Tables    Table 1: Summary of FAA’s Satisfaction of SEI’s Institutional               6
            Requisites
          Table I.1: Comparison of SEI’s Requisites for Reliable Estimating          46
            Processes and FAA Policies and Practices for ATC Projects
          Table II.1: A Corporate Memory                                             47
          Table II.2: Structured Processes for Estimating Software Size and          51
            Reuse
          Table II.3: Mechanisms for Extrapolating From Demonstrated                 52
            Accomplishments on Past Projects
          Table II.4: An Audit Trail                                                 53
          Table II.5: Integrity in Dealing With Dictated Costs and Schedules         53
          Table II.6: Data Collection and Feedback Processes That Foster             54
            Capturing and Correctly Interpreting Data From Work Performed
          Table III.1: SEI Checklist for Validating the Reliability of a             55
            Project’s Cost Estimate
          Table IV.1: Six ATC Projects’ Cost Estimates’ Satisfaction of SEI’s        59
            Checklist

Figures   Figure 1.1: Summary of ATC Over the Continental United States              16
            and Oceans
          Figure 1.2: Organizational Chart Highlighting ATC Modernization            20
            and Maintenance Management Structure
          Figure 2.1: Cost Estimating Accuracy Over Time                             35

          Abbreviations

          ACEIT        Automated Cost Estimating Integrated Tools
          APO          Office of Aviation Policy and Plans
          ARA          Associate Administrator for Research and Acquisitions
          ARINC        Aeronautical Radio Incorporated
          ASD-400      Investment Analysis and Operations Research
          ASR-9        Airport Surveillance Radar-9
          ATC          air traffic control
          ATS          Associate Administrator for Air Traffic Services
          CASE         Computer-Aided Software Engineering
          CBAS         Cost Benefit Analysis System
          CFO          Chief Financial Officer
          CMM          Capability Maturity Model
          COPS         Cost of Performance System
          CPMS         Cost Performance Management System



          Page 11                                     GAO/AIMD-97-20 Air Traffic Control
Contents




DAFIS      Departmental Accounting and Financial Information
                 System
DCCR       Display Channel Complex Rehost
DOT        Department of Transportation
DSR        Display System Replacement
FAA        Federal Aviation Administration
FASAB      Federal Accounting Standards Advisory Board
FMFIA      Federal Managers’ Financial Integrity Act of 1982
FMS        Financial Management System
F&E        Facilities and Equipment
GPS        Global Positioning System
IPDS       Integrated Product Development System
IPT        Integrated Product Team
ISSS       Initial Sector Suite System
JRC        Joint Resources Council
OIG        Office of the Inspector General
OMB        Office of Management and Budget
O&M        Operations and Maintenance
PCB&T      Personnel Compensation, Benefits, and Travel
RE&D       Research, Engineering, and Development
REDMACS    Research, Engineering, and Development Monitoring,
                 Analysis, and Control System
SEI        Software Engineering Institute
SFFAS 4    Statement of Federal Financial Accounting Standards no. 4
SLIM       Software Life Cycle Intermediate Model
STARS      Standard Terminal Automation Replacement System
TRACON     Terminal Radar Approach Control
VSCS       Voice Switching and Control System
WAAS       Wide Area Augmentation System




Page 12                                  GAO/AIMD-97-20 Air Traffic Control
Page 13   GAO/AIMD-97-20 Air Traffic Control
Chapter 1

Introduction


                      The Federal Aviation Administration’s (FAA) primary mission is to ensure
                      safe, orderly, and efficient air travel throughout the United States. FAA’s
                      ability to fulfill this mission depends on the adequacy and reliability of the
                      nation’s air traffic control (ATC) system, a vast network of computer
                      hardware, software, and communications equipment. Sustained growth in
                      air traffic and aging equipment have strained the current ATC system,
                      limiting the efficiency of ATC operations. To combat these trends, in 1981
                      FAA embarked on an ambitious ATC modernization program. FAA estimates
                      that it will spend about $34 billion on the program between 1982 and 2003.

                      Our work over the years has chronicled many FAA failures in meeting ATC
                      projects’ cost, schedule, and performance goals.1 As a result, we
                      designated FAA’s ATC modernization as a high-risk information technology
                      initiative in our 1995 report series on high-risk programs.


                      Automated information processing and display, communication,
ATC at a Glance       navigation, surveillance, and weather resources permit air traffic
                      controllers to view key information, such as aircraft location, aircraft flight
                      plans, and prevailing weather conditions, and to communicate with pilots.
                      These resources reside at, or are associated with, several ATC
                      facilities—flight service stations, air traffic control towers, terminal radar
                      approach control (TRACON) facilities, and air route traffic control centers
                      (en route centers). These facilities’ ATC functions are described below.

                  •   About 90 flight service stations provide pre-flight and in-flight services,
                      such as flight plan filing and weather report updates, primarily for general
                      aviation aircraft.
                  •   Airport towers control aircraft on the ground and before landing and after
                      take-off when they are within about 5 nautical miles of the airport, and up
                      to 3,000 feet above the airport. Air traffic controllers rely on a combination
                      of technology and visual surveillance to direct aircraft departures and
                      approaches; maintain safe distances between aircraft; and communicate
                      weather-related information, clearances, and other instructions to pilots
                      and other personnel.
                  •   Approximately 180 TRACONs sequence and separate aircraft as they
                      approach and leave busy airports, beginning about 5 nautical miles and
                      ending about 50 nautical miles from the airport, and generally up to 10,000
                      feet above the ground, where en route centers’ control begins.


                      1
                       Air Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-95-175FS, May 26, 1995); Air
                      Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-94-167FS, Apr. 15, 1994); and Air
                      Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-93-121FS, Apr. 16, 1993).



                      Page 14                                                     GAO/AIMD-97-20 Air Traffic Control
    Chapter 1
    Introduction




•   Twenty en route centers control planes over the continental United States
    in transit and during approaches to some airports. Each en route center
    handles a different region of airspace, passing control from one to another
    as respective borders are reached until the aircraft reaches TRACON
    airspace. En route center controlled airspace usually extends above 18,000
    feet for commercial aircraft. En route centers also handle lower altitudes
    when dealing directly with a tower, or when agreed upon with a TRACON.
•   Two en route centers—Oakland and New York—also control aircraft over
    the ocean. Controlling aircraft over oceans is radically different from
    controlling aircraft over land because radar surveillance only extends 175
    to 225 miles offshore. Beyond the radars’ sight, controllers must rely on
    periodic radio communications through a third party—Aeronautical Radio
    Incorporated (ARINC), a private organization funded by the airlines and FAA
    to operate radio stations—to determine aircraft locations.

    See figure 1.1 for a visual summary of the processes for controlling aircraft
    over the continental United States and oceans.




    Page 15                                       GAO/AIMD-97-20 Air Traffic Control
                                         Chapter 1
                                         Introduction




Figure 1.1: Summary of ATC Over the Continental United States and Oceans




                                                          Oceanic
                                                          En Route
                                                           Center



                                                                              ARINC




                                      En Route
                                       Center
                       TRACON




                                                 Flight Service                       En Route
                                                     Station                           Center




             Airport Tower




                                                                             TRACON

           Departure Control
                                                                  Airport Tower
           Approach Control
           Local and Ground Control

           Continental
           United States




                                         Page 16                                                 GAO/AIMD-97-20 Air Traffic Control
                      Chapter 1
                      Introduction




                      The ATC system of the late 1970s was a blend of several generations of
Overview of the ATC   automated and manual equipment, much of it labor-intensive and obsolete.
Modernization         FAA recognized that it could increase ATC operating efficiency by increasing

Program               automation. Additionally, FAA forecasted increased future demand for air
                      travel, brought on by airline deregulation of the late 1970s. It also
                      anticipated that meeting the demand safely and efficiently would require
                      improved and expanded services, additional facilities and equipment,
                      improved workforce productivity, and the orderly replacement of aging
                      equipment. Accordingly, in December 1981, FAA initiated its plan to
                      modernize, automate, and consolidate the existing ATC system by the year
                      2000.

                      This ambitious modernization program includes the acquisition of new
                      radars and automated data processing, navigation, and communication
                      equipment in addition to new facilities and support equipment. FAA
                      estimates that the modernization will cost over $34 billion through the
                      year 2003 and total over 200 separate projects. ATC information systems
                      make up a large portion of this total, accounting for 169 projects costing
                      $20.7 billion. The Congress will have provided FAA with approximately
                      $14.7 billion of the $20.7 billion through fiscal year 1997.

                      Over the past 15 years, FAA’s modernization projects have experienced
                      substantial cost overruns, lengthy schedule delays, and significant
                      performance shortfalls. To illustrate, the long-time centerpiece of this
                      modernization program—the Advanced Automation System—was
                      restructured in 1994 after estimated costs tripled from $2.5 billion to
                      $7.6 billion and delays in putting significantly less-than-promised system
                      capabilities into operation were expected to run 8 years or more. Similarly,
                      increases in per-unit costs for five other major ATC projects2 have ranged
                      from 50 to 511 percent, and schedule delays have averaged almost 4 years.

                      Our past work on the ATC modernization raised a number of concerns,
                      including concerns about the reliability of projects’ cost estimates. For
                      example, our review of FAA’s Oceanic Display and Planning System found
                      that cost and schedule estimates were questionable because they were
                      based solely on managers’ judgments and were not revised to reflect
                      changing project demands and conditions.3 Also, our work on AAS

                      2
                       The five projects and their respective percentage change in unit costs are the Voice Switching and
                      Control System (511 percent), the Integrated Terminal Weather System (129 percent), the Airport
                      Surface Detection Equipment (56 percent), the Aviation Weather Observing System (51 percent), and
                      the Terminal Doppler Weather Radar (50 percent).
                      3
                       Air Traffic Control: FAA Needs to Justify Further Investment in Its Oceanic Display System
                      (GAO/IMTEC-92-80, Sept. 30, 1992).



                      Page 17                                                       GAO/AIMD-97-20 Air Traffic Control
                         Chapter 1
                         Introduction




                         highlighted that FAA underestimated the complexity of the system
                         development and relied on its contractor’s deficient cost estimating
                         systems.4 Most recently, we reported that the organizational culture
                         intrinsic to FAA has been to deflate realistic estimates of cost and schedule
                         in order to gain funding approval.5 In its internal risk management
                         guidance, FAA acknowledges that it has a history of unreliable project cost
                         estimates, attributing some of its unenviable record in ATC projects’ cost
                         growth to setting unrealistically low cost estimates, either because of poor
                         cost estimating processes or inadequate system descriptions.6


ATC Modernization and    Two major FAA organizations play key roles in the modernization and
Maintenance Management   evolution of ATC systems—the Office of the Associate Administrator for
Structure                Research and Acquisitions (ARA) and the Office of the Associate
                         Administrator for Air Traffic Services (ATS). The first, ARA, manages the
                         research, development, and acquisition of modernization projects. Within
                         ARA, two groups are responsible for acquiring systems, while the others
                         handle cross-cutting management functions (e.g., budget formulation, cost
                         estimation, and program evaluation). Also, the William J. Hughes
                         Technical Center is the ATC system test and evaluation facility and
                         supports ATC systems’ research, engineering, and development.

                         ARA  employs an Integrated Product Development System (IPDS) approach.
                         A key component of IPDS is the use of Integrated Product Teams (IPT),
                         which are cross-functional teams aligned with major business and
                         functional areas (i.e., en route, terminal, weather and flight services, air
                         traffic management, oceanic, communications, navigation, surveillance,
                         infrastructure, and information systems). IPT members include systems
                         and specialty engineers, logistics personnel, lawyers, contract specialists,
                         and representatives from the organization responsible for the system’s
                         operations and maintenance. IPTs are responsible for systems research,
                         development, acquisition, and installation. Product teams within these IPTs
                         are responsible for individual ATC system acquisitions or projects. For
                         example, the en route IPT has product teams for the Display Channel
                         Complex Rehost, the Display System Replacement, the Voice Switching
                         and Control System, and several other en route systems.




                         4
                          Air Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-94-167FS, Apr. 15, 1994).
                         5
                          Aviation Acquisition: A Comprehensive Strategy Is Needed for Cultural Change at FAA
                         (GAO/RCED-96-159, Aug. 22, 1996).
                         6
                          Cost growth can occur for various reasons, including changing requirements, unstable funding
                         streams, poor project and/or contract management, and poor estimates.
                         Page 18                                                      GAO/AIMD-97-20 Air Traffic Control
Chapter 1
Introduction




The second major organization involved with ATC systems is ATS. ATS is
responsible for directing, coordinating, controlling, and ensuring the safe
and efficient utilization of the national airspace system. Organizations
within ATS are responsible for planning, operating, and maintaining ATC
systems. Responsibility for managing ATC systems is transferred from the
IPT to ATS once the systems have been installed and are operational. See
figure 1.2 for a visual summary of the ATC modernization and maintenance
management structure.




Page 19                                       GAO/AIMD-97-20 Air Traffic Control
                                                                     Chapter 1
                                                                     Introduction




Figure 1.2: Organizational Chart Highlighting ATC Modernization and Maintenance Management Structure


                                                                                           Administrator
                                                                                              AOA

                                                              Assistant
                                                            Administrator            Deputy Administrator
                                                          for System Safety                 ADA
                                                                 ASY

                                     Assistant                             Assistant                           Assistant                            Assistant
   Chief Counsel                                                         Administrator                                                          Administrator for
                                   Administrator                                                             Administrator
       AGC                                                           for Government and                                                        Policy, Planning and
                                  for Civil Rights                                                         for Public Affairs
                                                                        Industry Affairs                                                      International Aviation
                                       ACR                                   AGI                                 APA                                    API

                                                                                                                                  Office of Aviation         Europe, Africa and
                                                                                                                                  Policy and Plans           Middle East Office
                                                                                                                                        APO                        AEU

                                                                                                                                Office of Environment              Asia-Pacific
                                                                                                                                     and Energy                       Office
                                                                                                                                         AEE                          APC

                                                                                                                                      Office of                Latin America-
                                                                                                                                International Aviation        Caribbean Office
                                                                                                                                         AIA                        ALC


     Associate                                           Associate                 Associate                                                                      Associate
                            Associate
   Administrator                                       Administrator             Administrator                                                               Administrator for
                          Administrator
 for Administration                                  for Civil Aviation       for Regulation and                                                             Air Traffic Services
                           for Airports
       AAD                                                Security                Certification                                                                     ATSa
                               ARP
                                                           ACS                       AVR


  Office of Business      Office of Airport              Office of Civil                                                                                       Airway Facilities          Office of System
                                                                               Office of Accident      Air Traffic System
    Information &          Planning and                 Aviation Security                                                           Air Traffic Service
                                                                                 Investigation        Requirements Service                                         Service                    Capacity
     Consultation          Programming                    Intelligence                                                                     AAT
                                                                                      AAI                     ARS                                                   AAF                         ASC
         ABC                    APP                            ACI
                          Office of Airport              Office of Civil           Aircraft                                              Air Traffic                    NAS
  Office of Financial                                                                                      Airway Facilities                                                               Airport Capacity
                             Safety and                 Aviation Security         Certification                                          Resource                     Operations
       Services                                                                                             Requirements                                                                       Planning
                             Standards                     Operations              Service                                              Management                      AOP
         ABA                                                                                                    AFR                                                                            ASC-100
                                AAS                           ACO                     AIR                                                   ATX
   Office of Human                                      Office of Civil            Office of                  Air Traffic                Air Traffic                  Operational            Airspace
       Resource                                       Aviation Security                                      Requirements
                                                                               Aviation Medicine                                         Operations                    Support            Capacity Planning
    Management                                       Policy and Planning                                         ATR
                                                                                      AAM                                                   ATO                         AOS                  ASC-200
         AHR                                                 ACP

                                                                                Flight Standards                                     Air Traffic Airspace           NAS Transition
                                                                                                                                        Management                       and
                                                                                     Service
                                                                                                                                             ATA                    Implementation
                                                                                       AFS                                                                               ANS

                                                                                   Office of                                                                        Spectrum Policy
                                                                                  Rulemaking                                            Field Divisions            and Management
                                                                                     ARM                                                                                 ASR


                                                                                                                                                                      Resources
                                                                                                                                                                     Management
                                                                                                                                                                        AFZ

                                                                                                                                                                   Aviation System
                                                                                                                                                                      Standards
                                                                                                                                                                         AVN


                                                                                                                                                                    Field Divisions




                        Central                                                                            Northwest                                                                       Mike Monroney
     Alaskan                                  Eastern           Great Lakes       New England                                   Southern               Southwest        Western-Pacific
                        Region                                                                             Mountain                                                                         Aeronautical
     Region                                   Region              Region            Region                                       Region                 Region             Region
                         ACE                                                                                Region                                                                             Center
       AAL                                     AEA                 AGL               ANE                                          ASO                    ASW                AWP
                                                                                                             ANM                                                                                AMC



                                                                     Page 20                                                                                GAO/AIMD-97-20 Air Traffic Control
                                                                 Chapter 1
                                                                 Introduction




                                                                                                          Associate
                                                                                                        Administrator
                                                                                                      for Research and
                                                                                                        Acquisitions
                                                                                                            ARA


Office of Independent                            Office of Air Traffic       Office of Aviation            Office of                                    Office of System      William J. Hughes
                        Office of Acquisitions                                                                              Office of Information
  Operational Test                                     Systems                   Research             Comm., Navigation,                                Architecture and
                                 ASU                                                                                             Technology                                     Tech Center
    and Evaluation                                  Development                    AAR                 and Surveillance                               Investment Analysis
         ATQ                                             AUA                                            Systems, AND                 AIT                      ASD                    ACT

                                                                               Chief Scientist         Integrated Product        Corporate                  Architecture           Resource
                           Acquisition Policy
                                                   Integrated Product        for Human Factors              Team for            Information                 and System            Management
                            and Procedures
                                                   Team for En Route              Division                Infrastructure      Management Div.               Engineering             Division
                               Division
                                                        AUA-200                   AAR-100                   AND-100               AIT-100                    ASD-100               ACT-100
                               ASU-100
                                                                                                       Integrated Product     Integrated Product          Evaluation and
                                                                                  Research                  Team for                                       Configuration        ATC Engineering
                           Quality Assurance       Integrated Product                                                              Team for
                                                                                   Division             Communications                                     Management           and Test Division
                                Division           Team for Terminal                                                         Information Systems
                                                                                  AAR-200                   AND-300                                         ASD-200                 ACT-200
                               ASU-200                  AUA-300                                                                    AIT-200
                                                   Integrated Product            Research and          Integrated Product     Integrated Product         NAS Programming        CNS Engineering
                               Contracts           Team for Weather               Acquisitions              Team for              Team for IT              and Financial
                                Division                                                                                                                                        and Test Division
                                                   and Flight Service        International Division       Surveillance             Services                Management              ACT-300
                               ASU-300                                             AAR-300                  AND-400                 AIT-300                  ASD-300
                                                   Systems, AUA-400
                                                                              Airport and Aircraft     Integrated Product     Integrated Product        Investment Analysis        Facilities
                                                   Integrated Product
                                                                              Safety, Research           Team for GPS/            Team for IT             and Operations          Management
                                                   Team for Air Traffic
                                                                              and Development              Navigation             Acquisitions               Research               Division
                                                      Management
                                                                              Division, AAR-400             AND-500                AIT-400                   ASD-400               ACT-400
                                                        AUA-500
                                                                              Aviation Security        Integrated Product                                                      Aviation Simulation
                                                   Integrated Product           Research and                Team for                                                               and Human
                                                    Team for Oceanic            Development             Aircraft/Avionics                                                       Factors Division
                                                        AUA-600               Division, AAR-500             AND-600                                                                 ACT-500
                                                                                                                                                                               Airport Management
                                                                                                                                                                                 and Emergency
                                                                                                                                                                               Operations Division
                                                                                                                                                                                     ACT-600

                                                                 a
                                                                     ATS is currently being reorganized




                                                                 Page 21                                                                            GAO/AIMD-97-20 Air Traffic Control
                                  Chapter 1
                                  Introduction




                                  During our review, we assessed six major modernization projects. These
                                  projects were the

                              •   Voice Switching and Control System (VSCS), which provides
                                  air-to-ground voice communication services and ground-to-ground voice
                                  communication services between controllers, other ATC personnel, and
                                  others at the same and different en route centers and other ATC facilities;
                              •   Standard Terminal Automation Replacement System (STARS), which
                                  is to replace critical air traffic control computers with new traffic
                                  computers, displays, and software in TRACON facilities and towers;
                              •   Display System Replacement (DSR), which is to replace air traffic
                                  controllers’ existing display-related systems in each of the en route
                                  centers;
                              •   Airport Surveillance Radar-9 (ASR-9), which monitors aircraft
                                  movement and position within a radius of 60 miles of an airport terminal;
                              •   Wide Area Augmentation System (WAAS) for the Global Positioning
                                  System (GPS), which is to provide augmentations to the Department of
                                  Defense’s GPS in order to allow improved navigation on domestic and
                                  oceanic air routes; and
                              •   Display Channel Complex Rehost (DCCR), which is an interim
                                  replacement to the mainframe computer system that processes radar and
                                  other data into displayable images on controllers’ screens.


Roles and Responsibilities        The FAA organization responsible for estimating costs on the projects we
for Project Cost Estimating       assessed varied depending on the project’s stage in its life cycle. According
                                  to FAA acquisition rules in place when the latest life cycle cost estimate for
                                  the projects we assessed were developed, FAA’s Investment Analysis and
                                  Operations Research organization (ASD-400), which is one of the groups
                                  within ARA that handles cross-cutting management functions, developed
                                  life cycle cost estimates early in the project life cycle—both when mission
                                  needs were evaluated and again when evaluating the costs, benefits, and
                                  feasibility of project alternatives. Once the decision was made to invest in
                                  a given alternative, the IPTs assumed responsibility for updating the cost
                                  estimates. However, IPTs could also choose to develop their own cost
                                  estimates prior to the investment decision point, rather than have ASD
                                  prepare them.

                                  Of the six projects we reviewed, four used ASD-400 at some point in the
                                  project’s life cycle. The other two did not. In addition, some project
                                  managers updated the acquisition phase portion of the life cycle cost




                                  Page 22                                        GAO/AIMD-97-20 Air Traffic Control
Chapter 1
Introduction




estimates periodically between the times that the full life cycle cost
estimates were revised.

FAA’s organizational responsibilities for cost estimating have recently
changed. In October 1995, the Congress instructed FAA to develop and
implement a new acquisition management system, which would not be
subject to various existing acquisition laws. On November 15, 1995, the
President signed this bill into law.7 FAA began implementing this new
acquisition management system in April 1996 with the issuance of broad
policies, guiding principles, and internal procedures.8 While not yet fully
implemented, these specify that two key decisions be made at the
corporate level by the Joint Resources Council (JRC), a newly formed body
comprised of the associate administrators for operations and acquisitions
as well as officials responsible for acquisitions, financial services, and
legal counsel. These decisions are (1) whether mission needs warrant
entry into investment analysis and (2) whether to invest in the project at
the conclusion of investment analysis. FAA identified the latter as the most
important decision in the life cycle acquisition management process.
Accordingly, FAA plans to establish a “center of excellence” for investment
analysis with experts in cost estimating, risk assessment, market analysis,
and affordability analysis.

Under this new scenario, investment analysis will be conducted as a joint
enterprise by the FAA organization sponsoring the system (i.e., ATS) and the
FAA organization responsible for acquiring it (i.e., ARA). The purpose is
threefold: to ensure that (1) users buy into the solution, (2) acquisition
specialists have a voice in the cost, schedule, and performance baselines
they will have to live with, and (3) the investment analysis staff
understands the concerns of the operations and acquisitions organizations.
Under this approach, the sponsoring organization, with technical support
from the investment analysis staff, develops and approves its requirements
in the form of a Requirements Document. The investment analysis staff
leads the effort to identify and analyze candidate solutions through market
surveys, alternatives analysis, and affordability assessments, with support
from the sponsoring organization and the ARA IPT responsible for acquiring
it. This effort culminates in the Investment Analysis Report, which is to
contain comprehensive quantitative data for each alternative, such as life
cycle cost, cost-benefit ratios, and risk. The IPTs use this information to
generate cost and schedule baselines for each alternative in the form of an
Acquisition Program Baseline. At the investment decision point, the JRC

7
 1996 DOT Appropriations Act (Public Law 104-50), Section 348.
8
 Federal Aviation Administration Acquisition Management System, Apr. 1, 1996.



Page 23                                                     GAO/AIMD-97-20 Air Traffic Control
                       Chapter 1
                       Introduction




                       decides on an alternative; baselines the project’s requirements, costs,
                       schedules, performance, and benefits; and commits the agency to full
                       funding of the program. Thereafter, any changes to these baselines must
                       be approved by the JRC. In fact, no funding may be committed or obligated
                       that would exceed the program cost baseline until the increase is
                       approved by the JRC and included in agency plans and budgets.

                       The success of FAA’s new investment analysis and decision-making
                       approach depends on many factors, not the least of which is the reliability
                       of ATC project cost information discussed in this report.


                       Reliable cost estimates and monitoring of actual costs are essential to
Sound Investment       informed investment decision-making throughout the development and
Decisions Require      maintenance of capital items, such as ATC systems. In the case of FAA, they
Reliable Cost          are cornerstones to its aforementioned investment analysis and
                       decision-making processes.
Estimating and
Accounting Practices   In 1994, we reported on how leading organizations improved mission
                       performance through information technology. Among other things, we
and Data               reported that successful organizations manage information system
                       projects as investments, and continually assess the quality of projects’
                       estimated costs and carefully monitor projects’ actual costs against these
                       estimates. Furthering this initiative, OMB’s 1995 guidance, Evaluating
                       Information Technology Investments, calls for selecting information
                       technology project investments on the basis of cost, benefit, risk, and
                       return; controlling projects by comparing ongoing actual results being
                       achieved with projected costs, benefits, and risks; and, finally, evaluating
                       projects after they have been implemented to determine actual cost,
                       benefits, risks, and returns, and modifying the selection and control
                       processes based on lessons learned. This guidance has since been
                       embodied in (1) the Clinger-Cohen Act of 1996, which requires the
                       selection of information technology investments on the basis of competing
                       projects’ estimated costs, benefits, and risks, and (2) the Chief Financial
                       Officers (CFO) Act of 1990, which requires federal agencies to maintain
                       integrated accounting and financial management systems that permit
                       systematic and reliable measurement of projects’ cost and performance.
                       Additionally, OMB Circular A-11, Part 3, requires agencies to request full
                       up-front budget authority for all ongoing and new fixed assets (including
                       information technology) in their fiscal year 1998 budget submission. This
                       circular also requires a fixed asset plan and justification for major




                       Page 24                                       GAO/AIMD-97-20 Air Traffic Control
                       Chapter 1
                       Introduction




                       acquisitions, including, among other items, an analysis of full life cycle
                       costs and an estimate of the risk and uncertainty in meeting project goals.


Experts Advocate Key   The Software Engineering Institute’s (SEI) Capability Maturity Model
Estimating Process     (CMM), the standard used by government and industry to determine the
Requisites             maturity of an organizations’ software development processes, also
                       highlights the need for good estimates and good estimating processes.9
                       Three of the CMM’s key process areas for level 2 (repeatable) process
                       maturity are project planning, project tracking, and subcontract
                       management. These process areas must have reliable estimates for size,
                       effort, schedule, and cost if they are to be performed successfully. The CMM
                       further requires that the procedures that implement these key process
                       areas be documented.

                       To improve the state of practice for software cost and schedule estimating,
                       SEI developed and published (1) criteria for establishing sound estimating
                       processes and (2) a guide for managers to use in validating an individual
                       project’s estimate.10 These documents, in effect, describe “best practices”
                       used in industry and government for estimating software costs and
                       schedules. However, SEI found that the “best practices” are equally
                       applicable to hardware and integrated systems projects, and therefore
                       allows for substituting the word “system” for “software” throughout its
                       guides and checklists. SEI also noted that while the criteria target the
                       acquisition/development phase of a project’s life cycle, the concepts are
                       also applicable to other phases of the life cycle.

                       According to SEI’s Checklists and Criteria for Evaluating the Cost and
                       Schedule Estimating Capabilities of Software Organizations, in order to
                       have sound estimating processes, an organization should have six
                       attributes, or requisites, institutionally embedded in its policies and
                       procedures. These include (1) a corporate memory, or historical
                       database(s), for cataloging cost estimates, revisions, reasons for revisions,
                       actuals, and other contextual information, (2) structured processes for
                       estimating software size and the amount and complexity of existing
                       software that can be reused, (3) cost models calibrated/tuned to reflect
                       demonstrated accomplishments on similar past projects, (4) audit trails
                       that record and explain values used as cost model inputs, (5) processes for

                       9
                        Capability Maturity Model for Software, Version 1.1 (CMU/SEI-93-TR-24).
                       10
                        Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software
                       Organizations (CMU/SEI-95-SR-005, Jan. 1995) and A Manager’s Checklist for Validating Software Cost
                       and Schedule Estimates (CMU/SEI-95-SR-004, Jan. 1995).



                       Page 25                                                      GAO/AIMD-97-20 Air Traffic Control
                  Chapter 1
                  Introduction




                  dealing with externally imposed cost or schedule constraints in order to
                  ensure the integrity of the estimating process, and (6) data collection and
                  feedback processes that foster capturing and correctly interpreting data
                  from work performed. SEI provides detailed checklists for assessing an
                  organization’s satisfaction of each requisite.

                  These same six requisites are interwoven through seven questions SEI
                  poses in A Manager’s Checklist for Validating Software Cost and Schedule
                  Estimates. The seven questions are (1) Are the objectives of the estimate
                  clear and correct? (2) Has the task been appropriately sized? (3) Are the
                  estimated cost and schedule consistent with demonstrated
                  accomplishments on past projects? (4) Have the factors that affect the
                  estimate been identified and explained? (5) Have steps been taken to
                  ensure the integrity of the estimating process? (6) Is the estimate based on
                  reliable evidence of the organization’s past performance? and (7) Has the
                  situation remained unchanged since the estimate was prepared? Once
                  again, SEI provides detailed checklists for addressing these seven
                  questions. These SEI publications are further discussed in chapter 2 and in
                  appendixes I, II, and III.


Cost Accounting   Requirements for agency cost accounting have been evolving for decades.
Requirements      In a 1985 report, the Comptroller General presented a framework for
                  strengthening agencies’ financial management structure.11 This report
                  called for the integration of accounting and budgeting systems to better
                  monitor progress against estimates and to better estimate future program
                  costs. More specifically, it states that actual costs must be maintained and
                  monitored in order to effectively manage programs and control costs. This
                  approach was embodied in the CFO Act of 1990, which requires agencies to
                  develop and maintain integrated accounting and financial management
                  systems which provide for (1) the development and reporting of cost
                  information and (2) the systematic measurement of performance.

                  Most recently, the Statement of Federal Financial Accounting Standards
                  no. 4 (SFFAS 4), Managerial Cost Accounting Concepts and Standards for
                  the Federal Government, effective for fiscal years beginning after
                  September 30, 1996, was issued. This standard requires reporting entities
                  to regularly accumulate and report the full cost (including all direct,
                  indirect, and supporting costs) of its activities. These cost accounting
                  standards are further discussed in chapter 3.

                  11
                   Managing the Cost of Government: Building An Effective Financial Management Structure
                  (GAO/AFMD-85-35 and GAO/AFMD-85-35-A, Feb. 1985).



                  Page 26                                                   GAO/AIMD-97-20 Air Traffic Control
                         Chapter 1
                         Introduction




                         The objectives of our review were to determine if (1) FAA’s project cost
Objectives, Scope,       estimates are based on good estimating policies and practices and (2) the
and Methodology          actual costs of ATC modernization projects are being properly
                         accumulated.

                         To determine if FAA’s estimates were based on good policies and practices,
                         we

                     •   researched current literature and interviewed project estimating experts to
                         identify the key components of good cost estimating practices;
                     •   obtained and analyzed FAA’s policies and practices for estimating costs to
                         determine what criteria (directives, orders, instructions, and implementing
                         procedures), if any, FAA has in place to guide managers in developing
                         projects’ cost estimates;
                     •   assessed FAA’s cost estimating policies, practices, tools, and techniques to
                         determine if they incorporate the key components of good cost and
                         schedule estimating practices advocated by SEI and other experts; and
                     •   selected FAA’s five largest (based on latest life cycle cost estimates)
                         ongoing ATC modernization projects and one project that was the subject
                         of another GAO review, and interviewed project managers and assessed
                         project documentation on these six projects to determine (1) how the
                         current life cycle cost baseline was estimated and (2) how this estimating
                         approach compared to the SEI project-level questions.12 To do this, we
                         compared each projects’ documentation to SEI’s detailed checklists for
                         each question; determined if the project satisfied, partially satisfied, or did
                         not satisfy each checklist item and assigned points accordingly (1, .5, or 0
                         points, respectively); and then summed the points and presented them as a
                         portion of the total points available (e.g., 4/10). Because the focus of this
                         effort is to assess FAA’s cost estimating processes and not to validate the
                         accuracy or completeness of the estimates, we did not evaluate the quality
                         of the estimates.

                         To determine whether the actual costs of ATC modernization projects are
                         being properly accumulated, we

                     •   obtained and reviewed (1) selected reports and testimonies issued by
                         GAO, the Department of Transportation’s Office of the Inspector General,
                         and the Defense Contract Auditing Agency, (2) related policies and
                         procedures issued by the Department of Transportation, (3) applicable
                         accounting standards and guidance, and (4) applicable OMB directives;

                         12
                            A Manager’s Checklist for Validating Software Cost and Schedule Estimates (CMU/SEI-95-SR-004,
                         Jan. 1995).



                         Page 27                                                     GAO/AIMD-97-20 Air Traffic Control
    Chapter 1
    Introduction




•   reviewed FAA’s policies and procedures governing ATC financial
    management and interviewed program managers and financial accounting
    staff to determine (1) their roles and responsibilities for recording and
    managing ATC cost information and (2) the financial processes used to
    accumulate and record ATC costs; and
•   reviewed available information for the five largest (based on life cycle cost
    estimates) ongoing ATC projects and determined if costs are properly
    accumulated by (1) obtaining available financial information on the
    projects and identifying the cost elements included and excluded and
    (2) assessing reconciliation procedures among varying sources of
    information.

    We requested comments on a draft of this product from the Secretary of
    Transportation. On December 10, 1996, we obtained oral comments from
    Transportation and FAA officials, including representatives from the Office
    of the Secretary of Transportation, the Executive Assistant to the FAA
    Chief Financial Officer, the FAA Manager of the Cost Accounting System
    Division, and the FAA Program Director for Investment Analysis and
    Operations Research. Their comments are presented and addressed in
    chapters 2 and 3 of this report.

    We performed our work at the Federal Aviation Administration in
    Washington, D.C., and the Software Engineering Institute in Pittsburgh,
    Pennsylvania, between February and December 1996. Our work was
    performed in accordance with generally accepted government auditing
    standards.




    Page 28                                       GAO/AIMD-97-20 Air Traffic Control
Chapter 2

ATC Modernization Lacks the Institutional
Capacity to Produce Reliable Cost Estimates

                           Reliable estimates of projects’ expected costs are essential to decide
                           among alternative investments. According to SEI, consistently producing
                           them requires defined institutional processes for deriving estimates,
                           archiving them, and measuring actual performance against these
                           estimates. FAA’s cost estimating processes used on its ATC modernization
                           projects do not meet SEI criteria. These weaknesses are exacerbated by
                           FAA’s practice of presenting cost estimates as precise, point estimates. By
                           doing so, FAA obscures the estimates’ inherent uncertainty and may
                           mislead decisionmakers.


                           FAA’s institutional processes for estimating ATC projects’ costs do not fully
Policies and Practices     satisfy any of the six SEI requisites. According to SEI, all six must be
for Estimating Costs       satisfied to produce credible estimates. The six requisites are described
Do Not Satisfy “Best       below, along with our analysis showing that FAA’s institutional policies and
                           practices fail to meet them. (See appendixes I and II for more detail.) We
Practices” Requisites      shared this analysis with FAA’s cost estimating authority, who agreed that
                           FAA’s policies and practices do not meet SEI’s process requirements.



Requisite 1: A Corporate   According to SEI, estimating organizations should have a process for
Memory                     organizing and retaining project cost and schedule information in a
                           historical database, and for using it as an integral part of the estimating
                           process. This database should contain detailed information on projects’
                           original estimates, revised estimates, reasons for revisions, and actual
                           performance against the estimates. It should also contain descriptive
                           contextual information that enables people to understand and correctly
                           interpret the data in the database.

                           FAA has no institutional corporate memory, or historical database, on ATC
                           projects’ cost and schedule estimates and performance. While FAA has a
                           number of stand-alone databases within different groups, none provide a
                           complete picture of estimates, assumptions that make up the estimates,
                           revisions, and actual performance on projects. For example, the Cost
                           Benefit Analysis System (CBAS) is a database that contains some
                           information on projects’ cost estimates and planned budget levels.
                           However, it contains no information on how and why estimates are
                           revised, why budget streams differ from estimates, or what projects
                           actually cost.

                           Further, what limited information is available on actual cost performance
                           is not an integral part of the project estimating process, and the



                           Page 29                                        GAO/AIMD-97-20 Air Traffic Control
                           Chapter 2
                           ATC Modernization Lacks the Institutional
                           Capacity to Produce Reliable Cost Estimates




                           information on cost estimates that is retained by the central estimating
                           organization is not readily available to the project personnel that are
                           responsible for updating estimates after the initial investment decision has
                           been made.


Requisite 2: Structured    According to SEI, estimating organizations should follow well-defined,
Processes for Estimating   structured processes for estimating product size and the amount and
Product Size and the       complexity of existing software that can be reused. It should be clear what
                           is included and what is excluded from size estimates, and new estimates
Amount and Complexity of   should be checked by comparing them to measured sizes of existing
Existing Software That     software products.
Can Be Reused
                           FAA has no institutional process for estimating product size and the
                           amount of existing software that can be reused. Each project manager
                           decides to estimate software size and reuse as he or she chooses. Among
                           ATC projects, these processes range from simple lines-of-code estimates
                           using an individual’s personal knowledge of similar systems’ sizes to
                           sophisticated analysis based on project-unique variables. For example, the
                           original software size estimates for the Standard Terminal Automation
                           Replacement System (STARS) was a rough approximation based on the
                           number of lines of code in a predecessor system. A more recent, though
                           not yet official, estimating effort established size estimates for each
                           desired STARS software function and then compared the estimates to
                           known sizes of similar functions on two other completed projects. This
                           effort also used a checklist to ensure that no desired functions were
                           overlooked and accounted for differences between STARS and its
                           predecessor system. The latter, more careful size estimate is over three
                           times the original estimate for new and modified software.


Requisite 3: Mechanisms    According to SEI, estimating organizations should have documented
for Extrapolating From     processes for extrapolating from past experiences. These processes
Demonstrated               should include the use of cost models that are calibrated and validated on
                           the basis of actual experience. Further, differences between cost models’
Accomplishments on Past    outputs should be analyzed and explained.
Projects
                           FAA’s estimating guidance recommends 10 different cost models as
                           acceptable estimating tools; however, there is no requirement for project
                           estimators to use these models or to collect and use past experience to
                           calibrate these models. As a result, projects’ use and calibration of the
                           models is inconsistent. For example, estimators on three of the six



                           Page 30                                       GAO/AIMD-97-20 Air Traffic Control
                            Chapter 2
                            ATC Modernization Lacks the Institutional
                            Capacity to Produce Reliable Cost Estimates




                            projects we assessed did not use cost models in their estimates whereas
                            Display System Replacement (DSR) officials not only used four cost
                            models, but also calibrated them to past experiences on a predecessor
                            system.


Requisite 4: Audit Trails   According to SEI, estimating organizations should prepare adequate audit
                            trails of inputs to the estimates, including parameters used in cost models
                            and their rationales.

                            FAA estimating guidance requires that cost estimates be documented and
                            reproducible. However, the degree of documentation and the extent of any
                            accompanying explanation is left to the discretion of each project’s
                            estimators. As a result, the detail and quality of the audit trails on ATC
                            projects is inconsistent. For example, only handwritten notes document
                            how the Display Channel Complex Rehost (DCCR) estimate was derived.
                            On the other hand, the DSR estimate and the ongoing STARS estimating
                            effort are supported by volumes of documentation delineating the
                            assumptions, processes, and model inputs used.


Requisite 5: Integrity in   According to SEI, organizations should ensure that the effects of dictated,
Dealing With Dictated       or externally imposed, costs or schedules are determined and explicitly
Costs or Schedules          presented to management. Estimators should document and managers
                            should approve any changes made to model parameters to accommodate
                            dictated costs or schedules, the rationale for making the changes, and the
                            effect of the changes on other factors—cost, schedule, or risk.

                            FAA has no institutional process for ensuring integrity in dealing with
                            dictated costs or schedules. As a result, each project manager determines
                            his or her own response to externally imposed constraints. Only one of the
                            projects we assessed acknowledged working under an externally imposed
                            schedule. STARS project officials are preparing a cost estimate that shows
                            the cost of meeting the compressed schedule. However, without an
                            institutional policy requiring such action, there is no assurance that all
                            dictated constraints on all ATC projects will be handled so effectively.




                            Page 31                                       GAO/AIMD-97-20 Air Traffic Control
                          Chapter 2
                          ATC Modernization Lacks the Institutional
                          Capacity to Produce Reliable Cost Estimates




Requisite 6: Data         According to SEI, organizations should have a defined process for
Collection and Feedback   gathering information on ongoing and completed projects (including
Processes That Foster     original estimates, revised estimates, and post-mortem assessments) and
                          entering this information into the historical database.
Capturing and Correctly
Interpreting Data From    FAA has no institutional process for gathering information on completed
Work Performed            projects and entering it in a historical database. Post-mortem reviews are
                          performed rarely, and only on an ad hoc basis. Instead, each project
                          manager determines the type and amount of information retained.


                          Because FAA does not have well-defined, institutional processes for
Individual ATC            estimating information technology projects’ costs, the approaches used
Projects’ Cost            and the reliability of the estimates are inconsistent.
Estimating
                          To assist management in assessing the credibility of a given project cost
Approaches Are            estimate, SEI developed seven questions which must be answered. The
Inconsistent and          seven questions are (1) Are the objectives of the estimate clear and
                          correct? (2) Has the task been appropriately sized? (3) Are the estimated
Sometimes Unreliable      cost and schedule consistent with demonstrated accomplishments on
                          other projects? (4) Have the factors that affect the estimate been
                          identified and explained? (5) Have steps been taken to ensure the integrity
                          of the estimating process? (6) Is the estimate based on reliable evidence of
                          the organization’s past performance? and (7) Has the situation remained
                          unchanged since the estimate was prepared? SEI developed a detailed
                          checklist to assist in addressing each question. (See appendix III for more
                          information on SEI’s questions and checklists.)

                          We applied this checklist to the most recently baselined life cycle cost
                          estimate on six ATC projects (the five ongoing projects with the largest life
                          cycle cost estimates plus one ongoing project that was the subject of
                          another GAO review). These projects are the Voice Switching and Control
                          System (VSCS), the Standard Terminal Automation Replacement System
                          (STARS), the Display System Replacement (DSR), the Airport Surveillance
                          Radar-9 (ASR-9), the Wide Area Augmentation System for the Global
                          Positioning System (WAAS), and the Display Channel Complex Rehost
                          (DCCR).

                          Of the six systems, two (ASR-9 and DCCR) could not be assessed because
                          they lacked documentation describing how the latest official life cycle cost
                          estimates were derived. For example, the estimated cost to develop ASR-9 is




                          Page 32                                        GAO/AIMD-97-20 Air Traffic Control
Chapter 2
ATC Modernization Lacks the Institutional
Capacity to Produce Reliable Cost Estimates




$857 million.1 However, there is no documentation describing how ASR-9
software size estimates were determined or what assumptions were used
to estimate costs from these size estimates. As a result, there is no
analytical way to determine the credibility of the original estimate or
estimating approach. Moreover, without documentation, FAA cannot
systematically use its ASR-9 experience in other system estimating
endeavors.

Of the remaining four systems, we found that the approaches used on
three (STARS, WAAS, and VSCS) satisfied some but not all of SEI’s reliability
questions. To their credit, officials on two of these projects are currently
revising parts of the life cycle cost estimate using more rigorous
approaches. For example, the estimating approach used to establish the
current STARS life cycle cost estimate was performed very early in the
project’s life cycle, during the mission needs determination phase. Project
officials estimated the project’s software size on the basis of a predecessor
system’s size. However, the predecessor was so different from STARS that
its size was not a good indicator of the size of STARS’ software. Unlike the
predecessor, STARS is being developed to be easily maintainable, portable,
and expandable. Additionally, many significant functions, such as support
software for configuration management and simulation, were not
accounted for in the original software estimate. Project officials agree that
the early estimating process was not strong, and in fact, that they
underestimated the size and complexity of the software to be developed.
In an ongoing estimating effort, STARS estimators have been more thorough
and rigorous in estimating the software size and project costs. They
defined existing and missing STARS functions, comparing the functions to
two completed projects’ functions to help determine software size for each
function, and refining this estimate based on advice from experts
throughout FAA and performance demonstrations by competing vendors.
However, even this improved estimating approach falls short of SEI
standards. Specifically, STARS estimators did not calibrate the cost
estimating models it used to past FAA performance, choosing instead to use
the models’ default values. Calibrating models to an organization’s
historical experience (i.e., determining the productivity value of a
“man-month” or the time and cost to produce a “line of code” based on the
organization’s past performance on similar projects, or adding new cost
drivers that are specific to an organization’s business, such as security or
privacy) will provide more accurate estimates.



1
 A project official stated that they have no estimate of what it will cost to operate and maintain ASR-9
over its life cycle.



Page 33                                                         GAO/AIMD-97-20 Air Traffic Control
Chapter 2
ATC Modernization Lacks the Institutional
Capacity to Produce Reliable Cost Estimates




In addition, project estimators derived official WAAS life cycle cost
estimates based on experiences with the National Satellite Test Bed, a
precursor to the WAAS development. However, little documentation exists
on how the original WAAS size estimates were derived and what
assumptions and parameters were used to estimate costs. Project officials
are currently updating estimates using a published sizing methodology and
the Software Life Cycle Intermediate Model (SLIM) cost model, and stated
that they believe this is a much more rigorous approach than prior
estimating efforts. However, this more structured approach also falls short
of SEI requirements. For example, the project software estimator stated
that one cost model parameter (input variable) is a ranking of the
sophistication of the developer’s software environment. The estimator
scored the WAAS contractor as very high on this parameter and showed us
where this information was captured in the SLIM database. However, the
estimator went on to explain that he chose a high score based on the
contractor’s rating as a level 3 organization on SEI’s CMM and the fact that
they have and use Computer-Aided Software Engineering (CASE) tools.
This explanatory information was not recorded in the SLIM database or in
any other documentation, and thus is not available to anyone trying to
understand or validate the estimate or learn from this estimating
experience.

The current official estimate for the last of the six systems, DSR, was
derived using an approach that partially or fully satisfied six of the seven
SEI reliability questions. Examples of DSR’s adherence to SEI guidance
include (1) DSR’s software size estimate was developed by identifying
needed software functions, and then determining the amount of new code
needed and reusable code available for each of these software functions,
(2) DSR estimators calibrated cost estimating models to past experience on
DSR’s predecessor system, the Initial Sector Suite System (ISSS), and
(3) estimators used templates to ensure that key cost factors would not be
overlooked. However, DSR estimators did not record the rationales for the
parameters they used in their cost models or explain differences among
cost model results. (See appendix IV for further information on the results
of our assessment.)




Page 34                                       GAO/AIMD-97-20 Air Traffic Control
                                                         Chapter 2
                                                         ATC Modernization Lacks the Institutional
                                                         Capacity to Produce Reliable Cost Estimates




                                                         Software and systems development experts agree that early project
FAA’s Portrayal of                                       estimates are by definition imprecise, and that this inherent imprecision
ATC Projects’ Cost                                       decreases during the project’s life cycle as more information becomes
Estimates Implies an                                     known about the system. Some have described this phenomenon as a
                                                         “cone of uncertainty” that is widest early in the life cycle and narrows over
Unjustified Level of                                     time as more becomes defined and known about the project. (See figure
Precision                                                2.1.) These experts emphasize that each cost estimate should include an
                                                         indication of its degree of uncertainty, possibly as an estimated range or
                                                         qualified by some factor of confidence. For example, a cost estimate of
                                                         $1 million could be presented as a range from $750,000 to $1.25 million or
                                                         as $1 million with a confidence level of 90 percent, indicating that there is
                                                         a 10-percent chance that costs will exceed this estimate.


Figure 2.1: Cost Estimating Accuracy Over Time
   Range of uncertainty around a cost estimate (x)




                                                     x




                                                                    Systems Life Cycle




                                                         Page 35                                       GAO/AIMD-97-20 Air Traffic Control
                         Chapter 2
                         ATC Modernization Lacks the Institutional
                         Capacity to Produce Reliable Cost Estimates




                         FAA does not reveal its estimates’ degree of uncertainty to managers
                         involved in investment decisions. Instead, FAA presents its projects’ cost
                         estimates as unqualified point estimates, thereby suggesting an element of
                         precision that does not exist. A budget official stated that FAA presents
                         project cost estimates as such because “this is the way OMB and Congress
                         want to see it.” Further, he stated that in today’s environment of lean
                         budgets, the low-end of the estimate range is all that FAA can afford and all
                         that is salable, and therefore, this is what they present.

                         By presenting a point estimate instead of a range of estimates or a
                         realistically qualified estimate, FAA is not fully disclosing all relevant
                         information about the projects’ potential costs and inherent risk. The
                         result is uninformed, and thus potentially unwise, investment decisions.


                         During the course of our review, FAA organizations initiated several efforts
FAA Organizations        to improve their processes for estimating and archiving cost information.
Have Initiated Efforts   However, these efforts are still relatively new and have not yet been
to Improve Cost          institutionalized. We did not evaluate any of these efforts.

Estimating               FAA’s Office of Investment Analysis and Operations Research (ASD-400),
                         which has historically been responsible for early life cycle cost estimates,
                         has established an informal cost estimating work group to explore the
                         possibility of integrating cost estimating functions across FAA. To date, this
                         group has documented current cost estimating processes and identified
                         the multitude of organizations that participate in different components of
                         the estimating process. The group has also drafted a workbook to guide
                         estimators in developing estimates. However, neither the group nor the
                         workbook has not yet been officially sanctioned by FAA.

                         Additionally, FAA’s Office of Information Technology recently procured an
                         agencywide site license for another cost estimating tool, the Software Life
                         Cycle Intermediate Model (SLIM). This tool is one of several that estimators
                         can use to develop cost estimates.

                         FAA’s Office of Airway Facilities Requirements initiated a “grass roots”
                         effort to standardize on a single cost estimating and tracking tool, called
                         the Automated Cost Estimating Integrated Tools (ACEIT). Officials stated
                         that institutionalizing ACEIT would provide FAA a consistent framework for
                         estimating costs, including a documented audit trail. The group supporting
                         ACEIT is currently briefing other FAA organizations in an attempt to promote
                         adoption of this tool throughout FAA.



                         Page 36                                        GAO/AIMD-97-20 Air Traffic Control
                      Chapter 2
                      ATC Modernization Lacks the Institutional
                      Capacity to Produce Reliable Cost Estimates




                      The same organization is also testing the Cost and Performance
                      Management System (CPMS) to track operations and maintenance (O&M)
                      costs by project for the first time. An airways facilities manager stated that
                      CPMS will eventually be integrated with ACEIT to allow estimators to use
                      actual project cost information in estimating new projects’ O&M costs.


                      Multimillion dollar, and even billion-dollar, investment decisions on air
Conclusions           traffic control modernization projects are being made without reliable
                      information on the projects’ estimated and actual costs. FAA does not have
                      well-defined, structured estimating processes that are rigorously followed,
                      and does not disclose the estimates’ inherent uncertainty. Without better
                      estimates of cost, FAA’s new investment analysis and decision-making
                      processes are unlikely to be effective.


                      We recommend that the Secretary of Transportation direct the FAA
Recommendations       Administrator to institutionalize defined processes for estimating ATC
                      projects’ costs. At a minimum, these processes should include the
                      following SEI requisites, each of which are described in more detail in this
                      report:

                  •   a corporate memory (or historical database), which includes cost and
                      schedule estimates, revisions, reasons for revisions, actuals, and relevant
                      contextual information;
                  •   structured approaches for estimating software size and the amount and
                      complexity of existing software that can be reused;
                  •   cost models calibrated/tuned to reflect demonstrated accomplishments on
                      past projects;
                  •   audit trails that record and explain all values used as cost model inputs;
                  •   processes for dealing with externally imposed cost or schedule constraints
                      in order to ensure the integrity of the estimating process; and
                  •   data collection and feedback processes that foster capturing and correctly
                      interpreting data from work performed.

                      We also recommend that the Secretary direct the Administrator to
                      immediately begin disclosing the inherent uncertainty and range of
                      imprecision in all ATC projects’ official cost estimates presented to
                      executive oversight agencies or the Congress.

                      Additionally, we recommend that the Secretary direct the Administrator to
                      report to the Secretary and FAA’s authorizing and appropriation



                      Page 37                                        GAO/AIMD-97-20 Air Traffic Control
                  Chapter 2
                  ATC Modernization Lacks the Institutional
                  Capacity to Produce Reliable Cost Estimates




                  committees on progress being made on these recommendations as part of
                  the agency’s fiscal year 1999 budget submission.


                  DOT and FAA officials provided oral comments on a draft of this report.
Agency Comments   These officials concurred with the report’s findings, conclusions, and
                  recommendations on cost estimating. They also stated that this report will
                  be useful as FAA strives to improve its cost estimating capabilities.




                  Page 38                                       GAO/AIMD-97-20 Air Traffic Control
Chapter 3

FAA Is Not Adequately Accounting for ATC
Project Costs

                           Agencies are required to maintain adequate systems of accounting and
                           internal controls to provide managers and other decisionmakers with
                           reliable financial information to effectively measure performance and
                           make sound investment decisions. In the case of the ATC modernization
                           program, FAA is not satisfying this requirement. Specifically, ATC project
                           information does not include all relevant project costs, including internal
                           personnel compensation, benefits, and travel (PCB&T) costs, because FAA
                           lacks a cost accounting system to accumulate and allocate these costs to
                           specific projects. FAA’s internal accounting policies include a requirement
                           for a cost accounting system; however, this policy has not been
                           implemented. As a result, project managers are unable to measure actual
                           costs and their ability to make informed decisions is impaired. Moreover,
                           complete project information is not available to feed back into, and
                           thereby improve, future project cost estimates.


                           The Federal Managers’ Financial Integrity Act of 1982 (FMFIA) requires that
Agencies Are               agency systems of internal accounting and administrative control comply
Required to Maintain       with internal control standards prescribed by the Comptroller General and
Adequate Accounting        provide reasonable assurance that among other things, obligations and
                           costs comply with applicable law and revenues and expenditures
Systems                    applicable to agency operations are recorded and accounted for properly.
                           FMFIA also requires that agency heads issue an annual report, transmitted
                           to the President and the Congress, detailing whether their internal control
                           systems fully comply with the act’s requirements, including the
                           identification of material systems weaknesses and plans for corrective
                           actions.

                           The Chief Financial Officers (CFO) Act of 1990 requires agencies to develop
                           and maintain integrated agency accounting and financial management
                           systems that comply with applicable accounting principles, standards, and
                           requirements, including

                       •   the preparation of complete, reliable, consistent, uniform, and timely
                           information that is responsive to agency management’s financial
                           information needs;
                       •   the development and reporting of cost information;
                       •   the integration of accounting and budgeting information; and
                       •   the systematic measurement of performance.

                           Recently, the Statement of Federal Financial Accounting Standards no. 4
                           (SFFAS 4), Managerial Cost Accounting Concepts and Standards for the



                           Page 39                                       GAO/AIMD-97-20 Air Traffic Control
                            Chapter 3
                            FAA Is Not Adequately Accounting for ATC
                            Project Costs




                            Federal Government, was issued, effective for fiscal periods beginning
                            after September 30, 1996. These standards require a reporting entity to
                            accumulate and report the full cost of its activities regularly for
                            management information purposes. The full cost of a project is described
                            as the sum of (1) the costs of resources consumed by the project that
                            directly or indirectly contribute to the output and (2) the costs of
                            identifiable supporting services provided by other organizations within the
                            reporting entity and by other reporting entities. These standards also
                            require that the full costs of resources be assigned to outputs through
                            costing methodologies or cost finding techniques that are most
                            appropriate to the organization’s operating environment and that they be
                            followed consistently. While SFFAS 4 has only been effective for a short
                            time, and therefore was not applicable during the period of our review, it
                            provides cost accounting criteria which are now required to be
                            implemented by all agencies.

                            Additionally, the 104th Congress passed the Federal Financial
                            Management Improvement Act of 1996 which, among other provisions,
                            requires agencies to comply with federal accounting standards.


                            FAA financial systems supporting the ATC modernization program do not
ATC Project Costs Are       accumulate all project costs, and thus, managers do not receive all
Not Being Properly          relevant financial information needed to effectively manage their projects.
Accumulated                 Of the five projects whose financial information we reviewed, none of the
                            project managers could provide the total of all costs incurred from the
                            project’s inception. Instead, they provided:

                        •   contract numbers for their respective projects so that cost data for each
                            contract could be extracted from FAA’s Departmental Accounting and
                            Financial Information System (DAFIS) and aggregated to provide total
                            contract costs. However, these contract costs could be understated
                            because project officials could not verify that they provided us all
                            applicable contract numbers.
                        •   incomplete project costs. These costs were not complete because they did
                            not include (1) Personnel Compensation, Benefits, and Travel (PCB&T)
                            costs associated with the Facilities and Equipment (F&E) appropriations
                            account and (2) all costs paid out of the Operations and Maintenance
                            (O&M) appropriations account. PCB&T costs for the F&E appropriation
                            include internal FAA costs that are related to project design, contracting,
                            and contractor oversight. O&M costs are costs associated with the
                            administration, operation, repair, and maintenance of operating FAA



                            Page 40                                       GAO/AIMD-97-20 Air Traffic Control
                    Chapter 3
                    FAA Is Not Adequately Accounting for ATC
                    Project Costs




                    facilities and are generally the single largest component of an information
                    systems life cycle cost. In 1995, total PCB&T costs were approximately
                    $2 billion for all ATC projects. PCB&T and O&M costs are accounted for
                    separately and are not allocated to individual ATC projects. Because these
                    costs are not allocated to specific projects, full life cycle costs of projects
                    cannot be determined and may be significantly understated.

                    An additional limitation is that FAA does not carry over and report the costs
                    associated with terminated or redirected projects as part of the successor
                    projects’ costs, even though successor projects reuse parts of the
                    predecessors’ components. As a result, the full costs of “restructured” ATC
                    projects are understated. Accounting for the full costs of projects requires
                    that the costs related to usable portions of terminated or redirected
                    projects be included in the costs of the ongoing projects. In addition, full
                    project cost accounting information would require that costs of unused
                    parts of terminated or redirected projects be separately identifiable within
                    the “corporate memory.”

                    For example, one of the projects we reviewed, the Display System
                    Replacement (DSR), is a follow-on to an earlier terminated project known
                    as the Initial Sector Suite System (ISSS). According to FAA officials, DSR
                    software and hardware salvaged from ISSS accounts for about 19 percent of
                    DSR’s cost. However, these costs are not included in DSR’s accumulated and
                    reported costs because, according to an FAA official, these costs are
                    considered “sunk costs.” The term “sunk costs” is generally used to
                    describe costs that have been incurred in the past and have no relevance
                    to future decision-making. However, we believe these costs should be
                    considered a part of a project’s full cost since they would be instructive in
                    reliably estimating costs of similar systems. In addition, information about
                    the amount of costs associated with the unused portions of terminated
                    projects should be retained in the “corporate memory” to provide a full
                    picture of the real cost of development projects.


                    A managerial cost accounting system supports the collection,
FAA Lacks a Cost    measurement, accumulation, analysis, interpretation, and communication
Accounting System   of cost information to allow users to determine the cost of specific
for Its ATC         programs and activities and the composition of, and changes in, these
                    costs. As mentioned above, the CFO Act requires agencies to develop and
Modernization       maintain a cost accounting capability that captures both budgetary and
Program             financial accounting data and generates performance measures. FAA’s
                    internal policies require a cost accounting system and state that the cost



                    Page 41                                         GAO/AIMD-97-20 Air Traffic Control
    Chapter 3
    FAA Is Not Adequately Accounting for ATC
    Project Costs




    accounting system should be integrated with the general accounting
    system. However, these policies have not been implemented; thus FAA
    project managers do not have the capability to fully account for costs
    being incurred for the ATC modernization program.

    Instead of a cost accounting system, several financial management
    systems account for specific financial accounting and budgetary data, but
    these systems are not integrated and they do not provide the full cost
    information necessary for investment decisions. These systems include the
    following.

•   Departmental Accounting and Financial Information System (DAFIS): This
    system is the Department of Transportation’s core accounting system.
    However, this system is not a cost accounting system because not all cost
    information is captured by project. In addition, the Department’s Office of
    the Inspector General (OIG) reported that DAFIS data is unreliable and
    inaccurate.1 For example, the OIG reported that major balance
    discrepancies existed between DAFIS accounts and their supporting details.
    Further, in April 1996, a cost accounting systems consultant reported that
    DAFIS does not provide all levels of management with timely, accurate,
    relevant, and meaningful information.
•   Financial Management System (FMS): This FAA system is used by ATC
    project managers to establish quarterly obligation plans under the
    Facilities and Equipment (F&E) appropriation and to track actual
    obligations against these plans. However, FMS is not a cost accounting
    system because it (1) contains only obligation data and (2) does not
    contain all relevant cost data, such as those for PCB&T.
•   Cost of Performance System (COPS): This system is used by FAA
    organizations responsible for operating and maintaining ATC systems to
    allocate aggregated O&M obligation data to individual cost centers (that is,
    field maintenance organizations). COPS is not a cost accounting system
    because it does not contain information on actual costs. In addition, COPS’
    O&M cost data is not allocated to and accumulated for individual projects.
•   Research, Engineering and Development, Monitoring, Analysis, and
    Control System (REDMACS): This system is used by FAA organizations
    responsible for ATC research and development projects to establish
    quarterly obligation plans for the Research, Engineering, and Development
    (RE&D) appropriation and to track actual obligations against these plans.
    REDMACS is not a cost accounting system because it does not contain
    information on actual costs.

    1
     Supplementary Report on Internal Control Systems and Compliance Related to the Airport and
    Airway Trust Fund Portion of the Federal Aviation Administration FY 1993 Financial Statement
    (Department of Transportation - Office of Inspector General, AD-FA-5-005, Mar. 29, 1995).



    Page 42                                                     GAO/AIMD-97-20 Air Traffic Control
                  Chapter 3
                  FAA Is Not Adequately Accounting for ATC
                  Project Costs




                  Project managers have also developed their own unique systems to
                  account for F&E and RE&D obligations. These “cuff” systems range from
                  spreadsheets to more sophisticated financial management systems, but do
                  not include O&M costs and generally do not capture actual costs. None of
                  the systems listed above, either individually or combined, constitute a cost
                  accounting system because they do not provide for accumulation and
                  monitoring of total costs.

                  In recognition of the need for accurate and reliable cost information, FAA’s
                  Associate Administrator for Administration established a new Cost
                  Accounting Systems Division in August 1996 and engaged a consultant to
                  assist the agency in defining its cost accounting requirements and in
                  designing and implementing a system to meet these requirements.
                  According to the Manager of the Cost Accounting System Division, this
                  system will use data from several financial management systems, including
                  DAFIS, and is planned to be in place by October 1, 1997.




                  Reliable project cost information that is both complete and accurate is
Conclusions       needed to estimate future project costs, make sound investment decisions,
                  and effectively manage projects. FAA does not have a cost accounting
                  system capable of reliably accumulating full project cost information, and
                  therefore cannot reliably estimate future project costs, ensure that
                  investment decisions are sound, or manage projects effectively. Without
                  better cost information, FAA’s new investment analysis and
                  decision-making processes are unlikely to be effective.


                  In light of FAA’s weaknesses in accounting for and reporting ATC project
Recommendations   costs, we recommend that the Secretary of Transportation direct the FAA
                  Administrator to acquire or develop and implement a managerial cost
                  accounting capability that will satisfy the requirements of SFFAS 4,
                  Managerial Cost Accounting Concepts and Standards for the Federal
                  Government. This system capability should provide the cost accounting
                  and financial management information needed by FAA management and
                  those who make investment decisions. Such information should include
                  full life cycle costs, which include the costs of resources consumed by a
                  project that directly or indirectly contribute to the output and the costs of
                  identifiable supporting services provided by other organizations within the
                  reporting entity.




                  Page 43                                       GAO/AIMD-97-20 Air Traffic Control
                     Chapter 3
                     FAA Is Not Adequately Accounting for ATC
                     Project Costs




                     We further recommend that the Secretary report FAA’s lack of a cost
                     accounting capability for its ATC modernization as a material internal
                     control weakness in the Department’s fiscal year 1996 FMFIA report and in
                     subsequent annual FMFIA reports until the problem is corrected.

                     Also, we recommend that the Secretary direct the Administrator to report
                     to the Secretary and FAA’s authorizing and appropriation committees on
                     progress being made on these recommendations as part of the agency’s
                     fiscal year 1999 budget submission.


                     In providing oral comments on a draft of this report, DOT and FAA officials
Agency Comments      stated that since they are in the process of acquiring a cost accounting
and Our Evaluation   system and plan to have an “initial operating capability” by October 1,
                     1997, they do not agree with our recommendations and consider them
                     unnecessary. While we acknowledge and support FAA’s cost accounting
                     organizational and system initiatives, it is important to note that its cost
                     accounting system acquisition is still very early in its acquisition life cycle
                     and much remains to be accomplished before FAA can have the cost
                     accounting capability we recommend. In fact, FAA has yet to develop
                     detailed functional requirements for this system, thereby precluding our
                     analysis at this time of whether its plans will satisfy our recommendation.
                     Additionally, until FAA implements our recommendation and improves the
                     accuracy of underlying data in feeder systems like DAFIS, it will continue to
                     lack adequate cost information needed to effectively manage its ATC
                     system acquisitions. Disclosure of such a management control weakness is
                     one of the objectives of the FMFIA, and therefore we continue to believe
                     that FAA should report its lack of a cost accounting system as a material
                     weakness in its FMFIA reports until the problem is corrected.




                     Page 44                                         GAO/AIMD-97-20 Air Traffic Control
Page 45   GAO/AIMD-97-20 Air Traffic Control
Appendix I

Summary Comparison of SEI’s Requisites
for Reliable Estimating Processes and FAA’s
Institutional Policies and Practices
                                                Table I.1 identifies SEI’s requisites for reliable estimating processes and
                                                contrasts these with the institutional policies and practices currently in
                                                place at FAA. A more detailed comparison for each requisite is provided in
                                                appendix II.


Table I.1: Comparison of SEI’s Requisites for Reliable Estimating Processes and FAA Policies and Practices for ATC
Projects
                                                                                  FAA
SEI requisites                                  Institutional policies                      Actual practice
A corporate memory (historical database)        None.                                       Some information on project estimates and
containing original and revised cost and                                                    budgets are maintained in different,
schedule estimates and actual costs and                                                     unrelated databases. None of these
schedules on completed projects to be used                                                  databases, however, provide sufficient
as an integral part of the estimating process                                               data to support these estimates and
for new/updated projects.                                                                   assumptions and to document revisions to
(See appendix II, table II.1 for more details.)                                             estimates and actual performance against
                                                                                            estimates.

                                                                                            Projects vary in the type and extent of
                                                                                            historical information used.

                                                                                            The limited information available on actual
                                                                                            project cost is not used as part of the
                                                                                            estimating process.
Structured processes for estimating product     None.                                       Project teams follow whatever approach
size and reuse. (See appendix II, table II.2                                                they choose for estimating product size
for more details.)                                                                          and reuse. As a result, size and reuse
                                                                                            practices vary.
Mechanisms for extrapolating from               FAA policy approves parametric modeling     Project teams use a variety of models or
demonstrated accomplishments on past            as one of five acceptable cost estimating   none at all. The extent to which they use
projects—that is, tools for forecasting costs   methodologies. FAA guidance                 historical information varies depending on
and schedules that are calibrated to past       recommends, but does not require, using     their awareness of and access to such
experience. (See appendix II, table II.3 for    various estimating tools/models. FAA does   information.
more details.)                                  not have a policy regarding the
                                                calibration/tuning of the models using
                                                demonstrated accomplishments.
Audit trails detailing and explaining values    FAA guidance states that cost estimates be Audit trails vary among projects, ranging
used for the cost model parameters. (See        documented and reproducible, but does      from handwritten notes to detailed volumes.
appendix II, table II.4 for more details.)      not specify what that entails.
Integrity in dealing with dictated costs and    None.                                       Project teams determine how they deal with
schedules. (See appendix II, table II.5 for                                                 imposed cost and schedule objectives. The
more details.)                                                                              effect of imposed cost and schedule
                                                                                            constraints is not always explicitly
                                                                                            presented.
Data collection and feedback processes that     FAA policies require projects to identify   Project teams determine the type and
foster capturing and correctly interpreting     and analyze differences from earlier        amount of information they retain. Thus, the
data from work performed and entering them      estimates. FAA does not have a policy       type of information and its location varies
into the historical database. (See appendix     regarding data collection and feedback.     among projects.
II, table II.6 for more details.)




                                                Page 46                                             GAO/AIMD-97-20 Air Traffic Control
Appendix II

Detailed Comparison of SEI’s Checklist for
Each Requisite and FAA Practices for ATC
Projects
                                             The following tables provide a detailed comparison of FAA’s practices to
                                             the six SEI requisites’ components that were summarized in appendix I.


Table II.1: A Corporate Memory
Evidence of maturity                                            FAA practices
The organization has a process for organizing and retaining     FAA does not have an institutional process for creating,
information on completed projects (a historical database).      maintaining, and using a historical database(s) on ATC projects.

                                                                FAA has several processes for retaining different subsets of the
                                                                full set of information on completed project estimates and actuals.
                                                                That is, it has processes for populating several databases that
                                                                organize and retain some project information for use within certain
                                                                groups.a These individual processes are not coordinated with or
                                                                part of an institutional process for organizing and retaining all
                                                                relevant information on completed projects’ estimates and actuals.

                                                                None of these databases (either individually or combined) provide
                                                                a comprehensive cradle-to-grave understanding of a project’s
                                                                history from early estimates, through revisions, and to ultimate
                                                                completion.

                                                                Furthermore, the information catalogued in the existing databases
                                                                is limited to the perceived needs of the group that manages it.
                                                                That is, it is not always known and accessible to, or usable by,
                                                                estimators outside each group.




                                             Page 47                                           GAO/AIMD-97-20 Air Traffic Control
                                                  Appendix II
                                                  Detailed Comparison of SEI’s Checklist for
                                                  Each Requisite and FAA Practices for ATC
                                                  Projects




                                                                                                        FAA practices
                                                                                         FAA does not have an institutional historical
                            Evidence of maturity                                        database.b The databases noted above provide
           The information on completed projects includes:                                        the following information:
                                                                                       CBAS            F&E              DAFIS
1. the life cycle model used together with the portion covered by the                  yes             no               no
recorded schedule and costs
2. original size estimate                                                              yes             no               no
3. changes in size resulting from changes in requirements                              yesc            no               no
                                                                                            d
4. the original cost and schedule estimate, together with the values and               y/n             no               no
rationales used for cost model parameters
5. re-estimates and estimates-to-complete                                              yes             yes              no
6. reasons for re-estimates                                                            no              no               no
7. actual costs and schedules                                                          no              no               y/ne
8. actual size of delivered code                                                       no              no               no
9. staffing profile                                                                    no              no               no
10. labor mix                                                                          no              no               yesf
11. skill level of the project team, measured relative to the skill level of           no              no               no
the organization’s typical team
12. nonlabor costs                                                                     no              no               no
13. management costs                                                                   no              no               no
14. system integration costs                                                           no              no               no
15. an estimate at completion                                                          no              yes              yes
                                                                                                             g
16. extenuating circumstances or reasons for the differences between                   no              yes              no
the original and final estimates
17. a work breakdown structure or alternative description of the tasks                 no              no               no
included in the recorded costs
18. a work-flow schematic for the software process                                     no              no               no
19. a summary of significant deliverables produced by the project                      no              no               no
(software, documentation, etc.)
20. a summary of any unusual issues or contract factors that affected                  no              yes              no
cost or schedule
21. if multiple builds or releases were used, the size, cost, schedule, and            no              no               no
characteristics of each build or release




                                                  Page 48                                             GAO/AIMD-97-20 Air Traffic Control
                                               Appendix II
                                               Detailed Comparison of SEI’s Checklist for
                                               Each Requisite and FAA Practices for ATC
                                               Projects




Evidence of maturity                                                  FAA practices
The historical database is treated as an integral part of the         Because FAA does not have an institutional historical database, it
estimating process, and estimators have active roles in specifying    is not an integral part of the estimating process. With the
and sustaining the information it contains.                           exception of the CBAS database, none of the individual
                                                                      databases are used in the estimating process. CBAS is used at
                                                                      the discretion of the ASD-400 estimators if they are involved in
                                                                      developing a project’s cost estimate.

                                                                      With the exception of the ASD-400 estimators, estimators are not
                                                                      active in defining the information in the respective databases.
The database contains a useful set of completed projects.             FAA does not have an institutional historical database. Each of the
                                                                      individual databases contains a useful set of projects.
The elements included in (and excluded from) effort, cost,            Because FAA does not have an institutional historical database,
schedule, size, and reuse measures are clearly identified.            this type of information is not recorded therein. None of the
                                                                      individual databases provide this level of detail.

                                                                      The CBAS database provides a reference to documentation
                                                                      supporting the cost estimate, which includes information on cost
                                                                      and schedule elements, but not the elements included in effort,
                                                                      size, and reuse measures.

                                                                      Individual projects may have this information in their project files.
Schedule milestones (start and finish dates) are described in terms   Because FAA does not have an institutional historical database,
of criteria for initiation or completion.                             this type of information is not recorded therein. None of the
                                                                      individual databases provide this level of detail.

                                                                      Individual projects may have this information in their project files.
Effort and cost data clearly indicate which parts of the life cycle and Because FAA does not have an institutional historical database,
which activities are covered by the different categories of hours or this type of information is not recorded therein. None of the
costs recorded.                                                         individual databases provide this level of detail.

                                                                      Individual projects may have this information in their project files.
Records for projects indicate whether unpaid overtime was used.       Because FAA does not have an institutional historical database,
                                                                      this type of information is not recorded therein. None of the
                                                                      individual databases provide this level of detail. Further, none of
                                                                      the projects we reviewed recorded unpaid overtime.
Unpaid overtime, if used, is quantified, so that recorded data        Because FAA does not have an institutional historical database,
provide a valid basis for estimating future effort.                   this type of information is not recorded therein. None of the
                                                                      individual databases provide this level of detail. Further, none of
                                                                      the projects we reviewed recorded unpaid overtime.
Cost models are used to provide a consistent framework (standard      Because FAA does not have an institutional historical database,
terms and parameters) for recording historical data.                  this type of information is not recorded therein. FAA endorses a
                                                                      number of cost models, thus permitting differences among
                                                                      projects in the data that are recorded. Of the individual
                                                                      databases, only the CBAS database references models used by
                                                                      estimators. These models, however, are not used to record actual
                                                                      performance data on completed projects.
                                                                                                                                (continued)




                                               Page 49                                                 GAO/AIMD-97-20 Air Traffic Control
                                             Appendix II
                                             Detailed Comparison of SEI’s Checklist for
                                             Each Requisite and FAA Practices for ATC
                                             Projects




Evidence of maturity                                                       FAA practices
Historical data have been examined to identify inconsistencies, and Because FAA does not have an institutional historical database,
anomalies have been corrected or explained. (This is best done      this type of information is not recorded therein. Thus, these data
with the same cost models used for estimating.)                     are not examined for anomalies. None of the individual databases
                                                                    provide this level of detail.

                                                                           Individual projects may have this information in their project files
                                                                           and may correct anomalies.
Workflow schematics are used to describe similarities and                  Because FAA does not have an institutional historical database,
differences among projects.                                                this type of information is not recorded therein. None of the
                                                                           individual databases provide this level of detail.

                                                                           Individual projects may have this information in their project files.

                                             a
                                                 The databases include:

                                             —the Cost Benefit Analysis System (CBAS), which catalogues project estimates and provides a
                                             reference to the location of supporting documentation in a repository. It is only accessible by
                                             estimators in the Investment Analysis and Operations Research Division (ASD-400) of the Office
                                             of Systems Architecture and Investment Analysis;

                                             —the Facilities and Equipment Database, which tracks changes in F&E estimates over time, but
                                             whose supporting documentation is located in a number of places. It is used by one individual to
                                             answer ad hoc FAA and congressional management and oversight questions on how estimates
                                             have changed over time; and

                                             —several financial management systems, including REDMACS, DAFIS, FMS, and COPS, which
                                             provide budgetary performance information to project teams.
                                             b
                                                 Individual projects may contain some of this information in their project files.
                                             c
                                                 Only if a new cost estimate is generated.
                                             d
                                              The database repository contains cost and schedule estimates, but not the values and rationales
                                             used for cost model parameters.
                                             e
                                                 DAFIS can provide actual costs, but no schedule information.
                                             f
                                              This information could be pulled, with effort, from DAFIS.
                                             g
                                              The database provides a marker to when, or how, the estimate changed (i.e., at a major
                                             acquisition review), not the reason for the change.




                                             Page 50                                                            GAO/AIMD-97-20 Air Traffic Control
                                                Appendix II
                                                Detailed Comparison of SEI’s Checklist for
                                                Each Requisite and FAA Practices for ATC
                                                Projects




Table II.2: Structured Processes for Estimating Software Size and Reuse
Evidence of maturity                         FAA practices
The estimating processes for size and reuse     FAA does not have a documented institutional process for estimating size and reuse.
are documented.                                 Project teams are permitted to follow whatever approach they choose for estimating
                                                product size and reuse.
The estimating processes for size and reuse     Because FAA does not have a documented institutional process for estimating size and
are followed.                                   reuse, project teams do not follow such a process. Project teams are permitted to follow
                                                whatever approach they choose for estimating product size and reuse.
The descriptions of size and reuse identify     Because FAA does not have a documented institutional process for estimating size and
what has been included in (and excluded         reuse, project approaches differ in the extent to which they describe what has been
from) the size and reuse measures.              included and excluded from size and reuse measures.
The measures of reuse distinguish between       Because FAA does not have a documented institutional process for estimating size and
code that will be modified and code that will   reuse, project approaches differ in how reuse measures are described.
be integrated as is into the system.
Size estimates are checked by relating them Because FAA does not have a documented institutional process for estimating size and
to measured sizes of other software products reuse, project approaches differ in how size estimates are checked.
or components.
The size estimating process is checked          Because FAA does not have a documented institutional process for estimating size and
periodically by comparing its predictive        reuse, project approaches differ in how size estimating approaches are checked.
capabilities with measured sizes of
completed products.
Because size estimating is often the weakest FAA does not have any ongoing efforts that focus on improving its size estimating
link in cost and schedule estimating, the    process.
organization has a continuing effort that
focuses on improving its size estimating     Project-specific efforts may exist.
process.




                                                Page 51                                               GAO/AIMD-97-20 Air Traffic Control
                                                 Appendix II
                                                 Detailed Comparison of SEI’s Checklist for
                                                 Each Requisite and FAA Practices for ATC
                                                 Projects




Table II.3: Mechanisms for Extrapolating From Demonstrated Accomplishments on Past Projects
Evidence of maturity                        FAA practices
The extrapolation process is documented.         FAA does not have a documented institutional process requiring, or describing how to
                                                 go about, extrapolation from past projects. Project teams are permitted to follow
                                                 whatever approach they choose for their extrapolations, if in fact they choose to
                                                 extrapolate.
Cost models and other tools have been            FAA recommends a number of cost models to assist estimators. However, it does not
acquired or developed to assist estimators.      provide these models—estimators are allowed to pick any model they choose, if any,
                                                 and are individually responsible for obtaining any needed licenses.
The cost models have been calibrated to          Because FAA does not have a documented institutional process for extrapolating from
relevant historical data.                        past projects, project approaches differ based on if, and how, cost models are
                                                 calibrated.
Cost model calibrations are up to date.          Because FAA does not have a documented institutional process for extrapolating from
                                                 past projects, project approaches differ based on if, and how, cost models are
                                                 calibrated.
The cost and schedule models are used to         Because FAA does not have a documented institutional process for extrapolating from
quantify demonstrated organizational             past projects, project approaches differ in the extent to which they use cost models to
performance in ways that normalize for           quantify demonstrated organizational performance.
differences among software products and
projects.
The consistency that estimators achieve          Because FAA does not have a documented institutional process for extrapolating from
when fitting cost models to historical data is   past projects, project approaches differ in the extent to which they track cost model
measured and tracked.                            performance.
Values used for cost model parameters are    Because FAA does not have a documented institutional process for extrapolating from
validated by comparisons with past projects. past projects, project approaches differ in the extent to which they validate cost model
                                             parameters.
The methods used to account for reuse            Because FAA does not have a documented institutional process for extrapolating from
recognize that reuse is not free.                past projects, project approaches differ in their recognition of software reuse costs.
Extrapolations from past projects incorporate Because FAA does not have a documented institutional process for extrapolating from
measured trends in technology                 past projects, project teams are permitted to determine if they extrapolate from past
improvement, either within the cost models    projects, and what this extrapolation incorporates.
themselves or as inputs to them.
Estimators work jointly with project managers If used by the project team, ASD-400 or contract estimators work jointly with project
and experienced technical people to identify managers and technical personnel to identify similarities between current and prior
how the new work compares to work the         systems.
organization or others have done before.
More than one cost model or estimating           FAA policies require use of one or more approved cost estimating approaches.
approach is used, and differences among          However, FAA does not require that estimators analyze and explain differences in
results are analyzed and explained.              estimates. Project teams are permitted to determine if and how extensively they assess
                                                 any differences among estimates.
Trends in the organization’s process and         Because FAA does not have a documented institutional process for extrapolating from
performance parameters are tracked to            past project, project teams are permitted to determine if and how they will track and
identify their effects on cost model             identify the effects of trends in organizational parameters on cost model calibrations.
calibrations.




                                                 Page 52                                               GAO/AIMD-97-20 Air Traffic Control
                                                Appendix II
                                                Detailed Comparison of SEI’s Checklist for
                                                Each Requisite and FAA Practices for ATC
                                                Projects




Table II.4: An Audit Trail
Evidence of maturity                            FAA practices
The organization’s process documentation        FAA does not have an institutional process identifying who is responsible for preparing
identifies who is responsible for preparing     an audit trail for software estimates. Project teams are permitted to determine who
the audit trail for software estimates.         documents the audit trail and the extent of this documentation.
A list of parameter values and their rationales FAA does not have an institutional process requiring estimators to list all parameter
accompanies each estimate.                      values and their rationales. Project teams are permitted to determine the extent to which
                                                estimators document parameter values and their rationales.
A template or format is used to record the      FAA does not have an institutional process for recording cost model parameters and
values of cost model parameters and their       their rationales. Project teams are permitted to determine how they will record the values
rationales.                                     of cost model parameters and their rationales.
Uncertainties in parameter values are           FAA does not have an institutional process for dealing with uncertainties in parameter
identified and quantified.                      values. Project teams are permitted to determine the extent to which they identify and
                                                quantify uncertainties in parameter values.
The lists of parameter values and their         Because FAA does not have an institutional historical database, this type of information
rationales are retained in the organization’s   is not retained therein. None of the individual databases provide this level of detail.
historical database.



Table II.5: Integrity in Dealing With Dictated Costs and Schedules
Evidence of maturity                           FAA practices
Management reviews and agrees to                FAA does not have an institutional policy requiring managers to review and agree to
parameter values and rationales before          parameter values and rationales before costs are estimated. Project teams are permitted
costs are estimated.                            to determine if and when management approval is needed in developing cost estimates.
                                                FAA’s Office of Aviation Policy and Plans (APO) is responsible for reviewing the
                                                methodological soundness of cost estimates. However, APO does not evaluate
                                                parameter values and their rationales.
Reasons for changing parameter values from FAA does not have an institutional process on changing parameter values. Project teams
those identified in the calibration set are are permitted to determine which parameter values they will use.
documented.
Adjustments to cost model parameters to         FAA does not have an institutional process on dealing with imposed costs and
meet desired costs or schedules are             schedules. Project teams are permitted to determine how they will deal with cost or
accompanied by management actions that          schedule objectives.
make the parameter values realistic.
The actions that the organization intends to    Because FAA does not have an institutional process on dealing with imposed costs and
take to make its adjusted cost model            schedules, it does not require projects to justify how to make adjusted cost model
parameters valid are spelled out in the         parameters valid. Project teams are permitted to determine how they will deal with cost
project plan.                                   and schedule objectives.




                                                Page 53                                               GAO/AIMD-97-20 Air Traffic Control
                                               Appendix II
                                               Detailed Comparison of SEI’s Checklist for
                                               Each Requisite and FAA Practices for ATC
                                               Projects




Table II.6: Data Collection and Feedback Processes That Foster Capturing and Correctly Interpreting Data From Work
Performed
Evidence of maturity                        FAA practices
There is a defined process for gathering       Because FAA does not have an institutional historical database, it does not have a
information on completed projects and          process for gathering information on completed projects and entering it into the historical
entering it into the historical database.      database. Of the individual databases, only the financial databases have information on
                                               completed projects, and this information is strictly budgetary.

                                               Individual project teams may have this information in their project files.
Postmortems are held at the completion of      FAA does not have an institutional process for holding postmortem assessments at the
each project to ensure that (1) recorded data completion of each project. FAA performs project postmortem reviews on an ad hoc
are valid and (2) events that affected cost or basis.
schedule get recorded and described while
they are still fresh in people’s minds.
Estimates used for original project planning   Because FAA does not have an institutional historical database, this type of information
are saved and entered into the historical      is not recorded therein. Estimates for original project planning are retained by ASD-400,
database.                                      and filed in the CBAS system.
Re-estimates and estimates for changes to      Because FAA does not have an institutional historical database, this type of information
the product or process are recorded and        is not recorded therein. Project teams are required to file updated cost estimates with
saved in the historical database.              ASD-400, for input into the CBAS database.
Pilots and prototypes of new software        FAA does not have an institutional process for incorporating full-scale software process
processes are measured and tracked to        improvements into its estimating approaches.
capture information that can guide estimates
for full-scale processes.
Organizations that acquire software receive    FAA does not have an institutional process for receiving and saving copies of the
and save copies of the developer’s             developer’s postmortem reports. Project teams are permitted to determine the extent of
postmortem reports.                            the documentation they require from contractors and retain it.
There is a structured process for capturing    FAA policies require monitoring actual program cost, schedule,and technical
data on effort and cost from ongoing and       achievement. Additionally, all major project teams require contractors to submit contract
completed projects.                            performance reports detailing progress against cost, schedule, and technical goals. This
                                               information is reported at formal acquisition review meetings and retained by the project
                                               office.
The capturing of data for cost estimating and FAA does not have an institutional process for integrating its estimating/planning
planning is integrated with the measurement functions with its measurement processes used for tracking, oversight, and process
processes used for project tracking and       improvement on individual projects.
oversight and process improvement.
                                              Individual project teams may incorporate project tracking information in their
                                              re-estimates.
Estimates-to-complete are updated and          FAA requires that estimates-to-complete be updated at major acquisition reviews.
reviewed at regularly scheduled intervals
(e.g., monthly).
Estimates-to-complete are updated and          FAA requires project managers to monitor project performance against the cost,
reviewed whenever there is a major change      schedule, and technical boundaries in the acquisition program baseline and to promptly
to requirements, resources, priorities,        report any anticipated breaches to the acquisition executive.
commitments, assumptions, or
understanding of the project.
The processes for capturing, collecting, and   FAA does not have institutional processes for the automated capture, collection, and
disseminating measurement results and          dissemination of measurement results and descriptive data.
descriptive data are supported by
automation, so that opportunities for
misinformation, sloppiness, and indifference
are minimized.
                                               Page 54                                                 GAO/AIMD-97-20 Air Traffic Control
Appendix III

SEI Checklist for Validating the Reliability of
a Project’s Cost Estimate

                                               Table III.1 provides a summary of the checklist items supporting each SEI
                                               question.


Table III.1: SEI Checklist for Validating the Reliability of a Project’s Cost Estimate
SEI questions                                  Checklist items
Are the objectives of the estimate clear and   The objectives of the estimate are stated in writing.
correct?                                       The life cycle to which the estimate applies is clearly defined.
                                               The tasks and activities included in (and excluded from) the estimate are clearly
                                               identified.
                                               The tasks and activities included in the estimate are consistent with the objectives of the
                                               estimate.
Has the task been appropriately sized?         A structured process has been used to estimate and describe the size of the software
                                               product.
                                               A structured process has been used to estimate and describe the extent of reuse.
                                               The processes for estimating size and reuse are documented.
                                               The descriptions of size and reuse identify what is included in (and excluded from) the
                                               size and reuse measures used.
                                               The measures of reuse distinguish between code that will be modified and code that will
                                               be integrated as is into the system.
                                               The definitions, measures, and rules used to describe size and reuse are consistent with
                                               the requirements (and calibrations) of the models used to estimate cost and schedule.
                                               The size estimate was checked by relating it to measured sizes of other software
                                               products or components.
                                               The size estimating process was checked by testing its predictive capabilities against
                                               measured sizes of completed products.
                                                                                                                               (continued)




                                               Page 55                                                GAO/AIMD-97-20 Air Traffic Control
                                            Appendix III
                                            SEI Checklist for Validating the Reliability
                                            of a Project’s Cost Estimate




SEI questions                               Checklist items
Are the estimated cost and schedule         The organization has a structured process for relating estimates to actual costs and
consistent with demonstrated                schedules of completed work.
accomplishments on other projects?          —The process is documented.
                                            —The process was followed.
                                            The cost and schedule models that were used have been calibrated to relevant historical
                                            data. (Models of some sort are needed to provide consistent rules for extrapolating from
                                            previous experience.)
                                            The cost and schedule models quantify demonstrated organizational performance in
                                            ways that normalize for differences among software products and projects. (So that a
                                            simple, unnormalized, lines-of-code per staff-month extrapolation is NOT the basis for
                                            the estimate.)
                                            The consistency achieved when fitting the cost and schedule models to historical data
                                            has been measured and reported.
                                            The values used for cost and schedule model parameters appear valid when compared
                                            to values that fit the models well to past projects.
                                            The calibration of cost and schedule models was done with the same versions of the
                                            models that were used to prepare the estimate.
                                            The methods used to account for reuse recognize that reuse is not free. (The estimate
                                            accounts for activities such as interface design, modification, integration, testing, and
                                            documentation that are associated with effective reuse.)
                                            Extrapolations from past projects account for differences in application technology. (For
                                            example, data from projects that implemented traditional mainframe applications require
                                            adjustments if used as a basis for estimating client-server implementation. Some cost
                                            models provide capabilities for this, others do not.)
                                            Extrapolations from past projects account for observed, long-term trends in software
                                            technology improvement. (Although some cost models attempt this internally, the best
                                            methods are usually based on extrapolating measured trends in calibrated
                                            organizational performance.)

                                            Extrapolations from past projects account for the effects of introducing new software
                                            technology or processes. (Introducing a new technology or process can initially reduce
                                            an organizations’s productivity.)
                                            Work-flow schematics have been used to evaluate how this project is similar to (and how
                                            it differs from) projects used to characterize the organization’s past performance.
Have the factors that affect the estimate   A written summary of parameter values and their rationales accompanies the estimate.
been identified and explained?              Assumptions have been identified and explained.
                                            A structured process such as a template or format has been used to ensure that key
                                            factors have not been overlooked.
                                            Uncertainties in parameter values have been identified and quantified.
                                            A risk analysis has been performed, and risks that affect cost or schedule have been
                                            identified and documented. (Elements addressed include issues such as probability of
                                            occurrence, effects on parameter values, cost impacts, schedule impacts, and
                                            interactions with other organizations.)
                                                                                                                         (continued)




                                            Page 56                                              GAO/AIMD-97-20 Air Traffic Control
                                       Appendix III
                                       SEI Checklist for Validating the Reliability
                                       of a Project’s Cost Estimate




SEI questions                          Checklist items
Have steps been taken to ensure the    Management reviewed and agreed to the values for all descriptive parameters before
integrity of the estimating process?   costs were estimated.
                                       Adjustments to parameter values to meet a desired cost or schedule have been
                                       documented.
                                       If a dictated schedule has been imposed, the estimate is accompanied by an estimate of
                                       (1) the normal schedule and (2) the additional expenditures required to meet the
                                       dictated schedule.
                                       Adjustments to parameter values to meet a desired cost or schedule are accompanied
                                       by management action that makes the values realistic.
                                       More than one cost model or estimating approach has been used, and the differences in
                                       results have been analyzed and explained.
                                       People from related but different projects or disciplines were involved in preparing the
                                       estimate.
                                       At least one member of the estimating team is an experienced estimator, trained in the
                                       cost models that were used.
                                       Estimators independent of the performing organization concur with the reasonableness
                                       of the parameter values and estimating methodology.
                                       The groups that will be doing the work accept the estimate as an achievable target.
                                       Memorandums of agreement have been completed and signed with the other
                                       organizations whose contributions affect cost or schedule.
                                                                                                                   (continued)




                                       Page 57                                              GAO/AIMD-97-20 Air Traffic Control
                                              Appendix III
                                              SEI Checklist for Validating the Reliability
                                              of a Project’s Cost Estimate




SEI questions                                 Checklist items
Is the estimate based on reliable evidence of The estimating organization has a method for organizing and retaining information on
the organization’s past performance?          completed projects (a historical database).
                                              The database contains a useful set of completed projects.
                                              Elements included in (and excluded from) the effort, cost, schedule, size, and reuse
                                              measures in the database are clearly identified. (See, for example, the SEI checklist for
                                              defining effort, schedule, and size measures.)
                                              Schedule milestones (start and finish dates) are described in terms of criteria for
                                              initiation or completion, so that work accomplished between milestones is clearly
                                              bounded.
                                              Records for completed projects indicate whether unpaid overtime was used.
                                              Unpaid overtime, if used, has been quantified so that recorded data provide a valid
                                              basis for estimating future effort.
                                              Cost models that were used for estimating also have been used to provide consistent
                                              frameworks for recording historical data. (This helps ensure that comparable terms and
                                              parameters are used across all projects, and that recorded data are suitable for use in
                                              the estimating models.)
                                              The data in the historical database have been examined to identify inconsistencies, and
                                              anomalies have been corrected or explained. (This is best done with the same cost
                                              models used for estimating.)
                                              The organization has a structured process for capturing effort and cost data from
                                              ongoing projects.
                                              The producing organization holds postmortems at the completion of its projects to (1)
                                              ensure that recorded data are valid and (2) ensure that events that affected costs or
                                              schedules get recorded and described while they are still fresh in people’s minds.
                                              Information on completed projects includes
                                              —the life-cycle model used, together with the portion covered by the recorded cost and
                                              schedule;
                                              —actual (measured) size, cost and schedule;
                                              —the actual staffing profile;
                                              —an estimate at completion, together with the values for cost model parameters that
                                              map the estimate to the actual cost and schedule;
                                              —a work breakdown structure or alternative description of the tasks included in the
                                              recorded cost;
                                              —a work-flow schematic that illustrates the software process used;
                                              —nonlabor costs;
                                              —management costs;
                                              —a summary or list of significant deliverables (software and documentation) produced
                                              by the project; and
                                              —a summary of any unusual issues that affected cost or schedule.
                                              Evolution in the organization’s work-flow schematics shows steady improvement in the
                                              understanding and measurement of its software processes.
Has the situation remained unchanged since The estimate has not been invalidated by recent events, changing requirements, or
the estimate was prepared?                 management action (or inaction).
                                           The estimate is being used as the basis for assigning resources, deploying schedules,
                                           and making commitments.
                                           The estimate is the current baseline for project tracking and oversight.




                                              Page 58                                               GAO/AIMD-97-20 Air Traffic Control
Appendix IV

Six ATC Projects’ Cost Estimates’
Satisfaction of SEI’s Checklist

                                                Table IV.1 provides a summary comparison of six ATC projects’ estimates
                                                against SEI’s questions for assessing an estimate’s reliability. SEI provides
                                                detailed checklists for addressing these seven questions (see appendix III
                                                for a summary of the detailed checklists). We compared official project life
                                                cycle estimates to the detailed checklists to determine how well the
                                                projects satisfied each question. The fraction in each block shows the
                                                number of satisfactory (worth 1 point) or partially satisfactory responses
                                                (worth .5 points) to checklist items divided by the total number of items.
                                                Projects with insufficient documentation to support an assessment were
                                                given a “No Basis” rating, indicating that there was no basis for an
                                                evaluation. Because SEI checklists focus on key processes, we did not
                                                evaluate project cost estimates to determine if all applicable costs are
                                                included.

                                                An SEI expert agreed that this method was an acceptable and conservative
                                                approach to scoring projects. However, he cautioned that a project with an
                                                extremely high score could lack the one checklist item that is critical to
                                                that specific project, thus rendering the estimate unreliable.


Table IV.1: Six ATC Projects’ Cost Estimates’ Satisfaction of SEI’s Checklist
Projects SEI Questions                       VSCS           DSR          STARS             ASR-9         WAAS           DCCR
                                                                                                   a
Are the objectives of the estimate clear and        4/4              4/4          3/4      No Basis      4/4            No Basis
correct?
Has the task been appropriately sized?              No Basis         5.5/8        2/8      No Basis      No Basis       No Basis
Are the estimated cost and schedule                 No Basis         4.5/10       0.5/10   No Basis      2/10           No Basis
consistent with demonstrated
accomplishments on other projects?
Have the factors that affect the estimate been 2/5                   3.5/5        1/5      No Basis      3/5            No Basis
identified and explained?
Have steps been taken to ensure the integrity       5.5/7            5.5/7        2.5/7    No Basis      3/7            No Basis
of the estimating process?
Is the estimate based on reliable evidence of       1.5/12           2.5/12       0.5/12   No Basis      1/12           No Basis
the organization’s past performance?
Has the situation remained unchanged since          1/3              3/3          0/3      No Basis      0/3            No Basis
the estimate was prepared?
                                                a
                                                    No basis for an evaluation.




                                                Page 59                                            GAO/AIMD-97-20 Air Traffic Control
Appendix V

Major Contributors to This Report


                       Linda M. Calbom, Director
Accounting and         Randolph C. Hite, Senior Assistant Director
Information            John C. Fretwell, Assistant Director
Management Division,   Keith A. Rhodes Technical Assistant Director
                       Madhav S. Panwar, Senior Technical Advisor
Washington D.C.        Colleen M. Phillips, Senior Information Systems Analyst
                       Tomas Ramirez, Jr., Senior Business Process Analyst
                       Cynthia Jackson, Senior Auditor
                       Deborah R. Peay, Staff Auditor




(511402)               Page 60                                     GAO/AIMD-97-20 Air Traffic Control
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested