oversight

National Airspace System: Better Cost Data Could Improve FAA's Management of the Standard Terminal Automation Replacement System

Published by the Government Accountability Office on 2003-01-31.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States General Accounting Office

GAO            Report to the Honorable Ellen O.
               Tauscher, House of Representatives



January 2003
               NATIONAL
               AIRSPACE SYSTEM
               Better Cost Data
               Could Improve FAA’s
               Management of the
               Standard Terminal
               Automation
               Replacement System




GAO-03-343
               a
                                               January 2003


                                               NATIONAL AIRSPACE SYSTEM

                                               Better Cost Data Could Improve FAA’s
Highlights of GAO-03-343, a report to the      Management of the Standard Terminal
Honorable Ellen O. Tauscher, House of
Representatives                                Automation Replacement System



To enhance the capacity and safety             The reliability of FAA’s life-cycle cost estimate for STARS is uncertain. This
of the national airspace system, the           estimate includes estimates of the program’s development, operation, and
Federal Aviation Administration                technology upgrade costs, shown for the next 6 fiscal years in the figure
(FAA), within the Department of                below. The development cost estimate is based on the contractor’s
Transportation, is acquiring 74                projections, which FAA has not yet independently analyzed as its guidance
Standard Terminal Automation
Replacement Systems (STARS).
                                               directs. Furthermore, baseline data in cost performance reports that FAA
STARS will replace some outdated               obtains from the contractor are not accurate because the data do not reflect
air traffic control equipment. Since           the current status of the contract. As a result, FAA is limited in its ability to
1996, when FAA initiated this major            manage the contract effectively. FAA plans to address these problems. In
computer hardware and software                 addition, the program’s operation and technology upgrade cost estimates are
acquisition, the scope and                     based on limited experience with STARS and extend many years into the
estimated costs of STARS have                  future. However, the estimates do not reflect these uncertainties.
changed many times. FAA now
estimates that STARS’s remaining               Estimated STARS Funding, Fiscal Years 2004-2009
costs will total about $2.54 billion.
GAO was asked to assess the                    Cost element (Dollars in millions)
                                               100
reliability of FAA’s life-cycle cost
                                                90
estimate for STARS, determine the
impact of STARS’s estimated costs               80

on future FAA budgets, and identify             70
any alternatives to STARS that FAA              60
is considering. GAO based its                   50
analysis on published FAA cost                  40
data and the guidance FAA uses for              30
managing major acquisitions.                    20
                                                10
                                                 0
                                                     2004          2005       2006           2007         2008         2009
                                                     Fiscal year
To improve FAA’s management of
                                                     Development/Deployment          Operations/Maintenance      Technology upgrades
STARS and of subsequent terminal
                                               Sources: GAO and FAA.
modernization programs and to
provide the Congress with more                 For fiscal years 2004 through 2007, the years for which FAA provided budget
reliable information for oversight,            information, STARS’s estimated costs should have a declining impact on
GAO recommends that the                        FAA’s budgets because the program’s development is nearly over and its
Secretary of Transportation direct             operations are still limited. For fiscal years 2008 through 2030, the impact of
the FAA Administrator to maintain
                                               STARS’s estimated costs on FAA’s budgets is unknown because it is still too
accurate, current baseline data;
review baseline data within 6                  soon in the budget cycle for FAA to have developed detailed budgets for
months of any major modification               these years.
to ensure that the data reflect the
current status of the contract; and            After deploying STARS at the 74 terminal and support facilities included in
prepare a rigorous life-cycle cost             the program, FAA could use STARS, another contractor’s technology, or a
estimate that identifies the level of          combination of the two technologies for the nearly 100 remaining facilities.
uncertainty.                                   FAA is exploring the feasibility of combining the technologies and expects to
www.gao.gov/cgi-bin/getrpt?GAO-03-343.         announce its plans in April 2003.

To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Gerald L.
Dillingham at (202) 512-5555 or Keith A.
Rhodes at (202) 512-6412.
Contents



Letter                                                                                                1
                         Results in Brief                                                             2
                         Background                                                                   4
                         Reliability of FAA’s Life-Cycle Cost Estimate for STARS Is
                           Uncertain                                                                  8
                         Impact of STARS’s Estimated Costs on FAA’s Budgets Is Expected to
                           Decline through Fiscal Year 2007                                          13
                         After Deploying STARS at 74 Facilities, FAA May Modify Its
                           Approach for the Remaining Facilities                                     16
                         Conclusions                                                                 16
                         Recommendations for Executive Action                                        17
                         Agency Comments                                                             17
                         Scope and Methodology                                                       18


Appendix
           Appendix I:   GAO Contacts and Staff Acknowledgments                                      20
                         GAO Contacts                                                                20
                         Staff Acknowledgments                                                       20


Table                    Table 1: Proposed Funding for FAA’s Total Facilities and Equipment
                                  Budget and for STARS’s Development and Deployment                  15


Figures                  Figure 1: Standard Terminal Automation Replacement System
                                   (STARS)                                                            5
                         Figure 2: Life-Cycle Cost Estimate for STARS, Fiscal Years
                                   2004-2030                                                          9
                         Figure 3: Allocation of Facilities and Equipment Funding for Fiscal
                                   Year 2004 in FAA’s Capital Investment Plan                        14




                         Page i                                      GAO-03-343 National Airspace System
Contents




Abbreviations

ARTS         Automated Radar Terminal System
DCMA         Defense Contract Management Agency
DOD          Department of Defense
FAA          Federal Aviation Administration
STARS        Standard Terminal Automation Replacement System

 This is a work of the U.S. Government and is not subject to copyright protection in the
 United States. It may be reproduced and distributed in its entirety without further
 permission from GAO. It may contain copyrighted graphics, images or other materials.
 Permission from the copyright holder may be necessary should you wish to reproduce
 copyrighted materials separately from GAO’s product.




Page ii                                              GAO-03-343 National Airspace System
A
United States General Accounting Office
Washington, D.C. 20548



                                    January 31, 2003                                                                                 Leter




                                    The Honorable Ellen O. Tauscher
                                    House of Representatives

                                    Dear Ms. Tauscher:

                                    On November 17, 2002, the Federal Aviation Administration (FAA) began
                                    using the Standard Terminal Automation Replacement System (STARS) to
                                    control air traffic at the Philadelphia air route traffic control center, the
                                    first “busy” FAA facility to use this version of the new system.1 FAA’s
                                    current plan is to procure 74 STARS systems, including 70 for terminal
                                    facilities and 4 for support facilities.2 STARS will replace outdated
                                    computer equipment that is used to control air traffic within 5 to 50
                                    nautical miles of an airport.3 With STARS, air traffic controllers at these
                                    facilities will receive new hardware and software that produce color
                                    displays of aircraft position and flight information. In the future, FAA will
                                    be able to upgrade the software to provide air traffic control tools, such as
                                    those that will allow better spacing of aircraft as they descend into airports.
                                    STARS is complex, costly, and software-intensive. Since 1996, when FAA
                                    initiated STARS, the number of systems scheduled to be procured has
                                    ranged from as many as 188 to as few as 74,4 and the program’s cost and
                                    schedule have also varied considerably. Over the years, we have reported
                                    on these changes, most recently in September 2002.5

                                    As agreed with your office, this report addresses the following questions:




                                    1
                                     A previous version of STARS has been in use at smaller facilities since 1999. Philadelphia is
                                    the 14th-largest terminal facility in terms of handling operations under instrument flight
                                    rules. FAA’s data show that Philadelphia handled 686,000 operations between January and
                                    December 2000.
                                    2
                                     To support STARS operations at terminal facilities, FAA currently has four systems
                                    dedicated to maintenance.
                                    3
                                     The Department of Defense (DOD) is also procuring STARS for 153 of its facilities.
                                    4
                                     FAA currently plans to deploy STARS to a total of about 170 terminal and support facilities.
                                    5
                                     U.S. General Accounting Office, National Airspace System: Status of FAA’s Standard
                                    Terminal Automation Replacement System, GAO-02-1071 (Washington, D.C.: Sept. 17,
                                    2002).




                                    Page 1                                                 GAO-03-343 National Airspace System
                   • How reliable is FAA’s estimate of the life-cycle costs to develop and
                     deploy, operate and maintain, and upgrade STARS?

                   • What impact will STARS’s estimated cost have on future FAA budgets
                     given competing demands for funds to enhance aviation safety, security,
                     and capacity?

                   • What alternatives to STARS is FAA considering?

                   In addition, you asked whether STARS falls under the termination
                   provisions of Public Law 104-264, as an acquisition that is more than 50
                   percent over its cost goal or behind schedule. Our analysis of the law and
                   its legislative history indicates that STARS is not subject to these
                   termination provisions. According to our analysis, the termination
                   provisions apply to acquisitions initiated after October 9, 1996, the date of
                   the law’s enactment. Because FAA approved the initial acquisition plan for
                   STARS in March 1996 and signed the contract with Raytheon Corporation
                   in September 1996, STARS is not subject to those provisions.

                   This report covers cost and performance issues related to FAA’s
                   procurement of STARS for 74 terminal and support facilities.6 To conduct
                   our work, we reviewed FAA’s 5-year Capital Investment Plan, which
                   proposes funding for programs to modernize the national airspace system.
                   We also analyzed data from cost performance reports that the STARS
                   contractor developed for FAA. However, we did not independently verify
                   these cost and performance data.



Results in Brief   The reliability of FAA’s life-cycle cost estimate for STARS is uncertain.
                   According to FAA, the costs to develop and deploy, 7 operate and maintain,
                   and upgrade STARS will amount to about $2.54 billion for 74 systems for
                   fiscal years 2004 through 2030. More specifically, the costs of completing
                   STARS’s development and deployment will amount to about $153 million
                   for fiscal years 2004 through 2008, the costs of operating and maintaining
                   the systems at those facilities over their useful lives will add another $1.46


                   The report does not address DOD’s efforts to deploy the equipment.
                   6


                   7
                    Throughout this report, we use the term “deploy” to denote efforts by FAA to put STARS
                   software, hardware, and other supporting equipment into a facility to test it and eventually
                   use it to control traffic.




                   Page 2                                                GAO-03-343 National Airspace System
billion for fiscal years 2004 through 2030,8 and the costs of upgrading
STARS technology will amount to about $930 million over the same period.
FAA’s development cost estimate is based largely on the contractor’s recent
proposals and projections, which incorporate the costs of new work
specified in major modifications to the STARS contract. FAA has not yet
independently analyzed these proposals and projections as its guidance
directs and therefore does not know whether the development cost
estimate of $153 million is reliable. Furthermore, FAA has not worked with
the contractor to incorporate the requirements and costs of the new work
into cost performance reports, which it receives from the contractor. These
reports are intended to provide FAA with accurate, current data for
monitoring and overseeing the contractor’s progress and for estimating the
program’s remaining costs, but FAA is not using the reports because they
do not reflect the current status of the contract. FAA is now analyzing the
contractor’s cost data and working with the contractor to align the cost
performance reports with the current status of the contract. FAA expects
to complete these tasks in the spring of 2003 and then should be able to use
the cost performance reports as intended. The reliability of FAA’s estimates
of the life-cycle costs to operate, maintain, and upgrade STARS technology
is unknown, primarily because FAA has limited experience with STARS
equipment and the estimates extend nearly 30 years into the future.
However, despite these uncertainties, FAA has expressed its cost estimates
as point values, rather than as ranges. As a result, the estimates may imply
more certainty than is appropriate. Moreover, in this instance, the use of
point values limits disclosure of the program’s investment risks. We are
making recommendations to strengthen FAA’s management of STARS, and
FAA officials indicated that the recommendations were in line with the
agency’s ongoing and planned efforts. These recommendations, we believe,
will also help FAA better manage the planned modernization of terminal
facilities that are no longer included in the STARS program.

According to FAA’s latest budget planning documents, the impact of
STARS’s estimated costs on FAA’s budgets will decline for fiscal years 2004
through 2007; for later years, however, the impact of these estimated costs
is unknown. Since FAA has nearly finished developing STARS, the
program’s development costs are expected to decrease over the next 4
fiscal years, while its operation and maintenance costs are expected to
grow with increased deployment. According to FAA’s documents, the


8
 FAA noted that the $1.46 billion would be predominantly for labor costs that the agency
would incur for STARS or a similar automation system.




Page 3                                               GAO-03-343 National Airspace System
             funding proposed to develop and deploy STARS represents 4 percent of
             FAA’s proposed $3.06 billion facilities and equipment budget account for
             fiscal year 2004 and smaller percentages for fiscal years 2005 through 2007.
             This account funds the development, procurement, and installation of
             equipment to help increase the capacity and improve the safety, efficiency,
             and security of the national airspace system. Given these small and
             declining percentages for STARS, the program’s estimated costs should,
             over this period, have a small and declining impact on other aviation safety,
             security, and capacity efforts funded from the facilities and equipment
             account. FAA’s budget planning documents also show that as FAA deploys
             STARS at more facilities, more funding will come from the agency’s
             operations account, which supports training and compensation for the
             controllers and technicians who operate and maintain STARS. However,
             the impact of STARS’s estimated costs on FAA’s budgets for later years is
             currently unknown because it is too soon in the budget cycle for FAA to
             have developed detailed budgets beyond fiscal year 2007. In addition, as we
             previously noted, FAA has limited operational experience for projecting
             STARS’s budgetary impact, and long-range cost estimates are inherently
             uncertain.

             FAA is committed to deploying STARS at the 74 terminal and support
             facilities included in the STARS program, but for the nearly 100 other
             facilities that remain to be modernized, the agency could deploy STARS,
             another contractor’s hardware and software, or a combination of STARS
             and the other contractor’s technologies that are currently being used in
             FAA terminal facilities. FAA knows that each contractor’s technology
             works independently, and FAA is assessing the feasibility of operating
             Raytheon’s STARS display system with the processing system from
             Lockheed Martin Corporation’s Common Automated Radar Terminal
             System, which was recently installed at some terminal facilities. Combining
             the two technologies could be cost-effective because it would allow FAA to
             use both the customized display system that accounted for a substantial
             portion of STARS’s development costs and the recently acquired processing
             system. FAA plans to announce the results of this assessment in April 2003.



Background   STARS will replace controller workstations with new color displays,
             processors, and computer software at FAA and Department of Defense
             terminal air traffic control facilities. (See fig. 1.) FAA’s goal for STARS is to
             provide an open, expandable terminal automation platform that can
             accommodate future air traffic growth and allow for the introduction of
             new hardware- and software-based tools to promote safety, maximize



             Page 4                                          GAO-03-343 National Airspace System
operational efficiency, and improve controllers’ productivity. FAA believes
that STARS will facilitate efforts to optimally configure the terminal
airspace around the country, exchange digital information between pilots
and controllers, and introduce new position and surveillance capabilities
for pilots.



Figure 1: Standard Terminal Automation Replacement System (STARS)




  Source: FAA.



FAA has given high priority to STARS. Both the past and the current FAA
Administrator have emphasized the program’s importance to enhancing the
capacity of the national airspace system. In addition, STARS is 1 of 19
programs on FAA’s list of top programs. Recently, FAA gave priority
consideration to STARS so that it could meet its commitment to the
Congress to deploy STARS at the Philadelphia terminal, the first busy
facility, on November 17, 2002. Furthermore, according to FAA officials, the



Page 5                                       GAO-03-343 National Airspace System
agency remains committed to funding STARS until completion, even if that
means postponing the funding for other programs. Agency officials have
indicated, for example, that if the fiscal year 2004 budget falls below the
level shown in FAA’s planning documents, funds from other programs
would likely be cut to fully fund STARS.

For each acquisition program that FAA undertakes, it officially estimates
the program’s life-cycle costs (from development and deployment through
operations and maintenance), schedule, benefits, and performance in a
formal budget document called the acquisition program baseline. FAA uses
this document—which its acquisition decision-making body, the Joint
Resources Council, must approve—to decide whether to fund the program
and, if it is funded, to monitor its progress. FAA also uses the approved
acquisition program baseline to develop a 5-year budget-planning
document, called the Capital Investment Plan. Program managers rely on
the acquisition program baseline to oversee the program’s progress and to
ensure that no action is taken that would breach the approved baseline. To
support their oversight, program managers typically require contractors to
deliver cost performance reports, each of which includes a performance
measurement baseline for assessing the contractor’s progress in meeting
the contract’s cost, schedule, and technical performance goals. For STARS,
FAA has one contract with Raytheon, which accounts for 82 percent of the
funding approved in the acquisition program baseline. Most of the
remaining funding is used for contracts that support FAA’s internal program
management.

For guidance in managing its major acquisition programs, FAA relies
largely on two documents—its own Acquisition System Toolset, a “one-stop
acquisition information system” on FAA’s Web site that contains the
agency’s official acquisition policy and guidance, and the Defense Contract
Management Agency’s (DCMA) guide on implementing earned value
management.9 The Acquisition System Toolset includes policy, guidance,
instructions, examples, best practices, lessons learned, references, and
other related information tailored to each type of procurement contract.
The earned value management guide provides information on how to use
cost and performance measurement to manage acquisition programs.




9
 Defense Contract Management Agency, Earned Value Management Implementation Guide
(October 1997).




Page 6                                         GAO-03-343 National Airspace System
FAA funds STARS primarily through two of its budget accounts: (1)
facilities and equipment and (2) operations. The facilities and equipment
account covers the costs to develop, procure, and place the new equipment
in operation.10 After the equipment has been fully operating for at least a
year, the funding source shifts to the operations account, which covers the
costs to support and maintain the equipment over its life cycle. Operations
costs are, in large part, personnel costs—for the controllers who operate
and the technicians who maintain the equipment. Planned product
improvements and technology upgrades are primarily funded from the
facilities and equipment account.

Since 1996, when FAA initiated STARS, it has spent approximately $1.2
billion, or about 86 percent of the funding budgeted for the program, and it
has twice approved major changes to the program's cost and schedule
estimates. First, in October 1999, FAA modified its acquisition approach
(from off-the-shelf software only to a combination of customized and off-
the-shelf software) and increased to 188 the number of facilities scheduled
to receive STARS. We reported on these changes in September 2002.11 At
that time, FAA also concluded that it did not have adequate funding to
deploy STARS to all 188 facilities with the remainder of the STARS funding.
Instead, FAA decided to deploy STARS to 74 terminal and support facilities.
The selected facilities had frequent equipment failures, were new, or had
the digital radar needed to operate STARS. FAA has since reduced the total
number of facilities to about 170. The agency is currently studying options
for modernizing display systems at the nearly 100 remaining facilities and is
identifying the additional costs of upgrading the other facilities with STARS
or an alternative system. The STARS program office stated that these
additional costs, together with the funds already committed to STARS,
would more than likely exceed the $1.4 billion originally planned. The
program office is in the process of developing options and estimates for
these additional sites.




10
 This account funds security activities related mainly to the security of FAA facilities and
equipment. Funding for some airport security comes from another FAA account—Grants-in-
Aid for Airports.
11
     GAO-02-1071.




Page 7                                                GAO-03-343 National Airspace System
Reliability of FAA’s         FAA has estimated the life-cycle cost for STARS, including the costs to
                             develop and deploy systems to 74 terminal and support facilities and to
Life-Cycle Cost              operate, maintain, and upgrade the systems. However, the reliability of
Estimate for STARS Is        these cost estimates is uncertain. If FAA’s estimates are not reliable, both
                             the agency and the Congress will be limited in their ability to project and
Uncertain                    compare the costs and benefits of completing STARS and modernizing
                             other facilities as well as in their ability to budget realistically for other
                             capital investments.



FAA Has Estimated STARS’s    According to FAA’s estimates, the agency will need about $2.54 billion for
Remaining Life-Cycle Costs   STARS over the remaining life of the program. As indicated in figure 2, FAA
                             will need about (1) $153 million for fiscal years 2004 through 2008 to
                             complete the development and deployment of 74 systems; (2) $1.46 billion
                             for fiscal years 2004 through 2030—or about $54 million a year, on
                             average—to operate and maintain the systems;12 and (3) $930 million to
                             upgrade STARS technology over the same period. FAA’s operating cost
                             estimate assumes that STARS’s operating costs will grow with deployment
                             through about fiscal year 2007 and will subsequently keep pace with
                             inflation, at least through fiscal year 2009. FAA expects the costs of
                             technology upgrades to more than triple between fiscal years 2007 and 2008
                             because of a technology update cycle, which will peak when the hardware
                             and software are due for replacement. This projection assumes that STARS
                             software will be upgraded every 3 years.




                             12
                              As previously noted, FAA officials said the $1.46 billion would be labor costs that the
                             agency would incur for STARS or a similar automation system.




                             Page 8                                                 GAO-03-343 National Airspace System
Figure 2: Life-Cycle Cost Estimate for STARS, Fiscal Years 2004-2030
Dollars in millions

                                                                           Fiscal year
Cost
element               Appropriation     2004    2005        2006       2007        2008       2009            2010 – 2030            Total

Development           Facilities and    $39.5   $25.5      $12.5        $2.0       $1.1        $0.0                $0.0                $80.6
                      equipment

Deployment            Facilities and     54.6    18.0         0.0        0.0        0.0         0.0                 0.0                 72.6
                      equipment

Subtotal                               $94.1    $43.5     $12.5        $2.0       $1.1        $0.0                $0.0              $153.2

Technology            Facilities and    $19.9   $13.6      $12.6       $16.5      $58.8      $65.7              $742.5                $929.5
upgrades              equipment

Operations and        Operations         22.3    41.9       47.3        51.1       53.9        55.7             1,187.5              1,459.7
maintenance

Total                                  $136.3   $99.0      $72.4       $69.6     $113.8     $121.4            $1,930.0              $2,542.4

Source: FAA.


                                                Note: These data are from FAA’s March 2002 acquisition program baseline for STARS and have been
                                                adjusted to reflect the effects of inflation. Numbers may not add because of rounding.


                                                FAA does not yet know to what extent its estimate of STARS’s remaining
                                                development costs is reliable. The estimate is derived from the contractor’s
                                                proposals and projections, which reflect the costs of major modifications
                                                to the contract made since May 2000. However, FAA has not yet
                                                independently analyzed and validated the proposals and projections and
                                                therefore cannot assess the reliability of the development cost estimate.

                                                FAA obtains monthly cost performance reports from the contractor, which
                                                the agency should be able to use both to oversee the contractor’s
                                                performance and to estimate the program’s remaining development costs.
                                                However, FAA does not use these reports because they are not current.
                                                More specifically, their central component, the performance management
                                                baseline—which establishes performance, cost, and schedule milestones
                                                for the contract—has not been updated since May 2000 and therefore does
                                                not incorporate the effects of major contract modifications approved since
                                                that date. For example, the September 2002 cost performance report does
                                                not reflect FAA’s March 2002 reduction in the scope of STARS from 188 to
                                                74 systems, and the report does not include the costs of new work that FAA


                                                Page 9                                                  GAO-03-343 National Airspace System
                             authorized between May 2000 and September 2002. Consequently, the
                             report indicates that STARS is on schedule and within 1 percent of budget,
                             even though, compared with the program envisioned in May 2000, FAA is
                             now under contract to modernize fewer than half as many facilities at more
                             than twice the cost per facility.



Cost Performance Reports     The cost performance reports for STARS do not meet the criteria for such
Do Not Meet FAA’s Criteria   reports established in the guidance for managing major acquisitions that
                             FAA has adopted from DCMA.13 According to this guidance, cost
                             performance reports are valid only when

                             • a reliable performance measurement baseline is established and
                               maintained (i.e., regularly updated and validated),

                             • changes to the baseline are carefully controlled (i.e., negotiated and
                               approved before being authorized),

                             • an integrated baseline review14 takes place within 6 months of a
                               contract’s award or significant modification, and

                             • contract oversight occurs regularly.

                             FAA and the contractor have established a performance measurement
                             baseline for the STARS contract, but they have not satisfied the remaining
                             criteria. The baseline has not been updated since May 2000, and FAA has
                             not validated it. For example, FAA has not asked an independent
                             organization, such as the Defense Contract Audit Agency, to verify the
                             costs in the contractor’s cost performance reports by tracing the costs back
                             to the contractor’s accounting system. In addition, FAA has not controlled
                             changes to the baseline because, between May 2000 and September 2002, it
                             approved up to $179 million in authorized, unpriced work—that is,
                             additional work that FAA agreed to let the contractor perform without first
                             negotiating or independently verifying the costs. Furthermore, the
                             additional tasks and costs have not been incorporated in the baseline, even

                             13
                                  Earned Value Management Guide (October 1997).
                             14
                              The purpose of this review, conducted jointly by the agency and the contractor, is to
                             understand and assess the adequacy, accuracy, and risks of a performance measurement
                             baseline with respect to the contract’s work scope, schedule, technical requirements, and
                             resource availability.




                             Page 10                                              GAO-03-343 National Airspace System
though the DCMA guidance calls for processing contract modifications
expeditiously and incorporating them in a timely manner to maintain the
baseline’s integrity. Although the DCMA guidance does not define “a timely
manner,” experts generally agree that this term means no longer than 3
months. FAA has not maintained and controlled the baseline because,
according to program officials, it has been “schedule driven”—committed
to deploying STARS at the Philadelphia terminal by November 17, 2002.
FAA is currently analyzing the contractor’s cost data and working with the
contractor to incorporate modifications in the performance measurement
baseline.

Although the DCMA guidance calls for performing an integrated baseline
review within 6 months of awarding or significantly modifying a contract,
FAA has not performed such a review of STARS since August 2000, even
though it has subsequently made two major modifications to the contract.
An integrated baseline review is important to ensure that the contract’s
cost data are aligned with the current status of the program after a major
contract modification. According to the Manager of Terminal Automation,
who oversees STARS, FAA had planned to initiate an integrated baseline
review of the STARS contract in November 2002 and expects to begin this
effort as soon as funding is available.

To provide regular contract oversight as the DCMA guidance requires, FAA
and Raytheon meet monthly to, among other things, discuss the cost
performance reports. However, discussions of these reports do not, in our
view, constitute regular contract oversight because, without a current, valid
performance measurement baseline, FAA cannot compare what the
contractor has done with what the contractor agreed to do in the contract.
With a current, valid baseline, the reports would indicate when cost or
schedule thresholds had been exceeded, and FAA could then require the
contractor to explain the reasons for the variances and to identify and take
appropriate corrective actions. But because the baseline has not been
maintained and is not aligned with the current status of the program, the
reports are not useful for evaluating the contractor’s performance or for
projecting the contract’s remaining costs. The Manager of Terminal
Automation agreed that the current cost performance reports are not
useful for these purposes and said that the agency therefore uses the
monthly meetings with the contractor to discuss other program control and
financial issues.




Page 11                                      GAO-03-343 National Airspace System
FAA’s Cost Estimates Do      FAA has limited information for determining the reliability of the costs it
Not Reflect Levels of Risk   has estimated to operate and maintain STARS and to upgrade its
                             technology. The agency first projected the costs of operating and
                             maintaining STARS in March 2002 and said that if these costs were not
                             funded, essential STARS maintenance would not be completed. However,
                             FAA has just begun to operate STARS and has limited experience for
                             projecting future operating and maintenance costs. In addition, FAA did not
                             perform a risk assessment, as its Acquisition System Toolset guidance
                             specifies, to identify the minimum, most likely, and maximum expected
                             costs for the entire program. Although FAA performed such an assessment
                             for one facility, it did not extend this effort—again because of time
                             pressures, according to FAA officials. Because FAA has just begun to
                             operate STARS, it also has limited experience for projecting the costs of
                             technology upgrades over the life of the program. Additionally, both the
                             operating and the technology upgrade cost estimates are uncertain because
                             they extend many years into the future.

                             Despite the uncertainty inherent in estimates—especially long-range
                             ones—FAA has expressed its cost estimates for STARS as point values. As
                             we reported in 1997,15 point values imply certainty and therefore are not
                             suitable for expressing estimates. Instead, ranges or numbers with
                             confidence levels would be more appropriate. For instance, a cost estimate
                             of $1 million could be presented either as a range of $900,000 to $1.1 million
                             or as $1 million with a confidence interval of 90 percent, indicating that
                             there is a 10 percent chance that the cost will exceed the estimate.16
                             Because FAA did not perform a risk assessment for the entire STARS
                             program, it did not develop ranges or confidence levels that it could use to
                             express its cost estimates for STARS. And because FAA used point values
                             instead, it limited its disclosure of the program’s investment risks.

                             The reliability of FAA’s development and operating cost estimates is
                             important not only for managing STARS effectively but also for planning
                             appropriately for other terminal modernization efforts and other FAA
                             capital investments. Without reliable estimates for comparing STARS’s


                             15
                               U.S. General Accounting Office, Air Traffic Control: Improved Cost Information Needed
                             to Make Billion Dollar Modernization Investment Decisions, GAO/AIMD-97-20
                             (Washington, D.C.: Jan. 22, 1997).
                             16
                              The 90 percent confidence level is based on an analysis of the estimate’s uncertainty made
                             by using a Monte Carlo model.




                             Page 12                                              GAO-03-343 National Airspace System
                      costs and benefits and for anticipating future expenditures, FAA and the
                      Congress cannot make informed decisions about how best to modernize
                      the nearly 100 facilities that are not currently scheduled to receive STARS.
                      Furthermore, because FAA has given high priority to completing STARS
                      and has said that it will fund the program even if it has to postpone the
                      funding for other programs, FAA’s estimates for STARS will influence the
                      agency’s plans for funding other capital investments.



Impact of STARS’s     According to our analysis of budgetary data for fiscal years 2004 through
                      2007 that we obtained from FAA, the estimated costs of STARS should have
Estimated Costs on    a declining impact on FAA’s budgets during these years because the
FAA’s Budgets Is      program’s development phase is nearly over and its operations will still be
                      limited. For later years, the impact of these estimated costs on FAA’s
Expected to Decline   budgets is unknown because it is too soon in the budget cycle for FAA to
through Fiscal Year   have developed detailed budgets beyond fiscal year 2007.
2007
                      Because FAA has nearly completed the development of STARS and has
                      begun to install the equipment, the agency is budgeting less from the
                      facilities and equipment account for STARS. Therefore, the proposed
                      funding for STARS does not have a significant impact on the funding for
                      other efforts to improve the safety, security, and capacity of the national
                      airspace system. For fiscal year 2004, the $94.1 million planned to develop
                      and deploy STARS represents 3 percent of the total funding for facilities
                      and equipment proposed in FAA’s Capital Investment Plan. When the $19.9
                      million planned for technology upgrades is included in the projections for
                      fiscal year 2004, the estimated cost for STARS is about 4 percent of FAA’s
                      planned total facilities and equipment budget (see fig. 3).




                      Page 13                                      GAO-03-343 National Airspace System
Figure 3: Allocation of Facilities and Equipment Funding for Fiscal Year 2004 in
FAA’s Capital Investment Plan

 FY04 Investments at $3.062 Billion Level
                                                                                  Efficiency
                                                                                  3%
                                                                                  Security
                                                                                  4%
                                                                                  STARS

                                                                                  Safety
                                                            9%
                                  29%                                             Capacity
                                                               11%

                                                                                  Mission support
                            16%                              13%
                                               15%


                                                                                  Pay and benefits
                                                                                  Reliability
     Proposed funding for other safety, security, and capacity efforts.
     Proposed funding for STARS.

Sources: GAO and FAA.


Notes:
GAO analysis of data from FAA.
Beginning with the fiscal year 2003 budget, FAA showed its facilities and equipment budget in broad
performance (mission) outcome areas. Individual programs support these broad performance areas.
For example, STARS supports several mission areas, such as safety, security, and capacity. To
illustrate the impact of STARS on the other broad mission areas, this figure consolidates the allocation
for STARS in one percentage and compares this percentage with the allocations to the broad mission
areas.


According to FAA’s Capital Investment Plan, the proposed funding for FAA’s
total facilities and equipment budget will increase slightly during fiscal
years 2005 through 2007, while the proposed funding for STARS’s
development and deployment will continue to decline (see table 1).
Therefore, the impact of these estimated STARS costs on the facilities and
equipment budget for safety, security, and capacity would be even less
during these fiscal years. The estimated costs of technology upgrades, if



Page 14                                                                   GAO-03-343 National Airspace System
included in the facilities and equipment cost estimates for STARS for these
3 fiscal years, would likewise have a similarly small impact on the facilities
and equipment budget for these fiscal years.



Table 1: Proposed Funding for FAA’s Total Facilities and Equipment Budget and for
STARS’s Development and Deployment

Dollars in millions
Proposed funding               Fiscal year 2005         Fiscal year 2006          Fiscal year 2007
Total facilities and
equipment budget                           $3,129                  $3, 202                   $3, 277
STARS development
and deployment                               43.5                      12.5                          2.0
STARS technology
upgrades                                     13.6                      12.6                     16.5
Source: FAA.

Note: These data are from the Capital Investment Plan that FAA submitted with its fiscal year 2003
budget request and FAA’s March 2002 acquisition program baseline for STARS.


As FAA begins to operate STARS equipment at more terminal facilities, the
primary funding source for the program will shift from the facilities and
equipment budget account to the operations budget account. Although FAA
has projected that it will increasingly need funding from (1) the operations
account to cover the costs of operating and maintaining STARS and (2) the
facilities and equipment account to upgrade STARS technology, it has not
yet developed detailed estimates of the program’s operations and
maintenance and technology upgrade costs beyond fiscal year 2007. FAA
officials noted that FAA, like most federal agencies, develops its budget in
5-year plans and has not yet begun to develop detailed budgets beyond
fiscal year 2007. Consequently, FAA does not currently know what impact
the estimated costs of operating, maintaining, and upgrading STARS will
have on the agency’s future budgets. Furthermore, as we previously noted,
FAA has limited operational experience with STARS to use in estimating
and budgeting for the costs of the program’s operations, maintenance, and
technology upgrades. Finally, the uncertainty of long-range estimates
makes it difficult to determine the impact of STARS’s estimated costs on
FAA’s budgets for outlying fiscal years.




Page 15                                                    GAO-03-343 National Airspace System
After Deploying STARS   FAA is committed to deploying STARS at the 74 terminal and support
                        facilities included in the STARS program, but for the nearly 100 other
at 74 Facilities, FAA   facilities that remain to be modernized, the agency has at least three
May Modify Its          options: It could deploy Raytheon’s STARS, it could procure Lockheed
                        Martin Corporation’s Common Automated Radar Terminal System
Approach for the        (Common ARTS) technology, or it could combine Raytheon’s STARS
Remaining Facilities    display system with Lockheed Martin’s Common ARTS processing system.
                        Between 1997 and 1999, FAA deployed Common ARTS to 131 small to
                        medium-sized facilities and to 5 larger facilities. From its experience with
                        both systems, FAA knows that each works independently. However, by
                        combining the two technologies, FAA could both (1) take advantage of the
                        customized software developed to resolve complex computer-human
                        (controller and technician) interface issues that accounted for a significant
                        portion of STARS’s development costs and (2) continue to use portions of
                        the Common ARTS equipment that it recently deployed.

                        In October 2002, FAA asked Raytheon to consider the feasibility of merging
                        the STARS display system with the Common ARTS processing system.
                        According to the Manager of Terminal Automation, Raytheon finished
                        negotiating a subcontract with Lockheed Martin in January 2003, and FAA
                        anticipates that the two contractors will now begin analyzing the feasibility
                        of a merger of components of the two systems. After FAA receives the
                        results of this analysis, which it expects in April 2003, it plans to determine
                        the cost and schedule for modernizing the remaining terminal facilities.



Conclusions             FAA lacks accurate, valid, current data on the STARS program’s costs and
                        progress. Without such data, FAA is limited in its ability to effectively
                        oversee the contractor’s performance and reliably estimate future costs.
                        FAA cannot use the contractor’s cost performance reports for these
                        purposes until the contract’s performance measurement baseline has been
                        revised to incorporate the results of contract modifications, and FAA has
                        verified that the revised baseline is aligned with the current status of the
                        contract. Furthermore, without performing a risk assessment to identify
                        the program’s minimum, most likely, and maximum expected costs, FAA
                        cannot reliably determine the level of uncertainty inherent in its cost
                        estimates. Finally, by using point values, rather than ranges or other
                        appropriate measures, to express its cost estimates, FAA is not revealing
                        the extent of their uncertainty. Its current estimates, expressed as point
                        values, are misleading because they convey undue certainty and limit




                        Page 16                                       GAO-03-343 National Airspace System
                      disclosure of the program’s investment risks. As such, the estimates’
                      usefulness to program managers and congressional overseers is limited.

                      The earned value management guidance that FAA has adopted from DCMA
                      and the Acquisition System Toolset that it compiled for itself establish clear
                      procurement management policies and procedures that are applicable to
                      STARS and to subsequent terminal modernization programs. In light of
                      FAA’s incomplete or inconsistent application of this guidance to the STARS
                      program thus far and the resulting cost overruns and schedule delays, we
                      believe it is essential that the agency revisit the guidance, not only when it
                      updates its baseline and performs an integrated baseline review this spring,
                      but also throughout the remainder of STARS and throughout subsequent
                      terminal modernization programs.



Recommendations for   To improve FAA’s management of STARS and of subsequent terminal
                      modernization programs and to provide the Congress with more reliable
Executive Action      information for overseeing these programs, we recommend that the
                      Secretary of Transportation direct the FAA Administrator to follow the
                      agency’s guidance for managing major acquisition systems by

                      • establishing, maintaining, and controlling an accurate, valid, and current
                        performance measurement baseline, which would include negotiating
                        all authorized, unpriced work within 3 months;

                      • conducting an integrated baseline review of any major contract
                        modifications within 6 months; and

                      • preparing a rigorous life-cycle cost estimate, including a risk
                        assessment, in accordance with the Acquisition System Toolset’s
                        guidance and identifying the level of uncertainty inherent in the
                        estimate.



Agency Comments       We requested comments on a draft of this report from the Secretary of
                      Transportation or his designee. On January 23, 2003, FAA officials,
                      including the Manager of Terminal Automation, provided us with the
                      following oral comments on the draft. FAA agreed with the general tone
                      and intent of the draft report as well as with our conclusions and
                      recommendations. FAA noted that, in focusing its time and energy on
                      meeting the technical performance and schedule requirements of STARS, it



                      Page 17                                       GAO-03-343 National Airspace System
              spent less time and energy on its business management of the program.
              However, the manager noted, the agency is now taking steps to reduce the
              program’s cost risks, such as defining and negotiating with the prime
              contractor all of the contract work planned for fiscal years 2003 through
              2005. FAA also plans to conduct an integrated baseline review when it
              receives its appropriation for fiscal year 2003. According to FAA, our
              recommendations will further strengthen FAA’s commitment to improve
              the business management of the STARS program. FAA provided additional
              clarifying and technical information, which we incorporated as
              appropriate.



Scope and     To conduct our work, we reviewed FAA’s 5-year Capital Investment Plan,
              which proposes funding for programs to modernize the national airspace
Methodology   system. We also analyzed data from cost performance reports that the
              STARS contractor developed for FAA. We interviewed FAA officials
              responsible for air traffic control modernization planning and budgeting.
              We also reviewed data from the FAA STARS program, contracting officials,
              and the Department of Transportation’s Office of the Inspector General. We
              did not independently verify the cost and performance data we received
              from FAA. We performed our work from October 2002 through January
              2003 in accordance with generally accepted government auditing
              standards.


              As arranged with your office, unless you publicly announce its contents
              earlier, we plan no further distribution of this report until 5 days after the
              date of this letter. At that time, we will send copies to interested Members
              of Congress, the Secretary of Transportation, and the FAA Administrator.
              We will also make copies available to others on request. In addition, the
              report will be available at no charge on the GAO Web site at
              http://www.gao.gov.




              Page 18                                       GAO-03-343 National Airspace System
Should you or your staff have questions on matters discussed in this report,
please contact Gerald L. Dillingham at (202) 512-2834 or Keith A. Rhodes at
(202) 512-6412. We can also be reached by E-mail at dillinghamg@gao.gov
and rhodesk@gao.gov, respectively. GAO contacts and key contributors to
this report are listed in appendix I.

Sincerely yours,




Gerald L. Dillingham, Ph.D.
Director, Physical Infrastructure Issues




Keith A. Rhodes
Chief Technologist, Applied Research and
  Methods




Page 19                                     GAO-03-343 National Airspace System
Appendix I

GAO Contacts and Staff Acknowledgments                                                         AA
                                                                                                ppp
                                                                                                  ep
                                                                                                   ned
                                                                                                     n
                                                                                                     x
                                                                                                     id
                                                                                                      e
                                                                                                      x
                                                                                                      Iis




GAO Contacts      Belva Martin (202) 512-2834
                  Madhav Panwar (202) 512-6228



Staff             In addition to those individuals named above, Yvette Banks, Geraldine
                  Beard, Jennifer Echard, Elizabeth Eisenstadt, Elizabeth Marchak, and
Acknowledgments   Karen Richey made key contributions to this report.




(540041)          Page 20                                    GAO-03-343 National Airspace System
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300
Address Service Requested