oversight

Air Traffic Control Modernization: Management Challenges Associated with Program Costs and Schedules Could Hinder NextGen Implementation

Published by the Government Accountability Office on 2012-02-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Committees




February 2012
                AIR TRAFFIC
                CONTROL
                MODERNIZATION
                Management
                Challenges Associated
                with Program Costs
                and Schedules Could
                Hinder NextGen
                Implementation




GAO-12-223
                                           February 2012

                                           AIR TRAFFIC CONTROL MODERNIZATION
                                           Management Challenges Associated with Program
                                           Costs and Schedules Could Hinder NextGen
                                           Implementation
Highlights of GAO-12-223, a report to
congressional committees




Why GAO Did This Study                     What GAO Found
The Federal Aviation Administration        In a review of 30 major ATC acquisition programs, all of which will contribute to
(FAA), partnering with other federal       the transition to NextGen, GAO found that costs for 11 of the 30 programs have
agencies and the aviation industry, is     increased from their initial estimates by a total of $4.2 billion and 15 programs
implementing the Next Generation Air       experienced delays. The 11 acquisitions that experienced cost increases account
Transportation System (NextGen), a         for over 60 percent of FAA’s total acquisition costs ($11 billion of $17.7 billion) for
new satellite-based air traffic            the 30 programs. The 15 acquisitions that experienced schedule delays, of which
management system that will replace        10 also had cost increases, ranged from 2 months to more than 14 years and
the current radar-based system and is      averaged 48 months.
expected to enhance the safety and
capacity of the air transport system by    Cost increases and schedule delays occurred due to several factors, many of
2025. Concurrently, FAA continues to       which have been longstanding challenges for FAA. Specifically, these have
maintain and upgrade existing air          involved (1) additional or unanticipated system requirements; (2) insufficient
traffic control (ATC) systems that will    stakeholder involvement (such as controllers’ input) throughout system
also be needed for NextGen. This           development; (3) underestimating the complexity of software development; and
involves acquiring and implementing        (4) unanticipated events including funding shortfalls or work stoppages. These
new software and hardware.                 challenges, if they persist, will impede the implementation of NextGen, especially
                                           in light of the interdependencies among many acquisition programs, where cost
GAO was asked to determine (1) how,        increases or delays in one program can affect the costs and schedules of other
if at all, costs and schedules of FAA      programs.
ATC acquisitions programs, including
those related to NextGen, have             For the four programs GAO selected to analyze in depth, FAA is not consistently
changed since they were first              following the characteristics of high-quality cost estimates and scheduling best
submitted to Congress, (2) the reasons     practices that GAO previously identified. Regarding cost estimates, GAO found
for any such changes, and (3) the          that although all four of the programs generally provided well documented and
extent that selected ATC programs          comprehensive estimates, which are two of the four characteristics, no program
adhere to cost and schedule best           fully met the two other characteristics. Specifically, each program estimate was
practices. To do its work, GAO             not credible because each lacked an independent cost estimate, which provides
reviewed 30 programs and conducted         a check against FAA’s estimate and three programs lacked risk or uncertainty
cost and schedule analysis on four         analysis. The estimates also lacked accuracy because they were not updated
programs that had an approved              regularly or based on comparable programs. Regarding scheduling practices,
baseline and were NextGen related.         most programs did not substantially or fully meet the majority of the 9 best
GAO reviewed acquisition documents         practices GAO previously identified including developing a fully integrated master
and interviewed FAA officials.             schedule of all program activities and performing a schedule risk analysis. For
                                           example, without a schedule risk analysis, FAA is unable to predict, with any
What GAO Recommends                        degree of confidence, if the estimated completion dates are realistic. FAA is
To better estimate the cost and            implementing new processes and organizational changes to better manage
completion dates for major                 acquisitions. However, by not consistently following the characteristics of high-
acquisitions, FAA should, among other      quality cost estimate and scheduling best practices, FAA cannot provide
things, require cost and schedule risk     reasonable assurance to Congress and other stakeholders that NextGen and
analysis, independent cost estimates       other ATC programs will avoid additional cost increases or schedule delays.
and integrated master schedules. FAA
did not comment on whether or not it
agreed with the recommendations.



View GAO-12-223 or key components.
 For more information, contact Gerald L.
Dillingham, Ph.D., at
(202) 512-2834 or dillinghamg@gao.gov .
                                                                                     United States Government Accountability Office
Contents


Letter                                                                                         1
               Background                                                                     4
               Most Ongoing ATC Programs Remain within Budget and on
                 Schedule, but a Few Have Significantly Exceeded Initial
                 Estimates                                                                   12
               Schedules for Roughly Half of ATC Programs Are on Track, but
                 Half of ATC Programs Are Delayed                                            15
               Several Factors Contributed to Cost Increases and Schedule
                 Delays, and Some Could Hamper NextGen Implementation                        17
               For Programs We Reviewed, FAA Did Not Consistently Meet All
                 Characteristics of Quality Cost Estimates or Schedule Best
                 Practices                                                                   24
               Conclusions                                                                   39
               Recommendations for Executive Action                                          41
               Agency Comments                                                               42

Appendix I     Objectives, Scope and Methodology                                             45



Appendix II    Detailed Purpose and Status for 30 Acquisitions GAO Reviewed                  52



Appendix III   Assessments of Four FAA Program Cost Estimates                                82



Appendix IV    Assessments of Four FAA Program Schedules                                    102



Appendix V     GAO Contact and Staff Acknowledgments                                        142



Tables
               Table 1: FAA Acquisition Life Cycle                                            6
               Table 2: Characteristics of High-Quality Cost Estimates and Steps
                        Related to Each Characteristic                                       10
               Table 3: Eleven ATC Programs That Have Experienced Cost
                        Increases as of August 2011                                          14




               Page i                                GAO-12-223 Air Traffic Control Modernization
          Table 4: Fifteen ATC Programs That Have Experienced Delays
                   Based on Original Scheduled Completion Date                         16
          Table 5: Key Factors Contributing to Cost Growth, Schedule
                   Delays, or Both for ATC Systems                                     18
          Table 6: GAO Analysis of the Extent FAA Acquisition Cost
                   Estimates Met the Characteristics of High-Quality and
                   Reliable Cost Estimates                                             25
          Table 7: Extent Program Schedules Met Best Practices                         29
          Table 8: GAO’s Analysis of the FAA’s ADS-B Cost Estimates                    82
          Table 9: GAO’s Analysis of FAA’s CATMT Cost Estimates                        87
          Table 10: GAO’s Analysis of FAA’s SWIM Cost Estimates                        92
          Table 11: GAO’s Analysis of FAA’s WAAS Cost Estimates                        97
          Table 12: Extent to Which the ADS-B Schedule Met Best Practices             102
          Table 13: Extent to Which the CATMT Schedule Met Best Practices             109
          Table 14: Extent to Which the SWIM Schedule Met Best Practices              115
          Table 15: Extent to Which the FAA-Prepared WAAS Schedule Met
                   Best Practices                                                     123
          Table 16: Extent to Which the WAAS Contractor-Prepared
                   Schedule Met Best Practices                                        131
          Table 17: Probability of Project Completion                                 136
          Table 18: Prioritized Risks at the 80th Percentile                          141


Figures
          Figure 1: Acquisition Programs Reported by FAA in the CIP for
                   2012 through 2016                                                    5
          Figure 2: Improvements to Phases of Flight Expected under
                   NextGen                                                              8
          Figure 3: Automatic Dependent Surveillance-Broadcast (ADS-B)                 52
          Figure 4: Advanced Technologies and Oceanic Procedures (ATOP)                53
          Figure 5: Airport Surveillance Radar (ASR) Model 11 Tech Refresh
                   Segment 1                                                           54
          Figure 6: Aviation Surface Weather Observation Network
                   (ASWON) and Automated Surface Observing System
                   (ASOS) P31                                                          55
          Figure 7: Air Traffic Control Radar Beacon Interrogator (ATCBI-6)            56
          Figure 8: Collaborative Air Traffic Management Technologies
                   (CATMT), Work Packages 2 and 3 (WP2 and WP3)                        57
          Figure 9: En Route Communication Gateway (ECG) Tech Refresh                  58
          Figure 10: En Route Automation Modernization (ERAM)                          59
          Figure 11: Integrated Display Systems (IDS) Replacement                      60
          Figure 12: International Flight Inspection Aircraft (IFIA)                   61


          Page ii                              GAO-12-223 Air Traffic Control Modernization
Figure 13: Instrument Flight Procedure Automation (IFPA)                     62
Figure 14: Integrated Terminal Weather System (ITWS)                         63
Figure 15: Next Generation Air/Ground Communication System
         (NEXCOM) Segment 1a                                                 64
Figure16: Next Generation Weather Radar (NEXRAD) Dual
         Polarization                                                        65
Figure 17: Power Systems Sustained Support (PS3)                             66
Figure 18: Regulation and Certification Infrastructure for System
         Safety (RCISS) Segment 2                                            67
Figure 19: Runway Status Lights (RWSL)                                       68
Figure 20: Standard Terminal Automation Replacement System
         (STARS) (TAMR Phase 1)                                              69
Figure 21: System Wide Information Management (SWIM)
         Segment 1                                                           70
Figure 22: Terminal Automation Modernization and Replacement
         (TAMR)—Phase 2                                                      71
Figure 23: Tower Training Simulator Systems                                  72
Figure 24: Trajectory Management—Arrival Tactical Flow Time
         Based Flow Management                                               73
Figure 25: Terminal Doppler Weather Radar (TDWR) Service Life
         Extension Program (SLEP)                                            74
Figure 26: Traffic Flow Management System (TFMS) Tech Refresh                75
Figure 27: Ultra High Frequency (UHF) Radio Replacement                      76
Figure 28: Next Generation Voice Recorder Replacement Program
         (VRRP)                                                              77
Figure 29: Voice Switching and Control Switching System (VSCS)
         Tech Refresh Phase 2                                                78
Figure 30: Wide Area Augmentation System (WAAS)                              79
Figure 31: Weather and Radar Processor (WARP) Sustain                        80
Figure 32: Weather Camera Program (WCP)                                      81




Page iii                             GAO-12-223 Air Traffic Control Modernization
Abbreviations
ABRR            Airborne Reroute Execution
ADS-B           Automatic Dependent Surveillance-Broadcast
AirNav          Airports and Navigations Aids database
AMS             Acquisition Management System
APB             acquisition planning baseline
APTS            Automated Procedures Tracking System
ASDP            Advanced Signal Data Processor
ASOS            Automated Surface Observing System
ASR             Airport Surveillance Radar
ASWON           Aviation Surface Weather Observation Network
ATC             air traffic control
ATCBI-6         Air Traffic Control Radar Beacon Interrogator
ATCSCC          Air Traffic Control System Command Center
ATO             Air Traffic Organization
ATOP            Advanced Technologies and Oceanic Procedures
AUM             Arrival Uncertainty Management
CACR            Collaborative Airspace Constraint Resolution
CATMT           Collaborative Air Traffic Management Technologies
CIP             Capital Investment Plan
Cost Guide      GAO Cost Estimating and Assessment Guide
DataComm        Data Communications
DOD             Department of Defense
EBUS            Enhanced Back-up Surveillance
ECG             En Route Communication Gateway
ERAM            En Route Automation Modernization
ERIDS           En Route Information Display System
FAA             Federal Aviation Administration
F&E             facilities and equipment
I2I             Idea to In-Service Management
IDA             Investment Decision Authority
IDS             Integrated Display Systems
IFIA            International Flight Inspection Aircraft
IFP             Instrument Flight Procedures
IFPA            Instrument Flight Procedure Automation



Page iv                             GAO-12-223 Air Traffic Control Modernization
IP&A               Investment Planning and Analysis
IPDS               Instrument Procedure Development System
IT                 information technology
ITWS               Integrated Terminal Weather System
JRC                Joint Resources Council
NAS                National Airspace System
NextGen            Next Generation Air Transportation System
NEXCOM             Next Generation Air/Ground Communication System
NEXRAD             Next Generation Weather Radar
OE                 Obstacle Evaluation
OMB                Office of Management and Budget
P3I                Pre-Planned Product Improvements
PAMRI              Peripheral Adapter Module Replacement Item
PS3                Power Systems Sustained Support
RCISS              Regulation and Certification Infrastructure for System
                   Safety
RWSL               Runway Status Lights
SIP                SWIM implementation program
SLEP               Service Life Extension Program
SRA                schedule risk analysis
STARS              Standard Terminal Area Replacement System
SWIM               System Wide Information Management
TAMR               Terminal Automation Modernization and Replacement
TDWR               Terminal Doppler Weather Radar
Tech Center        FAA Technical Center
TERPS              Terminal Instrument Procedures
TFM                Traffic Flow Management
TFMS               Traffic Flow Management System
TRACON             Terminal Radar Approach Control
UHF                Ultra High Frequency
VRRP               Next Generation Voice Recorder Replacement Program
VSCS               Voice Switching and Control Switching System
WAAS               Wide Area Augmentation System
WAM                Colorado Wide Area Multilateration
WARP               Weather and Radar Processor
WCP                Weather Camera Program
WP2                Work Package 2
WP3                Work Package 3


This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page v                                        GAO-12-223 Air Traffic Control Modernization
United States Government Accountability Office
Washington, DC 20548




                                   February 16, 2012

                                   Congressional Committees

                                   To accommodate anticipated increases in air passenger traffic over time, 1
                                   the Federal Aviation Administration (FAA) has expanded its acquisitions
                                   program to sustain current—or legacy—air traffic control (ATC) facilities
                                   and systems while simultaneously replacing or supplementing those
                                   systems through transition to the satellite-based Next Generation Air
                                   Transportation System (NextGen). This modernization effort involves
                                   acquiring and implementing new, advanced air traffic management
                                   systems, including hardware and software, to dramatically change the
                                   way the current aviation system is operated. As the agency transitions to
                                   NextGen, which has significantly increased the number, cost, and
                                   complexity of FAA’s acquisition programs, it is imperative that these
                                   programs remain on time and within budget, particularly given current
                                   budget constraints and the interdependencies of many NextGen-related
                                   ATC programs. FAA has taken several steps to improve its acquisition
                                   management––including implementing a cost estimating methodology, a
                                   cost accounting system, and a business process, and developing an
                                   enterprise architecture––which resulted in the removal of its acquisition
                                   management from the GAO High-Risk list in 2009. 2 However, recent cost
                                   and scheduling problems among some major acquisition programs, such
                                   as the En Route Automation Modernization (ERAM), which is integral to
                                   ATC modernization, have renewed concerns about the agency’s ability to
                                   manage complex multibillion-dollar procurement programs.

                                   In response to your request, this report (1) describes how, if at all, the
                                   planned costs and schedules of current FAA ATC acquisition programs,
                                   including those related to NextGen, have changed since they were first
                                   submitted to Congress, (2) examines the reasons for any changes in
                                   planned costs and schedules, and (3) assesses the extent to which




                                   1
                                    The nation’s air traffic control system handles almost 30 million flights per year. FAA
                                   predicts that by 2025, the number of passengers will increase from about 700 million to
                                   about 1.1 billion per year, and the number of flights will increase from about 80,000 to
                                   more than 95,000 every 24 hours.
                                   2
                                       GAO, High Risk Series: An Update, GAO-09-271 (Washington, D.C.: January 2009).




                                   Page 1                                         GAO-12-223 Air Traffic Control Modernization
selected ATC programs adhere to best practices for determining
acquisition costs and schedules.

To determine the changes, if any, in the costs and schedules of FAA’s
ATC programs, we gathered and analyzed agency data on the estimated
costs and schedules of the 30 ATC programs 3 that had baselines—that
is, those programs whose estimated budget and schedule had received
FAA executive approval. 4 Eighteen of these programs are directly related
to the implementation of NextGen, and all are needed to maintain and
modernize the existing ATC system in order for it to operate in the
NextGen environment. We also drew upon past GAO work in which we
undertook detailed reviews of the status of ATC and other acquisition
programs 5 and obtained updated documentation as necessary from FAA.
We interviewed FAA officials to obtain information on their programs’ past
and current challenges and current status and summarized the status of
all 30 ATC programs, including their original and current cost estimates
and completion dates. For each program, we compared its initial
estimated cost at the time of its submission to Congress for approval
against its current cost estimate and compared its planned and actual
schedules.

To examine the reasons contributing to any changes in cost estimates
and schedules in the 30 baselined ATC programs, we interviewed FAA
officials and contractors and reviewed program documentation. We
analyzed information on cost increases and delays for the 30 baselined
programs to determine the factors that contributed to cost and schedule
changes.


3
    We requested the information on the programs in August 2010.
4
 According to FAA, baselined programs are acquisition programs with an agreed-to
description of the attributes and estimated costs at a particular point in time. The baseline
is a formal, management-approved document that serves as a starting point for tracking
any changes in the program as it is developed and implemented for performance
accountability.
5
 GAO, Department of Homeland Security: Assessments of Selected Complex
Acquisitions, GAO-10-588SP, (Washington, D.C.: June 30, 2010); Defense Acquisitions:
Assessments of Selected Weapon Programs, GAO-10-388SP (Washington, D.C.: Mar.
30, 2010); Air Traffic Control: FAA Reports FAA Reports Progress in System Acquisitions,
but Changes in Performance Measurement Could Improve Usefulness of Information,
GAO-08-42 (Washington, D.C.: Dec. 18, 2007); and National Airspace System: FAA Has
Made Progress but Continues to Face Challenges in Acquiring Major Air Traffic Control
Systems, GAO-05-331 (Washington, D.C.: June 10, 2005).




Page 2                                          GAO-12-223 Air Traffic Control Modernization
To assess the extent to which FAA’s cost estimating and scheduling
processes aligned with best practices, we conducted an in-depth review
of 4 of the 30 baselined programs: the Automatic Dependent
Surveillance-Broadcast (ADS-B) system, the Collaborative Air Traffic
Management Technologies (CATMT) system, the System Wide
Information Management (SWIM) system, and the Wide Area
Augmentation System (WAAS). 6 We selected these programs based on
the following criteria: the program had reached sufficient maturity such
that risks could be identified and the program was key to the
modernization of the air traffic control system. We conducted in-depth
interviews with FAA acquisition program managers for the programs
selected. In addition to interviews, we obtained and analyzed the most
current cost and schedule estimates for these programs. To assess the
extent to which FAA’s acquisition practices for these programs were
consistent with best practices, we used the 2009 GAO Cost Estimating
and Assessment Guide (Cost Guide). 7 In assessing each program’s cost
estimate, we used the Cost Guide to evaluate FAA’s estimating
methodologies, assumptions, and results to determine whether the cost
estimate was well-documented, comprehensive, accurate, and credible.
We also determined the extent to which the schedule was prepared in
accordance with best practices that are fundamental to having a reliable
schedule. In addition, we performed a schedule risk analysis on a WAAS
schedule prepared by the contractor to determine the high-priority risks
that the program may encounter that could affect the schedule and the
likelihood of the program finishing on time. We selected the WAAS
contractor schedule because we determined that it was the only schedule
of the four programs we reviewed in detail that was sufficiently reliable for
a risk analysis to be performed.

We conducted this performance audit from August 2010 to February 2012
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and




6
    The four initial cost estimates we reviewed were developed at various points in the past.
7
GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and
Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009).




Page 3                                            GAO-12-223 Air Traffic Control Modernization
             conclusions based on our audit objectives. Appendix I contains more
             detailed information on our objectives, scope, and methodology.



Background   FAA catalogs its acquisition programs in its annually updated Capital
             Investment Plan (CIP). The CIP identifies planned capital investment in
             the National Airspace System (NAS) for the next 5 years consistent with
             the amount requested in the agency’s annual budget submission.
             Appendix C of the CIP, which identifies the anticipated budget line items,
             is divided into five activities: (1) Engineering, Development, Test, and
             Evaluation; (2) ATC Facilities and Equipment; (3) Non-ATC Facilities and
             Equipment; (4) Facilities and Equipment Mission Support; and (5)
             Personnel Compensation, Benefits and Travel. The CIP for fiscal years
             2012 through 2016 contains 106 funded acquisition programs with
             estimated total budgets (through 2016) of more than $14 billion. Of these
             83 acquisition programs FAA considers ATC related, 18 involve
             Engineering, Development, Test and Evaluation and 65 involve ATC
             Facilities and Equipment. The 83 programs include 30 that have had
             program baselines approved by FAA’s Joint Resources Council (JRC), 8
             which is responsible for approving major programs. 9 These 30 baselined
             programs include communications, navigation, and surveillance systems
             that are key to ATC operations. FAA considers 5 of the programs to be
             foundational parts of NextGen, and all are key to modernizing the existing
             ATC system. Figure 1 illustrates the universe of FAA acquisitions for
             fiscal years 2012-2016.




             8
              The JRC is an FAA executive body consisting of associate and assistant administrators,
             acquisition program executives, the Chief Financial Officer, the Chief Information Officer,
             and legal counsel. The JRC makes executive-level decisions, including those that
             determine whether a program meets a mission need and should proceed.
             9
              As defined in Office of Management and Budget Circular A-11, Part 7, major programs
             are assets that require special management attention because of their importance to the
             agency’s mission. These include high development, operating, or maintenance costs; high
             risk; high return; or a significant role in the administration of agency programs, finances,
             property, or other resources.




             Page 4                                         GAO-12-223 Air Traffic Control Modernization
Figure 1: Acquisition Programs Reported by FAA in the CIP for 2012 through 2016




FAA has developed and uses its Acquisition Management System (AMS)
to provide policies and guidance for managing ATC system programs
through all phases of the programs’ life cycles (see table 1). The Air
Traffic Organization (ATO) within FAA is responsible for operating,
maintaining, and modernizing the nation’s current ATC system. 10




10
  FAA is implementing a reorganization of ATO that may alter the current alignment of
program acquisition responsibilities within ATO.




Page 5                                       GAO-12-223 Air Traffic Control Modernization
Table 1: FAA Acquisition Life Cycle

Phase/activity              What occurs during this phase or activity
Needs and solution identification
Mission analysis            FAA identifies a capability shortfall and determines that it
                            needs an investment to better carry out its mission.
                            Recently, FAA began analyzing its mission needs within
                            the context of its overall goals for the NAS.
Investment analysis         FAA, using an investment analysis team, evaluates
                            alternatives, selects practical and affordable solutions, and
                            develops a baseline of cost, schedule, and performance
                            requirements. This document is called the acquisition
                            program baseline.
Solution implementation
System integration          Both hardware and software components and subsystems
                            are integrated into a product. Also, intra- and inter-system
                            compatibility are tested and analyzed.
System demonstration        Tests show that the product can work as required and be
                            manufactured within targets.
System production           All activities are carried out to produce needed quantities.
                            Each end item is tested before it leaves the factory to
                            verify that it conforms to specifications and is free from
                            manufacturing defects.
In-service management       All required activities are carried out, including directly
                            operating, providing maintenance functions (both
                            scheduled and unscheduled), and furnishing technical and
                            logistics requirements.
Decommission                All unneeded assets are decommissioned and removed
                            from service at the end of their service lives.
Source: FAA.



The acquisition program baseline defines the cost, schedule and
performance baselines for the investment program. The JRC which
determines whether to approve a cost and schedule baseline also
approves rebaselining, through which the agency documents and
approves major changes to a program’s previously approved budget or
schedule. Rebaselining resets the estimated costs and schedule used to
determine how the program will be held accountable and can occur
before the program is deployed. Once a program is rebaselined, FAA
reports on the performance of the program based on the revised cost and
schedule. Although the rationale for rebaselining can be reasonable, as,
for example, when a program’s scope has been expanded, reporting a




Page 6                                      GAO-12-223 Air Traffic Control Modernization
program’s performance based on a rebaselined cost or schedule can also
skew or conceal from Congress and other stakeholders the program’s
actual total costs or overall timeline. We previously reported 11 that the
absence of this information on rebaselining in ATO’s performance
reporting could cause managers and other stakeholders, including
Congress, to think that performance was better than it actually was. We
recommended that FAA regularly report on the overall, long-term
performance in acquiring ATC systems by providing in FAA’s annual
Performance and Accountability Report the original budget and schedule
baselines for each rebaselined program and the reasons for the
rebaselining. In response to our recommendation, FAA currently provides
this information in Appendix D of its CIP, where it details baseline cost
and schedule information for major acquisition programs.

NextGen involves changes to every aspect of air transportation (see
fig. 2). NextGen requires the acquisition of new integrated systems (both
software and hardware), flight procedures, aircraft performance
capabilities, and supporting infrastructure to transform the current air
transportation system into one that uses satellite-based surveillance and
navigation operations instead of ground-based radar. 12 These changes
are intended to increase the efficiency and capacity of the air
transportation system while maintaining safety and accommodating
anticipated future growth. The planning for NextGen began in 2003 13 and
is now focused on implementing improvements in the midterm (by 2018)
and in the far term (by 2025). 14




11
     GAO-08-42.
12
 GAO, Next Generation Air Transportation System: Challenges with Partner Agency and
FAA Coordination Continue, and Efforts to Integrate Near, Mid-, and Long-term Activities
Are Ongoing, GAO-10-649T (Washington, D.C.: Apr. 21, 2010).
13
  Pub. L. No. 108-176, Vision 100—Century of Aviation Reauthorization Act Pub. L. No.
108-176, §§ 709-710, 117 Stat. 2490, 2582 (2003) authorized FAA to begin the NextGen
initiative.
14
   Midterm includes capabilities that are planned to be implemented by 2018, such as
improved aircraft operational procedures and automated data communication between
aircraft and controllers. Far term refers to the complete implementation of NextGen.




Page 7                                       GAO-12-223 Air Traffic Control Modernization
Figure 2: Improvements to Phases of Flight Expected under NextGen




                                       As noted previously, we selected 4 programs for in-depth review to
                                       determine the extent to which their cost estimates and schedules aligned
                                       with best practices:

                                       •   ADS-B will enable aircraft to continually broadcast flight data such as
                                           position, air speed, and altitude, among other types of information, to
                                           air traffic controllers and other aircraft. The program was baselined in
                                           2007, and FAA is currently installing the hardware and software at
                                           approximately 800 sites across the nation. The program is scheduled
                                           to be completed in 2014.

                                       •   CATMT will provide new functionality and other enhancements to the
                                           existing Traffic Flow Management System (TFMS), such as
                                           automated reroutes and improved data exchanges between ATC
                                           facilities. CATMT was first baselined in 2008 and is scheduled to be
                                           completed in 2015.

                                       •   SWIM will provide an information technology infrastructure that will
                                           enable the multiple systems that make up the NAS to share



                                       Page 8                                 GAO-12-223 Air Traffic Control Modernization
       information. As such, SWIM is a portfolio of capabilities that will be
       implemented by other systems. SWIM was first baselined in 2009 and
       is scheduled to be completed in 2020.

•      WAAS will provide aircraft with more accurate aircraft position
       information to facilitate more direct flight paths and precision
       approaches to airports. The initial WAAS program was baselined in
       1998. The third segment of the program is scheduled to be completed
       in 2013.

Successful acquisition program management depends, in part, upon
effective cost estimation. The cost estimate is the basis for establishing a
program’s detailed schedule (see following discussion of schedules), as
well as identifying the bounds for how much program costs can be
expected to vary. We have defined cost estimates as the summation of
individual cost elements, using established methods and valid data, to
estimate future program costs based on what is currently known. 15 As
such, cost estimating requires both science and judgment because
answers are seldom—if ever—entirely precise. The goal is to find a
reasonable estimate. 16 Reliable cost estimating is a critical function
without which agencies are at risk of experiencing cost overruns, missed
deadlines, and performance shortfalls.

Our Cost Guide identifies 12 steps consistently applied by cost-estimating
organizations throughout the federal government and industry and
considered best practices for developing cost estimates. For the purposes
of this review, we grouped these steps into four characteristics of high-
quality and reliable estimates: well-documented; comprehensive;
accurate, and credible. (See table 2.)




15
     GAO-09-3SP.
16
     GAO-09-3SP.




Page 9                                   GAO-12-223 Air Traffic Control Modernization
Table 2: Characteristics of High-Quality Cost Estimates and Steps Related to Each Characteristic

Characteristic       Explanation                                                               Step identified in Cost Guide
Well-documented      The documentation should address the purpose of the estimate, the         Step 1: Define the estimate’s purpose,
                     program background, a description of the system, the system’s             scope, and schedule
                     schedule, the scope of the estimate (in terms of time and what is         Step 3: Define the program
                     and is not included), the ground rules and assumptions, all data          characteristics
                     sources, the estimating and rationale, the results of the risk
                     analysis, and a conclusion about whether the cost estimate is             Step 5: Identify ground rules and
                     reasonable. Therefore, a good cost estimate—while taking the form         assumptions
                     of a single number—is supported by detailed documentation that            Step 6: Obtain the data
                     describes how it was derived and how the expected funding will be         Step 10: Document the estimate
                     spent in order to achieve a given objective. For example, the
                                                                                               Step 11: Present the estimate to
                     documentation should capture in writing such things as the source
                                                                                               management for approval
                     data used and their significance, the calculations performed and
                     their results, and the rationale for choosing a particular estimating
                     method or reference. Moreover, this information should be captured
                     in such a way that the data used to derive the estimate can be
                     traced back to and verified against their sources, allowing for the
                     estimate to be easily replicated and updated. Finally, the cost
                     estimate should be reviewed and accepted by management to
                     ensure that there is a high level of confidence in the estimating
                     process and in the estimate itself.
Comprehensive        The cost estimates should include both government and contractor          Step 2: Develop the estimating plan
                     costs of the program over its full life cycle, from inception through     Step 4: Determine the estimating
                     design, development, deployment, and operation and maintenance            structure
                     to retirement. The estimates should also completely define the
                     program, reflect its current schedule, and be technically reasonable.     Step 5: Identify ground rules and
                     Comprehensive cost estimates should provide a level of detail             assumptions
                     appropriate to ensure that cost elements are neither omitted nor
                     double-counted, and they should document all cost-influencing
                     ground rules and assumptions. Establishing a product-oriented
                     work breakdown structure is a best practice because it allows a
                     program to track cost and schedule by defined deliverables, such
                     as a hardware or software component.
Accurate             The cost estimates should provide for results that are unbiased,          Step 7: Develop the point estimate
                     and they should not be overly conservative or optimistic. Estimates       Step 12: Update the estimate to reflect
                     are accurate when they are based on an assessment of most likely          actual costs and changes
                     costs, adjusted properly for inflation, and contain few, if any, minor
                     mistakes. In addition, the estimates should be updated regularly to
                     reflect material changes in the program, such as when schedules or
                     other assumptions change, and actual costs so that the estimate is
                     always reflecting the program’s current status. Among other things,
                     the estimate should be grounded in documented assumptions and
                     a historical record of cost estimating and actual experiences on
                     other comparable programs.




                                           Page 10                                            GAO-12-223 Air Traffic Control Modernization
Characteristic   Explanation                                                            Step identified in Cost Guide
Credible         The cost estimates should discuss any limitations of the analysis      Step 7: Compare the point estimate to an
                 because of uncertainty or biases surrounding data or assumptions.      independent cost estimate
                 Major assumptions should be varied, and other outcomes                 Step 8: Conduct sensitivity analysis
                 recomputed to determine how sensitive they are to changes in the
                 assumptions. Risk and uncertainty analysis should be performed to      Step 9: Conduct risk and uncertainty
                 determine the level of risk associated with the estimate.              analysis
                 Furthermore, the estimate’s results should be crosschecked, and
                 an independent cost estimate conducted by a group outside the
                 acquiring organization should be developed to determine whether
                 other estimating methods produce similar results. For management
                 to make good decisions, the program estimate must reflect the
                 degree of uncertainty, so that a level of confidence can be given
                 about the estimate. Having a range of costs around a point estimate
                 is more useful to decision makers because it conveys the level of
                 confidence in achieving the most likely cost and also informs them
                 on cost, schedule, and technical risks.
                                      Source: GAO.



                                      The success of a program also depends in part on having an integrated
                                      and reliable schedule, which defines, among other things, when work
                                      activities will occur, how long they will take, and how they are related to
                                      one another. As such, the program schedule not only provides a road
                                      map for systematic program execution, but also provides the means by
                                      which to gauge progress, identify and address potential problems, and
                                      promote accountability. Accordingly, a schedule helps ensure that all
                                      stakeholders understand both the dates for major milestones and the
                                      activities that drive the schedule. If changes occur within a program, the
                                      schedule helps decision makers analyze how those changes affect the
                                      program. We have previously identified nine best practices that help
                                      ensure a reliable program schedule (these best practices are discussed
                                      later in this report). The reliability of the schedule will determine the
                                      credibility of the program’s forecasted dates, which are used for decision
                                      making. FAA is currently developing an integrated master schedule 17 for
                                      the NextGen initiative that will be built in part on individual program
                                      schedules. The NextGen integrated master schedule is intended to be a
                                      comprehensive framework to support planned and actual work, providing
                                      decision makers with the information needed to manage the overall effort
                                      effectively.




                                      17
                                       An integrated master schedule is required at the acquisition program level in order to
                                      meet our best practices, which we discuss later in this report.




                                      Page 11                                          GAO-12-223 Air Traffic Control Modernization
Most Ongoing ATC
Programs Remain
within Budget and on
Schedule, but a Few
Have Significantly
Exceeded Initial
Estimates

Most Cost Estimates          Of the 30 baselined FAA ATC programs we reviewed,19 have not
Remain on Target, but Cost   increased in cost, but 11 have experienced cost increases ranging from
Overruns for Three Key       $2 million to over $2 billion. And of the 19 programs whose costs have not
                             increased, 7 experienced a cost decrease while the remainder have not
Programs in Total            changed significantly. 18 However, the 11 programs that exceeded their
Exceeded $4 Billion          initial estimated costs account for over 60 percent of total program costs
                             for the 30 baselined programs—$11 billion of $17.7 billion. These 11
                             programs are among the most complex of FAA’s major acquisitions in
                             that each involves a large amount of software engineering. (See table 3.)

                             The 3 programs with the largest cost increases—totaling more than
                             $4 billion—are key to ATC modernization. Several factors contributed to
                             cost overruns for the Standard Terminal Automation Replacement
                             System (STARS), WAAS, and ERAM programs and required additional
                             congressional appropriations or reductions in program scope.

                             •      Our previous work disclosed that the near tripling of the STARS’s
                                    budget resulted from insufficient involvement of stakeholders and
                                    requirements growth. 19




                             18
                                Some programs had neither cost increases not schedule delays, including: En route
                             Communication Gateway, Integrated Display System, Regulation and Certification
                             Infrastructure for System Safety, Time Based Flow Manager, and Weather and Radar
                             Program.
                             19
                                  GAO-08-42.




                             Page 12                                      GAO-12-223 Air Traffic Control Modernization
•    The WAAS program began in 1998 with an initial cost estimate of $1
     billion 20 and a current estimate of $3 billion. We reported previously
     that FAA’s lack of scientific and technical expertise resulted in
     unplanned work and contributed to cost increases as well as delays in
     the deployment schedule. Additionally, FAA changed how it
     accounted for certain costs in the capital budget in 1999, 21 which
     further raised the cost estimate to $3.3 billion. FAA recently revised
     that estimate down to the current $3 billion during the 2009
     rebaselining because, according to FAA officials, they had met certain
     program requirements in 2006.

•    As previously mentioned, ERAM is a key modernization system and
     will be the backbone of the NextGen system. FAA originally submitted
     to Congress an estimated cost of $2.1 billion in 2003, and the
     program is now expected to cost about $2.4 billion––an increase of
     about $330 million. According to FAA, various software issues (e.g.,
     unsuccessful transmission messages and inaccurate data pairing of
     aircraft and traffic display), as well as problems interfacing with other
     facilities and systems, have contributed to the cost increases and
     delays. The extent to which unanticipated requirements, unplanned
     work, and underestimates of the complexity of software development,
     among other factors, have contributed to other FAA ATC acquisition
     program cost overruns and scheduling delays is discussed later in this
     report.




20
  In GAO-05-331, we noted that the original estimate for WAAS was $509 million for a
terminated contract with Wilcox Electric. Subsequent estimates and baselines consider
only the interim contract awarded to Hughes Aircraft in 1996.
21
 FAA changed how it accounted for the costs of satellite leases from the operations
account to the facilities and equipment account.




Page 13                                      GAO-12-223 Air Traffic Control Modernization
Table 3: Eleven ATC Programs That Have Experienced Cost Increases as of August 2011

Dollars in millions
                                                               Original           Initial        Current             Cost         Percentage
Program                                                       start date       estimate         estimate         increase          increases
                                   a
Wide Area Augmentation System                                        1998        $1,001           $3,008           $2,007                199%
Standard Terminal Automation Replacement System                      1996          $940           $2,719           $1,779                189%
En route Automation Modernization                                    2003        $2,154           $2,484              $330                15%
                                                 c
Automated Dependant Surveillance—Broadcast                           2007        $1,682           $1,726               $44                  3%
                                                                     2001
                                                 d
Aviation Surface Observation Weather Network                                       $351              $384              $33                10%
Runway Status Lights                                                 2010          $327              $352              $25                  8%
UHF Replacement                                                      2002             $85             $93               $8                  9%
                                       b
International Flight Inspection Aircraft                             2003             $27             $34               $6                23%
                                       b
Integrated Terminal Weather System                                   1997          $276              $282               $6                  2%
Terminal Doppler Weather Radar                                       2003             $75             $77               $2                  3%
Tower Training Simulators                                            2007             $34             $36               $2                  7%
Total                                                                            $6,952          $11,195           $4,243
                                           Source: GAO analysis of FAA data.
                                           a
                                            According to FAA, the WAAS program is currently divided into three phases—two have been
                                           completed and the third is projected to be completed by 2013. However, in the CIP for fiscal years
                                           2012 through 2016, the WAAS program is listed as a single program costing $3 billion.
                                           b
                                               Program completed during GAO review.
                                           c
                                            According to the FAA, the ADS-B current revised budget includes congressional directed spending
                                           of $9.4 million in fiscal year 2008 and $6.8 million in fiscal year 2009.
                                           d
                                               This program includes the ongoing pre planned product improvement segment.




                                           Page 14                                             GAO-12-223 Air Traffic Control Modernization
Schedules for
Roughly Half of ATC
Programs Are on
Track, but Half of ATC
Programs Are
Delayed

Half of ATC Programs Are   We found that 15 of the 30 baselined programs either have experienced
on or Ahead of Schedule    no change in schedule or were completed early or on time; however, the
and Half Are Delayed       other 15 programs are projected to be completed later than originally
                           estimated. These delayed programs range from the Integrated Display
                           System, which will consolidate information from several weather
                           subsystems into a single display, which FAA expects to be completed 2
                           months after its initial estimated completion date, to WAAS, which FAA
                           estimates will be completed in 2013—more than 14 years after its initial
                           estimated completion date (see table 4). Ten of the 15 programs with
                           schedule delays also experienced cost increases. However, even if a
                           schedule delay does not result in a direct cost increase to that program,
                           the delay can lead to increased costs for FAA because FAA staff must
                           continue to manage the acquisition over the longer term as it is being
                           implemented, as well as maintain any legacy system that the program is
                           replacing. Because of program interdependencies, a schedule delay can
                           also affect how and when other programs will be implemented.




                           Page 15                               GAO-12-223 Air Traffic Control Modernization
Table 4: Fifteen ATC Programs That Have Experienced Delays Based on Original Scheduled Completion Date

                                                                            Original planned    Projected               Delay or projected
Program                                     Original start date             completion date     completion date          delay (in months)
                                    a
Wide Area Augmentation System               January 1998                    August 1999         September 2013                            169
Air Traffic Control Beacon Interrogator-6   August 1997                     September 2004      January 2012                               88
                                        b
Integrated Terminal Weather System          June 1997                       July 2003           Aug 2010                                   85
Next Generation Air/Ground                  September 1998                  September 2008      September 2013                             60
Communication System Segment 1
                                d
Airport Surveillance Radar- 11              November 1997                   September 2005      June 2010                                  57
Terminal Doppler Weather Radar              February 2003                   December 2013       September 2017                              4
En route Automation Modernization           June 2003                       December 2010       August 2014                                44
Aviation Surface Weather Observation        August 2001                     September 2009      September 2012                             36
        c
Network
UHF Replacement                             November 2002                   September 2010      September 2013                             36
International Flight Inspection Aircraft    December 2003                   August 2009         May 2012                                   33
Voice Switching and Control System          August 2006                     June 2012           December 2014                              30
(Tech Refresh) Phase 2
Standard Terminal Automation                February 1996                   October 2005        June 2007                                  20
Replacement System
                           b
Tower Training Simulator                    December 2007                   September 2009      August 2010                                11
Runway Status Lights                        January 2010                    October 2015        June 2016                                   8
Integrated Display System Replacement       September 2008                  October 2015        December 2015                               2
                                            Source: GAO analysis of FAA data.
                                            a
                                             According to FAA, the WAAS program is divided into three phases—two have been completed and
                                            the third is projected to be completed by 2013. However, in the CIP for fiscal years 2012 through
                                            2016, the WAAS program is listed as a single program costing $3 billion.
                                            b
                                                Program completed during GAO review.
                                            c
                                             Program includes the ongoing pre planned product improvement segment.
                                            d
                                             The ASR-11 Program was re-baselined in September 2005 with a planned completion of September
                                            2009.




                                            Page 16                                            GAO-12-223 Air Traffic Control Modernization
Several Factors
Contributed to Cost
Increases and
Schedule Delays, and
Some Could Hamper
NextGen
Implementation

A Number of Factors Have   Cost increases and schedule delays occurred because of several factors,
Contributed to Cost        all of which have been long-standing challenges for FAA and some of
Increases and Delays       which continue to affect programs despite FAA efforts to mitigate the
                           factors. Specifically, these factors include (1) additional, unanticipated
                           system requirements work; (2) insufficient stakeholder involvement
                           throughout system development; (3) underestimates of the complexity of
                           software development; and (4) unanticipated events, including funding
                           decreases or work stoppages (see table 5). Of the 30 programs we
                           reviewed, 15 experienced cost increases, schedule delays, or both, 22 and
                           we were able to determine that cost increases or schedule delays for 11
                           were attributable to one or more of these factors. 23




                           22
                              Ten programs experienced cost delays and schedule delays, and five programs only
                           experienced schedule delays.
                           23
                                We did not categorize, or have available information for, the remaining four programs.




                           Page 17                                          GAO-12-223 Air Traffic Control Modernization
Table 5: Key Factors Contributing to Cost Growth, Schedule Delays, or Both for ATC Systems

                                                                                                  Underestimating
                                                    Unanticipated                 Insufficient    the complexity of
                                                   requirements or               stakeholder          software            Unanticipated
Name of ATC system                                 unplanned work                involvement        development              events
Airport Surveillance Radar-11                                                                                                  
EnRoute Automation Modernization                                                                         
Next Generation Air-Ground Communications                                                                                      
Runway Status Lights                                                                                                            
Voice Switching and Control System- Tech
Refresh Phase 2                                                                                                                

Air Traffic Control Beacon Intergrorator-6                                                                                     
Integrated Weather System                                                                                                     
International Flight Inspection Aircraft                      
Wide Area Augmentation System                                                                            
Standard Terminal Automation Replacement
System                                                                               

UHF Radio Replacement                                                                                                           
                                             Source: GAO analysis of FAA data.



                                             Following are some examples of how these contributing factors led to
                                             cost increases or schedule delays in some of FAA’s ATC baselined
                                             programs:

                                             •     Unanticipated requirements or work: For nine of the programs in table
                                                   5, FAA has had to undertake substantially more development work than
                                                   planned because FAA program officials originally misjudged the extent
                                                   to which commercial off-the-shelf nondevelopmental solutions, such as
                                                   those procured by another agency, would meet FAA’s needs. For
                                                   example, although WAAS was being developed by an integrated
                                                   product team that included representatives from several FAA offices,
                                                   the team did not effectively resolve problems in meeting a required
                                                   performance capability—that pilots be warned in a timely manner when
                                                   a system may be providing them with potentially misleading and
                                                   possibly hazardous information. These actions resulted in unanticipated
                                                   work and contributed to the rise in WAAS’s cost from the original
                                                   estimate of $509 million in 1994 to about $2 billion in 2005.




                                             Page 18                                             GAO-12-223 Air Traffic Control Modernization
•      Insufficient stakeholder involvement: As we previously reported, 24
       ERAM was designed at a time when air traffic controllers did not
       participate in efforts to design and test new systems. Because active
       users of the system from different locations could not provide insight
       early on, issues that could have been addressed early in the design
       phase were not addressed. In response, FAA has taken steps to
       improve the testing of new systems in order to reduce the likelihood of
       larger-than-anticipated software issues arising during system
       implementation. For example, FAA and the controllers’ union recently
       entered into a memorandum of understanding to bring controllers into
       the testing and evaluation phase of ERAM. 25 Under this agreement,
       the controllers’ union will have ERAM technical, evaluation, and
       training representatives, as well as a team of 16 controllers (including
       12 from en route facilities and 4 from terminal facilities), who will be
       detailed to test and validate software fixes with contractor engineers
       at the FAA Technical Center (Tech Center). In addition, our previous
       work disclosed that the near tripling of the Standard Terminal
       Automation Replacement System’s budget resulted from insufficient
       involvement of stakeholders and requirements growth—two systemic
       factors that we found led to acquisitions missing their budget and
       schedule targets. This, in turn, contributed to cost growth, schedule
       delays, and eventually a reduction in the number of systems to be
       deployed. 26

•      Underestimates of the complexity of software development: This
       factor contributed to cost increases and schedule delays for ERAM,
       as well as issues with costs, scheduling, or both for two other
       programs. In 2010, FAA tested ERAM at two key sites (the Seattle
       and Salt Lake en route centers) on live air traffic, usually late at night
       when air traffic volume was low. During this testing, FAA encountered
       both anticipated and unanticipated software issues, which prompted
       the test sites, at times, to revert to using FAA’s legacy en route
       computer system. Specifically, software instructions to a controller in
       one sector to hand off control of an aircraft to a controller in an


24
 GAO, Next Generation Air Transportation System: FAA’s Metrics Can Be Used to Report
on Status of Individual Programs, but Not of Overall NextGen Implementation or
Outcomes, GAO-10-629 (Washington, D.C.: July 27, 2010).
25
   Under the new contract between FAA and the controllers union, the union is to have full
participation in the development and implementation of air traffic modernization systems.
26
     GAO-08-42.




Page 19                                       GAO-12-223 Air Traffic Control Modernization
     adjacent sector failed, and flight data were lost or reassigned to
     another flight. While some testing at FAA’s Tech Center preceded
     testing at the two key sites, the Tech Center could only test limited
     scenarios, and none of the scenario testing identified this software
     error. In addition, as discussed earlier, ERAM was designed during a
     time when air traffic controllers did not participate in efforts to design
     and initially test new systems. FAA anticipated the potential for
     software issues at the outset of the program and initially scheduled
     approximately 6 to 9 months of contingency time between the time it
     achieved initial operating capability 27 and operational readiness
     demonstrations at these sites, leaving little buffer for any potential
     delays. FAA worked with its contractor to correct a number of software
     issues, but further testing on live air traffic at the two test sites
     continued to produce critical safety errors. As a result, in March 2010,
     FAA decided, with the support of the air traffic controllers’ union, to
     halt all ERAM testing on live traffic and to revise the deployment
     schedule. The program was rebaselined in June 2011, and the
     program’s completion date was extended from December 2010 to
     August 2014. As a result of the schedule delays, the rebaselined cost
     estimate increased from $ 2.1 billion to $2.4 billion. 28

•    Unanticipated events: Unanticipated events at implementation sites
     and unanticipated funding issues have delayed several programs’
     schedules and increased costs. For example, Airport Surveillance
     Radar-11 was originally scheduled to be completed in June 2009 but
     was delayed to June 2010. FAA reports indicated that the delay was
     due to an unusually protracted real estate acquisition at one site and
     issues involving validating performance during seasonal radar
     operations from other another site. Similarly, FAA’s Runway Status
     Lights program—which involves installing airport lighting equipment
     that visually signals to pilots when it is unsafe to enter, cross, or begin
     takeoff on a runway—has experienced schedule delays because of
     construction issues at five sites (Charlotte, North Carolina; Fort
     Lauderdale, Florida; Las Vegas; Minneapolis; and Washington-
     Dulles). FAA officials attributed some of these delays to the furlough



27
 Initial operating capability is the declaration by site personnel that the system is ready for
conditional operational use in the NAS.
28
 Recent reports from the Department of Transportation Inspector General and MITRE (a
not-for profit organization chartered to work in the public interest) indicate that these cost
and schedule estimates may be underestimated.




Page 20                                          GAO-12-223 Air Traffic Control Modernization
                                 of some FAA employees in July 2011 and a freeze on contractor
                                 funding during the furlough, which resulted in work stoppage orders
                                 for several projects—including Runway Status Lights. FAA program
                                 managers will need to assess the impact of the furlough on other
                                 programs that had experienced work stoppage orders, including ADS-
                                 B, the Standard Terminal Automation Replacement System, SWIM,
                                 WAAS, and various weather programs. 29


Interdependencies among   The interdependencies of ATC acquisition programs have become more
NextGen ATC Programs      prominent as the NextGen program shifts from planning to
Could Slow NextGen        implementation, so that cost increases and schedule delays in one
                          program could have a cascading effect on other programs. As discussed
Implementation            earlier, due to the integrated nature of NextGen, the development and
                          delivery of many of its component programs are mutually dependent on
                          the development and delivery of one or more other programs. For
                          example, ERAM, FAA’s new en route computer system, is critical to the
                          delivery of ADS-B capabilities such as broadcasting flight information.
                          ERAM is also pivotal to the on-time implementation of two other key
                          NextGen programs—Data Communications (DataComm), 30 which is
                          estimated to cost about $3 billion, and the NextGen information
                          technology architecture, SWIM, which is estimated to cost over $550
                          million. Due in part to ERAM’s delay, FAA was forced to delay the Data
                          Communications baseline date by approximately 6 months, rebaseline
                          SWIM segment 1, and delay the SWIM segment 2 baseline date to 2012.
                          The longer-term effects of these delays are unclear, but certain SWIM
                          capabilities could be delayed for several years, and the progress of other
                          programs that are dependent on SWIM’s system integration could be
                          hindered, as well. Thus, looking more broadly, the implementation of
                          NextGen—both midterm (now through 2018) and far-term (2019-2025)
                          schedules—will be affected by how well FAA manages program
                          interdependencies.

                          As we reported in 2010, 31 individual FAA program offices understand their
                          programs’ dependence on ERAM’s implementation, but FAA has not


                          29
                               http://www.faa.gov/news/media/workstop/.
                          30
                           DataComm is planned provide capabilities for pilots and controllers to transmit digital
                          messages and will eventually replace the analog voice communication system currently in
                          use.
                          31
                               GAO-10-629.




                          Page 21                                         GAO-12-223 Air Traffic Control Modernization
                       developed a full listing of how ERAM schedule slippages, or slippages in
                       other programs that are critical to NextGen, could either affect other
                       programs’ implementation schedules or delay the implementation of
                       capabilities and improvements. 32 In 2008, we recommended that FAA
                       improve the usefulness of ATO’s acquisition performance reporting by
                       including information in the agency’s Performance and Accountability
                       report or elsewhere on the potential effect of any budget or schedule
                       slippages on the overall transition to NextGen. 33 This recommendation
                       remains open, as FAA has not definitively indicated how it will track
                       slippages that will affect other dependent NextGen programs. FAA’s
                       acquisition management system was not designed for managing NextGen
                       programs in an integrated way. To assist in managing NextGen portfolios,
                       FAA is starting to monitor all the activities of a particular operational
                       improvement to ensure integration is on track. As we noted in the 2010
                       report, as this approach is more fully implemented, it will likely clarify the
                       impact of slippages in one program’s schedule on the implementation
                       status of other NextGen programs and operational capabilities. In
                       addition, as we will discuss in the next section, FAA is developing an
                       Integrated Master Schedule for the entire NextGen initiative that is, in
                       part, intended to show how changes in program schedules affect other
                       programs and the timelines for the NextGen initiative as a whole.
                       However, as we discuss later, the schedules for the four programs we
                       reviewed in detail are not reliable. Reliable schedules at the program level
                       will be needed to develop a reliable Integrated Master Schedule for
                       NextGen.


Program Management     According to FAA, it is taking actions to address the factors that have
Changes Are Aimed at   contributed to cost increases and schedule delays. In 2011, FAA
Addressing Past        assessed the NextGen effort as part of its Foundation for Success
                       initiative 34 and has implemented the “Idea to In-Service Management”
Challenges             (I2I) process, which it believes will result in improvements in the way the



                       32
                          FAA’s Enterprise Architecture for the national airspace system shows the
                       interdependencies and capabilities that may be affected by various programs, but this
                       document cannot indicate specific milestones that could be affected.
                       33
                            GAO-08-42.
                       34
                          The Foundation for Success initiative is aimed at improving certain governance, shared
                       services, human capital, and NextGen management structures to better manage FAA
                       functions.




                       Page 22                                       GAO-12-223 Air Traffic Control Modernization
FAA develops, acquires, and implements new NextGen capabilities from
conception through implementation. The I2I concept is intended to
improve collaboration early in the acquisition process, resulting in better
defined capabilities and an early indication of cost and benefits. These
enhancements are intended to resolve many of the challenges associated
with overall program management and enable FAA to focus on program
management best practices. FAA believes that I2I will also result in
improvements in specific areas that have presented challenges in the
past, such as cost estimating, anticipating requirements and work,
stakeholder collaboration, software development, and systems
integration.

Also in 2011, FAA implemented a reorganization of the NextGen
Operations and Planning Office and ATO which FAA believes will support
the I2I process and improve acquisitions of NextGen programs.
Specifically, FAA created a NAS Lifecycle Planning Division within the
NextGen Operations and Planning Office to focus the integration of
NextGen programs from a cost, schedule, and systems capability
perspective. Within ATO, FAA established a new Program Management
Office, which puts the responsibility for the program management of all
NextGen and other major ATC acquisitions within a single organization.
By combining program managers into one organization, FAA hopes to
create a stronger acquisition program and improve the consistency and
implementation of best practices. According to FAA, these organizational
changes allow responsibilities for acquisitions to be better defined to more
efficiently set strategic direction, define operational requirements, ensure
system integration, oversee implementation processes, and ensure
accountability throughout the acquisition life cycle.

To improve the acquisitions management process, FAA has also divided
large acquisition programs into segments. A segmented or phased
approach is being taken with programs like SWIM and CATMT. This
approach breaks a larger program into smaller and more manageable
pieces to lower the risk. In the past, we noted that this approach can
improve management by providing for midcourse corrections and, thus,
help FAA avoid costly late-stage changes. However, this approach can
also increase the duration and possibly the cost of the program. 35



35
 GAO, Air Traffic Control: FAA’s Modernization Efforts Past, Present, and Future,
GAO-04-227T (Washington, D.C.: Oct. 30, 2003).




Page 23                                      GAO-12-223 Air Traffic Control Modernization
                             According to FAA officials, a segmented approach allows the agency to
                             more effectively manage acquisitions at both the program and enterprise
                             architecture level. An enterprise architecture approach provides the
                             structure to relate organizational mission, vision, and goals to business
                             processes and the technical or information technology infrastructure
                             required to execute them. FAA officials stated that many of the factors we
                             identified that contributed to cost increases and schedule delays highlight
                             the need for an enterprise-level perspective throughout the acquisition
                             process. The I2I process is intended to provide an enterprise-level focus
                             and improve collaboration across related programs.



For Programs We
Reviewed, FAA Did
Not Consistently Meet
All Characteristics of
Quality Cost
Estimates or Schedule
Best Practices

Selected Cost Estimates      Our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates
Were Generally               showed that while each program followed at least some of the four
Comprehensive and Well-      characteristics of high-quality and reliable cost estimates—well-
                             documented, comprehensive, accurate, and credible—none of the
Documented, but Accuracy     programs adhered closely enough to those characteristics to create a
and Credibility Need to Be   reliable cost estimate. As previously noted, these characteristics
Improved                     incorporate the 12 steps consistently applied by cost-estimating
                             organizations throughout the federal government and industry and
                             considered best practices for developing cost estimates. The results of
                             our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates,
                             which are summarized in table 6, show that they were most aligned with
                             the characteristic of comprehensive cost estimates but need improvement
                             in the other three areas, particularly with the characteristics of accurate
                             and credible estimates. Imprecise estimates can result in Congress
                             unnecessarily authorizing and appropriating millions of dollars for
                             programs. As noted, in some cases, FAA, in order to stay within the
                             original cost estimate, modified a program’s requirements and Congress




                             Page 24                               GAO-12-223 Air Traffic Control Modernization
                                           had to appropriate more funds or reduce the scope to allow FAA to finish
                                           the program.

                                           FAA could better ensure that the cost estimates for these four programs,
                                           as well as its other major acquisition programs, are reliable. Our work
                                           shows that an assessment of these cost estimates for these programs, as
                                           well as FAA’s other major acquisition programs, would allow FAA to
                                           better understand if its cost estimation guidelines and our characteristics
                                           of high-quality cost estimates are in fact being followed (See table 6).

Table 6: GAO Analysis of the Extent FAA Acquisition Cost Estimates Met the Characteristics of High-Quality and Reliable
Cost Estimates

Characteristic    Characteristic description                                                            ADS-B       CATMT       SWIM      WAAS
Well-documented   The cost estimate should be supported by detailed documentation that
                  describes the purpose of the estimate, the program background and system
                  description, the scope of the estimate, the ground rules and assumptions,
                  all data sources, estimating methodology and rationale, and the results of
                  the risk analysis. Moreover, this information should be captured in such a
                                                                                                           ◑          ◑           ◕         ◕
                  way that the data used to derive the estimate can be traced back to, and
                  verified against, the sources.
Comprehensive     The cost estimates should include costs of the program over its full life
                  cycle, provide a level of detail appropriate to ensure that cost elements are
                  neither omitted nor double counted, and document all cost-influencing
                  ground rules and assumptions.
                                                                                                           ◕          ◕           ◕         ◕
Accurate          The cost estimate should be based on an assessment of most likely costs
                  (adjusted for inflation), documented assumptions, historical cost estimates,
                  and actual experiences on other comparable programs. Estimates should
                  be cross-checked against an independent cost estimate for accuracy,
                  double counting, and omissions. In addition, the estimate should be
                                                                                                           ◑          ◑           ◑         ◑
                  updated to reflect any changes.
Credible          The cost estimates should discuss any limitations of the analysis because
                  of uncertainty, or biases surrounding data or assumptions. Risk and
                  uncertainty analysis should be performed to determine the level of risk
                  associated with the estimate. Further, the estimate’s results should be
                                                                                                           ◑          ◔           ◔         ◑
                  cross-checked against an independent estimate.
                                           Source: GAO analysis of FAA documents.

                                            Met
                                           ◕ Substantially met
                                           ◑ Partially met
                                           ◔ Minimally met
                                            Not met
                                           Note: “Not met” means the program provided no evidence that satisfies any of the best practice
                                           criteria. “Minimally met” means the program provided evidence that satisfies a small portion of the
                                           criteria. “Partially met” means the program provided evidence that satisfies about half of the criteria.
                                           “Substantially met” means the program provided evidence that satisfies a large portion of the criteria.
                                           “Met” means the program provided evidence that completely satisfies the criteria.




                                           Page 25                                               GAO-12-223 Air Traffic Control Modernization
Because the four programs were generally similar in the extent to which
they met each of the four characteristics, the following discussion
summarizes the strengths and weaknesses we found for each
characteristic across the four programs. A more detailed discussion of our
findings is contained in appendix IV.

•   Well-documented. Two of the four cost estimates we analyzed
    substantially met the characteristic of being well-documented; the
    other two partially met this characteristic. A well-documented cost
    estimate is thoroughly documented, including identifying specific
    source data and their significance, detailing calculations and results,
    and explaining why particular cost estimating methods were chosen.
    In other words, sufficient documentation exists such that an unfamiliar
    analyst could recreate the cost estimate and arrive at the same
    results. For example, the SWIM estimate provided detailed
    documentation describing the program, in addition to the
    methodology, calculations, and quantities used to develop the
    estimate. However, none of the four estimates sufficiently captured
    the entire source data used, addressed its reliability, or described how
    various forms of data from disparate sources were normalized (i.e.,
    the data were described in like terms). For example, the WAAS
    estimate was based, in part, on actual labor costs from a previous
    contract, but the program office provided no evidence that these data
    were evaluated for reliability or accuracy. Similarly, the CATMT
    estimate routinely relied on subject matter expertise as a source for
    assumptions, such as the cost of labor, but the estimate did not
    document the experts’ qualifications, background, underlying
    assumptions, or data sources. Moreover, we noted that three of the
    four estimates often substantially relied on expert opinion rather than
    on data. While expert opinion can be useful in the absence of data, it
    is subjective and generally should be used sparingly for cost
    estimates. Since data are the foundation of every cost estimate, data
    quality affects the overall quality of the estimate. In addition, because
    data are gathered from a variety of sources and take many different
    forms, normalization helps to improve consistency with other cost
    information and enable valid comparisons and projections.

•   Comprehensive. All four cost estimates we analyzed substantially met
    the characteristic of being comprehensive. For an estimate to be
    comprehensive, it should include full life-cycle costs, completely
    define the program with sufficient detail, include cost elements that
    are traceable to the statement of work or objective to ensure they are
    neither omitted nor double counted, and document all cost-influencing
    ground rules and assumptions. We found that the ADS-B, CATMT,


Page 26                                GAO-12-223 Air Traffic Control Modernization
    and SWIM cost estimates included all life-cycle costs, regardless of
    program phase or funding source, and the ADS-B and SWIM cost
    estimates completely defined the program with an appropriate level of
    detail. In particular, the ADS-B cost estimate included cost estimates
    for both government and contractor costs, and the WAAS cost
    estimate thoroughly defined the program and reflected the current
    schedule. The four estimates did not fully meet the comprehensive
    characteristic because they lacked evidence that all cost influencing
    ground rules and assumptions were considered.

•   Accurate. None of the four cost estimates met or substantially met the
    characteristic of being accurate. The estimates generally adjusted
    costs for inflation and contained few computation or mathematical
    mistakes, but they were not regularly updated to reflect schedule and
    requirement changes, did not provide evidence of documenting or
    reviewing differences between planned and actual costs, and were
    not based on historical cost data from comparable programs. For
    example, the ADS-B, CATMT, and SWIM cost estimates provided no
    evidence that they were updated to reflect program changes, such as
    schedule slippages or varying assumptions, and did not include the
    current actual costs of the program. Although the WAAS estimate
    included evidence that it was updated to reflect major changes in
    technical and program requirements, such as the four rebaselinings
    the program has undergone since its 1998 inception, it did not include
    evidence that estimated costs were replaced with actual costs as the
    program advanced. Cost estimates that are not regularly updated with
    current information cannot provide decision makers with accurate
    information that is necessary, for example, when new system
    requirements are called for under tight budget conditions. In addition,
    comparing planned and actual costs enables cost estimators to
    measure the accuracy of their estimates and refine their processes. In
    addition, none of the four programs more than minimally used
    historical data to develop their cost estimates. Had historical data
    been used, the estimators would have had additional insight into
    actual costs on programs that used similar technologies, which could
    be used, for example, to challenge overly optimistic assumptions and
    bring more realism to the cost estimate.

•   Credible. None of the four cost estimates met or substantially met the
    characteristic of being credible, which includes obtaining an
    independent cost estimate from a group outside the acquiring
    organization, and cross-checking the major cost elements in that
    estimate against cost drivers identified through sensitivity and risk
    analyses. The ADS-B, CATMT, SWIM, and WAAS estimates lacked


Page 27                               GAO-12-223 Air Traffic Control Modernization
     credibility largely because FAA did not obtain an independent cost
     estimate for any of the programs. In addition the CATMT, SWIM, and
     WAAS estimates provided little evidence that it conducted sensitivity
     or risk analyses. Instead, each program received independent cost
     reviews as part of the investment decision process—even though
     such reviews are not required by FAA policy. FAA stated that the
     Investment, Planning and Analysis (IP&A) Office in the FAA Finance
     Organization does not prepare independent estimates, but it is
     organizationally independent of the acquisition programs and
     conducts independent reviews of all cost estimates. However, an
     independent cost review is less rigorous than an independent cost
     estimate. According to our cost guide, an independent cost estimate is
     often more accurate because the estimating team is further removed
     from the program office and less prone to accept overly optimistic
     assumptions or be burdened by organizational bias. 36 Other federal
     agencies, including the Department of Defense, require independent
     cost estimates. Had an independent cost estimate been completed,
     the estimating team and program team could have identified the major
     differences between their estimates, reconciled those differences
     where possible, and provided a synopsis of the two estimates and
     their differences to acquisition program management. In addition,
     without sensitivity and risk analyses, cost estimators cannot measure
     the effects of varying assumptions, and managers cannot determine,
     for example, the rational level of contingency reserves necessary to
     cover increased costs that may result from uncertainties such as
     unexpected design complexity, changes in requirements, or budget
     shortfalls––all of which FAA ATC programs, and in particular NextGen
     programs, have experienced in recent years. We found evidence that
     some level of risk analysis was conducted for ADS-B, CATMT, and
     SWIM, although the analysis was not sufficiently robust. For example,
     key cost drivers were not identified, and additional context about how
     the estimate could be affected by software design and development
     issues was not included.




36
 An independent cost estimate can be done by an independent cost estimating office
within an organization or by an outside contractor.




Page 28                                     GAO-12-223 Air Traffic Control Modernization
Selected Schedules Did                          We determined that the schedules for the four programs we reviewed are
Not Substantially Meet                          unreliable because none met or substantially met all nine of the best
Most Best Practices and                         practices for developing a reliable schedule. (see table 7). For example,
                                                none of the schedules fully met best practices for capturing all activities in
Are, Therefore, Unreliable                      an integrated master schedule, identifying critical paths and reasonable
                                                float for all activities, or assigning resources to those activities. Moreover,
                                                none of the schedules had documentation that provided more than
                                                minimal evidence that they conducted a schedule risk analysis. As was
                                                the case with our review of cost estimates for the four programs, our work
                                                regarding the schedules for these programs shows that an assessment of
                                                the schedules, as well as schedules for FAA’s other major acquisition
                                                programs, would allow FAA to understand if the nine best practices for
                                                reliable schedules are being followed.

Table 7: Extent Program Schedules Met Best Practices

                                                                  ADS-B        CATMT         SWIM          WAAS            WAAS
                                                                   (FAA      (contractor     (FAA           (FAA         (contractor
Best practice           Description                              prepared)    prepared)    prepared)      prepared)       prepared)
1.   Capturing all      A schedule should reflect all
     activities         activities defined in the program’s
                        work breakdown structure and
                        include all activities to be performed
                                                                    ◑           ◕             ◔              ◔               ◕
                        by the government and contractor.
2.   Sequencing all     The schedule should be planned so
     activities         that all activities are logically
                        sequenced in the order they are to
                        be carried out.
                                                                    ◔           ◑             ◔              ◔               ◕
3.   Assigning        The schedule should realistically
     resources to all reflect the resources (i.e., labor
     activities       material and overhead) needed to
                      do the work, whether all required
                      resources will be available when
                                                                    ◕           ◕                           ◔               ◑
                      needed, and whether any funding or
                      time constraints exist.
4.   Establishing       The schedule should reflect how
     the duration of
     all activities
                        long each activity will take to
                        execute.
                                                                    ◕                        ◑              ◑               ◕
5.   Integrating        The schedule should be horizontally
     schedule           and vertically integrated—that is, it
     activities         should link already sequenced
     horizontally and   activities with outcomes while also
     vertically         delineating the relation of
                        supporting tasks and subtasks to
                        upper-level milestones. Such
                                                                    ◑           ◑             ◔              ◑               ◕
                        mapping among levels enables
                        different groups to work to the same
                        master schedule.




                                                Page 29                                    GAO-12-223 Air Traffic Control Modernization
6.   Establishing      The schedule should identify the
     the critical path critical path, or those activities that,
                       if delayed, will negatively impact the
                       overall project completion date. The
                       critical path enables analysis of the
                                                                                         ◑                ◔                 ◔                ◕
                       effect delays may have on the
                       overall schedule.
7.   Identifying       The schedule should identify float—
     reasonable        the amount of time an activity can
     float             slip in the schedule before it affects
                       other activities—so that flexibility in
                       the schedule can be determined. As
                       a general rule, activities along the
                                                                                         ◔                ◔                 ◑                ◕
                       critical path typically have the least
                       amount of float.
8.   Conducting a      The schedule should include a
     schedule risk     schedule risk analysis that uses
     analysis          statistical techniques to predict the
                       probability of meeting a completion
                       date. A schedule risk analysis can
                       help management identify and
                                                                           ◔                              ◔                                 ◔
                       understand the most important risks
                       and focus on mitigating them.
9.   Updating the      The schedule should use realistic
     schedule using    durations for activities and be
     logic and         monitored to determine when
     durations to
     determine
     dates
                       forecasted completion dates differ
                       from the planned dates. This
                       analysis can be used to assess
                                                                           ◕              ◕                ◑                 ◑                ◑
                       whether schedule variances will
                       affect future work.
                                                 Source: GAO analysis of FAA documents.

                                                  Met
                                                 ◕ Substantially met
                                                 ◑ Partially met
                                                 ◔ Minimally met
                                                  Not met
                                                 Note: “Not met” means the program provided no evidence that satisfies any of the best practice
                                                 criteria. “Minimally met” means the program provided evidence that satisfies a small portion of the
                                                 criteria. “Partially met” means the program provided evidence that satisfies about half of the criteria.
                                                 “Substantially met” means the program provided evidence that satisfies a large portion of the criteria.
                                                 “Met” means the program provided evidence that completely satisfies the criteria.


                                                 Because the scheduling best practices are interrelated in such a way that
                                                 deficiencies in one best practice will cause deficiencies in the others, a
                                                 schedule must meet or substantially meet all nine practices to be reliable.
                                                 For example, preparing a schedule that is program-wide––including an
                                                 integrated breakdown of the work to be performed by both the
                                                 government and its contractors over the expected life of the program––is



                                                 Page 30                                               GAO-12-223 Air Traffic Control Modernization
        a best practice. If the schedule does not capture all activities, then there
        will be uncertainty about whether activities are sequenced correctly or if
        the schedule properly reflects the resources needed to accomplish the
        work, which is also a best practice. Logic and durations (that is, the time it
        takes to complete a specific activity) should be used and maintained to
        ensure realistic start and completion dates and to reflect the true status of
        the project––a necessary condition for conducting follow-on schedule risk
        analyses. Moreover, if activities are not properly sequenced with logical
        links, it will not be certain if the critical path—which represents the chain
        of dependent activities with the longest total duration—is valid.
        Collectively, the weaknesses in not fully meeting or substantially meeting
        all nine key practices increases the risk of schedule slippages and cost
        overruns since a well-defined schedule helps to identify the amount of
        human capital and fiscal resources that are needed to execute the
        program. Therefore, by not having reliable schedules, FAA cannot
        conduct meaningful oversight of an acquisition program’s progress or
        determine whether the program is achieving the desired results.

        The following discussion summarizes the extent to which the schedules
        for the four programs we examined met best practices. More detailed
        information for each program regarding scheduling best practices is
        presented in appendix V.

ADS-B   We reviewed the schedule prepared by FAA and found it did not fully
        meet any of GAO’s nine scheduling best practices, resulting in an
        unreliable schedule. Evidence provided in the ADS-B schedule indicates
        that it substantially met three of the nine best practices and partially,
        minimally, or did not meet the other six. For example, although the ADS-B
        schedule provided evidence of periodic updating, it did not capture all of
        the effort currently called for in the approved baseline for the entire ADS-
        B program and, therefore, was not a fully integrated schedule. Without
        fully integrating government activities with contractor activities, and
        thereby capturing all key activities, the schedule will not reliably estimate
        the program’s completion. In addition, the ADS-B schedule we reviewed
        did not identify critical paths or include a schedule risk analysis, which
        uses statistical techniques to predict a level of confidence in meeting a
        program’s completion date; did not logically sequence all activities and
        establish their durations; and had excessive float 37 on a majority of


        37
           The amount of time by which a predecessor activity can slip before the delay affects
        successor activities.




        Page 31                                        GAO-12-223 Air Traffic Control Modernization
        current and planned activities. According to program officials, a number of
        the issues our analysis identified were, in part, the result of the schedule’s
        limited time frame, which covered only a defined transitional period
        (October 2010 through April 2011) during which responsibility for about a
        third of the effort passed from the FAA to its prime contractor. Officials
        also stated that although their schedule contains critical activities, it has
        not had a traditional critical path since its contractor began managing the
        deployment of deliverables. The FAA uses contract options to order the
        scope, sequence, and requirements for key milestones. Within those
        options, the contractor has the authority to implement the sequence of
        more discrete activities in the order they deem most appropriate. FAA
        program officials plan to rectify this problem, noting that with negotiations
        now completed, they will in the near future identify a critical path to span
        all program milestones.

CATMT   Because the CATMT program did not prepare an FAA schedule and
        instead relied on its contractor schedule, we reviewed the contractor
        schedule, which we found to be unreliable. Our analysis found that the
        contractor’s CATMT schedule substantially or fully met four of the nine
        best practices: capturing all activities, assigning resources, establishing
        the durations, and updating the schedule. For example, the CATMT
        contactor schedule pertains to the current phase of the program that is
        being implemented in software releases, or phases. However, there was
        no overarching FAA government owned schedule that accounts for all
        software releases for the entire program and would thus delineate the
        relation of current software release tasks to the upper-level milestones for
        the overall CATMT program. The CATMT schedule included detailed
        resource information, and the program office provided evidence that
        resources are tracked in detailed labor-hour spreadsheets. We also found
        that 90 percent of the activities were of short duration and that the
        program office regularly reviews the schedule, which is in line with best
        practices. On the other hand, five of the nine best practices were either
        partially, minimally, or not met. Specifically, the CATMT schedule lacked
        evidence indicating that it established a critical path, accurately identified
        float between activities, integrated the schedule vertically and
        horizontally, sequenced all activities, or performed a schedule risk
        analysis. Regarding the critical path, our analysis determined that the
        CATMT schedule does not identify a critical path for the entire program.




        Page 32                                 GAO-12-223 Air Traffic Control Modernization
       Instead, the program is being accomplished multiple 6-month spirals 38;
       thus, there is only a critical path for each software release, not for the
       program as a whole. 39 Without a valid program-wide critical path FAA
       management cannot determine which tasks, if they slip, will have the
       most detrimental effects on the project finish date. We also found that 68
       percent of the remaining activities to be completed had unreasonably high
       float exceeding 1,000 days, meaning that those activities could slip about
       5 work years 40 without affecting the overall project finish date, a highly
       unlikely scenario. 41 The accurate identification of critical paths and float
       are inextricably linked. For example, if the schedule is missing activities or
       they are not correctly linked, float estimates will be miscalculated,
       resulting in an invalid critical path. Without a schedule that can produce a
       true critical path, the program office will neither be positioned to provide
       reliable timeline estimates nor be able to identify when problems or
       changes may occur and determine the impact they may have on
       subsequent work. CATMT program officials acknowledged that the
       schedule did not include program-wide critical paths but noted that a
       critical path exist for individual segments of the program. They also noted
       that a schedule risk analysis was not performed because it was not a
       contractual deliverable.

SWIM   Because the SWIM schedule did not fully or substantially meet any of
       GAO’s nine scheduling best practices, we found it to be unreliable. The
       SWIM program differs from the others in that it is an aggregation of
       NextGen acquisition programs, each developing an aspect of the SWIM
       information sharing capability. Because SWIM program managers are
       reliant on schedule information from a number of other programs, SWIM
       schedule integration is particularly important. However, our analysis found


       38
        Software spiral development is an incremental approach to reduce risk, so that user
       needs and requirements are better defined.
       39
        FAA officials said that because of the 6-month spiral development approach, the
       schedule cannot deliver a single critical path for the entire program. Instead, the critical
       paths are calculated and based by software releases. To calculate a critical path by each
       software release, the prime contractor uses an end constraint on the key deliverable
       milestone for each software release. For further explanation, see table 13 in appendix IV.
       40
          A work year is approximately 200 days since every organization works to a different
       calendar.
       41
          These float values are due mostly to activities being tied to the project finish milestone,
       which is constrained to start no earlier than July 1, 2016. For further explanation, see table
       13 in appendix IV.




       Page 33                                         GAO-12-223 Air Traffic Control Modernization
       that the SWIM schedule was not, by any measure, fully integrated
       because it provided only a synopsis of the individual system implementing
       program schedules and, thus, did not fully represent the work required to
       complete the overall SWIM program. This resulted in float calculations
       that were unrealistic, and the resulting critical path calculations were
       invalid. In addition, while the many missing activities negatively impacted
       the schedule logic and the accuracy of durations, it also made the
       accurate allocation of resources and comprehensive integration of
       schedule activities, both horizontally and vertically, impossible. We also
       noted that FAA made no effort to identify a program-wide critical path.
       Program officials said that because each of the system implementing
       program schedules has its own critical path, involve disparate capabilities,
       and are independent of one another their individual critical paths are not
       accessible through the SWIM schedule software. They therefore are not
       used for overall SWIM program management. We believe that the SWIM
       program itself should have its own critical path that includes, at a
       minimum, acceptance of major deliverables from the system
       implementing program schedules. Without a program-wide critical path,
       management does not have a clear picture of the underlying project tasks
       that must be performed to achieve the overall program target completion
       date. Finally, although there was no risk analysis conducted on this
       schedule, our analysis found that this best practice was minimally met
       because a risk analysis was conducted on a separate but related
       schedule, and the SWIM program office considered risk to some extent.

WAAS   Like the other three programs, we found the WAAS program schedule
       prepared by FAA unreliable because it did not fully or substantially meet
       any of GAO’s nine scheduling best practices; however, we also reviewed
       the contractor’s schedule for the same segment and found it fully or
       substantially met six best practices. 42 For example, FAA’s WAAS program
       schedule did not fully sequence activities in the order in which they are to
       be carried out. More specifically, the WAAS program schedule showed
       nearly half of the remaining activities were missing sequenced logic,
       causing us to question the calculated dates of activities. Logic is
       necessary for a schedule to show program managers when activities are
       expected to start and finish; when logic is missing, activity dates cannot


       42
         We reviewed two schedules for the WAAS program: one produced by the FAA program
       office and the other produced by its prime contractor. We evaluated the contractor
       prepared schedule, which was the most current schedule available for the purposes of the
       schedule risk analysis, which is discussed more fully later in this report.




       Page 34                                      GAO-12-223 Air Traffic Control Modernization
adjust correctly to changes in activities. To test the ability of the schedule
to dynamically update its dates due to changes, we artificially extended
the duration of an activity to 1,500 days, which changed the activity’s
finish date. However, the duration extension had no effect on successor
activities because this activity is not tied to any successor activities.
Extending the duration to 1,500 days also pushed the project planned
finish date from September 22, 2016, to June 29, 2017; however,
because the logic links are not in place, we questioned whether the
projected finish date under this scenario is reliable.

Moreover, the WAAS program schedule had too many artificial
constraints that were driving the start and finish dates for more than 70
percent of the remaining activities. 43 Constraints are usually substitutes
for logic and can mean that the schedule is not well planned or feasible.
Constraints also greatly reduce the ability of the program to take
advantage of possible time savings. Further, our analysis found that the
schedule did not fully capture or assign resources to all government and
contractor activities; it also did not accurately allocate resources or
consistently establish the duration of activities. In addition, while WAAS
program officials told us that the schedule was integrated vertically and
horizontally, we did not find evidence of such integration. Furthermore, we
found the WAAS program office’s schedule did not identify a critical path
for the entire program. As noted earlier, critical path and float
determinations are closely related. Our analysis of the WAAS program
office schedule found that more than half of the remaining activities had
float of more than 1,000 working days, which we believe to be
unreasonably high. Without proper determination of float, management
cannot properly reallocate resources from tasks to other tasks without
adversely affecting the overall completion date. Although program officials
said that they maintained a risk register listing the potential risks that
could impact the schedule and adjusted the schedule for these risks, we
did not find evidence that the program office had conducted a risk
analysis of its schedule.

While the schedule prepared by the contractor did not fully or
substantially meet three of the scheduling best practices, it fully or
substantially met six: capturing, sequencing, assigning resources to, and



43
 A constraint predefines the start, finish, or both dates of an activity. The schedule should
use logic and durations in order to reflect realistic start and completion dates for activities.




Page 35                                          GAO-12-223 Air Traffic Control Modernization
                           establishing the duration of all activities; establishing the critical path; and
                           identifying reasonable float between activities. For example, our analysis
                           found that all activity durations are consistently estimated in days and
                           adhere to a standard 5-day workweek that accounts for holidays, and no
                           activities were scheduled to begin on a weekend. Officials from the
                           contractor said duration estimates for the schedule are based on
                           historical information from past performance, comparable releases,
                           lessons learned, similar work, and other data requirements. In addition,
                           our analysis traced several critical paths in the schedule. Though we
                           found minor interruptions in the various critical paths, the schedule’s logic,
                           reasonable durations, and low total float estimates allow the calculation a
                           valid critical path.


WAAS Schedule Risk         As noted, FAA did not perform a complete schedule risk analysis for any
Analysis Indicates Risks   of the four programs we reviewed and, thus, cannot accurately estimate
That Managers Could        these programs’ completion dates with confidence. A schedule risk
                           analysis, which is one of our best practices for program scheduling, uses
Mitigate to Avoid Delays   statistical techniques to predict a level of confidence in meeting a
                           program’s completion date. The objective of the analysis is to develop a
                           probability distribution of possible completion dates that reflect the project
                           and its identified risks. This analysis can help program managers both
                           understand the most important risks to the program and focus on
                           mitigating those risks. Other federal agencies, including the Department
                           of Defense and the National Aeronautics and Space Administration,
                           require schedule risk analysis for major acquisitions; the Department of
                           Veterans Affairs, in response to a GAO recommendation, 44 plans to
                           require schedule risk analysis for major construction projects.

                           We conducted a schedule risk analysis on the WAAS contractor prepared
                           schedule, which we chose because it was relatively mature, it partially
                           met or substantially met six of the nine best practices (see table 7), and it
                           contained enough information to perform a schedule risk analysis. 45 We
                           reviewed the risk register that the contractor had developed, which



                           44
                              GAO, VA Construction: VA Is Working to Improve Initial Project Cost Estimates, but
                           Should Analyze Cost and Schedule Risks, GAO-10-189 (Washington, D.C.: Dec. 14,
                           2009).
                           45
                             The other three schedules did not have the required information to conduct a schedule
                           risk analysis.




                           Page 36                                       GAO-12-223 Air Traffic Control Modernization
showed four potential risks to the project. We then conducted interviews
with FAA program and contractor staff and asked them to discuss other
potential risks to the project, including how the risk would affect the
project’s timeline and the likelihood of the risk occurring. Using this
information we identified an additional 16 risks for a total of 20 risks. The
fact that our interviews identified a relatively large number of new risks
could be an indication that the contractor did not systematically analyze
the full range of risks when developing the program’s risk register. We
then consolidated the 20 risks into 14 broader risks and tested how each
would impact the duration of specific activities in the schedule. We then
ran a Monte Carlo simulation, 46 which consisted of the computer-
generated results of 3,000 estimates of the future schedule based on the
activities in the schedule, the chance that some of the activities would be
affected by some risks and the predicted effect of those risks on the
duration of each activity.

We then analyzed the potential impact of risks on the program schedule.
Since risks can effect the schedule in various ways––for example, risks
can have a large impact on the durations of activities they affect, or they
can introduce critical paths that are different from the baseline critical
path–– we analyzed the marginal impact of each of the risks we identified
to determine which would have the greatest effect on the overall
schedule. We found the following three key risks to the program, only the
first of which (limited WAAS program office resources) was originally
identified by the contractor. The three risks are

•    limited WAAS program office resources such as staffing;

•    delays in software yet to be released and additional changes to
     software already released and in use; and

•    a potentially optimistic schedule completion date.

Our schedule risk analysis showed the completion of the segment of the
WAAS program covered by the schedule could slip as much as 2 months.
Specifically, the analysis showed that there is less than a 5 percent
probability that the program segment would be completed by September
6, 2012, the current baselined date for completion. However, it appears


46
  A Monte Carlo simulation involves the use of random numbers and probability
distributions to examine potential outcomes.




Page 37                                      GAO-12-223 Air Traffic Control Modernization
                            that the segment will be completed close to the deadline since we found a
                            50 percent probability that the program segment will be completed by
                            October 23, 2012 (about 1.5 months after the current estimated date for
                            completion); and an 80 percent probability that the program will be
                            completed by November 13, 2012 (about 2.25 months after the current
                            estimated date for completion).

                            Although we did not conduct a schedule risk analysis for other FAA
                            programs, the result of our analysis provides examples of the types of
                            risks that major acquisition programs face and the impact those risks can
                            have on meeting acquisition program milestones, especially given the
                            interrelation and interdependencies among NextGen acquisitions
                            discussed earlier. More information on our schedule risk analysis can be
                            found in appendix V.


Lack of Reliable Program    FAA has begun developing an integrated master schedule for the entire
Schedules Will Hinder       NextGen initiative that would, in part, capture related NextGen program
Development of an Overall   schedules, governance activities, and other performance and financial
                            data to provide real-time monitoring of the overall NextGen effort.
NextGen Integrated Master   However, the unreliability of the four program schedules for programs that
Schedule                    are integral to the NextGen initiative puts this high-level master schedule
                            at risk. Having a reliable integrated master schedule would enable FAA to
                            determine how delays in one program impact other programs and the
                            overall NextGen implementation timeline. While it is encouraging that
                            FAA is beginning to develop an integrated NextGen master schedule, the
                            effort could be hampered by the lack of schedule integration at the
                            program level, as well as the failure of individual program schedules to
                            meet best practices. For example, since FAA does not perform schedule
                            risk analysis on individual programs, it cannot predict with certainty if any
                            of the programs will be completed on time. Therefore, the integrated
                            master schedule for NextGen would be built on schedules that may not
                            reflect accurate program completion dates. Similarly, none of the four
                            schedules we reviewed, which were for segments of the entire program,
                            had reflected how tasks for the segment affected milestones for the entire
                            program. Without integrated schedules at the program level, an integrated
                            master schedule at the NextGen initiative level would be problematic.




                            Page 38                                GAO-12-223 Air Traffic Control Modernization
FAA has Taken Steps to   In response to our review of the extent that the four selected acquisition
Improve Acquisition      programs met best practices for cost estimates and schedules, FAA
Program Cost Estimates   provided information on steps it is taking to improve its processes for both
                         cost estimates and schedules and noted that some of the cost estimates
and Schedules            and schedules we reviewed were developed before the improvements
                         were in place. FAA stated that strengthening its cost estimation process is
                         part of the seven key acquisition processes it has developed, including
                         program management, contractor management, requirements, risk
                         management, measurement and analysis, verification and validation, and
                         quality assurance. FAA stated that it has updated its Guidelines for FAA
                         Cost Estimating to be consistent with the GAO Cost Guide, filling in gaps
                         that it had identified during a comparison of its practices to those
                         contained in the Cost Guide. As of November 2011, 11 of the 12 best
                         practices are addressed in the guidelines. According to FAA officials, the
                         remaining best practice—involving the creation of independent cost
                         estimates—is unlikely to be implemented at FAA in the foreseeable future
                         because FAA believes the resources required to create independent
                         estimates are prohibitive in current budget environments. FAA has more
                         than tripled the number of cost estimators in the Investment Planning and
                         Analysis organization, many of which work with the acquisition program
                         offices to provide guidance on preparing estimates. Additionally, as part
                         of FAA’s effort to improve acquisition certification and training, the agency
                         is preparing to launch a cost estimating certification program. Coupled
                         with a competency-based training program, FAA believes the certification
                         program will enhance and improve consistency of the skills of FAA cost
                         estimators.

                         In describing its efforts to improve schedules, FAA stated that it views the
                         development and maintenance of integrated schedules as an inherent
                         and critical part of its seven key acquisition functions. FAA noted that
                         included in its standard process for acquisition schedules are toolkits that
                         require programs to develop integrated program schedules that address
                         all nine of GAO’s best practices. FAA stated that that the current
                         procedures for developing best practices were not fully in place when the
                         four programs we reviewed began the implementation phase.



Conclusions              FAA has made improvements in its management of air traffic control
                         modernization acquisitions, and most of the 30 we reviewed are currently
                         within the original cost estimate and half are on schedule. FAA is also
                         taking steps to address past issues to ensure cost estimates and
                         schedules are more accurate in the future, including incorporating best
                         practices in its acquisitions guidance and policies. Nevertheless, our


                         Page 39                                GAO-12-223 Air Traffic Control Modernization
review of the FAA acquisitions found that it has yet to fully implement
several GAO-identified best practices or follow others. Following best
practices is particularly important for FAA, which must manage large,
complex, and interdependent acquisitions associated with NextGen. Cost
estimates that are imprecise can result in Congress appropriating millions
of dollars for projects based on estimates that prove to be inaccurate, and
program schedule delays can increase costs and affect the
implementation of interdependent programs. In such cases, FAA will be
forced to reduce the scope of the programs to stay within the original
estimates or Congress will need to appropriate unanticipated funds to
complete the programs. Delays and cost increases in individual programs
could have a cascading effect on other programs and ultimately affect
FAA’s timelines and goals for NextGen implementation.

Our analysis of the cost estimates and schedules for the four programs
we reviewed indicates that FAA needs to further develop requirements for
critical cost estimation and schedule procedures. Independent cost
estimates can improve the accuracy and credibility of cost estimates and
better ensure that programs will be completed within budget. A schedule
risk analysis can help FAA determine the likelihood that a program will be
completed on time. FAA stated that it has no immediate plans to conduct
independent cost estimates due to current budgetary constraints. We
recognize that conducting independent cost estimates and schedule risk
analysis takes both financial resources and some time and that it may be
appropriate to limit one or both of these analyses to instances where a
program is particularly costly, complex, or on a compressed schedule.
However, conducting independent cost estimates, schedule risk analyses,
and other analyses called for in our best practices can not only help
minimize the risk of cost overruns and schedule delays, but also provide
FAA, congressional decision makers, and other stakeholders with
important information about these critical acquisitions.

It is also important that FAA develop master schedules at the individual
acquisition program level. FAA’s lack of a fully integrated master schedule
for the programs we reviewed hampers its ability to provide accurate
information on the schedule for these programs. This information will be
needed as FAA simultaneously works to develop an integrated master
schedule for the overall NextGen initiative. The use of an integrated
master schedule can assist FAA in monitoring a program, identifying
problems that could affect later stages of the program’s implementation,
improving the accuracy of cost estimates and schedules for individual
programs, and improving the accuracy of information FAA is compiling to
monitor the costs and schedules for the NextGen initiative.


Page 40                               GAO-12-223 Air Traffic Control Modernization
                      FAA has incorporated 11 of our 12 steps that are associated with the
                      characteristics of a high-quality and reliable cost estimate into their
                      acquisition guidelines. However, our analysis of the four major programs
                      indicates that FAA has not adequately integrated all of the steps for these
                      programs into its cost estimation processes, and thus the estimates are
                      not reliable. Similarly, although FAA addresses our nine scheduling best
                      practices in its acquisition guidelines, our analysis of the schedules for the
                      four programs indicates that the schedules are not adequately following
                      these best practices and are not reliable. Although the cost estimates and
                      schedules for some of the four programs were developed prior to FAA’s
                      revision of acquisition guidelines, our work shows that FAA needs to
                      assess its major acquisition programs to understand if its guidelines and
                      other best practices are, in fact, being followed. Such an assessment
                      would then allow FAA to better ensure that best practices for cost
                      estimates and schedules are being applied.



Recommendations for   To improve cost estimates and schedules for NextGen and other major
                      air traffic control acquisition programs, GAO recommends that the
Executive Action      Secretary of Transportation direct FAA to take the following three actions
                      when appropriate for major acquisition programs based on a program’s
                      cost, schedule, complexity, and risk:

                      •   Conduct independent cost estimates and schedule risk analysis for
                          major acquisition programs.

                      •   Require a fully integrated master schedule for each major acquisition
                          program, including those that are components of NextGen. An
                          integrated master schedule should horizontally and vertically link all
                          program activities and milestones, including government and
                          contractor schedules and program segments.

                      •   Conduct an assessment of major acquisition programs to ensure they
                          meet all of the established best practices for cost estimates and
                          schedules contained in GAO guidance.

                      Given constrained budgets, FAA should determine which programs
                      should be subject to these recommendations, such as those that are
                      particularly costly, complex, or on a compressed schedule.




                      Page 41                                 GAO-12-223 Air Traffic Control Modernization
Agency Comments   We provided a draft of this report to the Department of Transportation for
                  review and comment. DOT and FAA responded by email and did not
                  comment whether or not they agreed or disagreed with our
                  recommendations. DOT provided comments on the results of our analysis
                  of the cost estimates and schedules for the four programs we reviewed in
                  depth. In response to our finding that the ADS-B, CATMT, SWIM, and
                  WAAS estimates lacked credibility largely because FAA did not obtain an
                  independent cost estimate for any of the cost estimates, and provided
                  little evidence that they conducted sensitivity or risk analyses, FAA stated
                  it is not convinced that an independent organization will reduce the
                  uncertainty of cost estimates. FAA noted that it does not have an
                  independent organization such as the Department of the Navy’s Center
                  for Cost Analysis. However, FAA stated that the Finance Organization
                  within ATO assessed the ADS-B program office’s Basis of Estimate as
                  part of the JRC Decision and that this level of independence, combined
                  with specific entry and exit criteria, allowed the program offices to
                  manage these acquisitions so that costs were controlled, risks mitigated,
                  and technical parameters achieved, while adhering to the planned
                  milestone schedule. We agree that the Finance Organization assessment
                  of the two cost estimates provided some degree of independence and
                  may have improved the accuracy of the ADS-B estimates, but it is not
                  clear that such an independent review will guarantee similar results for
                  other programs. As we stated in the report, such an independent cost
                  review is less rigorous than an independent cost estimate. According to
                  our cost guide, an independent cost estimate is often more accurate
                  because the estimating team is further removed from the program office
                  and less prone to accept overly optimistic assumptions or be burdened by
                  organizational bias. DOT also provided technical clarifications, which we
                  incorporated into the report as appropriate.




                  Page 42                                GAO-12-223 Air Traffic Control Modernization
We are sending copies of this report to interested congressional
committees, the Secretary of Transportation, and the Acting Administrator
of FAA. In addition, the report is available at no charge on the GAO
website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact
me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made key contributions to this
report are listed in appendix V.




Gerald L. Dillingham, Ph.D.
Director, Physical Infrastructure Issues




Page 43                                GAO-12-223 Air Traffic Control Modernization
List of Committees

The Honorable Jay Rockefeller
Chairman
The Honorable Kay Bailey Hutchison
Ranking Member
Committee on Commerce, Science, and Transportation
United States Senate

The Honorable Ralph Hall
Chairman
The Honorable Eddie Bernice Johnson
Ranking Member
Committee on Science, Space, and Technology
House of Representatives

The Honorable Thomas Petri
Chairman
The Honorable Jerry Costello
Ranking Member
Subcommittee on Aviation
Committee on Transportation and Infrastructure
House of Representatives




Page 44                             GAO-12-223 Air Traffic Control Modernization
Appendix I: Objectives, Scope and
              Appendix I: Objectives, Scope and
              Methodology



Methodology

              In response to a congressional request, we examined the Federal
              Aviation Administration’s (FAA) ability to modernize, upgrade, and replace
              the National Airspace System’s (NAS) facilities and equipment to meet
              projected increases in traffic volumes, enhance the system’s safety, and
              increase the efficiency of the air traffic control (ATC) system—a principal
              component of the NAS. FAA’s ATC acquisitions are critical to maintaining
              the NAS and transitioning to the Next Generation Air Transportation
              System (NextGen) over the next 10 years. Given that some key legacy
              and NextGen acquisitions have experienced schedule delays and cost
              overruns, which may risk the timely implementation of NextGen, we (1)
              determined whether the planned costs and schedules of current FAA ATC
              acquisition programs have changed since they were first submitted to
              Congress; (2) examined the reasons for any changes in planned costs
              and schedules; and (3) assessed the extent to which select ATC
              programs adhered to best practices for determining acquisition costs and
              schedules.

              To describe any changes in costs and schedules of the current 30 FAA
              capital ATC acquisitions, we gathered and analyzed agency data on the
              estimated cost and schedules of these ATC acquisitions. 1 We drew upon
              past work in which we undertook detailed reviews of the status of ATC
              and other acquisition programs 2 and obtained updated documentation as
              necessary from FAA. We interviewed FAA officials to obtain information
              on FAA’s acquisition process and summarized the status of all
              acquisitions, including FAA’s original and current cost estimates and
              completion dates. For baselined acquisitions, 3 we compared estimated
              costs when they were submitted to Congress for approval against their
              current estimates, and we analyzed planned and actual schedules.

              To determine the reasons for changes in cost estimates and schedules,
              we interviewed FAA officials and FAA contractors and reviewed


              1
                  We requested the information on the programs in August 2010.
              2
                  GAO-10-588SP, GAO-10-388SP, GAO-08-42, and GAO-05-331.
              3
               According to FAA, baselined acquisitions, as opposed to nonbaselined acquisitions, are
              an agreed-to description of the attributes of a product, at a point in time, that serves as a
              basis for defining change; an approved and released document, or a set of documents,
              each of a specific revision—the purpose of which is to provide a defined basis for
              managing change; the currently approved and released configuration documentation; or a
              released set of files consisting of a software version and associated configuration
              documentation.




              Page 45                                         GAO-12-223 Air Traffic Control Modernization
Appendix I: Objectives, Scope and
Methodology




acquisition documentation. We analyzed information on cost increases
and delays to determine if systematic issues exist that have effects on
other FAA acquisitions.

To determine the extent to which select ATC programs adhered to best
practices for determining acquisition costs and schedules, we conducted
an in-depth review of 4 of the 30 acquisitions programs: The Automatic
Dependent Surveillance-Broadcast (ADS-B) system, the Collaborative Air
Traffic Management Technologies (CATMT) system, the System Wide
Information Management (SWIM) system, and the Wide Area
Augmentation System (WAAS). We selected these four acquisitions
based on the following criteria: (1) existence of baselining, (2) the
acquisition is at a point in the acquisition process where risks can be
identified, and (3) the acquisition is key to NextGen and legacy systems.
In addition to interviews, we collected documentation, and we analyzed
and summarized the views and information collected. We also identified
best practices that FAA could adopt or strengthen to improve its
acquisitions cost estimation and scheduling and ensured that acquisitions
follow cost and schedule best practices outlined in our Cost Estimating
and Assessment Guide. 4 We also performed a schedule risk analysis of
the WAAS program to determine the likelihood of the project finishing on
schedule.

We used our Cost Estimating and Assessment Guide 5 (Cost Guide) as a
source of criteria for analyzing cost estimates. As noted earlier in our
report, our Cost Guide identifies 12 steps consistently applied by cost-
estimating organizations throughout the federal government and industry
and considered best practices for developing cost estimates. For the
purposes of this review, we grouped these steps into four characteristics
of high-quality and reliable estimates—well-documented, comprehensive,
accurate, and credible—which can be summarized as follows:

Well-documented: The documentation should address the purpose of
the estimate, the project background and system description, its
schedule, the scope of the estimate (in terms of time and what is and is
not included), the ground rules and assumptions, all data sources, the
estimating methodology and rationale, the results of the risk analysis, and


4
    GAO-09-3SP.
5
    GAO-09-3SP.




Page 46                               GAO-12-223 Air Traffic Control Modernization
Appendix I: Objectives, Scope and
Methodology




a conclusion about whether the cost estimate is reasonable. Therefore, a
good cost estimate—while taking the form of a single number—is
supported by detailed documentation that describes how it was derived
and how the expected funding will be spent in order to achieve a given
objective. For example, the documentation should capture in writing such
things as the source data used and their significance, the calculations
performed and their results, and the rationale for choosing a particular
estimating method or reference. Moreover, this information should be
captured in such a way that the data used to derive the estimate can be
traced back to and verified against their sources. Finally, the cost
estimate should be reviewed and accepted by management to ensure
there is a high level of confidence in the estimate and the estimating
process.

Comprehensive: The cost estimates should include both government
and contractor costs of the project over its full life cycle, from inception
through design, development, deployment, operation, and maintenance to
retirement of the project. The cost estimate should be structured in
sufficient detail to ensure that cost elements are neither omitted nor
double counted, and they should document all cost-influencing ground
rules and assumptions.

Accurate: The cost estimates should provide for results that are
unbiased, and they should not be overly conservative or optimistic.
Estimates are accurate when they are based on an assessment of most
likely costs, adjusted properly for inflation, and contain few, if any, minor
mistakes. In addition, the estimates should be updated regularly to reflect
material changes in the project, such as when schedules or other
assumptions change so that the estimate is always reflecting the project’s
current status. Among other things, the estimate should be grounded in
documented assumptions and a historical record of cost estimating and
actual experiences on other comparable projects.

Credible: The cost estimates should discuss any limitations of the
analysis because of uncertainty or biases surrounding data or
assumptions. Major assumptions should be varied, and other outcomes
recomputed to determine how sensitive they are to changes in the
assumptions. Risk and uncertainty analysis should be performed to
determine the level of risk associated with the estimate. Furthermore, the
estimate’s results should be cross-checked, and an independent cost
estimate conducted by a group outside the acquiring organization should
be developed to determine whether other estimating methods produce
similar results.


Page 47                                GAO-12-223 Air Traffic Control Modernization
Appendix I: Objectives, Scope and
Methodology




After reviewing documentation submitted by FAA and information
obtained during interviews, we determined the extent to which the cost
estimates met the characteristics of cost-estimating best practices for the
four projects we reviewed.

Our review of project schedules was based on research that identified a
range of best practices associated with effective schedule estimating. In
addition, we obtained the consulting services of David Hulett, Ph.D., 6 to
assist in our risk analysis of the WAAS project schedule. 7 We also
conducted multiple interviews with project managers, contractors, and
schedulers to determine the extent to which current project schedules met
the best practices criteria. These nine practices are:

Capturing all activities: The schedule should reflect all activities (steps,
events, outcomes, and other factors) as defined in the project’s work
breakdown structure, including activities to be performed by both the
government and its contractors.

Sequencing all activities: The schedule should be planned so that it can
meet project-critical dates. To meet this objective, activities need to be
logically sequenced in the order that they are to be carried out. In
particular, activities that must finish prior to the start of other activities
(i.e., predecessor activities) and activities that cannot begin until other
activities are completed (i.e., successor activities) should be identified.
Identifying interdependencies among activities that collectively lead to the
accomplishment of events or milestones can be used as a basis for
guiding work and measuring progress.

Assigning resources to all activities: The schedule should realistically
reflect what resources (i.e., labor, material, and overhead) are needed to
do the work, whether all required resources will be available when they
are needed, and whether any funding or time constraints exist.




6
  Hulett & Associates, LLC Los Angeles. Dr. Hulett is the author of “Practical Schedule
Risk Analysis.”
7
 The WAAS contractor schedule was found reliable enough to conduct a schedule risk
analysis. Because we found the other three schedules were unreliable, a schedule risk
analysis could not be performed.




Page 48                                        GAO-12-223 Air Traffic Control Modernization
Appendix I: Objectives, Scope and
Methodology




Establishing the duration of all activities: The schedule should reflect
how long each activity will take to execute. In determining the duration of
each activity, the same rationale, data, and assumptions used for cost
estimating should be used for preparing the schedule. Furthermore, these
durations should be as short as possible and should have specific start
and end dates. Excessively long periods needed to execute an activity
should prompt further decomposition of the activity so that shorter
execution durations will result.

Integrating schedule activities horizontally and vertically: The
schedule should be horizontally integrated, meaning that it should link the
products and outcomes associated with already sequenced activities.
These links are commonly referred to as “hand-offs” and serve to verify
that activities are arranged in the right order to achieve aggregated
products or outcomes. The schedule should also be vertically integrated,
meaning that traceability exists among varying levels of activities and
supporting tasks and subtasks. Such mapping or alignment among levels
can enable different groups to work to the same master schedule.

Establishing the critical path for all activities: With the use of
scheduling software, the critical path—the longest-duration path through
the sequenced list of activities—should be identified. The establishment
of a project’s critical path is necessary for examining the effects of delays
in any activity along this path. Potential problems that may occur on or
near the critical path should also be identified and reflected in the
scheduling of the time for high-risk activities (see the next activity,
“Identifying float”).

Identifying reasonable float: The schedule should identify float—the
time that a predecessor activity can slip before the delay affects
successor activities—so that schedule flexibility can be determined. As a
general rule, activities along the critical path typically have the least
amount of float.

Conducting a schedule risk analysis: A schedule risk analysis uses a
good critical path method schedule and data about project schedule risks,
as well as Monte Carlo simulation techniques, to predict the level of
confidence in meeting a project’s completion date, the amount of time
contingency needed for a level of confidence, and the identification of
high-priority risks. This analysis should focus not only on critical path
activities but also on other schedule paths that may become critical. A
schedule/cost risk assessment recognizes the interrelationship between
schedule and cost and captures the risk that schedule durations and cost


Page 49                                 GAO-12-223 Air Traffic Control Modernization
Appendix I: Objectives, Scope and
Methodology




estimates may vary for a variety of reasons, including limited data,
optimistic estimating, technical challenges, lack of qualified personnel,
and other external factors. As a result, the baseline schedule should
include a buffer or a reserve of extra time. A reserve of extra time for
contingencies should be calculated by performing a schedule risk
analysis. As a general rule, the reserve should be held by the project
manager and applied as needed to those activities that take longer than
scheduled because of the identified risks. Reserves of time should not be
apportioned in advance to any specific activity since the risks that will
actually occur and the magnitude of their impact are not known in
advance.

Updating the schedule using logic and durations to determine the
dates: The schedule should use logic and durations in order to reflect
realistic start and completion dates for project activities. The schedule
should be continually monitored to determine when forecasted completion
dates differ from the planned dates. This information can be used to
determine whether schedule variances will affect downstream work.
Maintaining the integrity of the schedule logic is not only necessary to
reflect the project’s true status but is also required before conducting a
schedule risk analysis. The schedule should avoid logic overrides and
artificial constraint dates that are chosen to create a certain result on
paper. Individuals trained in critical path method scheduling should be
responsible for updating the schedule.

Based on our work, we determined the extent to which estimates and
schedules for the four projects we selected met each best practices
criterion:

•   Not Met—project officials provided no evidence that satisfies any
    portion of the criterion.

•   Minimally Met—project officials provided evidence that satisfies a
    small portion of the criterion.

•   Partially Met—project officials provided evidence that satisfies about
    half of the criterion.

•   Substantially Met—project officials provided evidence that satisfies a
    large portion of the criterion.

•   Met—project officials provided evidence that satisfies the entire
    criterion.



Page 50                                GAO-12-223 Air Traffic Control Modernization
Appendix I: Objectives, Scope and
Methodology




We conducted this performance audit from August 2010 to February 2012
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.




Page 51                             GAO-12-223 Air Traffic Control Modernization
Appendix II: Detailed Purpose and Status for
                                       Appendix II: Detailed Purpose and Status for
                                       30 Acquisitions GAO Reviewed



30 Acquisitions GAO Reviewed

                                       This appendix contains detailed information for 30 individual air traffic
                                       control programs. Each overview presents information and data that was
                                       provided by FAA. The overviews provide a description of the program and
                                       the cost and schedule status. The overviews are based on program office
                                       reported information as of August 2011. In most cases, we did not
                                       validate the data provided, but reviewed the data and performed various
                                       checks to determine they were reliable enough for our purposes.

Figure 3: Automatic Dependent Surveillance-Broadcast (ADS-B)




                                       Page 52                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 4: Advanced Technologies and Oceanic Procedures (ATOP)




                                      Page 53                                        GAO-12-223 Air Traffic Control Modernization
                                        Appendix II: Detailed Purpose and Status for
                                        30 Acquisitions GAO Reviewed




Figure 5: Airport Surveillance Radar (ASR) Model 11 Tech Refresh Segment 1




                                        Page 54                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 6: Aviation Surface Weather Observation Network (ASWON) and Automated Surface Observing System (ASOS) P31




                                      Page 55                                        GAO-12-223 Air Traffic Control Modernization
                                         Appendix II: Detailed Purpose and Status for
                                         30 Acquisitions GAO Reviewed




Figure 7: Air Traffic Control Radar Beacon Interrogator (ATCBI-6)




                                         Page 56                                        GAO-12-223 Air Traffic Control Modernization
                                       Appendix II: Detailed Purpose and Status for
                                       30 Acquisitions GAO Reviewed




Figure 8: Collaborative Air Traffic Management Technologies (CATMT), Work Packages 2 and 3 (WP2 and WP3)




                                       Page 57                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 9: En Route Communication Gateway (ECG) Tech Refresh




                                      Page 58                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 10: En Route Automation Modernization (ERAM)




                                      Page 59                                        GAO-12-223 Air Traffic Control Modernization
                                        Appendix II: Detailed Purpose and Status for
                                        30 Acquisitions GAO Reviewed




Figure 11: Integrated Display Systems (IDS) Replacement




                                        Page 60                                        GAO-12-223 Air Traffic Control Modernization
                                            Appendix II: Detailed Purpose and Status for
                                            30 Acquisitions GAO Reviewed




Figure 12: International Flight Inspection Aircraft (IFIA)




                                            Page 61                                        GAO-12-223 Air Traffic Control Modernization
                                        Appendix II: Detailed Purpose and Status for
                                        30 Acquisitions GAO Reviewed




Figure 13: Instrument Flight Procedure Automation (IFPA)




                                        Page 62                                        GAO-12-223 Air Traffic Control Modernization
                                        Appendix II: Detailed Purpose and Status for
                                        30 Acquisitions GAO Reviewed




Figure 14: Integrated Terminal Weather System (ITWS)




                                        Page 63                                        GAO-12-223 Air Traffic Control Modernization
                                     Appendix II: Detailed Purpose and Status for
                                     30 Acquisitions GAO Reviewed




Figure 15: Next Generation Air/Ground Communication System (NEXCOM) Segment 1a




                                     Page 64                                        GAO-12-223 Air Traffic Control Modernization
                                       Appendix II: Detailed Purpose and Status for
                                       30 Acquisitions GAO Reviewed




Figure16: Next Generation Weather Radar (NEXRAD) Dual Polarization




                                       Page 65                                        GAO-12-223 Air Traffic Control Modernization
                                       Appendix II: Detailed Purpose and Status for
                                       30 Acquisitions GAO Reviewed




Figure 17: Power Systems Sustained Support (PS3)




                                       Page 66                                        GAO-12-223 Air Traffic Control Modernization
                                         Appendix II: Detailed Purpose and Status for
                                         30 Acquisitions GAO Reviewed




Figure 18: Regulation and Certification Infrastructure for System Safety (RCISS) Segment 2




                                         Page 67                                        GAO-12-223 Air Traffic Control Modernization
                                         Appendix II: Detailed Purpose and Status for
                                         30 Acquisitions GAO Reviewed




Figure 19: Runway Status Lights (RWSL)




                                         Page 68                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 20: Standard Terminal Automation Replacement System (STARS) (TAMR Phase 1)




                                      Page 69                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 21: System Wide Information Management (SWIM) Segment 1




                                      Page 70                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 22: Terminal Automation Modernization and Replacement (TAMR)—Phase 2




                                      Page 71                                        GAO-12-223 Air Traffic Control Modernization
                                       Appendix II: Detailed Purpose and Status for
                                       30 Acquisitions GAO Reviewed




Figure 23: Tower Training Simulator Systems




                                       Page 72                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 24: Trajectory Management—Arrival Tactical Flow Time Based Flow Management




                                      Page 73                                        GAO-12-223 Air Traffic Control Modernization
                                       Appendix II: Detailed Purpose and Status for
                                       30 Acquisitions GAO Reviewed




Figure 25: Terminal Doppler Weather Radar (TDWR) Service Life Extension Program (SLEP)




                                       Page 74                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 26: Traffic Flow Management System (TFMS) Tech Refresh




                                      Page 75                                        GAO-12-223 Air Traffic Control Modernization
                                       Appendix II: Detailed Purpose and Status for
                                       30 Acquisitions GAO Reviewed




Figure 27: Ultra High Frequency (UHF) Radio Replacement




                                       Page 76                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 28: Next Generation Voice Recorder Replacement Program (VRRP)




                                      Page 77                                        GAO-12-223 Air Traffic Control Modernization
                                       Appendix II: Detailed Purpose and Status for
                                       30 Acquisitions GAO Reviewed




Figure 29: Voice Switching and Control Switching System (VSCS) Tech Refresh Phase 2




                                       Page 78                                        GAO-12-223 Air Traffic Control Modernization
                                     Appendix II: Detailed Purpose and Status for
                                     30 Acquisitions GAO Reviewed




Figure 30: Wide Area Augmentation System (WAAS)




                                     Page 79                                        GAO-12-223 Air Traffic Control Modernization
                                      Appendix II: Detailed Purpose and Status for
                                      30 Acquisitions GAO Reviewed




Figure 31: Weather and Radar Processor (WARP) Sustain




                                      Page 80                                        GAO-12-223 Air Traffic Control Modernization
                                     Appendix II: Detailed Purpose and Status for
                                     30 Acquisitions GAO Reviewed




Figure 32: Weather Camera Program (WCP)




                                     Page 81                                        GAO-12-223 Air Traffic Control Modernization
Appendix III: Assessments of Four FAA
                                            Appendix III: Assessments of Four FAA
                                            Program Cost Estimates



Program Cost Estimates

                                            This appendix provides the results of our analysis of the extent to which
                                            the processes and methodologies used to develop and maintain the four
                                            FAA cost estimates meet the characteristics of high-quality cost
                                            estimates. These characteristics incorporate the 12 steps consistently
                                            applied by cost-estimating organizations throughout the federal
                                            government and industry and considered best practices for developing
                                            cost estimates and that are listed in table 2 of the report. The following
                                            tables provide the detailed results of our analysis of the program cost
                                            estimates for Automatic Dependent Surveillance-Broadcast (ADS-B),
                                            Collaborative Air Traffic Management Technologies (CATMT), System
                                            Wide Information Management (SWIM), and Wide Area Augmentation
                                            System (WAAS). “Not met” means the program provided no evidence that
                                            satisfies any of the criteria. “Minimally met” means the program provided
                                            evidence that satisfies a small portion of the criterion. “Partially met”
                                            means the program provided evidence that satisfies about half of the
                                            criterion. “Substantially met” means the program provided evidence that
                                            satisfies a large portion of the criterion. “Fully met” means the program
                                            provided evidence that completely satisfies the criterion.

Table 8: GAO’s Analysis of the FAA’s ADS-B Cost Estimates

Characteristic of high-quality cost                Extent to which criterion
estimates/assessment criterion/explanation         was met                     Key examples of rationale for assessment
Well-documented                                    Partially met
•   Captures the source data used, the reliability                             •    Lists data sources, but does not provide
    of the data, and how the data were made                                         documentation of the source data, bringing into
    compatible with other data in the estimate.                                     question the reliability of the data. (Minimally
Data should be collected from primary sources.                                      meets.) Data are the foundation of every cost
The source, content, time, and units should be                                      estimate. Depending on data quality, an estimate
adequately documented. Data should also be                                          can range anywhere from a mere guess to a
analyzed to determine accuracy and reliability,                                     highly defensible cost position. Data are often in
and to identify cost drivers.                                                       many different forms and need to be adjusted
                                                                                    before being used. The cost estimator needs
                                                                                    information about the source and reliability of the
                                                                                    data in order to know whether the data collected
                                                                                    can be used directly or need to be modified.
•   Describes the calculations and the                                         •    Documentation does not fully explain how FAA
    methodology used to derive each element’s                                       derived estimated costs. For example, FAA relied
    cost.                                                                           on expert opinion to estimate several costs but
Documentation should describe what calculation                                      provided no historical data to back up the
methods are used, as well as how they were                                          opinions. There was also no supporting
applied, and explain any anomalies.                                                 information for the software cost estimates.
                                                                                    (Minimally meets.) Poorly documented cost
                                                                                    estimates can cause a program’s credibility to
                                                                                    suffer because the documentation cannot explain
                                                                                    the rationale for the methodology or the
                                                                                    calculations. Estimates that lack sufficient
                                                                                    documentation are not useful for updates or



                                            Page 82                                       GAO-12-223 Air Traffic Control Modernization
                                              Appendix III: Assessments of Four FAA
                                              Program Cost Estimates




Characteristic of high-quality cost                  Extent to which criterion
estimates/assessment criterion/explanation           was met                     Key examples of rationale for assessment
                                                                                      information sharing and can hinder understanding
                                                                                      and proper use.
•    Describes how the estimate was developed.                                   •    Cost calculations are described at a high level,
The data supporting the estimate should be                                            but the documentation does not provide enough
available and adequately documented so that the                                       detail so that a cost analyst unfamiliar with the
estimate can be easily documented to reflect                                          program could understand what was done and
actual costs or program changes.                                                      replicate it. Furthermore, we found an
                                                                                      inconsistency between dollar values stored in the
                                                                                      cost estimating tool and the Excel spreadsheets
                                                                                      used to report life-cycle costs. (Partially met.)
                                                                                      Without good documentation, management and
                                                                                      oversight officials will not be convinced that the
                                                                                      estimate is credible; supporting data, lessons
                                                                                      learned, and reasons why costs changed will not
                                                                                      be available for future use; questions about the
                                                                                      approach or data used to create the estimate
                                                                                      cannot be answered; and the scope of the
                                                                                      analysis cannot be thoroughly defined.
                                                                                 •    Technical details contained within the basis-of-
•    Discusses the technical baseline description.
                                                                                      estimate documentation are consistent with
A technical baseline description provides a                                           corresponding details in the technical baseline.
common definition of the program, including                                           (Substantially meets.)
detailed technical, program, and schedule
descriptions of the system, for a cost estimate to
be built on. The data in the technical baseline
should be consistent with the data used to
develop the cost estimate.
•    Provides evidnce of management review
                                                                                 •    The estimate was briefed to the Joint Resources
     and acceptance.
                                                                                      Council. The briefing included a discussion of the
There should be a briefing to management,                                             scope, justification, and cost of the program. It
including a clear explanation of how the cost                                         also contained an overview of the program’s
estimate was derived. Management’s                                                    technical requirements, life-cycle costs,
acceptance of the cost estimate should be                                             assumptions, and results of risk and sensitivity
documented.                                                                           analysis. (Met.)
Comprehensive                                        Substantially met
•     Includes all life-cycle costs.                                             •    Includes costs from fiscal year 2007 through fiscal
A life-cycle cost estimate provides a complete                                        year 2035 for both government and contractor
and structured accounting of all resources and                                        efforts across all phases of the program. (Met.)
associated cost elements required to develop,
produce, deploy, and sustain a particular
program. It should cover the program from its
inception through its retirement.
                                                                                 •    Reflects the current project schedule, but no single
•     Completely defines the program, reflects the
      current schedule, and is technically                                            technical baseline was provided and there was no
      reasonable.                                                                     evidence that the technical documents had been
                                                                                      updated to reflect changes. (Substantially met.)
The cost estimate should be based on a
documented technical baseline description,
which provides a common definition of the
program—including detailed technical, program,
and schedule descriptions of the system.



                                              Page 83                                        GAO-12-223 Air Traffic Control Modernization
                                              Appendix III: Assessments of Four FAA
                                              Program Cost Estimates




Characteristic of high-quality cost                  Extent to which criterion
estimates/assessment criterion/explanation           was met                     Key examples of rationale for assessment
•    Has a product-oriented work breakdown                                       •    Reflects the FAA standard work breakdown structure,
     structure, and is traceable to the program’s                                     which is not product-oriented; instead, it breaks work
     technical scope, at an appropriate level of                                      down into functional categories. While the cost
     detail.                                                                          estimate work breakdown structure matches the work
A work breakdown structure provides a basic                                           breakdown structure for earned value management, it
framework for a variety of related activities like                                    does not match the work breakdown structure for the
estimating costs, developing schedules,                                               schedule. Furthermore, there was no evidence that
identifying resources and potential risks, and                                        the work breakdown structure had been updated as
providing the means for measuring program                                             the program became better defined. (Partially met.)
status using earned value management. It is                                           Without a work breakdown structure, the program
product-oriented if it allows a program to track                                      lacks a framework to develop a schedule and cost
cost and schedule by defined deliverables, such                                       plan that can be used to easily track technical
as a hardware or software component.                                                  accomplishments. A standard product-oriented work
                                                                                      breakdown structure facilitates the tracking of
                                                                                      resource allocations and expenditures, which can
                                                                                      give an agency insight to reliably estimate the cost of
                                                                                      future similar programs.
                                                                                 •    While ground rules and assumptions were
•    Documents all cost-influencing ground rules                                      discussed, many of the assumptions did not include
     and assumptions.                                                                 supporting data. In addition, details supporting risk
                                                                                      assumptions were incomplete because the sources
Cost estimates are typically based on limited                                         for the risk ranges were not provided. (Partially met.)
information and therefore need to be bound by                                         Unless ground rules and assumptions are clearly
ground rules and assumptions. Ground rules are                                        documented, the cost estimate will not have a basis
a set of estimating standards that provide                                            for assessing potential risks. Furthermore, the
guidance and common definitions, while                                                estimate cannot be reconstructed when the original
assumptions are judgments about past, present,                                        estimators are no longer available.
or future conditions that may affect the estimate.
Accurate                                             Partially met
•     Produces unbiased results.                                                 •    The cost basis of estimate contains minimum,
Cost estimates should have an uncertainty                                             most likely, and maximum values to model
analysis, which determines where the estimate                                         uncertainties for each cost element and to provide
falls against the range of all possible costs.                                        a range of costs; however, no analysis has been
                                                                                      performed to determine the confidence level of
                                                                                      the estimate. (Partially met.) A cost estimate is
                                                                                      biased if the estimated work is overly
                                                                                      conservative or too optimistic. Unless the
                                                                                      estimate is based on an assessment of the most
                                                                                      likely costs and reflects the degree of uncertainty
                                                                                      given all of the risks considered, management will
                                                                                      not be able to make good decisions.
•     Is properly adjusted for inflation.                                        •    The cost estimate was adjusted for inflation
Cost data should be adjusted for inflation to                                         correctly, but the source data for the government
ensure that comparisons and projections are                                           salaries inflation index were not provided.
valid. Data should also be normalized to                                              (Substantially met.)
constant-year dollars to remove the effects of
inflation.
•     Contains few mistakes.                                                     •    We found no instances of incorrect formulas,
Results should be checked for accuracy, double                                        double-counted costs, or omitted costs.
counting, and omissions.                                                              Moreover, costs reported by fiscal year correctly
                                                                                      summed to their life-cycle totals. (Met.)




                                              Page 84                                        GAO-12-223 Air Traffic Control Modernization
                                              Appendix III: Assessments of Four FAA
                                              Program Cost Estimates




Characteristic of high-quality cost                   Extent to which criterion
estimates/assessment criterion/explanation            was met                     Key examples of rationale for assessment
•    Is regularly updated to reflect significant                                  •   There was no evidence that the cost estimate
     program changes.                                                                 was updated to reflect changes or actual costs to
The cost estimate should be updated to reflect                                        reflect the current program status. (Not met.) A
significant program changes, such as changes to                                       lack of cost estimate updates interferes with
schedules or other assumptions. Updates should                                        analysis of changes in program costs and hinders
also reflect actual costs so that the estimate                                        collection of cost and technical data to support
always reflects the current program status.                                           future estimates. The cost estimate should be
                                                                                      updated when the technical baseline changes;
                                                                                      otherwise, it will lack credibility. A properly
                                                                                      updated cost estimate can provide decision
                                                                                      makers with accurate information for assessing
                                                                                      alternative decisions.
•   Documents and explains variances between                                      •   Variances between planned and actual costs, and
    planned and actual costs.                                                         explanations for the variance, were documented
                                                                                      in earned value management data, but no
Variances between planned and actual costs
                                                                                      variance for total cost at completion was included
should be documented, explained, and reviewed.
                                                                                      in the cost estimate documentation. (Substantially
For any elements whose actual costs or
                                                                                      met.) Without a documented comparison between
schedules differ from the estimate, the estimate
                                                                                      the current estimate (updated with actual costs)
should discuss variances and lessons learned.
                                                                                      and the old estimate, cost estimators cannot
                                                                                      determine the level of variance between the two
                                                                                      estimates. That is, the estimators cannot see how
                                                                                      well they are estimating and how the program is
                                                                                      changing over time.
                                                                                  •   There is some evidence that the estimate was
•    Reflects cost-estimating experiences from
     comparable programs.                                                             based on a historical record of cost-estimating and
                                                                                      actual experiences from other analogous programs.
The estimate should be based on historical cost                                       However, the reliability, risks, and applicability of the
estimation data and actual experiences from                                           analogous data were not addressed. Also,
other comparable programs. These data should                                          comparisons were not made between ADS-B and
be reliable and relevant to the new program.                                          analogous programs. (Minimally met.) Historical
                                                                                      data provide the cost estimator with insight into
                                                                                      actual costs on similar programs, including any cost
                                                                                      growth that occurred after the original estimate. As a
                                                                                      result, historical data can be used to challenge
                                                                                      optimistic assumptions and bring more realism to a
                                                                                      cost estimate.
Credible:                                             Partially met
•   Includes a sensitivity analysis that identifies                               •   While a high-level sensitivity analysis was
    a range of possible costs based on varying                                        provided to the Joint Resources Council for the
    inputs.                                                                           equipage cost driver, other key cost drivers,
A sensitivity analysis examines how changes to                                        assumptions, and data inputs were not varied.
key assumptions and inputs affect the estimate.                                       (Minimally met.) Because uncertainty cannot be
The estimate should identify key cost drivers,                                        avoided, it is necessary to identify the cost
examine their parameters and assumptions, and                                         elements that represent the most risk. A
re-estimate the total cost by varying each                                            sensitivity analysis reveals how the cost estimate
parameter between its minimum and maximum                                             is affected by a change in a single assumption,
range.                                                                                which helps the cost estimator understand the
                                                                                      extent to which each variable affects the cost
                                                                                      estimate. Any sources of variation should be well-
                                                                                      documented and traceable.




                                              Page 85                                        GAO-12-223 Air Traffic Control Modernization
                                             Appendix III: Assessments of Four FAA
                                             Program Cost Estimates




Characteristic of high-quality cost                  Extent to which criterion
estimates/assessment criterion/explanation           was met                                  Key examples of rationale for assessment
•    Contains a risk and uncertainty analysis.                                                •   A risk analysis for each cost element was
A risk and uncertainty analysis recognizes the                                                    performed but the risk data were essentially the
potential for error and attempts to quantify it by                                                same for each element and there was no
identifying the effects of changing key cost                                                      documentation of correlations of risks between
drivers.                                                                                          the cost elements. (Substantially met.)
•    Includes cross-checking of major cost                                                    •   Contains no evidence that cross-checks were
     elements.                                                                                    performed. (Not met.) The main purpose of cross-
                                                                                                  checking is to determine whether alternative
A cross-check is done by using a different                                                        methods produce similar results. If so, then
method to see if it produces similar results.                                                     confidence in the estimate increases, leading to
Includes a comparison to an independent cost                                                      greater credibility.
estimate conducted by another organization.                                                   •   A rough order-of-magnitude estimate was
A second, independent cost estimate should be                                                     performed by another organization but addressed
performed by an organization outside of the                                                       only a part of the total program. (Partially met.) An
program office’s influence. It should be based on                                                 independent cost estimate is considered one of
the same technical baseline, ground rules, and                                                    the best and most reliable estimate validation
assumptions as the original estimate.                                                             methods. It provides an independent view of
                                                                                                  expected program costs that tests the program
                                                                                                  office’s estimate for reasonableness. Without an
                                                                                                  independent cost estimate, decisions makers will
                                                                                                  lack insight into a program’s potential costs
                                                                                                  because independent cost estimates frequently
                                                                                                  use different methods and are less burdened with
                                                                                                  organizational bias.
                                             Source: GAO analysis of FAA’s ADS-B cost estimate.




                                             Page 86                                                     GAO-12-223 Air Traffic Control Modernization
                                              Appendix III: Assessments of Four FAA
                                              Program Cost Estimates




Table 9: GAO’s Analysis of FAA’s CATMT Cost Estimates

Characteristic of high-quality cost                 Extent to which criterion
estimates/assessment criterion/explanation          was met                     Key examples of rationale for assessment
Well-documented:                                    Partially met
•    Captures the source data used, the reliability                             •     Lists data sources, the majority of which were
     of the data, and how the data were made                                          subject matter experts. No details on the
     compatible with other data in the estimate;                                      qualifications or background of these experts
Data should be collected from primary sources.                                        were provided, and there was no documentation
The source, content, time, and units should be                                        about the reliability of the data. (Partially met.)
adequately documented. The data should also                                           Data are the foundation of every cost estimate.
be analyzed to determine accuracy and                                                 Depending on data quality, an estimate can range
reliability, and to identify cost drivers.                                            anywhere from a mere guess to a highly
                                                                                      defensible cost position. Data are often in many
                                                                                      different forms and need to be adjusted before
                                                                                      being used. The cost estimator needs information
                                                                                      about the source and reliability of the data in
                                                                                      order to know whether the data collected can be
                                                                                      used directly or need to be modified.
                                                                                •     Documentation does not fully explain how FAA
•   Describes the calculations and the
                                                                                      derived estimated costs. For example, FAA relied
    methodology used to derive each element’s
                                                                                      on expert opinion to estimate several costs based
    cost.
                                                                                      on a percentage of other cost elements but
Documentation should describe what calculation                                        provided no historical data to back up the
methods are used, as well as how they were                                            opinions. There was also no supporting
applied, and explain any anomalies.                                                   information for the software cost estimates.
                                                                                      (Partially met.) Poorly documented cost estimates
                                                                                      can cause a program’s credibility to suffer
                                                                                      because the documentation cannot explain the
                                                                                      rationale for the methodology or the calculations.
                                                                                      Estimates that lack sufficient documentation are
                                                                                      not useful for updates or information sharing and
                                                                                      can hinder understanding and proper use.
•    Describes how the estimate was developed.                                  •     Cost calculations are described in detail for fiscal
                                                                                      year 2014, but documentation was missing for
The data supporting the estimate should be                                            fiscal years 2015 through 2022. As a result, the
available and adequately documented so that the                                       documentation does not provide enough detail so
estimate can be easily documented to reflect                                          that a cost analyst unfamiliar with the program
actual costs or program changes.                                                      could understand what was done and replicate it.
                                                                                      (Partially met.) Without good documentation,
                                                                                      management and oversight officials will not be
                                                                                      convinced that the estimate is credible;
                                                                                      supporting data, lessons learned, and reasons
                                                                                      why costs changed will not be available for future
                                                                                      use; questions about the approach or data used
                                                                                      to create the estimate cannot be answered; and
                                                                                      the scope of the analysis cannot be thoroughly
                                                                                      defined.
                                                                                •     Many inconsistencies were found between the
•   Discusses the technical baseline
    description.                                                                      cost basis of the estimate document and the cost
                                                                                      model. (Partially met.) Because the technical
A technical baseline description provides a                                           baseline is intended to serve as the basis for
common definition of the program, including                                           developing a cost estimate, it should be



                                              Page 87                                        GAO-12-223 Air Traffic Control Modernization
                                             Appendix III: Assessments of Four FAA
                                             Program Cost Estimates




Characteristic of high-quality cost                  Extent to which criterion
estimates/assessment criterion/explanation           was met                     Key examples of rationale for assessment
detailed technical, program, and schedule                                            discussed in the cost estimate documentation.
descriptions of the system, for a cost estimate to                                   Without a technical baseline, the cost estimate
be built on. The data in the technical baseline                                      will not be based on a comprehensive program
should be consistent with the cost estimate.                                         description and will lack specific information about
                                                                                     technical and program risks.
•    Provides evidence of management review                                      •   The estimate was presented to the Joint
     and acceptance.                                                                 Resources Council management during an
There should be a briefing to management,                                            investment decision briefing. The briefing
including a clear explanation of how the cost                                        included a program overview, life-cycle costs over
estimate was derived. Management’s                                                   time, risk analysis, budget justification, and
acceptance of the cost estimate should be                                            recommendation. The briefing did not include
documented.                                                                          detail on the estimating methodology for each
                                                                                     cost element; sensitivity, risk, and uncertainty
                                                                                     analysis; or an affordability analysis.
                                                                                     (Substantially met.)
Comprehensive:                                     Substantially met
•     Includes all life-cycle costs.                                             •   Includes costs from fiscal year 2009 through fiscal
A life-cycle cost estimate provides a complete                                       year 2022 for both government and contractor
and structured accounting of all resources and                                       efforts across all phases of the program.
associated cost elements required to develop,                                        However, some government labor costs for in-
produce, deploy, and sustain a particular                                            service management were not included.
program. It should cover the program from its                                        (Substantially met.)
inception through its retirement.
•     Completely defines the program, reflects the                               •   Reflects many detailed technical requirements,
      current schedule, and is technically                                           but no single technical baseline was provided.
      reasonable.                                                                    (Substantially met.)
The cost estimate should be based on a
documented technical baseline description,
which provides a common definition of the
program—including detailed technical, program,
and schedule descriptions of the system.
•     Has a product-oriented work breakdown                                      •   Reflects FAA’s standard work breakdown
      structure, and is traceable to the program’s                                   structure, which does not follow a product-oriented
      technical scope, at an appropriate level of                                    work breakdown structure. Instead, it breaks work
      detail.                                                                        down into functional categories. The cost estimate
                                                                                     work breakdown structure matches the schedule
A work breakdown structure provides a basic
                                                                                     work breakdown structure. However, we found no
framework for a variety of related activities like
                                                                                     evidence that the work breakdown structure had
estimating costs, developing schedules,
                                                                                     been updated as the program became better
identifying resources and potential risks, and
                                                                                     defined. (Substantially met.)
providing the means for measuring program
status using earned value management. It is
product-oriented if it allows a program to track
cost and schedule by defined deliverables, such
as a hardware or software component.
•     Documents all cost-influencing ground rules                                •   The estimate provides a limited set of cost-
      and assumptions.                                                               influencing ground rules and assumptions;
Cost estimates are typically based on limited                                        however, many of the assumptions did not
information and therefore need to be bound by                                        include supporting data. In addition, there was
ground rules and assumptions. Ground rules are                                       little evidence risks were identified if the
a set of estimating standards that provide                                           assumption did not hold. (Partially met.) Unless



                                             Page 88                                        GAO-12-223 Air Traffic Control Modernization
                                              Appendix III: Assessments of Four FAA
                                              Program Cost Estimates




Characteristic of high-quality cost                  Extent to which criterion
estimates/assessment criterion/explanation           was met                     Key examples of rationale for assessment
guidance and common definitions, while                                                ground rules and assumptions are clearly
assumptions are judgments about past, present,                                        documented, the cost estimate will not have a
or future conditions that may affect the estimate.                                    basis for assessing potential risks. Furthermore,
                                                                                      the estimate cannot be reconstructed when the
                                                                                      original estimators are no longer available.
Accurate:                                            Partially met
•     Produces unbiased results.                                                 •    The cost basis of estimate contains low, likely,
Cost estimates should have an uncertainty                                             and high estimates for software lines of code, but
analysis, which determines where the estimate                                         the estimate did not have a similar analysis for
falls against the range of all possible costs.                                        every assumption in the cost model. In addition,
                                                                                      we found no confidence level associated with the
                                                                                      estimate. (Partially met.) A cost estimate is biased
                                                                                      if the estimated work is overly conservative or too
                                                                                      optimistic. Unless the estimate is based on an
                                                                                      assessment of the most likely costs and reflects
                                                                                      the degree of uncertainty given all of the risks
                                                                                      considered, management will not be able to make
                                                                                      good decisions.
•     Is properly adjusted for inflation.                                        •    The cost estimate was adjusted for inflation
Cost data should be adjusted for inflation to                                         correctly and the source data for the inflation
ensure that comparisons and projections are                                           indexes were provided. (Met.)
valid. Data should also be normalized to
constant-year dollars to remove the effects of
inflation.
•     Contains few mistakes.                                                     •    While we found no apparent mistakes in the
Results should be checked for accuracy, double                                        calculations, as stated earlier, there were
counting, and omissions.                                                              inconsistencies in costs between the cost model
                                                                                      and the cost basis of estimate documentation.
                                                                                      (Partially met.) Validating that a cost estimate is
                                                                                      accurate requires thoroughly understanding and
                                                                                      investigating how the cost model was
                                                                                      constructed. Without access to a detailed cost
                                                                                      model and estimate details, calculations may not
                                                                                      be accurate or expressed consistently,
•    Is regularly updated to reflect significant                                 •    We found no evidence that the cost estimate was
     program changes.                                                                 updated to reflect changes or actual costs. (Not
The cost estimate should be updated to reflect                                        met.) A lack of cost estimate updates interferes
significant program changes, such as changes to                                       with analysis of changes in program costs and
schedules or other assumptions. Updates should                                        hinders collection of cost and technical data to
also reflect actual costs so that the estimate                                        support future estimates. The cost estimate
always reflects the current program status.                                           should be updated when the technical baseline
                                                                                      changes; otherwise, it will lack credibility. A
                                                                                      properly updated cost estimate can provide
                                                                                      decision makers with accurate information for
                                                                                      assessing alternative decisions.
•   Documents and explains variances between                                     •    There was no documentation of variances
    planned and actual costs.                                                         between planned and actual costs. (Not met.)
Variances between planned and actual costs                                            Without a documented comparison between the
should be documented, explained, and reviewed.                                        current estimate (updated with actual costs) and
For any elements whose actual costs or                                                the old estimate, the cost estimators cannot




                                              Page 89                                        GAO-12-223 Air Traffic Control Modernization
                                                Appendix III: Assessments of Four FAA
                                                Program Cost Estimates




Characteristic of high-quality cost                   Extent to which criterion
estimates/assessment criterion/explanation            was met                     Key examples of rationale for assessment
schedules differ from the estimate, the estimate                                        determine the level of variance between the two
should discuss variances and lessons learned.                                           estimates. That is, the estimators cannot see how
                                                                                        well they are estimating and how the program is
                                                                                        changing over time.
•    Reflects cost-estimating experiences from                                    •     There is some evidence that the estimate was
     comparable programs.                                                               based on a historical record of cost-estimating
The estimate should be based on historical cost                                         and actual experiences from other analogous
estimation data and actual experiences from                                             programs. For example, software costs were
other comparable programs. These data should                                            estimated using function points and the the
be reliable and relevant to the new program.                                            Constructive Cost Model (COCOMO). However,
                                                                                        no comparison was made between CATMT and
                                                                                        other comparable FAA programs for the purpose
                                                                                        of the validating the CATMT estimate. (Minimally
                                                                                        met.) Historical data provide the cost estimator
                                                                                        with insight into actual costs on similar programs,
                                                                                        including any cost growth that occurred after the
                                                                                        original estimate. As a result, historical data can
                                                                                        be used to challenge optimistic assumptions and
                                                                                        bring more realism to a cost estimate.
Credible:                                             Minimally met
•   Includes a sensitivity analysis that identifies                               •     A sensitivity analysis was provided on the
    a range of possible costs based on varying                                          discount rate used in the economic analysis, but
    inputs;                                                                             other key cost drivers like software design and
A sensitivity analysis examines how changes to                                          development, as well as assumptions and data
key assumptions and inputs affect the estimate.                                         inputs, were not varied. (Minimally met.) Because
The estimate should identify key cost drivers,                                          uncertainty cannot be avoided, it is necessary to
examine their parameters and assumptions, and                                           identify the cost elements that represent the most
re-estimate the total cost by varying each                                              risk. A sensitivity analysis reveals how the cost
parameter between its minimum and maximum                                               estimate is affected by a change in a single
range.                                                                                  assumption, which helps the cost estimator
                                                                                        understand the extent to which each variable
                                                                                        affects the cost estimate. Any sources of variation
                                                                                        should be well-documented and traceable.
•    Contains a risk and uncertainty analysis.                                    •     The documentation discusses some aspects of a
A risk and uncertainty analysis recognizes the                                          risk and uncertainty analysis, such as risk
potential for error and attempts to quantify it by                                      percents, but lacks some results, such as the
identifying the effects of changing key cost                                            confidence level for the cost estimate. (Partially
drivers.                                                                                met.) The program estimate should reflect the
                                                                                        degree of uncertainty, so that a level of
                                                                                        confidence can be provided to management
                                                                                        about the estimate. An estimate without risk and
                                                                                        uncertainty analysis is unrealistic because it does
                                                                                        not assess the variability in the cost estimate from
                                                                                        such effects as schedules slipping, missions
                                                                                        changing, and proposed solutions not meeting
                                                                                        users’ needs.
•   Cross-checking of major cost elements.                                        •     The documentation states that a function point
A cross-check is done by using a different                                              analysis and a historical cost estimation method
method to see if it produces similar results.                                           were used to cross-check the software cost
                                                                                        estimate. However, a comparison of the model
                                                                                        results was not provided, and we could not




                                                Page 90                                        GAO-12-223 Air Traffic Control Modernization
                                            Appendix III: Assessments of Four FAA
                                            Program Cost Estimates




Characteristic of high-quality cost                 Extent to which criterion
estimates/assessment criterion/explanation          was met                                 Key examples of rationale for assessment
                                                                                                 identify other examples of cross-checks.
                                                                                                 (Minimally met.) The main purpose of cross-
                                                                                                 checking is to determine whether alternative
                                                                                                 methods produce similar results. If so, then
                                                                                                 confidence in the estimate increases, leading to
                                                                                                 greater credibility.
•    A comparison to an independent cost                                                    •    There was no evidence that an independent cost
     estimate conducted by another organization.                                                 estimate was conducted. (Not met.) An
A second, independent cost estimate should be                                                    independent cost estimate is considered one of
performed by an organization outside of the                                                      the best and most reliable estimate validation
program office’s influence. It should be based on                                                methods. It provides an independent view of
the same technical baseline, ground rules, and                                                   expected program costs that tests the program
assumptions as the original estimate.                                                            office’s estimate for reasonableness. Without an
                                                                                                 independent cost estimate, decision makers will
                                                                                                 lack insight into a program’s potential costs
                                                                                                 because independent cost estimates frequently
                                                                                                 use different methods and are less burdened with
                                                                                                 organizational bias.
                                            Source: GAO analysis of FAA’s CATMT cost estimate.




                                            Page 91                                                    GAO-12-223 Air Traffic Control Modernization
                                            Appendix III: Assessments of Four FAA
                                            Program Cost Estimates




Table 10: GAO’s Analysis of FAA’s SWIM Cost Estimates

Characteristic of high-quality cost                  Extent to which criterion
estimates/assessment criterion/explanation           was met                     Key examples of rationale for assessment
Well-documented:                                   Substantially met
•   Captures the source data used, the reliability                               •   The documentation captures the majority of the
    of the data, and how the data were made                                          data sources used. However, it does not provide
    compatible with other data in the estimate.                                      specific backup of the data sources and does not
Data should be collected from primary sources.                                       capture the reliability or the accuracy of the data.
The source, content, time, and units should be                                       (Partially met.) Data are the foundation of every
adequately documented. Data should also be                                           cost estimate. Depending on data quality, an
analyzed to determine accuracy and reliability,                                      estimate can range anywhere from a mere guess
and to identify cost drivers.                                                        to a highly defensible cost position. Data are
                                                                                     often in many different forms and need to be
                                                                                     adjusted before being used. The cost estimator
                                                                                     needs information about the source and reliability
                                                                                     of the data in order to know whether the data
                                                                                     collected can be used directly or need to be
                                                                                     modified.
•   Describes the calculations and the
                                                                                 •   The documentation describes in detail the
    methodology used to derive each element’s
                                                                                     methodology, calculations, basis of data, and
    cost.
                                                                                     quantities used to determine most of the cost
Documentation should describe what calculation                                       estimate. However, there is little detail provided
methods are used, as well as how they were                                           about the cost elements that used the analogy
applied, and explain any anomalies.                                                  and parametric estimating methodologies.
                                                                                     (Substantially met.)
•    Describes how the estimate was developed.                                   •   The documentation provides enough detail so
The data supporting the estimate should be                                           that a cost analyst unfamiliar with the program
available and adequately documented so that the                                      could understand what was done and replicate it;
estimate can be easily documented to reflect                                         however, there was no sensitivity analysis
actual costs or program changes.                                                     performed. (Substantially met.)
•    Discusses the technical baseline description.                               •   The technical details contained within the basis of
A technical baseline description provides a                                          estimate are consistent with the corresponding
common definition of the program, including                                          details in the technical baseline, but there is no
detailed technical, program, and schedule                                            mapping between the basis of estimate, the
descriptions of the system, for a cost estimate to                                   technical baseline, and the life-cycle cost
be built on. The data in the technical baseline                                      estimate. (Substantially met.)
should be consistent with the cost estimate.
•    Provides evidence of management review                                      •   The estimate was presented to management
     and acceptance.                                                                 during a briefing and was accepted by the
There should be a briefing to management,                                            issuance of a Joint Resources Council record of
including a clear explanation of how the cost                                        decision. The briefing included a program
estimate was derived. Management’s acceptance                                        overview, a comparison between the 2007 and
of the cost estimate should be documented.                                           2009 estimates, a breakout of cost drivers, and a
                                                                                     business case analysis. In addition, a software
                                                                                     estimate briefing discussed the estimating
                                                                                     methodology used. However, more specific
                                                                                     items, such as data sources, risk information, and
                                                                                     a comparison to an independent cost estimate
                                                                                     were not provided. (Partially met.) A cost
                                                                                     estimate is not considered valid until
                                                                                     management has approved it. It is imperative that




                                            Page 92                                        GAO-12-223 Air Traffic Control Modernization
                                             Appendix III: Assessments of Four FAA
                                             Program Cost Estimates




Characteristic of high-quality cost                  Extent to which criterion
estimates/assessment criterion/explanation           was met                     Key examples of rationale for assessment
                                                                                     management understand how the estimate was
                                                                                     developed, including the risks associated with the
                                                                                     underlying data and methods.
Comprehensive:                                       Substantially met
•     Includes all life-cycle costs.                                             •   Includes all life-cycle costs for each of the seven
A life-cycle cost estimate provides a complete                                       SWIM Implementing Programs (SIP) from fiscal year
and structured accounting of all resources and                                       2011 through fiscal year 2033. However, costs for
associated cost elements required to develop,                                        fiscal year 2009 and fiscal year 2010 were not
produce, deploy, and sustain a particular                                            included because these costs were already
program. It should cover the program from its                                        baselined. The cost estimate also includes both
inception through its retirement.                                                    government and contractor efforts across all phases
                                                                                     of the program. (Substantially met.)
•    Completely defines the program, reflects the                                •   Reflects many detailed technical requirements,
     current schedule, and is technically                                            including scope and schedule requirements, but no
     reasonable.                                                                     single technical baseline was provided. Also, there
                                                                                     was no evidence that the technical baseline
The cost estimate should be based on a                                               requirements had been updated. (Substantially met.)
documented technical baseline description,
which provides a common definition of the
program—including detailed technical, program,
and schedule descriptions of the system.
•    Has a product-oriented work breakdown                                       •   Reflects FAA’s standard work breakdown structure,
     structure, and is traceable to the program’s                                    which does not follow a product-oriented work
     technical scope, at an appropriate level of                                     breakdown structure. Instead, it breaks work down
     detail.                                                                         into functional categories. The cost estimate work
A work breakdown structure provides a basic                                          breakdown structure also does not match either the
framework for a variety of related activities like                                   schedule or the earned value management work
estimating costs, developing schedules,                                              breakdown structure. In addition, we found no
identifying resources and potential risks, and                                       evidence that the work breakdown structure had
providing the means for measuring program                                            been updated as the program became better defined.
status using earned value management. It is                                          (Partially met.) Without a work breakdown structure,
product-oriented if it allows a program to track                                     the program lacks a framework to develop a
cost and schedule by defined deliverables, such                                      schedule and cost plan that can be used to easily
as a hardware or software component.                                                 track technical accomplishments. A standard,
                                                                                     product-oriented work breakdown structure facilitates
                                                                                     the tracking of resource allocations and expenditures,
                                                                                     which can give FAA insight into reliable estimates of
                                                                                     the cost of similar programs in the future.
•                                                                                •   The estimate documents all cost-influencing ground
     Documents all cost-influencing ground rules                                     rules and assumptions; however, in several
     and assumptions.                                                                instances, not all of the data sources supporting each
Cost estimates are typically based on limited                                        were provided. In addition, while there was a risk
information and therefore need to be bound by                                        adjustment section in the documentation, the majority
ground rules and assumptions. Ground rules are                                       of the risk adjustments were based on the judgment
a set of estimating standards that provide                                           of the SWIM program office team. (Partially met.)
guidance and common definitions, while                                               Unless ground rules and assumptions are clearly
assumptions are judgments about past, present,                                       documented, the cost estimate will not have a basis
or future conditions that may affect the estimate.                                   for assessing potential risks. Furthermore, the
                                                                                     estimate cannot be reconstructed when the original
                                                                                     estimators are no longer available.




                                             Page 93                                        GAO-12-223 Air Traffic Control Modernization
                                           Appendix III: Assessments of Four FAA
                                           Program Cost Estimates




Characteristic of high-quality cost                Extent to which criterion
estimates/assessment criterion/explanation         was met                     Key examples of rationale for assessment
Accurate:                                          Partially met
•     Produces unbiased results.                                               •   The cost basis of estimate contains minimum,
Cost estimates should have an uncertainty                                          most likely and maximum values to model
analysis, which determines where the estimate                                      uncertainties for each cost element. No historical
falls against the range of all possible costs.                                     data were provided to support any of the risk
                                                                                   adjustments, and no level of confidence was
                                                                                   associated with the estimate. (Minimally met.) A
                                                                                   cost estimate is biased if the estimated work is
                                                                                   overly conservative or too optimistic. Unless the
                                                                                   estimate is based on an assessment of the most
                                                                                   likely costs and reflects the degree of uncertainty
                                                                                   given all of the risks considered, its usefulness to
                                                                                   management for making decisions is limited.
•     Is properly adjusted for inflation.                                      •   The cost estimate was adjusted for inflation
Cost data should be adjusted for inflation to                                      correctly and the source data for the inflation
ensure that comparisons and projections are                                        indexes were provided for all elements except for
valid. Data should also be normalized to                                           government salaries. (Substantially met.)
constant-year dollars to remove the effects of
inflation.
•     Contains few mistakes.                                                   •   The estimate contains no mistakes, and we found
Results should be checked for accuracy, double                                     no evidence of double counting or omissions.
counting, and omissions.                                                           Additionally, the documentation, equations, and
                                                                                   results were consistent between the
                                                                                   documentation and the cost model. (Met.)
•    Is regularly updated to reflect significant                               •   There was no evidence that the cost estimate
     program changes.                                                              was recently updated to reflect changes or actual
The cost estimate should be updated to reflect                                     costs to reflect current program status. The cost
significant program changes, such as changes to                                    estimate was updated in 2009 from the original
schedules or other assumptions. Updates should                                     2007 estimate, which requested funding only for
also reflect actual costs so that the estimate                                     fiscal year 2009 through fiscal year 2010.
always reflects the current program status.                                        (Minimally met.) A lack of cost estimate updates
                                                                                   interferes with analysis of changes in program
                                                                                   costs and hinders collection of cost and technical
                                                                                   data to support future estimates. The cost
                                                                                   estimate should be updated when the technical
                                                                                   baseline changes; otherwise, it will lack
                                                                                   credibility. A properly updated cost estimate can
                                                                                   provide decision makers with accurate
                                                                                   information for assessing alternative decisions.
•   Documents and explains variances between                                   •   Earned value management system reports for July
    planned and actual costs.                                                      2010 through December 2010 contained actual
                                                                                   costs and variances. However, no recent earned
Variances between planned and actual costs
                                                                                   value management reports captured actual costs
should be documented, explained, and reviewed.
                                                                                   and variances. In addition, because the cost
For any elements whose actual costs or
                                                                                   estimate work breakdown structure did not match
schedules differ from the estimate, the estimate
                                                                                   the earned value management work breakdown
should discuss variances and lessons learned.
                                                                                   structure, it would be difficult to track variances from
                                                                                   the earned value management system back to the
                                                                                   cost estimate. (Minimally met.) Without a
                                                                                   documented comparison between the current
                                                                                   estimate (updated with actual costs) and the old



                                           Page 94                                        GAO-12-223 Air Traffic Control Modernization
                                                Appendix III: Assessments of Four FAA
                                                Program Cost Estimates




Characteristic of high-quality cost                   Extent to which criterion
estimates/assessment criterion/explanation            was met                     Key examples of rationale for assessment
                                                                                        estimate, cost estimators cannot determine the level
                                                                                        of variance between the two estimates. That is, the
                                                                                        estimators cannot see how well they are estimating
                                                                                        and how the program is changing over time.
                                                                                  •     There is some evidence that the estimate was
                                                                                        based on a historical record of cost estimating.
•    Reflects cost estimating experiences from                                          For example, per diem and hotel rates were
     comparable programs.                                                               obtained from a government website; however,
                                                                                        no such comparable data were provided for other
The estimate should be based on a historical
                                                                                        cost elements. (Minimally met.) Historical data
cost estimation data and actual experiences from
                                                                                        provide the cost estimator with insight into actual
other comparable programs. These data should
                                                                                        costs on similar programs, including any cost
be reliable and relevant to the new program.
                                                                                        growth that occurred after the original estimate.
                                                                                        As a result, historical data can be used to
                                                                                        challenge optimistic assumptions and bring more
                                                                                        realism to a cost estimate.
Credible:                                             Minimally met
•   Includes a sensitivity analysis that identifies                               •     While an independent evaluation includes a
    a range of possible costs based on varying                                          breakout of cost drivers as a percentage of total
    inputs.                                                                             cost, the cost estimate documentation does not
A sensitivity analysis examines how changes to                                          show evidence that a sensitivity analysis was
key assumptions and inputs affect the estimate.                                         performed. (Minimally met.) Because uncertainty
The estimate should identify key cost drivers,                                          cannot be avoided, it is necessary to identify the
examine their parameters and assumptions, and                                           cost elements that represent the most risk. A
re-estimate the total cost by varying each                                              sensitivity analysis reveals how the cost estimate
parameter between its minimum and maximum                                               is affected by a change in a single assumption,
range.                                                                                  which helps the cost estimator to understand the
                                                                                        extent to which each variable affects the cost
                                                                                        estimate. Any sources of variation should be well-
                                                                                        documented and traceable.
•    Contains a risk and uncertainty analysis.                                    •     The documentation discusses some aspects of a
A risk and uncertainty analysis recognizes the                                          risk and uncertainty analysis, and each element
potential for error and attempts to quantify it by                                      receives a risk adjustment. A risk briefing that
identifying the effects of changing key cost                                            showed cost risk ranges for all of the SIPs were
drivers.                                                                                also provided. However, the documentation did not
                                                                                        discuss what level of confidence was associated
                                                                                        with the cost estimate. (Partially met.).The
                                                                                        program estimate must reflect the degree of
                                                                                        uncertainty, so that a level of confidence can be
                                                                                        provided to management about the estimate. An
                                                                                        estimate without risk and uncertainty analysis is
                                                                                        unrealistic because it does not assess the
                                                                                        variability in the cost estimate from such effects as
                                                                                        schedules slipping, missions changing, and
                                                                                        proposed solutions not meeting users’ needs.
                                                                                  •     There was no evidence that cross-checks were
•   Includes cross-checking of major cost                                               performed. (Not met.) The main purpose of cross-
    elements.                                                                           checking is to determine whether alternative
A cross-check is done by using a different                                              methods produce similar results. If so, then
method to see if it produces similar results.                                           confidence in the estimate increases, leading to
                                                                                        greater credibility.




                                                Page 95                                       GAO-12-223 Air Traffic Control Modernization
                                            Appendix III: Assessments of Four FAA
                                            Program Cost Estimates




Characteristic of high-quality cost                  Extent to which criterion
estimates/assessment criterion/explanation           was met                                    Key examples of rationale for assessment
•    Contains a comparison to an independent                                                    •   There was no evidence that an independent cost
     cost estimate conducted by another                                                             estimate was conducted; however, there was an
     organization.                                                                                  independent assessment briefing that showed a
A second, independent cost estimate should be                                                       review of all SIP cost estimates. (Minimally met.)
performed by an organization outside of the                                                         An independent cost estimate is considered one
program office’s influence. It should be based on                                                   of the best and most reliable estimate validation
the same technical baseline, ground rules, and                                                      methods. It provides an independent view of
assumptions, as the original estimate.                                                              expected program costs that tests the program
                                                                                                    office’s estimate for reasonableness. Without an
                                                                                                    independent cost estimate, decisions makers will
                                                                                                    lack insight into a program’s potential costs
                                                                                                    because independent cost estimates frequently
                                                                                                    use different methods and are less burdened with
                                                                                                    organizational bias.
                                            Source: GAO analysis of FAA’s SWIM cost estimate.




                                            Page 96                                                       GAO-12-223 Air Traffic Control Modernization
                                                 Appendix III: Assessments of Four FAA
                                                 Program Cost Estimates




Table 11: GAO’s Analysis of FAA’s WAAS Cost Estimates

Characteristic of high-quality cost
estimates/assessment                              Extent to which criterion was
criterion/explanation                             met                           Key examples of rationale for assessment
Well-documented:                                  Substantially met
•    Captures the source data used, the                                            •     The documentation captures the majority of the
     reliability of the data, and how the data                                           data sources used; however, it does not capture
     were made compatible with other data                                                the reliability or the accuracy of the data. (Partially
     in the estimate.                                                                    met.) Data are the foundation of every cost
Data should be collected from primary                                                    estimate. Depending on data quality, an estimate
sources. The source, content, time, and                                                  can range anywhere from a mere guess to a highly
units should be adequately documented.                                                   defensible cost position. Data are often in many
Data should also be analyzed to determine                                                different forms and need to be adjusted before
accuracy and reliability, and to identify cost                                           being used. The cost estimator needs information
drivers.                                                                                 about the source and reliability of the data in order
                                                                                         to know whether the data collected can be used
                                                                                         directly or need to be modified.
                                                                                   •     The documentation describes in great detail the
                                                                                         methodology, calculations, basis of data, and
                                                                                         quantities used in the cost estimate. However,
•    Describes the calculations and the                                                  estimates for some elements were based on the
     methodology used to derive each                                                     judgment of subject matter experts. (Substantially
     element’s cost.                                                                     met.)
Documentation should describe what                                                 •     The documentation provides enough detail so that
calculation methods were used, as well as                                                a cost analyst unfamiliar with the program could
how they were applied, and explain any                                                   understand what was done and replicate it. There
anomalies.                                                                               was no sensitivity analysis performed, and some
•    Describes how the estimate was                                                      cost elements did not have a risk adjustment
     developed.                                                                          because they were considered ongoing efforts.
                                                                                         (Substantially met.)
The data supporting the estimate should be
available and adequately documented so                                             •     The technical details contained within the basis of
that the estimate can be easily documented                                               the estimate are consistent with the corresponding
to reflect actual costs or program changes.                                              details in the technical baseline. However, the cost
                                                                                         estimate does not reference the technical baseline
                                                                                         documents, and there is no mapping between the
•    Discusses the technical baseline                                                    basis of estimate, the technical baseline, and the
     description.                                                                        cost estimate. (Substantially met.)
A technical baseline description provides a                                        •     The estimate was presented to management
common definition of the program, including                                              during a briefing and was accepted by the
detailed technical, program, and schedule                                                issuance of a Joint Resources Council record of
descriptions of the system, for a cost                                                   decision. The briefing included a program overview
estimate to be built on. The data in the                                                 and specific details about the service area,
technical baseline should be consistent with                                             previous baselines, and a cost benefit analysis. In
the cost estimate.                                                                       addition, information about equipment
                                                                                         procurement, the schedule, and the spending plan
                                                                                         were also presented along with a risk analysis, an
•    Provides evidence of management
                                                                                         affordability analysis, and an independent
     review and acceptance.
                                                                                         evaluation. However, more specific items such as
There should be a briefing to management,                                                data sources for each cost element, a sensitivity
including a clear explanation of how the cost                                            analysis, an uncertainty analysis, and a
estimate was derived. Management’s                                                       comparison against an independent cost estimate
acceptance of the cost estimate should be                                                were not provided. (Substantially met.)



                                                 Page 97                                        GAO-12-223 Air Traffic Control Modernization
                                                Appendix III: Assessments of Four FAA
                                                Program Cost Estimates




Characteristic of high-quality cost
estimates/assessment                             Extent to which criterion was
criterion/explanation                            met                           Key examples of rationale for assessment
documented.
Comprehensive:                                    Substantially met
•     Includes all life-cycle costs.                                              •     Includes all life-cycle costs from fiscal year 2009
A life-cycle cost estimate provides a                                                   through fiscal year 2028 except sunk costs and
complete and structured accounting of all                                               disposition costs. The cost estimate covers both
resources and associated cost elements                                                  government and contractor efforts for the solution
required to develop, produce, deploy, and                                               development, implementation, and in-service
sustain a particular program. It should cover                                           management phases of the program. (Partially
the program from its inception through its                                              met.) A life-cycle cost estimate should
retirement.                                                                             encompass all past, present, and future costs for
                                                                                        every aspect of the program, regardless of
                                                                                        funding source—including all government and
                                                                                        contractor costs. Life-cycle cost estimates can
                                                                                        enhance decision making by allowing for design
                                                                                        trade-off studies to be evaluated on a total cost
                                                                                        basis as well as on a technical and performance
                                                                                        basis.
                                                                                  •     Reflects many detailed technical requirements,
•    Completely defines the program,                                                    including scope and schedule requirements, but
     reflects the current schedule, and is                                              no single technical baseline was provided.
     technically reasonable.                                                            Consistent with best practices, we found
The cost estimate should be based on a                                                  evidence that the technical baseline requirements
documented technical baseline description,                                              had been updated to reflect technical, program,
which provides a common definition of the                                               and schedule changes to the program.
program, including detailed technical,                                                  (Substantially met.)
program, and schedule descriptions of the                                         •     Reflects FAA’s standard work breakdown
system.                                                                                 structure, which does not follow a product-
•    Has a product-oriented work breakdown                                              oriented work breakdown structure. Instead, it
     structure, and is traceable to the                                                 breaks work down into functional categories. The
     program’s technical scope, at an                                                   cost estimate work breakdown structure also
     appropriate level of detail.                                                       does not match either the schedule or the earned
                                                                                        value management work breakdown structure;
A work breakdown structure provides a                                                   however, there was a mapping from the schedule
basic framework for a variety of related                                                work breakdown structure to the cost work
activities like estimating costs, developing                                            breakdown structure. In addition, we found
schedules, identifying resources and                                                    evidence that the work breakdown structure had
potential risks, and providing the means for                                            been updated as the program became better
measuring program status using earned                                                   defined. (Partially met.) Without a work
value management. It is product-oriented if it                                          breakdown structure, the program lacks a
allows a program to track cost and schedule                                             framework to develop a schedule and cost plan
by defined deliverables, such as a hardware                                             that can be used to easily track technical
or software component.                                                                  accomplishments. A standard, product-oriented
                                                                                        work breakdown structure facilitates the tracking
                                                                                        of resource allocations and expenditures, which
                                                                                        can give the agency a basis to reliably estimate
                                                                                        the cost of future similar programs.
•    Documents all cost-influencing ground                                        •     The estimate documents all cost-influencing
     rules and assumptions.                                                             ground rules and assumptions. In addition, there
Cost estimates are typically based on limited                                           were additional ground rules and assumptions
information and therefore need to be bound                                              found in the technical baseline’s authoritative




                                                Page 98                                       GAO-12-223 Air Traffic Control Modernization
                                               Appendix III: Assessments of Four FAA
                                               Program Cost Estimates




Characteristic of high-quality cost
estimates/assessment                            Extent to which criterion was
criterion/explanation                           met                           Key examples of rationale for assessment
by ground rules and assumptions. Ground                                                references. However, in several instances, not all
rules are a set of estimating standards that                                           of the data sources supporting the assumptions
provide guidance and common definitions,                                               were provided. In addition, while there was a risk
while assumptions are judgments about                                                  adjustment section in the documentation, the
past, present, or future conditions that may                                           majority of the risk adjustments were blank,
affect the estimate.                                                                   based on the assumption that the costs were well
                                                                                       defined by this point. (Substantially met.).
Accurate:                                        Partially met
•    Produces unbiased results.                                                  •     A minimal uncertainty analysis was performed for
Cost estimates should have an uncertainty                                              select cost elements with an 80 percent
analysis, which determines where the                                                   confidence level reported. No analysis was done
estimate falls against the range of all                                                to show the accuracy of the data sources used
possible costs.                                                                        for this analysis. (Minimally met.) A cost estimate
                                                                                       is biased if the work estimate is overly
Is properly adjusted for inflation.                                                    conservative or too optimistic. Unless the
Cost data should be adjusted for inflation to                                          estimate is based on an assessment of the most
ensure that comparisons and projections are                                            likely costs and reflects the degree of uncertainty
valid. Data should also be normalized to                                               given all of the risks considered, management will
constant-year dollars to remove the effects                                            not be able to make good decisions.
of inflation.                                                                    •     The cost estimate was adjusted for inflation
Contains few mistakes.                                                                 correctly, and the source data for the inflation
Results should be checked for accuracy,                                                indexes were provided. (Met.)
double counting, and omissions.                                                  •     The estimate contains no mistakes, and we found
•    Is regularly updated to reflect significant                                       no evidence of double counting or omissions.
     program changes.                                                                  Additionally, the documentation, equations, and
                                                                                       results were consistent between the
•    The cost estimate should be updated to                                            documentation and the cost model. (Met.)
     reflect significant program changes,
     such as changes to schedules or other                                       •     There was evidence that the cost estimate was
     assumptions. Updates should also                                                  updated to reflect changes in technical and
     reflect actual costs so that the estimate                                         program requirements. For example, the WAAS
     always reflects the current program                                               program has undergone four rebaselines from
     status.                                                                           fiscal year 1994 through fiscal year 2009. During
                                                                                       each rebaseline, the cost estimate was updated.
•    Documents and explains variances                                                  However, the cost model is not updated on a
     between planned and actual costs.                                                 frequent basis. It is only updated to reflect major
•    Variances between planned and actual                                              changes in technical and program requirements.
     costs should be documented, explained,                                            There was no evidence that the cost estimate
     and reviewed. For any elements whose                                              was updated with actual costs. (Partially met.) A
     actual costs or schedules differ from the                                         lack of cost estimate updates interferes with
     estimate, the estimate should discuss                                             analysis of changes in program costs and hinders
     variances and lessons learned.                                                    collection of cost and technical data to support
•    Reflects cost estimating experiences                                              future estimates. The cost estimate should be
     from comparable programs.                                                         updated when the technical baseline changes;
                                                                                       otherwise, it will lack credibility. A properly
The estimate should be based on a historical
                                                                                       updated cost estimate can provide decision
cost estimation data and actual experiences
                                                                                       makers with accurate information for assessing
from other comparable programs. These
                                                                                       alternative decisions.
data should be reliable and relevant to the
new program.                                                                     •     Earned value management system reports for
                                                                                       November 2008 through February 2009
                                                                                       contained actual costs and variances. However,




                                               Page 99                                       GAO-12-223 Air Traffic Control Modernization
                                             Appendix III: Assessments of Four FAA
                                             Program Cost Estimates




Characteristic of high-quality cost
estimates/assessment                          Extent to which criterion was
criterion/explanation                         met                           Key examples of rationale for assessment
                                                                                     no recent earned value management reports
                                                                                     captured actual costs and variances. In addition,
                                                                                     because the cost estimate work breakdown
                                                                                     structure did not match the earned value
                                                                                     management work breakdown structure, updating
                                                                                     the cost estimate would not be a straightforward
                                                                                     effort. (Partially met.) Without a documented
                                                                                     comparison between the current estimate
                                                                                     (updated with actual costs) and the old estimate,
                                                                                     cost estimators cannot determine the level of
                                                                                     variance between the two estimates. That is, the
                                                                                     estimators cannot see how well they are
                                                                                     estimating and how the program is changing over
                                                                                     time.
                                                                               •     There is some evidence that the estimate was
                                                                                     based on a historical record of cost estimating.
                                                                                     For example, software source lines of code were
                                                                                     estimated based on analogous historical data, but
                                                                                     there was no documentation to support these
                                                                                     costs. For the other cost elements, we did not
                                                                                     find sufficient documentation of historical costs
                                                                                     from comparable programs. (Minimally met.)
                                                                                     Historical data provide the cost estimator with
                                                                                     insight into actual costs on similar programs,
                                                                                     including any cost growth that occurred after the
                                                                                     original estimate. As a result, historical data can
                                                                                     be used to challenge optimistic assumptions and
                                                                                     bring more realism to a cost estimate.
Credible:                                     Partially met
•    Includes a sensitivity analysis that                                      •     Minimal evidence was provided to show that a
     identifies a range of possible costs                                            sensitivity analysis was provided for some cost
     based on varying inputs;                                                        elements, such as labor. However, we could not
A sensitivity analysis examines how                                                  determine if key cost drivers were identified
changes to key assumptions and inputs                                                across the entire program or if input ranges were
affect the estimate. The estimate should                                             varied. (Minimally met.) Because uncertainty
identify key cost drivers, examine their                                             cannot be avoided, it is necessary to identify the
parameters and assumptions, and re-                                                  cost elements that represent the most risk. A
estimate the total cost by varying each                                              sensitivity analysis reveals how the cost estimate
parameter between its minimum and                                                    is affected by a change in a single assumption,
maximum range.                                                                       which helps the cost estimator to understand the
                                                                                     extent to which each variable affects the cost
                                                                                     estimate. Any sources of variation should be well-
                                                                                     documented and traceable.
•    A risk and uncertainty analysis.                                          •     The documentation discusses some aspects of a
A risk and uncertainty analysis recognizes                                           risk and uncertainty analysis, such as a risk
the potential for error and attempts to                                              adjustment; however, only a few (4 out of 40) of
quantify it by identifying the effects of                                            the cost elements contained detailed
changing key cost drivers.                                                           documentation to support the risk adjustments.
                                                                                     The majority of the cost elements (36 out of 40)
                                                                                     did not have any risk adjustments applied. In
                                                                                     addition, the documentation did not discuss what



                                             Page 100                                      GAO-12-223 Air Traffic Control Modernization
                                                Appendix III: Assessments of Four FAA
                                                Program Cost Estimates




Characteristic of high-quality cost
estimates/assessment                              Extent to which criterion was
criterion/explanation                             met                           Key examples of rationale for assessment
                                                                                                    level of confidence was associated with the cost
                                                                                                    estimate. (Partially met.) The program estimate
                                                                                                    must reflect the degree of uncertainty, so that a
                                                                                                    level of confidence can be provided to
                                                                                                    management about the estimate. An estimate
                                                                                                    without risk and uncertainty analysis is unrealistic
                                                                                                    because it does not assess the variability in the
                                                                                                    cost estimate from such effects as schedules
                                                                                                    slipping, missions changing, and proposed
                                                                                                    solutions not meeting users’ needs.
•   Cross-checking of major cost elements.                                                      •   There was minimal evidence that cross-checks
A cross-check is done by using a different                                                          were identified. During the fiscal year 2004
method to see if it produces similar results.                                                       rebaseline, the basis of estimate was reviewed by
                                                                                                    several independent staff, but there was no
                                                                                                    evidence to supporting specific elements being
                                                                                                    cross-checked. (Minimally met.) The main
                                                                                                    purpose of cross-checking is to determine
                                                                                                    whether alternative methods produce similar
                                                                                                    results. If so, then confidence in the estimate
                                                                                                    increases, leading to greater credibility.
•    A comparison to an independent cost                                                        •   There was no evidence that an independent cost
     estimate conducted by another                                                                  estimate was conducted. However, as stated
     organization.                                                                                  earlier, there was an independent review during
A second, independent cost estimate should                                                          the fiscal year 2004 rebaseline; since that time,
be performed by an organization outside of                                                          the cost model has been reviewed several times.
the program office’s influence. It should be                                                        For example, in 2007 an independent evaluation
based on the same technical baseline,                                                               analyzed eight cost issues. (Partially met.) An
ground rules, and assumptions, as the                                                               independent cost estimate is considered one of
original estimate.                                                                                  the best and most reliable estimate validation
                                                                                                    methods. It provides an independent view of
                                                                                                    expected program costs that tests the program
                                                                                                    office’s estimate for reasonableness. Without an
                                                                                                    independent cost estimate, decisions makers will
                                                                                                    lack insight into a program’s potential costs
                                                                                                    because independent cost estimates frequently
                                                                                                    use different methods and are less burdened with
                                                                                                    organizational bias.
                                                Source: GAO analysis of FAA’s WAAS cost estimate.




                                                Page 101                                                  GAO-12-223 Air Traffic Control Modernization
Appendix IV: Assessments of Four FAA
                                          Appendix IV: Assessments of Four FAA
                                          Program Schedules



Program Schedules

                                          This appendix provides the results of our analysis of the extent to which
                                          the processes and methodologies used to develop and maintain four FAA
                                          integrated master schedules meet nine best practices associated with
                                          effective schedule estimating. The following tables provide the detailed
                                          results of our analyses of the schedules for the Automatic Dependent
                                          Surveillance-Broadcast (ADS-B), Collaborative Air Traffic Management
                                          Technologies (CATMT), System Wide Information Management (SWIM),
                                          and Wide Area Augmentation System (WAAS) programs compared to the
                                          nine best practices.

                                          “Not met” means the program provided no evidence that satisfies any
                                          portion of the criterion. “Minimally met” means the program provided
                                          evidence that satisfies a small portion of the criterion. “Partially met”
                                          means the program provided evidence that satisfies about half of the
                                          criterion. “Substantially met” means the program provided evidence that
                                          satisfies a large portion of the criterion. “Fully met” means the program
                                          provided evidence that satisfies the entire criterion.

Table 12: Extent to Which the ADS-B Schedule Met Best Practices

                                                              Extent to which
Best practice           Explanation                           criterion was met   GAO analysis
1.   Capturing all      The schedule should reflect all        Partially met      Our analysis found that detailed activities
     activities         activities as defined in the project’s                    within the schedule are mapped to a four-
                        work breakdown structure, which                           level work breakdown structure, an
                        defines in detail the work                                organization breakdown structure, and
                        necessary to accomplish a                                 unique control account numbers. Activities
                        project’s objectives—including                            are further mapped to service areas, control
                        activities to be performed by both                        account managers, and project leaders.
                        the owner and the contractors.                            Level-of-effort activities are included in the
                                                                                  schedule and are clearly marked as such. In
                                                                                  addition, several risk mitigation activities are
                                                                                  included in the schedule. For example,
                                                                                  activities associated with a contingency plan
                                                                                  for the 2011 Mississippi River flood are
                                                                                  included.
                                                                                  The government owns and maintains the
                                                                                  program schedule for the ADS-B program,
                                                                                  which covers all government and nonprime
                                                                                  contract work through fiscal year 2013. The
                                                                                  schedule is planned in 6-month rolling wave
                                                                                  periods. Officials stated that they have not
                                                                                  requested a detailed schedule deliverable
                                                                                  from the prime contractor because the
                                                                                  contractor portion represents work
                                                                                  performed in the fixed-price subscription
                                                                                  phase. The contractor owns a separate
                                                                                  schedule that is not a required deliverable




                                          Page 102                                GAO-12-223 Air Traffic Control Modernization
                                        Appendix IV: Assessments of Four FAA
                                        Program Schedules




                                                           Extent to which
Best practice         Explanation                          criterion was met   GAO analysis
                                                                               to the government. The government
                                                                               integrated master schedule includes prime
                                                                               contractor touch points that are updated
                                                                               monthly according to the contractor’s
                                                                               progress. However, without fully integrating
                                                                               all contractor and government activities, we
                                                                               cannot guarantee that the schedule has
                                                                               adequately captured all key activities
                                                                               necessary for the program’s completion.
                                                                               Officials stated that work needs to be
                                                                               accomplished in and beyond fiscal year
                                                                               2014, but the program schedule does not
                                                                               contain long-term planning activities beyond
                                                                               the current FAA-approved baseline. Effort
                                                                               from fiscal year 2015 to fiscal year 2020 is
                                                                               not included in the schedule because the
                                                                               program has no budget until Joint
                                                                               Resources Council approval of the next
                                                                               baseline. However, officials noted that the
                                                                               prime contract extends to 2025. Relevant
                                                                               guidance states that a comprehensive
                                                                               schedule should reflect all activities for a
                                                                               project and recognizes that there can be
                                                                               uncertainties and unknown factors in
                                                                               schedule estimates due to, among other
                                                                               things, limited data.
2.   Sequencing all   The schedule should be planned         Minimally met     Our analysis found 207 remaining activities,
     activities       so that critical project dates can be                    or work that remains to be accomplished
                      met. To meet this objective,                             (19 percent), are missing predecessor logic
                      activities need to be logically                          and 289 remaining activities (27 percent)
                      sequenced—that is, listed in the                         are missing successor logic. These counts
                      order in which they are to be                            include 50 remaining activities (5 percent)
                      carried out. In particular, activities                   that are missing both predecessor and
                      that must be completed before                            successor logic.
                      other activities can begin                               In addition, our analysis found a number of
                      (predecessor activities), as well as                     activities with dangling logic. Each
                      activities that cannot begin until                       activity—other than the start and finish
                      other activities are completed                           milestones—must have its start date
                      (successor activities), should be                        driven by a previous activity, and each
                      identified. This helps ensure that                       activity must drive the start date of another
                      interdependencies among activities                       activity. Tasks with dangling logic can
                      that collectively lead to the                            either carry on indefinitely without affecting
                      accomplishment of events or                              downstream activities (finish date has no
                      milestones can be established and                        successor) or must start earlier in order to
                      used as a basis for guiding work                         finish on time (start date has no
                      and measuring progress.                                  predecessor). The schedule includes 372
                                                                               remaining activities, or 34 percent, with
                                                                               dangling successors.
                                                                               The reliability of the schedule is also
                                                                               hindered by the presence of lags and
                                                                               leads (negative lags).




                                        Page 103                               GAO-12-223 Air Traffic Control Modernization
                              Appendix IV: Assessments of Four FAA
                              Program Schedules




                                                Extent to which
Best practice   Explanation                     criterion was met    GAO analysis
                                                                     Lags represent the passing of time
                                                                     between activities but are often misused to
                                                                     put activities on a specific date or to insert
                                                                     a buffer for risk. In particular, 22 percent of
                                                                     the remaining activities are affected by at
                                                                     least one lag or lead. Lags should be
                                                                     justified because they cannot have risk or
                                                                     uncertainty. Furthermore, lags often
                                                                     represent work or a delay whose duration
                                                                     is unknown, yet are hard-coded into the
                                                                     schedule. Using lags rather than activities
                                                                     to account for work or to insert a buffer for
                                                                     risks causes problems because lags
                                                                     persist even when activities are delayed.
                                                                     As a result, when activities take longer
                                                                     than planned, the lags may actually distort
                                                                     the true logic of the schedule. Negative
                                                                     lags are generally not valid because
                                                                     negative time is not demonstrable;
                                                                     additionally, a lead implies the ability to
                                                                     predict the finish date of a predecessor
                                                                     activity with certainty. In general, leads can
                                                                     be replaced with straightforward finish-to-
                                                                     start logic once tasks are broken down
                                                                     further.
                                                                     FAA officials said that there is broken and
                                                                     incomplete logic for the detail planning
                                                                     period (Wave “G”) that we assessed. They
                                                                     stated that the schedule we evaluated was
                                                                     in transition because of ongoing
                                                                     negotiations with the prime contractor over
                                                                     ADS-B service volume.
                                                                     Having all interdependencies between
                                                                     tasks identified is necessary for the
                                                                     schedule to properly calculate dates and
                                                                     predict changes in the future. Without the
                                                                     correct linkages, tasks that slip early in the
                                                                     schedule do not transmit delays to tasks
                                                                     that should depend on them. When this
                                                                     happens, the schedule will not provide a
                                                                     sufficient basis for understanding the
                                                                     program as a whole, and users of the
                                                                     schedule will lack confidence in the dates
                                                                     and the critical path. Unless complete
                                                                     network logic is established, the schedule
                                                                     cannot predict impacts on the project’s
                                                                     planned finish date of, among other things,
                                                                     misallocated resources, delayed tasks,
                                                                     external events, and unrealistic deadlines.




                              Page 104                               GAO-12-223 Air Traffic Control Modernization
                                           Appendix IV: Assessments of Four FAA
                                           Program Schedules




                                                              Extent to which
Best practice             Explanation                         criterion was met   GAO analysis
3.   Assigning resources The schedule should reflect what     Substantially met   The ADS-B integrated master schedule is
     to all activities   resources (e.g., labor, materials,                       resource-loaded at the work package level
                         and overhead) are needed to do                           by resource name, and contains
                         the work, whether all required                           approximately 250 resources, including
                         resources will be available when                         both government and nongovernment full-
                         needed, and whether any funding                          time-equivalents. Each has responsibility
                         or time constraints exist.                               for managing the resources at that level.
                                                                                  The program office leadership teams
                                                                                  define what work will be done in each
                                                                                  control account at a high level and which
                                                                                  resources are available. The control
                                                                                  account manager is then responsible for
                                                                                  assigning resources at the work package
                                                                                  level. Once the rolling wave planning is
                                                                                  approved, resources are assigned
                                                                                  accordingly with resource leveling
                                                                                  occurring outside of the schedule. Officials
                                                                                  use the integrated master schedule to
                                                                                  track all resources that are not related to
                                                                                  the prime contract (approximately 70
                                                                                  percent of the baseline work).
                                                                                  Allocation reports are run during the initial
                                                                                  rolling wave planning process to ensure
                                                                                  resources are neither underallocated nor
                                                                                  overallocated, but are not assessed
                                                                                  monthly to track resource allocations along
                                                                                  with schedule progress updates. However,
                                                                                  allocations are assessed whenever a
                                                                                  change request is initiated to ensure
                                                                                  resources are available for these additional
                                                                                  activities.
4.   Establishing the     The schedule should realistically   Substantially met   Our analysis found that approximately 75
     duration of all      reflect how long each activity will                     percent of the activities within the detail
     activities           take to execute. In determining the                     planning period of the schedule had
                          duration of each activity, the same                     original durations of 44 days or less, which
                          rationale, historical data, and                         generally meets best practices. Our
                          assumptions used for cost                               analysis also found that all durations are
                          estimating should be used.                              consistently estimated in days and adhere
                          Durations should be as short as                         to a standard 5-day workweek that
                          possible and have specific start                        accounts for holidays. Our analysis found
                          and end dates. The schedule                             no activities scheduled to begin on a
                          should be continually monitored to                      weekend. Program officials stated that
                          determine when forecasted                               long-duration planning packages exist in
                          completion dates differ from                            the schedule for the period fiscal year
                          planned dates; this information can                     2012 through fiscal year 2014 and that
                          be used to determine whether                            they hold high-level resources and budget
                          schedule variances will affect                          meetings until officials are able to plan
                          subsequent work.                                        future rolling wave periods.
                                                                                  Officials stated that duration estimates are
                                                                                  based on estimates from responsible
                                                                                  control account managers who follow
                                                                                  earned value management guidance.



                                           Page 105                               GAO-12-223 Air Traffic Control Modernization
                                               Appendix IV: Assessments of Four FAA
                                               Program Schedules




                                                                    Extent to which
Best practice                Explanation                            criterion was met   GAO analysis
                                                                                        However, these estimates are not formally
                                                                                        justified or documented in a basis of
                                                                                        estimate. The schedule aligns to an
                                                                                        earned value management system, and
                                                                                        the earned value technique used for each
                                                                                        activity is stored in the schedule.
5.   Integrating the         The schedule should be                 Partially met       The schedule is vertically integrated, with
     schedule                horizontally integrated, meaning                           low-level activities being traceable to
     horizontally and        that it should link products and                           higher-level summary activities in the work
     vertically              outcomes associated with other                             breakdown structure hierarchy. However,
                             sequenced activities. These links                          our analysis determined that the schedule
                             are commonly referred to as                                is not horizontally integrated because of
                             “hand-offs” and serve to verify that                       the issues described in Best Practice 2.
                             activities are arranged in the right                       Officials stated that all logic is not in place
                             order to achieve aggregated                                for the current detail planning period.
                             products or outcomes. The                                  Instead, the scheduler has used a
                             schedule should also be vertically                         combination of soft constraints, lags, and
                             integrated, meaning that the dates                         out-of-sequence updates in an attempt to
                             for starting and completing                                match as closely as possible actual
                             activities in the integrated master                        execution of activities while the
                             schedule should be aligned with                            deployment phase is under negotiation.
                             the dates for supporting tasks and                         Horizontal integration demonstrates that
                             subtasks. Such mapping or                                  the overall schedule is rational, planned in
                             alignment among levels enables                             a logical sequence, accounts for the
                             different groups to work to the                            interdependencies of activities, and
                             same master schedule.                                      provides a way to evaluate current status.
                                                                                        Therefore, if the schedule lacks horizontal
                                                                                        integration, the effects of slipped tasks on
                                                                                        downstream work cannot be determined
                                                                                        and there will be little confidence in the
                                                                                        calculated dates or critical path.
6.   Establishing the        Scheduling software should be        Not met               Officials stated that there is no critical path
     critical path for all   used to identify the critical path,                        in the current schedule because of the
     activities              which represents the chain of                              incomplete logic noted in Best Practice 2.
                             dependent activities with the                              Officials stated that once the program
                             longest total duration. Establishing                       began deploying service volumes, a critical
                             a project’s critical path is                               path was difficult to derive because the
                             necessary to examine the effects                           contractor has the authority to implement
                             of any activity slipping along this                        service volumes in any order it chooses.
                             path. Potential problems along or                          However, now that negotiations for service
                             near the critical path should also                         volumes are complete, program officials
                             be identified and reflected in                             plan to begin identifying a critical path to all
                             scheduling the duration of high-risk                       Office of Management and Budget (OMB)
                             activities.                                                Exhibit 300 (annual performance report)
                                                                                        milestones.
                                                                                        Without a valid program critical path,
                                                                                        management cannot determine which
                                                                                        slipped tasks will have detrimental effects
                                                                                        on the project finish date and it cannot
                                                                                        determine if float within the schedule can
                                                                                        be used to mitigate critical tasks by




                                               Page 106                                 GAO-12-223 Air Traffic Control Modernization
                                          Appendix IV: Assessments of Four FAA
                                          Program Schedules




                                                             Extent to which
Best practice           Explanation                          criterion was met   GAO analysis
                                                                                 reallocating resources from tasks that can
                                                                                 safely slip to tasks that must be completed
                                                                                 on time. Until the schedule can produce a
                                                                                 true critical path, the program office will not
                                                                                 be positioned to provide reliable timeline
                                                                                 estimates and it will not be able to identify
                                                                                 when problems or changes may occur and
                                                                                 what effect they may have on downstream
                                                                                 work.
7.   Identifying        The schedule should identify the    Not met              Officials stated that because of the missing
     reasonable float   float—the amount of time by which                        logic in the current detail planning period,
                        a predecessor activity can slip                          the float represented in the schedule is not
                        before the delay affects successor                       realistic. Our analysis confirmed this, as
                        activities—so that a schedule’s                          we found many activities within the
                        flexibility can be determined. As a                      schedule with extremely high float; that is,
                        general rule, activities along the                       unrealistically high float that is greater than
                        critical path have the least float.                      199 days. Specifically, there are 12 (1
                        Total float is the total amount of                       percent) in-progress activities and 745
                        time by which an activity can be                         planned activities (69 percent) that are
                        delayed without delaying the                             able to slip more than 199 days without
                        project’s completion, if everything                      affecting the end date of the program.
                        else goes according to plan.                             Program officials stated they do not yet
                                                                                 monitor float in the schedule because this
                                                                                 particular version of the ADS-B schedule is
                                                                                 in transition during deployment
                                                                                 negotiations with the prime contractor.
8.   Conducting a       A schedule risk analysis (SRA)         Minimally met     Officials stated that a schedule risk
     schedule risk      should be performed using                                analysis is not part of the rolling wave
     analysis           statistical techniques to predict the                    planning process. They perform
                        level of confidence in meeting a                         uncertainty analysis on the cost estimate,
                        project’s completion date. This                          but do not implement it on the schedule.
                        analysis focuses not only on critical                    Our analysis found this practice to be
                        path activities but also on activities                   minimally met because, as noted in Best
                        near the critical path, since they                       Practice 1, risk mitigation activities are
                        can affect the project’s status.                         entered into the schedule as baseline
                                                                                 change requests. The program scheduler
                                                                                 does not mark these as risk mitigation
                                                                                 activities per se, but the documentation
                                                                                 associated with a baseline change request
                                                                                 will list the additional activities. An example
                                                                                 of a risk mitigation activity is “Boxer
                                                                                 Contingency Plan,” which was added for
                                                                                 the 2011 flooding of the Mississippi River.




                                          Page 107                               GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                                       Extent to which
Best practice               Explanation                                criterion was met   GAO analysis
9.   Updating the           The schedule should be               Substantially met         Our analysis found that the schedule has a
     schedule using logic   continuously updated using logic                               valid and current status date. Our analysis
     and durations to       and durations to determine realistic                           also found there are no activities with start
     determine dates        start and completion dates for                                 or finish dates in the past that are missing
                            program activities. The schedule                               actual start and actual finish dates, and
                            should be analyzed continuously                                there are no activities with actual start and
                            for variances to determine when                                actual finish dates in the future.
                            forecasted completion dates differ                             Contractor updates are received by each
                            from planned dates. This analysis                              control account manager in the form of
                            is especially important for those                              expected finish dates, so the program
                            variations that affect activities                              scheduler is aware when a task is running
                            identified as being in a project’s                             late. Because the contractor has the
                            critical path and can influence a                              authority to deploy service volumes in any
                            scheduled completion date.                                     order, officials are not overly concerned
                                                                                           about the effect of a late task on the
                                                                                           program as a whole. They stated that the
                                                                                           control account managers know each
                                                                                           service volume negotiated deadline and
                                                                                           are aware of the effects of the slipped
                                                                                           tasks. However, too many activities
                                                                                           slipping into the future may create a bow-
                                                                                           wave of effort that the contractor may not
                                                                                           have the resources to complete near the
                                                                                           expected deployment completion date in
                                                                                           fiscal year 2013. In addition, activities
                                                                                           should be updated using actual duration or
                                                                                           effort and estimates of remaining duration
                                                                                           or effort. The scheduling software will
                                                                                           calculate the percentage of completed
                                                                                           activities and a forecasted finish date
                                                                                           based on actual effort and planned effort
                                                                                           remaining.
                                                                                           Our analysis found 60 activities (6 percent)
                                                                                           as occurring out of sequence. That is,
                                                                                           activities have started before their
                                                                                           predecessor was completed, contrary to
                                                                                           the network logic. Out-of-sequence
                                                                                           progress should be corrected, either by
                                                                                           changing the logical relationships to reflect
                                                                                           what has actually happened or stopping
                                                                                           work on the successor activity until the
                                                                                           predecessor activity is complete (called
                                                                                           “retained logic”). FAA officials stated they
                                                                                           expect many activities to be done out of
                                                                                           sequence as they adjust to the negotiated
                                                                                           deployment dates with the contractor. In
                                                                                           future rolling waves, as logic is put in place
                                                                                           and negotiated dates are firm, they expect
                                                                                           less out-of-sequence logic.
                                             Source: GAO analysis of ADS-B schedule.




                                             Page 108                                      GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




Table 13: Extent to Which the CATMT Schedule Met Best Practices

                                                               Extent to which
Best practice         Explanation                              criterion was met    GAO analysis
1.   Capturing all    The schedule should reflect all         Substantially met     The scope of work for the CATMT project, which
     activities       activities as defined in the project’s                        has no defined end state, is defined in 6-month
                      work breakdown structure, which                               increments. The schedule is owned and
                      defines in detail the work necessary                          maintained by the prime contractor, who is
                      to accomplish a project’s objectives—                         responsible for capturing all activities authorized in
                      including activities to be performed by                       the schedule. The schedule tracks government-
                      both the owner and the contractors.                           furnished information and equipment deliverables
                                                                                    and it covers all work in the statement of work. The
                                                                                    work breakdown structure maps to a standardized
                                                                                    FAA work breakdown structure but is functionally
                                                                                    oriented rather than product-oriented in
                                                                                    accordance with FAA standards.
                                                                                    Our analysis was able to successfully track detail
                                                                                    work activities in the schedule to statement of work
                                                                                    requirements, contract line item numbers, and task
                                                                                    orders.
2.   Sequencing all   The schedule should be planned so         Partially met       For the 481 remaining activities, or work that
     activities       that critical project dates can be met.                       remains to be completed, we found few issues
                      To meet this objective, activities need                       affecting the sequencing logic of the schedule. For
                      to be logically sequenced—that is,                            example, all activities in the schedule had at least
                      listed in the order in which they are to                      one predecessor and one successor activity.
                      be carried out. In particular, activities                     However, our analysis found several issues that
                      that must be completed before other                           would affect the ability of the schedule to
                      activities can begin (predecessor                             dynamically forecast future dates. For example, we
                      activities), as well as activities that                       found three Must Finish On constraints (1 percent
                      cannot begin until other activities are                       of the remaining activities). A Must Finish On
                      completed (successor activities),                             constraint means that the task, whether linked or
                      should be identified. This helps                              not, must finish on a certain date. That is, it
                      ensure that interdependencies                                 prevents the activity from finishing any earlier or
                      among activities that collectively lead                       later than a certain date, thereby overwriting
                      to the accomplishment of events or                            network logic. The use of hard constraints is
                      milestones can be established and                             essentially the same as marking a date on a
                      used as a basis for guiding work and                          calendar and, therefore, defeats the purpose of
                      measuring progress.                                           using a dynamic scheduling tool. Officials stated
                                                                                    that one Must Finish On constraint, the “Ready for
                                                                                    Operations” milestone for Release 5, is used to
                                                                                    create the critical path through Release 5, a viable
                                                                                    reason that was not justified in documentation.
                                                                                    Officials provided no justification for the remaining
                                                                                    two Must Finish On constraints. We also found 24
                                                                                    Start No Earlier Than constraints (5 percent of the
                                                                                    remaining activities). Start No Earlier Than
                                                                                    constraints are considered “soft” in that they allow
                                                                                    the activity to slip into the future if their
                                                                                    predecessor activity is delayed. However, the
                                                                                    activity cannot begin earlier than its constraint
                                                                                    date. Officials stated that the constraints were
                                                                                    justified but only provided justification for 3 of the
                                                                                    24 Start No Earlier Than constraints. For example,



                                             Page 109                                       GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                              Extent to which
Best practice           Explanation                           criterion was met     GAO analysis
                                                                                    officials stated “Tech Refresh Development Tech
                                                                                    Ops” is an obsolete task that will be deleted from
                                                                                    the schedule and removed from the contract. In
                                                                                    addition, officials said a Start No Earlier Than
                                                                                    constraint was in place to prevent multiple level-of-
                                                                                    effort tasks (management and other oversight
                                                                                    related activities that continue until the detail
                                                                                    activities they support have been completed) from
                                                                                    being identified as critical activities. Activities with
                                                                                    constraints typically are substitutes for logic and
                                                                                    can mean that the schedule is not well planned
                                                                                    and may not be feasible.
                                                                                    Our analysis also found 33 tasks with lags (7
                                                                                    percent of the remaining activities), including 28
                                                                                    activities with positive lags and 5 activities with
                                                                                    leads (negative lags). According to officials, the
                                                                                    lags are justified, but it is not their policy to
                                                                                    document justifications. Lags should be justified
                                                                                    because they cannot have risk or uncertainty. Lags
                                                                                    often represent work or a delay whose duration is
                                                                                    unknown, yet are hard-coded into the schedule.
                                                                                    Activities represented by lags are not, in fact, risk
                                                                                    free. Lags are often used to put activities on a
                                                                                    specific date or to insert a buffer for risk. These
                                                                                    lags persist even when activities are delayed and
                                                                                    use up the buffer. When that happens, the lag may
                                                                                    no longer be necessary. Leads are discouraged,
                                                                                    as negative time is not demonstrable. Leads can
                                                                                    often be replaced by additional tasks and linked
                                                                                    using the appropriate scheduling logic. If used
                                                                                    improperly, lags can distort float calculations in the
                                                                                    schedule and corrupt the calculation of the critical
                                                                                    path.
3.   Assigning          The schedule should reflect what       Substantially met    Our analysis found 48 resources in the schedule;
     resources to all   resources (e.g., labor, materials, and                      however, resources are actually tracked outside of
     activities         overhead) are needed to do the work,                        the schedule in detailed labor hour spreadsheets.
                        whether all required resources will be                      Officials said this way of managing resources
                        available when needed, and whether                          meets agency criteria. Moreover, they stated that
                        any funding or time constraints exist.                      because of the time frame represented by the
                                                                                    schedule, the program may have three software
                                                                                    releases in progress at any one time, which results
                                                                                    in an overlapping of resources. Staffing levels are
                                                                                    reviewed during the monthly management review
                                                                                    meetings to ensure resources identified by the
                                                                                    contractor map to the associated activities in the
                                                                                    schedule.
                                                                                    Because resources are tracked outside the
                                                                                    schedule rather than loaded within the schedule
                                                                                    itself, management’s ability to fully determine
                                                                                    whether all required resources will be available
                                                                                    when needed, and whether any funding or time




                                             Page 110                                       GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                              Extent to which
Best practice           Explanation                           criterion was met     GAO analysis
                                                                                    constraints exist, is hampered. A resource-loaded
                                                                                    schedule implies that all required labor and
                                                                                    significant materials, equipment, and other costs
                                                                                    are assigned to the appropriate activities within the
                                                                                    schedule. Assigning resources to activities ensures
                                                                                    that resources are used to determine activity
                                                                                    durations because resource requirements directly
                                                                                    relate to the duration of an activity.
4.   Establishing the The schedule should realistically        Met                  Excluding level-of-effort activities, more than 90
     duration of all  reflect how long each activity will take                      percent of the remaining detailed activities meet
     activities       to execute. In determining the                                best practices for having durations of 44 days or
                      duration of each activity, the same                           less. In addition, all activities are assigned to
                      rationale, historical data, and                               standard calendars that account for holidays.
                      assumptions used for cost estimating                          In estimating durations, program officials stated
                      should be used. Durations should be                           that initially they used production rates for software
                      as short as possible and have                                 lines of code to help determine durations. Now that
                      specific start and end dates. The                             the program is in the execution phase, officials
                      schedule should be continually                                have several years of history pertaining to
                      monitored to determine when                                   production, industry standards, and defect rates on
                      forecasted completion dates differ                            which to base duration estimates. Officials also
                      from planned dates; this information                          said they try to keep the durations as short as
                      can be used to determine whether                              possible, but activities such as coding are naturally
                      schedule variances will affect                                3-month-long tasks whose duration cannot
                      subsequent work.                                              artificially be shortened to 44 days.
5.   Integrating        The schedule should be horizontally Partially met           Vertical integration—that is, the ability to
     schedule           integrated, meaning that it should link                     consistently trace work breakdown structure
     activities         products and outcomes associated                            elements between detailed and summary master
     horizontally and   with other sequenced activities.                            schedules—is demonstrated in the schedule
     vertically         These links are commonly referred to                        because lower-level activities and milestones roll
                        as “hand-offs” and serve to verify that                     up into higher-level summary activities. In addition,
                        activities are arranged in the right                        the schedule shows the dependencies between
                        order to achieve aggregated products                        CATMT and other NextGen programs, such as
                        or outcomes. The schedule should                            EnRoute Automation Modernization (ERAM) and
                        also be vertically integrated, meaning                      SWIM. Officials said in order to help manage these
                        that the dates for starting and                             dependencies on ERAM and SWIM, CATMT
                        completing activities in the integrated                     development leads attend ERAM and SWIM
                        master schedule should be aligned                           weekly meetings and discuss risk dependencies.
                        with the dates for supporting tasks                         In addition, officials stated they also track the
                        and subtasks. Such mapping or                               ERAM and SWIM schedules and any related
                        alignment among levels enables                              schedule slips.
                        different groups to work to the same                        However, horizontal traceability is hampered by
                        master schedule.                                            the reliance on date constraints and lags as
                                                                                    discussed in Best Practice 2. For example,
                                                                                    extending the durations of selected activities in the
                                                                                    schedule more than 400 working days had no
                                                                                    effect on the project finish date. If the schedule
                                                                                    lacks horizontal integration, then the effects of
                                                                                    slipped tasks on downstream work cannot be
                                                                                    determined and there will be little confidence in the
                                                                                    calculated dates or critical path. Any logic errors
                                                                                    between detail, intermediate, and summary




                                             Page 111                                       GAO-12-223 Air Traffic Control Modernization
                                              Appendix IV: Assessments of Four FAA
                                              Program Schedules




                                                                Extent to which
Best practice          Explanation                              criterion was met    GAO analysis
                                                                                     schedules will cause inconsistent dates between
                                                                                     schedules and will cause different expectations
                                                                                     between senior management and activity owners.
6.   Establishing the Scheduling software should be used        Partially met        Officials said because of the 6-month spiral
     critical path for to identify the critical path, which                          development approach, the schedule cannot
     all activities    represents the chain of dependent                             deliver a single critical path for the entire program.
                       activities with the longest total                             Instead, the critical paths are calculated and based
                       duration. Establishing a project’s                            on by software releases. To calculate a critical
                       critical path is necessary to examine                         path by each software release, the prime
                       the effects of any activity slipping                          contractor uses an end constraint on the key
                       along this path. Potential problems                           deliverable milestone for each software release.
                       along or near the critical path should                        At the time of our analysis, officials stated that only
                       also be identified and reflected in                           work related to Release 5 was subject to a critical
                       scheduling the duration of high-risk                          path analysis because that was the focus of
                       activities.                                                   management. We were able to trace a contiguous
                                                                                     critical path for Release 5, beginning with the
                                                                                     project status date and ending with the Release 5
                                                                                     finish milestone. However, the validity of the
                                                                                     Release 5 critical path is hampered by the
                                                                                     existence of seven in-progress and remaining
                                                                                     detail activities within Release 5 that have over
                                                                                     1,300 working days of total float. We also
                                                                                     determined that a small number of activities
                                                                                     outside of Release 5 were currently under way (for
                                                                                     example, in other task orders). Because these
                                                                                     activities fall outside of Release 5, and therefore
                                                                                     outside the purview of a Release 5 critical path
                                                                                     analysis, management may not be fully aware of
                                                                                     the effect of any delay on these activities.
                                                                                     The critical path is directly related to the logical
                                                                                     sequencing of events and float calculations. If the
                                                                                     schedule is missing dependencies or if activities
                                                                                     are not linked correctly, float estimates will be
                                                                                     miscalculated. Incorrect float estimates will result
                                                                                     in an invalid critical path and will hinder the ability
                                                                                     of management to allocate resources from
                                                                                     noncritical activities to those that must be
                                                                                     completed on time. Without a valid critical path,
                                                                                     there is no true program control because
                                                                                     management cannot determine which slipped
                                                                                     tasks will have detrimental effects on the project
                                                                                     finish date. Until the schedule can produce a true
                                                                                     critical path, the program office will not be
                                                                                     positioned to provide reliable timeline estimates
                                                                                     and it will not be able to identify when problems or
                                                                                     changes may occur and what effect they may have
                                                                                     on downstream work.




                                              Page 112                                       GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                              Extent to which
Best practice          Explanation                            criterion was met     GAO analysis
7.   Identifying      The schedule should identify the          Minimally Met       While officials said the schedule does not contain
     reasonable float float—the amount of time by which a                           excessive float values and float values are
                      predecessor activity can slip before                          monitored as part of the schedule’s health
                      the delay affects successor                                   assessment, our analysis found unreasonable
                      activities—so that a schedule’s                               amounts of total float throughout the schedule. For
                      flexibility can be determined. As a                           example, 325 (68 percent) of the 481 remaining
                      general rule, activities along the                            activities have float values greater than 1,000
                      critical path have the least float. Total                     days. These unreasonable float values are due
                      float is the total amount of time by                          mostly to activities being tied to the project finish
                      which an activity can be delayed                              milestone, which is constrained to start no earlier
                      without delaying the project’s                                than July 1, 2016. Interim milestones that are
                      completion, if everything else goes                           scheduled to finish earlier, such as those marking
                      according to plan.                                            the end of task orders, are tied to the project finish
                                                                                    milestone as predecessors and are, therefore,
                                                                                    showing large amounts of float that do not reflect
                                                                                    actual flexibility in the schedule. The majority of
                                                                                    high-float activities are level-of-effort activities,
                                                                                    many of which are extended to the end of these
                                                                                    interim milestones and are thus associated with
                                                                                    unreasonable amounts of float as well. As noted in
                                                                                    our analysis of the schedule’s adherence to Best
                                                                                    Practice 6, a hard constraint is in place to calculate
                                                                                    a critical path through Release 5 activities;
                                                                                    however, our analysis also found unreasonable
                                                                                    amounts of total float in some of these activities.
                                                                                    For example, several activities have more than
                                                                                    1,000 days of float, including “Test and Deploy”—
                                                                                    which shows a total float value of 1,280 days.
                                                                                    Because of this excessive float, delays in the
                                                                                    activities above will have no effect on the finish
                                                                                    date of Release 5.
                                                                                    Float estimates are directly related to the logical
                                                                                    sequencing of activities and, therefore, if the
                                                                                    schedule is not properly sequenced, float
                                                                                    calculations will be miscalculated. Without reliable
                                                                                    float estimates, management may be unable to
                                                                                    allocate resources from noncritical activities to
                                                                                    activities that cannot slip without affecting the
                                                                                    project finish date.




                                             Page 113                                       GAO-12-223 Air Traffic Control Modernization
                                            Appendix IV: Assessments of Four FAA
                                            Program Schedules




                                                                  Extent to which
Best practice         Explanation                                 criterion was met   GAO analysis
8.   Conducting a     A schedule risk analysis should be       Not met                A risk analysis of the schedule’s vulnerability to
     schedule risk    performed using statistical techniques                          slippages in the completion of activities has not
     analysis         to predict the level of confidence in                           been performed. Officials stated that a schedule
                      meeting a project’s completion date.                            risk analysis was not conducted because it is not a
                      This analysis focuses not only on                               contractual deliverable. However, to address
                      critical path activities but also on                            schedule risk with software development, the
                      activities near the critical path, since                        contractor performed cross-check and risk
                      they can affect the project’s status.                           analyses on estimated lines of code using industry
                                                                                      standard software development estimating tools. In
                                                                                      addition, the officials stated they perform what-if
                                                                                      analyses and discuss risks during their monthly
                                                                                      earned value management analysis reviews.
                                                                                      However, a comprehensive schedule risk analysis
                                                                                      is an essential tool for decision makers. A
                                                                                      schedule risk analysis can be used to determine a
                                                                                      level of confidence in meeting the completion date
                                                                                      or whether proper reserves have been
                                                                                      incorporated into the schedule. A schedule risk
                                                                                      analysis will calculate schedule reserve, which can
                                                                                      be set aside for those activities identified as high
                                                                                      risk. Without this reserve, the program faces the
                                                                                      risk of delays to the scheduled completion date if
                                                                                      any delays occur on critical path activities.
9.   Updating the     The schedule should be continuously Substantially met           The schedule is updated by the contractor weekly,
     schedule using   updated using logic and durations to                            and the contractor also provides a formal update of
     logic and        determine realistic start and                                   the schedule to the program office on a monthly or
     durations to     completion dates for program                                    bimonthly basis as necessary. Officials stated that
     determine the    activities. The schedule should be                              during the monthly schedule reviews, FAA is an
     dates            analyzed continuously for variances                             active participant in discussions involving network
                      to determine when forecasted                                    logic and forecasted activity dates.
                      completion dates differ from planned                            We found 47 activities (10 percent of remaining
                      dates. This analysis is especially                              activities) with start dates in the past or with no
                      important for those variations that                             actual start dates and 49 activities (10 percent of
                      affect activities identified as being in                        remaining activities) with finish dates in the past
                      a project’s critical path and can                               with no actual finish dates. The program office said
                      influence a scheduled completion                                the majority of these data anomalies are
                      date.                                                           associated with “on-hold” activities. After reviewing
                                                                                      the schedule, we did find that majority of these
                                                                                      activities were labeled as “on-hold.” We found no
                                                                                      activities in the schedule with actual start dates in
                                                                                      the future or actual finish dates in the future.
                                            Source: GAO analysis of CATMT schedule.




                                            Page 114                                          GAO-12-223 Air Traffic Control Modernization
                                         Appendix IV: Assessments of Four FAA
                                         Program Schedules




Table 14: Extent to Which the SWIM Schedule Met Best Practices

                                                             Extent to which
Best practice           Explanation                          criterion was met   GAO analysis
1.   Capturing all      The schedule should reflect all        Minimally met     Our analysis found that, despite a substantial
     activities         activities as defined in the project’s                   amount of detail and clearly marked level-of-
                        work breakdown structure, which                          effort activities, the program schedule does not
                        defines in detail the work                               effectively represent the work required to
                        necessary to accomplish a                                complete SWIM, primarily because of the way
                        project’s objectives—including                           in which the schedule was created and the
                        activities to be performed by both                       limited authority of the SWIM program office.
                        the owner and the contractors.                           The Integrated Schedule is a synopsis of the
                                                                                 individual SWIM implementation program (SIP)
                                                                                 schedules and includes only activities whose
                                                                                 timeliness is assumed to be crucial to meeting
                                                                                 Joint Resources Council milestone dates,
                                                                                 resulting in a schedule that likely excludes a
                                                                                 portion of the program’s scope. Because of
                                                                                 missing activities, it is unlikely that this
                                                                                 schedule will highlight opportunities for process
                                                                                 improvement (e.g., identifying redundant
                                                                                 activities), what-if analysis, and risk mitigation.
                                                                                 The SWIM program office does not have
                                                                                 purview over the content that is included in the
                                                                                 individual SIP schedules. Instead, this
                                                                                 responsibility is held by FAA’s Joint Resources
                                                                                 Council secretariat. Thus, the SWIM program
                                                                                 office cannot enforce the use of a standard
                                                                                 work breakdown structure for all SIP schedules
                                                                                 or dictate the way in which the SIPs name their
                                                                                 activities. Our analysis found repetitive naming
                                                                                 of activities across SIP schedules that are not
                                                                                 associated with specific products or phases.
                                                                                 This makes communication difficult between
                                                                                 teams, particularly between team members
                                                                                 that are responsible for updating and
                                                                                 integrating multiple schedules.
                                                                                 If the schedule does not fully and accurately
                                                                                 reflect the project’s activities, it will not serve as
                                                                                 an appropriate basis for analysis and may
                                                                                 result in unreliable completion dates, time
                                                                                 extension requests, and delays. Since the
                                                                                 schedule is used for coordination, missing
                                                                                 elements will hinder coordination efforts,
                                                                                 increasing the likelihood of disruption and
                                                                                 delays. In addition, if activities are missing from
                                                                                 the schedule, then other best practices will not
                                                                                 be met. Without accounting for all necessary
                                                                                 activities, it is uncertain whether: all activities
                                                                                 are scheduled in the correct order, resources
                                                                                 are properly allocated, missing activities would
                                                                                 appear on the critical path, or a schedule risk
                                                                                 analysis accounts for all risk.




                                         Page 115                                   GAO-12-223 Air Traffic Control Modernization
                                       Appendix IV: Assessments of Four FAA
                                       Program Schedules




                                                           Extent to which
Best practice         Explanation                          criterion was met   GAO analysis
2.   Sequencing all   The schedule should be planned         Minimally met     Our analysis found that the activities in the
     activities       so that critical project dates can be                    SWIM Integrated Schedule were not properly
                      met. To meet this objective,                             sequenced because of missing dependencies
                      activities need to be logically                          and the presence of dangling activities, date
                      sequenced—that is, listed in the                         constraints, and lags. Dangling logic reduces the
                      order in which they are to be                            credibility of the calculated activity start and
                      carried out. In particular, activities                   finish dates and the identity of the critical paths
                      that must be completed before                            because the slip or elongation of an activity that
                      other activities can begin                               has no logical successor will not affect the
                      (predecessor activities), as well as                     scheduled dates of successor activities.
                      activities that cannot begin until                       In addition, the schedule contains 23 constraints
                      other activities are completed                           (6 percent of the remaining activities), or work
                      (successor activities), should be                        that remains to be completed, including 13 soft
                      identified. This helps ensure that                       constraints and 10 hard constraints. Program
                      interdependencies among activities                       officials stated that hard constraints were added
                      that collectively lead to the                            to enable the creation of the critical path for the
                      accomplishment of events or                              subprogram. However, officials stated that float
                      milestones can be established and                        is not monitored. If float is not monitored, the
                      used as a basis for guiding work                         critical path may be invalid, and there is thus no
                      and measuring progress.                                  credible reason to include these constraints in
                                                                               the schedule.
                                                                               Constraints jeopardize the ability of the schedule
                                                                               to remain dynamic—that is, to compute the
                                                                               correct delivery dates and critical path if
                                                                               durations change. Constraints artificially set
                                                                               dates in the software and may mislead users of
                                                                               the schedule, since they do not force the actual
                                                                               project to do anything. A large number of
                                                                               activities with constraints typically are substitutes
                                                                               for logic and can mean that the schedule is not
                                                                               well planned and may not be feasible.
                                                                               The quality of the schedule is significantly
                                                                               degraded by the presence of lags and leads
                                                                               (negative lags). In particular, 61 percent of the
                                                                               remaining activities are affected by at least one
                                                                               lag or lead. Program officials explained that lags
                                                                               are used to represent collections of activities that
                                                                               were present in the original SIP schedules but
                                                                               deemed noncrucial during the Integrated
                                                                               Schedule creation process. The purpose of
                                                                               these lags is then to preserve the underlying
                                                                               network logic and timeline. However, in this
                                                                               case, a summary activity would be a more
                                                                               appropriate modeling choice, since it would
                                                                               allow for logic preservation, but also maintain the
                                                                               dynamic nature of the schedule.
                                                                               Using lags rather than activities to account for
                                                                               work or to insert a buffer for risks causes
                                                                               problems because lags persist even when
                                                                               activities are delayed. For this reason, lags
                                                                               should be eliminated and replaced with activities
                                                                               so they can be tracked. Furthermore, lags



                                       Page 116                                   GAO-12-223 Air Traffic Control Modernization
                                            Appendix IV: Assessments of Four FAA
                                            Program Schedules




                                                                Extent to which
Best practice              Explanation                          criterion was met   GAO analysis
                                                                                    lessen the ability of the project schedule to
                                                                                    dynamically respond to changes in the status of
                                                                                    predecessor activities, and the lags’ need to be
                                                                                    constantly manually updated defeats the
                                                                                    purpose of a dynamic schedule and can cause
                                                                                    errors in the schedule when there are too many
                                                                                    lags to maintain.
3.   Assigning resources   The schedule should reflect what     Not met             The schedule is not resource-loaded.
     to all activities     resources (e.g., labor, materials,                       According to program officials, the SWIM
                           and overhead) are needed to do                           program is somewhat atypical with respect to
                           the work, whether all required                           resource loading. In particular, the substantive
                           resources will be available when                         effort for the program is handled by SIP
                           needed, and whether any funding                          support contractors and the SWIM program
                           or time constraints exist.                               office serves to monitor the SIPs’ progress and
                                                                                    ensure that deliverables are completed on
                                                                                    time. Moreover, in accordance with the
                                                                                    program’s federated approach, the SWIM
                                                                                    program office does not dictate the content that
                                                                                    is included in the individual SIP schedules. As
                                                                                    a result, the SIPs are not required to resource-
                                                                                    load their schedules and, thus, the Integrated
                                                                                    Schedule itself is not resource loaded.
                                                                                    Because this schedule does not have
                                                                                    resource assignments, it cannot be used to
                                                                                    monitor productivity or resource-constrained
                                                                                    activities, allocate idle resources, or level
                                                                                    resources across activities.
4.   Establishing the      The schedule should realistically   Partially met        The requirements for compliance with this best
     duration of all       reflect how long each activity will                      practice have been partially met, primarily
     activities            take to execute. In determining the                      because of the nature of the schedule creation
                           duration of each activity, the same                      and update processes. Durations for many of
                           rationale, historical data, and                          the activities in the schedule are significantly
                           assumptions used for cost                                longer than 2 working months (44 working
                           estimating should be used.                               days). Specifically, 190 (55 percent) of the
                           Durations should be as short as                          remaining activities have a duration longer than
                           possible and have specific start                         the threshold of 44 days, and half of the
                           and end dates. The schedule                              activities are scheduled for longer than 66
                           should be continually monitored to                       days. Furthermore, the average duration of all
                           determine when forecasted                                remaining activities is 158 days, or roughly 7
                           completion dates differ from                             working months. Such a schedule is likely to be
                           planned dates; this information can                      very challenging to manage and is indicative of
                           be used to determine whether                             insufficient detail. Long-duration activities also
                           schedule variances will affect                           make it difficult for management to gauge
                           subsequent work.                                         progress and can skew schedule risk analysis
                                                                                    results.
                                                                                    Officials stated that many of these lengthy
                                                                                    durations are the result of the way in which
                                                                                    the schedule was created. Those activities
                                                                                    that are not level-of-effort were originally
                                                                                    summary activities in the SIP schedule from
                                                                                    which they were extracted and, thus, are




                                            Page 117                                   GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                                  Extent to which
Best practice               Explanation                           criterion was met   GAO analysis
                                                                                      composed of many detailed activities with
                                                                                      shorter durations.
                                                                                      In addition, because the method by which
                                                                                      activity durations are estimated is not within
                                                                                      the purview of the SWIM program office, there
                                                                                      is little insight into how the durations are
                                                                                      derived. However, as noted in the discussion
                                                                                      of Best Practice 3, there is no requirement for
                                                                                      the SIP contractor schedules to be resource-
                                                                                      loaded. Because activity durations are based
                                                                                      on many underlying assumptions, including
                                                                                      the number of hours per workweek, number of
                                                                                      full-time-equivalent personnel, productivity
                                                                                      rates, and other factors, they should be
                                                                                      estimated under normal conditions, not
                                                                                      optimal conditions. If durations are not
                                                                                      realistic, program management does not have
                                                                                      the information necessary to properly allocate
                                                                                      resources to tasks and understand how those
                                                                                      tasks affect downstream work.
5.   Integrating the       The schedule should be                 Minimally met       This schedule suffers from a lack of horizontal
     schedule horizontally horizontally integrated, meaning                           integration for reasons identified in other Best
     and vertically        that it should link products and                           Practice responses. The schedule failed
                           outcomes associated with other                             several tests for horizontal integration, primarily
                           sequenced activities. These links                          because of the hard constraints on the SIP
                           are commonly referred to as                                completion milestones. For example, the
                           “hand-offs” and serve to verify that                       duration of a critical activity was increased by
                           activities are arranged in the right                       50 days, changing its float (and that of its
                           order to achieve aggregated                                predecessor) from 0 to 50 days and its finish
                           products or outcomes. The                                  date from February 28, 2011, to May 9, 2011,
                           schedule should also be vertically                         but without any effect on the project completion
                           integrated, meaning that the dates                         date. Further increasing this activity’s duration
                           for starting and completing                                by 1,500 days had the expected effect on the
                           activities in the integrated master                        finish dates of the activity and its successor,
                           schedule should be aligned with                            but left the project’s completion date
                           the dates for supporting tasks and                         unchanged and several years before the new
                           subtasks. Such mapping or                                  finish date of the critical activity. Horizontal
                           alignment among levels enables                             integration demonstrates that the overall
                           different groups to work to the                            schedule is rational, planned in a logical
                           same master schedule.                                      sequence, accounts for interdependent
                                                                                      activities, and provides a way to evaluate
                                                                                      current status. Therefore, if the schedule lacks
                                                                                      horizontal integration, the effects of slipped
                                                                                      tasks on subsequent work cannot be
                                                                                      determined and there will be little confidence in
                                                                                      the calculated dates or critical path.
                                                                                      Schedules that are not horizontally integrated
                                                                                      may not depict relationships between different
                                                                                      program elements and product hand-offs.
                                                                                      When this happens, product hand-offs of
                                                                                      project subcomponents cannot be fully traced
                                                                                      to the end product, leading to less effective



                                             Page 118                                    GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                                 Extent to which
Best practice                Explanation                         criterion was met   GAO analysis
                                                                                     management of the project. According to
                                                                                     officials, all parties involved in the creation
                                                                                     and updating of the SWIM program office
                                                                                     schedules meet routinely for status reviews
                                                                                     and have signed off on all “touch points” and
                                                                                     hand-offs. However, because of the structure
                                                                                     of the schedule, it is difficult to determine
                                                                                     where key hand-offs are made and how they
                                                                                     are related to top-level milestone dates.
                                                                                     As noted in the discussion of Best Practice 1,
                                                                                     there are several inconsistencies between
                                                                                     lower- and upper-level milestone dates. In
                                                                                     addition, this schedule does not contain all
                                                                                     lower-level activities but, rather, only a
                                                                                     selection of activities from the lower-level SIP
                                                                                     schedules. Any logic errors between detail,
                                                                                     intermediate, and summary schedules will
                                                                                     cause inconsistent dates between schedules,
                                                                                     as well as different expectations between
                                                                                     senior management and activity owners.
6.   Establishing the        Scheduling software should be        Minimally met      Based on the description of critical path
     critical path for all   used to identify the critical path,                     management practices provided by the SWIM
     activities              which represents the chain of                           program office, we analyzed this best practice
                             dependent activities with the                           from the traditional, top-level perspective, as
                             longest total duration. Establishing                    well as from that of the individual SIPs. In
                             a project’s critical path is                            both cases, we found that the requirements
                             necessary to examine the effects                        for compliance were only minimally met. For
                             of any activity slipping along this                     each analysis that we performed, we traced
                             path. Potential problems along or                       the paths of activities with the least amount of
                             near the critical path should also                      total float since they are considered “drivers”
                             be identified and reflected in                          of a critical finish milestone.
                             scheduling the duration of high-risk                    Program officials stated that, although the SIP
                             activities.                                             schedules are integrated in the schedule, the
                                                                                     SIPs are unrelated and, thus, each has its own
                                                                                     critical path. However, the full individual critical
                                                                                     paths are not accessible through the Integrated
                                                                                     Schedule, and the SIP schedules are not used
                                                                                     by the SWIM program office for management
                                                                                     purposes, so their existence offers little
                                                                                     advantage from a SWIM management
                                                                                     perspective. Furthermore, the SWIM program
                                                                                     itself should have its own critical path, which
                                                                                     includes, at the very least, acceptance of all
                                                                                     major deliverables from the SIPs.
                                                                                     We first examined the SWIM-wide critical path
                                                                                     identified by the scheduling software. This
                                                                                     critical path was found to be invalid for a variety
                                                                                     of reasons, including the inclusion of line-of-
                                                                                     effort activities, date constraints, activities with
                                                                                     excessive durations, and lags and leads.
                                                                                     Additionally, issues identified during the




                                             Page 119                                   GAO-12-223 Air Traffic Control Modernization
                                        Appendix IV: Assessments of Four FAA
                                        Program Schedules




                                                           Extent to which
Best practice           Explanation                        criterion was met   GAO analysis
                                                                               analysis of other best practices—particularly
                                                                               Best Practices 1 and 2—prevent this critical
                                                                               path from being valid. Without a valid program
                                                                               critical path, management cannot determine
                                                                               which slipped tasks will have detrimental
                                                                               effects on the project finish date.
                                                                               We also analyzed the schedule from the
                                                                               perspective of each SIP to determine the
                                                                               validity of individual critical paths. In each
                                                                               case, the critical path failed to meet the
                                                                               requirements for this best practice because of
                                                                               the presence of one or more of the following:
                                                                               line-of-effort activities, lags, leads, long
                                                                               durations, program management activities,
                                                                               and excessive float.
7.   Identifying        The schedule should identify the    Minimally met      Program officials noted that total float is within
     reasonable float   float—the amount of time by which                      acceptable limits for the project but that float
                        a predecessor activity can slip                        is not monitored by the SWIM program office.
                        before the delay affects successor                     However, our analysis found unreasonable
                        activities—so that a schedule’s                        float calculations in the schedule. In particular,
                        flexibility can be determined. As a                    107 (28 percent) of the 393 remaining detail
                        general rule, activities along the                     and milestone activities have float values
                        critical path have the least float.                    greater than or equal to 100 working days;
                        Total float is the total amount of                     that is, 27 percent of all remaining activities
                        time by which an activity can be                       can slip more than 4 working months without
                        delayed without delaying the                           affecting successor activities. Additionally, two
                        project’s completion, if everything                    activities with excessive negative float values
                        else goes according to plan.                           (95 working days) were discovered.
                                                                               The abnormal float values are primarily
                                                                               caused by the schedule integration process.
                                                                               In particular, these high values arise because
                                                                               of large differences between the finish dates
                                                                               of individual SIP capabilities and the SWIM
                                                                               program completion date. Program officials
                                                                               noted that adding links between the
                                                                               completion dates of the individual SIP
                                                                               capabilities and the project completion
                                                                               milestone (in order to lower the float values) is
                                                                               unnecessary since float management is not
                                                                               used by the SWIM program office. Officials
                                                                               stated that total float is not monitored because
                                                                               the schedule is not resource-loaded and the
                                                                               work is performed at the SIP level. In addition
                                                                               to the integration process, other possible
                                                                               causes of excessive float are the missing
                                                                               dependencies and constraints noted in the
                                                                               discussion of Best Practice 2.
                                                                               Float is directly related to the logical
                                                                               sequencing of events and the critical path. If
                                                                               the schedule is missing activities or
                                                                               dependencies, or links activities incorrectly,




                                        Page 120                                  GAO-12-223 Air Traffic Control Modernization
                                     Appendix IV: Assessments of Four FAA
                                     Program Schedules




                                                       Extent to which
Best practice        Explanation                       criterion was met    GAO analysis
                                                                            float estimates will not be accurate. Incorrect
                                                                            float estimates may result in an invalid critical
                                                                            path and an inaccurate assessment of project
                                                                            completion dates. In addition, inaccurate
                                                                            values of total float result in a false depiction
                                                                            of true project status, which could lead to
                                                                            decisions that may jeopardize the project.
8.   Conducting a    A schedule risk analysis should be Minimally met       No schedule risk analysis was conducted for
     schedule risk   performed using statistical                            this schedule. However, our analysis found
     analysis        techniques to predict the level of                     that this best practice was minimally met
                     confidence in meeting a project’s                      because a risk analysis was conducted on a
                     completion date. This analysis                         separate higher-level SWIM schedule and risk
                     focuses not only on critical path                      management has been considered, to some
                     activities but also on activities near                 extent, by the SWIM program office.
                     the critical path, since they can                      The schedule risk analysis was performed on
                     affect the project’s status.                           a high-level schedule, consisting of standard
                                                                            work packages designed to represent a
                                                                            generic software development program, in
                                                                            order to satisfy Joint Resources Council
                                                                            requirements. According to the SWIM
                                                                            program office, the SWIM Joint Resources
                                                                            Council met with each of the SIP program
                                                                            offices to develop high-level schedules that
                                                                            were risk adjusted and priced for the original
                                                                            cost estimate. These individual schedules
                                                                            were combined, resulting in a schedule with
                                                                            182 activities. This risk analysis does not
                                                                            comply with this best practice because the
                                                                            underlying schedule was created using
                                                                            standard work packages, not specific to
                                                                            SWIM, and does not accurately capture the
                                                                            relationships between the SIP capabilities and
                                                                            the success of the SWIM program.
                                                                            Furthermore, even if a risk analysis had been
                                                                            performed on the Integrated Schedule,
                                                                            several issues have been identified that would
                                                                            call its validity into question. If the schedule
                                                                            risk analysis is to be credible, the program
                                                                            must have a quality schedule that reflects
                                                                            reliable logic and clearly identifies the critical
                                                                            path—conditions that the SWIM program’s
                                                                            Integrated Schedule does not meet.
                                                                            Program officials have stated that it has not
                                                                            been practice to perform risk analysis on the
                                                                            Integrated Schedule because it is a derivative
                                                                            of several lower-level schedules and does not
                                                                            capture the work of the original program.
                                                                            However, a schedule risk analysis should be
                                                                            conducted on each schedule used for
                                                                            planning purposes, in order to provide a more
                                                                            realistic view of the expected project timeline.




                                     Page 121                                  GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                                        Extent to which
Best practice               Explanation                                 criterion was met   GAO analysis
9.   Updating the           The schedule should be               Partially met              Our analysis found that the requirements for
     schedule using logic   continuously updated using logic                                compliance with this best practice have been
     and durations to       and durations to determine realistic                            partially met. According to program officials,
     determine dates        start and completion dates for                                  the schedule is updated regularly during
                            program activities. The schedule                                monthly status meetings. This has been
                            should be analyzed continuously                                 partially confirmed by the latest schedule
                            for variances to determine when                                 status on January 31, 2011, shortly before the
                            forecasted completion dates differ                              program office transmitted it to us. Despite the
                            from planned dates. This analysis                               regularity of these meetings, however, the
                            is especially important for those                               update process described by program officials
                            variations that affect activities                               is quite labor-intensive and likely prone to
                            identified as being in a project’s                              error.
                            critical path and can influence a                               We discovered evidence of the inherent error
                            scheduled completion date.                                      in this process through our analysis. Although
                                                                                            the schedule was recently updated, there
                                                                                            were 32 activities with start dates in the past
                                                                                            but without a recorded actual start date, and
                                                                                            46 activities with finish dates in the past but
                                                                                            without an actual finish date entered in the
                                                                                            schedule. If these activities have started
                                                                                            (finished), their actual start (finish) date
                                                                                            should be updated to reflect this
                                                                                            accomplishment. If not, the start (finish) date
                                                                                            should be updated to reflect the current plan.
                                                                                            Additionally, our analysis identified 28
                                                                                            activities in the schedule with finish dates that
                                                                                            occurred before the status date, 58 activities
                                                                                            with remaining durations exceeding their
                                                                                            planned finish date, and 21 activities updated
                                                                                            out of sequence. The existence of these date
                                                                                            anomalies indicates error in the duration-
                                                                                            estimating methodology or neglect in the
                                                                                            schedule-updating process, and a schedule
                                                                                            with remaining out-of-sequence progress may
                                                                                            have the wrong logic in place and hence have
                                                                                            inaccurate critical paths and completion
                                                                                            dates.
                                             Source: GAO analysis of SWIM schedule.




                                             Page 122                                          GAO-12-223 Air Traffic Control Modernization
                                          Appendix IV: Assessments of Four FAA
                                          Program Schedules




Table 15: Extent to Which the FAA-Prepared WAAS Schedule Met Best Practices

                                                            Extent to which
Best practice        Explanation                            criterion was met    GAO analysis
1.   Capturing all   The schedule should reflect all         Minimally met       Our analysis found that the program schedule
     activities      activities as defined in the project’s                      is not fully integrated. Program officials said
                     work breakdown structure, which                             the government’s integrated master schedule,
                     defines in detail the work necessary to                     which is maintained by FAA, is made up of 15
                     accomplish a project’s objectives—                          subschedules; however, these subschedules
                     including activities to be performed by                     are not linked to the government’s integrated
                     both the owner and the contractors.                         master schedule. Furthermore, identification of
                                                                                 these subschedules in the government’s
                                                                                 integrated master schedule is difficult because
                                                                                 that schedule does not contain any custom
                                                                                 text fields, statements of work, or contract line
                                                                                 Item mappings. Without these mappings, it is
                                                                                 impossible to determine which activities in the
                                                                                 government’s integrated master schedule
                                                                                 belong to which subschedules and whether all
                                                                                 the work has been captured.
                                                                                 In addition, we found that the work breakdown
                                                                                 structure for the government’s integrated
                                                                                 master schedule does not match the work
                                                                                 breakdown structure used for the cost
                                                                                 estimate or the work breakdown structure
                                                                                 used in the program’s Earned Value
                                                                                 Management System Control Plan. Because
                                                                                 the work breakdown structure is not
                                                                                 standardized, we cannot be sure that the
                                                                                 schedule contains all of the work necessary to
                                                                                 accomplish the program’s objectives, and it is
                                                                                 difficult, if not impossible, to identify activities
                                                                                 across cost, schedule, and earned value
                                                                                 management reporting.
                                                                                 Without an integrated master schedule that
                                                                                 accounts for all planned government and
                                                                                 contractor effort, management is not able to
                                                                                 reliably estimate planned dates beyond the
                                                                                 current schedule’s end date of September
                                                                                 2016. Moreover, unless all activities are
                                                                                 accounted for, it is uncertain whether all
                                                                                 activities are properly sequenced, resources
                                                                                 are properly assigned, the critical path reflects
                                                                                 all activities, or a schedule risk analysis
                                                                                 accounts for all risk. Finally, if the schedule
                                                                                 does not fully and accurately reflect the
                                                                                 project’s activities, it will not serve as an
                                                                                 appropriate basis for analysis and may result
                                                                                 in unreliable completion dates, time extension
                                                                                 requests, and delays.




                                          Page 123                                  GAO-12-223 Air Traffic Control Modernization
                                            Appendix IV: Assessments of Four FAA
                                            Program Schedules




                                                              Extent to which
Best practice         Explanation                             criterion was met    GAO analysis
2.   Sequencing all   The schedule should be planned so         Minimally met      Our analysis found 116 remaining activities—
     activities       that critical project dates can be met.                      or remaining work to be completed. (36
                      To meet this objective, activities need                      percent) missing predecessor logic and 162
                      to be logically sequenced—that is,                           activities (48 percent) missing successor logic.
                      listed in the order in which they are to                     In addition, our analysis found a number of
                      be carried out. In particular, activities                    activities with constraints. The schedule
                      that must be completed before other                          includes 198 activities (49 percent) with Start
                      activities can begin (predecessor                            No Earlier Than constraints. These constraints
                      activities), as well as activities that                      are considered “soft” in that they allow the
                      cannot begin until other activities are                      activity to slip into the future if their
                      completed (successor activities),                            predecessor activity is delayed. However, the
                      should be identified. This helps ensure                      activity cannot begin earlier than its constraint
                      that interdependencies among                                 date. We also found 92 activities (23 percent)
                      activities that collectively lead to the                     with Finish No Earlier Than constraints. These
                      accomplishment of events or                                  constraints are also considered “soft” date
                      milestones can be established and                            constraints because they prevent activities
                      used as a basis for guiding work and                         from finishing earlier than their constraint date.
                      measuring progress.                                          The reliability of the schedule is also hindered
                                                                                   by the presence of lags and leads (negative
                                                                                   lags). In particular, 38 activities (9 percent) are
                                                                                   affected by at least one lag or lead. Lags
                                                                                   should be justified because they cannot have
                                                                                   risk or uncertainty. Furthermore, lags often
                                                                                   represent work or a delay whose duration is
                                                                                   unknown, yet are hard-coded into the schedule.
                                                                                   Using lags rather than activities to account for
                                                                                   work or to inset a buffer for risks causes
                                                                                   problems because lags persist even when
                                                                                   activities are delayed. As a result, when
                                                                                   activities take longer, the lags may actually
                                                                                   distort the true logic of the schedule. Negative
                                                                                   lags are generally not valid because time is not
                                                                                   demonstrable; moreover, a lead implies the
                                                                                   ability to predict the finish date of a predecessor
                                                                                   activity with certainty. In general, leads can be
                                                                                   replaced with straightforward finish-to-start logic
                                                                                   once tasks are broken down further.
                                                                                   Having all interdependencies between tasks
                                                                                   identified is necessary for the schedule to
                                                                                   properly calculate dates and predict changes.
                                                                                   Without the right linkages, tasks that slip early
                                                                                   in the schedule do not transmit delays to tasks
                                                                                   that should depend on them. When this
                                                                                   happens, the schedule will not provide a
                                                                                   sufficient basis for understanding the program
                                                                                   as a whole, and users of the schedule will lack
                                                                                   confidence in the dates and the critical path.
                                                                                   Unless complete network logic is established,
                                                                                   the schedule cannot predict effects on the
                                                                                   project’s planned finish date of, among other
                                                                                   things, misallocated resources, delayed tasks,
                                                                                   external events, and unrealistic deadlines.




                                            Page 124                                  GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                               Extent to which
Best practice          Explanation                             criterion was met    GAO analysis
3.   Assigning        The schedule should reflect what         Minimally met        Program officials stated that the schedule is
     resources to all resources (e.g., labor, materials, and                        not resource-loaded, and our analysis
     activities       overhead) are needed to do the work,                          confirmed that resources are not assigned to
                      whether all required resources will be                        activities. Program officials said the general
                      available when needed, and whether                            resource allocation process begins with the 5-
                      any funding or time constraints exist.                        year Strategic Plan that lays out the program’s
                                                                                    long-term goals.
                                                                                    Program officials said given the current
                                                                                    political and budget environment, it is very
                                                                                    difficult to hire government employees, and, as
                                                                                    a result, it is easier to fill resource gaps with
                                                                                    contractor personnel than with government
                                                                                    personnel.
                                                                                    Assigning resources to activities ensures that
                                                                                    resources are used to determine activity
                                                                                    durations because resource requirements
                                                                                    directly relate to the duration of an activity.
                                                                                    Labor, material, equipment, and funding
                                                                                    requirements are examined to determine the
                                                                                    feasibility of the schedule, so that resources
                                                                                    provide a benchmark of the total and per-
                                                                                    period cost of the project. If the current
                                                                                    schedule does not allow for insight into current
                                                                                    or projected allocation of resources, then the
                                                                                    risk of the program slipping is significantly
                                                                                    increased.
4.   Establishing the The schedule should realistically        Partially met        Our analysis found that more than 70 percent
     duration of all  reflect how long each activity will take                      of the remaining activities within the detail
     activities       to execute. In determining the duration                       planning period of the schedule had original
                      of each activity, the same rationale,                         durations of 44 days or less, which generally
                      historical data, and assumptions used                         meets best practices. Forty-one activities had
                      for cost estimating should be used.                           long durations, of between 200 and 630 days
                      Durations should be as short as                               that appear to be based on a fiscal year’s
                      possible and have specific start and                          period of work, without regard to deliverables
                      end dates. The schedule should be                             either at the end or at an intermediate point of
                      continually monitored to determine                            shorter duration. Usually activities in the
                      when forecasted completion dates                              schedule are organized around producing
                      differ from planned dates; this                               deliverables rather than on the end of the
                      information can be used to determine                          calendar or fiscal year, or of a 12-month
                      whether schedule variances will affect                        period.
                      subsequent work.                                              Program officials said the team uses past
                                                                                    experiences to estimate activity durations.
                                                                                    Officials also said durations are determined in
                                                                                    meetings between program engineers, control
                                                                                    account managers, and upper management.
                                                                                    We found no deadlines in the schedule, and
                                                                                    program officials noted that there is no
                                                                                    mandated date for the schedule.
                                                                                    Although the activity durations in the
                                                                                    program’s schedule appear to be reasonable,
                                                                                    not assigning resources in the schedule



                                             Page 125                                  GAO-12-223 Air Traffic Control Modernization
                                              Appendix IV: Assessments of Four FAA
                                              Program Schedules




                                                                Extent to which
Best practice           Explanation                             criterion was met    GAO analysis
                                                                                     directly affects the estimated work and
                                                                                     duration of activities. Assigning resources to
                                                                                     activities inside the schedule ensures that
                                                                                     resources are used to determine activity
                                                                                     durations because resource requirements
                                                                                     directly relate to the duration of an activity.
                                                                                     Therefore, because the schedule does not
                                                                                     include adequate resources, it is questionable
                                                                                     that the activity durations realistically reflect
                                                                                     how long each activity will take to execute.
                                                                                     Furthermore, only one activity had a calendar
                                                                                     assigned to it. All of the other activities had no
                                                                                     calendar assignment, which means that
                                                                                     holidays were not accounted for in this
                                                                                     schedule, immediately calling into question the
                                                                                     validity of the calculated dates.
5.   Integrating        The schedule should be horizontally     Partially met        The results for Best Practice 2, Sequencing of
     schedule           integrated, meaning that it should link                      All Activities, and Best Practice 7: Identifying
     activities         products and outcomes associated                             Reasonable Float, on activities and paths
     horizontally and   with other sequenced activities. These                       indicate that the logic is not in place to
     vertically         links are commonly referred to as                            demonstrate
                        “hand-offs” and serve to verify that                         vertical integration in the integrated master
                        activities are arranged in the right                         schedule.
                        order to achieve aggregated products                         The WAAS program office said vertical
                        or outcomes. The schedule should                             integration is demonstrated in the contractor
                        also be vertically integrated, meaning                       schedules, because the lower-level schedules
                        that the dates for starting and                              map to milestones in the integrated master
                        completing activities in the integrated                      schedule. However, we found that the dates
                        master schedule should be aligned                            for two key milestones in the detailed
                        with the dates for supporting tasks                          contractor schedule did not match the dates in
                        and subtasks. Such mapping or                                the government’s integrated master schedule,
                        alignment among levels enables                               indicating a problem with vertical integration
                        different groups to work to the same                         among the detailed and higher-level
                        master schedule.                                             schedules.
                                                                                     Program officials said horizontal integration is
                                                                                     identified in the schedule when activities are
                                                                                     handed off from givers to receivers, but our
                                                                                     analysis could not identify these hand-offs in
                                                                                     the schedule. In addition, the schedule does
                                                                                     not fully demonstrate horizontal traceability
                                                                                     because it does not include complete logic
                                                                                     from program start to program finish and does
                                                                                     not fully integrate the entire scope of work for
                                                                                     all parties involved in the program.
                                                                                     Program officials said the WAAS program
                                                                                     uses both the acquisition program baseline
                                                                                     and the Office of Management and Budget
                                                                                                                                a
                                                                                     (OMB) Exhibit 300 to track the program.
                                                                                     However, our analysis found that the WAAS
                                                                                     integrated master schedule only reflects work
                                                                                     approved through 2016, even though the




                                              Page 126                                  GAO-12-223 Air Traffic Control Modernization
                                              Appendix IV: Assessments of Four FAA
                                              Program Schedules




                                                                Extent to which
Best practice          Explanation                              criterion was met    GAO analysis
                                                                                     acquisition program baseline and OMB Exhibit
                                                                                     300 have estimated work planned beyond that
                                                                                     date. Without horizontal traceability, there is
                                                                                     no assurance that the forecasted dates within
                                                                                     the schedule will be determined by network
                                                                                     logic and progress to date rather than artificial
                                                                                     constraints.
                                                                                     We also tested horizontal integrity by
                                                                                     extending the durations of key activities for
                                                                                     Release 3B, the current release during the
                                                                                     GAO audit, to observe the effects on
                                                                                     successor activity dates and the project finish
                                                                                     date. Extending the durations of selected
                                                                                     activities in the schedule by more than 1,000
                                                                                     working days had no effect on the successor
                                                                                     activities or the project finish date.
                                                                                     Unless a schedule is fully horizontally
                                                                                     integrated, the effects of slipped tasks on
                                                                                     succeeding work cannot be determined.
                                                                                     Horizontal integration demonstrates that an
                                                                                     overall schedule is rational, planned in a
                                                                                     logical sequence, accounts for
                                                                                     interdependencies between work and planning
                                                                                     packages, and provides a way to evaluate a
                                                                                     program’s current status. When schedules are
                                                                                     not horizontally integrated, relationships
                                                                                     between different program teams cannot be
                                                                                     seen and product hand-offs cannot be
                                                                                     identified. Vertical integration traces the
                                                                                     consistency of data between work breakdown
                                                                                     structure elements within the layers of the
                                                                                     schedule—master, intermediate, detailed.
                                                                                     Without schedules that are vertically
                                                                                     integrated, lower-level schedules cannot be
                                                                                     clearly traced to upper-tiered milestones, and,
                                                                                     as a result, total schedule integrity is
                                                                                     compromised. This can hamper the ability of
                                                                                     different teams to use the schedule to manage
                                                                                     expectations.
6.   Establishing the Scheduling software should be used        Minimally met        Officials stated that there is no critical path in
     critical path for to identify the critical path, which                          the current schedule, but they know what the
     all activities    represents the chain of dependent                             critical activities are in the subschedules that
                       activities with the longest total                             make up the integrated master schedule. As
                       duration. Establishing a project’s                            evidence, they pointed out hard-copy versions
                       critical path is necessary to examine                         of five detail schedule critical paths for the
                       the effects of any activity slipping                          subschedules, including those for the safety
                       along this path. Potential problems                           computer. However, we were unable to find a
                       along or near the critical path should                        valid “critical path” in the schedule itself. Both
                       also be identified and reflected in                           critical paths in the schedule start with a Start
                       scheduling the duration of high-risk                          No Earlier Than constraint, which is not in line
                       activities.                                                   with best practices. Moreover, one of the
                                                                                     paths includes several activities with lags that



                                              Page 127                                  GAO-12-223 Air Traffic Control Modernization
                                              Appendix IV: Assessments of Four FAA
                                              Program Schedules




                                                                  Extent to which
Best practice          Explanation                                criterion was met   GAO analysis
                                                                                      equate to 39 months (approximately 854
                                                                                      working days) of unexplained duration.
                                                                                      Without a valid program critical path,
                                                                                      management cannot determine which slipped
                                                                                      tasks will have detrimental effects on the
                                                                                      project finish date and whether float within the
                                                                                      schedule can be used to mitigate delays in
                                                                                      critical tasks by reallocating resources from
                                                                                      tasks that can safely slip to tasks that must be
                                                                                      completed on time. Until the schedule can
                                                                                      produce a true critical path, the program office
                                                                                      will not be positioned to provide reliable
                                                                                      timeline estimates and it will not be able to
                                                                                      identify when problems or changes may occur
                                                                                      and how they may affect downstream work
7.   Identifying      The schedule should identify the            Partially met       Our analysis found excessive float values. We
     reasonable float float—the amount of time by which a                             found 238 activities (58 percent) with float
                      predecessor activity can slip before                            values greater than 1,000 working days, or 46
                      the delay affects successor                                     months. Our analysis also found 160 activities
                      activities—so that a schedule’s                                 (39 percent) with float values greater than 148
                      flexibility can be determined. As a                             working days, or 7 months.
                      general rule, activities along the                              Float estimates are directly related to the
                      critical path have the least float. Total                       logical sequencing of activities. Therefore, if
                      float is the total amount of time by                            the schedule is not properly sequenced, float
                      which an activity can be delayed                                estimates will be miscalculated. Without
                      without delaying the project’s                                  reliable float estimates, management may be
                      completion if everything else goes                              unable to allocate resources from noncritical
                      according to plan.                                              activities to activities that cannot slip without
                                                                                      affecting the project finish date. Furthermore,
                                                                                      because the critical path is directly related to
                                                                                      the logical sequencing of events and float
                                                                                      calculations, if the schedule is missing
                                                                                      dependencies or if activities are incorrectly
                                                                                      linked, float estimates will be miscalculated,
                                                                                      resulting in an invalid critical path.
8.   Conducting a      A schedule risk analysis should be         Not met             Officials stated that a schedule risk analysis
     schedule risk     performed using statistical techniques                         had not been performed on the integrated
     analysis          to predict the level of confidence in                          master schedule. Program officials said that
                       meeting a project’s completion date.                           although they do not have a WAAS
                       This analysis focuses not only on                              contingency schedule, they do manage risk
                       critical path activities but also on                           with a risk register with mitigation plans and
                       activities near the critical path, since                       that the integrated master schedule is also
                       they can affect the project’s status.                          risk-adjusted. Our analysis found no evidence
                                                                                      of risk mitigation activities in the schedule.
                                                                                      Performing a schedule risk analysis on the
                                                                                      integrated master schedule, using statistical
                                                                                      analysis, would allow the program office to
                                                                                      model the effects of work not yet on contract.
                                                                                      One way to estimate the effects of the scope
                                                                                      of work not on contract is through the use of a
                                                                                      Monte Carlo simulation (used to approximate



                                              Page 128                                   GAO-12-223 Air Traffic Control Modernization
                                          Appendix IV: Assessments of Four FAA
                                          Program Schedules




                                                           Extent to which
Best practice         Explanation                          criterion was met     GAO analysis
                                                                                 the probability outcomes of multiple trials by
                                                                                 generating random numbers) to estimate the
                                                                                 durations and start dates. This would give the
                                                                                 program office some visibility into risk impacts
                                                                                 (understanding of the effects of risk).
                                                                                 However, for a schedule risk analysis to be
                                                                                 credible, the program must have a good
                                                                                 schedule network that clearly identifies the
                                                                                 critical path and represents all work in the
                                                                                 schedule, neither of which is demonstrated in
                                                                                 the program’s schedule
                                                                                 A schedule risk analysis will calculate
                                                                                 schedule reserve, which can be set aside for
                                                                                 those activities identified as high risk. Without
                                                                                 this reserve, the program’s scheduled
                                                                                 completion date faces the risk of delays, if any
                                                                                 delays were to occur on critical path activities.
                                                                                 However, if the schedule risk analysis is to be
                                                                                 credible, the program must have a quality
                                                                                 schedule that reflects reliable logic and clearly
                                                                                 identifies the critical path—conditions that the
                                                                                 schedule does not meet.
9.   Updating the     The schedule should be continuously Partially Met          Program officials said the status of the
     schedule using   updated using logic and durations to                       schedule is updated monthly by incorporating
     logic and        determine realistic start and                              actual effort achieved by the contractors and
     durations to     completion dates for program                               the support team members. Whether the
     determine the    activities. The schedule should be                         schedule is properly updated is questionable.
     dates            analyzed continuously for variances to                     During a January 2011 meeting with the
                      determine when forecasted                                  program office, officials said they did not
                      completion dates differ from planned                       believe the schedule status date was correct
                      dates. This analysis is especially                         and, for the purposes of the audit, GAO should
                      important for those variations that                        use the date in the filename. The status date
                      affect activities identified as being in a                 is an input into the calculations used to update
                      project’s critical path and can                            and schedule remaining work. Neither the date
                      influence a scheduled completion                           on which someone is viewing the schedule nor
                      date.                                                      the latest save date of the file should be used
                                                                                 as a substitute for a valid status date. Program
                                                                                 officials said they have hired a dedicated
                                                                                 scheduler who is responsible for updating the
                                                                                 schedule.
                                                                                 Our analysis also found several data
                                                                                 anomalies. For example, the schedule
                                                                                 includes 19 remaining activities (5 percent)
                                                                                 with start dates in the past with no recorded
                                                                                 actual start dates; 17 remaining activities (4
                                                                                 percent) with finish dates in the past with no
                                                                                 recorded actual finish; 15 remaining activities
                                                                                 (4 percent) with actual start dates in the future;
                                                                                 and 12 remaining activities (3 percent) with
                                                                                 actual finish dates in the future.
                                                                                 Despite the program office’s efforts to keep




                                          Page 129                                  GAO-12-223 Air Traffic Control Modernization
                              Appendix IV: Assessments of Four FAA
                              Program Schedules




                                                      Extent to which
Best practice   Explanation                           criterion was met     GAO analysis
                                                                            the schedule updated, the integrated master
                                                                            schedule does not capture all activities and
                                                                            the schedule does not contain the complete
                                                                            network logic between all activities necessary
                                                                            for the schedule to correctly forecast the start
                                                                            and end dates of activities within the plan. As
                                                                            such, it is questionable that the schedule
                                                                            reflects the program’s true status.
                                                                            As a best practice, the schedule should be
                                                                            continually monitored to determine when
                                                                            forecasted completion dates differ from their
                                                                            planned dates. This information can be used
                                                                            to determine whether schedule variances will
                                                                            affect downstream work. Maintaining the
                                                                            integrity of the schedule logic is not only
                                                                            necessary to reflect the program’s true status,
                                                                            but is also required before conducting a
                                                                            schedule risk analysis. Without a documented,
                                                                            consistently applied schedule change control
                                                                            process, program managers can continually
                                                                            revise the schedule to match performance.
                                                                            Such adjustments hinder the project
                                                                            manager’s understanding of the project’s true
                                                                            performance. Good documentation helps with
                                                                            analyzing changes in the program schedule
                                                                            and identifying the reasons for variances
                                                                            between estimates and actual results, thereby
                                                                            contributing to the collection of cost, schedule,
                                                                            and technical data that can be used to support
                                                                            future estimates.
                              Source: GAO analysis of WAAS schedule.
                              a
                               An Exhibit 300—also called a Capital Asset Plan and Business Case—is used to justify resource
                              requests for major investments and is intended to enable an agency to demonstrate to its own
                              management, as well as to OMB, that a major project is well planned.




                              Page 130                                          GAO-12-223 Air Traffic Control Modernization
                                              Appendix IV: Assessments of Four FAA
                                              Program Schedules




Table 16: Extent to Which the WAAS Contractor-Prepared Schedule Met Best Practices

                                                                 Extent to which
Best practice           Explanation                              criterion was met   GAO analysis
1.   Capturing all      The schedule should reflect all        Substantially met     Our analysis found that the contractor’s schedule is
     activities         activities as defined in the project’s                       governed by a statement of work and the contract
                        work breakdown structure, which                              with FAA. Detailed activities are cross-referenced
                        defines in detail the work necessary                         to elements in the project’s work breakdown
                        to accomplish a project’s                                    structure that map to the task order structure in the
                        objectives—including activities to be                        schedule using custom text fields. Because the
                        performed by both the owner and the                          contractor did not provide a work breakdown
                        contractors.                                                 structure, although we found custom text fields in
                                                                                     the schedule, we cannot tell without a work
                                                                                     breakdown structure whether all the work is
                                                                                     included in the schedule.
2.   Sequencing all     The schedule should be planned so Substantially met          Our analysis found 6 remaining activities (1
     activities         that critical project dates can be met.                      percent) missing predecessor or successor logic.
                        To meet this objective, activities need                      In addition, our analysis found 6 remaining
                        to be logically sequenced—that is,                           activities (1 percent) with dangling logic where the
                        listed in the order in which they are to                     start of the activity had no predecessor (only F-F
                        be carried out. In particular, activities                    predecessors). Each activity—other than the start
                        that must be completed before other                          and finish milestones—must have its start date
                        activities can begin (predecessor                            driven by a previous activity, and each activity
                        activities), as well as activities that                      must drive the start date of another activity. Tasks
                        cannot begin until other activities are                      with dangling logic can either carry on indefinitely
                        completed (successor activities),                            without affecting downstream activities (finish date
                        should be identified. This helps                             has no successor) or must start earlier in order to
                        ensure that interdependencies                                finish on time (start date has no predecessor).
                        among activities that collectively lead                      Dangling activities are those activities with logic
                        to the accomplishment of events or                           that do not give the right activity start and finish
                        milestones can be established and                            dates when durations change.
                        used as a basis for guiding work and                         Though we found just a few missing dependencies
                        measuring progress.                                          and dangling activities, we identified 46 activities (7
                                                                                     percent) that are affected by at least one lag or
                                                                                     lead. Lags should be justified because they cannot
                                                                                     have risk or uncertainty. They often represent work
                                                                                     or a delay whose duration is unknown, yet are
                                                                                     hard-coded into the schedule. Negative lags—also
                                                                                     known as leads—are discouraged because
                                                                                     negative time is not demonstrable. Leads can often
                                                                                     be replaced by additional tasks and linked using
                                                                                     the appropriate F-S logic.
3.   Assigning          The schedule should reflect what         Partially met       Program officials stated that the schedule is not
     resources to all   resources (e.g., labor, materials, and                       resource-loaded and our analysis confirmed that
     activities         overhead) are needed to do the                               resources are not assigned to activities. Program
                        work, whether all required resources                         officials said resources are managed via their
                        will be available when needed, and                           earned value management system in which
                        whether any funding or time                                  resources are specified by resource group and skill
                        constraints exist.                                           types and are tied to the schedule by the work
                                                                                     breakdown structure.
                                                                                     Although the contractor may be conducting this
                                                                                     analysis of resources via its earned value




                                              Page 131                                       GAO-12-223 Air Traffic Control Modernization
                                             Appendix IV: Assessments of Four FAA
                                             Program Schedules




                                                              Extent to which
Best practice           Explanation                           criterion was met     GAO analysis
                                                                                    management system or its critical chain process, it
                                                                                    did not provide any supporting documentation to
                                                                                    validate this assertion.
                                                                                    Assigning resources to activities ensures that
                                                                                    resources are used to determine activity durations
                                                                                    because resource requirements directly relate to
                                                                                    the duration of an activity. Labor, material,
                                                                                    equipment, and funding requirements are
                                                                                    examined to determine the feasibility of the
                                                                                    schedule, so that resources provide a benchmark
                                                                                    of the total and per-period cost of the project. If the
                                                                                    current schedule does not allow for insight into
                                                                                    current or projected overallocation of resources,
                                                                                    then the risk of the program slipping is significantly
                                                                                    increased.
4.   Establishing the The schedule should realistically        Substantially Met    Our analysis found that more than half of the 154
     duration of all  reflect how long each activity will take                      remaining detailed activities (107 or 69 percent)
     activities       to execute. In determining the                                meet best practices for durations being less than
                      duration of each activity, the same                           44 days (or 2 working months). Our analysis found
                      rationale, historical data, and                               that all durations are consistently estimated in days
                      assumptions used for cost estimating                          and adhere to a standard 5-day workweek that
                      should be used. Durations should be                           accounts for holidays or uses calendars that
                      as short as possible and have                                 account for FAA moratoriums. Our analysis found
                      specific start and end dates. The                             no activities scheduled to begin on a weekend.
                      schedule should be continually                                Program officials said duration estimates for the
                      monitored to determine when                                   schedule are based on historical information from
                      forecasted completion dates differ                            past performance, comparable releases, lessons
                      from planned dates; this information                          learned, similar work, and other data requirements.
                      can be used to determine whether
                      schedule variances will affect
                      subsequent work.
5.   Integrating        The schedule should be horizontally Substantially met       FAA program officials said vertical integration is
     schedule           integrated, meaning that it should link                     demonstrated in the contractor’s schedule because
     activities         products and outcomes associated                            the lower-level schedules map to milestones in the
     horizontally and   with other sequenced activities.                            FAA integrated master schedule. However, we
     vertically         These links are commonly referred to                        found that dates for the same milestone activities
                        as “hand-offs” and serve to verify that                     in the contractor and FAA schedule did not match.
                        activities are arranged in the right                        Horizontal integration is also hampered in the
                        order to achieve aggregated products                        schedule. For example, extending the durations of
                        or outcomes. The schedule should                            selected activities in the schedule more than 500
                        also be vertically integrated, meaning                      working days had no effect on the project finish
                        that the dates for starting and                             date.
                        completing activities in the integrated                     Unless the schedule is fully horizontally integrated,
                        master schedule should be aligned                           the effects of slipped tasks on succeeding work
                        with the dates for supporting tasks                         cannot be determined. Horizontal integration
                        and subtasks. Such mapping or                               demonstrates that the overall schedule is rational,
                        alignment among levels enables                              planned in a logical sequence, accounts for
                        different groups to work to the same                        interdependencies between work and planning
                        master schedule.                                            packages, and provides a way to evaluate current
                                                                                    status. When schedules are not horizontally
                                                                                    integrated, relationships between different program




                                             Page 132                                       GAO-12-223 Air Traffic Control Modernization
                                              Appendix IV: Assessments of Four FAA
                                              Program Schedules




                                                                Extent to which
Best practice          Explanation                              criterion was met    GAO analysis
                                                                                     teams cannot be seen and product hand-offs
                                                                                     cannot be identified. Vertical integration traces the
                                                                                     consistency of data between work breakdown
                                                                                     structure elements within the layers of the
                                                                                     schedule—master, intermediate, detailed. Without
                                                                                     schedules that are vertically integrated, lower-level
                                                                                     schedules cannot be clearly traced to upper-tiered
                                                                                     milestones, and, as a result, total schedule integrity
                                                                                     is compromised. This can hamper the ability of
                                                                                     different teams to use the schedule to manage
                                                                                     expectations.
6.   Establishing the Scheduling software should be used        Substantially met    Program officials said the schedule contained a
     critical path for to identify the critical path, which                          critical path. Our analysis traced several critical
     all activities    represents the chain of dependent                             paths in the schedule. Though we found lags and
                       activities with the longest total                             deadlines, which create an interruption in the
                       duration. Establishing a project’s                            various critical paths, the schedule’s logic,
                       critical path is necessary to examine                         reasonable durations, and low total float estimates
                       the effects of any activity slipping                          allow the scheduling software to calculate a valid
                       along this path. Potential problems                           critical path.
                       along or near the critical path should
                       also be identified and reflected in
                       scheduling the duration of high-risk
                       activities.
7.   Identifying      The schedule should identify the          Substantially met    Of the 154 remaining detailed tasks, 148 or 96
     reasonable float float—the amount of time by which a                            percent, have float values of less than 100 days.
                      predecessor activity can slip before                           The other 6 activities, or 4 percent, have float
                      the delay affects successor                                    values of between 106 and 299 days. Though we
                      activities—so that a schedule’s                                found some negative float values in the schedule
                      flexibility can be determined. As a                            and a few activities with float values greater than
                      general rule, activities along the                             100 days, the logical sequencing of activities,
                      critical path have the least float. Total                      realistic durations, and evidence of accurate
                      float is the total amount of time by                           updating demonstrate that the total float values in
                      which an activity can be delayed                               this schedule are reasonable.
                      without delaying the project’s
                      completion, if everything else goes
                      according to plan.
8.   Conducting a      A schedule risk analysis should be       Minimally met        Program officials said there is no contractual
     schedule risk     performed using statistical                                   requirement to conduct a schedule risk analysis
     analysis          techniques to predict the level of                            and said that FAA has never expressed an interest
                       confidence in meeting a project’s                             in conducting such an analysis. Regarding
                       completion date. This analysis                                managing and assessing risks in the schedule,
                       focuses not only on critical path                             program officials said they use a monthly risk
                       activities but also on activities near                        management board process through which they
                       the critical path, since they can affect                      address schedule risk with FAA. Program officials
                       the project’s status.                                         said that, going forward, they plan to conduct a
                                                                                     Monte Carlo analysis of their schedules.
                                                                                     In our analysis, we found that the contractor’s
                                                                                     schedule includes a 22-day “buffer” activity; this is
                                                                                     a placeholder activity with a duration of 22 days
                                                                                     that is intended to be used as contingency should
                                                                                     delays occur within the release 3B effort. However,



                                              Page 133                                       GAO-12-223 Air Traffic Control Modernization
                                           Appendix IV: Assessments of Four FAA
                                           Program Schedules




                                                           Extent to which
Best practice         Explanation                          criterion was met      GAO analysis
                                                                                  our analysis found that the buffer does not produce
                                                                                  22 days of savings when removed from the
                                                                                  schedule. Removing the buffer from the schedule
                                                                                  saves 3 days because the buffer is not on the
                                                                                  current critical path.
                                                                                  A comprehensive schedule risk analysis is an
                                                                                  essential tool for decision makers. A schedule risk
                                                                                  analysis can be used to determine a level of
                                                                                  confidence in meeting the completion date or
                                                                                  whether proper reserves have been incorporated
                                                                                  into the schedule. A schedule risk analysis will
                                                                                  calculate schedule reserve, which can be set aside
                                                                                  for those activities identified as high-risk. Without
                                                                                  this reserve, the program faces the risk of delays to
                                                                                  the scheduled completion date if any delays occur
                                                                                  in critical path activities.
9.   Updating the     The schedule should be continuously Partially met           Our analysis found that the schedule has a valid
     schedule using   updated using logic and durations to                        and current status date. Our analysis also found
     logic and        determine realistic start and                               there are no activities with start or finish dates in
     durations to     completion dates for program                                the past that are missing actual start and actual
     determine the    activities. The schedule should be                          finish dates, and there are no activities with actual
     dates            analyzed continuously for variances                         start and actual finish dates in the future. Program
                      to determine when forecasted                                officials said that schedule updates are provided by
                      completion dates differ from planned                        the control account managers. The scheduler, who
                      dates. This analysis is especially                          is certified in earned value management and has
                      important for those variations that                         been doing scheduling for 2 years, provides the
                      affect activities identified as being in                    control account manager with a status update
                      a project’s critical path and can                           template that provides the scheduler with in-
                      influence a scheduled completion                            progress and percentage-complete activities. The
                      date.                                                       scheduler then reviews this information with the
                                                                                  control account manager for clarity and
                                                                                  completeness.
                                                                                  In addition to schedule updates, program officials
                                                                                  said they maintain a list of logic changes, which
                                                                                  include changes to the sequencing of activities or
                                                                                  delays in starting planned work. Program officials
                                                                                  said that they track both start and finish variances
                                                                                  in the Microsoft Project file as well as those tasks
                                                                                  whose status is unreported or that are late.
                                                                                  Program officials also said the schedules are
                                                                                  archived daily, weekly, and monthly. Though
                                                                                  program officials provided testimonial evidence
                                                                                  that they adhere to best practices when updating
                                                                                  their schedule, they did not provide any
                                                                                  documentation to validate these assertions.
                                                                                  As a best practice, the schedule should be
                                                                                  continually monitored to determine when
                                                                                  forecasted completion dates differ from the
                                                                                  planned dates. This information can be used to
                                                                                  determine whether schedule variances will affect
                                                                                  downstream work. Maintaining the integrity of the




                                           Page 134                                       GAO-12-223 Air Traffic Control Modernization
                              Appendix IV: Assessments of Four FAA
                              Program Schedules




                                                     Extent to which
Best practice   Explanation                          criterion was met             GAO analysis
                                                                                   schedule logic is not only necessary to reflect true
                                                                                   status, but is also required before conducting a
                                                                                   schedule risk analysis. Without a documented,
                                                                                   consistently applied schedule change control
                                                                                   process, project performers can continually revise
                                                                                   the schedule to match performance, thereby
                                                                                   hindering the project manager’s understanding of
                                                                                   the project’s true performance. Good
                                                                                   documentation helps with analyzing changes in the
                                                                                   program schedule and identifying the reasons for
                                                                                   variances between estimates and actual results. It
                                                                                   thus contributes to the collection of cost, schedule,
                                                                                   and technical data that can be used to support
                                                                                   future estimates.
                              Source: GAO analysis of WAAS contractor-prepared schedule.




Schedule Risk                 A best practice that the WAAS contractor schedule did not meet is
                              conducting a schedule risk analysis, which is not required by FAA’s
Analysis                      schedule specifications. FAA officials told us that they do not conduct
                              schedule risk analysis. In August and September 2011, we performed our
                              own schedule risk analysis on the WAAS contractor schedule, through
                              which we analyzed the latest version of the contractor’s schedule
                              available to us at the time of the analysis.

                              A schedule risk analysis uses statistical techniques to predict a level of
                              confidence in meeting a program’s completion date. This analysis focuses
                              on critical path activities and on near-critical and other activities since any
                              activity may potentially affect the program’s completion date. The
                              objective of the simulation is to develop a probability distribution of
                              possible completion dates that reflect the program and its quantified risks.
                              From the cumulative probability distribution, the organization can match a
                              date to its degree of risk tolerance. For instance, an organization might
                              want to adopt a program completion date that provides a 70 percent
                              probability that the program will finish on or before that date, leaving a 30
                              percent probability that it will extend beyond, or overrun that date, given
                              the schedule and the risks. The organization can thus adopt a plan
                              consistent with its desired level of confidence in the overall integrated
                              schedule. This analysis can give valuable insight into what-if drills and
                              quantify the effects of program changes.

                              In developing a schedule risk analysis, probability distributions for each
                              activity’s duration have to be established. Furthermore, risk in all activities
                              must be evaluated and included in the analysis. Some managers focus


                              Page 135                                                     GAO-12-223 Air Traffic Control Modernization
                                         Appendix IV: Assessments of Four FAA
                                         Program Schedules




                                         only on the critical path, but because we cannot be certain how long
                                         activities will take, we cannot know the true critical path. Consequently, it
                                         would be a mistake to focus only on the software-calculated critical path
                                         (those activities in which, if delayed, will negatively impact the overall
                                         project completion date) when some off-critical-path activity might
                                         become critical if a risk were to occur. Typically, three-point estimates—
                                         that is, best, most likely, and worst-case estimates—are used to develop
                                         the probability distributions for the duration of workflow activities.

                                         Once the distributions have been established, a Monte Carlo simulation
                                         uses random numbers to select specific durations from each activity
                                         probability distribution and calculates a new critical path and dates,
                                         including major milestone and program completion dates. The Monte
                                         Carlo simulation continues this random selection thousands of times,
                                         creating a new program duration estimate and critical path each time. The
                                         resulting frequency distribution displays the range of program completion
                                         dates along with the probabilities that these dates will occur. Table 17
                                         provides a range of dates and the probability of the project’s completion
                                         on those dates or earlier, based on our 3,000 iterations that are chosen at
                                         random during the Monte Carlo simulation. For example, according to our
                                         schedule risk analysis, there is a less than 5 percent chance that the
                                         project will be finished on or before September 13, 2012. Likewise, there
                                         is an 80 percent chance that the project will be finished on or before
                                         November 13, 2012.

                                         Because completion on any date is uncertain, it is more realistic to show
                                         a range of possible completion dates than to focus on a single date. In
                                         deciding which percentile to use for prudent scheduling, there is no
                                         international best practice standard. The chosen percentile depends on
                                         the riskiness and maturity of the project. For some projects, we
                                         emphasize the 80th percentile as a conservative promise-of-completion
                                         date. While the 80th percentile may appear overly conservative, it is a
                                         useful promise-of-completion date if a number of new but currently
                                         unknown risks (i.e., “unknown unknowns”) are anticipated. The 50th
                                         percentile date may expose the project to overruns.

Table 17: Probability of Project Completion
                                                                                           th              th               th              th
                                                                                          5            50               80              95
                                                        Baseline                   percentile     percentile       percentile      percentile
Finish date                                        Sept. 6, 2012           Sept. 18, 2012       Oct. 23, 2012   Nov. 13, 2012    Dec. 6, 2012
Calendar days beyond scheduled finish date                           -                    12              47               68              91
                                         Source: GAO analysis of contractor data




                                         Page 136                                                  GAO-12-223 Air Traffic Control Modernization
                   Appendix IV: Assessments of Four FAA
                   Program Schedules




                   In the case of the WAAS contractor schedule, our analysis concluded that
                   management should realistically expect cutover, or completion, between
                   October 23, 2012, and November 13, 2012, the 50th and 80th
                   percentiles, respectively. The artificial must-finish date constraint of
                   September 6, 2012, built into the schedule is unlikely. Our analysis shows
                   the probability of completion by September 6, 2012, is less than 5 percent
                   likely with the current schedule without risk mitigation. There are two
                   reasons why the planned end completion date is not likely to occur,
                   according to the results of our schedule risk analysis. First, most risks are
                   threats only. Only two opportunities were identified during the analysis:
                   (1) the estimating error of the schedule may be between -10 percent and
                   15 percent, and (2) there is a 65 percent chance that the 11-day formal
                   shadow test will not be needed. Second, there are parallel paths within
                   the structure of the schedule that lead to merge points. If several paths
                   converge to one milestone, the latest merging path determines the date.
                   This “merge bias” cannot accelerate schedule dates and usually adds
                   structural risk to the schedule.



Identified Risks   The contractor supplied six different risks that are currently identified in
                   the project’s risk register. Using these risks as a basis for discussion, we
                   interviewed 16 experts familiar with the project, including prime contractor
                   officials, FAA officials, and technical FAA consultants to identify any other
                   risks. Each interviewee was asked four questions to address four related
                   points.

                   •   To estimate the probability that an identified risk will occur on the
                       project in such a way that some activity durations are affected. The
                       estimated probability is translated into the percentage of iterations that
                       are chosen at random during the simulation. For example, if the
                       expert estimates weather will have a 10 percent chance of affecting
                       some activities, then, on average, weather risk will occur in 10 percent
                       of the Monte Carlo iterations.

                   •   If the interviewee believed the identified risk was likely, the
                       interviewee was asked to identify which activities’ durations would be
                       affected. For example, activities related to steel erection or concrete
                       pouring may be affected if the weather risk is realized and bad
                       weather occurs.

                   •   After the interviewee identified affected activities, the interviewee was
                       asked to provide a three-point estimate of the risk’s effect on
                       duration—low, most likely, and high. Estimates were provided as



                   Page 137                                GAO-12-223 Air Traffic Control Modernization
Appendix IV: Assessments of Four FAA
Program Schedules




    percentages, which were applied to the activity durations in the Monte
    Carlo simulation if the risk occurred. For example, if bad weather
    occurs, the duration of a 10-day steel erection activity may be affected
    a minimum of 110 percent, a most likely of 150 percent, or a
    maximum of 200 percent. These percentages translate into increases
    in the activity’s duration of 11 days under a low-risk scenario, 15 days
    under the most likely risk scenario, and 20 days under the maximum
    risk scenario. If the risk is not realized, there is no change to the
    activity’s original estimated duration.

•   Finally, the interviewee was asked to identify any unaccounted for
    risks.

We began the interviews with 6 risks and, through the interview process,
identified 16 more risks. During data analysis, some risks were
consolidated with others or eliminated because of limited data. In all, 14
risks were identified and incorporated into the Monte Carlo simulation.
These include 9 risk drivers, 4 existence risks, and 1 schedule duration
risk. 1

The final risk drivers used in the schedule risk analysis are as follows:

•   Some software Release 3B problem reports require testing with a live
    geostationary satellite; software simulation environment may not be
    able to test all requirements.

•   Potential difficulty repairing Fullerton contractor’s lab safety
    computers.

•   Software delays in software Release 3A and additional changes to
    software Release 2B are likely to delay the 3B schedule.

•   Parallel implementation of Releases 2B, 2C, 3A, and 3B create
    competition for FAA resources.

•   Release 3B problem report testing is more complex and subject to
    uncertainty.




1
 The schedule duration risk is applied to each activity’s duration to represent the inherent
inaccuracy of scheduling.




Page 138                                        GAO-12-223 Air Traffic Control Modernization
Appendix IV: Assessments of Four FAA
Program Schedules




•   FAA WAAS personnel move around, and inexperienced staff must be
    trained on WAAS.

•   Workshare strategy between FAA and its prime contractor affects the
    schedule for on-the-job training learning curve.

•   Changes to the simulation tools may be more difficult or more time-
    consuming to implement than anticipated.

The final existence risks are as follows:

•   FAA performs formal software release 3B shadow test.

•   Late problem report may come from FAA.

•   FAA program office is resource-limited.

Inserting existence risk activities does not affect dates within the baseline
schedule because the activities initially have zero duration. The activities
have duration only if they happen to occur during an iteration of the
simulation.

The final uncertainty risk is schedule without buffer may be optimistic.

Most risks were identified by multiple respondents during the interviews.
During data analysis, data from the interviews are combined and
analyzed to create ranges and probabilities for each of the 14 risks.

Because risks are multiplicative, several risks occurring on the same
activity may overestimate the true risk. That is, by default in the Monte
Carlo simulation, risks occur in a series, one after another, so that an
activity that has several risks may be unrealistically extended if all risks
occur. In reality, an activity may recover from two or more risks
simultaneously, so that the actual risk is not multiplicative. Therefore, to
avoid overestimation of risk, risks can explicitly be defined as occurring in
parallel rather than in series. Risks that occur in a series will occur one
after the other and add (or subtract) their respective effects on duration to
the affected activity. If risks occur in parallel, on the other hand, only the
maximum effect of all risks will affect the duration. For example, if the risk
of complexity adds 3 days to the duration of software development and
the risk of staff shortages adds 4 days, then development will extend 7
days if the risks are defined in series. However, the duration will extend
only 4 days if the risks are defined as parallel. This definition of parallel



Page 139                                GAO-12-223 Air Traffic Control Modernization
Appendix IV: Assessments of Four FAA
Program Schedules




risks helps temper any risk overestimated by a multiplication of risk
factors. We defined one risk in series: software delays in Release 3A and
additional changes to Release 2B are likely to delay Release 3B
schedule. All other risks were assumed to be parallel.

Most risks were assigned directly to existing activities in the schedule.
However, some risks required adjustments to the schedule. These
adjustments involved replacing lags with activities and inserting existence
risk activities.

Lags: During our initial analysis of the contractor’s schedule, we identified
25 remaining activities with lags and 21 remaining activities with leads
(negative lags). While lags represent the passing of time between activities,
they are often misused to put activities on a specific date or to insert a
buffer for risk. That number was reduced to 20 activities with lags and 6
activities with leads with the “Rev 1” schedule that was altered by the prime
contractor for our use in the schedule risk analysis. We replaced the 33-day
lag between the R3B Start Cutover and R3B End Cutover activities with an
actual activity, ID 258 “Cutover Task.” Replacing lags with actual activities
does not affect dates within the baseline schedule because the activities
have the same duration as the lags.

Existence risk: We identified some risks that would add an indeterminate
amount of time to the overall schedule if they were realized. For example,
if the Tijuana facility is not ready because software requirements are
misunderstood, it could add 4 to 26 days to the schedule in the form of
additional facility work.

Prioritizing risks and risk mitigation: Risks can affect the schedule in
several ways: They can have a high probability of occurring; have a large-
percentage impact on the durations of the activities they affect; or they
can apply to risk-critical paths, which may differ from the baseline
deterministic critical path. Beyond applying 14 risks to the schedule, we
were interested in identifying the marginal impact of each risk. That is, we
were interested in identifying which risks have the largest impact on the
schedule because these were the risks that should be targeted first for
mitigation.

To find the marginal impact of a risk on the total project risk at a certain
percentile, the Monte Carlo simulation was performed with the risk
removed. The difference between the finish dates of the simulation with
all the risks and the simulation with the missing risk yielded the marginal



Page 140                                GAO-12-223 Air Traffic Control Modernization
Appendix IV: Assessments of Four FAA
Program Schedules




impact of the risk. Table 18 gives the priority of risks at the 80th percentile
and the marginal impact of each risk.

Table 18: Prioritized Risks at the 80th Percentile

    Priority                                                                        Marginal impact
    level        Risk                                                              in calendar days
    1            FAA program office is resource-limited                                             13
    2            Software delays in Release 3A and additional changes                                 7
                 to Release 2B are likely to delay Release 3B schedule
    3            Release 3B problem report development and test are                                   7
                                                                     a
                 more complex than planned and subject to uncertainty
    4            FAA personnel move around and often get                                              8
                 inexperienced people to train
    5            Workshare strategy between FAA and prime contractor                                15
                 affects schedule for on-the-job training learning curve
    6            Late problem report may come from FAA                                                7
    7            Some Release 3B problem reports require testing with                                 6
                 a live geostationary satellite; software simulation
                 environment may not be able to test all requirements
    8            Schedule without buffer may be optimistic                                            5
    Total                                                                                           68
Source: GAO analysis of WAAS schedule
a
 These are two separate risks but are assigned the same priority because they are correlated 100
percent in our analysis. That is, we assumed if complexity affected one, it would necessarily affect the
other.


The marginal impact directly translates to potential calendar days saved if
the risk is mitigated. Once risks are prioritized at the percentile desired by
management, a risk mitigation workshop can be implemented to deal with
the high-priority risks in order. The prioritized list of risks will form the
basis of the workshop, and risk mitigation plans can be analyzed using
the risk model to determine how much time might be saved. Project
managers cannot expect to completely mitigate any one risk, and it is not
reasonable to expect to mitigate all risks. In addition, risk mitigation will
add to the project budget. However, some opportunities may be available
to partially mitigate risks.




Page 141                                             GAO-12-223 Air Traffic Control Modernization
Appendix V: GAO Contact and Staff
                  Appendix V: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

GAO Contact       Gerald Dillingham, Ph.D., (202) 512-2834 or dillinghamg@gao.gov



Staff             In addition to the contact named above, individuals making key
                  contributions to this report include Edward Laughlin and Karen Richey
Acknowledgments   (Assistant Directors), Lindsey Bach, David Brown, Pamela Davidson,
                  Tisha Derricotte, Kevin Egan, James Geibel, Bert Japikse, Delwen Jones,
                  Jason Lee, Dominic Nadarski, Josh Ormond, and Brian Welsh.




(540219)
                  Page 142                            GAO-12-223 Air Traffic Control Modernization
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

Obtaining Copies of   The fastest and easiest way to obtain copies of GAO documents at no
                      cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
Testimony             correspondence. To have GAO e-mail you a list of newly posted products,
                      go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
                      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.

To Report Fraud,      Contact:

Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional         Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
                      4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

Public Affairs        Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
                      U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.