oversight

Geostationary Weather Satellites: Design Progress Made, but Schedule Uncertainty Needs to be Addressed

Published by the Government Accountability Office on 2012-06-26.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States Government Accountability Office

GAO          A Report to the Committee on Science,
             Space, and Technology, House of
             Representatives


June 2012
             GEOSTATIONARY
             WEATHER
             SATELLITES

             Design Progress Made,
             but Schedule
             Uncertainty Needs to
             be Addressed




GAO-12-576
                                               June 2012

                                               GEOSTATIONARY WEATHER SATELLITES
                                               Design Progress Made, but Schedule Uncertainty
                                               Needs to be Addressed
Highlights of GAO-12-576, a report to the
Committee on Science, Space, and
Technology, House of Representatives




Why GAO Did This Study                         What GAO Found
The GOES-R series is a set of four             The Geostationary Operational Environmental Satellite-R series (GOES-R)
satellites intended to replace existing        program has made progress by completing its early design milestones and is
weather satellites that will likely reach      nearing the end of the design phase for its spacecraft, instrument, and ground
the end of their useful lives in about         system components. While the program continues to make progress, recent
2015. NOAA estimates the series to             technical problems with the instruments and spacecraft, as well as a significant
cost $10.9 billion through 2036.               modification to the ground project’s development plan, have delayed the
Because the transition to the series is        completion of key reviews and led to increased complexity for the development of
critical to the nation’s ability to maintain   GOES-R. The technical and programmatic challenges experienced by the flight
the continuity of data required for
                                               and ground projects have led to a 19-month delay in completing the program’s
weather forecasting, GAO reviewed
                                               preliminary design review. Nevertheless, program officials report that its planned
NOAA’s management of the GOES-R
program.
                                               launch date of October 2015 for the first satellite has not changed. While the
                                               program reports that approximately $1.2 billion is currently in reserve to manage
Specifically, GAO was asked to (1)             future delays and cost growth, significant portions of development remain for
assess NOAA’s progress in developing           major components. As a result, the program may not be able to ensure that it has
the GOES-R satellite program, (2)              adequate resources to cover ongoing challenges as well as unexpected
evaluate whether the agency has a              problems for the remaining development of all four satellites.
reliable schedule for executing the
program, and (3) determine whether             The success in management of a large-scale program depends in part on having
the program is applying best practices         a reliable schedule that defines, among other things, when work activities and
in managing and mitigating its risks.          milestone events will occur, how long they will take, and how they are related to
                                               one another. To its credit, the program has adopted key scheduling best
GAO analyzed program management,
                                               practices and has recognized certain scheduling weaknesses. It has also
acquisition, and cost data; evaluated
contractor and program-wide                    recently instituted initiatives to automate its integrated master schedule, correct
schedules against best practices;              integration problems among projects, and assess schedule confidence based on
analyzed program documentation                 risk. However, unresolved schedule deficiencies remain in its integrated master
including risk management plans and            schedule and the contractor schedules that support it, which have contributed to
procedures; and interviewed                    a re-plan of the schedule of the ground system and to the potential for delays to
government and contractor staff                satellite launch dates. The program recently determined that the likelihood of the
regarding program progress and                 first satellite meeting its planned October 2015 launch date is 48 percent. Based
challenges.                                    on this planned launch date, the program reports that there is a 37 percent
                                               chance of a gap in the availability of two operational GOES-series satellites,
What GAO Recommends                            which could result in the need for the National Oceanic and Atmospheric
                                               Administration (NOAA) to rely on older satellites that are not fully functional. Until
GAO is making recommendations to               its scheduling weaknesses are addressed, it will be more difficult for the program
NOAA to assess and report reserves             to know whether its planned remaining development is on schedule.
needed over the life of the program,
improve the reliability of its schedules,      NOAA has established policies and procedures that conform with recognized risk
and address identified program risks.          management best practices. For example, the program has documented a
NOAA concurred or partially concurred          strategy for managing risks that includes important elements, such as relevant
with GAO’s recommendations.                    stakeholders and their responsibilities and the criteria for evaluating,
                                               categorizing, and prioritizing risks. However, while the program has a well-
                                               defined risk management process, it has not been fully implemented. For
                                               example, the program has not provided adequate or timely evaluations for
                                               potential risks, did not always provide adequate rationale for the decision to close
                                               a risk, and has at least two critical risks in need of additional attention. Until all
                                               defined risk management practices are diligently executed and critical risks
View GAO-12-576. For more information,
contact David A. Powner at (202) 512-9286 or   adequately mitigated, the GOES-R program is at risk of exceeding cost and
pownerd@gao.gov.                               schedule targets, and launch dates could slip.

                                                                                         United States Government Accountability Office
Contents


Letter                                                                                       1
               Background                                                                    2
               Progress Made in Completing Early Design, but Milestones Were
                 Completed Late and Development Costs Have Increased                       12
               GOES-R Schedules Are Not Fully Reliable, Contributing to
                 Milestone and Potential Launch Date Delays                                23
               Risk Management Approach Is Well Defined but Not Fully
                 Implemented                                                               32
               Conclusions                                                                 38
               Recommendations for Executive Action                                        39
               Agency Comments and Our Evaluation                                          40

Appendix I     Objectives, Scope, and Methodology                                          43



Appendix II    Comments from the Department of Commerce                                    45



Appendix III   GAO Contact and Staff Acknowledgments                                       49



Tables
               Table 1: Summary of the Procurement History of GOES                           4
               Table 2: Originally Planned GOES-R Series Instruments, as of
                        August 2006                                                         6
               Table 3: Key Changes to the GOES-R Program                                   8
               Table 4: Delays in Program Milestones                                       14
               Table 5: Flight Project Development Efforts and Challenges
                        through February 2012                                              15
               Table 6: Ground Project Development Efforts and Challenges
                        through February 2012                                              17
               Table 7: Growth in Estimated Contract Cost for Major Program
                        Components                                                         18
               Table 8: Description of Scheduling Best Practices                           24
               Table 9: Practices Utilized in Selected GOES-R Schedules                    26
               Table 10: Description and Examples of Recognized Risk
                        Management Best Practices                                          33
               Table 11: Comparison of GOES-R Risk Management Policies and
                        Procedures to Recognized Practices                                 34



               Page i                               GAO-12-576 Geostationary Weather Satellites
          Table 12: GOES-R Implementation of Risk Practices                                         35
          Table 13: GOES-R Program Critical Risks, as of February 2012                              36


Figures
          Figure 1: Approximate GOES Geographic Coverage                                            3
          Figure 2: Generic GOES Data Relay Pattern                                                 5
          Figure 3: GOES-R Program Office Reporting, Structure, and
                   Staffing                                                                         10
          Figure 4: Recent Growth in Estimated Cost for Selected
                   Development Components                                                           19
          Figure 5: Changes in Program Reserves Over Time                                           21
          Figure 6: Changes in Planned Completion Dates for Key Milestones
                   of the First GOES-R Satellite                                                    30




          Abbreviations
          CDR          critical design review
          GOES         Geostationary Operational Environmental Satellite
          GOES-R       Geostationary Operational Environmental Satellite-R series
          IMS          integrated master schedule
          NASA         National Aeronautics and Space Administration
          NOAA         National Oceanic and Atmospheric Administration
          PDR          preliminary design review


          This is a work of the U.S. government and is not subject to copyright protection in the
          United States. The published product may be reproduced and distributed in its entirety
          without further permission from GAO. However, because this work may contain
          copyrighted images or other material, permission from the copyright holder may be
          necessary if you wish to reproduce this material separately.




          Page ii                                       GAO-12-576 Geostationary Weather Satellites
United States Government Accountability Office
Washington, DC 20548




                                   June 26, 2012

                                   The Honorable Ralph Hall
                                   Chairman
                                   The Honorable Eddie Bernice Johnson
                                   Ranking Member
                                   Committee on Science, Space, and Technology
                                   House of Representatives

                                   Operational geostationary environmental satellites play a critical role in
                                   our nation’s weather forecasting. These satellites—which are managed
                                   by the Department of Commerce’s National Oceanic and Atmospheric
                                   Administration (NOAA)—provide critical information on atmospheric,
                                   oceanic, climatic, and solar conditions that help meteorologists observe
                                   and predict global and local weather events. They also provide a means
                                   of identifying the large-scale evolution of severe storms, such as
                                   hurricane track and intensity forecasting and complementing radar in
                                   tornado or heavy precipitation forecasting.

                                   NOAA, through collaboration with the National Aeronautics and Space
                                   Administration (NASA), is procuring the next generation of geostationary
                                   satellites, called the Geostationary Operational Environmental Satellite-R
                                   (GOES-R) series. The GOES-R series is to replace the current series of
                                   satellites, which will likely begin to reach the end of their useful lives in
                                   approximately 2015. This new series is expected to mark the first major
                                   improvement in GOES instrumentation since 1994. It is also considered
                                   critical to the United States’ ability to maintain the continuity of data
                                   required for weather forecasting through the year 2036.

                                   This report responds to your request that we review NOAA’s
                                   management of the GOES-R program. Specifically, we were asked to (1)
                                   assess NOAA’s progress in developing the GOES-R satellite program, (2)
                                   evaluate whether the agency has a reliable schedule for executing the
                                   program, and (3) determine whether the program is applying best
                                   practices in managing and mitigating its risks.

                                   To assess NOAA’s progress in developing GOES-R, we analyzed
                                   program documentation, including monthly status reports, acquisition
                                   plans, and contractor performance reports on development efforts. To
                                   evaluate whether the agency has a reliable schedule for executing the
                                   program, we evaluated four contractor schedules including two
                                   instruments—the Advanced Baseline Imager and Geostationary Lightning


                                   Page 1                                  GAO-12-576 Geostationary Weather Satellites
             Mapper—the spacecraft, and the Core Ground System. We also analyzed
             programwide schedule initiatives. We compared contractor schedules and
             program initiatives to GAO-identified best practices in developing and
             maintaining schedules. To determine whether the program is applying
             best practices in managing and mitigating its risks, we analyzed program
             documentation, including the program’s risk management plan and
             outputs from its risk register. We compared program risk management
             policies and procedures to government and industry recognized risk
             management best practices and compared the program’s implementation
             of its risk management policies and procedures. We also interviewed
             government and contractor staff to discuss the program’s development
             progress and recent challenges as well as their scheduling and risk
             management practices.

             We conducted this performance audit from August 2011 to June 2012 in
             accordance with generally accepted government auditing standards.
             Those standards require that we plan and perform the audit to obtain
             sufficient, appropriate evidence to provide a reasonable basis for our
             findings and conclusions based on our audit objectives. We believe that
             the evidence obtained provides a reasonable basis for our findings and
             conclusions based on our audit objectives. Appendix I contains further
             details on our objectives, scope, and methodology.


             Since the 1960s, geostationary satellites have been used by the United
Background   States to provide meteorological data for weather observation, research,
             and forecasting. NOAA’s National Environmental Satellite, Data, and
             Information Service is responsible for managing the civilian operational
             geostationary satellite system, called GOES. Geostationary satellites can
             maintain a constant view of the earth from a high orbit of about 22,300
             miles in space.

             NOAA operates GOES as a two-satellite system that is primarily focused
             on the United States (see fig. 1). These satellites provide timely
             environmental data about the earth’s atmosphere, surface, cloud cover,
             and the space environment to meteorologists and their audiences. They
             also observe the development of hazardous weather, such as hurricanes
             and severe thunderstorms, and track their movement and intensity to
             reduce or avoid major losses of property and life. The ability of the
             satellites to provide broad, continuously updated coverage of atmospheric
             conditions over land and oceans is important to NOAA’s weather
             forecasting operations.



             Page 2                               GAO-12-576 Geostationary Weather Satellites
Figure 1: Approximate GOES Geographic Coverage




To provide continuous satellite coverage, NOAA acquires several
satellites at a time as part of a series and launches new satellites every
few years (see table 1). 1 NOAA’s policy is to have two operational
satellites and one backup satellite in orbit at all times.




1
 Satellites in a series are identified by letters of the alphabet when they are on the ground
(before launch) and by numbers once they are in orbit.




Page 3                                          GAO-12-576 Geostationary Weather Satellites
Table 1: Summary of the Procurement History of GOES

    Series name                           Procurement durationa                            Satellites
                     b
    Original GOES                                         1970-1987                 1, 2, 3, 4, 5, 6, 7
    GOES I-M                                              1985-2001                  8, 9, 10, 11, 12
    GOES N                                                1998-2010                    13, 14, 15, Qc
    GOES-R                                                2008-2024                        R, S, T, U
Source: GAO analysis of NOAA data.
a
    Duration includes time from contract award to final satellite launch.
b
 The procurement of these satellites consisted of four separate contracts for (1) two early prototype
satellites and GOES-1, (2) GOES-2 and -3, (3) GOES-4 through -6, and (4) GOES-G (failed on
launch) and GOES-7.
c
    NOAA decided not to exercise the option for this satellite.


Four GOES satellites–GOES-12, GOES-13, GOES-14, and GOES-15–
are currently in orbit. Both GOES-13 and GOES-15 are operational
satellites with GOES-13 covering the eastern United States and GOES-
15 in the western United States (see fig. 1). GOES-14 is currently in an
on-orbit storage mode and available as a backup for the other two
satellites should they experience any degradation in service. GOES-12 is
at the end of its service life, but it is being used to provide limited
coverage of South America. The GOES-R series is the next generation of
satellites that NOAA is planning; the first satellite in the series is planned
for launch beginning in 2015.

Each of the operational geostationary satellites continuously transmits
raw environmental data to NOAA ground stations. The data are
processed at these ground stations and transmitted back to the satellite
for broadcast to primary weather services and the global research
community in the United States and abroad. Raw and processed data are
also distributed to users via ground stations through other communication
channels, such as dedicated private communication lines and the
Internet. Figure 2 depicts a generic data relay pattern from the
geostationary satellites to the ground stations and commercial terminals.




Page 4                                                   GAO-12-576 Geostationary Weather Satellites
                         Figure 2: Generic GOES Data Relay Pattern




Overview of the GOES-R   NOAA plans for the GOES-R program to improve on the technology of
Program                  prior series, in terms of both system and instrument improvements. The
                         system improvements are expected to fulfill more demanding user
                         requirements by updating the satellite data more often and providing
                         satellite products to users more quickly. The instrument improvements
                         are expected to significantly increase the clarity and precision of the
                         observed environmental data. NOAA originally planned to acquire six
                         different types of instruments. Furthermore, two of these instruments—the
                         Advanced Baseline Imager and the Hyperspectral Environmental Suite—
                         were considered to be the most critical because they would provide data
                         for key weather products. Table 2 summarizes the originally planned
                         instruments and their expected capabilities.




                         Page 5                                      GAO-12-576 Geostationary Weather Satellites
Table 2: Originally Planned GOES-R Series Instruments, as of August 2006

Planned instrument                   Description
Advanced Baseline Imager             Expected to provide variable area imagery and radiometric information of the earth’s surface,
                                     atmosphere, and cloud cover. Key features include
                                     •   monitoring and tracking severe weather;
                                     •   providing images of clouds to support forecasts; and
                                     •   providing higher resolution, faster coverage, and broader coverage simultaneously.
Hyperspectral Environmental Suitea   Expected to provide information about the earth’s surface to aid in the prediction of weather
                                     and climate monitoring. Key features include
                                     •   providing atmospheric moisture and temperature profiles of the rapidly evolving pre-storm
                                         convective environment to support forecasts and warnings of high-impact weather
                                         phenomena;
                                     •   monitoring coastal regions for ecosystem health, water quality, coastal erosion, and
                                         harmful algal blooms; and
                                     •   providing higher resolution and faster coverage.
Geostationary Lightning Mapper       Expected to continuously monitor total lightning (in-cloud and cloud-to-ground) activity over
                                     the United States and adjacent oceans and to provide a more complete dataset than
                                     previously possible. Key features include
                                     •   detecting lightning activity as an indicator of severe storms and convective weather
                                         hazard impacts to aviation and
                                     •   providing a new capability to GOES for long-term mapping of total lightning that only
                                         previously existed on NASA low-earth-orbiting research satellites.
Magnetometer                         Expected to provide information on the general level of geomagnetic activity, monitor current
                                     systems in space, and permit detection of magnetopause crossings, sudden storm
                                     commencements, and substorms.
Space Environmental In-Situ Suite    Expected to provide information on space weather to aid in the prediction of particle
                                     precipitation, which causes disturbance and disruption of radio communications and
                                     navigation systems. Key features include
                                     •   measuring magnetic fields and charged particles;
                                     •   providing improved heavy ion detection, adding low-energy electrons and protons; and
                                     •   enabling early warnings for satellite and power grid operation, telecom services,
                                         astronauts, and airlines.
Solar Imaging Suiteb                 Expected to provide coverage of the entire dynamic range of solar X-ray features, from
                                     coronal holes to X-class flares, as well as estimate the measure of temperature and
                                     emissions. Key features include
                                     •   providing images of the sun and measuring solar output to monitor solar storms and
                                     •   providing improved imager capability.
                                         Source: GAO analysis of NOAA data.
                                         a
                                          The Hyperspectral Environmental Suite was cancelled in September 2006.
                                         b
                                          The Solar Imaging Suite was divided into two separate acquisitions: the Solar Ultraviolet Imager and
                                         the Extreme Ultraviolet/X-Ray Irradiance Sensor.




                                         Page 6                                              GAO-12-576 Geostationary Weather Satellites
In September 2006, NOAA decided to reduce the scope and technical
complexity of the GOES-R program because of expectations that total
costs, which were originally estimated to be $6.2 billion, could reach
$11.4 billion. 2 Specifically, NOAA reduced the minimum number of
satellites from four to two, cancelled plans for developing the
Hyperspectral Environmental Suite (which reduced the number of
planned satellite products from 81 to 68), and divided the Solar Imaging
Suite into two separate acquisitions. In light of the cancellation of the
Hyperspectral Environmental Suite, NOAA decided to use the planned
Advanced Baseline Imager to develop certain satellite data products that
were originally to be produced by this instrument. The agency estimated
that the revised program would cost $7 billion.

Subsequently, NOAA made several other important decisions about the
cost and scope of the GOES-R program. 3 In May 2007, NOAA had an
independent cost estimate completed for the GOES-R program. After
reconciling the program office’s cost estimate of $7 billion with the
independent cost estimate of about $9 billion, the agency established a
new program cost estimate of $7.67 billion. This was an increase of $670
million from the previous estimate. Further, in November 2007, to mitigate
the risk that costs would rise, program officials decided to remove
selected program requirements from the baseline program and treat them
as contract options that could be exercised if funds allowed. These
requirements included the number of products to be distributed, the time
to deliver the remaining products (product latency), and how often these
products would be updated with new satellite data (refresh rate). For
example, program officials eliminated the requirement to develop and
distribute 34 of the 68 envisioned products, including low cloud and fog,


2
 GAO, Geostationary Operational Environmental Satellites: Additional Action Needed to
Incorporate Lessons Learned from Other Satellite Programs, GAO-06-1129T
(Washington, D.C.: Sept. 29, 2006) and Geostationary Operational Environmental
Satellites: Steps Remain in Incorporating Lessons Learned from Other Satellite Programs,
GAO-06-993 (Washington, D.C.: Sept. 6, 2006).
3
 GAO, Geostationary Operational Environmental Satellites: Acquisition Has Increased
Costs, Reduced Capabilities, and Delayed Schedules, GAO-09-596T (Washington, D.C.:
Apr. 23, 2009); Geostationary Operational Environmental Satellites: Acquisition Is Under
Way, but Improvements Needed in Management and Oversight, GAO-09-323
(Washington, D.C.: Apr. 2, 2009); Geostationary Operational Environmental Satellites:
Further Actions Needed to Effectively Manage Risks, GAO-08-183T (Washington, D.C.:
Oct. 23, 2007); and Geostationary Operational Environmental Satellites: Progress Has
Been Made, but Improvements Are Needed to Effectively Manage Risks, GAO-08-18
(Washington, D.C.: Oct. 23, 2007).




Page 7                                        GAO-12-576 Geostationary Weather Satellites
                                              sulfur dioxide detection, and cloud liquid water. Program officials included
                                              the restoration of the requirements for the products, latency times, and
                                              refresh rates as options in the ground system contract that could be
                                              acquired at a later time. Program officials later reduced the number of
                                              products that could be restored as a contract option (called option 2) from
                                              34 to 31 because they determined that two products were no longer
                                              feasible and two others could be combined into a single product.

                                              Recently, NOAA restored two satellites to the program’s baseline, making
                                              GOES-R a four-satellite program once again. In February 2011, as part of
                                              its fiscal year 2012 budget request, NOAA requested funding to begin
                                              development for two additional satellites in the GOES-R series—GOES-T
                                              and GOES-U. The program estimates that the development for all four
                                              satellites in the GOES-R series—GOES-R, GOES-S, GOES-T, and
                                              GOES-U—is to cost $10.9 billion through 2036, an increase of $3.2 billion
                                              over its prior life cycle cost estimate of $7.67 billion for the two-satellite
                                              program. See table 3 for an overview of key changes to the GOES-R
                                              program.

Table 3: Key Changes to the GOES-R Program

                  August 2006
                  (baseline program)                                  September 2006                          November 2007     February 2011
Number of         4                                                   2                                       2                 4
satellites
Instruments or    •    Advanced Baseline Imager                       •      Cancelled Hyperspectral          No change         No change
instrument        •    Geostationary Lightning Mapper                        Environmental Suite
changes                                                               •      Decoupled Solar Imaging
                  •    Magnetometer
                                                                             Suite to the Solar Ultraviolet
                  •    Space Environmental In-Situ Suite                     Imager and Extreme
                  •    Solar Imaging Suite (which included the               Ultraviolet/X-Ray Irradiance
                       Solar Ultraviolet Imager, and Extreme                 Sensor
                       Ultraviolet/X-Ray Irradiance Sensor)
                  •    Hyperspectral Environmental Suite
Number of         81                                                  68                                      34 baseline       34 baseline
satellite                                                                                                     31 optional       31 optional
products
Life cycle cost   $6.2 billion—$11.4 billion (through 2034)           $7 billion (through 2028)               $7.67 billion     $10.9 billion
estimate (in                                                                                                  (through 2028)    (through 2036)a
then- year
dollars)
                                              Source: GAO analysis of NOAA data.
                                              a
                                               Based on NOAA’s fiscal year 2012 budget estimate, $7.64 billion of this cost estimate was for the
                                              first two satellites in the series, GOES-R and GOES-S. The cost for the remaining two satellites—
                                              GOES-T and GOES-U—was estimated at $3.22 billion.




                                              Page 8                                                  GAO-12-576 Geostationary Weather Satellites
Acquisition Strategy   NOAA’s original acquisition strategy was to award contracts for concept
                       development of the GOES-R system to several vendors who would
                       subsequently compete to be the single prime contractor responsible for
                       overall system development and production. In keeping with this strategy,
                       NOAA awarded contracts for concept development of the overall GOES-
                       R system to three vendors in October 2005. However, in March 2007,
                       NOAA revised its acquisition strategy for the development contract. In
                       response to recommendations by independent advisors, the agency
                       decided to separate the overall system development and production
                       contract into two separate contracts—the flight system and ground
                       system contracts. The flight system includes contracts for the
                       development of the five key instruments and spacecraft, while the ground
                       system includes contracts for the development of key systems needed for
                       the on-orbit operation of the satellites, receipt and processing of
                       information, and distribution of satellite data products to users.

                       In addition, to reduce the risks associated with developing technically
                       advanced instruments, the GOES-R program awarded contracts for
                       concept development for five of the planned instruments. It subsequently
                       awarded separate development contracts for five instruments and, upon
                       completion and approval, these instruments will be provided to the prime
                       contractor responsible for the spacecraft of the GOES-R program. NASA
                       will then work with the spacecraft contractor to integrate and test these
                       instruments. The sixth instrument, the magnetometer, is to be developed
                       as part of the spacecraft contract.




                       Page 9                                GAO-12-576 Geostationary Weather Satellites
Program Office Structure   NOAA is responsible for GOES-R program funding and overall mission
                           success. The NOAA Program Management Council, which is chaired by
                           NOAA’s Deputy Undersecretary, is the oversight body for the GOES-R
                           program. However, since it relies on NASA’s acquisition experience and
                           technical expertise to help ensure the success of its programs, NOAA
                           implemented an integrated program management structure with NASA for
                           the GOES-R program (see fig. 3). NOAA also located the program office
                           at NASA’s Goddard Space Flight Center. Within the program office, there
                           are two project offices that manage key components of the GOES-R
                           system. NOAA has entered into an agreement with NASA to manage the
                           Flight Project Office, including awarding and managing the spacecraft
                           contract and delivering flight-ready instruments to the spacecraft. The
                           Ground Project Office, managed by NOAA, oversees the Core Ground
                           System contract and satellite data product development and distribution.

                           Figure 3: GOES-R Program Office Reporting, Structure, and Staffing




                           Page 10                                  GAO-12-576 Geostationary Weather Satellites
Prior Reports Noted         We have previously reported on the impact of program milestone delays
Program Milestone Delays,   to GOES-R planned launch dates. In April 2009, we reported that the
Inadequate Continuity       program delayed the planned launch of its first satellite from December
                            2014 to April 2015. 4 Program officials attributed these delays to providing
Plans, and Insufficient
                            more stringent oversight before releasing the request for proposals for the
Communication with          spacecraft and ground system, additional time needed to evaluate the
GOES Data Users             contract proposals, and funding reductions in fiscal year 2008. In
                            September 2010, we reported that NOAA later approved a 6-month delay
                            in the planned launch of the first satellite, from April 2015 to October
                            2015, because work did not begin on the spacecraft until August 2009
                            due to a bid protest and NASA’s re-evaluation of proposals in response to
                            the protest. 5

                            In September 2010, we found that as a result of delays to planned launch
                            dates for the first two satellites in the GOES-R series, NOAA might not be
                            able to meet its policy of having a backup satellite in orbit at all times,
                            which could lead to a gap in satellite coverage if an existing satellite failed
                            prematurely. 6 Further, even though there could be a gap in backup
                            coverage, NOAA had not established adequate continuity plans to deal
                            with potential coverage gaps for its geostationary satellites. Specifically,
                            while NOAA had established a policy to always have a backup satellite
                            available, it did not have plans that included processes, procedures, and
                            resources needed to transition to a single or international satellite. We
                            recommended that NOAA develop and document plans for the operation
                            of geostationary satellites that included the implementation procedures,
                            resources, staff roles, and time tables needed to transition to a single
                            satellite, an international satellite, or other solution. NOAA has since
                            developed a continuity plan that generally includes the key elements we
                            recommended. As a result, NOAA has improved its ability to fully meet its
                            mission-essential function of providing continuous satellite imagery in
                            support of weather forecasting.

                            In addition, we found that while NOAA had identified GOES data users
                            and involved internal users in developing and prioritizing the GOES-R


                            4
                            GAO-09-323.
                            5
                             GAO, Geostationary Operational Environmental Satellites: Improvements Needed in
                            Continuity Planning and Involvement of Key Users, GAO-10-799 (Washington, D.C., Sept.
                            1, 2010). See also GAO-09-596T.
                            6
                            GAO-10-799.




                            Page 11                                    GAO-12-576 Geostationary Weather Satellites
                     requirements, it had not adequately involved other federal users that rely
                     on GOES data by documenting their input and communicating major
                     changes to the program that have occurred since 2006, such as the
                     removal of certain satellite data products. We recommended that NOAA
                     establish and implement processes to notify agencies of GOES-R
                     program status and changes. NOAA responded that the GOES-R
                     program would develop a communications plan for external stakeholders
                     and that the GOES-R System Program Director would provide an annual
                     program status briefing to the Office of the Federal Coordinator for
                     Meteorology, which would be responsible for its distribution to agencies
                     that would be users of GOES-R data and products.

                     In February 2012, the GOES-R program developed a communications
                     plan that described how external stakeholders would be notified of
                     GOES-R progress, status changes, and other relevant activities. Also, the
                     GOES-R System Program Director provided annual briefings to the Office
                     of the Federal Coordinator for Meteorology in October 2010 and
                     November 2011. These briefings communicated significant program
                     changes to federal GOES data users, such as completed and planned
                     milestones, the status of development efforts, and the removal of
                     additional products and latency improvements from the ground system
                     contract. As a result, federal users of GOES data have received more
                     information about changes to geostationary satellites that may affect their
                     ability to meet mission needs.


                     While the GOES-R program has made progress in completing its early
Progress Made in     design and is nearing the end of the design phase for its flight and ground
Completing Early     system components, it completed key design milestones later than
                     planned. Recent technical problems with the instruments and spacecraft,
Design, but          as well as a significant modification to the ground project’s development
Milestones Were      plan, have delayed the completion of key reviews and led to increased
Completed Late and   complexity for GOES-R’s development. Moreover, several instrument,
                     spacecraft, and ground system problems identified during design, have
Development Costs    not yet been resolved. In addition to the delays, the technical and
Have Increased       programmatic challenges experienced by GOES-R’s flight and ground
                     projects have led to increased costs for its development contracts.
                     Despite these problems, program officials report that planned launch
                     dates and cost estimates for the first two satellites have not changed, and
                     that approximately $1.2 billion is currently in reserve to manage future
                     delays and cost growth. However, the program has used approximately
                     30 percent of its reserves over the last 3 years and significant portions of
                     development remain for major components—including the spacecraft and


                     Page 12                                GAO-12-576 Geostationary Weather Satellites
                              Core Ground System. In addition, the program did not change its
                              reserves when it restored two satellites adding approximately $3.2 billion
                              to the program’s baseline. As a result, the program will be challenged in
                              completing its remaining development, particularly the final design and
                              testing of the spacecraft and ground system, within its cost and schedule
                              targets.


GOES-R Has Made               Two key types of development milestones, which are identified in the
Progress on Its Design, but   GOES-R December 2007 management control plan, 7 are the preliminary
Initial Design Milestones     design review (PDR) and a more detailed critical design review (CDR).
                              The PDR is an initial design milestone that assesses the readiness of the
Completed Later than
                              program to proceed with detailed design activities and the CDR is
Planned                       intended to demonstrate that the design is complete and appropriate to
                              support proceeding to full-scale software development, as well as
                              fabrication, assembly, integration, and testing. The program planned to
                              complete the flight project’s PDR in April 2010 and CDR in April 2011. It
                              planned to complete the ground project’s PDR in July 2010 and CDR in
                              July 2011. These project-level reviews are to be completed before the
                              program’s comprehensive PDR and CDR can be completed.

                              The program has demonstrated progress toward completing its design.
                              Specifically, the program and its subsequent projects have successfully
                              completed their PDRs, demonstrating that they are ready to proceed with
                              detailed design activities. The program and its projects are also currently
                              progressing towards the final design for the entire GOES-R system, which
                              is expected to be completed at the program’s CDR planned for
                              August 2012.

                              However, the program’s PDR milestones were completed later than their
                              2007 planned dates. Consequently, its CDR milestones were similarly
                              delayed. Table 4 highlights delays in key program milestones as of
                              April 2012.




                              7
                               The program established planned dates for key program and system milestones in its
                              management control plan documented in December 2007. This plan was approved by
                              NOAA’s Assistant Administrator for Satellite and Information Services and formed the
                              basis of flight and ground project plans prior to entering the development phase of the
                              program.




                              Page 13                                       GAO-12-576 Geostationary Weather Satellites
                              Table 4: Delays in Program Milestones

                                                                                  Date completed or     Delay from
                               Program milestone                   Date planned   planned               December 2007 plan
                               Flight project PDR                  April 2010     January 2011          9 months
                               Ground project PDR                  July 2010      December 2011         17 months
                               Program PDR                         July 2010      February 2012         19 months
                               Flight project CDR                  April 2011     April 2012            12 months
                               Ground project CDR                  July 2011      July 2012             12 months
                               Program CDR                         July 2011      August 2012           13 months
                              Source: GAO analysis of NOAA data.

                              Note: Program officials reported that the 2007 review target dates were revised in March 2010. Based
                              on its revised 2010 plans, the program missed its revised PDR milestone by 8 months and its revised
                              CDR milestone is 5 months later than planned.


                              Going forward, assembly and testing of all flight instruments for the first
                              satellite in the series is to be completed by August 2013. Both the flight
                              and ground projects’ development components are expected to be
                              complete by September 2015. Despite recent delays in program
                              milestones, NOAA still expects to meet an October 2015 launch date for
                              the first satellite in the series by utilizing planned schedule reserves.

Flight Project—Instrument     The Flight Project Office has made progress by completing the critical
Designs Are Complete but      design reviews for each of its five main instruments, which is significant
Technical Challenges Remain   because the instrument designs will be applied to each satellite in the
                              GOES-R series. The Geostationary Lightning Mapper was the most
                              recent instrument to complete its critical design review, which occurred in
                              August 2011—16 months after its planned completion date. 8 Instrument
                              fabrication, assembly, and testing activities are under way. The final
                              instrument scheduled for completion—the Geostationary Lightning
                              Mapper—is expected to be delivered for integration with the spacecraft by
                              August 2013.

                              Although instrument design milestones are complete and assembly and
                              testing activities are under way, each of the instruments and the
                              spacecraft has recently encountered technical challenges. The Flight


                              8
                               Completion of the Geostationary Lightning Mapper’s CDR was 16 months later than the
                              planned milestone documented in December 2007 when the program entered into its
                              development phase, and 13 months later than the revised milestone established in March
                              2010.




                              Page 14                                              GAO-12-576 Geostationary Weather Satellites
                                             Project Office has taken steps to resolve several technical problems
                                             through additional engineering support and redesign efforts. However,
                                             there are still important technical challenges to be addressed, including
                                             signal blurring problems for several of the Advanced Baseline Imager’s
                                             infrared channels and Geostationary Lightning Mapper emissions that are
                                             exceeding specifications. The project office is monitoring these problems
                                             and has plans in place to address them. The current development efforts
                                             and recent challenges for components of the flight project are described
                                             in table 5.

Table 5: Flight Project Development Efforts and Challenges through February 2012

Key components           Key efforts                                                Recent challenges
Instruments
Advanced Baseline        •   Completed critical design review                       •   Replaced printed wiring boards that were
Imager                   •   Testing under way; conducted initial bench testing         incompatible with specifications
                             for the first flight model                             •   Working to resolve signal problems in several of
                         •   Preparing for a planned pre-environmental reviewa          the imager’s infrared channels
                             in March 2012
Space Environmental      •   Completed certain hardware, including detectors        •   Corrected failures of its data processing unit
In-Situ Suite                and select spare parts                                     during testing
                         •   Developing software algorithms for the first flight
                             model
                         •   Performing environmental testing for three out of
                             five sensors on the instrument suite
                         •   Preparing for a planned pre-environmental review
                             in January 2013
Extreme Ultraviolet/X-   •   Completed individual instruments, which are being      •   Replaced filters that were damaged during
Ray Irradiance Sensor        integrated with the instrument suite                       vibration testing
                         •   Preparing for a planned pre-environmental review
                             in June 2012
Solar Ultraviolet        •   Completed software algorithms for the first flight     •   Investigated cracks in the camera’s electronic
Imager                       model                                                      box power control unit
                         •   Completed limited thermal vacuum testing on its        •   Working to resolve failures in the structural
                             engineering design unit                                    model analysis filter
                         •   Completing remaining work on the telescope and
                             sensor electronics prior to instrument-level testing
                         •   Preparing for a planned pre-environmental review
                             in September 2012
Geostationary            •   Completed integration of its engineering design        •   Rebuilt electronics unit and sensor unit power
Lightning Mapper             unit                                                       boards that failed during electrical systems
                         •   Completed software algorithms for the first flight         testing
                             model                                                  •   Working to resolve an image signal problem
                         •   Conducting electrical system testing                   •   Working to limit emissions that exceed
                         •   Preparing for a planned pre-environmental review           requirements
                             in December 2012



                                             Page 15                                         GAO-12-576 Geostationary Weather Satellites
Key components   Key efforts                                        Recent challenges
Spacecraft       •   Completed preliminary design review            •  Achieved sufficient massb margins
                 •   Received the Spacecraft Command and Telemetry •   Eliminated gaps in the flow down of mission
                     Simulator, which is a simulator to generate       assurance requirements to subcontractors
                     spacecraft telemetry and receive ground        •  Working to obtain adequate fault management
                     commands to be used in interface and              design specifications from the contractor
                     development testing
                                                                    •  Working to increase end-of-life power margins
                 •   Completing component and subsystem critical
                     design reviews in preparation for the system’s
                     critical design in April 2012
                                   Sources: GAO analysis of NOAA and NASA data.
                                   a
                                    A pre-environmental review ensures that the hardware or software to be tested, facility, personnel,
                                   and procedures are in place and ready for testing.
                                   b
                                    Mass is a measure of the amount of matter in a physical body that does not depend on location or
                                   gravitational pull.


Ground Project—Incomplete          The Ground Project Office has made progress in completing the ground
Requirements Definition and        system’s preliminary design. However, in doing so it experienced
Schedule Integration Have Led      problems in defining ground system software requirements and identified
to Revised Development Plan        problems with its dependencies on flight project schedules. Specifically,
                                   the project office discovered in early 2011 that software design
                                   requirements had not progressed enough to conduct the ground system’s
                                   preliminary design review. In addition, the ground system’s development
                                   schedule included software deliveries from flight project instruments that
                                   were not properly integrated—they had not yet been defined or could not
                                   be met. To address these problems, the Ground Project Office made
                                   significant revisions to the Core Ground System’s baseline development
                                   plan and schedule.

                                   In order to avoid potential slippages to GOES-R’s launch date, project
                                   officials decided to switch from plans to deliver software capabilities at
                                   major software releases to an approach where software capabilities could
                                   be delivered incrementally (prior to major releases) as the project
                                   received data inputs from the flight project. According to NOAA and
                                   program officials, the revised development approach is to provide
                                   flexibility in the ground system’s development schedule and reduce risk
                                   associated with the original waterfall schedule. However, the revised plan
                                   is expected to cost $85 million more than the original plan through the
                                   Core Ground System’s completion. This cost includes increased
                                   contractor and government staff, new oversight tools, and more
                                   verification and testing activities associated with an increased number of
                                   software deliveries. Program officials acknowledged that the revised plan
                                   intends to expend additional resources to reduce schedule risk and
                                   potential impacts to GOES-R’s launch date, but that it also introduces



                                   Page 16                                             GAO-12-576 Geostationary Weather Satellites
                                         new cost and schedule risks associated with incremental development,
                                         such as more software development and verification activities that require
                                         additional government oversight and continuous monitoring.

                                         In addition, the program has cancelled previously exercised options to the
                                         ground project that were once considered part of its original baseline. 9 In
                                         early 2011, the program determined that it could no longer fund Core
                                         Ground System contract options—which were estimated to cost
                                         approximately $50 million. Program officials stated that they cancelled the
                                         contract options due to approaching development commitments, including
                                         revisions to the ground system’s revised plan and schedule, and funding
                                         reductions from fiscal year 2011. According to program officials, the work
                                         to be performed under the cancelled contract options could be addressed
                                         by NOAA after GOES-R satellites are launched; however, there are
                                         currently no plans in place to do so.

                                         Table 6 describes the development efforts and recent challenges for
                                         ground project components.

Table 6: Ground Project Development Efforts and Challenges through February 2012

Key components         Key efforts                                            Recent challenges
Core Ground System     •   Completed preliminary design review                •  Working to avoid delays in the availability of a
                       •   Executing a revised development plan and              primary operations facility required for testing
                           schedule in preparation for critical design        •  Working to fully define software requirements,
                           review in April 2012                                  which are largely complete and resolve prior
                                                                                 problems, and integrate schedules with flight
                                                                                 project instruments
GOES-R Access          •   Completed critical design reviews for systems      No recent major challenges
Subsystem                  that will enable ground system product
                           processing and distribution
                       •   Completed trade studies for communication
                           services to be provided to local and federal
                           emergency managers
Antennas               •   Completed critical design review                   No recent major challenges
                       •   Conducting site preparation and construction
                           activities for multiple antenna sites
                                         Source: GAO analysis of NOAA data.




                                         9
                                          These options included 31 additional products—such as low cloud and fog, sulfur dioxide
                                         detection, and cloud liquid water—as well as enhancements to the time to deliver the
                                         remaining products and how often these products are updated with new satellite data.




                                         Page 17                                          GAO-12-576 Geostationary Weather Satellites
GOES-R Development                           The GOES-R program’s estimated costs include, among other things,
Costs Are Rising                             actual and estimated contract costs associated with design, development,
                                             integration, and testing activities for the instruments, spacecraft, and
                                             ground system as well as procurement of the satellites’ launch vehicles.
                                             The estimated costs also include government costs such as NOAA and
                                             NASA program management support and contingency reserves to be
                                             applied towards critical risks and issues as they arise. As of January
                                             2010, the program reported estimates of $3.3 billion for the flight project
                                             (including reserves of $598 million), $1.7 billion for the ground project
                                             (including reserves of $431 million), $2.0 billion for other program costs
                                             (including reserves of $617 million), and $748 million for operations and
                                             maintenance.

                                             Although NOAA has not changed its program cost estimates for the
                                             development of GOES-R and GOES-S, contract costs for the
                                             instruments, spacecraft, and ground system are rising. Specifically,
                                             contractor estimated costs for flight and ground project components grew
                                             by $757 million, or 32 percent, between January 2010 and January 2012.
                                             Table 7 identifies growth in estimated contract costs for major program
                                             components.

Table 7: Growth in Estimated Contract Cost for Major Program Components

                                                                          Contractor            Contractor         Contractor
                                 Original                Percent          estimate at           estimate at        estimate at       2-year     2-year
                                 contract              complete           completion            completion         completion       change     change
Major components                 award date           (Nov 2011)          (Jan 2010)a           (Jan 2011)a        (Jan 2012)a         ($M)        (%)
Advanced Baseline Imager         Sep 2004                     83%                $524M                 $581M            $672M      +$148M        +28%
Space Environmental In-Situ      Aug 2006                     54%                      69                    81             97          +28      +41%
Suite
Extreme Ultraviolet/X-Ray        Aug 2007                     58%                      72                    81             81           +9      +13%
Irradiance Sensor
Solar Ultraviolet Imager         Sep 2007                     62%                    139                    168            182          +43      +31%
Geostationary Lightning Mapper   Dec 2007                     57%                    157                    209            252          +95      +61%
Spacecraft                       Dec 2008                     32%                    711                    743            862         +151      +21%
Core Ground System               May 2009                     29%                    704                    792            976         +272      +39%
                                                                                                                                           c
Antennas                         July 2010                    37%                 Not                       119            130         +11        +9%
                                                                                     b
                                                                           applicable
Totals                                                                             2,376                   2,774         3,252        +757c     +32%c
                                             Sources: GAO analysis of NOAA and contractor-reported data.
                                             a
                                                 Contractor-reported most likely estimate at completion.
                                             b
                                                 The antenna contract was not awarded until July 2010.
                                             c
                                                 Total 2-year change includes the 1-year change in antenna contract costs.




                                             Page 18                                                         GAO-12-576 Geostationary Weather Satellites
Not only have development contracts experienced rising costs since
January 2010, but they have also experienced larger cost increases more
recently. For example, between January 2011 and January 2012,
contractors’ estimated costs increased by $184 million for the Core
Ground System, compared with $88 million for the period between
January 2010 and January 2011. Also, between January 2011 and
January 2012, contractors’ estimated costs for the spacecraft and the
Advanced Baseline Imager increased by $119 million and $91 million
compared with $32 million and $57 million, respectively, between January
2010 and January 2011. Figure 4 depicts recent growth in estimated
costs for these development components.

Figure 4: Recent Growth in Estimated Cost for Selected Development Components




Source: GAO analysis of contractors’ reported cost estimates.



The recent growth in contract costs is due in part to the additional labor
and engineering support needed to address technical and programmatic
problems experienced by flight and ground project components, including
the technical complexity associated with development of the Advanced
Baseline Imager and the spacecraft, and additional costs associated with
the Core Ground System’s revised development plan. NOAA stated that
some of the cost growth is attributed to scope changes, including


Page 19                                                         GAO-12-576 Geostationary Weather Satellites
                          instrument options that were exercised in 2010 and 2011. Based on our
                          analysis, approximately $60 million of the $757 million growth in
                          contractors’ estimated costs at completion from January 2010 through
                          November 2011 was due to scope changes associated with new
                          instrument flight model development. Given the recent increases in
                          contract costs, the program plans to determine how to cover these
                          increased costs by reducing resources applied to other areas of program
                          development and support, delaying scheduled work, or absorbing
                          additional life cycle costs.


Depleted Reserves Could   A contingency reserve (also called management reserve) is important
Impact Remaining          because it provides program managers ready access to funding in order
Development               to resolve problems as they occur and may be necessary to cover
                          increased costs resulting from unexpected design complexity, incomplete
                          requirements, or other uncertainties. 10 NOAA requires the program office
                          and flight and ground projects to maintain a reserve of funds until their
                          development is completed. Specifically, the flight project is to maintain 20
                          percent of planned remaining development costs as reserve, the ground
                          project is to maintain 30 percent of planned remaining development costs
                          as reserve, and the program office is to maintain 10 percent of planned
                          remaining development costs as reserve. 11

                          The program has allocated a proportion of its budget as reserves to
                          mitigate risks and manage problems as they surface during development.
                          As a result of changes in budget reserve allocations and reserve
                          commitments, 12 the program’s reserves have declined in recent years.
                          Between January 2009 and January 2012, the program reported that its
                          reserves fell from 42 percent of remaining development costs to 29
                          percent. Over the same period, the program reports that after accounting
                          for changes in reserve budgets and reserve commitments, reserves fell
                          from $1.7 billion to $1.2 billion, an approximately 30 percent reduction in its



                          10
                            GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and
                          Managing Capital Program Costs, GAO-09-3SP (Washington, D.C., Mar. 2009).
                          11
                            Reserve levels are to be based on all planned remaining development and do not
                          include operations and maintenance costs.
                          12
                            NOAA uses the terms liens and encumbrances to represent potential risks or issues for
                          which the program is to address with reserves. We refer to these as reserve
                          commitments.




                          Page 20                                      GAO-12-576 Geostationary Weather Satellites
uncommitted reserves. This is important to note since about two-thirds of
the development remains for the program’s two most expensive
components—the spacecraft and the Core Ground System. Recent
utilization of the program’s reserves included addressing unanticipated
problems with instrument and spacecraft design, using more than expected
labor required to accomplish program and project-related milestones, and
acquiring additional resources required to execute the revised development
plan and schedule for the Core Ground System. In this regard, the
program’s independent review board recently raised questions about the
sufficiency of the program’s near-term remaining reserves, and program
officials decided to establish contractual funding caps for the spacecraft,
three instruments—the Advanced Baseline Imager, the Solar Ultraviolet
Imager, and the Geostationary Lightning Mapper—and the Core Ground
System for fiscal year 2012, delaying work into future years. Figure 5
depicts changes in program reserves over time.

Figure 5: Changes in Program Reserves Over Time




At completion of the GOES-R program’s preliminary design review in
February 2012, the program reported that it was within its thresholds
because it had maintained 20 percent of planned remaining development
costs as reserve for the flight project, 32 percent of planned remaining



Page 21                                GAO-12-576 Geostationary Weather Satellites
development costs for the ground project, and 10 percent of planned
remaining development costs for the program office. The program will
soon enter the integration and test phase, when projects typically
experience cost and schedule growth and need additional funding—as
identified in our prior reviews of NASA acquisitions. 13 The program may
need to draw from its available remaining reserves to address a number
of situations, including

•    unanticipated problems during completion of the critical design
     reviews of the spacecraft and the ground system;

•    unanticipated problems during the integration and test of the
     instruments and spacecraft, such as required redesign;

•    potential additional labor required for ground system software
     development and testing;

•    potential delays in the readiness of NOAA’s Satellite Operations
     Facility for ground system testing, resulting in the use of contractor
     facilities; and

•    higher than expected costs for launch vehicles.

While the program reported that reserves were within accepted levels as
of February 2012, the reserves may not be matched to remaining
development. Although the program restored two satellites to its budget
baseline in February 2011, thereby adding approximately $3.2 billion to its
total budget, 14 it did not correspondingly change its program reserves.
The program did not report its rationale for maintaining reserves at the
two-satellite program or explain how these planned reserves were
intended to cover risks associated with the development of all four
satellites. As a result, there is limited assurance that the reserves are
appropriate for each satellite’s remaining development.




13
  See, for example, GAO, NASA: Assessments of Selected Large-Scale Projects,
GAO-11-239SP (Washington, D.C., March 3, 2011) and NASA: Assessments of Selected
Large-Scale Projects, GAO-12-207SP, (Washington, D.C., March 1, 2012).
14
  The restoration of GOES-T and GOES-U to the program resulted in an increase of $3.2
billion to the program’s cost estimate.




Page 22                                     GAO-12-576 Geostationary Weather Satellites
                            Whether the program will continue to stay within its budget depends in
                            part on whether officials have a full understanding of the reserves
                            required for remaining development. Given the program’s recent use of
                            reserves and the significant portions of development remaining for major
                            components, a complete understanding and proper management of
                            reserve levels will be critical to successfully completing all program
                            components. Unless NOAA assesses the reserve allocations across all of
                            the program’s development efforts, it may not be able to ensure that its
                            reserves will cover ongoing challenges as well as unexpected problems
                            for the remaining development of all four satellites in the series.


                            The success in management of a large-scale program depends in part on
GOES-R Schedules            having an integrated and reliable schedule that defines, among other
Are Not Fully               things, when work activities and milestone events will occur, how long
                            they will take, and how they are related to one another. Without such a
Reliable, Contributing      schedule, program milestones may slip. While the GOES-R program has
to Milestone and            adopted certain scheduling best practices at both the programwide and
Potential Launch Date       contractor levels, unresolved weaknesses also exist, some of which have
                            contributed to current program milestone delays and a replanning of the
Delays                      Core Ground System’s schedule. Without a proper understanding of
                            current program status that a reliable schedule provides, managing the
                            risks of the GOES-R program becomes more difficult and may result in
                            potential delays in GOES-R’s launch date.


GOES-R Integrated Master    Program schedules not only provide a road map for systematic program
Schedule and Some           execution, but also provide the means by which to gauge progress,
Subordinate Schedules Are   identify and address potential problems, and promote accountability.
                            Accordingly, a schedule helps ensure that all stakeholders understand
Unreliable                  both the dates for major milestones and the activities that drive the
                            schedule. If changes occur within a program, the schedule helps decision
                            makers analyze how those changes affect the program. The reliability of
                            the schedule will determine the credibility of the program’s forecasted
                            dates, which are used for decision making. Our work has identified nine
                            best practices 15 associated with developing and maintaining a reliable
                            schedule. These are (1) capturing all activities, (2) sequencing all
                            activities, (3) assigning resources to all activities, (4) establishing the



                            15
                             These practices are based on GAO-09-3SP.




                            Page 23                                     GAO-12-576 Geostationary Weather Satellites
                                               duration of all activities, (5) integrating schedule activities horizontally and
                                               vertically, (6) establishing the critical path for all activities, (7) identifying
                                               reasonable “float” between activities, (8) conducting a schedule risk
                                               analysis, and (9) updating the schedule using logic and durations. See
                                               table 8 for a description of each of these best practices.

Table 8: Description of Scheduling Best Practices

Practice                                       Description
Capturing all activities                       The schedule should reflect all activities (steps, events, outcomes, etc.) as defined in the
                                               program’s work breakdown structure to include activities to be performed by both the
                                               government and its contractors.
Sequencing all activities                      The schedule should sequence activities in the order that they are to be implemented. In
                                               particular, activities that must finish prior to the start of other activities (i.e., predecessor
                                               activities), as well as activities that cannot begin until other activities are completed (i.e.,
                                               successor activities) should be identified.
Assigning resources to all activities          The schedule should reflect who will do the work activities, whether all required resources
                                               will be available when they are needed, and whether any funding or time constraints
                                               exist.
Establishing the duration of all activities    The schedule should reflect the duration of each activity. These durations should be as
                                               short as possible and have specific start and end dates.
Integrating schedule activities horizontally   The schedule should be horizontally integrated, meaning that it should link the products
and vertically                                 and outcomes associated with sequenced activities. The schedule should also be
                                               vertically integrated, meaning that traceability exists among varying levels of activities and
                                               supporting tasks and subtasks.
Establishing the critical path for all         The critical path represents the chain of dependent activities with the longest total
activities                                     duration in the schedule.
Identifying reasonable float between           The schedule should identify a reasonable amount of float—the time that an activity can
activities                                     slip before the delay affects the finish milestone—so that schedule flexibility can be
                                               determined. As a general rule, activities along the critical path typically have the least
                                               amount of float.
Conducting a schedule risk analysis            A schedule risk analysis is used to predict the level of confidence in the schedule,
                                               determine the amount of time contingency needed, and identify high-priority schedule
                                               risks.
Updating the schedule using logic and          The schedule should use logic and durations in order to reflect realistic start and
durations to determine the dates               completion dates, be continually monitored to determine differences between forecasted
                                               completion dates and planned dates, and avoid logic overrides and artificial constraint
                                               dates.
                                               Source: GAO.



                                               The first seven practices are essential elements for creation of an
                                               integrated schedule. An integrated schedule that contains all the detailed
                                               tasks necessary to ensure program execution is called an integrated
                                               master schedule (IMS). The reliability of an integrated schedule depends
                                               in part on the reliability of its subordinate schedules. Automated
                                               integration of all activities into a single master schedule can prevent



                                               Page 24                                            GAO-12-576 Geostationary Weather Satellites
inadvertent errors when entering data or transferring them from one file to
another.

GOES-R has an IMS that is created manually once a month directly from
at least nine subordinate contractor schedules. 16 Due to anticipated
limitations related to the number of activities that would need to be
included, program officials stated that they did not intend to make the IMS
a live, integrated file. We believe that the lack of a dynamic IMS is an
acceptable schedule limitation for an organization of GOES-R’s size and
complexity. Program officials also stated that they are in the process of
creating an automated process for updating their IMS from contractor-
delivered files sometime in 2012, but this was not available in time for our
analysis.

To assess the reliability of the programwide IMS, we analyzed four
subordinate contractor schedules from July 2011 17 and found several best
practices that were met in multiple schedules. For example, similar to the
program-level IMS, three of the contractor IMS’s include all activities that
were supplemented with monthly data received directly from the
schedules of their subcontractors, and the fourth—the Core Ground
System schedule—included milestone activities for all subcontractors. All
four schedules also substantially or fully met the best practice for
regularly updating the schedule, including regular reporting of status and
keeping the number of activities completed out of sequence to a
minimum. Finally, three of the four schedules substantially met the best
practice for establishing accurate activity durations.

However, we also found weaknesses in each of the subordinate
schedules when compared to the best practices and, when viewed in
conjunction with manual program-level updates, we concluded that the
program-level schedule may not be fully reliable. For example, each of
the contractor schedules was either not resource loaded or had significant
overloading of resources. In addition, none of the contractors were able to


16
  The subordinate schedules used in creating the IMS each contain detailed activities for
discrete segments of the GOES-R program, such as instruments, which are assigned to a
specific contractor. We did not analyze the programwide IMS itself due to the limitations
inherent in manual creation of this schedule. However, conclusions drawn from analysis of
contractors’ schedules that feed directly into the programwide IMS can therefore be
applied to the program’s IMS as well.
17
 Our rationale for the selection and analysis of schedules is discussed in app. I.




Page 25                                       GAO-12-576 Geostationary Weather Satellites
                                                 provide information on schedule risk analyses conducted with risk
                                                 simulations.

                                                 A full set of analysis results is listed in table 9. Selected strengths and
                                                 weaknesses for each of the four schedules follow the table.

Table 9: Practices Utilized in Selected GOES-R Schedules

                                                                            Geostationary    Advanced                                                     Core Ground
                                                                              Lightning    Baseline Imager                          Spacecraft              System
Scheduling best practice                                                   Mapper schedule    schedule                               schedule              schedule
Best Practice 1: Capturing all activities
                                                                                      ●                           ◕                        ◕                       ◑
Best Practice 2: Sequencing all activities
                                                                                      ◕                           ◑                        ◑                       ◑
Best Practice 3: Assigning resources to all activities
                                                                                      ◔                           ◕                        ◔                       ◑
Best Practice 4: Establishing the duration of all activities
                                                                                      ◕                           ◑                        ◕                       ◕
Best Practice 5: Integrating schedule activities horizontally
and vertically                                                                        ◑                           ◑                        ◕                       ◑
Best Practice 6: Establishing the critical path for all activities
                                                                                      ◕                           ◑                        ◕                       ◔
Best Practice 7: Identifying float on activities and paths
                                                                                      ◕                           ◑                        ◑                       ◔
Best Practice 8: Conducting a schedule risk analysis
                                                                                      ◔                           ◔                        ◔                       ◔
Best Practice 9: Updating the schedule using logic and
durations to determine the dates                                                      ●                           ●                        ●                       ◕
                                                 Source: GAO analysis of schedules provided by GOES-R, documents and information received from GOES-R officials.

                                                 Key
                                                 ● = The agency/contractor has fully met the criteria for this best practice
                                                 ◕ = The agency/contractor has substantially met the criteria for this best practice
                                                 ◑ = The agency/contractor has partially met the criteria for this best practice
                                                 ◔ = The agency/contractor has minimally met the criteria for this best practice
                                                 ○ = The agency/contractor has not met the criteria for this best practice

                                                 Of the four component schedules, the Geostationary Lightning Mapper
                                                 schedule demonstrated the most comprehensive implementation of best
                                                 practices. For instance, all activities and durations were appropriately
                                                 captured in the schedule; logic links were included for over 99 percent of
                                                 activities; only one small gap was present in the critical path from the date
                                                 of the schedule through the end of the project; a high percentage of
                                                 activities had appropriate logic; and the schedule was updated using logic
                                                 and durations to determine dates. However, more than 10 percent of the
                                                 resources listed in the schedule were overloaded, meaning that the
                                                 schedule required more resources than were available. Also, changes in




                                                 Page 26                                                          GAO-12-576 Geostationary Weather Satellites
activity durations for one flight model did not result in a corresponding
change in the shipment date of that flight model.

The Advanced Baseline Imager schedule substantially or fully met three
of the best practices. As mentioned above, the Advanced Baseline
Imager schedule met the best practices for including all subcontractor
records as well as regular updating and reporting on the schedule. In
addition, it provided a substantial amount of information regarding the
resources in its schedule; the contractor created a series of over 900
named codes that denote detailed information such as resource type,
location, and labor rate. However, 18 percent of remaining activities had
incomplete or missing logic, and 43 percent had soft date constraints. If
the schedule is missing logic links between activities, float 18 estimates will
not be accurate. Incorrect total float values may in turn result in an invalid
critical path and an inaccurate assessment of project completion dates. In
the case of the Advanced Baseline Imager, many detailed activities had
large total float values throughout. Moreover, the Advanced Baseline
Imager schedule did not have a valid critical path for one of its three flight
models and did not have a critical path that spans the entire program.

The spacecraft schedule substantially met more than half of the best
practices. It had a valid critical path through the launch dates for the first
two satellites, included and mapped all activities appropriately, included
logic links for over 99 percent of its activities, and was appropriately
updated. However, more than 10 percent of its activities had constraints,
and nearly 10 percent of its activities had gaps between consecutive
activities, also known as lags. Lags should be minimal, and should not be
used in place of activities, as they cannot be easily monitored, cannot be
included in a risk assessment, and do not take resources. For this reason,
lags should be eliminated and replaced with activities so they can be
tracked. Also, the spacecraft schedule’s critical path included a year-long
current activity for which a detailed breakdown was not available, even
though the activity is in the current detailed planning period.

The Core Ground System schedule contained weaknesses across seven
of the nine best practice areas resulting in a score of partially or minimally
met. For instance, a valid critical path could not be traced between the



18
   Total float is the amount of time that an activity can slip before the delay affects the
finish milestone.




Page 27                                          GAO-12-576 Geostationary Weather Satellites
schedule’s latest status date and the launch date for either of the first two
satellites; 12 percent of remaining activities had incomplete or missing
logic and 13 percent had soft date constraints; and more than 75 percent
of activities had more than 100 days of total float. Also, not all
subcontractor detail activities were included in the schedule. Without
accounting for all necessary activities, it is uncertain whether all activities
are scheduled in the correct order, missing activities would appear on the
critical path, or a schedule risk analysis accounts for all risk.

Officials for all four contract teams suggested that certain schedule
weaknesses were unique to the July 2011 schedules they provided and
that the weaknesses would be remedied in December 2011 schedules.
Our subsequent analysis did find improvements for each of the four
contractors. For example, the Advanced Baseline Imager’s December
schedule had approximately 10 percent fewer activities missing
predecessors and successors than in July. The Core Ground System
schedule also performed better in three of the nine best practices,
including the presence of a full set of duration and total float information,
and better information on handoffs with external parties. However, many
weaknesses from the July schedules remained in the December
schedules. For example, officials stated that several schedule risk
analyses had been conducted for the Core Ground System schedule, but
also reported that the schedule risk analysis conducted by the Core
Ground System’s subcontractor was not valid and did not provide
accurate or constructive information.

Of particular importance is the absence of a valid critical path throughout
all the schedules. Establishing a valid program-level critical path depends
on the resolution of issues with the respective critical paths for the
spacecraft and Core Ground System components. Contract specifications
for all four contractors require that these schedules define critical paths
for their activities. Without a valid critical path, management cannot
determine which delayed tasks will have detrimental effects on the project
finish date. It also cannot determine if total float within the schedule can
be used to mitigate critical tasks by reallocating resources from tasks that
can be delayed without launch date impact. Unless the weaknesses in
these subordinate schedules are addressed, including the generation of
valid critical paths in all schedules, the programwide IMS that is derived
from them may not be sufficiently reliable.




Page 28                                  GAO-12-576 Geostationary Weather Satellites
Schedule Practice          Weaknesses in implementing scheduling best practices undermine the
Weaknesses Could Lead to   program’s ability to produce credible dates for planned milestones and
Further Schedule Delays    events, as illustrated by schedule discrepancies that occurred between
                           the ground project and the flight project and the subsequent replanning
                           that was required. The program has already demonstrated a pattern of
                           milestone delays during its development due in part to the scheduling
                           weaknesses we identified. Although the program has initiated two key
                           efforts that could address certain schedule weaknesses, other
                           weaknesses have not yet been resolved. Until the program’s scheduling
                           weaknesses are corrected, it may experience additional delays to its key
                           milestones.

                           The program has revised planned milestone and completion dates for
                           each of the instruments as well as the spacecraft and ground system
                           components by at least 3 months—and up to 2 years—since the program
                           originally estimated dates for key milestones in its December 2007
                           management control plan. Program officials noted that its December 2007
                           dates were notional estimates until integrated baseline reviews could be
                           conducted. However, delays occurred both before and after the
                           schedules for the instruments, spacecraft, and ground system
                           components were formalized. In certain cases, more recent changes
                           were due to delays in building and testing satellite components. For
                           example, the Solar Ultraviolet Imager experienced delays in 2011 and
                           2012 due in part to delays in software development and in procuring flight
                           model parts. Also, a failure in testing power supply boards caused rework
                           delays in 2011 and 2012 on the Geostationary Lightning Mapper
                           instrument. See figure 6 for a summary of changes in planned completion
                           dates for components of the flight and ground projects.




                           Page 29                               GAO-12-576 Geostationary Weather Satellites
Figure 6: Changes in Planned Completion Dates for Key Milestones of the First GOES-R Satellite




                                        a
                                         This chart shows estimated timing of GOES-R milestones based on NOAA’s initial 2007 estimate
                                        and monthly program status reports from 2010 and 2012. Antenna and the GOES-R Access
                                        Subsystem dates were not listed in the 2007 estimates.


                                        The potential for delays remains as GOES-R’s instruments, spacecraft,
                                        and ground system components complete their design and testing
                                        phases. According to program officials, the Geostationary Lightning
                                        Mapper shipment date remains at risk of a potential slip due to redesign
                                        efforts that have impacted the release of the build of the electronic board
                                        component of the instrument. The current projected delivery for this
                                        instrument is August 2013, leaving only 1 month before it is on the critical
                                        path for GOES-R’s launch readiness date. Moreover, weaknesses in
                                        implementing schedule best practices make meaningful measurement
                                        and oversight of program status and progress, as well as accountability
                                        for results, difficult to achieve—which can in turn reduce the timeliness
                                        and effectiveness of the understanding and mitigation of project risks.

                                        The program office has taken specific positive actions that address two of
                                        the scheduling weaknesses we identified. First, the GOES-R program
                                        implemented the Giver-Receiver Intersegment Database, a tool that tracks
                                        deliverables between the flight and ground projects. Giver-Receiver
                                        Intersegment Database items are formally reviewed by various working
                                        groups weekly and monthly before they are incorporated into the GOES-R
                                        IMS. This initiative is intended to address a program-recognized need for



                                        Page 30                                          GAO-12-576 Geostationary Weather Satellites
better horizontal integration (related to best practice 5). Second, the GOES-
R program implemented a Joint Cost and Schedule Confidence Level, a
set of parametric models designed to identify the probability that a given
program’s schedule values will be equal to or less than target values on a
specific date. 19 In the Joint Cost and Schedule Confidence Level,
simulations are run for the expected duration of activities based on
probabilities supplied by officials in the project’s cost and schedule division.
This initiative is intended to address a program-recognized need to conduct
a schedule risk analysis (related to best practice 8).

However, the possibility of future milestone delays remains. Initial results
from the Joint Cost and Schedule Confidence Level from January 2012
indicated that there is a 48 percent confidence level that the program will
meet its current launch readiness date of October 2015. Program officials
plan to consult with the NOAA Program Management Council to
determine the advisability of moving the launch readiness date to a 70
percent confidence level of February 2016. Given that scheduling
weaknesses remain unaddressed, even these confidence levels may be
unreliable. Establishing accurate confidence estimates depends on
reliable data that result from the implementation of a full set of scheduling
best practices.

Furthermore, delays in GOES-R’s launch date could impact current
operational GOES continuity and could produce milestone delays for
subsequent satellites in the series. Program documentation indicates that
with the current launch readiness date of October 2015, plus an on-orbit
testing period, there is a 37 percent chance of a gap in the availability of
two operational GOES-series satellites at any one time, assuming a
normal lifespan for the satellites currently on-orbit. Any delays in the
launch readiness date for GOES-R, which is already at risk due to the
increased cost growth and recent heavy use of program reserves
discussed previously, would further increase the probability of a gap in
satellite continuity. This could result in the need for NOAA to rely on older
satellites that are not fully functional. In addition, GOES-R’s schedule
reserve is being counted on to complete activities for GOES-S. As a



19
   The Joint Cost and Schedule Confidence Level is a probabilistic analysis that includes,
among other things, all cost and schedule elements, incorporates and quantifies potential
risks, assesses the impacts of cost and schedule to date, and addresses available annual
resources to arrive at development cost and schedule estimates associated with various
confidence levels.




Page 31                                       GAO-12-576 Geostationary Weather Satellites
                        result, delays to certain program schedule targets could impact milestone
                        commitments for GOES-S.

                        Until the program implements a full set of schedule best practices, and
                        uses it on succeeding schedule updates throughout the life of the
                        program, further delays in the program’s launch dates may occur. In
                        particular, without ensuring that all contractor and subcontractor
                        information is included in the IMS and conducting regular schedule risk
                        assessments, program management may not have timely and relevant
                        information at its disposal for decision making. Lack of the proper
                        understanding of current program status due to schedules that are not
                        fully reliable undercuts the ability of the program office to manage a high-
                        risk program like GOES-R.


                        Risk management is a continuous process to identify potential problems
Risk Management         before they occur. By identifying potential problems early, activities can
Approach Is Well        be planned and invoked as needed across the life of a project to avoid or
                        mitigate the adverse impacts of program problems. Effective risk
Defined but Not Fully   management involves early and aggressive risk identification through the
Implemented             collaboration and involvement of relevant stakeholders. Government and
                        industry risk management guidance divides risk management activities
                        into four key areas—preparing for risk management, identifying and
                        analyzing risks, mitigating risks, and reviewing risks with executive
                        oversight. Table 10 describes recognized best practices in these areas.




                        Page 32                                GAO-12-576 Geostationary Weather Satellites
Table 10: Description and Examples of Recognized Risk Management Best Practices

Risk practice                      Description
Risk preparation                   Risk preparation involves establishing and maintaining a strategy for identifying, analyzing, and
                                   mitigating risks. The risk management strategy addresses the specific actions and management
                                   approach used to apply and control the risk management program. It also includes identifying
                                   and involving relevant stakeholders in the risk management process.
Risk identification and analysis   Risks must be identified and described in an understandable way before they can be analyzed
                                   and managed properly. This includes identifying risks from both internal and external sources
                                   and evaluating each risk to determine its likelihood and consequences. Analyzing risks includes
                                   risk evaluation, categorization, and prioritization, and is used to determine when appropriate
                                   management attention is required.
Risk mitigation                    Risk mitigation involves developing alternative courses of action, workarounds, and fallback
                                   positions, with a recommended course of action for the most important risks to the project.
                                   Mitigation includes techniques and methods used to avoid, reduce, and control the probability of
                                   occurrence of the risk; the extent of damage incurred should the risk occur; or both. Examples of
                                   activities for mitigating risks include documented handling options for each identified risk; risk
                                   mitigation plans; contingency plans; a list of persons responsible for tracking and addressing
                                   each risk; and updated assessments of risk likelihood, consequence, and thresholds.
Executive risk oversight           Reviews of the project teams’ risk management activities, status, and results should be held on a
                                   periodic and event-driven basis. The reviews should include appropriate levels of management,
                                   such as key NOAA and NASA executives, who can provide visibility into the potential for project
                                   risk exposure and appropriate corrective actions.
                                          Source: GAO analysis based on the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition (Version
                                          1.3), the International Organization for Standardization, Defense Acquisition University, and GAO guidance on risk management
                                          practices.




GOES-R Has Defined a                      NOAA has established policies and procedures for effective risk
Comprehensive Risk                        management for GOES-R. For example, the program has documented a
Management Process                        strategy for managing risks that includes important elements, such as
                                          relevant stakeholders and their responsibilities and the criteria for
                                          evaluating, categorizing, and prioritizing risks. The program’s approved
                                          risk management plan also includes requirements for risk mitigation—
                                          such as required actions, deadlines, and assigned risk owners—as well
                                          as requirements that risks’ status and changes are periodically reviewed
                                          by appropriate managers, including senior NOAA and NASA managers.
                                          Table 11 compares GOES-R’s risk management policies and procedures
                                          with recognized risk management practices for four areas.




                                          Page 33                                                             GAO-12-576 Geostationary Weather Satellites
Table 11: Comparison of GOES-R Risk Management Policies and Procedures to Recognized Practices
                        GOES-R policies
Risk practice           and procedures       Rationale
Risk preparation        Fully meets criteria The program has established and maintains its strategy in an approved risk management
                                  ●          plan. The plan defines important aspects of the program’s risk process, including
                                             relevant stakeholders and designated responsibilities for risk management. It also
                                             defines other important aspects of the program’s risk process, relevant sources of risks,
                                             and the criteria to be used in risk evaluation, categorization, and prioritization. The
                                             approved plan requires the use of a risk database application as the official risk
                                             repository and formal training on the program’s risk process as needed.
Risk identification and Fully meets criteria The program’s approved risk plan requires projects to identify internal and external
analysis                          ●          program risks, defines the evaluation parameters and categories to be used in evaluating
                                             and grouping risks, and requires assessments of the relative priority of its risks. Also, the
                                             program’s schedule management plan requires the program to conduct a schedule risk
                                             assessment.
Risk mitigation         Fully meets criteria The program’s approved risk plan requires risks to have documented handling options
                                  ●          along with the required steps, deadlines, triggers, and assigned risk owners for mitigating
                                             the risk. The plan also requires program and project review boards to monitor risk status
                                             and make official decisions regarding risk review, acceptance, and closure as well as
                                             updates to track the status of mitigation efforts.
Executive risk          Fully meets criteria The program’s approved risk plan requires appropriate levels of management to review
oversight                         ●          the activities, status, and results of the risk management process covering the most
                                             critical risks, key risk parameters, and the status of mitigation efforts. It also requires that
                                             the program’s risk manager track actions assigned by a risk review board.
                                              Source: GAO analysis based on NOAA and NASA data.

                                              Key
                                              ●= The program has fully met the criteria for this practice
                                              ◕= The program has substantially met the criteria for this practice
                                              ◑= The program has partially met the criteria for this practice
                                              ◔= The program has minimally met the criteria for this practice
                                              ○= The program has not met the criteria for this practice

                                              With such policies and procedures in place, NOAA has established a
                                              comprehensive framework to support its identification, mitigation, and
                                              oversight of risks across the program, and laid a foundation for consistent
                                              implementation.


The Program Has Not                           While the program has a well-defined risk management process, this
Fully Implemented Its Risk                    process has not been fully implemented. Table 12 identifies the extent to
Management Process or                         which the program has implemented recognized risk management
                                              practices through its risk process. Of particular note, the program has not
Adequately Mitigated Key                      conducted and documented adequate or timely evaluations for all
Risks                                         potential risks—called candidate risks in the risk management plan—in its
                                              risk list. In addition, the program did not always document its risk handling
                                              strategies and time frames and did not always provide adequate rationale
                                              for its decision to close risks.


                                              Page 34                                              GAO-12-576 Geostationary Weather Satellites
Table 12: GOES-R Implementation of Risk Practices

                        GOES-R
                        application of its
Risk practice           risk process         Rationale
Risk preparation        Substantially        The program has established a risk database to communicate risk information and
                        meets criteria       provided risk training to the Ground Project Office in October 2011.
                                             However, the program has not provided formal risk training to the Flight Project Office.
                                ◕            Program officials stated that they plan to provide formal training to the flight project when its
                                             critical design is complete. Until the program completes training for both projects, the
                                             program will have less assurance that all staff are consistently applying its risk management
                                             process and associated tools for communicating risk assessments and status.
Risk identification and Partially meets      The program has identified certain internal and external program risks, has generally
analysis                criteria             evaluated open risks based on defined parameters, and has assessed the relative priority
                                             of its risks.
                                ◑            However, the program has not provided adequate or timely evaluations for certain
                                             candidate risks. Specifically, the program did not assess likelihood or potential impact for
                                             49 out of its 100 open candidate risks as required by the program’s risk plan. Further, 40
                                             candidate risks had not yet been examined by project review boards for more than 3
                                             months, and 15 of these were first identified in 2010. According to the program’s risk
                                             manager, candidate risks are generally to be assessed and presented to a project risk
                                             board within 1 to 3 months.
                                             In addition, the program is not categorizing its risks, thus managers may be limited in their
                                             ability to provide timely attention and efficient handling of these risks in context with the
                                             source of the risk. Also, the program only recently completed a schedule risk assessment in
                                             February 2012, the lack of which may have limited managers’ ability to understand which
                                             risks may be most critical to address in maintaining the program’s schedule.
Risk mitigation         Partially meets      The program has generally developed mitigation plans for critical risks to include required
                        criteria             steps and deadlines, assigned owners responsibility for managing risks, and monitored
                                             the status of risk mitigation efforts through monthly project review board meetings.
                                ◑            However, the program did not always document mitigation options for open and
                                             candidate risks and did not always include triggers or thresholds for when fallback actions
                                             were to be executed. In addition, the program did not include adequate rationale for its
                                             decision to close active risks. Thus, it is not clear whether these risks had been
                                             adequately mitigated prior to closure.
Executive risk          Substantially        The program and lower-level project review boards review risk status and results
oversight               meets criteria       monthly. Further, the NOAA Program Management Council reviews the risk status and
                                             results for the program’s top risks monthly.
                                ◕            However, the program has not tracked corrective actions from risk review board meetings
                                             or meetings with the NOAA Program Management Council. This may limit management’s
                                             ability to understand the number or status of outstanding actions and future GOES
                                             acquisitions’ ability to learn from the current program’s risk resolution approach.
                                             Source: GAO analysis based on NOAA and NASA data.

                                             Key
                                             ●= The program has fully met the criteria for this practice
                                             ◕= The program has substantially met the criteria for this practice
                                             ◑= The program has partially met the criteria for this practice
                                             ◔= The program has minimally met the criteria for this practice
                                             ○= The program has not met the criteria for this practice


                                             Page 35                                              GAO-12-576 Geostationary Weather Satellites
                                            The GOES-R Program Office has identified 11 risks that it considers
                                            critical (medium- or high-level risks) and could significantly impact the
                                            program’s development cost or schedule, documented mitigation
                                            approaches for each risk, and tracked mitigation progress. For these and
                                            its other risks, the program generally has mitigation plans in place that
                                            typically include actions to be taken, deadlines, and assigned
                                            responsibilities that could help to minimize or control the occurrence of
                                            these critical risks. Table 12 describes the program-identified critical risks,
                                            as of February 2012.

Table 13: GOES-R Program Critical Risks, as of February 2012

                                                                                                                         Program-
                                                                                                                         identified risk
Program-identified critical risks   Program risk description                                                             level
Program funding stability           If the program’s funding profile continues to be changed, then its life cycle cost   High
                                    will increase.
GOES-R development schedule         If flight and ground project developments (including their dependencies) are         Medium
                                    delayed, then development schedule will be impacted through delayed work or
                                    rework, which will increase costs and may adversely affect launch readiness.
Global positioning system (GPS)     If GPS timing or frequency interference occurs, then there will be a degradation Medium
timing and frequency interference   or loss of GOES-R capabilities with dependencies on GPS resulting in schedule
with GOES-R                         delays and cost increases from rework or redesign.
GOES-R user readiness               If user or NOAA infrastructure upgrades necessary to ensure compatibility with       Medium
                                    GOES-R do not align with planned deployment schedules, then there is a
                                    possibility that specific users will be unable to use the data products.
Interference at the spacecraft with If radiated emissions from instruments or other satellite hardware exceed        Medium
the GPS receiver                    specifications, then GOES-R satellite performance may be significantly degraded.
Impact of spacecraft changes on     If there are significant schedule changes in the final spacecraft and ground         Medium
launch critical release             system-related documents regarding telemetry tracking and control, spacecraft
                                    navigation, and product monitoring, then there could be impacts to a major
                                    ground software release.
Inertial Reference Unit delivery    If Inertial Reference Unit development and compliance continues to be delayed, Medium
                                    then the mission readiness review may not be successful, resulting in delayed
                                    GOES-R hardware delivery and cost and schedule impacts.
Geostationary Lightning Mapper      If the GLM instrument is delayed beyond planned delivery, then spacecraft            Medium
(GLM) late delivery                 integration and test will be impacted.
GLM ground processing algorithm If the GLM ground processing algorithm is not available from the contractor in           Medium
correction for ghosting         time, then the instrument’s algorithm will be degraded for post-launch testing.
Program budget carryover            If funding is not made available in a timely manner before carryover is              Medium
                                    expended, then there is the potential for additional program cost and schedule
                                    delays.
Program budget contingency          If the flight or ground project experiences future difficulties in maintaining       Medium
(reserve)                           agreed-to contingency levels, then additional adjustments or descopes may be
                                    required with impacts to program scope or launch readiness dates.
                                            Source: GAO analysis of NOAA and NASA data.




                                            Page 36                                          GAO-12-576 Geostationary Weather Satellites
While the program has documented mitigation plans in place for most of
its critical risks, the program is not mitigating its most critical risk—
program funding stability. Although the program has included this risk on
its top risk list and presented it to the NOAA Program Management
Council, it has not devised options for required replanning or functionality
descopes should the program experience reduced funding. Program
officials stated that the risk is external and beyond their control. However,
given that the program has made trade-off decisions regarding available
funding, functionality, and the timing of its work, it is reasonable to expect
the program to have plans in place that include possible trade-off
decisions based on different outcomes, including triggers for when
decisions need to be executed.

Further, there is at least one critical risk that is not on the program’s top
risk list that could further jeopardize the program’s launch readiness dates
and life cycle costs: GOES-S milestones may be affected by delays that
have occurred or may still occur during GOES-R development. As
discussed earlier in this report, further delays in the development of the
first satellite could result in problems for the second satellite’s scheduled
milestones since the program is planning to complete the second
satellite’s activities during periods of time set aside as reserve to
complete the first satellite’s activities. Program officials stated that a risk
has recently been added to the flight project’s risk list to reflect the
potential delays to the GOES-S development schedule. Until this risk is
added to the program’s risk list with a documented mitigation approach
and regular monitoring, NOAA could delay the analysis, planning, and
actions that would limit the impacts and occurrence of the risk, and would
thus be unprepared to face the significant consequences to the program
should this risk be realized.

While the program has well-defined policies and procedures, it has not
fully implemented its risk process. Fully implementing the recognized risk
management practices defined in GOES-R policies and procedures would
provide program officials with the assurance that all risks—including
those that are new and most critical—are adequately addressed.




Page 37                                 GAO-12-576 Geostationary Weather Satellites
              NOAA has made progress toward achieving GOES-R program goals by
Conclusions   completing certain preliminary and critical design milestones for its flight
              and ground projects. However, associated reviews, including the
              programwide critical design review planned for August 2012, have been
              delayed by many months. Progress was also accompanied by technical
              and programmatic challenges, such as initial failures of instrument
              components and replanning of ground project software releases.
              Although these two specific challenges are being addressed, others such
              as instrument signal problems and a 2-year cost growth of more than 30
              percent for program development contracts point to troubling patterns that
              will require ongoing remediation and monitoring. NOAA has allocated
              budget reserves for such situations according to its established guidelines
              and reports that project reserve levels are currently within those
              guidelines. However, the program has used approximately 30 percent of
              its reserves over the last 3 years and significant portions of development
              remain for major components—including the spacecraft and Core Ground
              System. In addition, the program did not change its reserves when it
              restored two satellites adding approximately $3.2 billion to the program’s
              baseline. Unless NOAA assesses the reserve allocations across all of the
              program’s development efforts, it may not be able to ensure that its
              reserves will cover ongoing challenges as well as unexpected problems
              for the remaining development of all four satellites in the series.

              The unreliability of the program’s integrated master schedule and some
              contractor schedules adds further uncertainty to whether the program will
              meet its commitments. The issues that exist among these schedules,
              such as a lack of a full and consistent allocation of resources, incomplete
              logic, and gaps in the critical path to project completion, are inconsistent
              with our and agency best practices. NOAA has taken steps to improve
              schedule reliability through more automated schedule integration, a
              cross-project deliverable-tracking database, and its first schedule risk
              assessment. However, unless the program addresses the full set of
              scheduling weaknesses we identified, its schedules may not provide a
              fully reliable basis for decision making.

              NOAA has defined the policies and procedures it needs to effectively
              manage and mitigate these and other program risks. Nevertheless,
              officials do not have a current and comprehensive view of program risk
              because these policies and procedures have not been fully implemented.
              Most significantly, not all known candidate risks have been evaluated;
              corrective actions have not been consistently tracked; and risks we
              identified in this review are not being tracked or adequately mitigated.
              Until program officials diligently execute all of the program’s defined risk


              Page 38                                GAO-12-576 Geostationary Weather Satellites
                      management practices and integrate these with improvements in the
                      management of reserves and schedules, the program is at risk of
                      exceeding cost and schedule targets and further slipping launch dates for
                      satellites in the GOES-R series.


                      To improve NOAA’s ability to execute GOES-R’s remaining planned
Recommendations for   development with appropriate reserves, improve the reliability of its
Executive Action      schedules, and address identified program risks, we recommend that the
                      Secretary of Commerce direct the NOAA Administrator to ensure that the
                      following four actions are taken:

                      •   Assess and report to the NOAA Program Management Council the
                          reserves needed for completing remaining development for each
                          satellite in the series.

                      •   For all satellites in the GOES-R program, including those for which
                          detailed scheduling has yet to begin, address shortfalls in the
                          schedule management practices we identified, including but not
                          limited to incorporating appropriate schedule logic, eliminating
                          unnecessary constraints, creating a realistic allocation of resources,
                          ensuring an unbroken critical path from the current date to the final
                          satellite launch, and ensuring that all subcontractor activities are
                          incorporated in contractors’ integrated master schedules.

                      •   Execute the program’s risk management policies and procedures to
                          provide more timely and adequate evaluations and reviews of newly
                          identified risks, documented handling strategies for all ongoing and
                          newly identified risks in the risk register, time frames for when risk
                          mitigation and fallback plans are to be executed, adequate rationale
                          for decisions to close risks, and documentation and tracking of action
                          items from risk review board meetings or other meetings with senior
                          NOAA and NASA managers through completion.

                      •   Given the potential impact to the program, add the risk that GOES-S
                          milestones may be affected by GOES-R development to the
                          program’s critical risk list, and ensure that this risk and the program-
                          identified funding stability risk are adequately monitored and
                          mitigated.




                      Page 39                                 GAO-12-576 Geostationary Weather Satellites
                     We received written comments on a draft of this report from the Secretary
Agency Comments      of Commerce, who transmitted NOAA’s comments. The department
and Our Evaluation   concurred with three of our recommendations and partially concurred with
                     one recommendation. It also provided general comments, which are
                     addressed below, and technical comments, which we have incorporated
                     into our report, as appropriate. A copy of NOAA’s comments is provided
                     in appendix II.

                     NOAA concurred with our first recommendation to assess and report on
                     the reserves needed for completing the remaining development for each
                     satellite in the series to the agency’s Program Management Council. It
                     stated that the GOES-R program would continue to provide status reports
                     on contingency reserves to the Goddard Space Flight Center and the
                     Program Management Council, and would work with NOAA to ensure
                     contingency reporting meets its needs. It also stated in its general
                     comments that contingency reserve for GOES-T and GOES-U
                     (amounting to 20 percent of their projected development costs) was
                     included when the $3.2 billion for these satellites was added to the
                     program budget baseline in February 2011. NOAA did not provide any
                     additional data on its fiscal year 2012 contingency budget allocations by
                     satellite to support this statement and, as we discussed in this report, the
                     program did not report a significant change in overall program reserve
                     levels when it revised the baseline from two satellites to four satellites.

                     NOAA concurred with our second recommendation to address shortfalls
                     that we identified in schedule management practices and stated that the
                     program will continue to bring down the number of errors in the schedules
                     and improve the fidelity of the program’s integrated master schedule.

                     NOAA partially concurred with our third recommendation to fully execute
                     the program’s risk management policies and procedures, to include timely
                     review and disposition of candidate risks. NOAA stated that it did not
                     consider the “concerns” listed in its risk database to be risks or candidate
                     risks, and that the risk management board actively determines whether
                     recorded concerns should be elevated to a risk. However, the program
                     has not treated concerns in accordance with its risk management plan,
                     which considers these to be “candidate risks” and requires their timely
                     review and disposition, as evidenced by the many concerns in the
                     database that were more than 3 months old and had not been assessed
                     or dispositioned. Unless NOAA follows its risk management plan by
                     promptly evaluating “concerns,” it cannot ensure that it is adequately
                     managing the full set of risks that could impact the program.



                     Page 40                                GAO-12-576 Geostationary Weather Satellites
NOAA concurred with our fourth recommendation to add to the program’s
critical risk list that GOES-S milestones may be affected by GOES-R
development and to ensure that this risk and the program-identified
funding stability risk are adequately monitored and mitigated. NOAA
stated that it currently has identified and reported on the GOES-S
milestone risk and that the program is working with NOAA to monitor and
mitigate the funding stability risk.

In a general comment on our discussion of contractors’ estimated cost
growth, NOAA stated that certain growth in contract costs was due to
scope changes, including instrument options that were exercised in 2010
and 2011, and stated that such changes should be distinguished from
cost growth due to efforts to resolve problems. We consider scope
changes to be a valid component of program cost growth. Nevertheless,
we revised our report to state that approximately $60 million of the $757
million growth in contractors’ estimated costs at completion from January
2010 through November 2011 was due to scope changes associated with
new instrument flight model development. If these scope changes are
excluded, contractors’ estimated costs for flight and ground project
components still grew by approximately $700 million, or approximately 30
percent, between January 2010 and January 2012.

NOAA commented that the GOES-R launch readiness date changed from
December 2014 to October 2015 due to a protest against the spacecraft
contract award. We acknowledged the reasons for delays in GOES-R’s
launch readiness date in our draft report submitted to NOAA and in our
prior reports on this program, and have added language to this report that
relates to the bid protest and NASA’s response in delaying the award.

NOAA also commented that the revised incremental schedule was less
risky than its original waterfall schedule. While we acknowledged the
reduced risk in our draft report and have further revised our report to
reflect NOAA’s comments on our draft, we also note that the revised
development methodology introduced other risks to the program—such
as additional contractor staff and software development and verification
activities that require government oversight and continuous monitoring.




Page 41                               GAO-12-576 Geostationary Weather Satellites
As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 1 day from the
report date. At that time, we will send copies to interested congressional
committees, the Secretary of Commerce, the Administrator of NASA, the
Director of the Office of Management and Budget, and other interested
parties. The report also will be available at no charge on the GAO website
at http://www.gao.gov.

If you or your staffs have any questions on the matters discussed in this
report, please contact me at (202) 512-9286 or pownerd@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who made
major contributions to this report are listed in appendix III.




David A. Powner
Director
Information Technology
   Management Issues




Page 42                               GAO-12-576 Geostationary Weather Satellites
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              Our objectives were to (1) assess the National Oceanic and Atmospheric
              Administration’s (NOAA) progress in developing the Geostationary
              Operational Environmental Satellite-R series (GOES-R) program, (2)
              evaluate whether the agency has a reliable schedule for executing the
              program, and (3) determine whether the program is applying best
              practices in managing and mitigating its risks.

              To assess progress in developing the GOES-R satellite program, we
              compared the program’s planned completion dates for key milestones
              identified in its management control plan and system review plan against
              actual and current estimated completion dates of milestones. We
              analyzed program monthly status briefings to identify the current status
              and recent development challenges of flight and ground project
              components and instruments. We also analyzed contractor- and program-
              reported data of development costs and reserves. Finally, we conducted
              several interviews with GOES-R program staff to better understand
              milestone time frames, to discuss current status and recent development
              challenges for work currently being performed on GOES-R, and to
              understand how the program reports costs and reserve totals.

              To evaluate whether NOAA has a reliable schedule for executing the
              program, we evaluated contractor schedules to determine the extent to
              which GOES-R is following our identified best practices in creating and
              maintaining its schedules. 1 We analyzed four contractor schedules—two
              of which represent the overall schedules for the flight and ground projects
              (spacecraft and Core Ground System), the program’s critical flight
              instrument (Advanced Baseline Imager), and an instrument that had been
              experiencing implementation issues at the time of our review
              (Geostationary Lightning Mapper). We also populated workbooks as a
              part of that analysis to highlight potential areas of strengths and
              weakness in schedule logic, use of resources, task duration, float, and
              task completion; analyzed programwide initiatives undertaken by GOES-
              R such as the Joint Cost and Schedule Confidence Level and the Giver-
              Receiver Intersegment Database; assessed GOES-R’s progress against
              their own scheduling requirements, and interviewed government and
              contractor officials regarding their scheduling practices.



              1
               The subordinate schedules analyzed were the Advanced Baseline Imager, the
              Geostationary Lightning Mapper, and the schedules for GOES-R’s flight and ground
              projects—the spacecraft and Core Ground System.




              Page 43                                     GAO-12-576 Geostationary Weather Satellites
Appendix I: Objectives, Scope, and
Methodology




To determine whether the GOES-R program is applying best practices in
managing and mitigating its risks, we analyzed the program’s risk
management plan to identify the program’s policies and procedures and
compared these to outputs from the program’s risk register and other
program risk management documentation, such as risk mitigation status
reports. We also reviewed documents such as monthly briefings to the
NOAA Program Management Council and program risk status reports to
determine how individual risks are managed and reviewed on a regular
basis. We assessed the extent to which the program’s policies and
procedures met government and industry recognized risk management
practices from the Software Engineering Institute’s Capability Maturity
Model® Integration for Acquisition (Version 1.3), the International
Organization for Standardization, the Defense Acquisition University, and
GAO, and whether the program fully implemented its policies and
procedures. We also examined the program’s risk status reports and risk
register to identify risks for which the program did not have an adequate
mitigation plan or were not being actively monitored by the program.
Finally, we interviewed GOES-R officials to gain further insight into the
program’s risk management process and the program’s application of its
process.

We primarily performed our work at NOAA and National Aeronautics and
Space Administration offices in the Washington, D.C., metropolitan area.
We conducted this performance audit from August 2011 to June 2012, in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.




Page 44                              GAO-12-576 Geostationary Weather Satellites
Appendix II: Comments from the Department
             Appendix II: Comments from the Department
             of Commerce



of Commerce




             Page 45                                     GAO-12-576 Geostationary Weather Satellites
Appendix II: Comments from the Department
of Commerce




Page 46                                     GAO-12-576 Geostationary Weather Satellites
Appendix II: Comments from the Department
of Commerce




Page 47                                     GAO-12-576 Geostationary Weather Satellites
Appendix II: Comments from the Department
of Commerce




Page 48                                     GAO-12-576 Geostationary Weather Satellites
Appendix III: GAO Contact and Staff
                  Appendix III: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  David A. Powner, (202) 512-9286 or pownerd@gao.gov
GAO Contact
                  In addition to the contact named above, individuals making contributions
Staff             to this report included Colleen Phillips (assistant director), Paula Moore
Acknowledgments   (assistant director), Shaun Byrnes, Nancy Glover, Franklin Jackson,
                  Jason Lee, and Josh Leiling.




311257
                  Page 49                               GAO-12-576 Geostationary Weather Satellites
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.