United States Government Accountability Office GAO A Report to the Committee on Science, Space, and Technology, House of Representatives June 2012 GEOSTATIONARY WEATHER SATELLITES Design Progress Made, but Schedule Uncertainty Needs to be Addressed GAO-12-576 June 2012 GEOSTATIONARY WEATHER SATELLITES Design Progress Made, but Schedule Uncertainty Needs to be Addressed Highlights of GAO-12-576, a report to the Committee on Science, Space, and Technology, House of Representatives Why GAO Did This Study What GAO Found The GOES-R series is a set of four The Geostationary Operational Environmental Satellite-R series (GOES-R) satellites intended to replace existing program has made progress by completing its early design milestones and is weather satellites that will likely reach nearing the end of the design phase for its spacecraft, instrument, and ground the end of their useful lives in about system components. While the program continues to make progress, recent 2015. NOAA estimates the series to technical problems with the instruments and spacecraft, as well as a significant cost $10.9 billion through 2036. modification to the ground project’s development plan, have delayed the Because the transition to the series is completion of key reviews and led to increased complexity for the development of critical to the nation’s ability to maintain GOES-R. The technical and programmatic challenges experienced by the flight the continuity of data required for and ground projects have led to a 19-month delay in completing the program’s weather forecasting, GAO reviewed preliminary design review. Nevertheless, program officials report that its planned NOAA’s management of the GOES-R program. launch date of October 2015 for the first satellite has not changed. While the program reports that approximately $1.2 billion is currently in reserve to manage Specifically, GAO was asked to (1) future delays and cost growth, significant portions of development remain for assess NOAA’s progress in developing major components. As a result, the program may not be able to ensure that it has the GOES-R satellite program, (2) adequate resources to cover ongoing challenges as well as unexpected evaluate whether the agency has a problems for the remaining development of all four satellites. reliable schedule for executing the program, and (3) determine whether The success in management of a large-scale program depends in part on having the program is applying best practices a reliable schedule that defines, among other things, when work activities and in managing and mitigating its risks. milestone events will occur, how long they will take, and how they are related to one another. To its credit, the program has adopted key scheduling best GAO analyzed program management, practices and has recognized certain scheduling weaknesses. It has also acquisition, and cost data; evaluated contractor and program-wide recently instituted initiatives to automate its integrated master schedule, correct schedules against best practices; integration problems among projects, and assess schedule confidence based on analyzed program documentation risk. However, unresolved schedule deficiencies remain in its integrated master including risk management plans and schedule and the contractor schedules that support it, which have contributed to procedures; and interviewed a re-plan of the schedule of the ground system and to the potential for delays to government and contractor staff satellite launch dates. The program recently determined that the likelihood of the regarding program progress and first satellite meeting its planned October 2015 launch date is 48 percent. Based challenges. on this planned launch date, the program reports that there is a 37 percent chance of a gap in the availability of two operational GOES-series satellites, What GAO Recommends which could result in the need for the National Oceanic and Atmospheric Administration (NOAA) to rely on older satellites that are not fully functional. Until GAO is making recommendations to its scheduling weaknesses are addressed, it will be more difficult for the program NOAA to assess and report reserves to know whether its planned remaining development is on schedule. needed over the life of the program, improve the reliability of its schedules, NOAA has established policies and procedures that conform with recognized risk and address identified program risks. management best practices. For example, the program has documented a NOAA concurred or partially concurred strategy for managing risks that includes important elements, such as relevant with GAO’s recommendations. stakeholders and their responsibilities and the criteria for evaluating, categorizing, and prioritizing risks. However, while the program has a well- defined risk management process, it has not been fully implemented. For example, the program has not provided adequate or timely evaluations for potential risks, did not always provide adequate rationale for the decision to close a risk, and has at least two critical risks in need of additional attention. Until all defined risk management practices are diligently executed and critical risks View GAO-12-576. For more information, contact David A. Powner at (202) 512-9286 or adequately mitigated, the GOES-R program is at risk of exceeding cost and firstname.lastname@example.org. schedule targets, and launch dates could slip. United States Government Accountability Office Contents Letter 1 Background 2 Progress Made in Completing Early Design, but Milestones Were Completed Late and Development Costs Have Increased 12 GOES-R Schedules Are Not Fully Reliable, Contributing to Milestone and Potential Launch Date Delays 23 Risk Management Approach Is Well Defined but Not Fully Implemented 32 Conclusions 38 Recommendations for Executive Action 39 Agency Comments and Our Evaluation 40 Appendix I Objectives, Scope, and Methodology 43 Appendix II Comments from the Department of Commerce 45 Appendix III GAO Contact and Staff Acknowledgments 49 Tables Table 1: Summary of the Procurement History of GOES 4 Table 2: Originally Planned GOES-R Series Instruments, as of August 2006 6 Table 3: Key Changes to the GOES-R Program 8 Table 4: Delays in Program Milestones 14 Table 5: Flight Project Development Efforts and Challenges through February 2012 15 Table 6: Ground Project Development Efforts and Challenges through February 2012 17 Table 7: Growth in Estimated Contract Cost for Major Program Components 18 Table 8: Description of Scheduling Best Practices 24 Table 9: Practices Utilized in Selected GOES-R Schedules 26 Table 10: Description and Examples of Recognized Risk Management Best Practices 33 Table 11: Comparison of GOES-R Risk Management Policies and Procedures to Recognized Practices 34 Page i GAO-12-576 Geostationary Weather Satellites Table 12: GOES-R Implementation of Risk Practices 35 Table 13: GOES-R Program Critical Risks, as of February 2012 36 Figures Figure 1: Approximate GOES Geographic Coverage 3 Figure 2: Generic GOES Data Relay Pattern 5 Figure 3: GOES-R Program Office Reporting, Structure, and Staffing 10 Figure 4: Recent Growth in Estimated Cost for Selected Development Components 19 Figure 5: Changes in Program Reserves Over Time 21 Figure 6: Changes in Planned Completion Dates for Key Milestones of the First GOES-R Satellite 30 Abbreviations CDR critical design review GOES Geostationary Operational Environmental Satellite GOES-R Geostationary Operational Environmental Satellite-R series IMS integrated master schedule NASA National Aeronautics and Space Administration NOAA National Oceanic and Atmospheric Administration PDR preliminary design review This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-12-576 Geostationary Weather Satellites United States Government Accountability Office Washington, DC 20548 June 26, 2012 The Honorable Ralph Hall Chairman The Honorable Eddie Bernice Johnson Ranking Member Committee on Science, Space, and Technology House of Representatives Operational geostationary environmental satellites play a critical role in our nation’s weather forecasting. These satellites—which are managed by the Department of Commerce’s National Oceanic and Atmospheric Administration (NOAA)—provide critical information on atmospheric, oceanic, climatic, and solar conditions that help meteorologists observe and predict global and local weather events. They also provide a means of identifying the large-scale evolution of severe storms, such as hurricane track and intensity forecasting and complementing radar in tornado or heavy precipitation forecasting. NOAA, through collaboration with the National Aeronautics and Space Administration (NASA), is procuring the next generation of geostationary satellites, called the Geostationary Operational Environmental Satellite-R (GOES-R) series. The GOES-R series is to replace the current series of satellites, which will likely begin to reach the end of their useful lives in approximately 2015. This new series is expected to mark the first major improvement in GOES instrumentation since 1994. It is also considered critical to the United States’ ability to maintain the continuity of data required for weather forecasting through the year 2036. This report responds to your request that we review NOAA’s management of the GOES-R program. Specifically, we were asked to (1) assess NOAA’s progress in developing the GOES-R satellite program, (2) evaluate whether the agency has a reliable schedule for executing the program, and (3) determine whether the program is applying best practices in managing and mitigating its risks. To assess NOAA’s progress in developing GOES-R, we analyzed program documentation, including monthly status reports, acquisition plans, and contractor performance reports on development efforts. To evaluate whether the agency has a reliable schedule for executing the program, we evaluated four contractor schedules including two instruments—the Advanced Baseline Imager and Geostationary Lightning Page 1 GAO-12-576 Geostationary Weather Satellites Mapper—the spacecraft, and the Core Ground System. We also analyzed programwide schedule initiatives. We compared contractor schedules and program initiatives to GAO-identified best practices in developing and maintaining schedules. To determine whether the program is applying best practices in managing and mitigating its risks, we analyzed program documentation, including the program’s risk management plan and outputs from its risk register. We compared program risk management policies and procedures to government and industry recognized risk management best practices and compared the program’s implementation of its risk management policies and procedures. We also interviewed government and contractor staff to discuss the program’s development progress and recent challenges as well as their scheduling and risk management practices. We conducted this performance audit from August 2011 to June 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix I contains further details on our objectives, scope, and methodology. Since the 1960s, geostationary satellites have been used by the United Background States to provide meteorological data for weather observation, research, and forecasting. NOAA’s National Environmental Satellite, Data, and Information Service is responsible for managing the civilian operational geostationary satellite system, called GOES. Geostationary satellites can maintain a constant view of the earth from a high orbit of about 22,300 miles in space. NOAA operates GOES as a two-satellite system that is primarily focused on the United States (see fig. 1). These satellites provide timely environmental data about the earth’s atmosphere, surface, cloud cover, and the space environment to meteorologists and their audiences. They also observe the development of hazardous weather, such as hurricanes and severe thunderstorms, and track their movement and intensity to reduce or avoid major losses of property and life. The ability of the satellites to provide broad, continuously updated coverage of atmospheric conditions over land and oceans is important to NOAA’s weather forecasting operations. Page 2 GAO-12-576 Geostationary Weather Satellites Figure 1: Approximate GOES Geographic Coverage To provide continuous satellite coverage, NOAA acquires several satellites at a time as part of a series and launches new satellites every few years (see table 1). 1 NOAA’s policy is to have two operational satellites and one backup satellite in orbit at all times. 1 Satellites in a series are identified by letters of the alphabet when they are on the ground (before launch) and by numbers once they are in orbit. Page 3 GAO-12-576 Geostationary Weather Satellites Table 1: Summary of the Procurement History of GOES Series name Procurement durationa Satellites b Original GOES 1970-1987 1, 2, 3, 4, 5, 6, 7 GOES I-M 1985-2001 8, 9, 10, 11, 12 GOES N 1998-2010 13, 14, 15, Qc GOES-R 2008-2024 R, S, T, U Source: GAO analysis of NOAA data. a Duration includes time from contract award to final satellite launch. b The procurement of these satellites consisted of four separate contracts for (1) two early prototype satellites and GOES-1, (2) GOES-2 and -3, (3) GOES-4 through -6, and (4) GOES-G (failed on launch) and GOES-7. c NOAA decided not to exercise the option for this satellite. Four GOES satellites–GOES-12, GOES-13, GOES-14, and GOES-15– are currently in orbit. Both GOES-13 and GOES-15 are operational satellites with GOES-13 covering the eastern United States and GOES- 15 in the western United States (see fig. 1). GOES-14 is currently in an on-orbit storage mode and available as a backup for the other two satellites should they experience any degradation in service. GOES-12 is at the end of its service life, but it is being used to provide limited coverage of South America. The GOES-R series is the next generation of satellites that NOAA is planning; the first satellite in the series is planned for launch beginning in 2015. Each of the operational geostationary satellites continuously transmits raw environmental data to NOAA ground stations. The data are processed at these ground stations and transmitted back to the satellite for broadcast to primary weather services and the global research community in the United States and abroad. Raw and processed data are also distributed to users via ground stations through other communication channels, such as dedicated private communication lines and the Internet. Figure 2 depicts a generic data relay pattern from the geostationary satellites to the ground stations and commercial terminals. Page 4 GAO-12-576 Geostationary Weather Satellites Figure 2: Generic GOES Data Relay Pattern Overview of the GOES-R NOAA plans for the GOES-R program to improve on the technology of Program prior series, in terms of both system and instrument improvements. The system improvements are expected to fulfill more demanding user requirements by updating the satellite data more often and providing satellite products to users more quickly. The instrument improvements are expected to significantly increase the clarity and precision of the observed environmental data. NOAA originally planned to acquire six different types of instruments. Furthermore, two of these instruments—the Advanced Baseline Imager and the Hyperspectral Environmental Suite— were considered to be the most critical because they would provide data for key weather products. Table 2 summarizes the originally planned instruments and their expected capabilities. Page 5 GAO-12-576 Geostationary Weather Satellites Table 2: Originally Planned GOES-R Series Instruments, as of August 2006 Planned instrument Description Advanced Baseline Imager Expected to provide variable area imagery and radiometric information of the earth’s surface, atmosphere, and cloud cover. Key features include • monitoring and tracking severe weather; • providing images of clouds to support forecasts; and • providing higher resolution, faster coverage, and broader coverage simultaneously. Hyperspectral Environmental Suitea Expected to provide information about the earth’s surface to aid in the prediction of weather and climate monitoring. Key features include • providing atmospheric moisture and temperature profiles of the rapidly evolving pre-storm convective environment to support forecasts and warnings of high-impact weather phenomena; • monitoring coastal regions for ecosystem health, water quality, coastal erosion, and harmful algal blooms; and • providing higher resolution and faster coverage. Geostationary Lightning Mapper Expected to continuously monitor total lightning (in-cloud and cloud-to-ground) activity over the United States and adjacent oceans and to provide a more complete dataset than previously possible. Key features include • detecting lightning activity as an indicator of severe storms and convective weather hazard impacts to aviation and • providing a new capability to GOES for long-term mapping of total lightning that only previously existed on NASA low-earth-orbiting research satellites. Magnetometer Expected to provide information on the general level of geomagnetic activity, monitor current systems in space, and permit detection of magnetopause crossings, sudden storm commencements, and substorms. Space Environmental In-Situ Suite Expected to provide information on space weather to aid in the prediction of particle precipitation, which causes disturbance and disruption of radio communications and navigation systems. Key features include • measuring magnetic fields and charged particles; • providing improved heavy ion detection, adding low-energy electrons and protons; and • enabling early warnings for satellite and power grid operation, telecom services, astronauts, and airlines. Solar Imaging Suiteb Expected to provide coverage of the entire dynamic range of solar X-ray features, from coronal holes to X-class flares, as well as estimate the measure of temperature and emissions. Key features include • providing images of the sun and measuring solar output to monitor solar storms and • providing improved imager capability. Source: GAO analysis of NOAA data. a The Hyperspectral Environmental Suite was cancelled in September 2006. b The Solar Imaging Suite was divided into two separate acquisitions: the Solar Ultraviolet Imager and the Extreme Ultraviolet/X-Ray Irradiance Sensor. Page 6 GAO-12-576 Geostationary Weather Satellites In September 2006, NOAA decided to reduce the scope and technical complexity of the GOES-R program because of expectations that total costs, which were originally estimated to be $6.2 billion, could reach $11.4 billion. 2 Specifically, NOAA reduced the minimum number of satellites from four to two, cancelled plans for developing the Hyperspectral Environmental Suite (which reduced the number of planned satellite products from 81 to 68), and divided the Solar Imaging Suite into two separate acquisitions. In light of the cancellation of the Hyperspectral Environmental Suite, NOAA decided to use the planned Advanced Baseline Imager to develop certain satellite data products that were originally to be produced by this instrument. The agency estimated that the revised program would cost $7 billion. Subsequently, NOAA made several other important decisions about the cost and scope of the GOES-R program. 3 In May 2007, NOAA had an independent cost estimate completed for the GOES-R program. After reconciling the program office’s cost estimate of $7 billion with the independent cost estimate of about $9 billion, the agency established a new program cost estimate of $7.67 billion. This was an increase of $670 million from the previous estimate. Further, in November 2007, to mitigate the risk that costs would rise, program officials decided to remove selected program requirements from the baseline program and treat them as contract options that could be exercised if funds allowed. These requirements included the number of products to be distributed, the time to deliver the remaining products (product latency), and how often these products would be updated with new satellite data (refresh rate). For example, program officials eliminated the requirement to develop and distribute 34 of the 68 envisioned products, including low cloud and fog, 2 GAO, Geostationary Operational Environmental Satellites: Additional Action Needed to Incorporate Lessons Learned from Other Satellite Programs, GAO-06-1129T (Washington, D.C.: Sept. 29, 2006) and Geostationary Operational Environmental Satellites: Steps Remain in Incorporating Lessons Learned from Other Satellite Programs, GAO-06-993 (Washington, D.C.: Sept. 6, 2006). 3 GAO, Geostationary Operational Environmental Satellites: Acquisition Has Increased Costs, Reduced Capabilities, and Delayed Schedules, GAO-09-596T (Washington, D.C.: Apr. 23, 2009); Geostationary Operational Environmental Satellites: Acquisition Is Under Way, but Improvements Needed in Management and Oversight, GAO-09-323 (Washington, D.C.: Apr. 2, 2009); Geostationary Operational Environmental Satellites: Further Actions Needed to Effectively Manage Risks, GAO-08-183T (Washington, D.C.: Oct. 23, 2007); and Geostationary Operational Environmental Satellites: Progress Has Been Made, but Improvements Are Needed to Effectively Manage Risks, GAO-08-18 (Washington, D.C.: Oct. 23, 2007). Page 7 GAO-12-576 Geostationary Weather Satellites sulfur dioxide detection, and cloud liquid water. Program officials included the restoration of the requirements for the products, latency times, and refresh rates as options in the ground system contract that could be acquired at a later time. Program officials later reduced the number of products that could be restored as a contract option (called option 2) from 34 to 31 because they determined that two products were no longer feasible and two others could be combined into a single product. Recently, NOAA restored two satellites to the program’s baseline, making GOES-R a four-satellite program once again. In February 2011, as part of its fiscal year 2012 budget request, NOAA requested funding to begin development for two additional satellites in the GOES-R series—GOES-T and GOES-U. The program estimates that the development for all four satellites in the GOES-R series—GOES-R, GOES-S, GOES-T, and GOES-U—is to cost $10.9 billion through 2036, an increase of $3.2 billion over its prior life cycle cost estimate of $7.67 billion for the two-satellite program. See table 3 for an overview of key changes to the GOES-R program. Table 3: Key Changes to the GOES-R Program August 2006 (baseline program) September 2006 November 2007 February 2011 Number of 4 2 2 4 satellites Instruments or • Advanced Baseline Imager • Cancelled Hyperspectral No change No change instrument • Geostationary Lightning Mapper Environmental Suite changes • Decoupled Solar Imaging • Magnetometer Suite to the Solar Ultraviolet • Space Environmental In-Situ Suite Imager and Extreme • Solar Imaging Suite (which included the Ultraviolet/X-Ray Irradiance Solar Ultraviolet Imager, and Extreme Sensor Ultraviolet/X-Ray Irradiance Sensor) • Hyperspectral Environmental Suite Number of 81 68 34 baseline 34 baseline satellite 31 optional 31 optional products Life cycle cost $6.2 billion—$11.4 billion (through 2034) $7 billion (through 2028) $7.67 billion $10.9 billion estimate (in (through 2028) (through 2036)a then- year dollars) Source: GAO analysis of NOAA data. a Based on NOAA’s fiscal year 2012 budget estimate, $7.64 billion of this cost estimate was for the first two satellites in the series, GOES-R and GOES-S. The cost for the remaining two satellites— GOES-T and GOES-U—was estimated at $3.22 billion. Page 8 GAO-12-576 Geostationary Weather Satellites Acquisition Strategy NOAA’s original acquisition strategy was to award contracts for concept development of the GOES-R system to several vendors who would subsequently compete to be the single prime contractor responsible for overall system development and production. In keeping with this strategy, NOAA awarded contracts for concept development of the overall GOES- R system to three vendors in October 2005. However, in March 2007, NOAA revised its acquisition strategy for the development contract. In response to recommendations by independent advisors, the agency decided to separate the overall system development and production contract into two separate contracts—the flight system and ground system contracts. The flight system includes contracts for the development of the five key instruments and spacecraft, while the ground system includes contracts for the development of key systems needed for the on-orbit operation of the satellites, receipt and processing of information, and distribution of satellite data products to users. In addition, to reduce the risks associated with developing technically advanced instruments, the GOES-R program awarded contracts for concept development for five of the planned instruments. It subsequently awarded separate development contracts for five instruments and, upon completion and approval, these instruments will be provided to the prime contractor responsible for the spacecraft of the GOES-R program. NASA will then work with the spacecraft contractor to integrate and test these instruments. The sixth instrument, the magnetometer, is to be developed as part of the spacecraft contract. Page 9 GAO-12-576 Geostationary Weather Satellites Program Office Structure NOAA is responsible for GOES-R program funding and overall mission success. The NOAA Program Management Council, which is chaired by NOAA’s Deputy Undersecretary, is the oversight body for the GOES-R program. However, since it relies on NASA’s acquisition experience and technical expertise to help ensure the success of its programs, NOAA implemented an integrated program management structure with NASA for the GOES-R program (see fig. 3). NOAA also located the program office at NASA’s Goddard Space Flight Center. Within the program office, there are two project offices that manage key components of the GOES-R system. NOAA has entered into an agreement with NASA to manage the Flight Project Office, including awarding and managing the spacecraft contract and delivering flight-ready instruments to the spacecraft. The Ground Project Office, managed by NOAA, oversees the Core Ground System contract and satellite data product development and distribution. Figure 3: GOES-R Program Office Reporting, Structure, and Staffing Page 10 GAO-12-576 Geostationary Weather Satellites Prior Reports Noted We have previously reported on the impact of program milestone delays Program Milestone Delays, to GOES-R planned launch dates. In April 2009, we reported that the Inadequate Continuity program delayed the planned launch of its first satellite from December 2014 to April 2015. 4 Program officials attributed these delays to providing Plans, and Insufficient more stringent oversight before releasing the request for proposals for the Communication with spacecraft and ground system, additional time needed to evaluate the GOES Data Users contract proposals, and funding reductions in fiscal year 2008. In September 2010, we reported that NOAA later approved a 6-month delay in the planned launch of the first satellite, from April 2015 to October 2015, because work did not begin on the spacecraft until August 2009 due to a bid protest and NASA’s re-evaluation of proposals in response to the protest. 5 In September 2010, we found that as a result of delays to planned launch dates for the first two satellites in the GOES-R series, NOAA might not be able to meet its policy of having a backup satellite in orbit at all times, which could lead to a gap in satellite coverage if an existing satellite failed prematurely. 6 Further, even though there could be a gap in backup coverage, NOAA had not established adequate continuity plans to deal with potential coverage gaps for its geostationary satellites. Specifically, while NOAA had established a policy to always have a backup satellite available, it did not have plans that included processes, procedures, and resources needed to transition to a single or international satellite. We recommended that NOAA develop and document plans for the operation of geostationary satellites that included the implementation procedures, resources, staff roles, and time tables needed to transition to a single satellite, an international satellite, or other solution. NOAA has since developed a continuity plan that generally includes the key elements we recommended. As a result, NOAA has improved its ability to fully meet its mission-essential function of providing continuous satellite imagery in support of weather forecasting. In addition, we found that while NOAA had identified GOES data users and involved internal users in developing and prioritizing the GOES-R 4 GAO-09-323. 5 GAO, Geostationary Operational Environmental Satellites: Improvements Needed in Continuity Planning and Involvement of Key Users, GAO-10-799 (Washington, D.C., Sept. 1, 2010). See also GAO-09-596T. 6 GAO-10-799. Page 11 GAO-12-576 Geostationary Weather Satellites requirements, it had not adequately involved other federal users that rely on GOES data by documenting their input and communicating major changes to the program that have occurred since 2006, such as the removal of certain satellite data products. We recommended that NOAA establish and implement processes to notify agencies of GOES-R program status and changes. NOAA responded that the GOES-R program would develop a communications plan for external stakeholders and that the GOES-R System Program Director would provide an annual program status briefing to the Office of the Federal Coordinator for Meteorology, which would be responsible for its distribution to agencies that would be users of GOES-R data and products. In February 2012, the GOES-R program developed a communications plan that described how external stakeholders would be notified of GOES-R progress, status changes, and other relevant activities. Also, the GOES-R System Program Director provided annual briefings to the Office of the Federal Coordinator for Meteorology in October 2010 and November 2011. These briefings communicated significant program changes to federal GOES data users, such as completed and planned milestones, the status of development efforts, and the removal of additional products and latency improvements from the ground system contract. As a result, federal users of GOES data have received more information about changes to geostationary satellites that may affect their ability to meet mission needs. While the GOES-R program has made progress in completing its early Progress Made in design and is nearing the end of the design phase for its flight and ground Completing Early system components, it completed key design milestones later than planned. Recent technical problems with the instruments and spacecraft, Design, but as well as a significant modification to the ground project’s development Milestones Were plan, have delayed the completion of key reviews and led to increased Completed Late and complexity for GOES-R’s development. Moreover, several instrument, spacecraft, and ground system problems identified during design, have Development Costs not yet been resolved. In addition to the delays, the technical and Have Increased programmatic challenges experienced by GOES-R’s flight and ground projects have led to increased costs for its development contracts. Despite these problems, program officials report that planned launch dates and cost estimates for the first two satellites have not changed, and that approximately $1.2 billion is currently in reserve to manage future delays and cost growth. However, the program has used approximately 30 percent of its reserves over the last 3 years and significant portions of development remain for major components—including the spacecraft and Page 12 GAO-12-576 Geostationary Weather Satellites Core Ground System. In addition, the program did not change its reserves when it restored two satellites adding approximately $3.2 billion to the program’s baseline. As a result, the program will be challenged in completing its remaining development, particularly the final design and testing of the spacecraft and ground system, within its cost and schedule targets. GOES-R Has Made Two key types of development milestones, which are identified in the Progress on Its Design, but GOES-R December 2007 management control plan, 7 are the preliminary Initial Design Milestones design review (PDR) and a more detailed critical design review (CDR). The PDR is an initial design milestone that assesses the readiness of the Completed Later than program to proceed with detailed design activities and the CDR is Planned intended to demonstrate that the design is complete and appropriate to support proceeding to full-scale software development, as well as fabrication, assembly, integration, and testing. The program planned to complete the flight project’s PDR in April 2010 and CDR in April 2011. It planned to complete the ground project’s PDR in July 2010 and CDR in July 2011. These project-level reviews are to be completed before the program’s comprehensive PDR and CDR can be completed. The program has demonstrated progress toward completing its design. Specifically, the program and its subsequent projects have successfully completed their PDRs, demonstrating that they are ready to proceed with detailed design activities. The program and its projects are also currently progressing towards the final design for the entire GOES-R system, which is expected to be completed at the program’s CDR planned for August 2012. However, the program’s PDR milestones were completed later than their 2007 planned dates. Consequently, its CDR milestones were similarly delayed. Table 4 highlights delays in key program milestones as of April 2012. 7 The program established planned dates for key program and system milestones in its management control plan documented in December 2007. This plan was approved by NOAA’s Assistant Administrator for Satellite and Information Services and formed the basis of flight and ground project plans prior to entering the development phase of the program. Page 13 GAO-12-576 Geostationary Weather Satellites Table 4: Delays in Program Milestones Date completed or Delay from Program milestone Date planned planned December 2007 plan Flight project PDR April 2010 January 2011 9 months Ground project PDR July 2010 December 2011 17 months Program PDR July 2010 February 2012 19 months Flight project CDR April 2011 April 2012 12 months Ground project CDR July 2011 July 2012 12 months Program CDR July 2011 August 2012 13 months Source: GAO analysis of NOAA data. Note: Program officials reported that the 2007 review target dates were revised in March 2010. Based on its revised 2010 plans, the program missed its revised PDR milestone by 8 months and its revised CDR milestone is 5 months later than planned. Going forward, assembly and testing of all flight instruments for the first satellite in the series is to be completed by August 2013. Both the flight and ground projects’ development components are expected to be complete by September 2015. Despite recent delays in program milestones, NOAA still expects to meet an October 2015 launch date for the first satellite in the series by utilizing planned schedule reserves. Flight Project—Instrument The Flight Project Office has made progress by completing the critical Designs Are Complete but design reviews for each of its five main instruments, which is significant Technical Challenges Remain because the instrument designs will be applied to each satellite in the GOES-R series. The Geostationary Lightning Mapper was the most recent instrument to complete its critical design review, which occurred in August 2011—16 months after its planned completion date. 8 Instrument fabrication, assembly, and testing activities are under way. The final instrument scheduled for completion—the Geostationary Lightning Mapper—is expected to be delivered for integration with the spacecraft by August 2013. Although instrument design milestones are complete and assembly and testing activities are under way, each of the instruments and the spacecraft has recently encountered technical challenges. The Flight 8 Completion of the Geostationary Lightning Mapper’s CDR was 16 months later than the planned milestone documented in December 2007 when the program entered into its development phase, and 13 months later than the revised milestone established in March 2010. Page 14 GAO-12-576 Geostationary Weather Satellites Project Office has taken steps to resolve several technical problems through additional engineering support and redesign efforts. However, there are still important technical challenges to be addressed, including signal blurring problems for several of the Advanced Baseline Imager’s infrared channels and Geostationary Lightning Mapper emissions that are exceeding specifications. The project office is monitoring these problems and has plans in place to address them. The current development efforts and recent challenges for components of the flight project are described in table 5. Table 5: Flight Project Development Efforts and Challenges through February 2012 Key components Key efforts Recent challenges Instruments Advanced Baseline • Completed critical design review • Replaced printed wiring boards that were Imager • Testing under way; conducted initial bench testing incompatible with specifications for the first flight model • Working to resolve signal problems in several of • Preparing for a planned pre-environmental reviewa the imager’s infrared channels in March 2012 Space Environmental • Completed certain hardware, including detectors • Corrected failures of its data processing unit In-Situ Suite and select spare parts during testing • Developing software algorithms for the first flight model • Performing environmental testing for three out of five sensors on the instrument suite • Preparing for a planned pre-environmental review in January 2013 Extreme Ultraviolet/X- • Completed individual instruments, which are being • Replaced filters that were damaged during Ray Irradiance Sensor integrated with the instrument suite vibration testing • Preparing for a planned pre-environmental review in June 2012 Solar Ultraviolet • Completed software algorithms for the first flight • Investigated cracks in the camera’s electronic Imager model box power control unit • Completed limited thermal vacuum testing on its • Working to resolve failures in the structural engineering design unit model analysis filter • Completing remaining work on the telescope and sensor electronics prior to instrument-level testing • Preparing for a planned pre-environmental review in September 2012 Geostationary • Completed integration of its engineering design • Rebuilt electronics unit and sensor unit power Lightning Mapper unit boards that failed during electrical systems • Completed software algorithms for the first flight testing model • Working to resolve an image signal problem • Conducting electrical system testing • Working to limit emissions that exceed • Preparing for a planned pre-environmental review requirements in December 2012 Page 15 GAO-12-576 Geostationary Weather Satellites Key components Key efforts Recent challenges Spacecraft • Completed preliminary design review • Achieved sufficient massb margins • Received the Spacecraft Command and Telemetry • Eliminated gaps in the flow down of mission Simulator, which is a simulator to generate assurance requirements to subcontractors spacecraft telemetry and receive ground • Working to obtain adequate fault management commands to be used in interface and design specifications from the contractor development testing • Working to increase end-of-life power margins • Completing component and subsystem critical design reviews in preparation for the system’s critical design in April 2012 Sources: GAO analysis of NOAA and NASA data. a A pre-environmental review ensures that the hardware or software to be tested, facility, personnel, and procedures are in place and ready for testing. b Mass is a measure of the amount of matter in a physical body that does not depend on location or gravitational pull. Ground Project—Incomplete The Ground Project Office has made progress in completing the ground Requirements Definition and system’s preliminary design. However, in doing so it experienced Schedule Integration Have Led problems in defining ground system software requirements and identified to Revised Development Plan problems with its dependencies on flight project schedules. Specifically, the project office discovered in early 2011 that software design requirements had not progressed enough to conduct the ground system’s preliminary design review. In addition, the ground system’s development schedule included software deliveries from flight project instruments that were not properly integrated—they had not yet been defined or could not be met. To address these problems, the Ground Project Office made significant revisions to the Core Ground System’s baseline development plan and schedule. In order to avoid potential slippages to GOES-R’s launch date, project officials decided to switch from plans to deliver software capabilities at major software releases to an approach where software capabilities could be delivered incrementally (prior to major releases) as the project received data inputs from the flight project. According to NOAA and program officials, the revised development approach is to provide flexibility in the ground system’s development schedule and reduce risk associated with the original waterfall schedule. However, the revised plan is expected to cost $85 million more than the original plan through the Core Ground System’s completion. This cost includes increased contractor and government staff, new oversight tools, and more verification and testing activities associated with an increased number of software deliveries. Program officials acknowledged that the revised plan intends to expend additional resources to reduce schedule risk and potential impacts to GOES-R’s launch date, but that it also introduces Page 16 GAO-12-576 Geostationary Weather Satellites new cost and schedule risks associated with incremental development, such as more software development and verification activities that require additional government oversight and continuous monitoring. In addition, the program has cancelled previously exercised options to the ground project that were once considered part of its original baseline. 9 In early 2011, the program determined that it could no longer fund Core Ground System contract options—which were estimated to cost approximately $50 million. Program officials stated that they cancelled the contract options due to approaching development commitments, including revisions to the ground system’s revised plan and schedule, and funding reductions from fiscal year 2011. According to program officials, the work to be performed under the cancelled contract options could be addressed by NOAA after GOES-R satellites are launched; however, there are currently no plans in place to do so. Table 6 describes the development efforts and recent challenges for ground project components. Table 6: Ground Project Development Efforts and Challenges through February 2012 Key components Key efforts Recent challenges Core Ground System • Completed preliminary design review • Working to avoid delays in the availability of a • Executing a revised development plan and primary operations facility required for testing schedule in preparation for critical design • Working to fully define software requirements, review in April 2012 which are largely complete and resolve prior problems, and integrate schedules with flight project instruments GOES-R Access • Completed critical design reviews for systems No recent major challenges Subsystem that will enable ground system product processing and distribution • Completed trade studies for communication services to be provided to local and federal emergency managers Antennas • Completed critical design review No recent major challenges • Conducting site preparation and construction activities for multiple antenna sites Source: GAO analysis of NOAA data. 9 These options included 31 additional products—such as low cloud and fog, sulfur dioxide detection, and cloud liquid water—as well as enhancements to the time to deliver the remaining products and how often these products are updated with new satellite data. Page 17 GAO-12-576 Geostationary Weather Satellites GOES-R Development The GOES-R program’s estimated costs include, among other things, Costs Are Rising actual and estimated contract costs associated with design, development, integration, and testing activities for the instruments, spacecraft, and ground system as well as procurement of the satellites’ launch vehicles. The estimated costs also include government costs such as NOAA and NASA program management support and contingency reserves to be applied towards critical risks and issues as they arise. As of January 2010, the program reported estimates of $3.3 billion for the flight project (including reserves of $598 million), $1.7 billion for the ground project (including reserves of $431 million), $2.0 billion for other program costs (including reserves of $617 million), and $748 million for operations and maintenance. Although NOAA has not changed its program cost estimates for the development of GOES-R and GOES-S, contract costs for the instruments, spacecraft, and ground system are rising. Specifically, contractor estimated costs for flight and ground project components grew by $757 million, or 32 percent, between January 2010 and January 2012. Table 7 identifies growth in estimated contract costs for major program components. Table 7: Growth in Estimated Contract Cost for Major Program Components Contractor Contractor Contractor Original Percent estimate at estimate at estimate at 2-year 2-year contract complete completion completion completion change change Major components award date (Nov 2011) (Jan 2010)a (Jan 2011)a (Jan 2012)a ($M) (%) Advanced Baseline Imager Sep 2004 83% $524M $581M $672M +$148M +28% Space Environmental In-Situ Aug 2006 54% 69 81 97 +28 +41% Suite Extreme Ultraviolet/X-Ray Aug 2007 58% 72 81 81 +9 +13% Irradiance Sensor Solar Ultraviolet Imager Sep 2007 62% 139 168 182 +43 +31% Geostationary Lightning Mapper Dec 2007 57% 157 209 252 +95 +61% Spacecraft Dec 2008 32% 711 743 862 +151 +21% Core Ground System May 2009 29% 704 792 976 +272 +39% c Antennas July 2010 37% Not 119 130 +11 +9% b applicable Totals 2,376 2,774 3,252 +757c +32%c Sources: GAO analysis of NOAA and contractor-reported data. a Contractor-reported most likely estimate at completion. b The antenna contract was not awarded until July 2010. c Total 2-year change includes the 1-year change in antenna contract costs. Page 18 GAO-12-576 Geostationary Weather Satellites Not only have development contracts experienced rising costs since January 2010, but they have also experienced larger cost increases more recently. For example, between January 2011 and January 2012, contractors’ estimated costs increased by $184 million for the Core Ground System, compared with $88 million for the period between January 2010 and January 2011. Also, between January 2011 and January 2012, contractors’ estimated costs for the spacecraft and the Advanced Baseline Imager increased by $119 million and $91 million compared with $32 million and $57 million, respectively, between January 2010 and January 2011. Figure 4 depicts recent growth in estimated costs for these development components. Figure 4: Recent Growth in Estimated Cost for Selected Development Components Source: GAO analysis of contractors’ reported cost estimates. The recent growth in contract costs is due in part to the additional labor and engineering support needed to address technical and programmatic problems experienced by flight and ground project components, including the technical complexity associated with development of the Advanced Baseline Imager and the spacecraft, and additional costs associated with the Core Ground System’s revised development plan. NOAA stated that some of the cost growth is attributed to scope changes, including Page 19 GAO-12-576 Geostationary Weather Satellites instrument options that were exercised in 2010 and 2011. Based on our analysis, approximately $60 million of the $757 million growth in contractors’ estimated costs at completion from January 2010 through November 2011 was due to scope changes associated with new instrument flight model development. Given the recent increases in contract costs, the program plans to determine how to cover these increased costs by reducing resources applied to other areas of program development and support, delaying scheduled work, or absorbing additional life cycle costs. Depleted Reserves Could A contingency reserve (also called management reserve) is important Impact Remaining because it provides program managers ready access to funding in order Development to resolve problems as they occur and may be necessary to cover increased costs resulting from unexpected design complexity, incomplete requirements, or other uncertainties. 10 NOAA requires the program office and flight and ground projects to maintain a reserve of funds until their development is completed. Specifically, the flight project is to maintain 20 percent of planned remaining development costs as reserve, the ground project is to maintain 30 percent of planned remaining development costs as reserve, and the program office is to maintain 10 percent of planned remaining development costs as reserve. 11 The program has allocated a proportion of its budget as reserves to mitigate risks and manage problems as they surface during development. As a result of changes in budget reserve allocations and reserve commitments, 12 the program’s reserves have declined in recent years. Between January 2009 and January 2012, the program reported that its reserves fell from 42 percent of remaining development costs to 29 percent. Over the same period, the program reports that after accounting for changes in reserve budgets and reserve commitments, reserves fell from $1.7 billion to $1.2 billion, an approximately 30 percent reduction in its 10 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C., Mar. 2009). 11 Reserve levels are to be based on all planned remaining development and do not include operations and maintenance costs. 12 NOAA uses the terms liens and encumbrances to represent potential risks or issues for which the program is to address with reserves. We refer to these as reserve commitments. Page 20 GAO-12-576 Geostationary Weather Satellites uncommitted reserves. This is important to note since about two-thirds of the development remains for the program’s two most expensive components—the spacecraft and the Core Ground System. Recent utilization of the program’s reserves included addressing unanticipated problems with instrument and spacecraft design, using more than expected labor required to accomplish program and project-related milestones, and acquiring additional resources required to execute the revised development plan and schedule for the Core Ground System. In this regard, the program’s independent review board recently raised questions about the sufficiency of the program’s near-term remaining reserves, and program officials decided to establish contractual funding caps for the spacecraft, three instruments—the Advanced Baseline Imager, the Solar Ultraviolet Imager, and the Geostationary Lightning Mapper—and the Core Ground System for fiscal year 2012, delaying work into future years. Figure 5 depicts changes in program reserves over time. Figure 5: Changes in Program Reserves Over Time At completion of the GOES-R program’s preliminary design review in February 2012, the program reported that it was within its thresholds because it had maintained 20 percent of planned remaining development costs as reserve for the flight project, 32 percent of planned remaining Page 21 GAO-12-576 Geostationary Weather Satellites development costs for the ground project, and 10 percent of planned remaining development costs for the program office. The program will soon enter the integration and test phase, when projects typically experience cost and schedule growth and need additional funding—as identified in our prior reviews of NASA acquisitions. 13 The program may need to draw from its available remaining reserves to address a number of situations, including • unanticipated problems during completion of the critical design reviews of the spacecraft and the ground system; • unanticipated problems during the integration and test of the instruments and spacecraft, such as required redesign; • potential additional labor required for ground system software development and testing; • potential delays in the readiness of NOAA’s Satellite Operations Facility for ground system testing, resulting in the use of contractor facilities; and • higher than expected costs for launch vehicles. While the program reported that reserves were within accepted levels as of February 2012, the reserves may not be matched to remaining development. Although the program restored two satellites to its budget baseline in February 2011, thereby adding approximately $3.2 billion to its total budget, 14 it did not correspondingly change its program reserves. The program did not report its rationale for maintaining reserves at the two-satellite program or explain how these planned reserves were intended to cover risks associated with the development of all four satellites. As a result, there is limited assurance that the reserves are appropriate for each satellite’s remaining development. 13 See, for example, GAO, NASA: Assessments of Selected Large-Scale Projects, GAO-11-239SP (Washington, D.C., March 3, 2011) and NASA: Assessments of Selected Large-Scale Projects, GAO-12-207SP, (Washington, D.C., March 1, 2012). 14 The restoration of GOES-T and GOES-U to the program resulted in an increase of $3.2 billion to the program’s cost estimate. Page 22 GAO-12-576 Geostationary Weather Satellites Whether the program will continue to stay within its budget depends in part on whether officials have a full understanding of the reserves required for remaining development. Given the program’s recent use of reserves and the significant portions of development remaining for major components, a complete understanding and proper management of reserve levels will be critical to successfully completing all program components. Unless NOAA assesses the reserve allocations across all of the program’s development efforts, it may not be able to ensure that its reserves will cover ongoing challenges as well as unexpected problems for the remaining development of all four satellites in the series. The success in management of a large-scale program depends in part on GOES-R Schedules having an integrated and reliable schedule that defines, among other Are Not Fully things, when work activities and milestone events will occur, how long they will take, and how they are related to one another. Without such a Reliable, Contributing schedule, program milestones may slip. While the GOES-R program has to Milestone and adopted certain scheduling best practices at both the programwide and Potential Launch Date contractor levels, unresolved weaknesses also exist, some of which have contributed to current program milestone delays and a replanning of the Delays Core Ground System’s schedule. Without a proper understanding of current program status that a reliable schedule provides, managing the risks of the GOES-R program becomes more difficult and may result in potential delays in GOES-R’s launch date. GOES-R Integrated Master Program schedules not only provide a road map for systematic program Schedule and Some execution, but also provide the means by which to gauge progress, Subordinate Schedules Are identify and address potential problems, and promote accountability. Accordingly, a schedule helps ensure that all stakeholders understand Unreliable both the dates for major milestones and the activities that drive the schedule. If changes occur within a program, the schedule helps decision makers analyze how those changes affect the program. The reliability of the schedule will determine the credibility of the program’s forecasted dates, which are used for decision making. Our work has identified nine best practices 15 associated with developing and maintaining a reliable schedule. These are (1) capturing all activities, (2) sequencing all activities, (3) assigning resources to all activities, (4) establishing the 15 These practices are based on GAO-09-3SP. Page 23 GAO-12-576 Geostationary Weather Satellites duration of all activities, (5) integrating schedule activities horizontally and vertically, (6) establishing the critical path for all activities, (7) identifying reasonable “float” between activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations. See table 8 for a description of each of these best practices. Table 8: Description of Scheduling Best Practices Practice Description Capturing all activities The schedule should reflect all activities (steps, events, outcomes, etc.) as defined in the program’s work breakdown structure to include activities to be performed by both the government and its contractors. Sequencing all activities The schedule should sequence activities in the order that they are to be implemented. In particular, activities that must finish prior to the start of other activities (i.e., predecessor activities), as well as activities that cannot begin until other activities are completed (i.e., successor activities) should be identified. Assigning resources to all activities The schedule should reflect who will do the work activities, whether all required resources will be available when they are needed, and whether any funding or time constraints exist. Establishing the duration of all activities The schedule should reflect the duration of each activity. These durations should be as short as possible and have specific start and end dates. Integrating schedule activities horizontally The schedule should be horizontally integrated, meaning that it should link the products and vertically and outcomes associated with sequenced activities. The schedule should also be vertically integrated, meaning that traceability exists among varying levels of activities and supporting tasks and subtasks. Establishing the critical path for all The critical path represents the chain of dependent activities with the longest total activities duration in the schedule. Identifying reasonable float between The schedule should identify a reasonable amount of float—the time that an activity can activities slip before the delay affects the finish milestone—so that schedule flexibility can be determined. As a general rule, activities along the critical path typically have the least amount of float. Conducting a schedule risk analysis A schedule risk analysis is used to predict the level of confidence in the schedule, determine the amount of time contingency needed, and identify high-priority schedule risks. Updating the schedule using logic and The schedule should use logic and durations in order to reflect realistic start and durations to determine the dates completion dates, be continually monitored to determine differences between forecasted completion dates and planned dates, and avoid logic overrides and artificial constraint dates. Source: GAO. The first seven practices are essential elements for creation of an integrated schedule. An integrated schedule that contains all the detailed tasks necessary to ensure program execution is called an integrated master schedule (IMS). The reliability of an integrated schedule depends in part on the reliability of its subordinate schedules. Automated integration of all activities into a single master schedule can prevent Page 24 GAO-12-576 Geostationary Weather Satellites inadvertent errors when entering data or transferring them from one file to another. GOES-R has an IMS that is created manually once a month directly from at least nine subordinate contractor schedules. 16 Due to anticipated limitations related to the number of activities that would need to be included, program officials stated that they did not intend to make the IMS a live, integrated file. We believe that the lack of a dynamic IMS is an acceptable schedule limitation for an organization of GOES-R’s size and complexity. Program officials also stated that they are in the process of creating an automated process for updating their IMS from contractor- delivered files sometime in 2012, but this was not available in time for our analysis. To assess the reliability of the programwide IMS, we analyzed four subordinate contractor schedules from July 2011 17 and found several best practices that were met in multiple schedules. For example, similar to the program-level IMS, three of the contractor IMS’s include all activities that were supplemented with monthly data received directly from the schedules of their subcontractors, and the fourth—the Core Ground System schedule—included milestone activities for all subcontractors. All four schedules also substantially or fully met the best practice for regularly updating the schedule, including regular reporting of status and keeping the number of activities completed out of sequence to a minimum. Finally, three of the four schedules substantially met the best practice for establishing accurate activity durations. However, we also found weaknesses in each of the subordinate schedules when compared to the best practices and, when viewed in conjunction with manual program-level updates, we concluded that the program-level schedule may not be fully reliable. For example, each of the contractor schedules was either not resource loaded or had significant overloading of resources. In addition, none of the contractors were able to 16 The subordinate schedules used in creating the IMS each contain detailed activities for discrete segments of the GOES-R program, such as instruments, which are assigned to a specific contractor. We did not analyze the programwide IMS itself due to the limitations inherent in manual creation of this schedule. However, conclusions drawn from analysis of contractors’ schedules that feed directly into the programwide IMS can therefore be applied to the program’s IMS as well. 17 Our rationale for the selection and analysis of schedules is discussed in app. I. Page 25 GAO-12-576 Geostationary Weather Satellites provide information on schedule risk analyses conducted with risk simulations. A full set of analysis results is listed in table 9. Selected strengths and weaknesses for each of the four schedules follow the table. Table 9: Practices Utilized in Selected GOES-R Schedules Geostationary Advanced Core Ground Lightning Baseline Imager Spacecraft System Scheduling best practice Mapper schedule schedule schedule schedule Best Practice 1: Capturing all activities ● ◕ ◕ ◑ Best Practice 2: Sequencing all activities ◕ ◑ ◑ ◑ Best Practice 3: Assigning resources to all activities ◔ ◕ ◔ ◑ Best Practice 4: Establishing the duration of all activities ◕ ◑ ◕ ◕ Best Practice 5: Integrating schedule activities horizontally and vertically ◑ ◑ ◕ ◑ Best Practice 6: Establishing the critical path for all activities ◕ ◑ ◕ ◔ Best Practice 7: Identifying float on activities and paths ◕ ◑ ◑ ◔ Best Practice 8: Conducting a schedule risk analysis ◔ ◔ ◔ ◔ Best Practice 9: Updating the schedule using logic and durations to determine the dates ● ● ● ◕ Source: GAO analysis of schedules provided by GOES-R, documents and information received from GOES-R officials. Key ● = The agency/contractor has fully met the criteria for this best practice ◕ = The agency/contractor has substantially met the criteria for this best practice ◑ = The agency/contractor has partially met the criteria for this best practice ◔ = The agency/contractor has minimally met the criteria for this best practice ○ = The agency/contractor has not met the criteria for this best practice Of the four component schedules, the Geostationary Lightning Mapper schedule demonstrated the most comprehensive implementation of best practices. For instance, all activities and durations were appropriately captured in the schedule; logic links were included for over 99 percent of activities; only one small gap was present in the critical path from the date of the schedule through the end of the project; a high percentage of activities had appropriate logic; and the schedule was updated using logic and durations to determine dates. However, more than 10 percent of the resources listed in the schedule were overloaded, meaning that the schedule required more resources than were available. Also, changes in Page 26 GAO-12-576 Geostationary Weather Satellites activity durations for one flight model did not result in a corresponding change in the shipment date of that flight model. The Advanced Baseline Imager schedule substantially or fully met three of the best practices. As mentioned above, the Advanced Baseline Imager schedule met the best practices for including all subcontractor records as well as regular updating and reporting on the schedule. In addition, it provided a substantial amount of information regarding the resources in its schedule; the contractor created a series of over 900 named codes that denote detailed information such as resource type, location, and labor rate. However, 18 percent of remaining activities had incomplete or missing logic, and 43 percent had soft date constraints. If the schedule is missing logic links between activities, float 18 estimates will not be accurate. Incorrect total float values may in turn result in an invalid critical path and an inaccurate assessment of project completion dates. In the case of the Advanced Baseline Imager, many detailed activities had large total float values throughout. Moreover, the Advanced Baseline Imager schedule did not have a valid critical path for one of its three flight models and did not have a critical path that spans the entire program. The spacecraft schedule substantially met more than half of the best practices. It had a valid critical path through the launch dates for the first two satellites, included and mapped all activities appropriately, included logic links for over 99 percent of its activities, and was appropriately updated. However, more than 10 percent of its activities had constraints, and nearly 10 percent of its activities had gaps between consecutive activities, also known as lags. Lags should be minimal, and should not be used in place of activities, as they cannot be easily monitored, cannot be included in a risk assessment, and do not take resources. For this reason, lags should be eliminated and replaced with activities so they can be tracked. Also, the spacecraft schedule’s critical path included a year-long current activity for which a detailed breakdown was not available, even though the activity is in the current detailed planning period. The Core Ground System schedule contained weaknesses across seven of the nine best practice areas resulting in a score of partially or minimally met. For instance, a valid critical path could not be traced between the 18 Total float is the amount of time that an activity can slip before the delay affects the finish milestone. Page 27 GAO-12-576 Geostationary Weather Satellites schedule’s latest status date and the launch date for either of the first two satellites; 12 percent of remaining activities had incomplete or missing logic and 13 percent had soft date constraints; and more than 75 percent of activities had more than 100 days of total float. Also, not all subcontractor detail activities were included in the schedule. Without accounting for all necessary activities, it is uncertain whether all activities are scheduled in the correct order, missing activities would appear on the critical path, or a schedule risk analysis accounts for all risk. Officials for all four contract teams suggested that certain schedule weaknesses were unique to the July 2011 schedules they provided and that the weaknesses would be remedied in December 2011 schedules. Our subsequent analysis did find improvements for each of the four contractors. For example, the Advanced Baseline Imager’s December schedule had approximately 10 percent fewer activities missing predecessors and successors than in July. The Core Ground System schedule also performed better in three of the nine best practices, including the presence of a full set of duration and total float information, and better information on handoffs with external parties. However, many weaknesses from the July schedules remained in the December schedules. For example, officials stated that several schedule risk analyses had been conducted for the Core Ground System schedule, but also reported that the schedule risk analysis conducted by the Core Ground System’s subcontractor was not valid and did not provide accurate or constructive information. Of particular importance is the absence of a valid critical path throughout all the schedules. Establishing a valid program-level critical path depends on the resolution of issues with the respective critical paths for the spacecraft and Core Ground System components. Contract specifications for all four contractors require that these schedules define critical paths for their activities. Without a valid critical path, management cannot determine which delayed tasks will have detrimental effects on the project finish date. It also cannot determine if total float within the schedule can be used to mitigate critical tasks by reallocating resources from tasks that can be delayed without launch date impact. Unless the weaknesses in these subordinate schedules are addressed, including the generation of valid critical paths in all schedules, the programwide IMS that is derived from them may not be sufficiently reliable. Page 28 GAO-12-576 Geostationary Weather Satellites Schedule Practice Weaknesses in implementing scheduling best practices undermine the Weaknesses Could Lead to program’s ability to produce credible dates for planned milestones and Further Schedule Delays events, as illustrated by schedule discrepancies that occurred between the ground project and the flight project and the subsequent replanning that was required. The program has already demonstrated a pattern of milestone delays during its development due in part to the scheduling weaknesses we identified. Although the program has initiated two key efforts that could address certain schedule weaknesses, other weaknesses have not yet been resolved. Until the program’s scheduling weaknesses are corrected, it may experience additional delays to its key milestones. The program has revised planned milestone and completion dates for each of the instruments as well as the spacecraft and ground system components by at least 3 months—and up to 2 years—since the program originally estimated dates for key milestones in its December 2007 management control plan. Program officials noted that its December 2007 dates were notional estimates until integrated baseline reviews could be conducted. However, delays occurred both before and after the schedules for the instruments, spacecraft, and ground system components were formalized. In certain cases, more recent changes were due to delays in building and testing satellite components. For example, the Solar Ultraviolet Imager experienced delays in 2011 and 2012 due in part to delays in software development and in procuring flight model parts. Also, a failure in testing power supply boards caused rework delays in 2011 and 2012 on the Geostationary Lightning Mapper instrument. See figure 6 for a summary of changes in planned completion dates for components of the flight and ground projects. Page 29 GAO-12-576 Geostationary Weather Satellites Figure 6: Changes in Planned Completion Dates for Key Milestones of the First GOES-R Satellite a This chart shows estimated timing of GOES-R milestones based on NOAA’s initial 2007 estimate and monthly program status reports from 2010 and 2012. Antenna and the GOES-R Access Subsystem dates were not listed in the 2007 estimates. The potential for delays remains as GOES-R’s instruments, spacecraft, and ground system components complete their design and testing phases. According to program officials, the Geostationary Lightning Mapper shipment date remains at risk of a potential slip due to redesign efforts that have impacted the release of the build of the electronic board component of the instrument. The current projected delivery for this instrument is August 2013, leaving only 1 month before it is on the critical path for GOES-R’s launch readiness date. Moreover, weaknesses in implementing schedule best practices make meaningful measurement and oversight of program status and progress, as well as accountability for results, difficult to achieve—which can in turn reduce the timeliness and effectiveness of the understanding and mitigation of project risks. The program office has taken specific positive actions that address two of the scheduling weaknesses we identified. First, the GOES-R program implemented the Giver-Receiver Intersegment Database, a tool that tracks deliverables between the flight and ground projects. Giver-Receiver Intersegment Database items are formally reviewed by various working groups weekly and monthly before they are incorporated into the GOES-R IMS. This initiative is intended to address a program-recognized need for Page 30 GAO-12-576 Geostationary Weather Satellites better horizontal integration (related to best practice 5). Second, the GOES- R program implemented a Joint Cost and Schedule Confidence Level, a set of parametric models designed to identify the probability that a given program’s schedule values will be equal to or less than target values on a specific date. 19 In the Joint Cost and Schedule Confidence Level, simulations are run for the expected duration of activities based on probabilities supplied by officials in the project’s cost and schedule division. This initiative is intended to address a program-recognized need to conduct a schedule risk analysis (related to best practice 8). However, the possibility of future milestone delays remains. Initial results from the Joint Cost and Schedule Confidence Level from January 2012 indicated that there is a 48 percent confidence level that the program will meet its current launch readiness date of October 2015. Program officials plan to consult with the NOAA Program Management Council to determine the advisability of moving the launch readiness date to a 70 percent confidence level of February 2016. Given that scheduling weaknesses remain unaddressed, even these confidence levels may be unreliable. Establishing accurate confidence estimates depends on reliable data that result from the implementation of a full set of scheduling best practices. Furthermore, delays in GOES-R’s launch date could impact current operational GOES continuity and could produce milestone delays for subsequent satellites in the series. Program documentation indicates that with the current launch readiness date of October 2015, plus an on-orbit testing period, there is a 37 percent chance of a gap in the availability of two operational GOES-series satellites at any one time, assuming a normal lifespan for the satellites currently on-orbit. Any delays in the launch readiness date for GOES-R, which is already at risk due to the increased cost growth and recent heavy use of program reserves discussed previously, would further increase the probability of a gap in satellite continuity. This could result in the need for NOAA to rely on older satellites that are not fully functional. In addition, GOES-R’s schedule reserve is being counted on to complete activities for GOES-S. As a 19 The Joint Cost and Schedule Confidence Level is a probabilistic analysis that includes, among other things, all cost and schedule elements, incorporates and quantifies potential risks, assesses the impacts of cost and schedule to date, and addresses available annual resources to arrive at development cost and schedule estimates associated with various confidence levels. Page 31 GAO-12-576 Geostationary Weather Satellites result, delays to certain program schedule targets could impact milestone commitments for GOES-S. Until the program implements a full set of schedule best practices, and uses it on succeeding schedule updates throughout the life of the program, further delays in the program’s launch dates may occur. In particular, without ensuring that all contractor and subcontractor information is included in the IMS and conducting regular schedule risk assessments, program management may not have timely and relevant information at its disposal for decision making. Lack of the proper understanding of current program status due to schedules that are not fully reliable undercuts the ability of the program office to manage a high- risk program like GOES-R. Risk management is a continuous process to identify potential problems Risk Management before they occur. By identifying potential problems early, activities can Approach Is Well be planned and invoked as needed across the life of a project to avoid or mitigate the adverse impacts of program problems. Effective risk Defined but Not Fully management involves early and aggressive risk identification through the Implemented collaboration and involvement of relevant stakeholders. Government and industry risk management guidance divides risk management activities into four key areas—preparing for risk management, identifying and analyzing risks, mitigating risks, and reviewing risks with executive oversight. Table 10 describes recognized best practices in these areas. Page 32 GAO-12-576 Geostationary Weather Satellites Table 10: Description and Examples of Recognized Risk Management Best Practices Risk practice Description Risk preparation Risk preparation involves establishing and maintaining a strategy for identifying, analyzing, and mitigating risks. The risk management strategy addresses the specific actions and management approach used to apply and control the risk management program. It also includes identifying and involving relevant stakeholders in the risk management process. Risk identification and analysis Risks must be identified and described in an understandable way before they can be analyzed and managed properly. This includes identifying risks from both internal and external sources and evaluating each risk to determine its likelihood and consequences. Analyzing risks includes risk evaluation, categorization, and prioritization, and is used to determine when appropriate management attention is required. Risk mitigation Risk mitigation involves developing alternative courses of action, workarounds, and fallback positions, with a recommended course of action for the most important risks to the project. Mitigation includes techniques and methods used to avoid, reduce, and control the probability of occurrence of the risk; the extent of damage incurred should the risk occur; or both. Examples of activities for mitigating risks include documented handling options for each identified risk; risk mitigation plans; contingency plans; a list of persons responsible for tracking and addressing each risk; and updated assessments of risk likelihood, consequence, and thresholds. Executive risk oversight Reviews of the project teams’ risk management activities, status, and results should be held on a periodic and event-driven basis. The reviews should include appropriate levels of management, such as key NOAA and NASA executives, who can provide visibility into the potential for project risk exposure and appropriate corrective actions. Source: GAO analysis based on the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition (Version 1.3), the International Organization for Standardization, Defense Acquisition University, and GAO guidance on risk management practices. GOES-R Has Defined a NOAA has established policies and procedures for effective risk Comprehensive Risk management for GOES-R. For example, the program has documented a Management Process strategy for managing risks that includes important elements, such as relevant stakeholders and their responsibilities and the criteria for evaluating, categorizing, and prioritizing risks. The program’s approved risk management plan also includes requirements for risk mitigation— such as required actions, deadlines, and assigned risk owners—as well as requirements that risks’ status and changes are periodically reviewed by appropriate managers, including senior NOAA and NASA managers. Table 11 compares GOES-R’s risk management policies and procedures with recognized risk management practices for four areas. Page 33 GAO-12-576 Geostationary Weather Satellites Table 11: Comparison of GOES-R Risk Management Policies and Procedures to Recognized Practices GOES-R policies Risk practice and procedures Rationale Risk preparation Fully meets criteria The program has established and maintains its strategy in an approved risk management ● plan. The plan defines important aspects of the program’s risk process, including relevant stakeholders and designated responsibilities for risk management. It also defines other important aspects of the program’s risk process, relevant sources of risks, and the criteria to be used in risk evaluation, categorization, and prioritization. The approved plan requires the use of a risk database application as the official risk repository and formal training on the program’s risk process as needed. Risk identification and Fully meets criteria The program’s approved risk plan requires projects to identify internal and external analysis ● program risks, defines the evaluation parameters and categories to be used in evaluating and grouping risks, and requires assessments of the relative priority of its risks. Also, the program’s schedule management plan requires the program to conduct a schedule risk assessment. Risk mitigation Fully meets criteria The program’s approved risk plan requires risks to have documented handling options ● along with the required steps, deadlines, triggers, and assigned risk owners for mitigating the risk. The plan also requires program and project review boards to monitor risk status and make official decisions regarding risk review, acceptance, and closure as well as updates to track the status of mitigation efforts. Executive risk Fully meets criteria The program’s approved risk plan requires appropriate levels of management to review oversight ● the activities, status, and results of the risk management process covering the most critical risks, key risk parameters, and the status of mitigation efforts. It also requires that the program’s risk manager track actions assigned by a risk review board. Source: GAO analysis based on NOAA and NASA data. Key ●= The program has fully met the criteria for this practice ◕= The program has substantially met the criteria for this practice ◑= The program has partially met the criteria for this practice ◔= The program has minimally met the criteria for this practice ○= The program has not met the criteria for this practice With such policies and procedures in place, NOAA has established a comprehensive framework to support its identification, mitigation, and oversight of risks across the program, and laid a foundation for consistent implementation. The Program Has Not While the program has a well-defined risk management process, this Fully Implemented Its Risk process has not been fully implemented. Table 12 identifies the extent to Management Process or which the program has implemented recognized risk management practices through its risk process. Of particular note, the program has not Adequately Mitigated Key conducted and documented adequate or timely evaluations for all Risks potential risks—called candidate risks in the risk management plan—in its risk list. In addition, the program did not always document its risk handling strategies and time frames and did not always provide adequate rationale for its decision to close risks. Page 34 GAO-12-576 Geostationary Weather Satellites Table 12: GOES-R Implementation of Risk Practices GOES-R application of its Risk practice risk process Rationale Risk preparation Substantially The program has established a risk database to communicate risk information and meets criteria provided risk training to the Ground Project Office in October 2011. However, the program has not provided formal risk training to the Flight Project Office. ◕ Program officials stated that they plan to provide formal training to the flight project when its critical design is complete. Until the program completes training for both projects, the program will have less assurance that all staff are consistently applying its risk management process and associated tools for communicating risk assessments and status. Risk identification and Partially meets The program has identified certain internal and external program risks, has generally analysis criteria evaluated open risks based on defined parameters, and has assessed the relative priority of its risks. ◑ However, the program has not provided adequate or timely evaluations for certain candidate risks. Specifically, the program did not assess likelihood or potential impact for 49 out of its 100 open candidate risks as required by the program’s risk plan. Further, 40 candidate risks had not yet been examined by project review boards for more than 3 months, and 15 of these were first identified in 2010. According to the program’s risk manager, candidate risks are generally to be assessed and presented to a project risk board within 1 to 3 months. In addition, the program is not categorizing its risks, thus managers may be limited in their ability to provide timely attention and efficient handling of these risks in context with the source of the risk. Also, the program only recently completed a schedule risk assessment in February 2012, the lack of which may have limited managers’ ability to understand which risks may be most critical to address in maintaining the program’s schedule. Risk mitigation Partially meets The program has generally developed mitigation plans for critical risks to include required criteria steps and deadlines, assigned owners responsibility for managing risks, and monitored the status of risk mitigation efforts through monthly project review board meetings. ◑ However, the program did not always document mitigation options for open and candidate risks and did not always include triggers or thresholds for when fallback actions were to be executed. In addition, the program did not include adequate rationale for its decision to close active risks. Thus, it is not clear whether these risks had been adequately mitigated prior to closure. Executive risk Substantially The program and lower-level project review boards review risk status and results oversight meets criteria monthly. Further, the NOAA Program Management Council reviews the risk status and results for the program’s top risks monthly. ◕ However, the program has not tracked corrective actions from risk review board meetings or meetings with the NOAA Program Management Council. This may limit management’s ability to understand the number or status of outstanding actions and future GOES acquisitions’ ability to learn from the current program’s risk resolution approach. Source: GAO analysis based on NOAA and NASA data. Key ●= The program has fully met the criteria for this practice ◕= The program has substantially met the criteria for this practice ◑= The program has partially met the criteria for this practice ◔= The program has minimally met the criteria for this practice ○= The program has not met the criteria for this practice Page 35 GAO-12-576 Geostationary Weather Satellites The GOES-R Program Office has identified 11 risks that it considers critical (medium- or high-level risks) and could significantly impact the program’s development cost or schedule, documented mitigation approaches for each risk, and tracked mitigation progress. For these and its other risks, the program generally has mitigation plans in place that typically include actions to be taken, deadlines, and assigned responsibilities that could help to minimize or control the occurrence of these critical risks. Table 12 describes the program-identified critical risks, as of February 2012. Table 13: GOES-R Program Critical Risks, as of February 2012 Program- identified risk Program-identified critical risks Program risk description level Program funding stability If the program’s funding profile continues to be changed, then its life cycle cost High will increase. GOES-R development schedule If flight and ground project developments (including their dependencies) are Medium delayed, then development schedule will be impacted through delayed work or rework, which will increase costs and may adversely affect launch readiness. Global positioning system (GPS) If GPS timing or frequency interference occurs, then there will be a degradation Medium timing and frequency interference or loss of GOES-R capabilities with dependencies on GPS resulting in schedule with GOES-R delays and cost increases from rework or redesign. GOES-R user readiness If user or NOAA infrastructure upgrades necessary to ensure compatibility with Medium GOES-R do not align with planned deployment schedules, then there is a possibility that specific users will be unable to use the data products. Interference at the spacecraft with If radiated emissions from instruments or other satellite hardware exceed Medium the GPS receiver specifications, then GOES-R satellite performance may be significantly degraded. Impact of spacecraft changes on If there are significant schedule changes in the final spacecraft and ground Medium launch critical release system-related documents regarding telemetry tracking and control, spacecraft navigation, and product monitoring, then there could be impacts to a major ground software release. Inertial Reference Unit delivery If Inertial Reference Unit development and compliance continues to be delayed, Medium then the mission readiness review may not be successful, resulting in delayed GOES-R hardware delivery and cost and schedule impacts. Geostationary Lightning Mapper If the GLM instrument is delayed beyond planned delivery, then spacecraft Medium (GLM) late delivery integration and test will be impacted. GLM ground processing algorithm If the GLM ground processing algorithm is not available from the contractor in Medium correction for ghosting time, then the instrument’s algorithm will be degraded for post-launch testing. Program budget carryover If funding is not made available in a timely manner before carryover is Medium expended, then there is the potential for additional program cost and schedule delays. Program budget contingency If the flight or ground project experiences future difficulties in maintaining Medium (reserve) agreed-to contingency levels, then additional adjustments or descopes may be required with impacts to program scope or launch readiness dates. Source: GAO analysis of NOAA and NASA data. Page 36 GAO-12-576 Geostationary Weather Satellites While the program has documented mitigation plans in place for most of its critical risks, the program is not mitigating its most critical risk— program funding stability. Although the program has included this risk on its top risk list and presented it to the NOAA Program Management Council, it has not devised options for required replanning or functionality descopes should the program experience reduced funding. Program officials stated that the risk is external and beyond their control. However, given that the program has made trade-off decisions regarding available funding, functionality, and the timing of its work, it is reasonable to expect the program to have plans in place that include possible trade-off decisions based on different outcomes, including triggers for when decisions need to be executed. Further, there is at least one critical risk that is not on the program’s top risk list that could further jeopardize the program’s launch readiness dates and life cycle costs: GOES-S milestones may be affected by delays that have occurred or may still occur during GOES-R development. As discussed earlier in this report, further delays in the development of the first satellite could result in problems for the second satellite’s scheduled milestones since the program is planning to complete the second satellite’s activities during periods of time set aside as reserve to complete the first satellite’s activities. Program officials stated that a risk has recently been added to the flight project’s risk list to reflect the potential delays to the GOES-S development schedule. Until this risk is added to the program’s risk list with a documented mitigation approach and regular monitoring, NOAA could delay the analysis, planning, and actions that would limit the impacts and occurrence of the risk, and would thus be unprepared to face the significant consequences to the program should this risk be realized. While the program has well-defined policies and procedures, it has not fully implemented its risk process. Fully implementing the recognized risk management practices defined in GOES-R policies and procedures would provide program officials with the assurance that all risks—including those that are new and most critical—are adequately addressed. Page 37 GAO-12-576 Geostationary Weather Satellites NOAA has made progress toward achieving GOES-R program goals by Conclusions completing certain preliminary and critical design milestones for its flight and ground projects. However, associated reviews, including the programwide critical design review planned for August 2012, have been delayed by many months. Progress was also accompanied by technical and programmatic challenges, such as initial failures of instrument components and replanning of ground project software releases. Although these two specific challenges are being addressed, others such as instrument signal problems and a 2-year cost growth of more than 30 percent for program development contracts point to troubling patterns that will require ongoing remediation and monitoring. NOAA has allocated budget reserves for such situations according to its established guidelines and reports that project reserve levels are currently within those guidelines. However, the program has used approximately 30 percent of its reserves over the last 3 years and significant portions of development remain for major components—including the spacecraft and Core Ground System. In addition, the program did not change its reserves when it restored two satellites adding approximately $3.2 billion to the program’s baseline. Unless NOAA assesses the reserve allocations across all of the program’s development efforts, it may not be able to ensure that its reserves will cover ongoing challenges as well as unexpected problems for the remaining development of all four satellites in the series. The unreliability of the program’s integrated master schedule and some contractor schedules adds further uncertainty to whether the program will meet its commitments. The issues that exist among these schedules, such as a lack of a full and consistent allocation of resources, incomplete logic, and gaps in the critical path to project completion, are inconsistent with our and agency best practices. NOAA has taken steps to improve schedule reliability through more automated schedule integration, a cross-project deliverable-tracking database, and its first schedule risk assessment. However, unless the program addresses the full set of scheduling weaknesses we identified, its schedules may not provide a fully reliable basis for decision making. NOAA has defined the policies and procedures it needs to effectively manage and mitigate these and other program risks. Nevertheless, officials do not have a current and comprehensive view of program risk because these policies and procedures have not been fully implemented. Most significantly, not all known candidate risks have been evaluated; corrective actions have not been consistently tracked; and risks we identified in this review are not being tracked or adequately mitigated. Until program officials diligently execute all of the program’s defined risk Page 38 GAO-12-576 Geostationary Weather Satellites management practices and integrate these with improvements in the management of reserves and schedules, the program is at risk of exceeding cost and schedule targets and further slipping launch dates for satellites in the GOES-R series. To improve NOAA’s ability to execute GOES-R’s remaining planned Recommendations for development with appropriate reserves, improve the reliability of its Executive Action schedules, and address identified program risks, we recommend that the Secretary of Commerce direct the NOAA Administrator to ensure that the following four actions are taken: • Assess and report to the NOAA Program Management Council the reserves needed for completing remaining development for each satellite in the series. • For all satellites in the GOES-R program, including those for which detailed scheduling has yet to begin, address shortfalls in the schedule management practices we identified, including but not limited to incorporating appropriate schedule logic, eliminating unnecessary constraints, creating a realistic allocation of resources, ensuring an unbroken critical path from the current date to the final satellite launch, and ensuring that all subcontractor activities are incorporated in contractors’ integrated master schedules. • Execute the program’s risk management policies and procedures to provide more timely and adequate evaluations and reviews of newly identified risks, documented handling strategies for all ongoing and newly identified risks in the risk register, time frames for when risk mitigation and fallback plans are to be executed, adequate rationale for decisions to close risks, and documentation and tracking of action items from risk review board meetings or other meetings with senior NOAA and NASA managers through completion. • Given the potential impact to the program, add the risk that GOES-S milestones may be affected by GOES-R development to the program’s critical risk list, and ensure that this risk and the program- identified funding stability risk are adequately monitored and mitigated. Page 39 GAO-12-576 Geostationary Weather Satellites We received written comments on a draft of this report from the Secretary Agency Comments of Commerce, who transmitted NOAA’s comments. The department and Our Evaluation concurred with three of our recommendations and partially concurred with one recommendation. It also provided general comments, which are addressed below, and technical comments, which we have incorporated into our report, as appropriate. A copy of NOAA’s comments is provided in appendix II. NOAA concurred with our first recommendation to assess and report on the reserves needed for completing the remaining development for each satellite in the series to the agency’s Program Management Council. It stated that the GOES-R program would continue to provide status reports on contingency reserves to the Goddard Space Flight Center and the Program Management Council, and would work with NOAA to ensure contingency reporting meets its needs. It also stated in its general comments that contingency reserve for GOES-T and GOES-U (amounting to 20 percent of their projected development costs) was included when the $3.2 billion for these satellites was added to the program budget baseline in February 2011. NOAA did not provide any additional data on its fiscal year 2012 contingency budget allocations by satellite to support this statement and, as we discussed in this report, the program did not report a significant change in overall program reserve levels when it revised the baseline from two satellites to four satellites. NOAA concurred with our second recommendation to address shortfalls that we identified in schedule management practices and stated that the program will continue to bring down the number of errors in the schedules and improve the fidelity of the program’s integrated master schedule. NOAA partially concurred with our third recommendation to fully execute the program’s risk management policies and procedures, to include timely review and disposition of candidate risks. NOAA stated that it did not consider the “concerns” listed in its risk database to be risks or candidate risks, and that the risk management board actively determines whether recorded concerns should be elevated to a risk. However, the program has not treated concerns in accordance with its risk management plan, which considers these to be “candidate risks” and requires their timely review and disposition, as evidenced by the many concerns in the database that were more than 3 months old and had not been assessed or dispositioned. Unless NOAA follows its risk management plan by promptly evaluating “concerns,” it cannot ensure that it is adequately managing the full set of risks that could impact the program. Page 40 GAO-12-576 Geostationary Weather Satellites NOAA concurred with our fourth recommendation to add to the program’s critical risk list that GOES-S milestones may be affected by GOES-R development and to ensure that this risk and the program-identified funding stability risk are adequately monitored and mitigated. NOAA stated that it currently has identified and reported on the GOES-S milestone risk and that the program is working with NOAA to monitor and mitigate the funding stability risk. In a general comment on our discussion of contractors’ estimated cost growth, NOAA stated that certain growth in contract costs was due to scope changes, including instrument options that were exercised in 2010 and 2011, and stated that such changes should be distinguished from cost growth due to efforts to resolve problems. We consider scope changes to be a valid component of program cost growth. Nevertheless, we revised our report to state that approximately $60 million of the $757 million growth in contractors’ estimated costs at completion from January 2010 through November 2011 was due to scope changes associated with new instrument flight model development. If these scope changes are excluded, contractors’ estimated costs for flight and ground project components still grew by approximately $700 million, or approximately 30 percent, between January 2010 and January 2012. NOAA commented that the GOES-R launch readiness date changed from December 2014 to October 2015 due to a protest against the spacecraft contract award. We acknowledged the reasons for delays in GOES-R’s launch readiness date in our draft report submitted to NOAA and in our prior reports on this program, and have added language to this report that relates to the bid protest and NASA’s response in delaying the award. NOAA also commented that the revised incremental schedule was less risky than its original waterfall schedule. While we acknowledged the reduced risk in our draft report and have further revised our report to reflect NOAA’s comments on our draft, we also note that the revised development methodology introduced other risks to the program—such as additional contractor staff and software development and verification activities that require government oversight and continuous monitoring. Page 41 GAO-12-576 Geostationary Weather Satellites As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 1 day from the report date. At that time, we will send copies to interested congressional committees, the Secretary of Commerce, the Administrator of NASA, the Director of the Office of Management and Budget, and other interested parties. The report also will be available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or email@example.com. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. David A. Powner Director Information Technology Management Issues Page 42 GAO-12-576 Geostationary Weather Satellites Appendix I: Objectives, Scope, and Appendix I: Objectives, Scope, and Methodology Methodology Our objectives were to (1) assess the National Oceanic and Atmospheric Administration’s (NOAA) progress in developing the Geostationary Operational Environmental Satellite-R series (GOES-R) program, (2) evaluate whether the agency has a reliable schedule for executing the program, and (3) determine whether the program is applying best practices in managing and mitigating its risks. To assess progress in developing the GOES-R satellite program, we compared the program’s planned completion dates for key milestones identified in its management control plan and system review plan against actual and current estimated completion dates of milestones. We analyzed program monthly status briefings to identify the current status and recent development challenges of flight and ground project components and instruments. We also analyzed contractor- and program- reported data of development costs and reserves. Finally, we conducted several interviews with GOES-R program staff to better understand milestone time frames, to discuss current status and recent development challenges for work currently being performed on GOES-R, and to understand how the program reports costs and reserve totals. To evaluate whether NOAA has a reliable schedule for executing the program, we evaluated contractor schedules to determine the extent to which GOES-R is following our identified best practices in creating and maintaining its schedules. 1 We analyzed four contractor schedules—two of which represent the overall schedules for the flight and ground projects (spacecraft and Core Ground System), the program’s critical flight instrument (Advanced Baseline Imager), and an instrument that had been experiencing implementation issues at the time of our review (Geostationary Lightning Mapper). We also populated workbooks as a part of that analysis to highlight potential areas of strengths and weakness in schedule logic, use of resources, task duration, float, and task completion; analyzed programwide initiatives undertaken by GOES- R such as the Joint Cost and Schedule Confidence Level and the Giver- Receiver Intersegment Database; assessed GOES-R’s progress against their own scheduling requirements, and interviewed government and contractor officials regarding their scheduling practices. 1 The subordinate schedules analyzed were the Advanced Baseline Imager, the Geostationary Lightning Mapper, and the schedules for GOES-R’s flight and ground projects—the spacecraft and Core Ground System. Page 43 GAO-12-576 Geostationary Weather Satellites Appendix I: Objectives, Scope, and Methodology To determine whether the GOES-R program is applying best practices in managing and mitigating its risks, we analyzed the program’s risk management plan to identify the program’s policies and procedures and compared these to outputs from the program’s risk register and other program risk management documentation, such as risk mitigation status reports. We also reviewed documents such as monthly briefings to the NOAA Program Management Council and program risk status reports to determine how individual risks are managed and reviewed on a regular basis. We assessed the extent to which the program’s policies and procedures met government and industry recognized risk management practices from the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition (Version 1.3), the International Organization for Standardization, the Defense Acquisition University, and GAO, and whether the program fully implemented its policies and procedures. We also examined the program’s risk status reports and risk register to identify risks for which the program did not have an adequate mitigation plan or were not being actively monitored by the program. Finally, we interviewed GOES-R officials to gain further insight into the program’s risk management process and the program’s application of its process. We primarily performed our work at NOAA and National Aeronautics and Space Administration offices in the Washington, D.C., metropolitan area. We conducted this performance audit from August 2011 to June 2012, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 44 GAO-12-576 Geostationary Weather Satellites Appendix II: Comments from the Department Appendix II: Comments from the Department of Commerce of Commerce Page 45 GAO-12-576 Geostationary Weather Satellites Appendix II: Comments from the Department of Commerce Page 46 GAO-12-576 Geostationary Weather Satellites Appendix II: Comments from the Department of Commerce Page 47 GAO-12-576 Geostationary Weather Satellites Appendix II: Comments from the Department of Commerce Page 48 GAO-12-576 Geostationary Weather Satellites Appendix III: GAO Contact and Staff Appendix III: GAO Contact and Staff Acknowledgments Acknowledgments David A. Powner, (202) 512-9286 or firstname.lastname@example.org GAO Contact In addition to the contact named above, individuals making contributions Staff to this report included Colleen Phillips (assistant director), Paula Moore Acknowledgments (assistant director), Shaun Byrnes, Nancy Glover, Franklin Jackson, Jason Lee, and Josh Leiling. 311257 Page 49 GAO-12-576 Geostationary Weather Satellites GAO’s Mission The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no Obtaining Copies of cost is through GAO’s website (www.gao.gov). Each weekday afternoon, GAO Reports and GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, Testimony go to www.gao.gov and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov. Contact: To Report Fraud, Waste, and Abuse in Website: www.gao.gov/fraudnet/fraudnet.htm E-mail: email@example.com Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, firstname.lastname@example.org, (202) 512- Congressional 4400, U.S. Government Accountability Office, 441 G Street NW, Room Relations 7125, Washington, DC 20548 Chuck Young, Managing Director, email@example.com, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Please Print on Recycled Paper.
Geostationary Weather Satellites: Design Progress Made, but Schedule Uncertainty Needs to be Addressed
Published by the Government Accountability Office on 2012-06-26.
Below is a raw (and likely hideous) rendition of the original report. (PDF)