oversight

James Webb Space Telescope: Actions Needed to Improve Cost Estimate and Oversight of Test and Integration

Published by the Government Accountability Office on 2012-12-03.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Committees




                JAMES WEBB SPACE
December 2012



                TELESCOPE

                Actions Needed to
                Improve Cost
                Estimate and
                Oversight of Test and
                Integration




GAO-13-4
                                               December 2012

                                               JAMES WEBB SPACE TELESCOPE
                                               Actions Needed to Improve Cost Estimate and
                                               Oversight of Test and Integration
Highlights of GAO-13-4, a report to
congressional committees




Why GAO Did This Study                         What GAO Found
JWST is one of NASA’s most                     The National Aeronautics and Space Administration (NASA) has provided
expensive and technologically                  significantly more time and money to the James Webb Space Telescope (JWST)
advanced science projects, intended to         than previously planned and expressed high confidence in the project's new
advance understanding of the origin of         baselines. Its current cost estimate reflects some features of best practices for
the universe. In 2011, JWST was                developing reliable and credible estimates. For example, the estimate
rebaselined with a life cycle cost             substantially meets one of four cost characteristics—comprehensive—that GAO
estimate of $8.8 billion and a launch          looks for in a reliable cost estimate, in part because all life cycle costs were
readiness date in October 2018—                included. The estimate, however, only partially met the other three
almost nine times the cost and more            characteristics—well documented, accurate, and credible—which detracts from
than a decade later than originally            its reliability. For example, the estimate's accuracy, and therefore the confidence
projected in 1999. Concern about the           level assigned to the estimate, was lessened by the summary schedule used for
magnitude of JWST's cost increase              the joint cost and schedule risk analysis because it did not provide enough detail
and schedule delay and their effects on        to determine how risks were applied to critical project activities. The estimate's
NASA’s progress on other high-priority
                                               credibility was also lessened because officials did not perform a sensitivity
missions led conferees for the
                                               analysis that would have identified key drivers of costs, such as workforce size.
Consolidated and Further Continuing
Appropriations Act, 2012, to direct
                                               Program officials believe that it would have been difficult to fully address all best
GAO to report on the project.                  practice characteristics. GAO believes there is time to improve the estimate and
Specifically, GAO assessed (1) the             enhance the prospects for delivering the project according to plan.
extent to which NASA’s revised cost
and schedule estimates are reliable            Project officials report that the JWST schedule has 14 months of reserve, which
based on best practices, (2) the major         meets Goddard guidance for schedule reserve; however, only 7 of the 14 months
risks and technological challenges             are likely to be available for the last three of JWST’s five complex integration and
JWST faces, and (3) the extent to              test efforts. GAO's prior work shows that the integration and test phases are
which NASA has improved oversight of           where problems are commonly found and schedules tend to slip. Given that
JWST. To do this, GAO compared                 JWST has a challenging integration and test schedule, this could particularly be
NASA’s revised cost and schedule               likely. The project has made some significant progress in the past year, notably
estimates with best practice criteria,         successfully completing development of the 18 primary mirror segments—
reviewed relevant contractor and               considered JWST's top technical risk. Nevertheless, ongoing challenges are
NASA documents, and interviewed                indicative of the kinds of issues that can require significant effort to address. For
project and contractor officials.              example, instrument challenges have delayed the first integration and test effort.
What GAO Recommends                            In addition, key long-term risks on subsystems with a significant amount of work
                                               remaining will not be retired until 2016. Currently, NASA’s plan for project
GAO recommends NASA take six                   oversight calls for one independent mission-level system integration review about
actions including, among others, to            13 months before launch. While this is consistent with what NASA requires for its
take steps to improve its cost estimate;       projects, this approach may not be sufficient for a project as complex as JWST.
to conduct an additional, earlier
independent review of test and                 JWST has taken several steps to improve communications and oversight of the
integration activities; and to develop a       project and its contractors—such as taking over responsibility for mission
long-term oversight plan that                  systems engineering from the prime contractor; instituting meetings that include
anticipates planned travel budget              various levels of NASA, contractor, and subcontractor management; and
reductions. In commenting on a draft of        implementing a new risk management system to allow for better tracking of risks.
this report, NASA fully or partially           The enhancements to the oversight of the project are steps in the right direction,
concurred with the recommendations             but it will take time to assess their effectiveness and ensure that the efforts are
citing ongoing efforts, but GAO                sustained by the project in the future. Reductions in travel budgets, however,
believes some do not go far enough.            could require the project to adjust the oversight approach that was adopted as a
View GAO-13-4. For more information, contact   result of the replan. Additional reductions in travel budgets are anticipated in
Cristina Chaplain at (202) 512-4841 or
chaplainc@gao.gov.
                                               future years, but officials do not have a plan to address such reductions and their
                                               potential impact on continuing the current oversight approach.
                                                                                        United States Government Accountability Office
Contents


Letter                                                                                   1
               Background                                                                3
               Revised JWST Cost Estimate Is Not Fully Consistent with Best
                 Practices for Reliable and Credible Estimates and the Integrated
                 Master Schedule Is Not Finalized                                        9
               Technically Challenging JWST Project Lacks the Schedule Reserve
                 Flexibility and Commensurate Oversight of Integration and Test
                 Efforts                                                                16
               JWST Project Has Taken Steps to Enhance Communications with
                 and Oversight of Its Contractors                                       26
               Conclusions                                                              32
               Recommendations for Executive Action                                     33
               Agency Comments and Our Evaluation                                       34

Appendix I     Objectives, Scope, and Methodology                                      40



Appendix II    Subsystems of the JWST Observatory                                      44



Appendix III   Our Evaluation of JWST’s Cost Estimate Process                          45



Appendix IV    Comments from the National Aeronautics and Space
               Administration                                                          47



Appendix V     GAO Contact and Staff Acknowledgments                                   51



Tables
               Table 1: Issues with JWST Science Instruments Included in ISIM           21
               Table 2: ICRP Recommendations Concerning Oversight and
                        Communications and the Actions Taken by NASA                    27
               Table 3: NASA Office of Evaluation Review Team Concerns                  31
               Table 4: Summary Assessment of JWST Cost Estimate Compared
                        to Best Practices                                               45




               Page i                                   GAO-13-4 James Webb Space Telescope
Figures
          Figure 1: Subsystems of the JWST                                                          6
          Figure 2: Organizational Chart for the JWST Program                                       8
          Figure 3: Summary of Results of GAO Assessment of JWST Cost
                   Estimate Based on Best Practices Criteria                                        10
          Figure 4: Project Integration and Test Time Line                                          17
          Figure 5: Percentage of Budgeted Development Work Remaining
                   for Major Subsystems                                                             19
          Figure 6: Current Mass Estimate for Spacecraft Subsystem                                  20
          Figure 7: Changes in ISIM Schedule Since the Project Replan                               24
          Figure 8: Subsystems of the JWST: Interactive Information                                 44




          Abbreviations

          CSA               Canadian Space Agency
          ESA               European Space Agency
          FGS               Fine Guidance Sensor
          ICRP              Independent Comprehensive Review Panel
          ISIM              Integrated Science Instrument Module
          JCL               Joint Cost and Schedule Confidence Level
          JWST              James Webb Space Telescope
          KDP               key decision point
          MIRI              Mid-Infrared Instrument
          NASA              National Aeronautics and Space Administration
          NIRCam            Near-Infrared Camera
          NIRISS            Near-Infrared Imager and Slitless Spectrograph
          NIRSpec           Near-Infrared Spectrograph
          NPR               NASA Procedural Requirements
          OTE               Optical Telescope Element
          OTIS              Optical Telescope Element and Integrated Science
                             Instrument Module

          This is a work of the U.S. government and is not subject to copyright protection in the
          United States. The published product may be reproduced and distributed in its entirety
          without further permission from GAO. However, because this work may contain
          copyrighted images or other material, permission from the copyright holder may be
          necessary if you wish to reproduce this material separately.




          Page ii                                            GAO-13-4 James Webb Space Telescope
United States Government Accountability Office
Washington, DC 20548




                                   December 3, 2012

                                   Congressional Committees

                                   The James Webb Space Telescope (JWST) is one of the National
                                   Aeronautics and Space Administration’s (NASA) most expensive and
                                   technologically advanced science projects. It is intended to revolutionize
                                   our understanding of star and planet formation and advance the search
                                   for the origins of our universe. The project has been hampered by
                                   ineffective management, unrealistic estimates of costs, and insufficient
                                   resources, and is projected to cost significantly more and to take much
                                   longer to complete than expected when it was authorized to proceed in
                                   1999. In 2011, JWST was rebaselined with a lifecycle cost estimate of
                                   $8.8 billion and a launch date in October 2018, which is almost nine times
                                   the cost and more than a decade later than originally projected. The
                                   magnitude of JWST’s cost increase and schedule delay has and will
                                   continue to have a major effect on NASA’s ability to implement or begin
                                   high-priority science missions for years to come.

                                   NASA has expressed high confidence in being able to execute the project
                                   within its new cost and schedule baselines, and officials report that the
                                   project is being executed to its revised estimates in the year since the
                                   rebaseline. The project, however, is less than halfway through its
                                   development with known and unknown technical risks and challenges
                                   remaining to be addressed. The on-time and on-budget delivery of JWST
                                   is a high congressional priority. In November 2011, the Joint Explanatory
                                   Statement of the Committee of Conference for the Consolidated and
                                   Further Continuing Appropriations Act, 2012, recommended that JWST’s
                                   formulation and development costs, not including the cost to operate the
                                   mission after launch, be no more than $8 billion. The conferees also
                                   directed GAO to assess the program annually and to report to the
                                   Committees on Appropriations on key issues relating to program and risk
                                   management, achievement of cost and schedule goals, and program
                                   technical status. This report responds to that direction. Specifically, we
                                   assessed (1) the extent to which NASA’s revised cost and schedule
                                   estimates are reliable based on GAO best practices, (2) the major risks
                                   and technological challenges the JWST project faces, and (3) the extent
                                   to which NASA has improved the oversight of the JWST project.




                                   Page 1                                   GAO-13-4 James Webb Space Telescope
Our approach included an examination of the revised cost and schedule
estimates NASA developed during the replanning process using GAO’s
best practice criteria. 1 This included review of documentation of the inputs
that supported the detailed cost and schedule estimates used by
decision-makers to rebaseline the program and interviews with project
officials involved in developing the estimates. We also reviewed the
proposal from the JWST prime contractor used as input to the replan and
interviewed contractor officials. We did not assess the most current JWST
integrated master schedule because of ongoing negotiations for contract
modifications associated with the replan. To identify the major short and
long term risks and technological challenges facing the project, we
interviewed project officials for the major subsystems as well as
contractor officials. We also reviewed the project’s risk database, monthly
status reviews, and other documentation provided by the project officials.
To understand the steps NASA implemented to improve the oversight
being performed on the project, we interviewed project and contractor
officials and reviewed documentation from independent reviews prior to
the replan and actions taken by NASA in response. We also interviewed
the Director of NASA’s Office of Evaluation concerning a recent
independent review of the JWST project performed by his office. We took
appropriate steps to address data reliability, such as clarifying data
discrepancies and corroborating NASA-generated data with other sources
where applicable. NASA provided comments and technical clarifications
on a draft of this report, which were incorporated as appropriate.
Appendix I contains detailed information on our scope and methodology.

We conducted this performance audit from February 2012 to December
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




1
 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and
Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: Mar. 2, 2009); and
GAO Schedule Assessment Guide: Best Practices for Project Schedules, GAO-12-120G
(Washington, D. C.: May 30, 2012).




Page 2                                        GAO-13-4 James Webb Space Telescope
                         Initially referred to as the “Next Generation Space Telescope,” JWST is a
Background               large deployable, infrared-optimized space telescope intended to be the
                         successor to the aging Hubble Space Telescope. JWST is designed to be
                         a 5-year mission to find the first stars and trace the evolution of galaxies
                         from their beginning to their current formation, and is intended to operate
                         in an orbit approximately 1.5 million kilometers—or 1 million miles—from
                         the Earth. In a 2001 decadal survey, the National Research Council rated
                         the JWST as the top-priority new initiative for astronomy and physics.
                         With its 6.5-meter primary mirror, JWST will be able to operate at 100
                         times the sensitivity of the Hubble Space Telescope. A tennis-court-sized
                         sunshield will protect the mirrors and instruments from the sun’s heat to
                         allow the JWST to look at very faint infrared sources. The Hubble Space
                         Telescope operates primarily in the visible and ultraviolet regions. 2


History of Cost Growth   JWST has experienced significant increases to project costs and
and Schedule Delays      schedule delays. Prior to being approved for development, cost estimates
                         of the project ranged from $1 billion to $3.5 billion with expected launch
                         dates ranging from 2007 to 2011. In March 2005, NASA increased the
                         JWST’s life-cycle cost estimate to $4.5 billion and slipped the launch date
                         to 2013. We reported in 2006 that about half of the cost growth was due
                         to schedule slippage—a 1-year schedule slip because of a delay in the
                         decision to use a European Space Agency-supplied Ariane 5 launch
                         vehicle and an additional 10-month slip caused by budget profile
                         limitations in fiscal years 2006 and 2007. More than a third of the cost
                         increase was caused by requirements and other changes. An increase in
                         the program’s contingency funding accounted for the remainder—about
                         12 percent—of the growth. NASA Headquarters chartered an
                         Independent Review Team to evaluate the project that same year. In April
                         2006, the review team’s assessment confirmed that the program’s
                         technical content was complete and sound, but expressed concern over
                         the project’s contingency reserve funding—funding used to mitigate
                         issues that arise but which were previously unknown—reporting that it
                         was too low and phased in too late in the development life cycle. The
                         team reported that for a project as complex as the JWST, a 25 to 30



                         2
                          The electromagnetic spectrum is the wavelengths of all the visible and invisible light. The
                         infrared part of the spectrum, also known as radiant heat, has wavelengths that go from
                         about 0.75 microns to a few hundred microns. The Hubble is designed to operate primarily
                         in the ultraviolet and visible wavelengths of the spectrum from 0.1 to 0.8 microns. Humans
                         cannot see in the ultraviolet region.




                         Page 3                                             GAO-13-4 James Webb Space Telescope
percent total contingency was appropriate. At that time, JWST’s total
contingency was about 19 percent. The team cautioned that this
contingency compromised the project’s ability to resolve issues, address
risk areas, and accommodate unknown problems. The team also
concluded that the 2013 launch date was not viable for the project based
on its anticipated budget. It recommended that before the project was
formally approved for development and baselined, NASA should take
steps to provide the JWST project with adequate time-phased reserve
funding to secure a stable launch date. Additional reserves were added
and the project was baselined in April 2009 with a life-cycle cost estimate
of $4.964 billion and a launch date in June 2014.

Shortly after JWST was approved for development and its cost and
schedule estimates were baselined, project costs continued to increase.
In 2010, Senator Barbara Mikulski, chair of the Senate Committee on
Appropriations, Subcommittee on Commerce, Justice, Science, and
Related Agencies, asked NASA to initiate another independent review in
response to the project’s cost increases and reports that the June 2014
launch date was in jeopardy. The Independent Comprehensive Review
Panel (ICRP) was commissioned by NASA and began its review in
August 2010. In October 2010, the ICRP issued its report and cited
several reasons for the project’s problems including management,
budgeting, oversight, governance and accountability, and communication
issues. 3 The panel concluded JWST was executing well from a technical
standpoint, but that the baseline funding did not reflect the most probable
cost with adequate reserves in each year of project execution, resulting in
an unexecutable project. The review panel recommended that additional
resources be considered along with organizational and management
restructuring. Following this review, the JWST program underwent a
replan in 2011. In November 2011, the JWST project was reauthorized,
but not before it was recommended for termination by the House
Appropriations Committee. On the basis of the replan, NASA announced
that the project would be rebaselined at $8.835 billion—a 78 percent
increase to the project’s life-cycle cost from the confirmed baseline—and
would launch in October 2018—a delay of 52 months. The revised life-
cycle cost estimate included 13 months of funded schedule reserve. In
the President’s Fiscal Year 2013 budget request, NASA reported a 66



3
 James Webb Space Telescope (JWST) Independent Comprehensive Review Panel
(ICRP): Final Report (Oct. 29, 2010).




Page 4                                     GAO-13-4 James Webb Space Telescope
               percent joint cost and schedule confidence level associated with these
               estimates. A joint cost and schedule confidence level (JCL) is the process
               NASA uses to assign a percentage to the probable success of meeting
               cost and schedule targets and is part of the project’s estimating process.


Current JWST   The JWST project is divided into three major segments: the launch
Organization   segment, the ground segment, and the observatory segment. The launch
               segment is primarily provided by the European Space Agency (ESA),
               which is contributing the Ariane 5 launch vehicle and launch site
               operations in French Guiana. The ground segment will be responsible for
               collecting the data obtained by JWST in space and making it usable for
               scientists and researchers. This includes the development of software
               that will translate data into usable formats as well as operation of the
               software once the telescope is in space. The Space Telescope Science
               Institute, operated by the Association of Universities for Research in
               Astronomy (AURA) on a contract awarded by NASA, which currently
               performs science operations for the Hubble Space Telescope, is
               developing the science and operations and flight operations center for
               JWST and will conduct the first 6 months of flight and science operations.
               The NASA contract with the Space Telescope Science Institute extends
               through the first 6 months of JWST operations. A contract to manage the
               long term operations of JWST is planned to be awarded approximately 2
               years prior to launch. The observatory segment will be launched into
               space and includes five major subsystems. 4 These subsystems are being
               developed through a mixture of NASA, contractor, and international
               partner efforts. See figure 1.




               4
                The hardware configuration created when the Optical Telescope Element and the
               Integrated Science Instrument Module are integrated, referred to as OTIS, is not
               considered a subsystem by NASA, but we categorize it as such for ease of discussion.




               Page 5                                           GAO-13-4 James Webb Space Telescope
Figure 1: James Webb Space Telescope

 Interactive Graphic   Rollover the white dots to see description. See appendix II for the non-interactive, printer-friendly version.




                            Optical Telescope &
                             Integrated Science                                                         Optical
                             Instrument Module                                                          Telescope
                                                                                                        Element
                        Integrated Science
                        Instrument Module




                                                                                                                                        Sunshield


                                 Ground systems

                                                                                  Spacecraft




                                                     Sources: GAO (analysis); NASA (data and images).




                                                    Page 6                                                             GAO-13-4 James Webb Space Telescope
JWST is a single project program reporting directly to the NASA
Associate Administrator for programmatic oversight and to the Associate
Administrator for the Science Mission Directorate for technical and
analysis support. 5 Goddard Space Flight Center is the NASA center
responsible for the management of JWST. See figure 2 for the current
JWST organizational chart.




5
 A project typically reports to a program within a mission directorate, such as the Science
Mission Directorate. A similar reporting structure was also instituted in the past with both
the Hubble Space Telescope and the Mars Exploration Program when they began
experiencing significant cost and schedule issues. For JWST, the change was made in
response to recommendations from the ICRP.




Page 7                                              GAO-13-4 James Webb Space Telescope
Figure 2: Organizational Chart for the JWST Program




Page 8                                       GAO-13-4 James Webb Space Telescope
                         Our analysis of JWST’s revised cost estimate showed that it is not fully
Revised JWST Cost        consistent with best practices for developing reliable and credible
Estimate Is Not Fully    estimates, although project officials took some steps in line with best
                         practices in the development of the estimate. For example, as part of its
Consistent with Best     cost estimation process, the project conducted a joint cost and schedule
Practices for Reliable   risk analysis, or joint cost and schedule confidence level (JCL), which
                         assigned a 66 percent confidence level to the estimate. 6 In addition, we
and Credible             found that the cost estimate included all life cycle costs for the project.
Estimates and the        Although NASA’s methods for developing the JWST cost estimate reflect
                         some features of best practices, our review of the estimate showed that
Integrated Master        based on best practice criteria, it did not fully meet the four characteristics
Schedule Is Not          of a reliable estimate. See figure 3.
Finalized




                         6
                          The JCL is a quantitative probability analysis that requires the project to combine its cost,
                         schedule, and risks into a complete quantitative picture to help assess whether the project
                         will be successfully completed within cost and on schedule. NASA introduced the analysis
                         in 2009, and it is among the agency’s initiatives to reduce acquisition management risk.
                         The move to probabilistic estimating marks a major departure from NASA’s prior practice
                         of establishing a point estimate and adding a percentage on top of that point estimate to
                         provide for contingencies. NASA’s procedural requirements state that Mission Directorates
                         should plan and budget programs and projects based on a 70 percent JCL, or at a
                         different level as approved by the Decision Authority—which in the case of the JWST
                         project would be the Agency Program Management Council—and any JCL approved at
                         less than 70 percent must be justified and documented. NASA Procedural Requirements
                         (NPR) 7120.5E, NASA’s Space Flight Program and Project Management Requirements,
                         paragraph 2.4.4 (Aug. 14, 2012).




                         Page 9                                              GAO-13-4 James Webb Space Telescope
Figure 3: Summary of Results of GAO Assessment of JWST Cost Estimate Based on Best Practices Criteria




                                       Note: Met: NASA provided complete evidence that satisfies the entire criterion; Substantially Met:
                                       NASA provided evidence that satisfies a large portion of the criterion; Partially Met: NASA provided
                                       evidence that satisfies about half of the criterion; Minimally Met: NASA provided evidence that
                                       satisfies a small portion of the criterion; and Not Met: NASA provided no evidence that satisfies any of
                                       the criterion. See appendix III for our evaluation of JWST’s cost estimate process.


                                       Specifically, the project’s estimate was found to substantially meet the
                                       best practice criteria for being comprehensive, and the remaining three
                                       characteristics of being well documented, accurate, and credible were
                                       found to be only partially met. For example, the accuracy of the cost
                                       estimate, and therefore the confidence level assigned to the estimate,
                                       was lessened by the schedule used in the JCL analysis because it
                                       prevented us from, among other things, identifying the activities that were
                                       on the critical path—defined as time associated with activities that drive
                                       the overall schedule. The credibility of the estimate was lessened


                                       Page 10                                                   GAO-13-4 James Webb Space Telescope
because project officials did not perform a sensitivity analysis that would
have identified key drivers of costs, such as workforce size. Although
NASA is not required to adhere to these best practices, our prior work has
shown that not following best practices for cost estimating can make the
cost estimate less reliable, putting projects at risk of experiencing cost
overruns, missed deadlines, and performance shortfalls. 7 The best
practices stem from practices federal cost estimating organizations and
industry use to develop and maintain reliable cost estimates, including the
Department of Defense and NASA. According to program officials, it
would have been difficult, if not impossible, for the project to have met all
of the best practice criteria given the complexity of the project and that
some elements of the project are quite mature in their development.
Instead, the program manager stated that the project followed a tailored
process to develop the cost estimate that was appropriate for the project.
Furthermore, officials report the project is currently meeting a majority of
its milestones and executing as planned to the revised estimates for the
JWST.

Comprehensive: The JWST cost estimate substantially met the criteria
for being comprehensive because it included all life-cycle costs,
documented cost-influencing ground rules and assumptions, and used a
work breakdown structure, or the structure used to define in detail the
work necessary to accomplish its objectives, that is detailed and traceable
to the cost of each work element and the contract statement of work. 8 The
cost estimate is not considered fully comprehensive, however, because
we were unable to align the schedule to the cost estimate and could not
align all of the subcontractor work to the work breakdown structure of the
cost estimate. Although NASA officials believe that the project has
visibility into, and can map, subcontractor activities, we were unable to
confirm that cost elements were neither omitted nor double counted. In
addition, we were unable to map the project’s earned value management
data to the work breakdown structure because information between the




7
GAO-09-3SP.
8
 A work breakdown structure reflects the requirements and what must be accomplished to
develop a program, and it provides a basis for identifying resources and tasks for
developing a program cost estimate. The work breakdown structure should be used to
define all program activities and tasks to ensure that the schedule encompasses the entire
work.




Page 11                                           GAO-13-4 James Webb Space Telescope
two was not compatible. 9 Finally, although the project outlined and
documented the ground rules and assumptions, we were unable to
determine whether risks associated with any assumptions were identified
and traced to specific elements.

Well documented: The JWST cost estimate only partially met the criteria
for being well documented because it did not include a step-by-step
description of how the estimate was developed, the raw data used to
develop the estimate, or the calculations and estimating methodology for
specific cost elements of the work breakdown structure. Without good
documentation, a cost analyst unfamiliar with the program will not be able
to replicate the estimate, because he or she will not understand the logic
behind it. Good documentation, for example, assists management and
oversight in assessing the credibility of the estimate, helps to keep a
history of reasons for cost changes and to record lessons learned,
defines the scope of the analysis, and answers questions about the
approach or data used to create the estimate. Project documentation,
however, does provide evidence that NASA management reviewed and
accepted the cost estimate because managers were briefed on the
technical aspects of the estimate and were provided an overview of the
joint cost and schedule risk analysis that was conducted.

Accurate: The JWST cost estimate only partially met the criteria for being
accurate because the projected costs of schedule reserve did not reflect
actual data, the summary schedule used to derive the JCL prevented us
from sufficiently understanding how risks were incorporated, and the
project did not provide evidence that it regularly updates the estimate or
plans to conduct another JCL. For example, using historical actual cost
data from Northrop Grumman, we estimated that 13 months of schedule
reserve is likely to be $204 million instead of NASA’s estimate of $121
million—a potential underestimation of 69 percent related to the schedule
reserve. Project officials, however, believe they have adequate reserves
available to offset any underestimation. In addition, the summary


9
 Earned value management is a project management tool that integrates the technical
scope of work with schedule and cost elements and compares the value of work
accomplished in a given period with the value of the work expected in that period. When
used properly, earned value management can provide objective assessments of project
progress, produce early warning signs of impending schedule delays and cost overruns,
and provide unbiased estimates of anticipated costs at completion. As a best practice the
work breakdown structure should match the schedule, cost estimate, and earned value
management system at a high level so that it clearly reflects the work to be done.




Page 12                                           GAO-13-4 James Webb Space Telescope
schedule the project used as an input to the JCL, although deemed
acceptable by NASA, contained many long-duration activities, some with
1,000 days or more. Because of these long durations in the summary
schedule used for the JCL, the lack of detail prevented us from identifying
the activities that were on the critical path, as well as which risks were
applied to remaining activities. 10 As a result, there is no way to ensure
that risks were appropriately assigned to activities in the schedule to
account for the impact of the risks during the JCL analysis.

Finally, it was unclear whether the cost estimate was regularly updated to
reflect material changes in actual costs and in the project itself, such as
when schedules or other assumptions change, due to a lack of detailed
documentation for the cost estimate. Project officials stated that in
keeping with NASA policy they do not plan, nor are they required, to
conduct another JCL analysis. GAO’s cost estimating best practices call
for estimates to be continually updated through the life of the project,
ideally every month as actual costs are reported in earned value
management reports, and that a risk analysis and risk simulation
exercise—like the JCL analysis—be conducted periodically through the
life of the program, as risks can materialize or change throughout the life
of a project. Unless properly updated on a regular basis, the cost estimate
cannot provide decision makers with accurate information to assess the
current status of the project. NASA officials state that the life-cycle cost
estimate is updated annually for the budgeting process, and that historical
records such as earned value data were used to develop the estimate.
They also stated that this information is updated in several different
documents being provided to management; however, we were unable to
determine how this information was used in updating the cost estimate on
a regular basis.

Credible: The JWST cost estimate only partially met the criteria for being
credible because project officials did not adequately test and verify the
reasonableness of the cost estimate and the schedule used in conducting
the JCL did not have a valid critical path and contained durations that
were too long to properly account for risks. For example, project officials
said they did not perform a sensitivity analysis for the cost estimate. A
sensitivity analysis identifies key elements that drive cost and permits


10
  A summary schedule is derived from the integrated master schedule and is often used in
a JCL analysis when, for example, time or resources needed to simulate the full detailed
schedule are limited.




Page 13                                          GAO-13-4 James Webb Space Telescope
analysis of different outcomes and is often used to develop cost ranges
and risk reserves. NASA officials stated that the largest cost driver for the
JWST project is the size of the workforce, which could have been
subjected to a sensitivity analysis; yet, the cost model did not include a
sensitivity analysis that would show how staff increasing or decreasing
over time affects cost. In addition, NASA officials believe that all risks
were sufficiently accounted for when conducting the JCL, however, the
software used to conduct the JCL analysis does not recognize certain
risks that officials had placed on activities in the project schedule and,
therefore, some risks were discarded during the simulation.

The schedule used to conduct the JCL was also summarized at such a
high level that the durations were too long to effectively model the risks.
For example, one of the activities that drove the launch date was over 4
years in duration and should have been broken down further prior to
conducting the simulation. Moreover, the critical path in the JCL schedule
consisted of six level of effort activities all with the same duration of 2,238
days in length. 11 Level of effort activities should never be on the critical
path because support activities should never drive any milestone finish
date. As a result of the schedule used in the JCL not fully meeting best
practices, we question the results of the analysis. Furthermore, the risk of
having to carry the JWST workforce to support the project if delayed was
not included since a sensitivity analysis was not performed. Project
officials report that, instead, risk associated with the workforce was
factored in when establishing cost reserves.

In addition, project officials did not commission an independent cost
estimate, which is considered one of the best and most reliable estimate
validation methods because it shows whether other estimating
procedures produce similar results, and it provides an independent view
of expected program costs that tests the program office’s estimate for
reasonableness. 12 An estimate that has not been reconciled with an
independent cost estimate has an increased risk of being underfunded


11
  Level of effort activities are those activities that have no measurable output and cannot
be associated with a physical product or defined deliverable. These activities are typically
related to management and other oversight that continues until the detailed activities they
support have been completed.
12
  An independent cost estimate is conducted by a group outside the acquiring
organization that uses the same detailed technical information as the program estimate to
determine whether other estimating methods produce similar results.




Page 14                                             GAO-13-4 James Webb Space Telescope
                             because the independent cost estimate provides an objective and
                             unbiased assessment of whether the project estimate can be achieved.
                             Notably, however, project officials provided evidence that an independent
                             cost assessment was done for the project at the request of the JWST
                             Standing Review Board, the independent review team for the project, and
                             the assessment was within 2 percent of the project’s estimated cost for
                             the rebaseline. Project officials contend that the approach they used in
                             developing the life-cycle cost estimate for the project is more accurate
                             than the types of approaches often used to develop and independent
                             estimate.


Complete Schedule Not        We did not conduct a full schedule assessment to determine the reliability
Finalized; Full Assessment   of the revised schedule based on best practices due to on-going contract
of Schedule Needed           negotiations. 13 The project has an integrated master schedule developed
                             as part of the replan; however, it is not finalized because major contract
                             modifications have yet to be negotiated and definitized. 14 Specifically, the
                             modification to the Northrop Grumman contract, which accounts for
                             approximately 40 percent of the total project cost and spans much of the
                             work on the spacecraft and OTE, remains undefinitized more than a year
                             after the project was rebaselined. Once the project completes
                             negotiations for the contract modification and all schedule dates are set,
                             the project can then have a measurable integrated master schedule.
                             Project officials stated that the negotiation process and updating of
                             associated schedules are planned to be complete in January 2013 for the
                             Northrop Grumman contract modification—a year after submission of the
                             latest update to its proposal for the replan. The project also reported that
                             multiple audits of the proposals submitted by Northrop Grumman and its
                             subcontractor by the Defense Contract Audit Agency have delayed
                             definitization. Negotiations for the modification to NASA’s contract with
                             the Space Telescope Science Institute to incorporate the October 2018



                             13
                              GAO-12-120G.
                             14
                               An integrated master schedule constitutes a program schedule that includes the entire
                             required scope of effort, including the effort necessary from all government, contractor,
                             and other key parties for a program’s successful execution from start to finish. As a key
                             focal point for management, the integrated master schedule documents and integrates the
                             planned work, the resources necessary to accomplish that work, and the associated
                             budget. It also consists of logically related activities whose forecasted dates are
                             automatically recalculated when activities change allowing for management to respond to
                             the consequences of these changes.




                             Page 15                                           GAO-13-4 James Webb Space Telescope
                          launch readiness date are not scheduled to be complete until spring
                          2013.

                          Once all the contracts have been definitized and the project’s integrated
                          master schedule is baselined, we plan to conduct a comprehensive best
                          practices assessment of the reliability of the project’s schedule estimates.


                          Project officials report that the JWST schedule has 14 months of reserve,
Technically               which meets Goddard guidance for schedule reserve; however, only 7 of
Challenging JWST          the 14 months are likely to be available for the last three of JWST’s five
                          complex integration and test efforts. GAO’s prior work shows that it is
Project Lacks the         during integration and test where problems are commonly found and
Schedule Reserve          schedules tend to slip. Given that JWST has a challenging integration and
                          test schedule, this could particularly be the case. The project has made
Flexibility and           some significant progress in the past year, notably successfully
Commensurate              completing development of the 18 primary mirror segments—considered
                          JWST’s top technical risk. Nevertheless, ongoing challenges are
Oversight of              indicative of the kinds of issues that can require a significant amount of
Integration and Test      effort to address. For example, instrument challenges have delayed the
                          first integration and test effort. In addition, key long-term risks on
Efforts                   subsystems with a significant amount of work remaining will not be retired
                          until 2016. Currently, NASA’s plan for project oversight calls for one
                          independent system integration review about 13 months before launch.
                          While this is consistent with what NASA requires for its projects, this
                          approach may not be sufficient for a project as complex as JWST. As a
                          result, the current plan may be inadequate to ensure key technical and
                          management issues are identified early enough to be addressed within
                          the current integration and test phase schedule.


Test and Integration      JWST has a complex and lengthy integration and test phase, which
Schedule Lacks Schedule   includes five major integration and test efforts—ISIM, OTE, OTIS,
Reserve Late in the       spacecraft, and observatory. See figure 4 for the project reported dates
                          for the major integration and test efforts and the schedule reserve
Process                   allocated for each effort.




                          Page 16                                    GAO-13-4 James Webb Space Telescope
Figure 4: Project Integration and Test Time Line




                                          Overall, project officials report that the critical path schedule has 14
                                          months of reserve with 7 months after the ISIM and OTE integration and
                                          test efforts. If these efforts are delayed beyond those 7 months, they will
                                          impinge on the schedule for the remaining three integration and test
                                          efforts. Project officials stated that the baseline plan is for the OTIS
                                          integration and test effort to not begin earlier than May 2016. These
                                          officials reported it is likely that all of the 7 months of schedule reserve
                                          held by the OTE subsystem will be utilized during its integration and test
                                          prior to delivery to OTIS and that the OTE effort is on the critical path for
                                          the project. Therefore, the remaining integration and test efforts—OTIS,
                                          Spacecraft, and Observatory—will likely have at most 7 months divided
                                          among them to use if issues are found during integration and test.

                                          In addition to not likely being able to conserve any of the unused first 7
                                          months of schedule reserve, the project has limited time allocated to the
                                          final three integration and test efforts, with between 2 to 4 months for
                                          each. This time could be used easily by the project if an issue were to
                                          arise during integration and test. An example of this is seen in the OTIS



                                          Page 17                                     GAO-13-4 James Webb Space Telescope
                         integration and test schedule, which currently has 3 months of schedule
                         reserve. The final event in the OTIS integration and test effort is a lengthy
                         cryo-vacuum test—the first time that the optics integrated with the
                         instruments will be tested at operational temperatures near absolute zero
                         (less than -400 degrees Fahrenheit)—that takes approximately 3 months,
                         due to the requirements of the test. If an issue were to arise during this
                         test that requires shutting the test down and working on the hardware, the
                         chamber would have to be slowly warmed to a temperature safe for
                         removal of the hardware from the chamber, work would be performed,
                         and the 3-month test process would need to begin again. This could
                         easily exhaust the available schedule reserve. Prior GAO work shows
                         that it is during integration and test when problems are commonly found,
                         and schedules tend to slip. A project official confirmed that this is the case
                         because during integration and test the process is more sequential and
                         there is less flexibility to move work around if problems are found. A
                         NASA Inspector General report on the Mars Science Laboratory, another
                         complex and high-cost mission, found that historically the probability that
                         schedule-impacting problems will arise is commensurate with the
                         complexity of the project. 15 JWST is one of NASA’s most technologically
                         complex projects to date.


Despite Overcoming       The project has made significant progress overcoming several technical
Several Technical        challenges over the last year. In December 2011, for example, the project
Challenges, JWST Has a   completed development of the 18 segments of the primary mirror—the
                         project’s primary technology risk—approximately 6 weeks ahead of
Significant Amount of    schedule. In addition, project officials stated that during the last year they
Work with Complex        were also able to accelerate other optics-related work, which added one
Technical Challenges     month of funded reserve to the schedule, bringing the total to 14 months.
Remaining                Finally, the project successfully addressed an increase in the estimated
                         amount of heat on the instruments, which otherwise could have pushed
                         observatory temperatures close to where the optics would not function
                         correctly.

                         Although technical challenges are being overcome, the project will likely
                         continue to experience additional challenges over the remainder of the
                         project, given the significant portion and complexity of the work



                         15
                           NASA, Office of Inspector General, NASA’s Management of the Mars Science
                         Laboratory Project. IG-11-019. (Washington, D.C.: June 2011).




                         Page 18                                        GAO-13-4 James Webb Space Telescope
remaining. Four of six major subsystems have nearly 50 percent or more
of their development work remaining based on its current budget
information, although the dollar amounts associated with the work vary.
See figure 5.

Figure 5: Percentage of Budgeted Development Work Remaining for Major
Subsystems




Currently, the project is experiencing several technical issues that have
required a significant amount of time and effort to address. For example,
the spacecraft subsystem, which experienced delays in development prior
to the replan, is currently estimated to be heavier than its mass limit. 16
Spacecraft development has lagged behind other subsystems because it
was viewed as a lower risk part of the project and was therefore not
allocated funding when budgets were limited prior to the replan. In March
2010, the project passed its mission critical design review, which
evaluated the project design and its ability to meet mission requirements
and indicated that the design was ready for fabrication phase; however,


16
  Mass is a measurement of how much matter is in an object. It is related to an object’s
weight and is mathematically equal to mass multiplied by acceleration due to gravity. The
project uses mass for JWST because when it goes into space, its weight changes with
gravity, but its mass stays the same.




Page 19                                           GAO-13-4 James Webb Space Telescope
the spacecraft was not included in this review due to its delayed
development. Under the initial replan, which had constrained funding in
fiscal years 2011 and 2012, the spacecraft critical design review was
scheduled for June 2014; however, due to additional funding in the final
agency-approved replan, the project was able to accelerate work and this
review is now planned for December 2013.

Project officials have been concerned with the mass of JWST since its
inception because of the telescope size and the limits of available launch
vehicles. Accordingly, mass limits have been allocated for each
subsystem, including the spacecraft. Project officials stated that they
expected to encounter mass growth on the spacecraft, but that the
magnitude of the mass growth on the spacecraft was unexpected. As
shown in figure 6, the current spacecraft projected mass exceeds its
mass allocation.

Figure 6: Current Mass Estimate for Spacecraft Subsystem




Primary drivers of the mass growth on the spacecraft are increases in the
estimated weight of the wiring harnesses, which distribute power and
electric signals between different parts of the observatory, the solar array,
and other structures that make up the spacecraft. The burden to find



Page 20                                      GAO-13-4 James Webb Space Telescope
                                           ways to reduce mass has been primarily placed with the spacecraft
                                           because it was assessed by the project to have the least technical risk
                                           and because it is the least mature subsystem and can more easily
                                           accommodate design changes. Over 100 kilograms, or 220 pounds, of
                                           mass savings options are being evaluated by the project and Northrop
                                           Grumman, which is developing the spacecraft. Potential mass solutions
                                           have been identified by Northrop Grumman and the project; however,
                                           cost and risk vary with each solution and the project is still evaluating the
                                           trade-offs of the various solutions. Project officials stated that final
                                           decisions for all tradeoffs will need to occur before spacecraft critical
                                           design review in December 2013.

                                           The ISIM subsystem is experiencing technology and engineering
                                           challenges that resulted in the use of 18 of ISIM’s 26 months of schedule
                                           reserve. The schedule for the instruments needed for ISIM continues to
                                           slip, which could result in use of more schedule reserve. Based on the
                                           replan, all four instruments were to be delivered by September 2012;
                                           however, only two instruments were delivered by that time and those still
                                           have issues that must be addressed. The remaining two instruments are
                                           currently scheduled to be delivered at least 11 months late. See table 1
                                           below for the instrument specific issues.

Table 1: Issues with JWST Science Instruments Included in ISIM

Instrument        Instrument specific issues
MIRI              MIRI was delivered in May 2012, 1 month later than the replan date, which did not impact the ISIM integration and
                  test schedule. However, the instrument team is still working to address an instrument sensitivity issue. The
                  instrument team is conducting data analysis to determine a root cause for why the instrument is less sensitive than
                  required. A formal report on the root cause and recommended corrective action is expected in November 2012.
                  NASA officials report that preliminary findings indicate that hardware changes will not be necessary.
NIRSpec           NIRSpec is currently planned for delivery in August 2013, a delay of 11 months from the replan date as a result of
                  several technical issues. First, in July 2011, three cracks were found in the part that holds the optics components
                  for the instrument. A failure review board was held in January 2012. ESA, which is building the instrument, will
                  need to reassemble the instrument using the flight spare part. This part cannot be rebuilt in time to include the
                  instrument in the first of three cryo-vacuum tests of ISIM. The project team adjusted the integration and test
                  schedule to be able to accept delivery of NIRSpec in February 2013. However, additional issues during testing,
                  including failure of the microshutters to close and slower than planned progress on the reintegration by the ESA
                  contractor, have further delayed NIRSpec until August 2013. The microshutters were provided to ESA by Goddard
                  Space Flight Center and enable the instrument to study 100 targets simultaneously. To mitigate schedule risk, the
                  project is studying an option to build new microshutters that will replace the current ones, if necessary. Delivery of
                  NIRSpec after August 2013 may threaten ISIM’s critical path schedule.




                                           Page 21                                              GAO-13-4 James Webb Space Telescope
Instrument   Instrument specific issues
FGS/NIRISS   FGS/NIRISS was delivered as planned in July 2012; however, it was delivered with a known gear motor issue that
             could result in the early loss of NIRISS in space. In July 2012, the Canadian Space Agency (CSA), which built the
             instrument, began to redesign the gear motors and they are being rebuilt. Around the time of the replan, CSA had
             to redesign its instrument because the original design was not meeting performance requirements. Part of
             instrument design was simplified, which resulted in some loss of expected science performance; however, NIRISS
             science is not required for JWST mission success.
NIRCam       NIRCam is currently planned for delivery in August 2013, a delay of 11 months from the replan date as a result of
             technical and workmanship issues. The project determined the need to electrically ground a mirror on the
             instrument and the team is reviewing several potential options to address this issue. Workmanship issues, such as
             poor soldering quality of the wires in the instrument controls electronic box, resulted in the need for corrective
             actions. The project has decided to run the first ISIM cryo-vacuum test without NIRCam.
                                     Source: GAO analysis of NASA information.


                                     In addition to the instrument delays, two other technical challenges
                                     associated with ISIM are: (1) the detectors used by three of the four
                                     instruments to capture infrared light in space are degrading and may
                                     need to be replaced, resulting in the addition of another round of cryo-
                                     vacuum testing—in which a test chamber is used to simulate the near
                                     absolute zero temperatures in space, and (2) issues with the
                                     development of the cryo-cooler system that removes heat and cools MIRI.
                                     In December 2010 the project became aware that the detectors in three
                                     of the instruments were degrading. 17 As a result, approximately $42
                                     million and 15 months of schedule reserve to replace the detectors were
                                     included in the replan. These additions covered the cost of manufacturing
                                     the detectors; fabrication, assembly, and test of new focal plane
                                     assemblies; changing the detectors on three instruments, and the
                                     addition of a third ISIM cryo-vacuum test. The manufacturing process for
                                     new detectors takes approximately 30 months, which means that they
                                     cannot be delivered until after the second round of ISIM cryo-vacuum
                                     testing in 2014. As a result, $2 million of the $42 million in the replan was
                                     used to add a third round of cryo-vacuum testing for ISIM. The third test
                                     will validate the performance requirement of the ISIM and is the only time
                                     the instruments are tested with the flight detectors. Changing the
                                     detectors requires disassembling the instruments from ISIM, a process
                                     that will risk damage to the structure and instruments. Project officials
                                     stated that they will continue to monitor the degradation rate of the current
                                     detectors because if the degradation rate is low, they may not replace the
                                     detectors.



                                     17
                                      NIRCam, NIRSpec, and FGS/NIRISS use detectors fabricated by the same vendor;
                                     whereas MIRI uses different detectors that do not need to be replaced.




                                     Page 22                                            GAO-13-4 James Webb Space Telescope
Development issues with a part of the cryo-cooler needed for MIRI have
delayed its delivery to ISIM. In 2010, project officials realized that an
essential valve in the cryo-cooler was leaking at rates that exceeded
requirements. Following the results of a failure review board, the
contractor manufactured a newly designed valve, but it also did not meet
leak rate requirements. Project officials stated that a new valve design will
not be manufactured in time for use in the first ISIM cryo-vacuum test.
The project is concurrently developing three alternatives and authorized
manufacturing for one of the alternatives in October 2012. Project officials
stated that the MIRI cryo-cooler is particularly complex because it spans
approximately 10 meters—or approximately 33 feet—through the entire
JWST observatory. These issues combined required the use of 18
months of schedule reserve, which reduced ISIM’s schedule reserve from
the 26 months established in the replan to 8 months before it is needed
for integration with the OTIS subsystem.

These types of issues are not uncommon among NASA programs as
technical issues tend to arise when disparate parts are integrated and
tested together for the first time. Given the complexity and cutting edge
technology developed and used on JWST, it is expected that these kinds
of issues will continue to materialize as the program moves through its
complex integration and test program. Figure 7 shows the delay of
instrument deliveries as well as changes to the ISIM integration and test
and final delivery dates over the last year.




Page 23                                    GAO-13-4 James Webb Space Telescope
Figure 7: Changes in ISIM Schedule Since the Project Replan




Until the project is able to overcome the major issues with the instruments
and other parts of the ISIM, it is likely that the schedule would continue to
slip and may begin to affect the overall project schedule. ISIM still has 8
months of schedule reserve before the slipping of its schedule would
affect the schedule for the remainder of the project. The instrument,
detector, and cryo-cooler issues have all contributed to the delay in the
ISIM integration and test schedule and the reduction of objectives that
can be achieved in the first two rounds of cryo-vacuum testing. The first
round of testing will not include two instruments, a final design of the cryo-
cooler hardware, or new detectors. As a result project officials will only be
able to gather risk reduction information on the FGS/NIRISS, MIRI, test
procedures, and test support equipment from the first cryo-vacuum test.

The project also has several known long term risks and challenges
remaining. For example, risks related to OTIS, the sunshield, and the


Page 24                                      GAO-13-4 James Webb Space Telescope
                              ground system subsystems are not scheduled to be addressed until late
                              in project development. As of October 2012, seven of the top 10 project
                              risks were related to the long-term risks associated with the OTIS and
                              sunshield, most of which will not be resolved until 2016 or later. For
                              example, several risks relating to OTE are not scheduled to be closed
                              until the OTIS testing in the chamber at Johnson Space Center in
                              February 2017. Project officials are adding risk mitigation through early
                              and additional testing, where possible, to these subsystems. Prior to the
                              replan, the ground system software was at high risk for not being
                              completed before launch and many tasks were planned for completion
                              after launch. Space Telescope Science Institute officials stated that the
                              replan allows them to plan for completion of their work before launch on a
                              more realistic time schedule, which decreases schedule and operational
                              risk. A continuing challenge on the ground system is that some
                              development and testing is dependent on the final design of subsystems
                              such as the instruments, which continue to slip delivery dates.


Independent and               The project plans to hold independent and management reviews required
Management Oversight          for all projects during the integration and test phase, but this phase for
during Test and Integration   JWST is particularly complex. JWST has five major integration and test
                              efforts that span 7 years and only one independent mission-level
May Not Be                    technical review—the system integration review. The system integration
Commensurate with JWST        review evaluates the readiness of the project and associated supporting
Test and Integration          infrastructure to begin system assembly, integration, and test, and
Complexity                    evaluates whether the remaining project development can be completed
                              within available resources. For JWST, this review is scheduled in
                              September 2017, only 13 months prior to launch. Projects we reviewed
                              that had recently launched, however, held their system integration review
                              on average approximately 22 months prior to launch. The project has an
                              internal review with participation from standing review board members
                              planned before the beginning of OTIS integration and test activities begin,
                              and it will be subject to independent lower level reviews conducted by the
                              Goddard Systems Review Office of the integration and test process. In
                              addition, key decision point D (KDP-D)—when the senior agency decision
                              authority would approve the project to proceed into the system integration
                              and test phase—is scheduled for December 2017, 3 months after the
                              commencement of the final major integration and test activity. According
                              to NASA policy, this review should be held prior to the start of the system




                              Page 25                                   GAO-13-4 James Webb Space Telescope
                       integration and test phase of the project. 18 Project budget information
                       shows that over 90 percent of expected integration and test funding will
                       be spent on four major integration and test activities prior to the
                       scheduled mission-level system integration review and KDP-D approval
                       by NASA senior management. As a result, the current plan may be
                       inadequate to ensure that key technical and management issues are
                       identified early enough to be addressed within the current integration and
                       test phase schedule.


                       The JWST project has taken steps to improve communications and
JWST Project Has       oversight of its contractors as part of the replanning activities. For
Taken Steps to         example, based on recommendations from the ICRP, the project has
                       instituted meetings at various levels throughout NASA and its contractors
Enhance                and subcontractors. In addition, the project has added personnel at
Communications with    contractor facilities, which has allowed for more direct interaction and
                       quicker resolution of issues. The project also assumed responsibility of
and Oversight of Its   the mission-level systems engineering function from Northrop Grumman,
Contractors            a move that shifts the authority to make trades or decisions to NASA. An
                       independent NASA review of the project conducted in May 2012 found,
                       however, that agencywide reductions in travel budgets have put the
                       effectiveness of the JWST project’s oversight plans in jeopardy. While the
                       project received partial relief from travel budget reductions in fiscal year
                       2012, project officials are concerned that the current level of oversight will
                       not be sustained if similar cuts in travel funding occur in future years as
                       anticipated. The project is also taking steps to enhance its oversight of
                       project risks by implementing a new risk management system. The new
                       project manager found that the previous system lacked rigor and was
                       relatively ineffective for managing project risks, especially for a project as
                       complex as JWST. The new system should allow for better tracking of
                       risks than did the previous system. While these enhancements to the
                       oversight of the project are steps in the right direction, it will take time to
                       assess their effectiveness.




                       18
                        NPR 7120.5E, table 2-5 and appendix A (Aug. 14, 2012).




                       Page 26                                        GAO-13-4 James Webb Space Telescope
NASA Responded to                            Based on recommendations in the ICRP report, NASA has taken action to
Recent Independent                           enhance oversight and communications. See table 2 for the ICRP
Review Panel Report with                     recommendations and actions taken by NASA in response.
Increased Communication
and Oversight

Table 2: ICRP Recommendations Concerning Oversight and Communications and the Actions Taken by NASA

ICRP recommendation                          Action taken
Move the JWST management and                 Completed: The management of JWST at Headquarters has been reorganized. JWST is
accountability from the Astrophysics         now a stand-alone program reporting to the NASA Associate Administrator for
Division to a new organizational entity at   programmatic oversight and to the Associate Administrator for the Science Mission
NASA headquarters that has responsibility    Directorate for technical and analysis support. A similar approach was taken in the past
only for the management and execution of     with both the Hubble Space Telescope and the Mars Exploration Program.
JWST.
Assign management and execution              Completed: NASA has reorganized the JWST project at Goddard to report directly to the
responsibility for the JWST Project to the   Center Director, who is responsible for the management and execution of the JWST
Goddard Center Director, with                Project and accountable to the NASA Administrator.
accountability to the Science Mission
Directorate Associate Administrator at
NASA headquarters.
Assign at least one senior Goddard           In progress: In August 2012, the Deputy Observatory Manager began spending at least
project person to be resident at Northrop    half time at the Northrop Grumman facility, with a permanent move scheduled in early
Grumman throughout the Project.              2013.
Improve communications between the           Completed: The JWST project office provides weekly updates of risks and new issues to
JWST Project and both Goddard                the Goddard Center Director. In addition, the Project Manager at Goddard has daily
management and NASA headquarters             interaction with the Program Manager at NASA HQ.
Science Mission Directorate.
Conduct monthly or bi-monthly JWST           Completed: Meetings were first held in December 2010 between the NASA Associate
Project Executive meetings, attended by      Administrator, the new JWST Program Director, and the Goddard Center Director and
the NASA Associate Administrator and the     senior management of the prime contractor. Since that time, senior level management
President of Northrop Grumman                reviews have occurred on a quarterly basis and are open to subcontractors as well.
Aerospace Systems.                           Topics at these meeting have included issues such as the spacecraft mass problems,
                                             instrument delivery delays, and the process to find alternative solutions for the MIRI cryo-
                                             cooler valve problem.
Strengthen the role and the independent      Completed: The project added a Deputy Senior Project Scientist/Technical position to the
voice of the science team in the Project.    project science team. This individual:
                                             •    is responsible for day-to-day interactions with senior project management on all
                                                  aspects of the mission; scientific, technical, budgetary, and schedule.
                                             •    meets regularly with other members of the project science team to ensure better
                                                  communication with other functions in the project.
                                             •    assists the Senior Project Scientist to facilitate better integration of the science
                                                  activities with the hardware development activities and enable closer coordination
                                                  and understanding of technical drivers to science performance so fully informed
                                                  decisions can be made.
                                             Source: GAO analysis of NASA data.




                                             Page 27                                              GAO-13-4 James Webb Space Telescope
NASA has taken steps to increase communication between the project
and its contractors and subcontractors in an effort to enhance oversight.
According to project officials, the increased communication has allowed
them to better identify and manage project risks by having more visibility
into contractors’ activities. The project reports that a great deal of
communication existed across the project prior to the ICPR and replan;
however, improvements have been made. For example, monthly
meetings between project officials at Goddard and all of the contractors
have continued on a regular basis and include half-day sessions devoted
to business discussions. The project reports that these meetings have
benefits over other forms of communication. For example, it was through
dialogue with several technical leads at Northrop Grumman during
detailed reviews of analytical models that the project identified that the
mass issue on the spacecraft was likely to occur.

In addition, the project has increased its presence at contractor facilities
as necessary to provide assistance with issues. For example, the project
has had two engineers working on a recurring basis at Lockheed Martin
to assist in solving problems with the NIRCam instrument. The ISIM
manager said that these engineers have insight into Lockheed Martin’s
work and are having a positive effect as they offer technical help and are
involved in devising the solutions to issues. He added that that these
engineers have authority to make decisions on routine issues to allow the
work flow to continue, but decisions that are more complex or require a
commitment of funds are communicated to project management for
disposition. The project reports that the Jet Propulsion Laboratory,
responsible for NASA contribution to the MIRI instrument and its
associated cryo-cooler, has an in-house representative in the responsible
Northrop Grumman division to monitor the work being performed on the
cryo-cooler.

The JWST project also assumed full responsibility for the mission system
engineering functions from Northrop Grumman in March 2011. NASA and
Northrop Grumman officials both said that NASA is better suited to
perform these tasks. Project officials stated the systems engineering
requires the ability to make trades and decisions across the entire
observatory, and because Northrop Grumman is only responsible for
portions of the observatory, it did not have the authority to make trades or
decisions for areas outside of its control. Although responsibility for the
overall mission systems engineering function was removed from Northrop
Grumman, it retains system engineering responsibility for work still under
its contract, such as development of the spacecraft and sunshield. The
ICRP noted that a highly capable, experienced systems engineering


Page 28                                    GAO-13-4 James Webb Space Telescope
                           group is fundamental to project success and appropriate to ensure
                           accountability especially for a project of JWST’s complexity and visibility.

                           While these enhancements to the oversight of the project are steps in the
                           right direction, it will take time to assess their effectiveness. In addition,
                           sustainment of these efforts on the part of the project will be important.
                           Project and contractor officials we spoke with believe that the increased
                           communication has had a positive effect on the relationships between
                           them. We will continue to monitor the interaction between the project and
                           its contractors and its frequency in future reviews to identify whether the
                           changes have had the desired results.


Travel Budget Reductions   The JWST project reported that its travel budget was reduced by
May Hamper Planned         approximately $200,000 from the $1.2 million planned in fiscal year 2012
Oversight Activities       as a result of NASA’s implementation of an Executive Order to promote
                           more efficient spending. 19 According to project officials, the changes in
                           oversight necessitated by a reduction in travel funds represent a major
                           shift away from the management paradigm adopted during the replan.
                           Proposed reductions in future fiscal years could significantly reduce the
                           project’s travel budget. The project reports that the travel requirements for
                           fiscal years 2013 through 2015 are $1.6 million, $1.7 million, and $1.8
                           million, respectively. Officials reported that while travel is a small
                           percentage of the project’s annual budget, the majority of expected
                           travel—about 87 percent—is for oversight functions put in place as a
                           result of the ICRP recommendations, such as having a permanent on-site
                           presence at Northrop Grumman. These oversight functions include
                           attending and participating in contractor monthly programmatic and
                           technical reviews, technical interface meetings, recurring on-site
                           presence at contractor facilities for quality assurance reviews and
                           inspection of hardware. JWST project officials are concerned that
                           decreased oversight could translate into the project increasing its use of
                           cost and schedule reserves as they will not be conducting planned
                           oversight to better ensure success. A recent NASA Office of Evaluation
                           review concluded that by not having an adequate travel budget, the
                           project is at risk of cost/schedule growth and/or technical risk due to the


                           19
                             Executive Order 13589, Promoting Efficient Spending, The White House, November 9,
                           2011. NASA provided allocations to the Centers based on this order. In turn, Goddard
                           conducted an internal travel prioritization process and center management determined the
                           allocation for JWST and other Goddard projects.




                           Page 29                                          GAO-13-4 James Webb Space Telescope
                            late identification of issues or timely resolution strategies. The project has
                            made adjustments to absorb the reduction in fiscal year 2012 and plans
                            to identify instances of increased cost or schedule risk due to late
                            identification of issues. However, the project does not have a strategy to
                            address anticipated future reductions. Ensuring adequate oversight is
                            particularly important as the project begins its complex and lengthy test
                            and integration phase, where issues will likely surface.


NASA Headquarters           As part of NASA’s approach to increase oversight of the project at
Independent Review of the   headquarters, NASA’s Office of Evaluation recently conducted an
JWST Program Identified     independent review of the JWST project to assess the progress since the
                            September 2011 rebaseline was approved. According to the Director of
Concerns and                the Office of Evaluation, the goal of the review was not to reproduce the
Recommends Indicators to    replan assessment, but rather to assess progress based on cost,
Measure Progress            schedule, and technical performance of the project and the status of
                            oversight functions within NASA headquarters, the JWST Program Office,
                            and Goddard Space Flight Center. The intended outcome of the review
                            was 1) to obtain a snapshot of performance to determine if the program
                            was progressing in accordance with its plan, and 2) to identify leading
                            indicators for upper management to use when tracking future
                            performance. The review team identified several areas of concern within
                            the program, many of which we have highlighted, and recommended a list
                            of leading indicators that project management should consider tracking.

                            The Director of the Office of Evaluation said that the project is generally
                            performing the activities and maintaining the schedule set forth in the
                            replan; however, the team identified key areas that should be monitored
                            as the project moves forward. The review team also recommended a set
                            of leading indicators for project management to consider tracking to
                            measure and monitor progress. The Director added that these indicators
                            are for the project to use and would not be specific criteria for use by
                            independent review boards such as the Standing Review Board. These
                            indicators are a positive step to ensure that NASA management has the
                            information necessary to monitor the progress of the JWST project. See
                            table 3 below for the concerns raised by the review team.




                            Page 30                                     GAO-13-4 James Webb Space Telescope
Table 3: NASA Office of Evaluation Review Team Concerns

Concern identified                           NASA Office of Evaluation Review Team finding
                                      a
Resolving the on-going mass issue.           The technical and programmatic risks involved in possible solutions need to be explicitly
                                             understood.
Understanding root causes for the            There needs to be a better understanding of the root causes and investigate what would
NIRSpec Optical Bench cracking issue.        constitute a flyable structure.
Monitoring of schedule erosion by the        There are several competing critical paths for the JWST program to include development
project and recognition of the dynamics      and integration and test of the OTE, ISIM, sunshield, and the spacecraft.
                                  a
of four competing critical paths.
Monitoring the stability of the schedule     The master schedule took 8 months to baseline and stability of the schedule going forward
going forward.                               is unknown. The project will need an accurate reporting system at the integrated master
                                             schedule level.
Ensuring that Northrop Grumman will          The project needs to work with Northrop Grumman to ensure that it has an adequate
have an adequate workforce and that          workforce and skill sets available to accomplish the planned ramp up for fiscal year 2013.
                                         a
skills are available in fiscal year 2013.
Monitoring the increased risk of             Without an adequate travel budget to execute the original plan, there is an increased risk of
cost/schedule growth and technical risk.     cost or schedule growth and technical risks due to late identification of issues or timely
                                             resolution strategies.
                                              Source: GAO analysis of NASA data.
                                              a
                                              These issues were also identified as leading indicators the review team recommended that project
                                              management should consider tracking.



Changes Made to Project                       The new JWST project manager re-emphasized the importance of the
Risk Management System                        project’s risk management system and, in August 2012, a new risk
                                              management database was implemented to support the system. The
                                              project manager told us that he evaluated the risk management system
                                              being utilized by the project when he assumed his position and found it to
                                              be ineffective and not robust, especially for a project as complex as
                                              JWST. While the basic risk management methodology remains
                                              unchanged, the project manager wanted a more regimented system. For
                                              example, the project utilizes a hierarchy of risk boards that periodically
                                              reviews and provides disposition of all new and existing risks. These risk
                                              boards reviewed and assessed new risks and lower level risk board
                                              actions and met on an ad hoc basis. The project manager instituted a
                                              more regimented system that re-emphasized and revised the weekly
                                              project risk board meetings. Lower level risk boards meet a minimum of
                                              once a month depending on activity.

                                              The project manager also determined that a new risk management
                                              database needed to be put in place that would bring more rigor to the risk
                                              management process. The project manager told us that he directed an
                                              overhaul of the risk management database to provide more complete
                                              information to management on the purpose and history for each risk. The


                                              Page 31                                                GAO-13-4 James Webb Space Telescope
              goal was to improve consistency in how the project determined the
              potential for a risk to occur and its impact, and provide greater detail on
              mitigation and better tracking of the status for each risk. For example, the
              new system puts more emphasis on understanding and capturing the key
              events in the mitigation plan that are intended to result in a change in
              likelihood or consequence of a risk. The new system has a provision
              where the mitigation plan will be entered and updated over time, and the
              capability to store data such as mitigation steps throughout the life of the
              risk. In addition, the new system now archives data automatically to
              provide a traceable history of the risk. The prior data system did not have
              as robust of an archiving function. Furthermore, the project manager
              wanted to improve the linkage between the risk database entries and
              financial records to ensure consistency of the data in the risk database
              with regard to cost and schedule for risk mitigations with project office
              financial records. As the changes to the risk management system and
              database, as well as other changes we identified that were put in place to
              enhance oversight were just recently implemented, we will continue to
              monitor their continued use and assess the impact they may be having on
              the project.


              The JWST project is among the most challenging and high-risk projects
Conclusions   NASA has pursued in recent years. It is also one of the most expensive,
              with a recent major replan resulting in a total cost of $8.8 billion. The
              reasons for cost and schedule growth were largely recognized by an
              independent review team to be rooted in ineffective funding,
              management, communication, and oversight. NASA has invested
              considerable time and resources replanning the project and instituting
              management and oversight improvements in order to ensure that it (1)
              can be executed within its new estimates and (2) has addressed the
              majority of issues raised in the recent independent review. It appears that
              communications with contractors and within NASA have improved, that a
              more robust risk mitigation system is in place, that more is known about
              what it will take to complete the project and how much it will cost, and that
              the project is currently meeting the majority of its milestones.
              Nevertheless, over the course of the next several years, the project will
              be executing a large amount of work with several extremely complex and
              challenging integration and test efforts. Because three major test and
              integration efforts must be completed in the last 2 years of the JWST
              schedule, it is essential that issues are identified and addressed early
              enough to be handled within the project’s current schedule. While the
              JWST oversight plan is consistent with NASA’s requirements for all
              project’s required reviews, a single independent review scheduled just


              Page 32                                    GAO-13-4 James Webb Space Telescope
                      over a year before launch may not be sufficient to identify and resolve
                      problems early for a project of this magnitude. A key element of
                      overseeing project progress is monitoring how the project is executing to
                      its cost baseline. To that end, while NASA took some steps that were in
                      line with best practices to develop its revised baseline, some of the
                      deficiencies we found in its process could impact the reliability of the cost
                      estimate and the joint cost and schedule confidence level that was
                      provided to headquarters decision-makers. Without higher-fidelity,
                      regularly updated information related to costs, as well as an oversight
                      regime during later phases of test and integration that is commensurate
                      with the complexity of that effort, NASA risks late identification of
                      technical and cost issues that could delay the launch of JWST and
                      increase project costs beyond established baselines. Also important to
                      oversight for the remainder of the project is the ability of officials to
                      sustain improvements to communication with and oversight of
                      contractors. Anticipated travel restrictions, however, could decrease the
                      project team’s ability to sustain these actions. Without a plan to address
                      such reductions in future years, the project could once again become
                      susceptible to communication and oversight problems identified in earlier
                      reviews, which could also have a detrimental impact on continued project
                      performance.


                      To ensure that the JWST life-cycle cost estimate conforms to best
Recommendations for   practices, GAO recommends that the NASA Administrator direct JWST
Executive Action      officials to take the following three actions to provide high-fidelity cost
                      information for monitoring project progress:

                      •   improve cost estimate documentation and continually update it to
                          reflect earned value management actual costs and record any
                          reasons for variances,
                      •   conduct a sensitivity analysis on the number of staff working on the
                          program to determine how staff variations affect the cost estimate,
                          and
                      •   perform an updated integrated cost/schedule risk analysis, or joint
                          cost and schedule confidence level analysis, using a schedule that
                          meets best practices and includes enough detail so that risks can be
                          appropriately mapped to activities and costs; historical, analogous
                          data should be used to support the risk analysis.

                      To ensure that technical risks and challenges are being effectively
                      managed and that sufficient oversight is in place and can be sustained,




                      Page 33                                     GAO-13-4 James Webb Space Telescope
                     GAO recommends that the NASA Administrator direct JWST officials to
                     take the following three actions:

                     •   conduct a separate independent review prior to the beginning of the
                         OTIS and spacecraft integration and test efforts to allow the project’s
                         independent standing review board the opportunity to evaluate the
                         readiness of the project to move forward, given the lack of schedule
                         flexibility once these efforts are under way,
                     •   schedule the management review and approval to proceed to
                         integration and test (key decision point D or KDP-D) prior to the start
                         of observatory integration and test effort, and
                     •   devise an effective, long-term plan for project office oversight of its
                         contractors that takes into consideration the anticipated travel budget
                         reductions.

                     NASA provided written comments on a draft of this report. These
Agency Comments      comments are reprinted in appendix IV. NASA also provided technical
and Our Evaluation   comments, which were incorporated as appropriate.

                     In responding to a draft of this report, NASA concurred with three
                     recommendations and partially concurred with three other
                     recommendations and commented on actions in process or planned in
                     response. In some cases, these actions meet the intent and are
                     responsive to issues we raise; however, some of the responses do not
                     fully address the issues we raised in the report.

                     NASA partially concurred with our recommendation to improve the cost
                     estimate documentation of the JWST project, and to continually update it
                     to reflect earned value management actual costs and record any reasons
                     for variances between planned and actual costs. In response to this
                     recommendation, NASA officials stated that the project currently receives
                     earned value data from some of its contractors and performs monthly
                     analysis of that data to understand the contractors’ estimates at
                     completion, and then compares these numbers to similar figures
                     independently assessed by the JWST project. NASA also highlighted its
                     efforts to improve the agency’s documentation of the earned value
                     variances and to extend the earned value management analysis to areas
                     where it is not yet implemented, such as ground systems development at
                     the Space Telescope Science Institute. In addition, NASA responded that
                     its annual budget process generates a requirements-driven budget plan
                     consistent with the rebaseline. NASA stated that this information is
                     updated in several different documents that are provided to management



                     Page 34                                    GAO-13-4 James Webb Space Telescope
and it does not plan to revise its JCL documentation developed during the
replan. Despite these steps, we could not independently confirm that they
were leading to an updated cost estimate, which is the basis of our
recommendation. If the estimate is not updated, it will be difficult to
analyze changes in project costs and collecting cost and technical data to
support future estimates will be hindered. Furthermore, if not properly
updated on a regular basis, the cost estimate cannot provide decision
makers with accurate information for assessing alternative decisions.
Without a documented comparison between the current estimate
(updated with actual costs) and the old estimate, the cost estimator
cannot determine the level of variance between the two estimates and
cannot see how the project is changing over time. Therefore, we continue
to believe NASA will be well served by following best practices and
updating its cost estimate with current information and documenting
reasons for any variances. We encourage the project to improve the cost
estimate documentation and record any reasons for variances between
planned and actual costs and we intend to review the documentation as a
part of our ongoing review of the project.

NASA officials partially concurred with our recommendation that the
project conduct a sensitivity analysis on the number of staff working on
the project to determine how staff variations affect the cost estimate. In its
response, the agency stated that it believes it met the intent of this
recommendation when staffing levels were determined in the 2011 JWST
rebaseline based on programmatic experience from the accomplishment
of similar activities. To accommodate the possibility of increased costs
based on increased staffing hours, NASA reports that funded schedule
reserve was built into the JWST rebaseline, in addition to unallocated
future expenses being held at various levels of the organization. NASA
believes that these reserves will be sufficient to cover increases for the
duration of specific activities that result in increased staffing cost, and that
an additional workforce sensitivity analysis is not warranted. NASA added
that the joint cost and schedule confidence level analysis performed
provided a de facto workforce sensitivity analysis and does not plan any
further action. A joint cost and schedule confidence level analysis,
however, is not the same as a sensitivity analysis wherein the sources of
the workforce variation should be well documented and traceable. While
we appreciate the steps NASA took to account for workforce variation, the
JWST cost model does not show how staff levels increasing or
decreasing over time affects cost. Furthermore, best practices call for a
risk analysis to be conducted in conjunction with a sensitivity analysis, not
to be a substitute for it. As a best practice, a sensitivity analysis should be
included in all cost estimates because it examines the effects of changing


Page 35                                      GAO-13-4 James Webb Space Telescope
assumptions and ground rules. Since uncertainty cannot be avoided, it is
necessary to identify the cost elements that represent the most risk and, if
possible, cost estimators should quantify the risk. Without performing a
sensitivity analysis that reveals how the cost estimate is affected by a
change in a single assumption, such as workforce size, the cost estimator
will not fully understand which variable most affects the cost estimate.
Therefore, we continue to believe that NASA should conduct a sensitivity
analysis for the JWST project, given the large number of staff working on
the program, to determine how staff variations positively or negatively
affect the cost estimate rather than relying on schedule reserve and
unallocated future expenses to offset any shortfall.

NASA concurred with our recommendation to perform an updated
integrated cost and schedule risk analysis using a schedule that meets
best practices and includes enough detail so that risks can be
appropriately mapped to activities and costs. In response to this
recommendation, NASA stated that the agency is already using tools and
a method to conduct programmatic assessments of projects after the
baseline was established using the JCL methodology. While these may
be good tools, the key point is the need to address shortcomings of the
schedule that supports the baseline itself. For example, the lack of detail
in the summary schedule used for the joint cost and schedule risk
analysis prevented us from sufficiently understanding how risks were
incorporated; therefore, we question the results of that analysis. Since the
JCL was a key input to the decision process of approving the project’s
new cost and schedule baseline estimates, we maintain that the JWST
project should perform an updated JCL analysis using a schedule with
sufficient detail to map risks to activities and costs. Doing so could help
increase the reliability of the cost estimate and the confidence level of the
JCL. Furthermore, risk management is a continuous process that
constantly monitors a project’s health. Given that JWST is many years
from launch and the risks that the project faces are likely to change, a risk
analysis should be conducted periodically throughout the life of the
project.

NASA concurred with our recommendation to conduct a separate
independent review prior to the beginning of the OTIS and spacecraft
integration and test efforts. In response to this recommendation, NASA
stated that it will request members of the independent JWST Standing
Review Board participate in OTIS Pre-Environmental Review scheduled
prior to the beginning of OTIS environmental testing. A member of the
Standing Review Board will co-chair this review and report its findings to
the NASA Associate Administrator, which is the practice of all Standing


Page 36                                    GAO-13-4 James Webb Space Telescope
Review Board reviews. In addition, NASA plans to direct Northrop
Grumman, the spacecraft developer, to add members of the Standing
Review Board, as well as members of the Goddard Independent Review
Team, to the spacecraft element integration readiness review and report
their findings to the NASA Associate Administrator. We believe these
actions meet the intent of our recommendation and will afford an
independent evaluation of the readiness of the project to move forward
with its major integration and test efforts.

NASA partially concurred with our recommendation to schedule the
management review and approval to proceed to integration and test
(KDP-D) prior to the start of the observatory integration and test effort. In
response to this recommendation, NASA stated that it will reduce the 3-
month gap between the scheduled system integration review and the
KDP-D review, which it believes will provide NASA management and the
NASA Associate Administrator with the full independent assessment
earlier than currently planned. While we agree that this change will move
the review earlier than previously planned, based on its response, NASA
still plans to hold the review after the observatory integration and test is
already underway. Holding this review after the observatory integration
and test effort is already underway does not meet agency policy and will
lessen the impact of the review as it may be inadequate to ensure key
technical and management issues are identified early enough to be
addressed. KDP-D is the point in which management approval is given to
transition to the test and integration phase. We reiterate our
recommendation that NASA should hold this important key decision point
prior to the beginning of this last major integration and test effort, as
required by agency policy.

NASA concurred with our recommendation to devise an effective, long-
term plan for project office oversight of its contractors that takes into
consideration the anticipated travel budget reductions. In response to this
recommendation, NASA stated that it will develop a plan based on fiscal
year 2013 travel allocations and will take into consideration anticipated
travel budget reductions. In addition, NASA stated that the plan will
enable the project to maintain oversight of JWST contractors and their
ability to meet performance and delivery deadlines and work closely with
the international partners. We believe such a plan will be critical to
ensuring adequate oversight, which is particularly important as the project
enters into the complex integration and test efforts where issues will likely
surface. In addition, we agree with the concerns of project officials that
the current efforts to increase communication and oversight may not be
sustained if reductions to future travel budgets occur as anticipated. We


Page 37                                    GAO-13-4 James Webb Space Telescope
encourage the project to complete this plan in a timely manner and intend
to review it as a part of our ongoing assessment of the project’s oversight
efforts.


We will send copies of the report to NASA’s Administrator and interested
congressional committees. We will also make copies available to others
upon request. In addition, the report will be available at no charge on
GAO’s web-site at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this
report, please contact me at (202) 512-4841 or chaplainc@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who made
major contributions to this report are listed in appendix V.




Cristina T. Chaplain
Director
Acquisition and Sourcing Management




Page 38                                   GAO-13-4 James Webb Space Telescope
List of Committees

The Honorable Barbara A. Mikulski
Chairwoman
The Honorable Kay Bailey Hutchison
Ranking Member
Subcommittee on Commerce, Justice, Science, and Related Agencies
Committee on Appropriations
United States Senate

The Honorable Frank R. Wolf
Chairman
The Honorable Chaka Fattah
Ranking Member
Subcommittee on Commerce, Justice, Science, and Related Agencies
Committee on Appropriations
House of Representatives

The Honorable Ralph M. Hall
Chairman
The Honorable Eddie Bernice Johnson
Ranking Member
Committee on Science, Space, and Technology
House of Representatives




Page 39                              GAO-13-4 James Webb Space Telescope
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              Our objectives were to assess (1) the extent to which NASA’s revised
              cost and schedule estimates are reliable based on GAO best practices,
              (2) the major risks and technological challenges the James Webb Space
              Telescope (JWST) project faces, and (3) the extent to which the National
              Aeronautics and Space Administration (NASA) has improved the
              oversight of the JWST project. In assessing the project’s cost and
              schedule estimates, we performed various checks to determine that the
              provided data were reliable enough for our purposes. Where we
              discovered discrepancies, we clarified the data accordingly. Where
              applicable, we confirmed the accuracy of NASA-generated data with
              multiple sources within NASA.

              To assess the current life cycle cost estimate of the JWST project, we
              used the GAO cost guide to evaluate the estimating methodologies,
              assumptions, and results to determine whether the cost estimate was
              comprehensive, accurate, well-documented, and credible. 1 We developed
              standardized data collection instruments to request relevant cost and
              schedule documentation and questionnaires for project officials to
              complete. Through the data collection instruments and questionnaires, we
              gathered basic information pertaining to the project’s cost and schedule
              estimation process. We examined documents from the JWST replan
              efforts pertaining to the revised cost estimate and the joint cost and
              schedule confidence level (JCL) analysis, including detailed spreadsheets
              that contained cost, schedule and project risk information. Project
              documents we reviewed included basis of estimates for the replan from
              contractors and the project office, contractor engineering change
              proposals, the JWST program plan and mission requirements, as well as
              the NASA cost estimating handbook. 2 To assess how management and
              independent review teams were involved in the cost estimate process, we
              also reviewed the project’s monthly status presentations to Goddard
              management, Standing Review Board reports concerning the project’s
              revised baseline estimates, cost estimates conducted by NASA’s
              Independent Program Assessment Office, and the Agency Program
              Management Council’s decision memo for the rebaselined estimate and
              JCL analysis. We supplemented our analysis by interviewing officials from



              1
               GAO-09-3SP. For the cost guide, GAO cost experts assessed measures consistently
              applied by cost-estimating organizations throughout the federal government and industry
              and considered best-practices for the development of reliable cost-estimates.
              2
               NASA, 2008 NASA Cost Estimating Handbook, (2008).




              Page 40                                           GAO-13-4 James Webb Space Telescope
Appendix I: Objectives, Scope, and
Methodology




Northrop Grumman, the Space Telescope Science Institute, and the
JWST program and project offices. After reviewing cost estimate
documentation submitted by NASA and conducting numerous interviews
with relevant sources within the project office, we calculated the
assessment rating of each criteria within the four characteristics by
assigning each individual assessment rating: Not Met = 1, Minimally Met
= 2, Partially Met = 3, Substantially Met = 4, and Met = 5. We then took
the average of the individual assessment ratings for the criteria to
determine the overall rating for each of the four characteristics. The
resulting average becomes the “Overall Assessment” as follows: Not Met
= 1.0 to 1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4,
Substantially Met = 3.5 to 4.4, and Met = 4.5 to 5.0. We discussed the
results of our assessments with officials within the program office at
NASA headquarters and the project office at Goddard Space Flight
Center.

We supplemented the assessment of the revised 2011 cost estimate with
an assessment of the summary schedule used for the JCL, which was a
part of the project’s cost estimation process, and followed criteria laid out
in the GAO schedule guide. 3 These practices address whether the
schedule (1) captured all activities; (2) sequenced all activities—that is,
listed in the order in which they are to be carried out; (3) assigned
resources to all activities; (4) established the duration of all activities; (5)
integrated schedule activities horizontally and vertically, which identifies
whether products and outcomes associated with other sequenced
activities are arranged in the right order, and that varying levels of
activities and supporting subactivities are also aligned properly; (6)
established for all activities, the critical path, which is the longest
continuous sequence of activities that is necessary to examine the effects
of activities slipping in the schedule; (7) identified between activities float,
which is the amount of time by which a predecessor activity can slip
before the delay affects the program’s estimated finish date; (8) identified
a level of confidence using a schedule risk analysis; and (9) was updated
using logic and durations to determine dates. We also reviewed the inputs
to the JCL model, the document outlining the methodology of the analysis
that accompanied the electronic files, and interviewed cognizant project
officials to discuss their use of the summary schedule.




3
GAO-12-120G.




Page 41                                      GAO-13-4 James Webb Space Telescope
Appendix I: Objectives, Scope, and
Methodology




Because the project’s detailed integrated master schedule has not been
finalized because of ongoing negotiations and contract modifications, we
did not conduct a complete schedule analysis using the GAO schedule
assessment guide. We plan to perform this assessment in a subsequent
review of the JWST project.

To assess the major short- and long-term risks and technological
challenges facing the project, we reviewed the project’s risk list, monthly
status reviews, and other documentation provided by projects and
contractor officials. This information covered the risks, mitigation plans,
and timelines for addressing risk and technological challenges. We also
interviewed project officials for each major observatory subsystems to
clarify information and to obtain additional information on risks and
technological challenges. Further, we interviewed officials from the Jet
Propulsion Laboratory, Northrop Grumman Aerospace Systems,
Lockheed Martin Advanced Technology Company, Teledyne Imaging
Sensors, the University of Arizona, and the Space Telescope Science
Institute concerning risks and challenges on the subsystems, instruments,
or components they were developing. We reviewed GAO’s prior work on
NASA Large Scale Acquisitions, NASA Office of Inspector General
reports, and NASA’s Space Flight Program and Project Management
Requirements and Systems Engineering Processes and Requirements
policy documents. 4 We compared NASA’s controls as outlined in these
agency policies with the project plan to assess the extent to which the
JWST’s plan followed the intent of the policies with regard to independent
oversight and management approval processes.

To assess the extent to which NASA is performing enhanced oversight of
the JWST project, we reviewed documentation from the Independent
Comprehensive Review Panel and the project to determine actions taken
by NASA in response to the panel’s recommendations. We interviewed
project officials to understand the impact of these changes on the
oversight processes for the project and communication between the
project and its contractors. We also interviewed officials from the Jet
Propulsion Laboratory, Northrop Grumman Aerospace Systems,
Lockheed Martin Advanced Technology Company, Teledyne Imaging
Sensors, the University of Arizona, and the Space Telescope Science



4
 NPR 7120.5E (Aug. 14, 2012) and NASA Procedural Requirements 7123.1A, NASA
Systems Engineering Processes and Requirements with Change 1. (Nov.4, 2009).




Page 42                                      GAO-13-4 James Webb Space Telescope
Appendix I: Objectives, Scope, and
Methodology




Institute concerning project oversight of work they were performing and
the effectiveness of oversight changes. In addition, we reviewed a
presidential directive and Office of Management and Budget and project
documentation and interviewed project officials concerning the reductions
to travel budgets and their impact on project oversight activities. We
interviewed the Director of NASA’s Office of Evaluation about a recent
internal review of the JWST project and reviewed documentation from
that review. We also reviewed documentation and interviewed project
officials concerning the changes made to the project’s risk management
system.

Our work was performed primarily at NASA Headquarters in Washington,
D.C., and Goddard Space Flight Center in Greenbelt, Maryland. We also
visited Johnson Space Center in Houston, Texas, and the Jet Propulsion
Laboratory in Pasadena, California. In addition, we met with
representatives from Northrop Grumman Aerospace Systems, Lockheed
Martin Advanced Technology Company, Teledyne Imaging Sensors, the
University of Arizona, and the Space Telescope Science Institute.

We conducted this performance audit from February 2012 to December
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 43                                  GAO-13-4 James Webb Space Telescope
Appendix II: Subsystems of the JWST
                                        Appendix II: Subsystems of the JWST
                                        Observatory



Observatory

Figure 8: Subsystems of the JWST: Interactive Information




                                        Note: The hardware configuration created when the Optical Telescope Element and the Integrated
                                        Science Instrument Module are integrated, referred to as OTIS, is not considered a subsystem by
                                        NASA, but we categorize it as such for ease of discussion.




                                        Page 44                                                GAO-13-4 James Webb Space Telescope
Appendix III: Our Evaluation of JWST’s Cost
                                       Appendix III: Our Evaluation of JWST’s Cost
                                       Estimate Process



Estimate Process

                                       In determining that the National Aeronautics and Space Administration’s
                                       (NASA) processes for developing the James Webb Space Telescope
                                       (JWST) cost estimate do not fully comply with best practices, we
                                       evaluated the project’s cost estimation methods against our 2009 Cost
                                       Estimating and Assessment Guide. (See table 4.) We applied the
                                       following scale across the four categories of best practices:

                                       •    Not met: NASA provided no evidence that satisfies any portion of the
                                            criterion.
                                       •    Minimally met: NASA provided evidence that satisfies less than one-
                                            half of the criterion.
                                       •    Partially met: NASA provided evidence that satisfies about one-half of
                                            the criterion.
                                       •    Substantially met: NASA provided evidence that satisfies more than
                                            one-half of the criterion.
                                       •    Met: NASA provided complete evidence that satisfies the entire
                                            criterion.


Table 4: Summary Assessment of JWST Cost Estimate Compared to Best Practices

                  Overall                                                                                    Individual
Characteristic    assessment          Best practice                                                          assessment
Comprehensive     Substantially met   The cost estimate includes all life-cycle costs.                       Met
                                      The cost estimate completely defines the program, reflects the         Substantially met
                                      current schedule, and is technically reasonable.
                                      The cost estimate work breakdown structure is product oriented,        Substantially met
                                      traceable to the statement of work/objective, and at an appropriate
                                      level of detail to ensure that cost elements are neither omitted nor
                                      double counted.
                                      The estimate documents all cost-influencing ground rules and           Substantially met
                                      assumptions.
Well documented   Partially met       The documentation should capture the source data used, the             Partially met
                                      reliability of the data, and how the data were normalized.
                                      The documentation describes in sufficient detail the calculations      Minimally met
                                      performed and the estimating methodology used to derive each
                                      element’s cost.
                                      The documentation describes step-by-step how the estimate was          Minimally met
                                      developed so that a cost analyst unfamiliar with the program could
                                      understand what was done and replicate it.
                                      The documentation discusses the technical baseline description, and Substantially met
                                      the data in the baseline are consistent with the estimate.
                                      The documentation provides evidence that the cost estimate was         Substantially met
                                      reviewed and accepted by management.




                                       Page 45                                              GAO-13-4 James Webb Space Telescope
                                  Appendix III: Our Evaluation of JWST’s Cost
                                  Estimate Process




                 Overall                                                                                 Individual
Characteristic   assessment      Best practice                                                           assessment
Accurate         Partially met   The cost estimate results are unbiased, not overly conservative or      Partially met
                                 optimistic and based on an assessment of most likely costs.
                                 The estimate has been adjusted properly for inflation.                  Partially met
                                 The estimate contains few, if any, minor mistakes.                      Partially met
                                 The cost estimate is regularly updated to reflect significant changes   Partially met
                                 in the program so that it is always reflecting current status.
                                 Variances between planned and actual costs are documented,              Minimally met
                                 explained, and reviewed.
                                 The estimate is based on a historical record of cost estimating and     Partially met
                                 actual experiences from other comparable programs.
Credible         Partially met   The cost estimate includes a sensitivity analysis that identifies a     Minimally met
                                 range of possible costs based on varying major assumptions,
                                 parameters, and data inputs.
                                 A risk and uncertainty analysis was conducted that quantified the       Partially met
                                 imperfectly understood risks and identified the effects of changing
                                 key cost driver assumptions and factors.
                                 Major cost elements were cross-checked to see whether results           Minimally met
                                 were similar.
                                 An independent cost estimate was conducted by a group outside the Partially met
                                 acquiring organization to determine whether other estimating
                                 methods produce similar results.
                                  Source: GAO analysis of NASA data.




                                  Page 46                                               GAO-13-4 James Webb Space Telescope
Appendix IV: Comments from the National
              Appendix IV: Comments from the National
              Aeronautics and Space Administration



Aeronautics and Space Administration




              Page 47                                   GAO-13-4 James Webb Space Telescope
Appendix IV: Comments from the National
Aeronautics and Space Administration




Page 48                                   GAO-13-4 James Webb Space Telescope
Appendix IV: Comments from the National
Aeronautics and Space Administration




Page 49                                   GAO-13-4 James Webb Space Telescope
Appendix IV: Comments from the National
Aeronautics and Space Administration




Page 50                                   GAO-13-4 James Webb Space Telescope
Appendix V: GAO Contact and Staff
                  Appendix V: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov
GAO Contact
                  In addition to the contact named above, Shelby S. Oakley, Assistant
Staff             Director; Karen Richey, Assistant Director; Richard A. Cederholm; Laura
Acknowledgments   Greifner; Cheryl M. Harris; David Hulett; Jason Lee; Kenneth E. Patton;
                  Sylvia Schatz; Stacey Steele; Roxanna T. Sun; Jay Tallon; and Jade A.
                  Winfree made key contributions to this report.




(121048)
                  Page 51                                 GAO-13-4 James Webb Space Telescope
                      The Government Accountability Office, the audit, evaluation, and
GAO’s Mission         investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
                      Website: http://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in   E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.