oversight

NASA: Earned Value Management Implementation across Major Spaceflight Projects Is Uneven

Published by the Government Accountability Office on 2012-11-19.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Requesters




                NASA
November 2012




                Earned Value
                Management
                Implementation
                across Major
                Spaceflight Projects Is
                Uneven




GAO-13-22
                                                 November 2012

                                                 NASA
                                                 Earned Value Management Implementation across
                                                 Major Spaceflight Projects Is Uneven
Highlights of GAO-13-22, a report to
congressional requesters




Why GAO Did This Study                           What GAO Found
NASA historically has experienced cost           The National Aeronautics and Space Administration's (NASA) 10 major
growth and schedule slippage in its              spaceflight projects discussed in this report have not yet fully implemented
portfolio of major projects and has              earned value management (EVM). As a result, NASA is not taking full advantage
taken actions to improve in this area,           of opportunities to use an important tool that could help reduce acquisition risk.
including adopting the use of EVM.               GAO assessed the 10 projects against three fundamental EVM practices that,
EVM is a tool developed to help project          according to GAO’s best practices cost guide, are necessary for maintaining a
managers monitor risks. GAO was                  reliable EVM system. GAO found shortfalls in two of three fundamental practices.
asked to examine (1) the extent to               Specifically, we found that
which NASA is using EVM to manage
its major space flight acquisitions, (2)         •   More than half of the projects did not use an EVM system that was fully
the challenges that NASA has faced in                certified as compliant with the industry EVM standard.
implementing an effective EVM                    •   Only 4 of the 10 projects established formal surveillance reviews, which
system, and (3) NASA’s efforts to                    ensure that key data produced by the system was reliable. The remaining 6
improve its use of EVM. To address                   projects provided evidence of monthly EVM data reviews; however, the rigor
these questions, GAO obtained                        of both the formal and informal surveillance reviews is questionable given the
contractor and project EVM data and
                                                     numerous data anomalies GAO found.
used established formulas and tools to
analyze the data and assess NASA’s               GAO also found that 3 projects had reliable EVM data while 7 had only partially
implementation of EVM on 10 major                reliable data. For the EVM data to be considered reliable per best practices it
spaceflight projects; interviewed                must be complete and accurate with all data anomalies explained.
relevant NASA headquarters, center
and mission directorate officials on             NASA EVM focal points, headquarters officials, project representatives, and
their views on EVM; and reviewed prior           program executives cited cultural and other challenges as impediments to the
reports on EVM and organizational                effective use of EVM at the agency. Traditionally, NASA's culture has focused on
transformations. GAO compared NASA               managing science and engineering challenges and not on monitoring cost and
policies and guidance on EVM to best             schedule data, like an effective EVM system produces. As a result, several
practices contained in GAO’s cost                representatives said this information traditionally has not been valued across the
estimating best practices guide.                 agency. This sentiment was also echoed in a NASA study of EVM
                                                 implementation. Also cited as a challenge to the effective use of EVM was
What GAO Recommends                              NASA’s insufficient number of staff with the skills to analyze EVM data. Without a
GAO recommends that NASA                         sufficient number of staff with such skills, NASA’s ability to conduct a sound
establish a time frame for requiring             analysis of the EVM data is limited. However, NASA has not conducted an EVM
new spaceflight projects to implement            skills gap analysis to determine the extent of its workforce needs.
its new EVM system; conduct an EVM
skills gap assessment; develop a                 NASA has undertaken several initiatives aimed at improving the agency's use of
change management plan for EVM;                  EVM. For example, NASA strengthened its spaceflight management policy to
and strengthen its EVM requirements              reflect the industry EVM standard and has developed the processes and tools for
by requiring projects to implement               projects to meet these standards through its new EVM system. While these are
formal EVM surveillance. NASA                    positive steps, the revised policy contains only the minimum requirements for
concurred with two recommendations               earned value management. For example, it lacks a requirement for rigorous
and partially concurred with two others          surveillance of how projects are implementing EVM and also does not require
citing resource constraints. Despite             use of the agency’s newly developed EVM system to help meet the new
NASA's plans to address some issues              requirements. NASA has attempted to address EVM shortcomings through policy
GAO identified, not addressing all key           changes over the years, but these efforts have failed to adequately address the
issues lessens the usefulness of EVM             cultural resistance to implementing EVM.
at NASA.
View GAO-13-22. For more information,
contact Cristina Chaplain at (202) 512-4841 or
chaplainc@gao.gov.

                                                                                         United States Government Accountability Office
Contents


Letter                                                                               1
               Background                                                            4
               NASA Projects Have Not Consistently Implemented Key EVM
                 Practices and Most Did Not Have Access to Reliable EVM Data         8
               Cultural, Technical, and Other Challenges Seen as Impediments to
                 EVM Implementation                                                 18
               NASA Policy Is in Line with Best Practices but Implementation
                 Remains the Challenge                                              22
               Conclusions                                                          26
               Recommendations for Executive Action                                 27
               Agency Comments and Our Evaluation                                   28

Appendix I     Scope and Methodology                                                31



Appendix II    Importance of Earned Value Management                                35



Appendix III   Case Studies of Selected Projects’ Implementation of Earned
               Value Management                                                     39



Appendix IV    Comments from the National Aeronautics and Space
               Administration                                                     101



Appendix V     GAO Contact and Staff Acknowledgments                              104



Tables
               Table 1: EVM Standards, Regulations, and Requirements Applicable
                        to NASA                                                      5
               Table 2: Summary of the 10 NASA Spaceflight Projects Use of
                        Three Fundamental EVM Practices and Reliability of the
                        Data                                                         9
               Table 3: Examples of NASA EVM System Compliance with
                        ANSI/EIA Guidelines                                         24
               Table 4: NASA 10 Spaceflights Project EVM Data Summary               40


               Page i                                                    GAO-13-22 NASA
          Table 5: Assessment of GPM EVM Practices                            45
          Table 6: Assessment of JWST EVM Practices                           49
          Table 7: Assessment of Landsat OLI EVM Practices                    55
          Table 8: Assessment of LADEE EVM Practices                          58
          Table 9: Assessment of MMS EVM Practices                            62
          Table 10: Assessment of MAVEN EVM Practices                         69
          Table 11: Assessment of OCO-2 EVM Practices                         81
          Table 12: Assessment of RBSP EVM Practices                          87
          Table 13: Assessment of SOFIA EVM Practices                         91
          Table 14: Assessment of TDRS EVM Practices                          96


Figures
          Figure 1: Example of a Project Work Breakdown Structure             11
          Figure 2: GAO Earned Value Analysis of GMI-1 Data                   47
          Figure 3: GAO Earned Value Analysis of Observatory Data             51
          Figure 4: GAO Earned Value Analysis of NIRcam Data                  53
          Figure 5: GAO Earned Value Analysis of Operational Land Imager
                   Data                                                       57
          Figure 6: GAO Earned Value Analysis of LADEE Project Data           61
          Figure 7: GAO Earned Value Analysis of MMS Project Data             65
          Figure 8: GAO Earned Value Analysis of Instrument Suite Data        68
          Figure 9: GAO Earned Value Analysis of MAVEN Project Data           72
          Figure 10: GAO Earned Value Analysis of Spacecraft Data             75
          Figure 11: GAO Earned Value Analysis of Science Operations
                   Center Data                                                78
          Figure 12: GAO Earned Value Analysis of Remote Sensing
                   Instrument Data                                            79
          Figure 13: GAO Earned Value Analysis of Langmuir Probe and
                   Waves Data                                                 80
          Figure 14: GAO Earned Value Analysis of OCO-2 Project Data          84
          Figure 15: GAO Earned Value Analysis of the Spacecraft Data         86
          Figure 16: GAO Earned Value Analysis of Project Data                90
          Figure 17: GAO Earned Value Analysis of the SOFIA Project Data      94
          Figure 18: GAO Earned Value Analysis of the SOFIA Project Data      95
          Figure 19: GAO Earned Value Analysis of the TDRS Spacecraft
                   Project Data                                               98
          Figure 20: GAO Earned Value Analysis of the TDRS Sustainment
                   Data                                                      100




          Page ii                                                  GAO-13-22 NASA
Abbreviations

ANSI           American National Standards Institute
BAC            budget at completion
CPR            contract performance report
DCMA           Defense Contract Management Agency
EIA            Electronic Industries Alliance
EVM            earned value management
FAR            Federal Acquisition Regulation
GPM            Global Precipitation Measurement
GSFC           Goddard Space Flight Center
IBR            integrated baseline review
ICESat-2       Ice, Cloud, and Land Elevation Satellite2
IMS            integrated master schedule
JWST           James Webb Space Telescope
LADEE          Lunar Atmosphere and Dust Environment Explorer
LDCM           Landsat Data Continuity Mission
LOE            level of effort
MAVEN          Mars Atmosphere and Volatile EvolutioN Mission
MMS            Magnetospheric Multiscale
NASA           National Aeronautics and Space Administration
NID            NASA Interim Directive
NPR            NASA Procedural Requirements
OCO-2          Orbiting Carbon Observatory 2
OMB            Office of Management and Budget
RBSP           Radiation Belt Storm Probes
SOFIA          Stratospheric Observatory for Infrared Astronomy
TDRS           Tracking and Data Relay Satellite System
VAC            variance at completion
WBS            work breakdown structure



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page iii                                                                  GAO-13-22 NASA
United States Government Accountability Office
Washington, DC 20548




                                   November 19, 2012

                                   The Honorable Kay Bailey Hutchison
                                   Ranking Member
                                   Committee on Commerce, Science, and Transportation
                                   United States Senate

                                   The Honorable Ralph M. Hall
                                   Chairman
                                   The Honorable Eddie Bernice Johnson
                                   Ranking Member
                                   Committee on Science, Space, and Technology
                                   House of Representatives

                                   The Honorable Bill Nelson
                                   Chairman
                                   Subcommittee on Science and Space
                                   Committee on Commerce, Science, and Transportation
                                   United States Senate

                                   The National Aeronautics and Space Administration (NASA) has a
                                   portfolio of 21 major spaceflight projects that are expected to cost $19.4
                                   billion to develop. 1 Historically, NASA has experienced problems with
                                   persistent cost growth and schedule slippage in the majority of its major
                                   projects. As a result, NASA acquisition management has remained on
                                   GAO’s high risk list since it was first introduced in 1990. 2 Our work has
                                   shown that several factors contribute to NASA’s cost and schedule
                                   performance, including poor cost estimating and underestimating risks
                                   associated with the development of its major systems.

                                   Over the years, NASA has undertaken a number of actions to improve its
                                   acquisition management function and has shown some progress in
                                   improving its performance. One of these actions has been to adopt



                                   1
                                     GAO, NASA: Assessments of Selected Large-Scale Projects, GAO-12-207SP
                                   (Washington, D.C.: March 1, 2012). Each of these major spaceflight projects have a
                                   lifecycle cost of $250 million or more. Since we began our review, five of the 21 projects
                                   have launched, one was canceled, and two other projects have entered the portfolio.
                                   2
                                    GAO, High-Risk Series: An Update, GAO-11-278 (Washington, D.C.: Feb. 2011).




                                   Page 1                                                                     GAO-13-22 NASA
earned value management (EVM). EVM is a project management tool
developed by the Department of Defense in the 1960s to help project
managers monitor risks. EVM measures the value of work accomplished
in a given period and compares it with the planned value of work
scheduled for that period and the actual cost of work accomplished.
EVM’s intended purpose is to integrate a project’s cost, schedule, and
technical efforts for management and provide reliable data to decision
makers.

Starting in 1997, NASA began to require its projects to implement EVM
on all significant contracts. 3 In 2005, NASA broadened its application of
EVM to encompass significant project efforts implemented by in-house
civil service and associated support contractor personnel. NASA has had
uneven success in effectively implementing EVM, according to GAO and
NASA Inspector General reports conducted since 1999. 4 NASA has
recognized a need for improved EVM implementation, and has an
ongoing effort focused on doing so. In light of these issues, the House
Science, Space, and Technology Committee and the Senate Committee
on Commerce, Science, and Transportation asked GAO to examine the
use of EVM at NASA. Specifically, we assessed (1) the extent to which
NASA is using EVM to manage its major space flight acquisitions, (2) the
challenges that NASA faces in implementing an effective EVM system,
and (3) NASA’s efforts to improve its use of EVM.

To determine the extent to which NASA’s major spaceflight projects are
using EVM to manage the acquisition, we reviewed 10 major spaceflight
projects, each with a life cycle cost estimate of more than $250 million
that had been approved to proceed into the implementation phase of



3
 The NASA Inspector General report notes that NASA Policy Directive 9501.3, Earned
Value Performance Measurement, issued in February 1997, established the basis for
applying EVM to its contracts. NASA considered a contract significant if it was a
production contract with an estimated value of $250 million or more or an RDT&E contract
with an estimated value of $60 million or more with a period of performance greater than 1
year. In March 2006, this policy was canceled and was replaced by NASA Procedural
Requirements 7120.5.
4
 NASA IG-99-058, Earned Value Management at NASA, September 30, 1999; GAO,
NASA: Lack of Disciplined Cost Estimating Processes Hinders Effective Program
Management; GAO-04-642 (Washington, D.C., May 28, 2004); and GAO, Information
Technology: Agencies Need to Improve the Implementation and Use of Earned Value
Techniques to Help Manage Major System Acquisitions; GAO-10-2 (Washington, D.C.,
Oct. 8, 2009).




Page 2                                                                   GAO-13-22 NASA
development. Collectively, the projects we reviewed will cost over $14
billion to develop (See table 2 for a list of the projects). 5 We collected all
available EVM data for the 10 projects for the period of August 2010 to
August 2011 and used established earned value formulas and tools to
identify cost and schedule variances and trends. We also assessed the
projects’ implementation of three fundamental EVM practices that,
according to GAO’s Cost Estimating and Assessment Guide, are
necessary for maintaining a reliable EVM system—using a certified
system, conducting integrated baseline reviews, and performing
surveillance. 6 We analyzed monthly project management review briefings
to support our analysis. To assess the reliability of the cost data, we
electronically tested the data for anomalies, and reviewed relevant project
documentation and interviewed agency and project officials about the
data. To determine the challenges the agency faces in its efforts to
implement an effective EVM system, we developed standard sets of
questions and interviewed NASA headquarters officials, program
executives, and EVM focal points at each center and the Human
Exploration and Operations and Science Mission Directorates to discuss
their roles as well as the extent to which EVM data is used to inform
decision making. We also reviewed prior GAO and relevant NASA
Inspector General reports on EVM. Also, we submitted written questions
to the project offices to obtain their views on implementing EVM on their
projects. To determine the steps that NASA is taking to improve its use of
EVM, we compared NASA policies and guidance intended to improve the
agency’s implementation of EVM to best practices for earned value
management as documented in GAO’s cost guide. We also interviewed
agency officials responsible for developing and implementing the NASA’s
new system to discuss ongoing initiatives. For additional details on our
scope and methodology, see appendix I.




5
 NASA's life cycle for flight systems is defined by two phases—formulation and
implementation. The implementation phase, preceded by the formulation phase, is defined
as the execution of approved plans for the development and operation of the program or
project, and the use of control systems to ensure performance to approved plans and
requirements and continued alignment with the agency’s strategic goals. NASA
Procedural Requirements 7120.5E, NASA Space Flight Program and Project
Management Requirements, figure 2-4 and paragraph 1.3.1(c) (Aug. 14,
2012).(Hereinafter cited as NPR 7120.5E (Aug. 14, 2012.))
6
 GAO, GAO Cost Estimating and Assessment Guide, GAO-09-3SP (Washington, D.C.:
Mar. 2009).




Page 3                                                                GAO-13-22 NASA
             We conducted this performance audit from July 2011 through November
             2012 in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objectives. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objectives.


             EVM is a project management tool that, when properly used, can provide
Background   accurate assessments of project progress, produce early warning signs of
             impending schedule delays and cost overruns, and provide unbiased
             estimates of anticipated costs at completion. Pulling together essential
             cost, schedule, and technical information in a meaningful, coherent
             fashion is a challenge for most projects. Without such information,
             managers can have a distorted view of a project’s status and risks. EVM
             also allows individuals outside the project to see a standardized metric
             describing the cost and schedule performance of that particular project
             and compare it consistently with other projects.

             EVM measures the value of work accomplished in a given period and
             compares it with the planned value of work scheduled for that period and
             with the actual cost of work accomplished. Differences in these values are
             measured in both cost and schedule variances. Positive variances
             indicate that activities are costing less or are completed ahead of
             schedule. Negative variances indicate activities are costing more or are
             falling behind schedule. For example, if a contractor completed $5 million
             worth of work and the work actually cost $6.7 million, there would be a
             $1.7 million negative cost variance. Schedule variances are also
             measured in dollars, but they compare the earned value of the work
             completed with the value of work that was expected to be completed. For
             example, if a contractor completed $5 million worth of work at the end of
             the month but was budgeted to complete $10 million worth of work, there
             would be a $5 million negative schedule variance.

             Earned value provides information that is necessary for understanding the
             health of a project and an objective view of project status. Cost and
             schedule variances can also be used in estimating the cost and time
             needed to complete the project. While some data that an EVM system
             produces are retrospective and indicate performance to date, EVM data
             can also be used to predict future performance. For example, estimates
             at completion for a project can be calculated by using efficiency indices



             Page 4                                                      GAO-13-22 NASA
                                         which are based on a project’s past cost and schedule performance. See
                                         appendix II for additional information on the importance of EVM.

                                         A project’s EVM data comes from multiple sources. For example, each
                                         contractor that supports a project will produce and deliver EVM reports to
                                         the project for the work they and their subcontractors perform, if the
                                         contract so requires. In addition, the project will collect EVM information
                                         for the work that it performs in-house at a NASA center. All of this lower
                                         level EVM data can then be consolidated at the project level to produce a
                                         project level EVM report. Pulling together EVM data from multiple levels
                                         into a project level report gives the project a comprehensive outlook of its
                                         cost and schedule, and provides the project manager with early warning
                                         of potential cost and schedule overruns.

                                         EVM has evolved from an industrial engineering tool to a government and
                                         industry best practice, providing improved information to conduct
                                         oversight of acquisition programs. As such, it is guided by industry best
                                         practices and standard, and is required by regulations and requirements
                                         at the federal government and the agency level at NASA. These
                                         requirements and standards are summarized in table 1 below.

Table 1: EVM Standards, Regulations, and Requirements Applicable to NASA

American National Standards       •   Regarded as the national standard and an industry best practice for EVM systems.
Institute/Electronic Industries   •   Describes 32 guidelines that a certified EVM system must meet in the areas of organization;
Alliance (ANSI/EIA)-748               planning, scheduling, and budgeting; accounting; analysis and management reports; and
                                      revisions and data maintenance.
Office of Management and Budget   •   EVM is to be applied for parts of a major acquisition with developmental effort and is to be
(OMB) Circular A-11 and Capital       used for both in-house government and contractor work.
Programming Guide                 •   EVM system must be in compliance with the guidelines in ANSI/EIA-748.
                                  •   When there is both government and contractor work, the data from the two EVM systems
                                      must be consolidated at the reporting level for total program management and visibility.
Federal Acquisition Regulation    •   EVM system required for major acquisitions in development.
(FAR), Subpart 34.2               •   Contractors and subcontractors are required to report EVM on a monthly basis.
                                  •   When an EVM system is required, an integrated baseline review is required to verify technical
                                      content and realism of related performance budgets, resources, and schedules.
NASA FAR Supplement, Subpart      •   EVM system is required on acquisitions for development or production work for flight and
1834.2                                ground support systems and components, prototypes, and institutional investments, such as
                                      facilities and IT infrastructure.
                                  •   Certified ANSI/EIA-compliant EVM system (as determined by cognizant Federal agency) is
                                      required for cost or fixed-price incentive contracts and subcontracts valued at $50 million or
                                      more.
                                  •   ANSI/EIA-compliant EVM system (as determined by the contracting officer) required for cost
                                      or fixed-price incentive contracts and subcontracts valued at $20 million or more but less than
                                      $50 million.




                                         Page 5                                                                     GAO-13-22 NASA
NASA Procedural Requirements   •   NPR 7120.5E strengthens the previous version by requiring projects to perform EVM with a
(NPR) 7120.5                       system that complies with the guidelines in ANSI/EIA-748. This directive establishes the
                                   requirements by which NASA will formulate and implement spaceflight programs and projects
                                   including flowing down EVM system requirements to applicable suppliers.a
                               •   Requires an integrated review of project baselines as part of their preparations for the
                                   confirmation review to ensure that the project’s work is properly linked with its cost, schedule,
                                   and risk and that the systems are in place to conduct EVM.
                                      Source: GAO analysis of standards, regulations, and requirements.
                                      a
                                       For some of its projects, international partners contribute elements of a mission. According to NASA,
                                      the agreements reached with its international partners do not require the international partners to
                                      share EVM data for their contribution to a project. When NASA collaborates with an international
                                      partner, it is done on a no exchange of funds basis. Each partner commits to providing certain
                                      elements of the overall mission, such as an instrument, spacecraft bus, or launch vehicle, and are
                                      responsible for delivering the element(s) at the appropriate time. Each partner determines how they
                                      want to manage their development efforts and assumes all costs for their work. Partners are not
                                      required to share any information on how much their efforts cost.


                                      Almost two decades of NASA’s past efforts to improve its use of earned
                                      value management have had uneven success. An EVM Focal Point
                                      Council was created in 1996 and focal points were designated at each
                                      NASA center and the Office of Procurement and the Office of the Chief
                                      Financial Officer to provide an open forum to share experiences and
                                      develop a network of support within the NASA EVM community. In 1997,
                                      the agency issued NASA Policy Directive 9501.3 “Earned Value
                                      Management Performance” which established the basis for applying EVM
                                      to NASA contracts. Prior to this policy, centers used their own individual
                                      policies on performance measurement systems. However, in 1999, the
                                      NASA Inspector General reported that EVM policy was not consolidated
                                      as an overall program management responsibility and that the Agency
                                      Program Management Council did not receive comprehensive EVM
                                      information. 7 As a result, in 2003, NASA shifted responsibility for the EVM
                                      policy from the Office of the Chief Financial Officer to the Office of the
                                      Chief Engineer to emphasize EVM as a project management tool, rather
                                      than a financial management tool.

                                      In 2004, GAO reported that only 2 of 10 NASA projects reviewed used
                                      EVM consistently and appropriately. 8 Several barriers to EVM
                                      implementation were identified, such as lack of reliable financial data,


                                      7
                                       The Program Management Council has the responsibility for periodically evaluating the
                                      technical, safety, and programmatic performance (including cost, schedule, and risk) and
                                      content of a program or project under their purview. These evaluations focus on whether
                                      the program or project is meeting its commitments to the agency.
                                      8
                                          GAO-04-642.




                                      Page 6                                                                             GAO-13-22 NASA
trained EVM staff, data analysis tools, and incentives. Among other
things, GAO recommended that NASA take action to ensure that a true
EVM system is used as an organizational management tool to bring cost
to the forefront in NASA’s decision-making process and that acquisition
and EVM management policies and procedures be enforced. In response
to our recommendations, NASA stated that it was updating NASA
Procedural Requirements 7120.5, its program and project management
processes and requirements policy, to improve its cost estimating and
ensure that its cost estimate and earned value analyses were effectively
used, and the updated policy was issued in 2005. Similar to other
agencies in our 2009 report on governmentwide use of EVM, we reported
on weaknesses in NASA’s EVM policies and practices and recommended
that the agency modify its policies governing EVM to ensure that they are
consistent with best practices. 9 In particular, we found problems with the
EVM training requirements for personnel responsible for investment
oversight and management responsibilities. In addition, NASA’s policy for
revising project cost and schedule baselines did not have adequately
defined criteria on the acceptable reasons for permitting a rebaselining. 10
We also found weaknesses in how the NASA projects we reviewed
implemented EVM and managed their negative performance trends.
NASA acknowledged the identified weaknesses and stated that it was
revising its NASA procedural requirements for programs and projects to
include expanded and strengthened policies governing EVM application
and processes, and revised policies for rebaselining projects.

In January 2010, the Agency Program Management Council approved the
funding for the EVM capability project to develop an EVM system that
complies with the guidelines in ANSI/EIA-748. Another goal of the project
was to determine whether it was feasible to implement a single EVM
system that would integrate the scope, schedule, and budget of EVM data
for NASA’s in-house managed efforts and contractor data across the



9
 GAO-10-2. As part of the audit we selected NASA projects that rely on various elements
of information technology and reviewed both the agency’s spaceflight and information
technology specific guidance.
10
  A rebaseline is when cost, resources, and/or schedule commitments are revised. NASA
projects are rebaselined when their estimated development cost exceeds NASA’s
baseline commitment development cost by 30 percent or more and Congress has
reauthorized the project; events external to NASA make a rebaseline appropriate; or a
NASA Associate Administrator determines that the project’s scope changed from the
approved project baseline.




Page 7                                                                 GAO-13-22 NASA
                     agency. According to agency officials, NASA invested about $2 million
                     into the capability project to pilot the EVM system through two projects:
                     the Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) and the
                     Constellation Extra-Vehicular Activity project. 11 Through the pilots, the
                     capability project demonstrated that an agency-wide EVM system was
                     feasible. As a result, a finalized set of processes, tools, guidance, and
                     training products that compose NASA’s new EVM system was developed.
                     This new system was peer reviewed and approved by a panel of EVM
                     experts.

                     Our assessment of 10 major spaceflight projects showed that NASA has
NASA Projects Have   not yet fully implemented EVM and thus is not taking full advantage of an
Not Consistently     important tool that could help reduce acquisition risk. GAO found that the
                     projects had shortfalls in two of the three fundamental practices that we
Implemented Key      assessed. Specifically, we found that half of the projects did not use an
EVM Practices and    EVM system that was certified as compliant with the ANSI/EIA-748
                     standard. Most of the projects conducted an integrated baseline review
Most Did Not Have    (IBR), a practice that ensures the performance measurement baseline
Access to Reliable   reflects all requirements and that resources are adequate to complete the
                     work. Specifically, 9 projects conducted an IBR for their overall efforts;
EVM Data             however, the Stratospheric Observatory for Infrared Astronomy (SOFIA)
                     project office only conducted an IBR of one of its major contractors. In
                     addition, we found that only 4 of the 10 projects had established formal
                     independent surveillance reviews to ensure that key elements of the EVM
                     process were maintained over time so that the data produced by the
                     system provided timely indications of actual or potential problems. For the
                     6 projects that did not have formal independent surveillance in place,
                     each provided evidence that they instituted monthly EVM data reviews,
                     which according to project officials, helps them to continually monitor cost
                     and schedule performance. However, the rigor of both the formal and
                     informal surveillance reviews is questionable given the numerous EVM
                     data anomalies we found in the monthly EVM reports. Specifically, we
                     found many unexplained anomalies, such as the presence of negative
                     numbers or missing data, which caused us to question the reliability of the
                     data. Out of the 10 projects we reviewed, we found that just 3 projects
                     had reliable EVM data while the remaining 7 had only partially reliable
                     data.


                     11
                       The second EVM pilot on the Constellation Extra-Vehicular Activity Project was not
                     completed due to the termination of the Constellation program.




                     Page 8                                                                  GAO-13-22 NASA
                                          Overall, the projects are using EVM, but NASA has not consistently
                                          implemented EVM across these projects. For example, we found that
                                          several projects were not implementing EVM at the project level, which is
                                          considered a best practice. Table 2 summarizes the performance of each
                                          of the 10 projects in meeting the three fundamental EVM practices and
                                          the reliability of the data.

Table 2: Summary of the 10 NASA Spaceflight Projects Use of Three Fundamental EVM Practices and Reliability of the Data

                                    Used A certified EVM            Conducted an              EVM System              Data resulting from
                                    system compliant with           integrated baseline       surveillance            the EVM system
Projects                            ANSI/EIA standard               review                    is being performed      are reliable
Global Precipitation Measurement
                                                                                                                              ◐
James Webb Space Telescope
                                                                                                                              ◐
Landsat Data Continuity Mission
                                                                                                                              ◐
Lunar Atmosphere and Dust
Environment Explorer                                                                                 ◐                         ◐
Magnetospheric Multiscale
                                                                                                     ◐                         ◐
Mars Atmosphere and Volatile
Evolution Mission                              ◐                                                      ◐                         
                                a
Orbiting Carbon Observatory 2
                                                                                                     ◐                         
Radiation Belt Storm Probes
                                                                                                     ◐                         
Stratospheric Observatory for
Infrared Astronomy Project                     ◐                                  ◐                    ◐                         ◐
Tracking and Data Relay Satellite
Replenishment                                                                                                                 ◐
                                          Source: GAO analysis of NASA and contractor data.

                                          Notes: For detailed information on the EVM performance of the projects selected for our case studies,
                                          see appendix III.
                                          a
                                           At the time of our review, the Orbital Sciences Corporation did not have a certified EVM system that
                                          complied with the ANSI/EIA-748 standard; however, the Defense Contract Management Agency
                                          certified their EVM system in January 2012.

                                          Key:       The agency met this criterion


                                                    ◐    The agency partially met this criterion


                                                     The agency did not meet this criterion




                                          Page 9                                                                           GAO-13-22 NASA
Few Projects Had EVM                      Of the 10 projects we reviewed, 4 projects had a certified EVM system, 3
Systems Compliant with                    did not, and 3 had a mixture in which some contractors and
the Industry Standard                     subcontractors had certified systems and some did not. When an EVM
                                          system is certified, the agency has assurance that the implemented
                                          system was validated for compliance with the ANSI/EIA-748 standard by
 Criteria
 •    Project obtained EVM data from
                                          independent and qualified staff and therefore can be considered to
      certified system(s).                provide reliable and valid data from which to manage a project. The
 •    Project shows a consistent work     Global Precipitation Measurement (GPM), Tracking and Data Relay
      breakdown structure between the     Satellite System (TDRS), Landsat Data Continuity Mission (LDCM), and
      EVM report and the schedule.
                                          James Webb Space Telescope (JWST) were the only projects that
 •    Project schedule identifies
      significant task                    provided evidence that the contract performance reports provided came
      interdependencies.                  from EVM systems that were certified as compliant with the ANSI/EIA-748
 •    Project uses a time-phased          standard. The Lunar Atmosphere and Dust Environment Explorer
      budget baseline for tracking cost   (LADEE), Magnetospheric Multiscale (MMS) and Radiation Belt Storm
      and schedule variances.
                                          Probes (RBSP) projects did not have EVM systems that were certified to
                                          be compliant with the ASNI/EIA-748 standard. Finally, the Jet Propulsion
                                          Laboratory, a federally funded research and development center that the
                                          California Institute of Technology manages under a contract with NASA,
                                          was the only NASA Center with a certified EVM system. The Jet
                                          Propulsion Laboratory is responsible for managing the Orbiting Carbon
                                          Observatory 2 (OCO-2) project. The Mars Atmosphere and Volatile
                                          Evolution Mission (MAVEN) and SOFIA prime contractors also had
                                          certified systems; however, their project offices did not. NASA does not
                                          require a certified EVM system for their in-house work.

                                          Using the project’s integrated master schedule and contract performance
                                          reports, we assessed the EVM data provided by the projects against
                                          selected fundamental ANSI/EIA guidelines to determine the extent to
                                          which each project’s EVM system, whether certified or not, was meeting
                                          them. The guidelines we reviewed included whether the work breakdown
                                          structure (WBS)—which provides the basis of the project schedule—was
                                          consistent between the EVM report and the schedule, whether the
                                          schedule identified significant task interdependencies, and whether the
                                          project had identified a time-phased budget baseline for tracking cost and
                                          schedule variances. As shown in figure 1, a work breakdown structure
                                          breaks down product-oriented elements into a hierarchical structure that
                                          shows how elements relate to one another as well as to the overall end
                                          product. By subdividing a project into smaller elements, management can
                                          more easily plan and schedule the program’s activities and assign
                                          responsibility for the work.




                                          Page 10                                                     GAO-13-22 NASA
Figure 1: Example of a Project Work Breakdown Structure




We found that even for the projects that had certified systems, there were
problems with consistency between the WBS and the EVM report and the
schedule. For example, we found discrepancies in the hierarchical



Page 11                                                     GAO-13-22 NASA
structure and numbering of WBS elements for JWST, an $8.8 billion
project. Specifically, the project’s WBS dictionary showed mission
assurance efforts numbered differently than contractor reports for two
contractors, each of which had mission assurance labeled with different
WBS numbers. 12 NASA officials explained that neither the spacecraft nor
the near infrared camera contractor was required to follow the project-
level WBS structure or numbering scheme. NASA officials explained that
while it is not a requirement for the project and contractor WBSs to be the
same, it is recommended that the prime contractor lower-level WBS
numbering scheme be consistent with the overall project WBS numbering
format. Doing so allows easier total project integration of cost and EVM
data for project reporting. Consistency of the WBS element between the
cost estimate and the schedule facilitates updating the cost estimate with
actual costs and ensures there is correlation between the cost estimate
and schedule.

Our review of the project schedules also revealed that about half of the
schedules were missing predecessor and/or successor dependencies
and had constraints that prevented the schedule from responding
properly to updates. Since the schedule is the foundation for the EVM
baseline, it must be properly sequenced. This means knowing how one
activity (the predecessor) affects another (the successor) and how each
affects the critical path. 13 When the schedule is not sequenced correctly,
the reliability of the EVM data is called into question. Our review found
that the MMS project was missing dependencies for 31 percent of its
remaining activities for its instrument suite contract and 36 percent of the
remaining activities for the instrument suite were constrained. 14 Due to


12
  EVM data are typically provided to NASA in the form of a standard report called the
contract performance report. This report is the primary source of cost and schedule status
and provides the information needed for effective program control. The contract
performance report provides cost and schedule variances, based on actual performance
against the plan, which can be further examined to understand the causes of any
differences.
13
  The critical path method is used to derive the critical activities—that is, activities that
cannot be delayed without delaying the end date of the program. The amount of time an
activity can slip before the program’s end date is affected is known as “total float.”
14
  Constraints are generally used to demonstrate an external event’s effect on the
schedule. However, because they prevent activities from responding dynamically to
network logic, including actual progress and availability of resources, they can affect float
calculations and the identification or continuity of the critical path and can mask actual
progress or delays in the schedule. Date constraints should be minimized because they
restrict the movement of activities and can cause false dates in a schedule.




Page 12                                                                      GAO-13-22 NASA
the major sequencing issues in the MMS instrument schedule, we
questioned the reliability of the overall network and the schedule’s ability
to correctly calculate float values and the critical path. MMS project
officials said they believed many of the constrained activities we found
were not valid because they reside in another schedule. In addition,
officials said some of the constraints found are in the Harness area, and if
removed, these constraints would have no effect on the overall schedule.
Furthermore, MMS officials said some of the sequencing issues may be
caused by manual integration because some instrument provider
schedules are in Microsoft Project and others are in Primavera, and
therefore it is not possible to ensure all tasks have been linked properly.
As part of their schedule health check process, the MMS project
scheduler tests the schedule for missing dependencies, logic errors, and
reasonable durations and the results are shared with the project office
and the contractor so that appropriate action can be taken. However,
when we removed the 15 level of effort type activities from the missing
dependencies count, the schedule still showed 28 percent of the
remaining activities missing dependencies. We also removed the 14 level
of effort type activities and 4 Harness from the constraint count, the
schedule still showed 33 percent of the remaining activities were
constrained. MMS program officials said that they have a process in place
to manage float and the critical path, however, the schedule we reviewed
still showed significant sequencing issues.

Finally, we found that 4 of the 14 the schedules we analyzed were not
resource loaded. This means that the schedule did not have the required
labor, materials, equipment, and other resources assigned to the
appropriate activities. When the schedule is not resource loaded, then
costs need to be spread over time using some other method that may not
be as straightforward as having the costs integrated directly within the
schedule. Having a resource-loaded schedule is a best practice for
developing the time phased budget baseline. The time-phased budget
baseline represents the plan that performance is measured against for
the life of a project. It takes into account that program activities occur in a
sequenced order, based on finite resources, with budgets representing
those resources spread over time. Deviations from the baseline identify
areas where management should focus their attention.




Page 13                                                         GAO-13-22 NASA
Majority of Projects                      In keeping with best practices, 9 of the 10 projects conducted integrated
Conducted an Integrated                   baseline reviews. An IBR is an evaluation of the performance
Baseline Review                           measurement baseline—the foundation for an EVM system—to
                                          determine whether all project requirements have been addressed, risks
                                          have been identified, mitigation plans are in place, and available and
 Criteria                                 planned resources are sufficient to complete the work. Conducting an IBR
 •    Project performed an IBR to
      verify that the performance
                                          increases confidence that the performance measurement baseline
      measurement baseline is realistic   provides reliable cost and schedule data for managing the project and
      and to ensure that the contractor   that it projects accurate estimated costs at completion. OMB has
      and government mutually             endorsed the IBR as a critical process for risk management on major
      understand the potential project
      risks.                              investments and requires agencies to conduct IBRs for all contracts that
                                          require EVM. Since an IBR’s goal is to verify that the technical baseline’s
                                          budget and schedule are adequate for performing the work, it offers many
                                          key benefits such as laying a solid foundation for successfully executing
                                          the project and enabling better understanding of the risks.

                                          Officials for the SOFIA project did not conduct an integrated baseline
                                          review at the project level; however, its prime contractor for the
                                          engineering and modification of the airborne observatory platform did
                                          conduct an integrated baseline review. According to project officials, the
                                          lack of a project-level IBR is related to the EVM system being
                                          implemented “on the fly” late in the development phase for SOFIA, as a
                                          result of an audit recommendation in 2010. 15 However, project officials
                                          noted that the EVM baseline was established concurrently with an agency
                                          approved re-plan and Joint Cost and Schedule Confidence Level analysis
                                          in 2009 and 2010 and was reviewed by a Standing Review Board as part
                                          of that process. 16




                                          15
                                             NASA Inspector General, Final Memorandum on Audit of the Stratospheric Observatory
                                          for Infrared Astronomy (SOFIA) Program Management Effectiveness, IG-09-013,
                                          (Washington, D.C., Mar. 27, 2009).
                                          16
                                            The Joint Cost and Schedule Confidence Level is a cost estimation tool that involves a
                                          probabilistic analysis of cost, schedule, and risk inputs to arrive at development cost and
                                          schedule estimates associated with various confidence levels.




                                          Page 14                                                                    GAO-13-22 NASA
Majority of Projects Do                     Four of the 10 projects we assessed had a comprehensive EVM
Not Have a Comprehensive                    surveillance system in place. Of the remaining 6 projects, 1 had formal
Surveillance System in                      surveillance at the project level but its contractor did not, and 2 projects
                                            did not have a formal surveillance system at the project level, only their
Place                                       prime contractors did. The remaining 3 projects did not have any formal
                                            surveillance, but provided evidence that EVM data, such as cost and
 Criteria                                   schedule variances, were being reviewed during their monthly status
 •    Project verifies that EVM             reviews. Beyond reviewing cost and schedule variances and variances at
      processes and procedures              completion, formal surveillance reviews ensure that the processes and
      continue to satisfy the ANSI/EIA      procedures continue to satisfy the ANSI/EIA EVM guidelines. A formal
      EVM guidelines.
 •    Independent surveillance
                                            surveillance plan involves establishing an independent surveillance
      organization conducts reviews to      organization with members who have practical experience using EVM.
      ensure the integrity of the           This organization then conducts periodic surveillance reviews to ensure
      organization’s EVM system.            the integrity of the contractor’s EVM system and where necessary
                                            discusses corrective actions to mitigate risks and manage cost and
                                            schedule performance. Effective surveillance ensures that the key
                                            elements of the EVM process are maintained over time and on
                                            subsequent applications.

                                            NASA delegates surveillance of contractor EVM systems to the Defense
                                            Contract Management Agency (DCMA); however, NASA has no entity to
                                            perform independent surveillance reviews to ensure that the ANSI/EIA-
                                            748 standard is being met for EVM efforts performed in-house or by
                                            nonprofit organizations. Without an independent surveillance function, an
                                            organization’s ability to use EVM as intended may be hampered since
                                            surveillance monitors problems with the performance measurement
                                            baseline and EVM data. If the kinds of problems that formal surveillance
                                            can identify go undetected, EVM data may be distorted and may not be
                                            meaningful for decision making.

Unreliable EVM Data Limit                   Only 3 of the 10 projects we reviewed, MAVEN, RBSP, and OCO-2,
NASA’s Ability to Measure                   produced fully reliable data for managing the project and reporting status.
Project Performance                         The other projects only partially met the criterion to have an EVM system
                                            in place that produces reliable data. If done correctly, EVM data can
 Criteria                                   provide an objective means for measuring project status and forecasting
 •    Project EVM system(s) produces        potential project cost overruns and schedule slippages so that timely
      reports that are complete and         action can be taken to minimize their impact. To do so, however, the
      accurate.
 •    All reported data anomalies are       underlying EVM data must be reliable, meaning that they are complete
      explained.                            and accurate and all data anomalies are explained. In our analysis, we
 •    EVM data can be mapped                found multiple cases of data anomalies that caused us to question the
      through various levels to             reliability of the EVM data. For example, we found several EVM reports
      precisely identify root cause(s) of
      issues.
                                            where a contractor reported that no work was planned or accomplished,



                                            Page 15                                                       GAO-13-22 NASA
but actual costs were incurred without an explanation in the variance
analysis report to say why this happened. Additionally, we found cases
where a contractor reported that work was planned and actual costs were
incurred, but a negative amount of work was performed—work that was
previously reported as completed was now reported as not completed.
Further, we also found several instances where a project reported an
estimate at completion but no budget at completion. 17 Finally, we found
instances of negative values in the EVM reports. When explanations were
provided in the variance analysis reports, the reasons for these anomalies
included use of estimated, rather than actual costs, or adjustments from
prior periods due to mistakes or errors with the underlying EVM systems.
For example, the SOFIA project said that many of the negative values in
its EVM reports were due to over-reporting of earlier progress,
mischarges by employees, delayed cost postings, inappropriate use of
charge codes. When there are data anomalies such as those we
identified for SOFIA, the EVM data can become skewed and can distort
true performance.

In some cases, the projects provided no explanations for the anomalies
we found. In other cases, the projects said that the size of the anomalies
did not breach the variance reporting threshold and therefore no
explanation was required. 18 However, whether the variance breaches the
threshold or not, cost and schedule deviations from the baseline plan give
management information about where corrective action plans are needed
to bring the project back on track or to update completion dates. If
programs only address those issues that breach the thresholds,
management may be losing valuable insight into project risks and causes.
EVM data should be valid and free from unexplained anomalies (e.g.,
missing or negative values) because without reliable performance data,
project management’s ability to identify potential problems and intervene
early to mitigate the problems can be limited. Therefore, anomalies



17
   An estimate at completion is an independent assessment of the cost to complete
authorized work based on the project’s historical EVM performance. Budget at completion
is the sum of all estimated budgets, representing at the project level the cumulative value
of the budgeted cost of the work scheduled over the life of the project or at lower levels,
such as a control account or work breakdown structure element.
18
  Variance thresholds try to quantify an acceptable range of deviation; those that do not
exceed a threshold are usually not a concern while those that do are worthy of further
inspection to determine the best course of action to minimize any negative impacts to the
cost and schedule objectives.




Page 16                                                                   GAO-13-22 NASA
should be minimized and the reason for each should be fully explained in
the monthly EVM variance analysis reports. To do less limits the
completeness and accuracy of the EVM data and makes the resulting
variance determinations unreliable. While an industry standard for what
constitutes an acceptable volume of anomalies does not exist, EVM
experts in the public and private sectors stated that the occurrence of
EVM data anomalies should be rare.

For four projects that provided subcontractor EVM data, we tried to map
EVM data at the project level to lower level EVM data at the subcontractor
level. However, we were only able to successfully map the data for one of
the projects. This mapping allows project managers to track cost and
schedule by defined deliverables to more precisely identify which
components are causing cost or schedule overruns and to more
effectively mitigate the root cause of the overruns. When the reports do
not allow for traceability, project managers are not able to effectively
measure progress, use the reports to monitor and control costs based on
the original baseline, and/or to track where and why there were
differences. For example, when we attempted to map the EVM data in the
lower level reports for MAVEN’s spacecraft, science operations center,
remote sensing instrument, and Langmuir Probes and Waves instrument
efforts to the overall MAVEN project EVM report, we were not able to see
how the costs tracked from one report to another and therefore could not
reconcile the costs between the reports. Such issues raise the question of
which reports contain the true costs for these efforts. However, MAVEN
officials walked us through the process they use to ensure that lower-level
reports map to the project level reports. Furthermore, MAVEN officials
said they do not mandate that their contractors follow a certain reporting
format, instead any adjustments necessary to ensure that the lower-level
reports map to the project-level reports are made by the project office.
Though the MAVEN project does not prescribe to a standard reporting
format, attempting to manually resolve incompatible pieces of data can
become time-consuming, expensive and can lead to data reliability
issues. We also had similar difficulty mapping various levels of EVM data
for the MMS project. The MMS project was able to demonstrate how the
Southwest Research Institute (SwRI) budget at completion in the lower-
level report mapped to the SwRI budget at completion in the MMS project
report, but because of the way the contractor submits their data, project
officials said that the two reports will never match. Although the project
was able to explain how the data tracked, again, attempting to manually
resolve incompatible pieces of data can become time-consuming and can
lead to data reliability issues. Since our review, MMS officials said the



Page 17                                                     GAO-13-22 NASA
                                         project is working to capture data at lower WBS levels which will allow for
                                         a closer tie between the cost and schedule data.


Projects Have Not                        All the projects we reviewed were using some EVM to manage their work.
Consistently Applied EVM                 However, the extent to which EVM is implemented across NASA’s in-
                                         house projects and their contractors varies by project and center. For
                                         example, 3 of the 10 projects we reviewed did not report project level
Criteria                                 EVM data. 19 Implementing EVM at the project level rather than just for the
•    Project uses an integrated EVM      contract is considered a best practice. In addition, OMB policy requires
     system across the entire project
     to measure cost, schedule, and      the use of an EVM system for both in-house and contractor work and
     performance goals.                  when there is both government and contractor work, the data from the
•    Project uses the 32 guidelines in   two EVM systems must be consolidated at the reporting level for total
     ANSI/EIA-748.                       program management and visibility. Integrating government and
                                         contractor cost, schedule, and performance status at the project level
                                         should result in better project execution through more effective
                                         management. In addition, some of the in-house projects we analyzed
                                         were only required to meet EVM principles and gather “EVM-like” data.
                                         Further, contractors such as the nonprofit organization managing the
                                         MMS instrument suite, do not report its EVM in the standard contract
                                         performance report format because they are only required to meet the
                                         intent of the standard. Other contractors for the JWST and MAVEN
                                         projects are required to provide EVM reports that show cost and schedule
                                         data by WBS elements.


                                         NASA EVM focal points, headquarters officials, project representatives,
Cultural, Technical,                     and program executives cited cultural and technical challenges, as well
and Other Challenges                     as other challenges, as impediments to the effective use of EVM at the
                                         agency. NASA’s culture traditionally has focused on solving science and
Seen as Impediments                      engineering challenges and not on monitoring cost and schedule data
to EVM                                   such as data produced by an effective EVM system. Technical challenges
                                         were also cited as an impediment to effective EVM use, but opinions
Implementation                           differed within NASA on the extent of their impact. The technical
                                         challenges cited involved difficulty in gathering sufficiently detailed data
                                         for timely inclusion and analysis in an EVM system. In addition, though
                                         NASA has not conducted an EVM skills gap analysis, NASA


                                         19
                                           GPM collects EVM data at the project level, however, this data was provided to GAO in
                                         a format we could not analyze. We included GPM in the 7 projects that had project level
                                         data.




                                         Page 18                                                                GAO-13-22 NASA
                           representatives said it is a challenge for the agency to implement EVM
                           effectively due to a lack of sufficient staff with the skills and experience to
                           analyze EVM data.


NASA Culture Seen as Not   Almost half of the more than 30 NASA representatives we interviewed,
Valuing EVM                including a large number of those charged with implementing EVM
                           agencywide—known as EVM focal points—and officials from the Chief
                           Engineer’s office, cited NASA’s culture as a challenge to the effective use
                           of EVM at the agency. Specifically, several NASA representatives said
                           that historically, NASA’s culture has not focused on or valued the kind of
                           information that EVM can highlight. For example, a NASA EVM analyst
                           said the culture has been focused on science and engineering and that
                           accomplishment of that work has been the first priority for managers.
                           Discussion of the cost of the work has been a secondary concern. A
                           NASA EVM focal point described overcoming a culture that is based on
                           applied and basic research rather than discrete tasks with discrete
                           deliverables, which are among the requirements for effective EVM
                           implementation. Further, a senior official at NASA headquarters told us
                           that in project reviews, discussion of cost and schedule information, like
                           EVM data and analysis, tends to be pushed to the very end of the review
                           meeting and generally is not discussed in detail. Because EVM data is
                           not universally valued within NASA, in some cases, the data generated to
                           satisfy a project’s EVM requirement may be of limited use. For example,
                           one NASA EVM focal point said some managers are just “checking the
                           box” with respect to using EVM. The task is performed but the
                           requirement to collect EVM data was viewed as a nuisance that ultimately
                           did not provide worthwhile information. Nonetheless, several of those we
                           interviewed said persistent inquiries about EVM data from senior
                           management at headquarters, especially over the last couple of years,
                           are having a positive impact on the culture and forcing projects to pay
                           more attention to the data.


Technical and Other        The NASA representatives we interviewed also cited technical challenges
Challenges Cited as        as having an impact on the effective use of EVM, although their views
Making EVM Use More        varied on the extent of these challenges. For example, about half of the
                           focal points we interviewed reported that a challenge to using an EVM
Difficult                  system at NASA was aligning it with the agency’s accounting system, the
                           SAP Core Financial System. One of the problems cited was that EVM
                           data collection may require more detailed data than a project has
                           collected for the agency’s accounting system and this may require the
                           use of estimated costs instead of actual costs. Estimating these costs can


                           Page 19                                                         GAO-13-22 NASA
create additional work for the project, delay the production of EVM data,
and limit the reliability of the EVM data that is produced. Nonetheless, a
NASA manager said, projects could more effectively plan their work to
better accommodate the accounting system. For example, NASA’s
accounting system is set up to measure and report on labor in terms of
full-time equivalents. A project, however, may have set up its earned
value management system with a different measure for labor, such as
productive hours. As a result, the accounting system cannot fill in the
proper numbers for an earned value analysis, potentially causing more
work for the project. Further, the EVM data could be less accurate due to
the use of estimates rather the actual figures. However, if the project had
planned from the outset to have the same measure of labor as the
accounting system, there would not be a problem having this data fit the
EVM system.

An Office of the Chief Financial Officer representative did not believe that
the projects have consistently demonstrated that the accounting system is
a problem, but nonetheless agreed that potential work-arounds and slight
changes to processes are potential solutions for these issues. The Office
of the Chief Financial Officer has started an initiative to address both the
level of detail of the data and improve the monitoring of contractor cost
performance at levels that may be lower than levels at which obligations
are made and costs are reported in the financial system. 20 A report on a
NASA EVM pilot project noted that the greatest impediment for
implementing EVM is cultural resistance, not technical challenges.
Specifically, it noted that “it’s not the EVM Process. It’s not the EVM
Tools. It’s not the SAP Accounting System. It is the NASA culture.”

NASA’s use of contracts with one entity to provide goods or services to
several different NASA projects was also cited as a challenge to use of
EVM. For example, NASA centers may have a contract with one firm to
provide engineering support services. Multiple projects may seek services
using a single task order on this contract. Because of the way the NASA
accounting system is configured, this approach can create artificial
variances when looking at EVM data on a month-to-month basis.
According to a NASA representative, contractual requirements can
correct these issues and allow for a closer accounting of the funds for


20
  As part of the initiative, the Office of the Chief Financial Officer piloted an approach for
collecting cost at the level necessary to manage the work performed to support the work
elements for effective project EVM.




Page 20                                                                       GAO-13-22 NASA
                       EVM purposes and in fact, one NASA center has already instituted such
                       requirements in a new contract for services. NASA plans to address this
                       issue as current service contracts expire, but it will take time for the new
                       data requirements that would provide the desired data to be implemented.

                       Other challenges cited include the difficulty of gathering sound EVM data
                       from nonprofit subcontractors, such as universities. One project, for
                       example, reported that EVM data from nonprofit subcontractors were
                       immature or non-existent. The nonprofits may be doing a significant
                       amount of work for a center, but are not equipped to collect earned value
                       data at the level of detail needed, Office of the Chief Financial Officer and
                       center representatives reported. EVM focal points said the problem of
                       collecting EVM data from nonprofits to feed into a larger project-level
                       EVM system could be mitigated through contract language that clearly
                       specifies what data are required from contractors. According to these
                       officials, this kind of language has been included in Jet Propulsion
                       Laboratory contracts and has been successfully demonstrated as a result.
                       Furthermore, NASA is concerned that if nonprofits and small businesses
                       are required to have a fully compliant or certified EVM system, they may
                       not be able to bid on the work. However, the Federal Acquisition
                       Regulation is clear that no offeror can be eliminated from consideration
                       for a contract award because the entity does not have a compliant or
                       certified EVM system. 21


Understanding of EVM   NASA representatives we interviewed said there was a need for improved
Varies Widely Across   abilities across the agency to analyze EVM data and implement EVM
NASA                   systems. Specifically, several focal points said the challenge for NASA is
                       not as much in obtaining EVM data because most of the information
                       comes from private contractors responsible for much of NASA’s work, but
                       in having a staff that can analyze the data and integrate it at the project
                       level. A senior NASA official also noted that the career civil servants, who
                       typically are the first level of review for EVM data, do not have
                       background or training in EVM and cannot conduct a sound analysis of
                       the data. A project representative echoed this comment and noted a
                       general awareness of EVM within projects but a shortage of in-depth
                       knowledge to understand EVM fundamentals and how to interpret the
                       data it produces. For example, some projects seek to reset EVM



                       21
                        FAR § 34.201(b).




                       Page 21                                                       GAO-13-22 NASA
                    baselines to match funding allocations, which thwarts efforts to examine
                    cost and schedule trends. One EVM focal point told us it has been difficult
                    to convince project managers that EVM can predict what will happen in
                    their projects given the highly technical nature of their work. For example,
                    a senior manager of a program that experienced significant schedule
                    delays and cost overruns stated that he is an “EVM skeptic” and that he
                    does not see EVM data as helpful in helping him track the performance of
                    a project.

                    Additionally, the employee skill sets available to analyze and implement
                    EVM vary widely from center to center, headquarters officials said. In
                    recent years, NASA has provided EVM training to a large number of
                    employees; however, the agency has not conducted a skills gaps
                    analysis, which could help to determine the number and extent of the
                    staff’s EVM expertise. NASA centers may have staff skill levels reflective
                    of the level of EVM use at the center. Some centers have many projects
                    producing EVM data while others may only rarely work on a project that
                    uses EVM. Without a sufficient number of trained staff to analyze
                    contractor data and implement in-house EVM efforts, NASA will likely
                    continue to struggle to effectively use EVM as a valuable project
                    management tool.


                    NASA has undertaken several initiatives aimed at improving the agency’s
NASA Policy Is in   use of EVM. For example, NASA strengthened its spaceflight
Line with Best      management policy to require projects to comply with the 32 ANSI/EIA-
                    748 guidelines and has developed the processes and tools for projects to
Practices but       meet this requirement through its new EVM system. While these are
Implementation      positive steps, the policy continues to lack a requirement for rigorous
                    oversight or surveillance of how projects are implementing EVM and
Remains the         NASA does not require projects to use the new EVM system to implement
Challenge           the EVM requirement of the revised policy. In addition, the issues that
                    have impeded NASA’s ability to effectively implement EVM, such as its
                    culture, are longstanding and, as a result, NASA has not had much
                    success implementing EVM in the past.

                    The agency’s recent revision of NASA Procedural Requirements
                    7120.5—the policy that governs NASA’s spaceflight projects and contains
                    project EVM requirements—strengthened the EVM requirements over
                    prior versions of the policy. For example, the revised policy requires all




                    Page 22                                                      GAO-13-22 NASA
spaceflight projects to demonstrate compliance with each of the 32 EVM
guidelines as set forth in ANSI/EIA-748, whereas the prior policy only
required projects to comply with seven high-level EVM principles. 22 The
new requirements took effect through the release of an interim directive
on September 28, 2011 and have since been made final in NASA’s most
recent update to 7120.5. As a result, projects meeting EVM reporting
thresholds that enter implementation after that date are required to
comply with the new requirement. According to an agency official, the
Office of the Chief Engineer and the mission directorates will determine
which projects that began development under the prior policy must
comply with the new EVM requirements. At major milestones, Office of
the Chief Engineer representatives will review whether the projects have
implemented the 32 EVM guidelines.

However, the new policy still only contains the minimum requirements for
earned value management, such as the thresholds for implementing EVM
and the requirement to comply with ANSI/EIA-748 guidelines. 23 The policy
does not require projects to implement formal independent surveillance of
their EVM systems. Without effective surveillance, agencies cannot
ensure they are meeting the ANSI/EIA-748 guidelines because internal
management systems are not being reviewed to determine if they are
providing reliable cost, schedule and technical performance data. In
addition, effective surveillance helps pinpoint problems, and is useful for
verifying the effectiveness of corrective action plans used to mitigate EVM
system deficiencies. While projects are not required to implement formal
independent surveillance, NASA does plan to conduct periodic
surveillance of project EVM systems. For example, NASA plans to
conduct EVM assessments at two key decision point life cycle reviews




22
  The seven EVM principles as defined by ANSI/EIA-748 entail planning and integrating
work scope, actual costs, and schedule into a performance management baseline against
which accomplishments may be measured and assessed and the resulting information
used to inform management’s decision-making.
23
  Additional, detailed, guidance is provided in an expanded NASA EVM Implementation
Handbook. For example, the handbook describes roles and responsibilities of key project
officials with respect to EVM, direction on how to implement EVM on the agency’s in-
house developed projects, and guidance on how EVM surveillance is to be conducted, as
well as other important considerations. Projects are not required to follow the guidance in
the handbook.




Page 23                                                                   GAO-13-22 NASA
                                              and through the Office of the Chief Engineer compliance surveys. 24 While
                                              these methods will increase the agency’s surveillance efforts, best
                                              practices call for project level surveillance to be an ongoing, continuous
                                              process conducted by an independent surveillance function.

                                              The policy also does not require projects to use NASA’s new EVM
                                              system, although the system was designed to help projects meet the
                                              ANSI/EIA-748 guidelines. We found that the system meets the intent of
                                              the ANSI/EIA-748 guidelines. Examples of how NASA’s EVM system will
                                              satisfy three key ANSI/EIA-748 guidelines are summarized in table 3
                                              below.

Table 3: Examples of NASA EVM System Compliance with ANSI/EIA Guidelines

ANSI/EIA guideline                                            NASA EVM system description
1. Define the authorized work elements for the                •  Requires a work breakdown structure and outlines how the work will be
program. A work breakdown structure, tailored for                structured and controlled.
effective internal management control, is commonly
used in this process.
6. Schedule the authorized work in a manner that              •      Outlines how work will be scheduled including the methodology used to
describes the sequence of work and identifies                        create the schedule.
significant task interdependencies required to meet the       •      Explains how the project will document activity and milestone sequences
requirements of the program.                                         through the assignment of predecessor and successor
                                                                     interdependencies within detailed Control Account schedules.
                                                              •      Establishes a process for updating and making changes.
8. Establish and maintain a time-phased budget                •      Outlines the Performance Measurement Baseline, which is the time-
baseline, at the control account level, against which                phased budget expressed in dollars with separate identification of cost
program performance can be measured. Initial budgets                 elements. It applies to the entire scope of authorized work and spans the
established for performance measurement will be                      project’s period of performance.
based on either internal management goals or the              •      Outlines the process and steps for creating a Performance Measurement
external customer negotiated target cost, including                  Baseline that is time-phased via integration with the network schedule at
estimates for authorized but undefinitized work. Budget              the Work Package or Planning Package level. The Performance
for far-term efforts may be held in higher level                     Measurement Baseline consists of the aggregation of individual time-
accounts until an appropriate time for allocation at the             phased Control Account budgets, including all burdens, for current
control account level. If an over-target baseline is used            project plans.
for performance measurement reporting, prior
                                                              •      Includes how the control accounts should reflect the budget for the work
notification must be provided to the customer.
                                                                     planned and outlines who is responsible for completing the tasks.
                                              Source: GAO analysis of draft NASA EVM system description.




                                              24
                                                These surveys review center processes and infrastructure for compliance with Office of
                                              the Chief Engineer requirements, policy, procedures, processes, statutes, and regulations.
                                              A sample of projects is selected across NASA’s centers every 2 years for review. The
                                              results are used to give centers feedback on areas where NASA policy and requirements
                                              should be modified and to respond to OMB and GAO concerns over the implementation of
                                              requirements.




                                              Page 24                                                                        GAO-13-22 NASA
                           For the projects required to comply with the new policy, use of the
                           agency-developed EVM system would meet the ANSI/EIA guidelines;
                           however, some projects will be permitted to continue using their individual
                           EVM systems as long as the 32 guidelines are met. According to agency
                           officials, while future revisions to the policy may require use of the
                           standardized agency-developed EVM system by all projects, at this time,
                           the agency does not plan to require projects to use the agency-developed
                           system in order to meet the guidelines. Instead, senior managers will
                           determine on a case by case basis whether a project will use the
                           agency’s new EVM system. Currently, only the Space Launch System
                           and ICESat-2 projects have been selected to implement the new EVM
                           system. According to an agency senior official, the Agency Program
                           Management Council approved a phased rollout of the new system
                           because NASA does not have the resources to implement it agency-wide.
                           For example, there are not enough NASA subject matter experts to
                           provide the support needed by the projects when applying the new EVM
                           system and to build the institutional capability at the centers. Their
                           approach aims to incrementally build the capacity to do EVM, and seek
                           increased acceptance of EVM as the requirement for its use is expanded.


Strong Leadership Needed   Over the years, NASA has attempted to address its EVM shortcomings
to Fully Implement EVM     through a series of policy changes, but these efforts have failed to
                           adequately address the cultural resistance to implementing EVM
                           highlighted by many of the NASA officials we interviewed. NASA has
                           made uneven progress since we reported in 2004 that the agency needed
                           to improve its use of EVM as a project management tool. Furthermore, a
                           2008 NASA internal study noted that projects needed to be educated on
                           the value and approaches for using EVM and to provide support for
                           setting up EVM within the projects early. Also, an internal agency briefing
                           on EVM stated that a change management initiative would be necessary
                           in order to successfully implement EVM at NASA centers. Our work has
                           also shown that implementing a large-scale initiative, such as EVM,
                           requires more than just policy changes. 25 To see real change and, in
                           effect, a cultural shift at NASA, top leadership must provide to employees
                           a succinct and compelling reason that effective implementation of EVM is
                           important. Articulating a compelling reason for implementing EVM



                           25
                            GAO, Results-Oriented Cultures: Implementation Steps to Assist Mergers and
                           Organizational Transformations, GAO-03-669 (Washington, D.C.: July 2, 2003).




                           Page 25                                                              GAO-13-22 NASA
              enables employees and other stakeholders to understand the expected
              outcome of the management initiative and engenders not only their
              cooperation, but also their ownership of the outcome, which our work has
              shown can take at least 5 to 7 years to fully implement. NASA, by having
              a policy that is not comprehensive, allowing projects to opt out of using
              the new EVM system, and not committing resources to adequately train
              staff, continues to limit progress in the cultural change needed to
              implement EVM. Without breaking through the cultural resistance to EVM
              and committing to efforts intended to strengthen the use of EVM, NASA is
              missing an opportunity to make full use of a key tool that could help it to
              manage its projects more effectively.


              Implementing an effective earned value management system and using it
Conclusions   across a large federal agency with well-established processes is without
              doubt a challenging task. However, NASA has had uneven progress to
              date. NASA acknowledges that EVM can be a valuable tool for monitoring
              project development and has initiated an effort to implement an
              agencywide system. Currently, only a few of the 10 major spaceflight
              projects we reviewed were able to produce reliable EVM data, raising
              concern that they cannot produce reliable estimates of cost at completion.
              Moreover, until the data are sufficiently reliable, NASA, as well as
              external stakeholders, lose valuable insights into project performance that
              EVM provides. A sound EVM system is not merely an accounting tool; it
              can alert managers to developing problems so that they can be proactive
              in reducing the project’s cost and schedule overruns. However, NASA is
              not making full use of a key tool that could help it address the cost and
              schedule issues that have kept NASA acquisition management on GAO’s
              high risk list for more than 20 years.

              Though NASA’s recent efforts to improve its EVM capability and
              strengthen its policy are steps in the right direction, implementation—fully
              integrating EVM into management processes—has been the biggest
              challenge and remains so today. NASA faces cultural and technical
              challenges that it must overcome to successfully implement an earned
              value system and to use this data on a regular basis to inform decision-
              making. Managing change will be key if NASA’s latest effort to overcome
              these challenges and implement an agencywide EVM plan is to succeed.
              To accomplish effective earned value management, strong leadership is
              required to set an expectation that reliable and credible data are
              necessary to manage a successful project. This should be buttressed with
              a sound EVM policy and system that are required, and a commitment of
              resources to enable staff. Without sustained momentum and commitment,


              Page 26                                                      GAO-13-22 NASA
                      its current efforts could suffer the same consequence as those in the
                      past.


                      To improve NASA management and oversight of its spaceflight projects,
Recommendations for   we recommend that the NASA Administrator direct the appropriate offices
Executive Action      to take the following four actions:

                      •   Establish a time frame by which all new spaceflight projects will be
                          required to implement NASA’s newly developed EVM system, unless
                          the project is proposing to use a certified system, to ensure that in-
                          house efforts are compliant with ANSI/EIA-748. The time frame
                          selected should take in to account the need to increase NASA’s
                          institutional capability for conducting EVM and analyzing and reporting
                          the data.
                      •   Conduct an EVM skills gap analysis to identify areas requiring
                          augmented capability across the agency. Based on the results of the
                          assessment, develop a workforce training plan to address any
                          deficiencies.
                      •   Develop an EVM change management plan to assist managers and
                          employees throughout the agency with accepting and embracing
                          earned value techniques while reducing the operational impact on the
                          agency. The plan should include a strategy for having the agency’s
                          senior leadership communicate their commitment to implementation of
                          EVM.
                      •   To improve the reliability of project EVM data, NASA Procedural
                          Requirements (NPR) 7120.5 should be modified to require projects to
                          implement a formal surveillance program that:
                          •   Ensures anomalies in contractor-delivered and in-house monthly
                              earned value management reports are identified and explained,
                              and report periodically to the center and mission directorate’s
                              leadership on relevant trends in the number of unexplained
                              anomalies.
                          •   Ensures consistent use of WBSs for both the EVM report and the
                              schedule.
                          •   Ensures that lower level EVM data reconcile to project level EVM
                              data using the same WBS structure.
                          •   Improves underlying schedules so that they are properly
                              sequenced using predecessor and successor dependencies and
                              are free of constraints to the extent practicable so that the EVM
                              baseline is reliable.




                      Page 27                                                     GAO-13-22 NASA
                     We provided a draft of this report to NASA for comment. In its written
Agency Comments      comments, reproduced in appendix IV, NASA’s Chief Engineer stated that
and Our Evaluation   the agency concurred with two recommendations and partially concurred
                     with two other recommendations. In particular, the agency agreed with
                     our recommendation to perform an EVM skills gap analysis and develop a
                     workforce training plan to address any deficiencies identified. To that end,
                     NASA plans to conduct a skills gap assessment and to augment its EVM
                     training program to address the gaps identified. In addition, the agency
                     also concurred with our recommendation to develop an EVM change
                     management plan and is planning to develop a strategy targeted at all
                     levels of the workforce from project team members to the agency’s
                     leadership.

                     The agency partially concurred with our recommendation that NASA
                     establish a time frame by which all new spaceflight projects will be
                     required to implement NASA's newly developed EVM system, stating that
                     they already require projects to perform EVM with an ANSI/EIA 748
                     compliant system. NASA stated that its phased rollout approach for
                     implementing the agency’s EVM system is based on available resources,
                     budgetary constraints, and institutional and project needs. However,
                     NASA's approach does not include a timeframe for when projects will be
                     required to use the new system. We recommended that NASA establish a
                     timeframe for rolling out the system to all projects because a large
                     number of projects are not in compliance with NASA’s requirement, and
                     very few are implementing the new EVM system. Using the newly
                     developed EVM system could help projects better ensure NASA's
                     projects are using a system that is compliant with the ANSI/EIA standard.
                     The agency also noted its disagreement with the notion that all projects,
                     in particular those that have a skilled EVM workforce and a compliant
                     system in place, should be forced to use the agency's new system.
                     Accordingly, we acknowledged in our report that there may be situations
                     where a project should not be required to use the agency’s EVM system,
                     such as when a project already uses a certified system or for current,
                     ongoing projects. Furthermore, we reported that NASA lacks the
                     appropriate level of surveillance of its projects’ EVM systems to monitor
                     project adherence to the EVM standard; in addition, the extent to which
                     EVM has been effectively implemented across NASA's projects varies. If
                     NASA chooses not to require projects to use its new system it should take
                     steps to ensure that it monitors their compliance with the standard.
                     Finally, while we appreciate that NASA must balance its resources with its
                     needs, the benefits that an effective EVM system can provide, such as
                     allowing project managers to identify cost growth and take actions to stem



                     Page 28                                                      GAO-13-22 NASA
further growth, warrants prioritization of resources to ensure earlier
widespread implementation of NASA's EVM system.

The agency also partially concurred with our recommendation that NPR
7120.5 be modified to require projects to implement a formal EVM
surveillance program. Citing resource constraints, NASA commented that
it does not plan to implement a formal surveillance program, but agreed
that the reliability and utility of the EVM data needed to be improved. As a
result, the agency plans to establish a surveillance process, expand the
workforce’s EVM skills, and provide analytical tools including developing
an EVM System Acceptance and Surveillance Guide. Furthermore, NASA
said that it was not appropriate to incorporate the surveillance
requirement in NPR 7120.5 because of the level of detail associated with
requirements in that policy. The most important part of our
recommendation is that EVM surveillance should be required to ensure
better quality data. We reported that only 4 of the 10 projects we
assessed had a comprehensive EVM surveillance system in place and
the others had limited or no surveillance being performed and only 3 of
the 10 projects had fully reliable data. Without an effective surveillance
program, NASA cannot ensure its projects are meeting the ANSI/EIA-748
standard because internal management systems are not being reviewed
to determine if they are providing reliable cost, schedule and technical
performance data.

In its response, NASA also noted that the project data we used in our
report is over a year old and does not take in to account progress made
over the past year. We disagree and note that in the report we discuss
progress the agency has made over the past year, such as strengthening
the EVM requirements in its policy and developing its new EVM system.
Furthermore, we did not solely rely on project EVM data to develop our
findings. For example, interviews with project officials and additional
documentation they provided further validated our findings. Finally, it is
important to note that NASA Acquisition Management has been on GAO’s
High Risk list for many years due to the agency’s cost and schedule
performance on its major projects. EVM is an important project
management tool that can assist project managers in managing and
assessing performance. Not addressing key issues that impact the
availability of accurate and reliable data could lessen the usefulness of
this key project management tool.

NASA also provided technical comments, which have been addressed in
the report, as appropriate.



Page 29                                                       GAO-13-22 NASA
We are sending copies of this report to interested congressional
committees, NASA’s Administrator, and other interested parties. In
addition, this report will also be available at no charge on the GAO
website at http://www.gao.gov.

Should you or your staff have any questions concerning this report,
please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs may
be found on the last page of this report. GAO staff who made key
contributions to this report are listed in appendix V.




Cristina T. Chaplain
Director
Acquisition and Sourcing Management




Page 30                                                      GAO-13-22 NASA
Appendix I: Scope and Methodology
             Appendix I: Scope and Methodology




             To determine to what extent NASA’s major spaceflight projects are using
             earned value management (EVM) to manage acquisitions, we reviewed
             all NASA major spaceflight projects with a life cycle cost of over $250
             million that were in the implementation phase and thus required to report
             EVM. There were 13 projects that met these criteria. Of these, 2 projects
             had recently launched and the launch of a 3rd was imminent. These 3
             were excluded from our assessment because the work on these projects
             was nearly complete. Collectively, the 10 projects we reviewed will cost
             over $14 billion to develop.

             Our review looked at EVM data for the period of August 2010 to August
             2011. While the majority of the 10 projects we reviewed had at least 6
             months of EVM data, a few did not because the project had only recently
             entered the implementation phase. Additionally, some projects were
             undergoing a re-plan and, therefore, were not required to provide EVM
             data for certain periods of time. In particular, the Tracking and Data Relay
             Satellite Sustainment contract had 5 months of data, the Mars
             Atmosphere Volatile Evolution Mission project had only 3 months of data,
             and the Orbiting Carbon Observatory 2 Orbital contract had just 4 months
             of data. The James Webb Space Telescope contract had EVM data for
             the whole period; however, the contractor underwent a major replan in
             which all EVM data except for the reporting of actual costs were
             suspended from January 2011 to April 2011. Although the Global
             Participation Measurement (GPM) project provided EVM reports for the
             entire project, we did not conduct an analysis of the project EVM data,
             because the performance reports did not contain the detailed data we
             needed for our analysis. However, we were able to assess the
             performance data for GPM’s Microwave Imager Instrument.

             To determine cost and schedule performance for the selected major
             projects based on an evaluation of the earned value data, we analyzed
             project and contractor data and documentation including contract
             performance reports; project work breakdown structures; project
             schedules; integrated baseline review briefings; the extent to which
             surveillance of the EVM system was occurring; and monthly management
             briefings for the 10 major spaceflight projects. Specifically, we compared
             project documentation with EVM and scheduling best practices as
             identified in GAO’s Cost Estimating and Assessment Guide and Schedule




             Page 31                                                      GAO-13-22 NASA
Appendix I: Scope and Methodology




Assessment Guide. 1 To the extent practicable, we assessed how each of
the 10 projects’ EVM data adhered to 3 of the American National
Standard Institute’s (ANSI) and Electronic Industries Alliance (EIA) 32
guidelines. In addition, we assessed the projects against 3 fundamental
EVM practices that we believe are necessary for maintaining a reliable
EVM system, as identified in our cost guide. 2 We also analyzed the
contract performance reports for each project to determine the level of
data reliability. Specifically, we identified instances of the following: (1)
negative planned value, earned value, or actual cost; (2) planned value
and earned value without actual cost; (3) earned value and actual cost
without planned value; (4) actual cost without planned value or earned
value; (5) earned value without planned value and actual cost; (6)
inconsistencies between the estimated cost at completion and the
planned cost at completion; (7) actual cost exceeding estimated cost at
completion; and (8) planned or earned values exceeding planned cost at
completion.

For the contracts that had more than 6 months of data, we used contract
performance report data in order to generate our estimated overrun or
underrun of the contract cost at completion by using formulas accepted
by the EVM community and printed in the GAO Cost Estimating and
Assessment Guide. To perform this analysis, we examined contractor
performance reports over the period for which we had data to show
trends in cost and schedule performances. We generated multiple
formulas for the projected contract cost at completion that were based on
how much of the contract had been completed up to August 2011 or
earlier for some projects. The ranges in the estimates at completion are
driven by using different efficiency indices based on the project’s past
cost and schedule performance to forecast the cost of the remaining work
and adding that cost to the actual costs to date. The efficiency indices
capture how the project has performed in the past and can be useful in
predicting how it will perform in the future.




1
 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and
Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009), and
GAO Schedule Assessment Guide: Best Practices for Project Schedules—Exposure
Draft, GAO-12-120G (Washington, D.C.: May 2012).
2
 The three fundamental EVM practices we assessed were using a certified EVM system,
conducting integrated baseline reviews, and performing surveillance of the EVM system.




Page 32                                                                GAO-13-22 NASA
Appendix I: Scope and Methodology




We also analyzed monthly project management review briefings to
support our analysis. Finally, we analyzed the earned value data
contained in EVM performance reports obtained from the projects. To
perform this analysis, we compared the cost of work completed with
budgeted costs for scheduled work to show trends in cost and schedule
performances.

To assess the reliability of the cost data, in addition to electronically
testing the data for anomalies, we also reviewed relevant project
documentation and interviewed agency and project officials about the
data. We then followed up on these anomalies with the project offices that
manage each of the spaceflight projects by sharing our preliminary
analysis for each of the 10 projects. When warranted, we updated our
analyses based on the agency’s response and additional documentation
provided to us. The data that we used were sufficiently reliable for how
we portrayed them in our report and we are making recommendations to
the agency to improve NASA’s data reliability based on the findings
discussed in our report. We did not test the adequacy of the agency or
contractor accounting systems.

To support and clarify information in our documentation reviews, we
interviewed agency officials at NASA headquarters and EVM Focal Point
Working Group members—the agency officials that are responsible for
developing an integrated, consistent approach for implementing EVM
throughout NASA, as well as addressing EVM review and surveillance
issues and activities—at each center and the Human Exploration and
Operations and Science mission directorates to discuss their roles as well
as the extent to which EVM data is used to inform decision making. We
interviewed officials at NASA headquarters in Washington, D.C.; and
officials from Ames Research Center in Moffett Field, California; Dryden
Flight Research Center in Edwards, California; Glenn Research Center in
Cleveland, Ohio; Goddard Space Flight Center in Greenbelt, Maryland;
Johnson Space Center in Houston, Texas; the Jet Propulsion Laboratory
in Pasadena, California; Kennedy Space Center in Florida; Langley
Research Center in Hampton, Virginia; Marshall Space Flight Center in
Huntsville, Alabama; and Stennis Space Center in Mississippi.
Additionally, we received responses to questions regarding the EVM data
from each of the 10 projects we selected for review. These questions
addressed how EVM practices are implemented at the project level and
how the project utilizes EVM data.

To determine the challenges that NASA has faced in implementing an
effective EVM system, we interviewed NASA headquarters personnel to


Page 33                                                     GAO-13-22 NASA
Appendix I: Scope and Methodology




discuss the status and plans for implementing the agency-wide EVM
system. We developed a standard set of questions and interviewed EVM
Focal Point Working Group members at each center and the Human
Exploration and Operations and Science mission directorates to assess
the challenges of implementing EVM at individual centers and across the
agency. We also interviewed a selection of senior officials and program
executives at NASA headquarters that represent projects from each
mission directorate and NASA center included in our engagement to
obtain their perspective on the challenges of implementing and using
EVM on their projects. We also reviewed prior GAO and NASA Inspector
General reports that discuss the agency’s prior efforts to implement EVM.
We examined GAO and NASA Inspector General reports that discuss the
importance of effective organizational change. Additionally, we received
written responses to a standard set of questions regarding the challenges
associated with implementing EVM from each of the 10 projects we
selected for review.

To determine the steps that NASA is taking to improve its use of earned
value management, we examined the results of NASA’s EVM capability
pilot projects and draft policies and guidance and compared these with
best practices in EVM as discussed in GAO’s Cost Estimating and
Assessment Guide, 3 the ANSI/EIA-748 standard, and OMB Circular A-11,
Preparation, Submission, and Execution of the Budget and the Capital
Programming Guide. In addition, we interviewed NASA headquarters
personnel and EVM Focal Point Working Group members at each center
and the Human Exploration and Operations and Science mission
directorates to discuss and obtain information on ongoing initiatives the
agency has undertaken.

We conducted this performance audit from June 2011 to November 2012
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.




3
GAO-09-03SP.




Page 34                                                    GAO-13-22 NASA
Appendix II: Importance of Earned Value
              Appendix II: Importance of Earned Value
              Management



Management

              Pulling together essential cost, schedule, and technical information in a
              meaningful, coherent fashion is always a challenge for any project.
              Without this information, management of the project will be fragmented,
              presenting a distorted view of project status. For several decades, the
              Department of Defense (DOD) has utilized a tool called earned value
              management (EVM) to compare the value of work performed to the
              work’s actual cost. Earned value management goes beyond the two-
              dimensional approach of comparing budgeted costs to actual costs. It
              attempts to compare the value of work accomplished during a given
              period with the work scheduled for that period. By using the value of
              completed work as a basis for estimating the cost and time needed to
              complete the project, the earned value concept should alert project and
              senior managers to the potential problems early in the project.

              In 1996, DOD adopted 32 criteria for evaluating the quality of earned
              value management systems. These 32 criteria are organized into 5 basic
              categories: organization, planning and budgeting, accounting
              considerations, analysis and management reports, and revisions and data
              maintenance. In general terms, the criteria require contractors to

                  •     define the contractual scope of work using a work breakdown
                        structure;
                  •     identify organizational responsibility for the work;
                  •     integrate internal management subsystems;
                  •     schedule and budget authorized work;
                  •     measure the progress of work based on objective indicators;
                  •     collect the cost of labor and materials associated with the work
                        performed;
                  •     analyze any variances from planned cost and schedules;
                  •     forecast costs at contract completion; and
                  •     control changes.
              The criteria have evolved to become an American National Standards
              Institute (ANSI) and Electronic Industries Alliance (EIA) standard for EVM,
              which has been adopted by major U.S. government agencies, industry,
              and the governments of Canada and Australia. The full application of
              EVM system criteria is appropriate for large-cost reimbursable contracts
              where the government bears the cost risk. For such contracts, the
              management discipline described by the criteria is essential. In addition,
              data from an EVM system have been proven to provide objective reports
              of contract status, allowing numerous indices and performance measures
              to be calculated. These can then be used to develop accurate estimates




              Page 35                                                        GAO-13-22 NASA
Appendix II: Importance of Earned Value
Management




of anticipated costs at completion, providing early warning of impending
schedule delays and cost overruns.

The standard format for tracking earned value is through a Contract
Performance Report (CPR). The CPR is a monthly compilation of cost,
schedule and technical data which displays the performance
measurement baseline, any cost and schedule variances from that
baseline, the amount of management reserve used to date, the portion of
the contract that is authorized unpriced work, and the contractor’s latest
revised estimate to complete the project.

As a result, the CPR can be used as an effective management tool
because it provides the project manager with early warning of potential
cost and schedule overruns. Using data from the CPR, a project manager
can assess trends in cost and schedule performance. This information is
useful because trends can be difficult to reverse. Studies have shown that
once projects are 15 percent complete, the performance indicators are
indicative of the final outcome. For example, a CPR showing a negative
trend for schedule status would indicate that the project is behind
schedule. By analyzing the CPR, one could determine the cause of the
schedule problem such as delayed flight tests, changes in requirements,
or test problems because the CPR contains a section that describes the
reasons for the negative status. A negative schedule condition is a cause
for concern, because it can be a predictor of later cost problems since
additional spending is often necessary to resolve problems. For instance,
if a project finishes 6 months later than planned, additional costs will be
expended to cover the salaries of personnel and their overhead beyond
what was originally expected. CPR data provides the basis for
independent assessments of a project’s cost and schedule status and can
be used to project final costs at completion in addition to determining
when a project should be completed.

Examining a project’s management reserve is another way that a project
can use a CPR to determine potential issues early on. Management
reserves, which are funds that may be used as needed, provide flexibility
to cope with problems or unexpected events. EVM experts agree that
transfers of management reserve should be tracked and reported
because they are often problem indicators. An alarming situation arises if
the CPR shows that the management reserve is being used at a faster
pace than the project is progressing toward completion. For example, a
problem would be indicated if a project has used 80 percent of its
management reserve but only completed 40 percent of its work. A
project’s management reserve should contain at least 10 percent of the


Page 36                                                     GAO-13-22 NASA
Appendix II: Importance of Earned Value
Management




cost to complete a project so that funds will always be available to cover
future unexpected problems that are more likely to surface as the project
moves into the testing and evaluation phase.

An Integrated Baseline Review (IBR) is conducted to ensure the reliability
of the EVM data and that the performance measurement baseline
accurately captures all the work to be accomplished. Data from the CPR
can then be used to assess project status—typically, monthly. Cost and
schedule variances are examined and various estimates at completion
are developed and compared to available funding. The results are shared
with management for evaluating contractor performance. Finally, because
EVM requires detailed planning for near-term work, as time progresses,
planning packages are converted into detailed work packages. This cycle
continues until all work has been planned and the project is complete.

An IBR is an evaluation of the performance measurement baseline to
determine whether all project requirements have been addressed, risks
identified, and mitigation plans put in place and all available and planned
resources are sufficient to complete the work. Too often, projects overrun
because estimates fail to account for the full technical definition,
unexpected changes, and risks. Using poor estimates to develop the
performance measurement baseline will result in an unrealistic baseline
for performance measurement.

After the CPRs start being delivered to the government, it is important to
oversee the project by conducting surveillance of the EVM system.
Surveillance is reviewing a contractor’s EVM system as it is applied to
one or more projects. Its purpose is to focus on how well a contractor is
using its EVM system to manage cost, schedule, and technical
performance. For instance, surveillance checks whether the contractor’s
EVM system summarizes timely and reliable cost, schedule, and
technical performance information directly from its internal management
system; complies with the contractor’s implementation of ANSI/EIA-748
guidelines; provides timely indications of actual or potential problems by
performing spot checks, sample data traces, and random interviews;
maintains baseline integrity; gives information that depicts actual
conditions and trends; and provides comprehensive variance analyses at
the appropriate levels, including corrections for cost, schedule, technical,
and other problem areas.

Effective surveillance ensures that the key elements of the EVM process
are maintained over time and on subsequent applications. EVM system
surveillance ensures that the contractor is following its own corporate


Page 37                                                       GAO-13-22 NASA
Appendix II: Importance of Earned Value
Management




processes and procedures and confirms that the contractor’s processes
and procedures continue to satisfy the ANSI guidelines.

The surveillance team designated to perform project reviews should
consist of a few experienced staff who fully understand the contractor’s
EVM system and the processes being reviewed. The surveillance
organization should appoint the team leader and ensure that all
surveillance team members are independent. This means that they
should not be responsible for any part of the projects they assess. Key
activities on the surveillance team’s agenda include reviewing documents,
addressing government project office concerns, and discussing prior
surveillance findings and any open issues. Sufficient time should be
allocated to all these activities to complete them. The documents for
review should give the team an overview of the project’s implementation
of the EVM process.

Successful surveillance is predicated on access to objective information
that verifies that the project team is using EVM effectively to manage the
contract and complies with company EVM procedures. Objective
information includes project documentation created in the normal conduct
of business. Besides collecting documentation, the surveillance team
should interview control account managers and other project staff to see if
they can describe how they comply with EVM policies, procedures, or
processes. During interviews, the surveillance team should ask them to
verify their responses with objective project documentation such as work
authorizations, cost and schedule status data, variance analysis reports,
and back-up data for any estimates at completion.




Page 38                                                     GAO-13-22 NASA
Appendix III: Case Studies of Selected
              Appendix III: Case Studies of Selected
              Projects’ Implementation of Earned Value
              Management


Projects’ Implementation of Earned Value
Management
              We conducted case studies of 10 major spaceflight system acquisition
              projects. This appendix provides a brief description of each project,
              including an analysis of the project’s earned value data and trends. As
              part of our analysis, we assessed the projects’ implementation of three
              fundamental earned value management (EVM) practices that we believe
              are necessary for maintaining a reliable EVM system—using a certified
              American National Standards Institute (ANSI) and Electronic Industries
              Alliance (EIA) compliant system, performing surveillance, and conducting
              integrated baseline reviews.

              We also assessed the projects’ EVM data against three ANSI and EIA
              guidelines. These guidelines state that the authorized work elements for
              the project should be defined typically using a work breakdown structure
              (WBS) that has been tailored to the project and that the WBS is the same
              for the cost estimate, schedule, and EVM. The ANSI/EIA guidelines also
              state that projects should have a schedule that describes the sequence of
              work by listing activities in the order in which they are to be carried out
              and identifying significant task interdependencies required to meet project
              requirements. Finally, the ANSI/EIA guidelines state that a project should
              establish and maintain a time-phased budget baseline to track cost and
              schedule variances in an EVM system.

              As mentioned above, this appendix includes an analysis of each project’s
              earned value trends from August 2010 to August 2011. These data and
              trends are often described in terms of cost and schedule variances. Cost
              variances compare the earned value of the completed work with the
              actual cost of the work performed. Schedule variances are also measured
              in dollars, but they compare the earned value of the completed work with
              the value of the work that was expected to be completed. Positive
              variances are good—they indicate that activities are costing less than
              expected or are completed ahead of schedule. Negative variances are
              bad—they indicate activities are costing more than expected or are falling
              behind schedule. Variances are merely measures that indicate that work
              is not being performed according to plan and that it must be assessed
              further to understand why. Although our EVM cost projections may show
              that a project is experiencing negative cost variances and schedule
              slippages, this does not mean that a project has exceeded its agency
              baseline commitment and will require additional funds to complete the




              Page 39                                                      GAO-13-22 NASA
                                           Appendix III: Case Studies of Selected
                                           Projects’ Implementation of Earned Value
                                           Management




                                           project. 1 These estimates use a project’s EVM baseline, which represents
                                           only a portion of the agency baseline commitment for a project. The EVM
                                           baseline contains the cost and schedule contained in a project’s
                                           management agreement minus unallocated future expenses and
                                           schedule margin held by the project and others above the project. 2

                                           As of August 2011, the budget at completion for the 10 projects was
                                           estimated to be $6.4 billion. To estimate the project variance at
                                           completion, we examined the trends for the earned value data for the
                                           entire project, if data was collected at that level, or elements of the
                                           project. 3 Table 4 provides a summary of the projects’ implementation of
                                           each EVM best practice we assessed and projected costs.

Table 4: NASA 10 Spaceflights Project EVM Data Summary

Dollars in millions
                                               Used a
                                               certified                           Data
                                               EVM                                 resulting
                                               system      Conducted EVM           from the
                                               compliant   an         system       EVM
                                  Data         with        integrated surveillance system                 Cumulative
                                  review       ANSI/EIA    baseline   is being     are                    cost
                  Subcomponenta                                                                                              b
Projects                          period       standard    review     performed reliable BAC              variance   GAO VAC
Global            Overall
Precipitation
Measurement
              c                                                                         ◐



                                           1
                                            The agency baseline commitment is the integrated set of requirements, cost, schedule,
                                           technical content, and an agreed-to joint confidence level that forms the basis for NASA’s
                                           commitment with OMB and Congress. NPR 7120.5E, appendix A (Aug. 14, 2012).
                                           2
                                             Unallocated future expenses are costs that are expected to be incurred but cannot yet be
                                           allocated to a specific work breakdown structure subelement of a project’s plan.
                                           Management control of some unallocated future expenses may be retained above the
                                           level of the project (i.e., Agency, Mission Directorate, or program). The management
                                           agreement defines the parameters and authorities over which the program or project
                                           manager has management control.
                                           3
                                            The Landsat Data Continuity Mission (LDCM) and James Webb Space Telescope
                                           (JWST) projects do not collect project level data and GPM did not provide project level
                                           data in a format we could analyze. Therefore, for these projects, we were only able to
                                           analyze EVM data for segments of the project that reported EVM. Also, these projects had
                                           project segments, such as instruments, that did not produce any EVM data.




                                           Page 40                                                                  GAO-13-22 NASA
                                              Appendix III: Case Studies of Selected
                                              Projects’ Implementation of Earned Value
                                              Management




Dollars in millions
                                                  Used a
                                                  certified                           Data
                                                  EVM                                 resulting
                                                  system      Conducted EVM           from the
                                                  compliant   an         system       EVM
                                     Data         with        integrated surveillance system         Cumulative
                                     review       ANSI/EIA    baseline   is being     are            cost
Projects          Subcomponenta      period       standard    review     performed reliable BAC      variance   GAO VACb
                  GPM Microwave      August
                  Imager             2010-July
                                     2011
                                                                                      ◐   $159      -$13      -$14 to -$22

James Webb        Overall
Space
Telescope
                                                                               ◐       ◐
                  Observatory        May 2011-
                                     July 2011                                        ◐   2,967       2           NA

                  Near infrared      August
                  camera             2010-July
                                     2011
                                                                               ◐       ◐   205        -33       -34 to -48

Landsat Data      Overall
Continuity
Mission
                                                                                      ◐
                  Operational Land August
                  Imager           2010-
                                   August                                             ◐   129        -45       -49 to -52
                                   2011
Lunar             Overall
Atmosphere
and Dust
Environment
                                                                               ◐       ◐
Explorer
                  Project            September
                                     2010-June
                                     2011
                                                                               ◐       ◐   140         3       -0.1 to -13

Magnetospheric Overall
Multiscale                                                                     ◐       ◐
                  Project            August
                                     2010-July
                                     2011
                                                                               ◐       ◐   1,037      -18       -47 to -80

                  Instrument Suite   August
                                     2010-
                                                                               ◐       ◐   233d       -4        -10 to -24
                                     August
                                     2011
Mars              Overall
Atmosphere
and Volatile
Evolution
                                                        ◐                       ◐       

Mission




                                              Page 41                                                         GAO-13-22 NASA
                                            Appendix III: Case Studies of Selected
                                            Projects’ Implementation of Earned Value
                                            Management




Dollars in millions
                                                 Used a
                                                 certified                           Data
                                                 EVM                                 resulting
                                                 system      Conducted EVM           from the
                                                 compliant   an         system       EVM
                                   Data          with        integrated surveillance system      Cumulative
                                   review        ANSI/EIA    baseline   is being     are         cost
Projects          Subcomponenta    period        standard    review     performed reliable BAC   variance   GAO VACb
                  Project          June
                                   2011-
                                   August                                    ◐          259         5           NA
                                   2011
                  Spacecraft       January
                                   2011-
                                                                                       169   6           1 to 14
                                   August
                                   2011
                  Science          November
                  Operations       2010-
                  Center
                         e
                                   August             —                       ◐          3     5           0 to 0.4
                                   2011
                  Remote sensing   November
                         e
                  Package          2010-
                                   August             —                       ◐          20    1           1 to 7
                                   2011
                  Langmuir Probe   November
                  and Waves        2010-
                  Instrumen
                           e
                                   August             —                       ◐          5     -1          -2 to -3
                                   2011
Orbiting Carbon Overall
Observatory 2f                                                               ◐       

                  Project          December
                                   2010-July                                           129   1           -1 to -26
                                   2011
                  Spacecraft       April 2011-
                                   July 2011                                 ◐          30    0            NA

Radiation Belt    Overall
Storm Probes                                                                 ◐       

                  Project          August
                                   2010-
                                   August                                    ◐          291   -32         -40 to -41
                                   2011g
Stratospheric   Overall
Observatory for
Infrared
Astronomy
                                                      ◐          ◐             ◐       ◐
Project




                                            Page 42                                                        GAO-13-22 NASA
                                             Appendix III: Case Studies of Selected
                                             Projects’ Implementation of Earned Value
                                             Management




Dollars in millions
                                                    Used a
                                                    certified                               Data
                                                    EVM                                     resulting
                                                    system          Conducted EVM           from the
                                                    compliant       an         system       EVM
                                    Data            with            integrated surveillance system                  Cumulative
                                    review          ANSI/EIA        baseline   is being     are                     cost
Projects          Subcomponenta     period          standard        review     performed reliable BAC               variance   GAO VACb
                  Project           August
                                    2010-
                                    August                                                     ◐     ◐   384      4              -1.4 to -76
                                    2011
                  Airborne System   August
                                    2010-July                                                          38       3              3 to 4
                                    2011
Tracking and      Overall
Data Relay
Satellite
                                                                                                    ◐
                  Spacecraft        August
                                    2010-
                                                                                                         698        -131        -152 to -185
                                    August
                                    2011
                  Sustainment       August
                                    2010-
                                    August                                                          ◐      5           0              NA
                                    2011
                                             Source: GAO analysis of NASA and contractor data.
                                             a
                                               Items in italics represent subcomponents that are accounted for in higher level EVM reports.
                                             b
                                               Forecast range based on Actual cost + Budgeted Cost of Work Remaining / Performance factor
                                             where Performance factor was either the cumulative cost performance index (CPI) * schedule
                                             performance index (SPI), the current CPI, or current CPI * SPI.
                                             c
                                               In October 2011, the Global Precipitation Measurement mission was approved for a replan.
                                             d
                                               Only reflects Phases B-D.
                                             e
                                               The contract for the Science Operations Center, Remote sensing Package, Langmuir Probe and
                                             Waves Instrument does not exceed $50 million. Therefore, the supplier is not required to have a
                                             certified system for this contract.
                                             f
                                               Orbiting Carbon Observatory 2 is in the process of being rebaselined due to a change in the launch
                                             vehicle.
                                             g
                                               Missing data for October and November 2010.
                                             NA - Not applicable means there was not enough data to forecast an Estimate at Completion.

                                             Key:       The agency met this criterion

                                                       ◐     The agency partially met this criterion


                                                        The agency did not meet this criterion

                                             With timely and effective action taken by project and executive
                                             management, it is possible to reverse negative performance trends so
                                             that the projected negative cost variances at completion may be reduced.



                                             Page 43                                                                           GAO-13-22 NASA
                       Appendix III: Case Studies of Selected
                       Projects’ Implementation of Earned Value
                       Management




                       To get such results, management needs to obtain reliable EVM data from
                       EVM systems that adhere to the ANSI/EIA-748 standard for informed
                       decision making. Until project offices undertake a rigorous validation of
                       their EVM data, NASA faces an increased risk that managers may not be
                       receiving the information they need to effectively manage their projects.

                       The following information describes the key that we used in tables 5
                       through 14 to convey the results of our assessment of the 10 case study
                       projects’ implementation of EVM practices.

                           Key description                                     Key symbol

                           Fully met this criterion                            


                           Partially met this criterion                        ◐
                           Did not meet this criterion                         



                       The Global Precipitation Measurement (GPM) mission, a joint NASA and
Global Precipitation   Japan Aerospace Exploration Agency (JAXA) project, seeks to improve
Measurement            the scientific understanding of the global water cycle and the accuracy of
                       precipitation forecasts. The GPM is composed of a core spacecraft
                       carrying two main instruments: a Dual-frequency Precipitation Radar and
                       a GPM Microwave Imager (GMI). GPM builds on the work of the Tropical
                       Rainfall Measuring Mission, and will provide an opportunity to calibrate
                       measurements of global precipitation when it launches in 2014.

                       This analysis focuses only on the GMI-1 effort. Ball Aerospace and
                       Technology Company is the prime contractor for GMI. GMI’s current
                       contract value is $217 million, which represents approximately 23 percent
                       of the total GPM project budget of $932.8 million. The GMI instrument
                       was delivered to Goddard Space Flight Center in February 2012 for
                       integration into NASA’s upcoming Earth science spacecraft. All remaining
                       effort for GMI-1 is post delivery support, which is all level of effort. 4 The
                       GPM project provided EVM reports for the entire project, but we did not



                       4
                        NASA was also developing a second Microwave Imager instrument for flight on another
                       spacecraft—GMI-2—that was descoped from the project in 2012. However, the science
                       requirements for GPM can still be met without flying the GMI-2 instrument.




                       Page 44                                                              GAO-13-22 NASA
                                       Appendix III: Case Studies of Selected
                                       Projects’ Implementation of Earned Value
                                       Management




                                       conduct an analysis of the project EVM data because the performance
                                       reports did not contain the detailed data we needed for our analysis.

Table 5: Assessment of GPM EVM Practices

Dollars in millions
                      Used a certified
                         EVM system                                                                         Data resulting
                       compliant with  Conducted an                  EVM system                             from the EVM
                            ANSI/EIA      integrated               surveillance is              Budget at      system are GAO variance at
                            standard baseline review             being performed               completion          reliable   completion
GPM Microwave                                                                         
Imager                                                                                               $159             ◐       -$14 to -$22

                                       Source: GAO analysis of contractor EVM data as of July 2011



                                       The GMI-1 contractor met the three fundamental ANSI/EIA-748 practices
GPM Microwave                          necessary for a reliable EVM system. The Defense Contract Management
Imager Contractor                      Agency (DCMA) certified that the GMI-1 contractor’s EVM system
                                       complied with the ANSI/EIA standard in April 2008. Though the contractor
Uses a Certified EVM                   has a certified system, the implementation of that system is questionable
System Compliant                       based on our findings below. We assessed GMI-1’s EVM data against
                                       three ANSI/EIA guidelines. These guidelines state that the authorized
with the ANSI/EIA                      work elements for the project should be defined typically using a WBS
Standard                               that has been tailored to the project and the WBS is the same for the cost
                                       estimate, schedule, and EVM. Our review found that the WBS in the GMI-
                                       1 schedule did not match the WBS used for the EVM data. Project
                                       officials said that the WBS for the project schedule did not match the
                                       WBS used in GMI contractor’s EVM reports because GMI-1 is only one
                                       element of the total project and the project level schedule has a simplified
                                       summary of the GMI-1 schedule that was used for completeness. The
                                       ANSI/EIA guidelines also state that projects should have a schedule that
                                       describes the sequence of work by listing activities in the order in which
                                       they are to be carried out and identifying significant task
                                       interdependencies required to meet project requirements. Our review of
                                       the GMI-1 schedule found some sequencing issues. For example, about
                                       5 percent of the remaining activities were missing predecessor and
                                       successor links, which are necessary for properly sequencing work so
                                       that the schedule will update in response to changes. We also found that
                                       19 percent of the remaining activities had date constraints, which also
                                       hinder the schedule’s ability to respond dynamically to status updates
                                       resulting in an artificial or unrealistic view of the project plan. These
                                       sequencing issues and constraint dates within the schedule affect the
                                       reliability of the overall network and the schedule’s ability to correctly



                                       Page 45                                                                             GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




                            calculate float values and the critical path. Project officials agreed with our
                            findings but said that these issues are corrected as they are discovered,
                            so there is no impact to the project. Finally, though resource loading the
                            schedule is not required to meet the ANSI/EIA guidelines, it is a best
                            practice and therefore resources should be accounted for in the schedule
                            in order to develop this baseline, according to the GAO cost guide. We
                            found that the GPM schedule was not resource loaded, it is GAO’s
                            assessment that the project did not show evidence that it had established
                            and maintained a time-phased budget baseline.


Project Conducted an        The project conducted an integrated baseline review (IBR) in January
Integrated Baseline         2006. From that review, officials believed the results were less than
Review                      satisfactory due to the contractor’s inability to demonstrate the integration
                            of contract schedule and cost in accordance with their EVM system
                            description. In particular, 64 areas of concern were identified during the
                            IBR, including major concerns with data continuity, cost/schedule risk,
                            and EVM processes. As a result of these issues and changes to the
                            contract another IBR was performed in January 2011.


EVM Surveillance Is Being   Joint surveillance reviews of the EVM data are being performed by DCMA
Performed                   and the contractor. According to DCMA, although they have found some
                            deficiencies in the contractor’s EVM data, the contractor has responded
                            with acceptable corrective action plans.


Data Resulting from the     We reviewed contract performance reports from August 2010 to July
EVM System Are              2011. Our review of the GMI-1 EVM data found various data anomalies
Somewhat Reliable           that call into question the reliability of the data. For example, we found
                            negative values for EVM data without any explanation in three monthly
                            reports. Project officials responded that the negative values all fell within
                            the contract threshold and therefore the GMI-1 contractor did not need to
                            provide an explanation in the variance analysis report. In addition, there
                            were many instances of costs and performances being recorded when no
                            work had been scheduled. In response to this finding, project officials
                            explained that this work had been accounted for in previous months,
                            which explain the missing values. Anomalous EVM data prevents the
                            project from being able to gain meaningful and proactive insight into
                            potential cost and schedule performance shortfalls, and take corrective
                            action to avoid shortfalls in the future. Figure 2 below illustrates that as of
                            July 2011, the project was reporting a negative cumulative cost variance



                            Page 46                                                         GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




of $13 million and a negative cumulative schedule variance of
$3 million.

Figure 2: GAO Earned Value Analysis of GMI-1 Data




GPM project officials said the cumulative negative cost and schedule
variances were due to slips caused by suppliers not delivering flight
hardware as planned, which pushed uncompleted work into the future.
The negative variances were also caused by tasks being worked that
were not included in the baseline. Officials also said the project
experienced unfavorable variances in labor costs across all integrated
project teams, which were further affected by the unfavorable 2010 year
end indirect rate adjustment. 5 To address the negative schedule variance,
officials said the contractor increased staffing and added extra shifts,


5
 Year-end indirect rate adjustments often happen when financial accounting assumptions
about forward pricing do not hold true. When a company reconciles its books at the year
end, it often finds that during the year it forecasted a larger business base than the actual
business base, and at the end of the year, indirect rate adjustments have to be made,
which can result in additional cost variances for all contracts.




Page 47                                                                     GAO-13-22 NASA
                   Appendix III: Case Studies of Selected
                   Projects’ Implementation of Earned Value
                   Management




                   which increased labor costs thereby increasing the contract value.
                   Officials noted, however, that the increase in contract value did not
                   translate into an increase in the baseline, just an increase in the project
                   funding. Due to both the negative cost and schedule variances, we are
                   forecasting a negative variance ranging from $14 million to $22 million at
                   contract completion. 6 According to NASA, the project is not overrunning
                   its approved baseline commitment, because the EVM baseline does not
                   include unallocated future expenses held at the project and headquarters
                   level.


                   The James Webb Space Telescope (JWST) is a large, infrared-optimized
James Webb Space   space telescope that is designed to find the first galaxies that formed in
Telescope          the early universe. Its focus will include searching for first light, assembly
                   of galaxies, origins of stars and planetary systems, and origins of the
                   elements necessary for life. Scheduled to launch in October 2018,
                   JWST’s instruments will be designed to work primarily in the infrared
                   range of the electromagnetic spectrum, with some capability in the visible
                   range. JWST will have a large primary mirror composed of 18 smaller
                   mirrors, measuring 6.5 meters (21.3 feet) in diameter, and a sunshield
                   that is the size of a tennis court. A successor to the Hubble Space
                   Telescope and the Spitzer Space Telescope, JWST will reside in an orbit
                   about 1 million miles from the Earth.

                   NASA rebaselined JWST in September 2011 and made changes in the
                   project’s management in response to cost and schedule performance
                   issues and the recommendations of the Independent Comprehensive
                   Review Panel report. As part of the rebaseline, NASA took the lead role
                   for systems engineering from the prime contractor. The telescope, along
                   with a segmented primary mirror, will deliver infrared light to the Fine
                   Guidance Sensor and fine pointing updates to the Observatory and four
                   scientific instruments including the Near-Infrared Camera (NIRCam), the
                   Near-Infrared Spectrograph, the Mid-Infrared Instrument and the Fine
                   Guidance Sensor/Near InfraRed Imager and Slitless Spectrograph. For
                   work being performed by its international partners, such as the Near-
                   Infrared Spectrograph, EVM data is not collected.




                   6
                    Variance at completion is calculated by taking the difference between the estimate at
                   completion and the budget at completion.




                   Page 48                                                                  GAO-13-22 NASA
                                        Appendix III: Case Studies of Selected
                                        Projects’ Implementation of Earned Value
                                        Management




                                        At the time of our review, there was no EVM data for the overall JWST
                                        project or for the work done in-house on the Integrated Science
                                        Instrument Module. According to project officials, the Integrated Science
                                        Instrument Module effort began over a decade ago, and significant parts
                                        of the project, particularly those undertaken in-house at GSFC were not
                                        structured to enable EVM to be implemented easily. However, the JWST
                                        project office is collecting EVM data from Northrop Grumman Aerospace
                                        Systems and obtains copies of the Lockheed Martin Space Systems-
                                        Advanced Technology EVM data, which is the University of Arizona’s
                                        prime contractor. Northrop Grumman Aerospace Systems is responsible
                                        for developing and launching the JWST Observatory, which comprises
                                        the spacecraft, sunshield, and the optical telescope element, systems
                                        integration and test observatory verification, observatory commissioning,
                                        and ground and launch support equipment. The University of Arizona is
                                        responsible for developing the Near-Infrared Camera science instrument.

Table 6: Assessment of JWST EVM Practicesa

Dollars in millions
                         Used a certified     Conducted an                                                    Data resulting
                            EVM system           integrated               EVM system                          from the EVM
                          compliant with           baseline             surveillance is             Budget at    system are GAO variance at
                       ANSI/EIA standard             review           being performed              completion        reliable   completion
Observatory                                                                                            $2,967                            NAb
                                                                                           
                                                                                                                         ◐
Near infrared camera                                                                                    $205                    -$34 to -$48
                              
                                                               ◐                             ◐                           ◐
                                        Source: GAO analysis of contractor EVM data as of July 2011.
                                        a
                                            All budget and variance at completion data is before the JWST re-plan.
                                        b
                                            NA - Not applicable means there was not enough data to forecast an Estimate at Completion.



                                        The observatory contractor met all three fundamental ANSI/EIA-748
Observatory                             practices necessary for a reliable EVM system. For the observatory
Contractor Has a                        portion of the JWST project, Northrop Grumman Aerospace Systems has
                                        a certified EVM System, which it uses to fulfill the earned value reporting
Certified EVM System                    requirement. Though the contractor has a certified system, the
Compliant with                          implementation of that system is questionable based on our findings
                                        below. We assessed the observatory contractor’s EVM data against three
ANSI/EIA Standard                       fundamental ANSI/EIA guidelines. These guidelines state that the
                                        authorized work elements for the project should be defined typically using
                                        a WBS that has been tailored to the project and the WBS is the same for
                                        the cost estimate, schedule, and EVM. We found that while consistent
                                        naming conventions existed between the WBS and contract performance



                                        Page 49                                                                              GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




                            reports, there were discrepancies in the hierarchical structure and
                            numbering of the WBS elements. For example, the WBS dictionary shows
                            the Mission Assurance listed as 3.0, while the contract performance
                            report provided by the observatory contractor has Mission Assurance
                            listed as 2.0. NASA officials responded that the observatory contractor is
                            not required to follow the project level WBS hierarchical structure nor the
                            WBS numbering scheme. They further stated that the project and the
                            observatory WBS structures are not identical because procurement of the
                            observatory is only one element of the overall JWST project. NASA did
                            not provide us with a schedule in a format that would allow us to
                            determine if the schedule had the proper sequencing in place. As a result,
                            we were unable to determine if significant task interdependencies such as
                            predecessor and successor links were in place to ensure that the
                            schedule would update in response to changes. In addition, because we
                            could not view the schedule in its native software, we were unable to
                            determine if the schedule was resource loaded, which is a best practice
                            for establishing and maintaining a time-phased budget baseline.


Project Conducted an        An Integrated Baseline Review was conducted in February 2010;
Integrated Baseline         however, because of the rebaseline in September 2011, according to the
Review                      project an additional IBR was held for the observatory in October 2012.


EVM Surveillance Is Being   Surveillance is being performed by DCMA. In addition, EVM data is
Performed                   reviewed monthly by multiple individuals on the project office as well as at
                            higher levels of NASA headquarters and the Goddard Space Flight
                            Center.


Data Resulting from the     We reviewed contract performance reports from August 2010 to July
EVM System Was              2011. Our review of the Observatory EVM data found various data
Somewhat Reliable           anomalies that call into question the reliability of the data. For example,
                            we found actual costs recorded without any work being performed,
                            inconsistencies between the reported estimate at completion and budget
                            at completion, large month to month performance swings, and
                            unexplained variances. NASA officials explained that during this time
                            period the observatory contractor was engaged in re-planning efforts so
                            NASA did not want them to expend resources reporting performance
                            management data to an outdated performance measurement baseline
                            that did not reflect the new rebaseline assumptions. Further, while
                            variances analyses were provided in the variance analysis reports for
                            WBS elements that exceeded contractual thresholds, there was no


                            Page 50                                                      GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




explanation for the anomalies we found. A variance analysis report
provides a detailed, narrative report explaining significant cost and
schedule variances and other contract problems and topics. Without this
information, management cannot understand the reasons for the
variances and the contractor’s plan for fixing them. When information is
missing in a variance analysis report, the EVM data will not be meaningful
or useful as a management tool.

As of July 2011, the observatory portion of the JWST Project was 51
percent complete with a positive cumulative cost variance of $2.4 million.
For the same period the project was also experiencing a positive
cumulative schedule variance of $0.9 million as seen in figure 3 below. In
January 2011, the observatory contractor began replanning the remaining
effort in order to meet the October 2018 launch readiness date. As a
result, NASA suspended performance measurement reporting during the
period of January 2011 through April 2011. However, the observatory
contractor was still required to submit contract performance reports
depicting actual cost and estimate at completion data.

Figure 3: GAO Earned Value Analysis of Observatory Data




Page 51                                                     GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




                          Since there were only 3 months of EVM data after the rebaseline, we
                          were not able to forecast a variance at completion. Since our
                          assessment, JWST project officials said they have made some significant
                          improvements in implementation and use of EVM that includes an EVM
                          approach for the in-house work that will provide EVM metrics to measure
                          progress. In addition, the officials said the project is also doing
                          managerial analysis on its contracts and project components and
                          producing independent estimate of completion each month based on the
                          EVM data.


                          The Near-Infrared Camera (NIRCam) contractor, Lockheed Martin, met
Near-Infrared Camera      one of the three fundamental ANSI/EIA-748 practices necessary for a
Contractor Has a          reliable EVM system. The contractor has a certified EVM System that it is
                          using to report EVM data for NIRCam. Our review found similar problems
Certified EVM System      with the WBS hierarchical structure and numbering of the elements. For
Compliant with            example, the WBS dictionary shows Mission Assurance listed as 3.0
                          while the NIRCam contract performance report shows this effort under
ANSI/EIA Standard         element 5.5. NASA officials explained that the NIRCam contractor is not
                          required to follow the project level WBS hierarchical structure or
                          numbering of elements. As stated above, we did not receive a schedule in
                          its native software, so we were unable to confirm whether the schedule
                          was sequenced using predecessor and successor links or if it was
                          resource loaded, necessary for establishing the time-phased budget
                          baseline.


Project Conducted an      As stated above, while the JWST project conducted an IBR in February
Integrated Baseline       2010, the rebaseline necessitated a new IBR, which occurred in March
Review                    2012.


EVM Surveillance Is not   While formal surveillance is not occurring for the Lockheed Martin EVM
Being performed           data, monthly reviews of the EVM data are performed by both the project
                          and program offices and by independent groups. As part of these
                          reviews, trending metrics are prepared and presented to management as
                          part of the internal project reviews and monthly status reviews.


Data Resulting from the   We reviewed contract performance reports from August 2010 through
EVM System Are            July 2011 and found various data anomalies that call into question the
Somewhat Reliable         reliability of the data. For example, we found EVM data with negative
                          values, no work scheduled but work performed, and actual costs being


                          Page 52                                                    GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




incurred without any work being performed. NASA officials stated that
variances that exceeded contractual thresholds should be reflected in the
variance analysis reports, however many of these anomalies did not
breach a variance threshold so the report provided no explanation. In
addition, NASA officials explained that many of these anomalies occurred
during the project replan, which was formally approved in September
2011, when the project was rebaselined. As a result of the replan, NASA
suspended EVM data reporting, which resulted in many of the anomalies
we found.

As of July 2011, the NIRCam portion of the JWST project was 98 percent
complete. In July 2011, there was a negative cumulative cost variance of
$33 million and a negative cumulative schedule variance of $4.4 million
as seen in figure 4 below. The reasons for the downward trend reflected
in the graph below were not explained because variance analysis
reporting was suspended during the replan period.

Figure 4: GAO Earned Value Analysis of NIRcam Data




Based on the downward trend, we are forecasting a negative variance at
completion ranging anywhere from $34 million to $48 million. This



Page 53                                                    GAO-13-22 NASA
                     Appendix III: Case Studies of Selected
                     Projects’ Implementation of Earned Value
                     Management




                     analysis is based on information prior to the project’s 2011 replan. The
                     variances projected do not take in to account the establishment of a new
                     EVM baseline as a result of the replan.

                     The Landsat Data Continuity Mission (LDCM) is a joint mission between
Landsat Data         NASA and the United States Geological Survey (USGS) that seeks to
Continuity Mission   extend the ability to detect and quantitatively characterize changes on the
                     global land surface at a scale where natural and man-made causes of
                     change can be detected and differentiated. It is the successor mission to
                     Landsat 7. The Landsat data series, begun in 1972, has provided the
                     longest continuous record of changes in the Earth’s surface as seen from
                     space.

                     Landsat data is a resource for people who work in agriculture, geology,
                     forestry, regional planning, education, mapping, and global change
                     research. The LDCM provides remotely sensed, highly calibrated,
                     moderate resolution, multispectral imagery affording systematic global
                     coverage of the Earth’s land surfaces on a seasonal basis and makes the
                     data readily available for large-scale and long-term Earth System Science
                     and Land use/land cover change research and management. The project
                     plans to launch early in February 2013.

                     LDCM consists of an Operational Land Imager (OLI) and a Thermal
                     Infrared Sensor (TIRS) science instrument, a spacecraft, and a mission
                     operations element. LDCM does not collect EVM data at the project level.
                     The decision not to perform EVM at the project level was reviewed
                     extensively prior to proceeding into the design and development phase,
                     according to project officials. Also, there is no EVM data for the
                     spacecraft effort because this work is being done under a firm fixed price
                     contract and NASA regulations do not require EVM for firm fixed price
                     contracts. The TIRS instrument, built in-house at NASA’s Goddard Space
                     Flight Center, was added late in the formulation phase with an aggressive
                     delivery schedule and delivered in February 2012. The Ground System is
                     being built and delivered by the USGS. The developmental part of the OLI
                     contract with Ball Aerospace and Technology Corporation was completed
                     with delivery of the instrument in early October 2011. Following on-orbit
                     checkout, the contract will transfer to USGS for management. The OLI
                     instrument is the only part of the project that performed EVM.




                     Page 54                                                      GAO-13-22 NASA
                                         Appendix III: Case Studies of Selected
                                         Projects’ Implementation of Earned Value
                                         Management




Table 7: Assessment of Landsat OLI EVM Practices

Dollars in millions
                        Used a certified
                           EVM system    Conducted an                 EVM system              Data resulting from
                         compliant with      integrated             surveillance is Budget at    the EVM system GAO variance
                      ANSI/EIA standard baseline review           being performed completion          are reliable at completion
Operational Land                                                                                          $129          -$49 to -$52
Imager                                                                                                        ◐
                                         Source: GAO analysis of contractor EVM data as of August 2011.



                                         The OLI contractor met two of the three fundamental ANSI/EIA-748
Operational Land                         practices necessary for a reliable EVM system. In 2007, the Defense
Imager Contractor                        Contract Management Agency (DCMA) certified Ball Aerospace and
                                         Technology Corporation’s EVM system. Though the contractor has a
Has a Certified EVM                      certified system, the implementation of that system is questionable based
System Compliant                         on our findings below. We assessed the contractor EVM data against
                                         three ANSI/EIA guidelines. These guidelines state that the authorized
with ANSI/EIA                            work elements for the project should be defined typically using a WBS
Standard                                 that has been tailored to the project and the WBS is the same for the cost
                                         estimate, schedule, and EVM. Our review found that the WBS in the OLI
                                         schedule was consistent with the WBS used for the EVM data. The
                                         ANSI/EIA guidelines also state that projects should have a schedule that
                                         describes the sequence of work by listing activities in the order in which
                                         they are to be carried out and identifying significant task
                                         interdependencies required to meet project requirements. Our analysis
                                         found 11 percent of the remaining activities were missing dependencies,
                                         13 percent had lags, and 24 percent had constraints, among other things.
                                         When schedules are not sequenced properly, float values and the
                                         calculated critical path will not be valid. Project officials said the schedule
                                         sequencing is driven by external forces such as facilities availability,
                                         spacecraft and ground system interfaces, DCMA inspections, and so
                                         forth. The effects of these forces on schedule sequencing and critical path
                                         are reviewed extensively and the validity of the critical path is not typically
                                         an issue for project management. Finally, the ANSI/EIA guidelines state
                                         that a project should establish and maintain a time-phased budget
                                         baseline to track cost and schedule variances in an EVM system. Though
                                         resource loading the schedule is not required to meet the ANSI/EIA
                                         guideline, it is a best practice and therefore resources should be
                                         accounted for in the schedule in order to develop this baseline, according
                                         to the GAO cost guide. Our analysis found that the OLI schedule was
                                         resource loaded.




                                         Page 55                                                                     GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




Project Conducted an        The project conducted an IBR in 2007. The IBR identified 80 areas of
Integrated Baseline         concern and as a result the LDCM Project did not accept the
Review                      Performance Measurement Baseline at the IBR. However since 2007,
                            Ball Aerospace and Technology Corporation has addressed the areas of
                            concern.


EVM Surveillance Is Being   Joint surveillance reviews are being conducted on Ball Aerospace and
performed                   Technology Corporation’s EVM system by DCMA and the Defense
                            Contract Audit Agency.


Data Resulting from the     We reviewed contract performance reports from August 2010 to August
EVM System Are              2011. While the EVM data reflected several data anomalies, Ball
Somewhat Reliable           Aerospace provided detailed explanations for each of them. For example,
                            negative values were attributed to year end rate savings, labor
                            corrections, or material transfers that were greater than the current month
                            actual costs which resulted in a negative number.

                            The EVM data assessed below reflects all work associated with the
                            LDCM OLI instrument. Figure 5 below illustrates that as of August 2011,
                            the project was a reporting a negative cumulative cost variance of
                            approximately $46 million and a negative cumulative schedule variance of
                            $1.3 million.




                            Page 56                                                      GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 5: GAO Earned Value Analysis of Operational Land Imager Data




The negative cumulative cost and schedule variances were due to
various technical challenges experienced during instrument development,
including detector fabrication issues, coatings issues that necessitated
the build of a second calibration subassembly, and instrument integration
onto the baseplate taking two weeks longer than expected to complete
fabrication. Due to the negative variances, we are forecasting a negative
variance at completion ranging from $49 million to $52 million. According
to a project official, the project is not overrunning its agency baseline
commitment, because the EVM baseline does not include unallocated
future expenses held by the project or NASA headquarters. The
developmental part of the OLI instrument contract with Ball Aerospace
and Technology Corporation was completed with delivery of the
instrument in early October 2011.




Page 57                                                         GAO-13-22 NASA
                                          Appendix III: Case Studies of Selected
                                          Projects’ Implementation of Earned Value
                                          Management




                                          The Lunar Atmosphere and Dust Environment Explorer (LADEE) will
Lunar Atmosphere                          determine the global density, composition, and time variability of the lunar
and Dust                                  atmosphere. LADEE’s measurements will determine the size, charge, and
                                          spatial distribution of electrostatically transported dust grains. Additionally,
Environment                               it will carry an optical laser communications demonstrator that will test
Explorer                                  high-bandwidth communication from lunar orbit. 7 Finally, it will broaden
                                          the scientific understanding of other planetary bodies regarding
                                          exospheres or very thin atmospheres, like the moon.




Table 8: Assessment of LADEE EVM Practices

Dollars in millions
                    Used a certified                                                                        Data resulting
                       EVM system       Conducted an               EVM system                               from the EVM
                     compliant with         integrated           surveillance is                Budget at      system are     GAO variance at
                  ANSI/EIA standard    baseline review         being performed                 completion          reliable       completion
Project                                                                                             $140                          -$0.1 to -$13
                                                                                   ◐                                   ◐
                                          Source: GAO analysis of NASA data as of June 2011.




                                          7
                                           The Lunar Laser Communications Demonstration instrument is being funded by NASA’s
                                          Human Exploration and Operations Mission Directorate at a cost of approximately $65
                                          million. This cost is not included in the $262.9 million LADEE cost baseline.




                                          Page 58                                                                              GAO-13-22 NASA
                       Appendix III: Case Studies of Selected
                       Projects’ Implementation of Earned Value
                       Management




                       LADEE met only one of three key fundamental ANSI/EIA-748 practices
Project Office Does    for reliable EVM system. NASA’s EVM guidance says that projects must
not Have a Certified   start reporting EVM data once the project enters the project
                       implementation phase, if the project’s life-cycle cost is at or greater than
EVM System             $20 million. While LADEE may not have a contract that exceeds $20
Compliant with the     million, the overall LADEE project cost is about $262.9 million.
                       Nevertheless, project officials said LADEE is responsible only for
ANSI/EIA Standard      gathering “EVM-like” data at the project level. The “EVM-like” data is
                       collected using the “EVM Lite” process, which attempts to meet the
                       ANSI/EIA-748 standard where applicable. Project officials said that when
                       LADEE was initiated in February 2008, NPR 7120.5D was still in effect
                       which required application of EVM principles. This is, in effect, “EVM-Lite”
                       or “EVM-Like.” Officials further stated that prior to August 2011, the
                       LADEE project evaluated candidate EVM techniques using sample data
                       from January to March 2011. Based on that evaluation period, LADEE
                       decided to use the “EVM Lite” technique to collect the necessary data to
                       manage the project. From April to June 2011, additional evaluations of
                       this technique continued. Therefore, standard reporting of the LADEE
                       “EVM-like” project level data did not begin until August 1, 2011. We
                       reviewed all available EVM data from September 2010 to June 2011,
                       which was before the standard EVM reporting period began.

                       Though LADEE does not have a certified system, we assessed how well
                       LADEE project was meeting three ANSI/EIA guidelines. These guidelines
                       state that the authorized work elements for the project should be defined
                       typically using a WBS tailored to the project and the WBS should be the
                       same for the schedule, cost estimate, and EVM. Our analysis found that
                       the WBS in the LADEE schedule matched the WBS used for EVM data.
                       The ANSI/EIA guidelines also states that projects should have a schedule
                       that describes the sequence of work by listing activities in the order in
                       which they are to be carried out and identifies significant task
                       interdependencies required to meet project requirements. Our review of
                       the LADEE schedule found 3 percent of the remaining activities were
                       missing predecessor or successor links, which cause the schedule to not
                       properly update in response to changes. We also found that about 6
                       percent of the remaining activities had date constraints and/or lags, which
                       also hinder the schedule from responding dynamically to changes and
                       can portray an artificial or unrealistic view of the project plan. While these
                       issues may be relatively small, any missing dependencies, constraints,
                       and lags may disrupt the reliability of the overall network. Finally, the
                       ANSI/EIA guidelines state that a project should establish and maintain a
                       time-phased budget baseline to track cost and schedule variances in an
                       EVM system. Though resource loading the schedule is not required to


                       Page 59                                                        GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




                          meet the ANSI guidelines, it is a best practice and therefore resources
                          should be accounted for in the schedule in order to develop this baseline,
                          according to the GAO cost guide. We found that the LADEE schedule
                          was not resource-loaded, therefore, it is GAO’s assessment that the
                          project did not show evidence that it had established and maintained a
                          time-phased budget baseline.


Project Conducted an      LADEE conducted an IBR in December 2010. From that review, a list of
Integrated Baseline       concerns about the EVM data and schedule were identified. Specifically,
Review                    the project was not using a consistent approach to collect EVM data,
                          calling into question the credibility of the data. In addition, it was unclear if
                          an objective assessment of cost and schedule performance could be
                          made using data generated by the “EVM Lite” approach. Since that
                          review, officials said all issues and actions have been addressed and
                          were formally closed by the IBR panel in July 2011.


EVM Surveillance Is Not   While formal surveillance is not occurring, EVM data assurance reviews
Being Performed           are being performed during monthly management reviews with the Lunar
                          Quest Program office. 8


Data Resulting From the   We reviewed contract performance reports from September 2010 to June
EVM System Are            2011. Our review of LADEE’s EVM data found various data anomalies
Somewhat Reliable         that call into question the reliability of the data. For example, from
                          October 2010 to December 2010 there were some instances of negative
                          values in the EVM reports that were unexplained. Since a variance
                          analysis report provides a detailed narrative explaining significant cost
                          and schedule variances, when this information is missing, management
                          cannot understand the reasons for variances and the plan for fixing them.
                          Also, the EVM data provided by the project office was not presented in a
                          standard EVM format. This could be attributed to the fact that LADEE is
                          required to provide only “EVM-like” data.




                          8
                           The Lunar Quest Program consists of flight missions and instruments for lunar missions
                          of opportunity, as well as research and analysis efforts.




                          Page 60                                                                 GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 6 below illustrates that as of June 2011, the LADEE project was
reporting a positive cumulative cost variance of $3 million and a negative
cumulative schedule variance of $10 million.

Figure 6: GAO Earned Value Analysis of LADEE Project Data




LADEE project officials provided no information regarding positive cost
and negative schedule variance drivers. As such, we have no insight into
what could be causing deviations from the plan. Based on the positive
cost variance and negative schedule variance thus far, we are forecasting
a negative variance at completion ranging from $0.1 million to $13 million
dollars. According to NASA, the project is not overrunning its commitment
because the EVM baseline does not include unallocated future expenses
held at the project and NASA headquarters level.




Page 61                                                      GAO-13-22 NASA
                                         Appendix III: Case Studies of Selected
                                         Projects’ Implementation of Earned Value
                                         Management




                                         The Magnetospheric Multiscale (MMS) is made up of four identically
Magnetospheric                           instrumented spacecraft. The mission will use the Earth’s magnetosphere
Multiscale                               as a laboratory to study the microphysics of magnetic reconnection,
                                         energetic particle acceleration, and turbulence. Magnetic reconnection is
                                         the primary process by which energy is transferred from solar wind to
                                         Earth’s magnetosphere and is the physical process determining the size
                                         of a space weather storm. The four spacecraft will fly in a pyramid
                                         formation, adjustable over a range of 10 to 400 kilometers. The data from
                                         MMS will be used as a basis for predictive models of space weather in
                                         support of exploration.

                                         The MMS spacecraft is being designed, developed, and tested in-house
                                         at Goddard Spaceflight Center (GSFC) while instrument development
                                         activities are under contract with Southwest Research Institute (SwRI).
                                         The Mission Operations Center and the Flight Dynamics Operations Area
                                         will be developed and operated at GSFC. The Science Operations Center
                                         for the instruments will be developed and operated at the Laboratory for
                                         Atmospheric and Space Physics at the University of Colorado and is
                                         under contract to SwRI. The MMS project office is collecting EVM data
                                         both at the project level as well as from SwRI, which is responsible for the
                                         entire instrument suite. Therefore, the SwRI Instrument Suite effort is a
                                         subset of the overall MMS project level EVM report.

Table 9: Assessment of MMS EVM Practices

Dollars in millions
                      Used a certified
                         EVM system                                                                              Data resulting
                       compliant with  Conducted an               EVM system                                     from the EVM
                            ANSI/EIA      integrated            surveillance is              Budget at              system are              GAO estimate at
                            standard baseline review          being performed               completion                  reliable                completion
Project Level                                                                                      $1,037                                       -$47 to -$80
                                                                                  ◐                                              ◐
Instrument                                                                                           $233a                                      -$10 to -$24
Suite                                                                             ◐                                              ◐
                                         Source: GAO analysis of NASA EVM data as of July 2011 and contractor EVM data as of August 2011.
                                         a
                                          The SwRI budget at completion is only for Phases B-D.




                                         Page 62                                                                                             GAO-13-22 NASA
                       Appendix III: Case Studies of Selected
                       Projects’ Implementation of Earned Value
                       Management




                       At the project level, MMS met one of three fundamental ANSI/EIA-748
Project Office Does    practices for a reliable EVM system. MMS does not have a certified EVM
Not Have a Certified   system that complies with the ANSI/EIA-748 standard. NASA project
                       officials said in-house projects are required only to be ANSI/EIA
EVM System             compliant, and are not required to have a certified system. Although the
Compliant with the     MMS project does not have a certified system, we assessed how well the
                       MMS project was meeting three ANSI/EIA guidelines. These guidelines
ANSI/EIA Standard      state that the authorized work elements for the project should be defined
                       typically using a WBS that has been tailored to the project and the WBS
                       is the same for the cost estimate, schedule, and EVM. Our review found
                       that the WBS in the MMS schedule did not match the WBS used for the
                       EVM data, which is not in line with best practices. According to project
                       officials, the MMS project was started before the requirements for earned
                       value management were developed. As a result, the schedule and WBS
                       were created without significant consideration of a one-to-one correlation
                       between the two. A project official stated that MMS has retrofitted its EVM
                       system to provide as close a correlation as possible without having to
                       rebuild the WBS.

                       The ANSI/EIA guidelines also state that projects should have a schedule
                       that describes the sequence of work by listing activities in the order in
                       which they are to be carried out and identifying significant task
                       interdependencies required to meet project requirements. Our review of
                       the MMS schedule found some sequencing issues within the schedule.
                       For example, 9 percent of predecessor and successor tasks were not
                       linked to one another, which is necessary for properly sequencing work
                       so that the schedule will update properly once changes are made. This
                       number accounts for removing all external tasks and level of effort (LOE)
                       type activities. We also found that 21 percent of the remaining activities
                       were constrained. This number also does not include LOE type activities.
                       In fact, the majority of these constraints were hard constraints. Hard
                       constraints can sometimes be impossible to meet, given the network
                       characteristics, and can thereby result in schedules that are logically
                       impossible to carry out. The presence of constraints also impacts the
                       schedule’s ability to respond dynamically to changes and may portray an
                       unrealistic view of the project plan. As a result, these sequencing issues
                       and date constraints within the schedule affect the reliability of the overall
                       network and the schedule’s ability to correctly calculate float values and
                       the critical path. Finally, the ANSI/EIA guidelines state that a project
                       should establish and maintain a time-phased budget baseline to track
                       cost and schedule variances in an EVM system. Though resource loading
                       the schedule is not required to meet the ANSI/EIA guideline, it is a best
                       practice and therefore resources should be accounted for in the schedule


                       Page 63                                                        GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




                          in order to develop this baseline, according to the GAO cost guide. We
                          found that that the MMS schedule was resource loaded.


Project Conducted an      The MMS project office conducted an IBR in June 2010. This resulted in
Integrated Baseline       positive comments, general observations, and constructive
Review                    recommendations by the review team.


EVM Surveillance Is Not   While formal surveillance is not occurring at the project level, EVM data
being Performed           assurance reviews are being performed by the Explorers and
                          Heliophysics Program Office, GSFC Flight Projects Directorate
                          managers, GSFC Chief Financial Officer’s Office, Standing Review
                          Board, and NASA headquarters.


Data Resulting from the   We reviewed contract performance reports from August 2010 to July
EVM System Are            2011. Our review of the MMS project level EVM data found some minor
Somewhat Reliable         data reliability issues. The 12 months of project level EVM data reflect all
                          of the performed in-house effort at GSFC for the project and the
                          instrument suite contractor’s summary level effort. Also, the data provided
                          was not reported in the standard contract performance report format.
                          Project officials said a defined contract performance report was not
                          dictated to the MMS project, but that all reporting has the same
                          information as a “standard” contract performance report even though the
                          formatting may be different. However, beginning in October 2011, MMS
                          began reporting with the standard format 1 contract performance report,
                          which provides cost and schedule data for each element in the project’s
                          product-oriented WBS.

                          We tried to map the EVM data in the lower level report for the instrument
                          suite to the MMS project level report and we were not able to see how the
                          costs tracked from one report to another. For example, the project level
                          EVM data showed that the instrument suite contractor’s July 2011 budget
                          at completion was $296 million, whereas the lower level EVM data in the
                          instrument suite contractor’s report showed the budget at completion to
                          be $217 million. The MMS project was able to demonstrate how the SwRI
                          budget at completion in the lower-level report mapped to the SwRI budget
                          at completion in the MMS project report. However, officials said because
                          of the way the contractor submits their data the two reports will never
                          match. Though we acknowledge that the project was able to explain how
                          the data tracked, attempting to manually resolve incompatible pieces of
                          data can become time-consuming, expensive and can lead to data


                          Page 64                                                      GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




reliability issues. Project officials said that MMS is working to capture data
at lower WBS levels, which will allow for a closer tie between the cost and
schedule data. In addition, MMS said that in addition to receiving the
SwRI EVM reports, the project now internally calculates earned value
metrics on the contractor provided instrument suite EVM data, which
gives MMS completely internally derived earned value performance
reports, based on the project team’s assessment without bias from
contractor data.

Figure 7 below illustrates that as of July 2011, the project was reporting a
negative cumulative cost variance of $18 million and a negative
cumulative schedule variance of $25 million.

Figure 7: GAO Earned Value Analysis of MMS Project Data




In October 2010, the project was in the midst of a replan and not all data
was available at the time of report submission to generate detailed
variance explanations. The replan was conducted so that the earned
value baseline was the same as the cost plan required by the agency for
monthly plan versus actual reporting. The goal of the replan was to
prevent the project from having to report variances against two different



Page 65                                                        GAO-13-22 NASA
                       Appendix III: Case Studies of Selected
                       Projects’ Implementation of Earned Value
                       Management




                       plans. Since the replan, however, the project has experienced a
                       downward trend in both cost and schedule performance. As a result, we
                       are forecasting a negative variance at completion ranging from $47
                       million to $80 million dollars. NASA stated that the project is not
                       overrunning its approved baseline commitment, because the EVM
                       baseline does not include unallocated future expenses held at the project
                       and headquarters level.


                       SwRI, the contractor responsible for the entire instrument suite for MMS,
Instrument Suite       met one of the three fundamental ANSI/EIA-748 practices for a reliable
Contractor Does not    EVM system. SwRI does not have a certified EVM system that complies
                       with the ANSI/EIA-748 standard. According to a project official, the SwRI
Have a Certified EVM   contract does not require SwRI to have a certified system but only to be
System Compliant       compliant with the ANSI/EIA-748. NASA convened an independent team
                       to review the contractor’s readiness for EVM system certification and
with ANSI/EIA          concluded that while the contractor has qualified people to support
Standard               implementation of EVM, a single point of failure exists without a
                       documented process. Not documenting the process is a problem because
                       if the people who know the process leave, new staff will not know what to
                       do. In addition, the team found that even though the right software tools
                       are in place to support EVM, more integration is needed to reduce
                       manual inputs. Finally, the team reported that compliance with the
                       ANSI/EIA-748 standard would not be achievable without management
                       support and resources. Despite these findings, the project office believed
                       that contractor’s EVM data is useful in examining trends and overall
                       performance of the instrument suite effort.

                       Though SwRI does not have a certified system, we assessed how well
                       the contractor was meeting three ANSI/EIA guidelines. These guidelines
                       state that the authorized work elements for the project should be defined
                       typically using a WBS that has been tailored to the project and the WBS
                       is the same for the cost estimate, schedule, and EVM. Our review found
                       that the WBS used in the schedule was not consistent with the WBS used
                       for the EVM data. The ANSI/EIA guidelines also state that projects should
                       have a schedule that describes the sequence of work by listing activities
                       in the order in which they are to be carried out and identifying significant
                       task interdependencies required to meet project requirements. Our review
                       of the schedule found some sequencing issues. For example, the
                       Primavera schedule provided showed 31 percent of the remaining
                       activities were missing dependencies and 36 percent were constrained.
                       Officials said majority of the constrained tasks were due to external
                       dependencies. Due to the major sequencing issues in the MMS SwRI


                       Page 66                                                      GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




                          schedule, we question the reliability of the overall network and the
                          schedule’s ability to correctly calculate float values and the critical path.
                          MMS project officials said they believed many of the constrained activities
                          are not valid because they reside in another schedule. Also, some
                          constraints, in the Harness area for example, if removed, have no effect
                          on the overall schedule. In addition, officials also said some of the
                          sequencing issues may be caused by the manual integration because
                          since some instrument provider schedules are in Microsoft Project and
                          others are in Primavera, and therefore it is not possible to ensure all tasks
                          have been linked properly. However, the MMS project scheduler tests the
                          schedule for missing dependencies, logic errors, and reasonable
                          durations and the results are shared with the project office and the
                          contractor so that appropriate action can be taken. Lastly, officials said
                          several of the activities identified in our analysis are not really schedule
                          items but level of effort type activities. When we removed the 15 LOE
                          type activities from the missing dependencies count, the schedule still
                          showed 28 percent of the remaining activities were missing
                          dependencies. When we removed the 14 LOE type activities and the 4
                          Harness activities from the constraint count, the schedule still showed 33
                          percent of remaining activities were constrained. Because the schedule is
                          the foundation for the EVM baseline, we question the reliability of the
                          Instrument Suite EVM data. Finally, the ANSI/EIA guidelines state that a
                          project should establish and maintain a time-phased budget baseline to
                          track cost and schedule variances in an EVM system. Though resource
                          loading the schedule is not required to meet the ANSI/EIA guidelines, it is
                          a best practice and therefore resources should be accounted for in the
                          schedule in order to develop this baseline, according to the GAO cost
                          guide. We found that the MMS schedule was resource-loaded.


Instrument Suite          The MMS project conducted an IBR of the MMS instrument suite effort in
Contractor Conducted an   January 2010 and all action items have been closed out.
Integrated Baseline
Review

EVM Surveillance Is Not   While the instrument suite contractor does not have a formal surveillance
Being Performed           program, the MMS project office has an EVM analyst, schedule team,
                          resources team, instrument management team, and project management
                          team who all review the instrument suite EVM data on a monthly basis. In
                          addition, the Solar Terrestrial Probes project office, Science Mission
                          Directorate management, and The Aerospace Corporation review the
                          instrument suite EVM data monthly.


                          Page 67                                                       GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




Data Resulting from the   We reviewed contract performance reports from August 2010 to August
EVM System Are            2011. Our review of the instrument suite EVM data found various data
Somewhat Reliable         anomalies that call into question the reliability of the data. For example,
                          there were negative numbers reported for EVM data in four of the months
                          that we reviewed. For some of the negative numbers, there was no
                          explanation for the cause. For others, the negative values were due to
                          correcting several months of translation errors brought on by a known
                          issue with importing data from the schedule into the EVM system
                          software. There were also errors such as inflated EVM data that once
                          corrected, resulted in negative values. Figure 8 below illustrates that as
                          August 2011, the project was reporting a negative cumulative cost
                          variance of
                          $4 million and negative cumulative schedule variance of $6 million.

                          Figure 8: GAO Earned Value Analysis of Instrument Suite Data




                          The cost variance in August 2011 dramatically improved from the
                          downward trend during the previous months due to the project applying
                          almost $13 million from its management reserve to the instrument suite
                          contract. However, due to the negative cumulative cost and schedule



                          Page 68                                                        GAO-13-22 NASA
                                             Appendix III: Case Studies of Selected
                                             Projects’ Implementation of Earned Value
                                             Management




                                             variances, we are forecasting a negative variance at completion ranging
                                             from $10 million to $24 million.


                                             The Mars Atmosphere and Volatile EvolutioN (MAVEN) mission is part of
Mars Atmosphere and                          NASA’s Mars Scout program, a robotic orbiter mission that will provide a
Volatile Environment                         comprehensive picture of the Mars upper atmosphere, ionosphere, solar
                                             energetic drivers, and atmospheric losses. Set to launch in 2013, MAVEN
                                             will deliver comprehensive answers to long-standing questions regarding
                                             the loss of Mars’ atmosphere, climate history, liquid water, and
                                             habitability. MAVEN will provide the first direct measurements ever taken
                                             to address key scientific questions about Mars’ evolution.

                                             Lockheed Martin is building the MAVEN spacecraft and will carry out
                                             mission operations for MAVEN. NASA’s Jet Propulsion Laboratory will
                                             navigate the spacecraft. The Laboratory for Atmospheric and Space
                                             Physics (LASP) at the University of Colorado at Boulder will coordinate
                                             the science team and science operations and lead the education and
                                             public outreach activities. NASA’s Goddard Spaceflight Center will
                                             provide management and technical oversight for the mission and will also
                                             provide mission systems engineering, mission design, and safety and
                                             mission assurance.

                                             The MAVEN project office is using EVM at the project level as well as
                                             collecting EVM data from Lockheed Martin, the spacecraft contractor, and
                                             LASP, which is responsible for the Science Operations Center, Remote
                                             Sensing, and Langmuir Probe and Waves instrument efforts. Both the
                                             Lockheed Martin and LASP contracts are subsets of the overall MAVEN
                                             Project EVM report.

Table 10: Assessment of MAVEN EVM Practices

Dollars in millions
                                             Conducted an       EVM system
                      Used a certified EVM      integrated    surveillance is                    Data resulting from          GAO
                        system compliant          baseline             being         Budget at      the EVM system      variance at
                       with ANSI standard           review        performed         completion           are reliable   completion
Project level                                                                            $259                                  NAa
                                                                          ◐                                      


Spacecraft                                                                            $169                            $1 to $14




                                             Page 69                                                                GAO-13-22 NASA
                                             Appendix III: Case Studies of Selected
                                             Projects’ Implementation of Earned Value
                                             Management




Dollars in millions
                                             Conducted an              EVM system
                      Used a certified EVM      integrated           surveillance is                                   Data resulting from                GAO
                        system compliant          baseline                    being                 Budget at             the EVM system            variance at
                       with ANSI standard           review               performed                 completion                  are reliable         completion
Science                                —                                                                        $3                                   $0 to $0.4
Operations                                                                             ◐                                                       
Centerb
Remote sensing                         —                                                                       $20                                     $1 to $7
Packageb                                                                               ◐                                                       


Langmuir Probe                         —                                                                        $5                                    -$2 to -$3
and Waves                                                                              ◐                                                       
Instrumentb
                                             Source: GAO Analysis of NASA EVM data as of July 2011 and contractor EVM data as of August 2011.
                                             a
                                             NA - Not applicable means there was not enough data to complete a forecast.
                                             b
                                             A certified EVM system is not required due to the contract value.



                                             MAVEN fully met one of the three key practices for implementing EVM at
Project Does Not                             the project level. Specifically, the project did not have a certified EVM
Have a Certified EVM                         system and is not required to have a certified system. Nevertheless, we
                                             assessed how well the MAVEN Project was meeting three ANSI/EIA
System Compliant                             guidelines. As part of our analysis, we assessed MAVEN’s EVM data
with ANSI/EIA                                against three ANSI/EIA guidelines. These guidelines state that the
                                             authorized work elements for the project should be defined typically using
Standard                                     a WBS that has been tailored to the project and that the WBS should be
                                             the same for the cost estimate, schedule, and EVM. We found that the
                                             project’s WBS was consistent between the schedule and EVM data.

                                             The ANSI/EIA guidelines also state that projects should have a schedule
                                             that describes the sequence of work by listing activities in the order in
                                             which they are to be carried out and identifying significant task
                                             interdependencies required to meet project requirements. Our review
                                             found some sequencing issues in the schedule. For example, 5 percent of
                                             the activities were missing dependencies, 6 percent had open ended logic
                                             links, and 20 percent had constraints, among other things. When
                                             schedules are not sequenced properly, float values and the calculated
                                             critical path will not be valid. In addition, the project conducted an
                                             integrated baseline review in July 2011. The MAVEN project office
                                             provided a May 2012 Schedule Health Check Report that showed the
                                             number of missing dependencies and constraints had been reduced
                                             considerably. Though we cannot validate the improvement in the
                                             schedule without performing our own assessment, we believe that
                                             MAVEN is working towards producing a more reliable schedule. This


                                             Page 70                                                                                            GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




                          review found that the project schedule and technical design were in good
                          shape, but noted concerns that more resources were needed to
                          implement and maintain EVM, there were cost and schedule integration
                          issues that caused the budgets for some work packages to not be in sync
                          with the schedule, and reliable critical path analysis was at risk because
                          of missing schedule links and constraints. Finally, a project should
                          establish and maintain a time-phased budget baseline at the control
                          account level, against which performance can be measured. Though
                          resource loading the schedule is not required to meet the ANSI/EIA
                          guidelines, it is a best practice and therefore resources should be
                          accounted for in the schedule in order to develop this baseline, according
                          to the GAO cost guide. We found that the schedule was resource-loaded.


Project Conducted an      Project conducted an IBR in July 2011 and all nine areas of concern were
Integrated Baseline       addressed and closed.
Review
EVM Surveillance Is Not   While formal surveillance is not occurring at the project level, EVM data
Being Performed           assurance reviews are being performed by the Mars Program Office
                          representatives, MAVEN standing Review Board representatives, and
                          Aerospace Corporation representatives at both the project level and for
                          LASP efforts.


Data Resulting from the   We reviewed contract performance reports from June 2011 to August
EVM System Is Reliable    2011. EVM data prior to spring 2011 was not available because MAVEN
                          had not been confirmed into the implementation phase. Our review of the
                          MAVEN project EVM data found that there was a mistake causing the
                          costs to be overstated by twice their actual amount. Other than this
                          mistake, which affected most of the data in the report, no other errors
                          were found. The project EVM data reflects all work on this project
                          including effort related to the project office, two in-house instruments, and
                          the Space Sciences Laboratory, as well as major efforts from Lockheed
                          Martin and LASP. Consolidated reporting of all components at a summary
                          level began with the May 2011 data, with the first full summary report
                          delivered on July 15, 2011. Therefore, we had only 3 months of data to
                          review. Figure 9 below illustrates that as of August 2011, the project was
                          reporting a positive cumulative cost variance of $5 million while also
                          experiencing a negative cumulative schedule variance of $5 million.




                          Page 71                                                       GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 9: GAO Earned Value Analysis of MAVEN Project Data




The positive cumulative cost variance was due to a decrease in labor
charges, delayed material costs, a reduction in re-work, and the
leveraging of common engineering products from other projects. The
negative schedule variance was being driven by the Neutral Gas and Ion
Mass Spectrometer instrument, which experienced technical issues such
as vendor machines not being manufactured to specifications. Because
we had only 3 months of data, we did not forecast an estimate at
completion.

In addition, we tried to map the EVM data in the lower level reports for
spacecraft, Science Operations Center, Remote Sensing, and Langmuir
Probe and Waves efforts to the overall MAVEN project EVM report and in
some cases we were not able to see how the costs tracked from one
report to another. For example, while we could easily trace the costs for
the Science Operations Center effort from the lower level EVM report to
the overall MAVEN project report, we could not clearly map the costs for
the spacecraft, Remote Sensing or Langmuir Probe and Waves efforts. In
particular, for Remote Sensing and Langmuir Probe and Waves efforts,
the lower level EVM report cost elements did not have their costs



Page 72                                                     GAO-13-22 NASA
                      Appendix III: Case Studies of Selected
                      Projects’ Implementation of Earned Value
                      Management




                      burdened at the WBS level, which could account for some of the
                      differences between the lower level report costs and the overall MAVEN
                      project costs for those elements. MAVEN project officials walked us
                      through their process of how they ensure lower-level reports map to the
                      project level reports. In addition, MAVEN project provided supporting
                      documentation that validated this assertion. Though MAVEN project
                      officials helped explain the mapping, officials said they do not mandate
                      that their contractors follow a certain reporting format, instead any
                      adjustments necessary to ensure that the lower-level reports map to the
                      project-level reports are made manually by the project office. Though
                      MAVEN project does not prescribe to a standard reporting format,
                      attempting to manually resolve incompatible pieces of data can become
                      time-consuming, expensive and can lead to data reliability issues.
                      Although the agency provided explanations for the mapping issues, the
                      ability to reconcile the costs between the reports should be evident,
                      without additional explanations.


                      Lockheed Martin, the spacecraft contractor, met the three fundamental
Lockheed Martin Has   ANSI/EIA practices necessary for a reliable EVM system. In August 2008,
a Certified EVM       the Defense Contract Management Agency (DCMA) certified that
                      Lockheed Martin’s EVM system is compliant with the ANSI/EIA standard.
System Compliant      However, the implementation of that EVM system is questionable based
with the ANSI/EIA     on our findings. We assessed the contractor’s data against the three
                      ANSI/EIA guidelines. These guidelines state that the authorized work
Standard              elements for the project should be defined typically using a WBS that has
                      been tailored to the project and the WBS is the same for the cost
                      estimate, schedule, and EVM. We found that the project’s WBS in the
                      schedule was consistent with the WBS used in the EVM data. However,
                      we found issues with the schedule. The ANSI/EIA guidelines also state
                      that projects should have a schedule that describes the sequence of work
                      by listing activities in the order in which they are to be carried out and
                      identifying significant task interdependencies required to meet project
                      requirements. Our review found some sequencing issues in the schedule.
                      For example, 2 percent of the activities remaining were missing
                      predecessor and successor links, which are necessary for properly
                      sequencing work so that the schedule will update in response to changes.
                      We also found that 37 percent of the remaining activities had constraints,
                      which also hinder the schedule’s ability to respond dynamically to status
                      updates resulting in an artificial or unrealistic view of the project plan.
                      These issues with the schedule affected the reliability of the overall
                      network and the schedule’s ability to correctly calculate float values and
                      the critical path. Project officials further explained that some of the “start


                      Page 73                                                        GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




                            no earlier than” constraints were due to resource availability and “finish no
                            later than” constraints were used intentionally to plan task activities to
                            occur as late as possible. Finally, the ANSI/EIA guidelines state that a
                            project should establish and maintain a time-phased budget baseline to
                            track cost and schedule variances in an EVM system. Though resource
                            loading the schedule is not required to meet the ANSI/EIA guidelines, it is
                            a best practice and therefore resources should be accounted for in the
                            schedule in order to develop this baseline, according to the GAO cost
                            guide. We found that the contractor schedule was resource loaded.


Project Conducted an        The project conducted the spacecraft’s IBR in April 2011. During the
Integrated Baseline         review, several areas of concern regarding the schedule were identified.
Review                      The project office said that during the integrated baseline review, the
                            review team identified many of the same observations with the schedule
                            as our findings. As a result, the project office directed the contractor to
                            eliminate the constraints, lags, and missing logic links in their integrated
                            master schedule. Since the November 2011 schedule submittal, the
                            contractor has decreased the number of sequencing issues in the
                            schedule, according to project officials. The project office also said that
                            the contractor continues to conduct schedule health checks to uncover
                            any sequencing issues and provide the project office with schedule
                            variance reports and critical path analysis, which are discussed during
                            monthly management meetings.


EVM Surveillance Is Being   Joint surveillance reviews of the EVM data are being performed by
Performed                   DCMA, MAVEN project officials, and the contractor.


Data Resulting from the     We reviewed contract management reports from January 2011 to August
EVM System Is Reliable      2011. Our review of the EVM data found no major issues with data
                            reliability. However, from January to May 2011, there were no variance
                            analysis reports produced to explain significant cost and schedule
                            variances and other contract problems and topics because they did not
                            meet reporting thresholds. Without this information, however,
                            management cannot understand the reasons for variances and the
                            contractor’s plan for fixing them.

                            Figure 10 below illustrates that as of August 2011, the project was
                            reporting a positive cumulative cost variance of $6 million and a negative
                            cumulative schedule variance of $3 million.



                            Page 74                                                        GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 10: GAO Earned Value Analysis of Spacecraft Data




One reason for the positive cost variance was due to the ability to
leverage a lower subcontractor rate than planned, which resulted in a cost
savings. The negative schedule variance was also being driven by the
mechanism subsystem falling behind schedule due to the shop being
overloaded with work and the mechanism designers supporting other
NASA efforts, among other things. Due to the positive cost variance, we
are forecasting a positive variance at completion ranging from $1 million
to $14 million.




Page 75                                                     GAO-13-22 NASA
                       Appendix III: Case Studies of Selected
                       Projects’ Implementation of Earned Value
                       Management




                       LASP, the contractor responsible for the Science Operations Center,
Laboratory for         Remote Sensing package, and Langmuir Probe and Waves instrument
Atmospheric and        efforts met one of three fundamental ANSI/EIA-748 practices necessary
                       for a reliable EVM system. To date, LASP does not have a certified EVM
Space Physics Does     system. Though the contractor is not required to have a certified system,
Not Have a Certified   we assessed how well the contractor was meeting three ANSI/EIA
                       guidelines. These guidelines state that the authorized work elements for
EVM System             the project should be defined typically using a WBS that has been tailored
Compliant with         for effective internal management control of the project and the WBS is
                       the same for the cost estimate, schedule, and EVM. We found some
ANSI/EIA Standard      slight inconsistencies in the WBS numbering between the EVM report and
                       the schedule for the Remote Sensing package, which calls into question
                       the reliability of their overall schedule network. Moreover, the Langmuir
                       Probe and Waves Instrument had issues with consistency between the
                       WBS and the schedule. In particular, there was varying levels of
                       information between the two WBSs, making it hard to use the WBS as a
                       common thread between the EVM data and the schedule. Since the WBS
                       is a critical component of EVM, it should be the same for developing the
                       EVM performance measurement baseline and the schedule. Without a
                       common link between these two features, project managers cannot fully
                       understand project cost and schedule variances.

                       The ANSI/EIA guidelines also state that projects should have a schedule
                       that describes the sequence of work by listing activities in the order in
                       which they are to be carried out and identifying significant task
                       interdependencies required to meet project requirements. We found some
                       sequencing issues with the Langmuir Probe and Waves Instrument
                       schedule. For example, 47 percent of the remaining activities had
                       constraints, which defeated the purpose of a using a dynamic schedule.
                       The quality of the schedule was also hampered by the presence of
                       schedule lags on 18 percent of the remaining activities. Schedule lags
                       must be justified because they cannot be easily monitored or included in
                       risk assessments. Finally, the ANSI/EIA guidelines state that a project
                       should establish and maintain a time-phased budget baseline to track
                       cost and schedule variances in an EVM system. Though resource loading
                       the schedule is not required to meet the ANSI/EIA guidelines, it is a best
                       practice and therefore resources should be accounted for in the schedule
                       in order to develop this baseline, according to the GAO cost guide. We
                       found that all three components showed evidence of a time-phased
                       budget baseline.




                       Page 76                                                     GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




Project Conducted an      The project conducted an IBR in March 2011 and several areas of
Integrated Baseline       concern were noted. MAVEN project officials stated that many of the
Review                    schedule issues we found were also discovered during the IBR and have
                          now been corrected.


EVM Surveillance Is Not   While formal surveillance is not occurring, EVM data assurance reviews
Being Performed           are being performed by the, project office, Mars Program Office, and
                          MAVEN Standing Review Board representatives. Also, The Aerospace
                          Corporation is working as an advisor to Science Mission Directorate’s
                          Planetary Systems Division.


Data Resulting from the   Our review of MAVEN’s Science Operations Center, Remote Sensing
EVM System Is Reliable    package, and Langmuir Probe and Waves instrument EVM data found no
                          major issues with data reliability. However, there was a lack of variance
                          analysis reports for these efforts. For example, Science Operations
                          Center had no variance analysis reports with the explanations for any of
                          the months reported. In addition, the Remote Sensing package, and
                          Langmuir Probe and Waves instrument variance analysis reports did not
                          provide any explanation for major performance swings from one month to
                          another.

                          Figure 11 below illustrates that as of August 2011, the Science
                          Operations Center portion of the MAVEN project was reporting a positive
                          cumulative cost variance of $0.01 million and a slightly negative
                          cumulative schedule variance of $0.01 million.




                          Page 77                                                    GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 11: GAO Earned Value Analysis of Science Operations Center Data




However, since the variance analysis reports provided no information
regarding what is driving the positive cost and slightly negative schedule
variances, we have no insight into the causes for deviations from the
plan. Based on the positive cost variance thus far, we are forecasting a
positive variance at completion of less than $0.5 million at contract
completion.

Figure 12 below illustrates that as of August 2011 the Remote Sensing
package portion of the MAVEN project was reporting a positive
cumulative cost variance of $0.8 million and a slightly negative cumulative
schedule variance of $0.6 million.




Page 78                                                         GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 12: GAO Earned Value Analysis of Remote Sensing Instrument Data




Factors behind the positive cost variance include an accounting lag on
the invoicing and payment process associated with the procurements,
which results in the appearance of cost efficiency. This issue has been
occurring for several months and is causing the EVM metrics to be
skewed to reflect false positive cost variances. The variance analysis
reports do not give any explanation for why there is a negative schedule
variance situation as of August 2011. As a result of the positive cost
variance, we are forecasting a positive variance at completion ranging
from $1 million to $7 million.

Figure 13 below illustrates that as of August 2011 the Langmuir Probe
and Waves portion of the MAVEN project was reporting a negative
cumulative cost variance of $0.6 million and negative cumulative
schedule variance of $0.3 million.




Page 79                                                        GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 13: GAO Earned Value Analysis of Langmuir Probe and Waves Data




Reasons for the negative cost and schedule variances are due to costs
for outside services and materials being more than planned as well as
additional work required to troubleshoot problems and mitigate risks. Due
to these problems, we are forecasting a negative variance at completion
ranging from $2 million and $3 million. According to NASA, the project is
not overrunning its commitment because the EVM baseline does not
include unallocated future expenses held at the project and headquarters
level.




Page 80                                                        GAO-13-22 NASA
                                         Appendix III: Case Studies of Selected
                                         Projects’ Implementation of Earned Value
                                         Management




                                         NASA’s Orbiting Carbon Observatory 2 (OCO-2) is designed to enable
Orbiting Carbon                          more reliable predictions of climate change and is based on the original
Observatory 2                            OCO mission that failed to reach orbit in 2009. It will make precise, time-
                                         dependent global measurements of atmospheric carbon dioxide. These
                                         measurements will be combined with data from a ground-based network
                                         to provide scientists with the information needed to better understand the
                                         processes that regulate atmospheric carbon dioxide and its role in the
                                         carbon cycle. NASA expects enhanced understanding of the carbon cycle
                                         will improve predictions of future atmospheric carbon dioxide increases
                                         and the potential impact on the climate.

                                         The OCO-2 mission consists of a dedicated spacecraft with a single
                                         instrument, flying in a near-polar, sun-synchronous orbit. The Jet
                                         Propulsion Laboratory (JPL) has overall responsibility for project
                                         management. The OCO-2 spacecraft is being built by Orbital Sciences
                                         Corporation while the instrument is being built in-house at JPL. Orbital
                                         Sciences Corporation submits spacecraft effort EVM data monthly and
                                         the project incorporates that data into the overall project EVM report. The
                                         project is facing a launch delay because the Taurus XL launch vehicle
                                         failed on the Glory Mission, and the contract was terminated. The project
                                         will be rebaselined as a result of NASA having to select a new launch
                                         vehicle. The current $477.2 million total project cost is a preliminary
                                         amount pending the outcome of the rebaseline process.

Table 11: Assessment of OCO-2 EVM Practicesa

Dollars in millions
                      Used a certified                EVM system                                                Data resulting
                         EVM system    Conducted an surveillance is                                             from the EVM
                       compliant with     integrated         being                          Budget at              system are     GAO variance at
                       ANSI standard baseline review    performed                          completion                  reliable       completion
Project                                                                                           $129                             -$1 to -$26
             b
Spacecraft                                                                                            $30                                     NAc
                                                                               ◐                                          
                                         Source: GAO analysis of JPL and contractor EVM data as of July 2011.
                                         a
                                           All the budget and variance data reported is prior to the rebaseling resulting from changing the
                                         launch vehicle.
                                         b
                                           DCMA certified Orbital Sciences Corporation’s EVM system in January 2012.
                                         c
                                           NA means that there was not enough EVM data available for us to forecast an estimate at
                                         completion.




                                         Page 81                                                                                   GAO-13-22 NASA
                       Appendix III: Case Studies of Selected
                       Projects’ Implementation of Earned Value
                       Management




                       The OCO-2 project met all three fundamental ANSI/EIA-748 practices
Project Using a        necessary for a reliable EVM system. JPL has a certified EVM system
Certified EVM System   that complies with the ANSI/EIA standard. However, the implementation
                       of its EVM system is questionable based on our findings below. We
Compliant with the     assessed the JPL EVM data against three ANSI/EIA guidelines. These
ANSI/EIA Standard      guidelines state that the authorized work elements for the project should
                       be defined typically using a WBS that has been tailored to the project and
                       the WBS is the same for the cost estimate, schedule, and EVM. We found
                       that the project had a work breakdown structure that was consistent with
                       the WBS used for the EVM data. The ANSI/EIA guidelines also state that
                       projects should have a schedule that describes the sequence of work by
                       listing activities in the order in which they are to be carried out and
                       identifying significant task interdependencies required to meet project
                       requirements. Our review found some sequencing issues in the schedule.
                       In particular, 14 percent of predecessor and successor tasks were not
                       linked to one another, which is necessary for properly sequencing work
                       so that it will update in response to changes. We also found 15 percent of
                       the remaining activities were constrained, which also hinders the
                       schedule from responding dynamically to changes and can portray an
                       artificial or unrealistic view of the project plan. These sequencing issues
                       and constraint dates within the schedule affect the reliability of the overall
                       network and the schedule’s ability to correctly calculate float values and
                       the critical path. Project officials said the missing dependencies are
                       mainly handoffs produced by level of effort (LOE) activities. Officials said
                       these activities are not necessary for valid schedule network logic, and
                       under no circumstances do these activities drive the critical path. They
                       also said the constrained activities are largely composed of mandated
                       delivery dates, which JPL uses as control points to manage subsystem
                       schedule performance prior to assembly, test, and launch operations
                       delivery, and coordinate major meeting logistics. Finally, the ANSI/EIA
                       guideline states that a project should establish and maintain a time-
                       phased budget baseline to track cost and schedule variances in an EVM
                       system. Though resource loading the schedule is not required to meet the
                       ANSI guidelines, it is a best practice and therefore resources should be
                       accounted for in the schedule in order to develop this baseline, according
                       to the GAO cost guide. To develop the baseline, resources must be
                       accounted for in the schedule and our review found that the schedule was
                       resource loaded.




                       Page 82                                                        GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




Project Conducted an        The project conducted an integrated baseline review in March 2011.
Integrated Baseline
Review

EVM Surveillance Is Being   JPL also has a formal surveillance plan in place for monitoring the EVM
Performed                   data. In particular, project officials said each month detailed earned value
                            data analysis is performed on each cost account and each work package
                            so that conclusive understanding of the performance status can be
                            reached and communicated within the project team. In addition, during
                            the project monthly management reviews, both cost and schedule
                            variances and reasons causing them are presented.


Data Resulting from the     We reviewed contractor management reports from December 2010 to
EVM System Are Reliable     July 2011. Though we found no major data reliability issues when we tried
                            to map the EVM data in the lower level spacecraft report to the JPL
                            project level report, we could not understand how the costs tracked from
                            one report to another. In a subsequent interview, officials explained, with
                            supporting documentation, how the lower level Orbital Sciences budget at
                            completion mapped to the budget at completion found in the JPL Project
                            level EVM report. Though we appreciate the explanations provided by
                            project officials regarding the differences between the two reports, the
                            issue remains that without additional documentation and explanations by
                            project officials, GAO or another independent party could not have
                            reconciled the data.

                            Figure 14 below illustrates that as of July 2011, the project was reporting
                            a positive cumulative cost variance of $0.6 million and a negative
                            schedule variance of $8 million as seen in the graph below. According to
                            project officials, the schedule variance was being caused by spectrometer
                            slit instability and an incompatible memory chip in the Remote Electronics
                            Module.




                            Page 83                                                       GAO-13-22 NASA
                        Appendix III: Case Studies of Selected
                        Projects’ Implementation of Earned Value
                        Management




                        Figure 14: GAO Earned Value Analysis of OCO-2 Project Data




                        Due to the positive cost variance and negative schedule variance we are
                        forecasting a negative variance at completion ranging from $1 million to
                        $26 million. Project officials said the project is not overrunning its
                        commitment because the EVM baseline provided to GAO does not
                        include unallocated future expenses held at the project and headquarters
                        level.


                        The spacecraft contractor, Orbital Sciences Corporation, met two of the
Spacecraft Contractor   three fundamental ANSI/EIA-748 practices necessary for a reliable EVM
Has a Certified EVM     System. At the time of our review, the Orbital Sciences did not have a
                        certified EVM system that complied with the ANSI/EIA-748 standard. In
System Compliant        January 2012, DCMA certified the Orbital Sciences EVM System. Though
with ANSI/EIA           the contractor now has a certified system, the implementation of that
                        system is questionable based on our findings below. We assessed how
Standard                well the contractor’s EVM system was meeting three ANSI/EIA
                        guidelines. These guidelines state that the authorized work elements for
                        the project should be defined typically using a WBS that has been tailored
                        to the project and the WBS is the same for the cost estimate, schedule,



                        Page 84                                                      GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




                          and EVM. We found that the spacecraft’s WBS in the schedule was not
                          consistent with the WBS used for the EVM data. The ANSI/EIA guidelines
                          also state that projects should have a schedule that describes the
                          sequence of work by listing activities in the order in which they are to be
                          carried out and identifying significant task interdependencies required to
                          meet project requirements. We found some sequencing issues in the
                          spacecraft contractor schedule. We found that 27 percent of the
                          remaining activities in the spacecraft’s schedule were missing
                          predecessor and successor links, which are necessary for properly
                          sequencing work so that the schedule will update in response to changes.
                          In addition, 24 percent of the remaining activities had date constraints,
                          which also hinder the schedule’s ability to respond dynamically to status
                          updates resulting in an artificial or unrealistic view of the project plan.
                          These sequencing issues and constraint dates within the schedule affect
                          the reliability of the overall network and the schedule’s ability to correctly
                          calculate float values and the critical path.


Spacecraft Contractor     Orbital Sciences conducted an integrated baseline review of the
Conducted an Integrated   spacecraft effort in March 2011.
Baseline Review
EVM Surveillance Is Not   While formal surveillance is not occurring, project officials stated that
Being Performed           EVM performance data is reviewed during the monthly status reviews.


Data Resulting from the   We reviewed contract performance reports from April 2011 to July 2011.
EVM System Are Reliable   Because we had only 4 months of data to review, we were unable to
                          forecast estimate variance at completion. As of July 2011, the project was
                          reporting a positive cumulative cost variance of $0.4 million and a
                          negative cumulative schedule variance of $2 million as seen in the graph
                          below. The positive cost variance was being driven by lower than
                          expected burden rates and less staff supporting project management,
                          flight assurance, and systems engineering efforts.




                          Page 85                                                        GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 15: GAO Earned Value Analysis of the Spacecraft Data




The positive cost variance was being driven by lower than expected
burden rates and less staff supporting project management, flight
assurance, and systems engineering efforts.




Page 86                                                       GAO-13-22 NASA
                                          Appendix III: Case Studies of Selected
                                          Projects’ Implementation of Earned Value
                                          Management




                                          The Radiation Belt Storm Probes (RBSP) mission will explore the Sun’s
Radiation Belt Storm                      influence on the Earth and near-Earth space by studying the planet’s
Probes                                    radiation belts at various scales of space and time. This insight into the
                                          physical dynamics of the Earth’s radiation belts will provide scientists with
                                          data to make predictions of changes in this little understood region of
                                          space. Understanding the radiation belt environment has practical
                                          applications in the areas of spacecraft system design, mission planning,
                                          spacecraft operations, and astronaut safety. The RBSP project built two
                                          spacecraft that will be used to measure the particles, magnetic and
                                          electric fields, and waves that reside in the Van Allen radiation belts.
                                          RBSP launched on August 30, 2012 on a two-year prime mission.

                                          The RBSP spacecrafts and ground system are being designed,
                                          developed, and tested by the John Hopkins University’s Applied Physics
                                          Laboratory. 9 The RBSP project office is collecting EVM data at the project
                                          level.




Table 12: Assessment of RBSP EVM Practices

Dollars in millions
                      Used a certified                                                                           Data resulting
                         EVM system       Conducted an                 EVM system                                from the EVM
                       compliant with         integrated             surveillance is                Budget at       system are GAO variance at
                       ANSI standard     baseline review           being performed                 completion           reliable   completion
Project
                                                                                        ◐               $291                    -$40 to -$41

                                          Source: GAO analysis of contractor EVM data as of April 2011.




                                          9
                                           The Applied Physics Laboratory is organized as a Limited Liability Company, wholly
                                          owned by Johns Hopkins University and operated as a division of the university.




                                          Page 87                                                                               GAO-13-22 NASA
                       Appendix III: Case Studies of Selected
                       Projects’ Implementation of Earned Value
                       Management




                       At the project level, the Applied Physics Laboratory only fully met one and
Project Does Not       partially met another of the three fundamental practices necessary for a
Have a Certified EVM   reliable EVM system. RBSP is the first full NASA mission to use EVM at
                       the Applied Physics Laboratory. According to the RBSP project manager,
System Compliant       the RBSP project implemented a limited earned value management
with the ANSI/EIA      system in Phase B as a risk mitigation activity for Phase C/D. This early
                       implementation was a risk mitigation activity, which allowed the project’s
Standard               control account manager, instrument provider, and project office to better
                       understand the reporting process and the use of the EVM system. The
                       use of EVM during this phase was also intended to allow for timely,
                       accurate, and useful EVM reporting during the formal reporting in later
                       phases of the project. Since then Applied Physics Laboratory has made
                       good progress and is in the process of meeting the intent of being
                       compliant with the 32 ANSI/EIA-748 guidelines.

                       We assessed how well the Applied Physics Laboratory was meeting the
                       three ANSI/EIA guidelines. These guidelines state that the authorized
                       work elements for the project should be defined typically using a WBS
                       that has been tailored to the project and the WBS is the same for the cost
                       estimate, schedule, and EVM. Our review, we found that that the WBS
                       used in the schedule was not consistent with the WBS used for the EVM
                       data. Project officials said that the project utilizes the Applied Physics
                       Laboratory WBS for all earned value management activity, which is then
                       mapped to the NASA WBS for reporting to the sponsor. Also, the Applied
                       Physics Laboratory WBS is uniformly utilized and consistent across all
                       control accounts in the EVM system. This internal WBS ties into both the
                       contract performance report and the integrated master schedule utilized
                       on the project.

                       The ANSI/EIA guidelines also state that projects should have a schedule
                       that describes the sequence of work by listing activities in the order in
                       which they are to be carried out and identifying significant task
                       interdependencies required to meet project requirements. Our review
                       found some sequencing issues in the schedule. For example, 23 percent
                       of the remaining activities were missing predecessor and successor links,
                       which are necessary for properly sequencing work so that the schedule
                       will update in response to changes. We also found that 29 percent of the
                       remaining activities had date constraints, which also hinder the
                       schedule’s ability to respond dynamically to status updates resulting in an
                       artificial or unrealistic view of the project plan. When schedules are not
                       sequenced properly, float values and the calculated critical path will not
                       be valid. Project officials explained that sequencing issues were a result
                       of the project consciously including constrained instrument deliveries and


                       Page 88                                                      GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




                          deliverables, level of effort activities and material and subcontractor
                          expenditures in the integrated master schedule, and though the RBSP
                          integrated master schedule had these issues it was able to monitor the
                          critical and near critical paths of all spacecraft systems and subsystems.
                          Finally, the ANSI/EIA guidelines state that a project should establish and
                          maintain a time-phased budget baseline to track cost and schedule
                          variances in an EVM system. Though resource loading the schedule is
                          not required to meet the ANSI/EIA guidelines, it is a best practice and
                          therefore resources should be accounted for in the schedule in order to
                          develop this baseline, according to the GAO cost guide. Our review found
                          that the schedule was resource loaded.


Project Conducted an      NASA conducted an integrated baseline review in August 2009. Of the
Integrated Baseline       seven IBR objectives identified, two were partially met and five were met.
Review                    In December 2010, the IBR deputy chief notified the Applied Physics
                          Laboratory that all areas of concern had been closed. As a result, the
                          overall consensus of the government review team was that the integrated
                          baseline review was successful.


EVM Surveillance Is Not   While formal surveillance is not being performed, the project office
Being Performed           reviews the EVM data on a monthly basis. Although they do not perform
                          formal surveillance, an Applied Physics Laboratory official said that they
                          performed additional monthly independent reviews of the RBSP EVM
                          system throughout Phase C/D.


Data Resulting from the   We reviewed contract performance reports from August 2010 to April
EVM System Are Reliable   2011 and July 2011 to August 2011. We were not provided reports for
                          May and June 2011. This was due to a re-plan determined necessary by
                          the Living with a Star Program Office and NASA headquarters in May
                          2011 due to changes in the launch manifest. Figure 16 below illustrates
                          that as of August 2011, the project was reporting a negative cumulative
                          cost variance of approximately $32 million and a negative cumulative
                          schedule variance of $3 million.




                          Page 89                                                      GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 16: GAO Earned Value Analysis of Project Data




The negative cumulative cost variance was caused by sustained effort on
the radio frequency communications, as well as by work on the avionics
equipment, and ground system software launch and post launch
components being behind schedule. The schedule variance is minimal
since the project is nearing completion. Due to the negative cost variance
we are forecasting a negative variance at completion from $40 million to
$41 million. Project officials noted the forecasted variance at completion
is below the revised contract value of $351.1 million, although above the
project’s estimated budget at completion. In addition, officials said this is
a project-level variance, and does not account for the application of
unallocated future expenses to fund the movement of the launch date and
to keep the project on track. When the project launched in August 2012,
its estimate at completion was below the $351.1 million budget at
completion.




Page 90                                                       GAO-13-22 NASA
                                          Appendix III: Case Studies of Selected
                                          Projects’ Implementation of Earned Value
                                          Management




                                          The Stratospheric Observatory for Infrared Astronomy (SOFIA) is a joint
Stratospheric                             project between NASA and Deutsche Zentrum für Luft-und Raumfahrt
Observatory for                           (DLR), the German space agency, to install a 2.5 meter telescope, as well
                                          as other scientific instruments capable of infrared and sub-millimeter
Infrared Astronomy                        observations, in a specially modified Boeing 747SP aircraft. This airborne
                                          observatory is designed to provide routine access to the visual, infrared,
                                          far-infrared, and sub-millimeter parts of the electromagnetic spectrum. Its
                                          mission objectives include studying many different kinds of astronomical
                                          objects and phenomena, including star birth and death; the formation of
                                          new solar systems; planets, comets, and asteroids in our solar system;
                                          and black holes at the center of galaxies. Currently, five U.S. and two
                                          German funded interchangeable instruments for the observatory are
                                          being developed to allow a range of scientific measurement to be taken
                                          by SOFIA.

                                          The SOFIA project office is using EVM at the project level as well as
                                          collecting EVM data from the L-3 Communications Integrated Systems
                                          L.P. (L-3), which is responsible for the airborne system platform effort.
                                          The EVM data provided to the project for the airborne observatory
                                          platform effort is a subset of the overall SOFIA project level EVM report.
                                          The German component of the SOFIA project does not generate earned
                                          value data and are not part of the project’s budget baseline. The
                                          University Space Research Association (USRA) has a support contract to
                                          help the Ames Research Center manage SOFIA’s science and mission
                                          operations in cooperation with the Deutsches SOFIA Institut. The USRA
                                          contract was established before NASA began requiring earned value
                                          management compliance. As a result, they are not required to generate
                                          earned value data. However, all of these components are subsets of the
                                          overall SOFIA project.

Table 13: Assessment of SOFIA EVM Practices

Dollars in millions
                      Used a certified                                                                                    Data resulting
                         EVM system       Conducted an                 EVM system                                         from the EVM
                       compliant with         integrated             surveillance is                Budget at                system are               GAO variance at
                       ANSI standard     baseline review           being performed                 completion                    reliable                 completion
Project level                                                                                                 $384                                        -$1.4 to -$76
                                                                                         ◐                                                 ◐
 Airborne system                                                                                             $38                                          $3 to $4


                                          Source: GAO analysis of EVM data as of August 2011 for the project and July 2011 for the contractor data.




                                          Page 91                                                                                                     GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




                            SOFIA project did not meet any of the three fundamental ANSI/EIA-748
Project does not have       practices necessary for a reliable EVM system. Project officials said in-
a Certified EVM             house projects are not required to have certified EVM systems. Though
                            the SOFIA project does not have a certified system, we assessed how
System Compliant            well the project was meeting three ANSI/EIA guidelines. These guidelines
with the ANSI/EIA           state that the authorized work elements for the project should be defined
                            typically using a WBS that has been tailored to the project and the WBS
Standard                    is the same for the cost estimate, schedule, and EVM. We found that the
                            WBS used in the SOFIA schedule was consistent with the WBS used for
                            the EVM data. The ANSI/EIA guidelines also state that projects should
                            have a schedule that describes the sequence of work by listing activities
                            in the order in which they are to be carried out and identifying significant
                            task interdependencies required to meet project requirements. Our review
                            found some sequencing issues in the schedule. For example, 16 percent
                            of the remaining activities had open ended logic links, and 1 percent had
                            date constraints, among other things. SOFIA project officials explained
                            that the majority of the missing links were due to using hammock
                            activities to resource load the schedule, which resulted in missing
                            successors. 10 Out of all of the activities missing successors, only 11 were
                            not hammocked activities, which accounted for a very small amount. In
                            addition, SOFIA project officials said that constraints in the schedule were
                            justified since they represented external deliveries or fiscal year funding
                            availability. They added that the one hard constraint, Must Start On, was
                            used to represent a fixed date for an international visit. While the majority
                            of these explanations seem reasonable, when schedules are not
                            sequenced properly, the slack values and the calculated critical path will
                            not be valid.


Project Did Not Conducted   An IBR was not conducted at the project level. Project officials said
an Integrated Baseline      SOFIA, an in-house project, did not begin collecting EVM data until very
Review                      late in the development phase, and because of this an IBR was not
                            conducted. Officials added although SOFIA did not conduct a project-
                            level IBR, the EVM baseline was established concurrently with an Agency
                            approved re-plan and joint confidence level analysis in the 2009/2010
                            time frame, and was reviewed by a Standing Review Board. Though the


                            10
                              A hammock activity is a way to represent level of effort activities in the schedule.
                            Having no set duration of its own, the hammock activity duration is determined by the
                            number of days between the beginning of the first activity and the end of the last activity in
                            the group-not the sum of the activities' durations.




                            Page 92                                                                     GAO-13-22 NASA
                          Appendix III: Case Studies of Selected
                          Projects’ Implementation of Earned Value
                          Management




                          Standing Review Board review satisfied some of the objectives of an IBR,
                          including confirmation of the schedule and budget baselines (e.g.,
                          schedule review, risk review, key milestones identified), it did not address
                          the full IBR checklist (e.g. work authorizations, control account plans,
                          EVM system description).


EVM Surveillance Is Not   While formal surveillance is not occurring at the project level, EVM data
being Performed           assurance reviews are being performed monthly and quarterly by the
                          SOFIA project office representatives.


Data Resulting from the   We reviewed contract performance reports from August 2010 to August
EVM System Are            2011. We found various issues that bring into question the reliability of the
Somewhat Reliable         SOFIA Project EVM data. For example, we found negative values due to
                          an “over-reporting” of progress in previous months that was caused by a
                          problem with translating percent complete progress from the University
                          Space Research Association schedule to the SOFIA integrated master
                          schedule as well as corrections/modifications in costs posted for support
                          service contracts. In addition, we found other anomalies in the data that
                          SOFIA project officials explained were most likely due to mischarges by
                          employees, delayed cost postings, or employees continuing to use
                          charge codes inappropriately. While the cost impact of these problems
                          was not that large for any one WBS element, each of these issues causes
                          us to question the reliability of the data. According to project officials, the
                          variances that caused these anomalies did not meet the reporting
                          threshold. A variance analysis report provides a detailed, narrative report
                          explaining significant cost and schedule variances and other contract
                          problems and topics. Without this information, management cannot
                          understand the reasons for variances and the contractor’s plan for fixing
                          them. When variance analysis reports are not produced, the EVM data
                          will not be meaningful or useful as a management tool.

                          Figure 17 below illustrates that as of August 2011, the SOFIA project was
                          reporting a positive cumulative cost variance of $3.5 million and a
                          negative cumulative schedule variance of $6.6 million.




                          Page 93                                                         GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




                            Figure 17: GAO Earned Value Analysis of the SOFIA Project Data




                            Due to the cost and schedule variances we are forecasting a negative
                            variance at completion ranging from $1.4 million to $76 million.


                            The contractor met all three fundamental ANSI/EIA-748 practices
Airborne System             necessary for a reliable EVM system. In 2002, DCMA certified that the
Platform Contractor         contractor has an EVM system compliant with the ANSI/EIA standard.
Has a Certified EVM
System Compliant
with the ANSI/EIA
Standard

Project Conducted an        The project conducted an IBR in November 2007.
Integrated Baseline
Review of Airborne System
Platform Effort


                            Page 94                                                          GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




EVM Surveillance Is Being   It is unclear if EVM surveillance is being performed on the L-3 EVM data.
Performed

Data Resulting from the     We reviewed contract performance reports from August 2010 to July
EVM System Are              2011. We found various data anomalies that call into question the
                            reliability of the contractor’s EVM data for the contract performance
Somewhat Reliable           reports we reviewed. For example, we found negative values, actual cost
                            of work performed being reported without work being scheduled and/or
                            performed. Project officials stated that these anomalies were caused by
                            performance being taken in later months and also because these issues
                            did not trip the threshold reporting.

                            Figure 18 below illustrates that as of July 2011, the airborne system
                            platform contractor was reporting a positive cumulative cost variance of
                            $3.2 million and a negative schedule variance of $0.6 million.

                            Figure 18: GAO Earned Value Analysis of the SOFIA Airborne System Data




                            The favorable cost variance is due to efficiencies in project oversight and
                            engineering. The unfavorable cumulative schedule variance is being
                            driven by a lack of government furnished equipment supposed to be



                            Page 95                                                         GAO-13-22 NASA
                                           Appendix III: Case Studies of Selected
                                           Projects’ Implementation of Earned Value
                                           Management




                                           received from NASA that is causing a backlog of tasks. Due to the cost
                                           and schedule variances we are forecasting a positive variance at
                                           completion ranging from $3 million to $4 million.


                                           The Tracking and Data Relay Satellite (TDRS) System consists of in-orbit
Tracking and Data                          communication satellites stationed at geosynchronous altitude coupled
Relay Satellite                            with two ground stations located in New Mexico and Guam. The satellite
                                           network and ground stations provide mission services for near-Earth user
Replenishment                              satellites and orbiting vehicles. TDRS-K and L are the 11th and 12th
                                           satellites, respectively, to be built for the TDRS system. They will
                                           contribute to the existing network by providing continuous high bandwidth
                                           digital voice, video, and mission payload data, as well as health and
                                           safety data relay services to Earth-orbiting spacecraft such as the
                                           International Space Station and the Hubble Space Telescope. NASA is
                                           planning to launch TDRS-K in December 2012 followed by the TDRS-L
                                           launch in February 2014.

                                           NASA is collecting EVM data from both the spacecraft and sustainment
                                           efforts. In December 2007, NASA awarded a fixed price incentive contract
                                           to design, develop, fabricate, integrate, test, ship, provide launch support,
                                           conduct on-orbit checkout operations and provide sustaining engineering
                                           support for two spacecraft, TDRS-K and TDRS-L, to Boeing Satellite
                                           Systems, Inc (Boeing).




Table 14: Assessment of TDRS EVM Practices

Dollars in millions
                              Used a
                      certified EVM
                             system
                          compliant     Conducted an               EVM system              Data resulting from
                      with ANSI/EIA         integrated           surveillance is Budget at    the EVM system             GAO Variance at
                           standard    baseline review         being performed completion          are reliable              completion
Spacecraft                                                                                           $698               -$152 to -$185
Sustainment                                                                                               $5                            NAa
                                                                                                                 ◐
                                           Source: GAO analysis of contractor data as of August 2011.
                                           a
                                            NA - Not applicable means there was not enough data to forecast an estimate at completion




                                           Page 96                                                                         GAO-13-22 NASA
                        Appendix III: Case Studies of Selected
                        Projects’ Implementation of Earned Value
                        Management




                        The spacecraft contractor, Boeing, met the three fundamental ANSI/EIA-
Spacecraft Contractor   748 practices necessary for a reliable EVM system. Boeing has a certified
Has a Certified EVM     EVM system that complies with the ANSI/EIA EVM standard. Though the
                        contractor has a certified system, the implementation of that system is
System Compliant        questionable based on our findings below. We also assessed whether the
with the ANSI/EIA       spacecraft contractor’s EVM data against three ANSI/EIA guidelines.
                        These guidelines state that the authorized work elements for the project
Standard                should be defined typically using a WBS that has been tailored to the
                        project and the WBS is the same for the cost estimate, schedule, and
                        EVM. We found that the WBS used in the spacecraft integrated master
                        schedule was consistent with the WBS used for the EVM data.

                        The ANSI/EIA guidelines also state that projects should have a schedule
                        that describes the sequence of work by listing activities in the order in
                        which they are to be carried out and identifying significant task
                        interdependencies required to meet project requirements. We found some
                        sequencing issues in the contractor’s schedule. For example, 13 percent
                        of the remaining activities were constrained. When schedules are not
                        sequenced properly, float values and the calculated critical path will not
                        be valid. Project officials acknowledged the constraints and said they are
                        a result of having to adjust support activities due to spacecraft integration
                        and test delays as well as alignments to the current/actual manifest dates,
                        which differ from the Boeing contractual and launch readiness dates. In
                        addition, critical path metrics are generated and analyzed monthly to track
                        the Boeing performance against the critical path activities. Finally, the
                        ANSI/EIA guidelines state that a project should establish and maintain a
                        time-phased budget baseline to track cost and schedule variances in an
                        EVM system. Though resource loading the schedule is not required to
                        meet the ANSI/EIA guidelines, it is a best practice and therefore
                        resources should be accounted for in the schedule in order to develop
                        this baseline, according to the GAO cost guide. We found that the
                        schedule was not resource loaded. Project officials said the integrated
                        master schedule is resource loaded but not inside the Microsoft Project
                        schedule because it does not directly interface with the Boeing
                        financial/EVM system. They stated that the integrated master schedule is
                        produced using Microsoft Project and imported into a planning software
                        tool. After the resource loading effort is performed, the planning data is
                        transferred into the EVM system. The tool integrates the Microsoft Project
                        schedule with the Boeing financial system and the resource allocations
                        for each task. When adjustments are required Boeing again utilizes the
                        planning tool.




                        Page 97                                                       GAO-13-22 NASA
                            Appendix III: Case Studies of Selected
                            Projects’ Implementation of Earned Value
                            Management




Project Conducted an        An integrated baseline review was conducted in 2008 where 211 issues
Integrated Baseline         were raised. All of these issues have since been resolved and closed.
Review
EVM Surveillance Is Being   DCMA prepares a monthly EVM analysis report and performs continuous
Performed                   surveillance of Boeing’s EVM implementation by sampling various control
                            account managers for interviews about the process.


Data Resulting from the     We examined contract performance reports from August 2010 through
EVM System Are              August 2011. Figure 19 illustrates that as of August 2011, the project was
Somewhat Reliable           reporting a negative cumulative cost variance of approximately $131
                            million and a negative cumulative schedule variance of $7 million.

                            Figure 19: GAO Earned Value Analysis of the TDRS Spacecraft Project Data




                            The negative cumulative cost variance was being driven by higher staffing
                            levels to support integration, the Preliminary Design Review and Critical
                            Design Review, as well as an incorrect assessment of project
                            requirements and the inability to use heritage specifications. Labor costs
                            were also higher than expected due to part failures and the late



                            Page 98                                                          GAO-13-22 NASA
                     Appendix III: Case Studies of Selected
                     Projects’ Implementation of Earned Value
                     Management




                     completion of component qualifications. Extended test activities also
                     contributed to the cumulative negative cost variance. Finally, more than
                     expected resources were needed to complete board and slice designs,
                     generate drawings, and assemble and test components because of the
                     complexity of the design. Due to the negative cost and schedule variance
                     we are forecasting a variance at completion ranging from $152 million to
                     $185 million. Project officials said GAO’s independent variance at
                     completion gives the impression that NASA may request additional
                     funding to complete the TDRS K and L. Because this is a fixed price,
                     incentive fee contract, NASA officials said the agency is obligated only to
                     pay up to the price ceiling of the contract.


                     Boeing is the contractor for both the spacecraft and sustainment efforts.
TDRS Sustainment     As noted above, Boeing met all three fundamental practices for a reliable
Effort EVM Summary   EVM system.

                     In addition, the 2008 integrated baseline review was conducted for both
                     the spacecraft and sustainment effort. As well, the formal surveillance
                     performed by Boeing and the Defense Contract Management Agency
                     applies to the sustainment effort

                     We reviewed contract performance reports from April 2011 to August
                     2011. Because we had only 5 months of data, we were not able to
                     forecast an independent estimate at completion. Figure 20 below
                     illustrates that as of August 2011, the project was reporting a positive
                     cumulative cost variance of $0.08 million. Because there were no
                     variance analysis reports accompanying the sustaining effort, we were
                     unable to determine what was causing the positive cost variance. Project
                     officials stated that the Performance Measurement Baseline for this effort
                     was almost entirely level of effort, so minimal variances would be
                     occurring. They added that during the August 2011 time period, the
                     contract reflected an April 2012 launch date even though the launch was
                     being delayed. Consequently, Boeing’s reports were reflecting work
                     scheduled to occur in support of the earlier launch date when in fact very
                     little effort was being done. As a result, since minimal costs were
                     incurred, this resulted in a positive cumulative cost variance.




                     Page 99                                                      GAO-13-22 NASA
Appendix III: Case Studies of Selected
Projects’ Implementation of Earned Value
Management




Figure 20: GAO Earned Value Analysis of the TDRS Sustainment Data




Page 100                                                       GAO-13-22 NASA
Appendix IV: Comments from the National
              Appendix IV: Comments from the National
              Aeronautics and Space Administration



Aeronautics and Space Administration




              Page 101                                  GAO-13-22 NASA
Appendix IV: Comments from the National
Aeronautics and Space Administration




Page 102                                  GAO-13-22 NASA
Appendix IV: Comments from the National
Aeronautics and Space Administration




Page 103                                  GAO-13-22 NASA
Appendix V: GAO Contact and Staff
                  Appendix V: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov
GAO Contact
                  In addition to the contact named above, Shelby S. Oakley and Karen
Staff             Richey (Assistant Directors), Greg Campbell; Jennifer K. Echard; Tisha D.
Acknowledgments   Derricotte; Laura Greifner; Kristine R. Hassinger; Ben Jaskiewicz; William
                  Laing; Richard Lee; Eric Lofgren; Kenneth E. Patton; Jose A. Ramos;
                  Carrie W. Rogers; Stacey L. Steele; Roxanna T. Sun; and Umesh
                  Thakkar made key contributions to this report.




(121002)
                  Page 104                                                    GAO-13-22 NASA
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
                      Website: http://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in   E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.