oversight

Business Modernization: Disciplined Processes Needed to Better Manage NASA's Integrated Financial Management Program

Published by the Government Accountability Office on 2003-11-21.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to the Committee on Commerce,
                Science, and Transportation,
                U.S. Senate, and the Committee on
                Science, House of Representatives

November 2003
                BUSINESS
                MODERNIZATION
                Disciplined Processes
                Needed to Better
                Manage NASA’s
                Integrated Financial
                Management Program




GAO-04-118
                                                November 2003


                                                BUSINESS MODERNIZATION

                                                Disciplined Processes Needed to Better
Highlights of GAO-04-118, a report to           Manage NASA’s Integrated Financial
the Senate Committee on Commerce,
Science, and Transportation, and the            Management Program
House Committee on Science




The National Aeronautics and                    The uncertain reliability of cost estimates, optimistic schedules, and
Space Administration (NASA) has                 insufficient processes for ensuring adequate funding reserves have put
struggled to implement a fully                  NASA’s latest financial management modernization effort at risk. Over the
integrated financial management                 past several years, IFMP’s life-cycle cost estimates have fluctuated, and
system. The lack of such a system               NASA’s current estimate is 14 percent greater than the previous estimate.
has affected the agency’s ability to
control program costs, raising
                                                The reliability of these estimates is uncertain because disciplined cost-
concerns about the management of                estimating processes required by NASA and recognized as best practices
its most costly programs, including             were not used in preparing them. For example, IFMP’s current life-cycle cost
the space shuttle program and the               estimate did not include the full cost likely to be incurred during the life of
International Space Station.                    the program, including certain operations costs and costs to retire the
                                                system. In addition, NASA did not consistently use breakdowns of work in
In April 2000 NASA initiated the                preparing the cost estimate, as recommended by NASA guidance. In cases
Integrated Financial Management                 where work breakdowns were used, the agency did not always show the
Program (IFMP)—its third effort to              connection between the work breakdown estimates and the official program
improve the agencywide                          cost estimate. This has been a weakness since the inception.
management of its resources.
Implementation is expected by
fiscal year 2006 with an estimated
                                                Although more than half of the IFMP modules have been implemented—
life-cycle cost of nearly $1 billion.           including the Core Financial module, which is considered the backbone of
                                                IFMP—the system may not be fully implemented by the end of fiscal year
This report (1) assesses NASA’s                 2006 as planned. Efforts to complete the integrated system as quickly as
methodology for preparing the                   possible might have resulted in schedule margins that are insufficient to
current life-cycle cost estimate for            manage program challenges—such as personnel shortages, uncertainties
implementing IFMP, (2) determines               about software availability, and Office of Management and Budget (OMB)
whether NASA’s current schedule                 initiatives to implement electronic systems for agency business processes
is reasonable, and (3) evaluates                governmentwide. These OMB initiatives have put IFMP in a reactive mode
NASA’s processes for ensuring                   and are already affecting planning for the payroll, procurement, and travel
adequate cost contingencies.                    components of the integrated system, which could result in additional
                                                schedule delays and cost growth.

GAO is recommending that IFMP                   Finally, reserve funding for IFMP contingencies may be insufficient, which
follow best practices and NASA                  is particularly problematic, given the program’s unreliable cost estimates
guidance in preparing and updating              and optimistic schedule. One module—Budget Formulation—is already
the life-cycle cost estimate and                experiencing potential shortfalls in its reserves, and project officials
establish additional processes that             expressed concerns that the module’s functionality may have to be reduced.
would enable the agency to more                 Yet the program continues to establish funding reserves based on reserve
accurately estimate program cost
                                                levels set by other high-risk NASA programs, such as NASA’s space flight
and predict the impact of possible
undesired events. NASA concurred                program—not on analyses of the potential cost impact of risks and
with GAO’s recommendations for                  unknowns specific to IFMP, as required by NASA guidance. Moreover, the
corrective action.                              program did not quantify the cost impact of high-criticality risks—also
                                                required by NASA—or link its risks to funding reserves to help IFMP develop
                                                realistic budget estimates.
www.gao.gov/cgi-bin/getrpt?GAO-04-118.

To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Allen Li at
(202) 512-4841 or lia@gao.gov.
Contents


Letter                                                                                    1
              Results in Brief                                                            2
              Background                                                                  5
              Reliability of IFMP’s Current Life-Cycle Cost Estimate Is Uncertain
                Owing to a Lack of Disciplined Processes                                 7
              Program Schedule May Be Optimistic                                        11
              Processes Insufficient to Ensure Adequate Funding Set Aside
                for Contingencies                                                       13
              Conclusion                                                                16
              Recommendations for Executive Action                                      16
              Agency Comments and Our Evaluation                                        17
              Scope and Methodology                                                     18

Appendix I    Comments from the National Aeronautics and
              Space Administration                                                       20



Appendix II   GAO Contact and Staff Acknowledgments                                      23



Tables
              Table 1: IFMP’s System Modules and Their Status                             6
              Table 2: IFMP’s Cost Estimates for Life Cycle                               9
              Table 3: IFMP’s and NSSC’s Share of Life-Cycle Costs in the May
                       2003 Budget for Fiscal Years 1999-2010                             9


Figure
              Figure 1: IFMP’s Life-Cycle Cost Estimate Trend                             8




              Page i                                      GAO-04-118 Business Modernization
Abbreviations

GAO               General Accounting Office
IFMP              Integrated Financial Management Program
NASA              National Aeronautics and Space Administration
NSSC              NASA Shared Services Center
OMB               Office of Management and Budget
SAP               System Applications and Products
SEI               Software Engineering Institute
WBS               Work Breakdown Structure




This is a work of the U.S. government and is not subject to copyright protection in the
United States. It may be reproduced and distributed in its entirety without further
permission from GAO. However, because this work may contain copyrighted images or
other material, permission from the copyright holder may be necessary if you wish to
reproduce this material separately.




Page ii                                              GAO-04-118 Business Modernization
United States General Accounting Office
Washington, DC 20548




                                   November 21, 2003

                                   The Honorable John McCain
                                   Chairman
                                   The Honorable Ernest F. Hollings
                                   Ranking Minority Member
                                   Committee on Commerce, Science,
                                   and Transportation
                                   United States Senate

                                   The Honorable Sherwood L. Boehlert
                                   Chairman
                                   The Honorable Ralph M. Hall
                                   Ranking Minority Member
                                   Committee on Science
                                   House of Representatives

                                   About 90 percent of the National Aeronautics and Space Administration’s
                                   (NASA) annual budget is spent on contractors. Yet since 1990 we have
                                   identified NASA’s contract management as a high-risk area—in part,
                                   because the agency has failed to implement a modern, fully integrated
                                   financial management system. As we reported in January 2003, the lack of
                                   such a system has hampered NASA’s ability to oversee contracts; control
                                   program costs; and ensure an effective human capital management
                                   strategy, raising serious concerns about NASA’s management of its largest
                                   and most costly programs, including the space shuttle program and the
                                   International Space Station.1

                                   In April 2000 NASA initiated its third and most recent effort to implement
                                   a modernized financial management system: the Integrated Financial
                                   Management Program (IFMP).2 Through IFMP, NASA plans to employ
                                   multiple software applications to improve the agencywide management of
                                   its financial, physical, and human resources. NASA expects to complete
                                   IFMP’s implementation in fiscal year 2006 with an estimated life-cycle cost


                                   1
                                    See U.S. General Accounting Office, Major Management Challenges and Program
                                   Risks: National Aeronautics and Space Administration, GAO-03-114 (Washington, D.C.:
                                   January 2003).
                                   2
                                    NASA abandoned two earlier efforts after spending about $180 million over 12 years.



                                   Page 1                                               GAO-04-118 Business Modernization
                   totaling nearly $1 billion, including operations and maintenance costs.
                   However, several of the system’s modules remain to be developed, and the
                   program faces significant challenges in completing them as planned. Given
                   its previous failures, NASA is under tremendous pressure to ensure that
                   IFMP is implemented quickly and can achieve its goals.

                   In April 2003 we issued an interim report on IFMP,3 which found that
                   NASA was not following key best practices for acquiring and
                   implementing IFMP. As agreed, we continued our review in three areas:
                   (1) whether NASA is acquiring and implementing IFMP in the context of
                   an enterprise architecture, (2) the extent to which the Core Financial
                   module will address NASA’s external reporting requirements, and
                   (3) NASA’s life-cycle cost estimate and schedule for IFMP. We are
                   responding to the first two issues in separate reports.4 This report
                   addresses the third issue—IFMP’s life-cycle cost estimate and schedule.
                   Specifically, you asked that we (1) assess the reliability of NASA’s
                   methodology for preparing the current cost estimate for implementing
                   IFMP, (2) determine whether NASA’s current schedule is reasonable in
                   terms of progress to date and available resources, and (3) evaluate NASA’s
                   processes for ensuring the adequacy of reserve funding for contingencies
                   to mitigate the potential impact of identified program risks and unknowns.
                   In addition, we have summarized our findings on the three areas
                   previously cited in a separate report.5


                   The uncertain reliability of cost estimates, optimistic schedules, and
Results in Brief   insufficient processes for ensuring the adequacy of funding reserves have
                   put NASA’s latest financial management modernization effort at risk of
                   schedule delays and cost growth. The reliability of the current estimate—
                   which is 14 percent greater than the previous estimate established in


                   3
                    See U.S. General Accounting Office, Business Modernization: Improvements Needed
                   in Management of NASA’s Integrated Financial Management Program, GAO-03-507
                   (Washington, D.C.: Apr. 30, 2003).
                   4
                    See U.S. General Accounting Office, Business Modernization: NASA’s Integrated
                   Financial Management Program Does Not Fully Address Agency’s External Reporting
                   Issues, GAO-04-151 (Washington, D.C.: Nov. 21, 2003). Also, see U.S. General Accounting
                   Office, Information Technology: Architecture Needed to Guide NASA’s Financial
                   Management Modernization, GAO-04-43 (Washington, D.C.: Nov. 21, 2003).
                   5
                   See U.S. General Accounting Office, Business Modernization: NASA Challenges in
                   Managing Its Integrated Financial Management Program, GAO-04-255 (Washington,
                   D.C.: Nov. 21, 2003).




                   Page 2                                              GAO-04-118 Business Modernization
February 2002—is uncertain because disciplined cost-estimating processes
required by NASA and recognized as best practices were not used in
preparing the estimate. Specifically, IFMP’s life-cycle cost6 estimate did
not include the full cost of all resources likely to be incurred during the
life of the program. In addition, NASA did not consistently use
breakdowns of the work to be performed in preparing the cost estimate, as
recommended by NASA guidance. Without using the work breakdowns to
prepare the cost estimate, NASA cannot ensure that all costs are
accounted for. In cases where work breakdowns were used, the agency
did not always provide a clear audit trail between the work breakdown
estimates and the program’s life-cycle cost estimate.

IFMP is scheduled for completion at the end of fiscal year 2006. However,
efforts to complete the integrated system as quickly as possible might
have resulted in schedule margins that are insufficient to manage program
challenges—such as personnel shortages and uncertainties about
software’s availability. In addition, Office of Management and Budget
(OMB) initiatives to implement electronic systems for agency business
processes---which are expected to improve agency operations
governmentwide—are affecting planning for the payroll, procurement, and
travel modules of the integrated system, which could result in additional
schedule delays and cost growth.

Finally, reserve funding for IFMP contingencies may be insufficient—
which is particularly problematic, given the uncertain reliability of cost
estimates and optimistic schedule for the program—because the program
did not consistently perform in-depth analyses of the potential cost impact
of risks and unknowns specific to IFMP, as required by NASA’s guidance.
Instead, the program established funding reserves on the basis of reserve
levels set by other high-risk NASA programs. Moreover, the program
did not quantify the cost impact of identified risks or link its risks to
funding reserves.

To help decision makers better assess all costs associated with operating
and implementing IFMP, we are recommending that the program use
current processes dictated by best practices and NASA guidance for
preparing and updating the life-cycle cost estimate as well as establish


6
 Life-cycle cost is the total of the direct, indirect, recurring, nonrecurring, and other
related expenses incurred or estimated to be incurred in the design, development,
verification, production, operation, maintenance, support, and retirement of a system
over its planned life.




Page 3                                                  GAO-04-118 Business Modernization
additional disciplined processes to better ensure that the agency more
accurately estimate program cost, predict the impact of possible undesired
events, and plan accordingly.

In written comments on a draft of this report, NASA’s Deputy
Administrator concurred with our recommendations for corrective action
but noted that all IFMP projects completed to date stayed within budget,
were completed ahead of schedule, and delivered the committed scope.
We do not agree that this is the case for the Core Financial module, which
NASA describes as the “backbone” of the program. When NASA
announced the implementation of the Core Financial module at all of its
centers, only about two-thirds of the financial events needed for day-to-
day financial operations and external reporting had been implemented. In
addition, we found that NASA deferred the implementation of other key
Core Financial module capabilities and created new problems in recording
certain financial transactions.7 Thus, full functionality of the system has
been deferred, increasing the risk of additional costs and potentially
affecting the implementation of future modules. NASA’s detailed
comments also noted that the program used a business case methodology
and professional cost estimators to perform cost and risk assessments. As
discussed in this report, the audit trail from these assessments to the
program’s cost estimates required by NASA guidance and best practices is
not clear. NASA’s detailed comments are included as appendix I.




7
See GAO-04-151.




Page 4                                      GAO-04-118 Business Modernization
             For more than a decade, we have reported that the lack of a modern
Background   integrated financial management system to produce accurate and reliable
             information has hampered NASA’s ability to oversee contracts and
             develop good cost estimates for NASA’s programs. In 1990 NASA’s lack of
             effective systems and processes for overseeing contractor’s activities
             prompted us to identify NASA’s contract management as a high-risk area.
             In July 2002 we reported that the accuracy of NASA’s $5 billion cost
             growth estimate for the International Space Station was questionable and
             that the agency might have difficulty preparing a reliable life-cycle cost
             estimate because a modern integrated financial management system
             was not available to track and maintain the data needed for estimating and
             controlling costs. 8 NASA’s lack of a fully integrated financial management
             system has also hurt the agency’s ability to collect, maintain, and report
             the full cost of its projects and programs. For example, in March 2002 we
             testified that NASA was unable to provide us with detailed support for the
             amounts that it reported to the Congress as obligated against space station
             and related shuttle program cost limits as required by the National
             Aeronautics and Space Administration Authorization Act of 2000.9

             IFMP is designed as an integrated system to replace the separate
             and incompatible financial management systems used by NASA’s
             10 centers.10 According to the IFMP Program Director, the new system
             will provide better decision data, consistent information across centers,
             and improved functionality.

             Unlike NASA’s previous efforts to modernize its financial management
             system, IFMP does not rely on a single contractor. NASA selected System
             Applications and Products (SAP) to provide its “best of suite” software11


             8
              See U.S. General Accounting Office, Space Station: Actions Under Way to Manage Cost,
             but Significant Challenges Remain, GAO-02-735 (Washington, D.C.: July 17, 2002).
             9
              See U.S. General Accounting Office, National Aeronautics and Space Administration:
             Leadership and Systems Needed to Effect Financial Management Improvements,
             GAO-02-551T (Washington, D.C.: Mar. 20, 2002).
             10
               NASA is composed of headquarters offices; nine centers located around the country; and
             the Jet Propulsion Laboratory, which is operated by the California Institute of Technology.
             For the purpose of this report, we treat the Jet Propulsion Laboratory as a center.
             11
               When acquiring and implementing commercial hardware and software solutions,
             organizations can generally pursue one of two basic approaches: An organization can opt
             for a single package of already integrated software components, which is referred to as the
             “best of suite” approach, or it can opt for different software components from different
             vendors, which is referred to as the “best of breed” approach. “Best of suite” components
             are easier and less costly to integrate.



             Page 5                                                GAO-04-118 Business Modernization
                                            and contracted for implementation services under a separate contract.
                                            NASA has also broken the project into modules that will be implemented
                                            individually—instead of all at once—on the basis of the availability of
                                            proven commercial-off-the-shelf software products. IFMP initially
                                            segmented implementation into 14 modules but has since reorganized the
                                            program into 9. Some of these modules may be further broken out and
                                            others added, depending on the scope of OMB’s e-Government initiatives12
                                            and other considerations. Table 1 describes the modules that currently
                                            comprise the system and their status.

Table 1: IFMP’s System Modules and Their Status

 Module                    Description                                                                    NASA-reported status
 Position Description      Web-based tool that enables the generation and classification of job           Implemented September
 Management                descriptions and automatic generation of associated documents.                 2002.
 Resume Management         Web-based application that allows applicants to apply for jobs online.         Implemented November 2001.
 Erasmus                   Web-based system providing financial performance information on NASA’s Implemented November 2002.
                           programs and projects in a standardized format.
 Travel Management         Comprehensive system to streamline and unify the NASA employee travel          Implemented April 2003.
                           system and to improve traveler and vendor reimbursement.
 Core Financial            Accounting and financial reporting system that serves as the “backbone” to Implemented June 2003.
                           the integrated system.
 Budget Formulation        Web-based tool to formulate project, program, institutional, enterprise, and   Planned completion of
                           agency-level budget requirements.                                              February 2004.
 Integrated Asset          System to manage NASA’s physical assets through functions such as              To begin in late 2003.
 Management                physical inventory and financial reporting.
 Contract Administration   Comprehensive tool to support procurement, receiving, invoicing, and           To begin in late 2004.
                           payment of materials for NASA.
 Human Resources           System allowing managers to fill positions with staff that possess the         To begin in late 2004.
 Management                appropriate skill sets and career goals.
Source: NASA.

                                            When NASA announced in June 2003 that the Core Financial module had
                                            been implemented at all of its centers, only about two-thirds of the
                                            financial events needed for day-to-day financial operations and external
                                            reporting had been implemented. In addition, we found that NASA
                                            deferred implementation of other key core financial module capabilities




                                            12
                                              OMB’s Electronic Government—or “e-Government”—initiatives advocate the use of
                                            Internet-based technologies governmentwide for agency business processes, such as
                                            payroll, travel management, and recruiting. The goal of these initiatives is to eliminate
                                            redundant systems and improve the government’s quality of customer service.




                                            Page 6                                                  GAO-04-118 Business Modernization
                        and created new problems in recording certain financial transactions.13
                        Thus, full functionality of the system has been deferred, increasing the risk
                        of additional costs and potentially affecting the implementation of future
                        modules.

                        As we reported in April 2003, NASA is not following key best practices for
                        acquiring and implementing IFMP. For example, NASA has not analyzed
                        the interdependencies between selected and proposed IFMP components,
                        and it does not have a methodology for doing so. By acquiring IFMP
                        components without first understanding system component relationships,
                        NASA has increased its risk of implementing a system that will not
                        optimize mission performance and will cost more and take longer to
                        implement than necessary. In addition, in implementing the Core Financial
                        module, NASA faces risks in the areas of user needs and requirements
                        management because the agency did not consider the information needs
                        of key system users and is relying on a requirements management process
                        that does not require the documentation of detailed system requirements
                        prior to system implementation and testing.


                        The reliability of the current life-cycle cost estimate—which has fluctuated
Reliability of IFMP’s   since the initial estimate and is 14 percent greater than the previous
Current Life-Cycle      estimate established in February 2002—is uncertain because disciplined
                        cost-estimating processes required by NASA and recognized as best
Cost Estimate Is        practices were not used in preparing the estimate. Specifically, IFMP’s
Uncertain Owing to a    life-cycle cost estimate did not include the full cost likely to be incurred
                        during the life of the program. In addition, breakdowns of work to
Lack of                 be performed—or Work Breakdown Structure (WBS)14—were not
Disciplined Processes   consistently used in preparing the cost estimate. In cases where work
                        breakdowns were used to prepare the estimate, the agency did not always
                        provide a clear audit trail. NASA has made some improvements in the
                        program’s financial management, such as hiring personnel to provide
                        oversight and consistency for the cost-estimating process. However, until
                        NASA uses more disciplined processes such as breakdowns of work in
                        preparing the program’s cost estimate, the reliability of the life-cycle cost
                        estimate will be uncertain and the program will have difficulty with
                        controlling costs.


                        13
                         See GAO-04-151.
                        14
                          A WBS is a method of organizing a program into logical subdivisions at lower and lower
                        levels of detail.




                        Page 7                                               GAO-04-118 Business Modernization
IFMP’s Life-Cycle     Since the program began, cost estimates for IFMP’s 10-year life cycle—
Cost Estimates Show   fiscal years 2001 through 2010—have fluctuated and increased overall, as
Overall Increase      shown in figure 1.

                      Figure 1: IFMP’s Life-Cycle Cost Estimate Trend




                      NASA’s current IFMP life-cycle cost estimate totals $982.7 million—an
                      increase of $121.8 million, or 14 percent, over the previous IFMP life-cycle
                      cost estimate. The estimate comprises IFMP direct program costs, NASA’s
                      enterprise support,15 and civil service salaries/benefits. (See table 2.)




                      15
                        NASA is organized into six strategic enterprises that function as primary business areas
                      for implementing NASA’s mission. Each enterprise draws on the capabilities of several
                      NASA centers, while each center contributes to multiple enterprises. For example, the
                      Space Flight Enterprise has oversight over NASA’s human space flight program and
                      exercises management authority over the Johnson Space Center, Kennedy Space Center,
                      Marshall Space Flight Center, and Stennis Space Center.




                      Page 8                                                GAO-04-118 Business Modernization
Table 2: IFMP’s Cost Estimates for Life Cycle

 Dollars in millions of then-year dollars
 IFMP’s estimate component                       Feb. 2002 estimate         May 2003 estimate          Change in dollars       Percent change
 Direct program                                                 $644.8                    $635.3                   $(9.5)                 (1.5)
 Enterprise support                                              164.8                      189.4                    24.6                 15.0
 Civil Service salaries/benefits                                   51.3                     158.0                   106.7                208.0
 Total life-cycle cost                                          $860.9                    $982.7                  $121.8                  14.1
Sources: NASA (data); GAO (analysis).

                                                 Although direct program costs decreased by $9.5 million, these costs
                                                 were shifted to the enterprise support component of the estimate with
                                                 the program’s decision to fund only 1 year’s worth of operations and
                                                 maintenance, rather than 2 years’ worth from the direct program budget.
                                                 In addition, NASA anticipates that operations costs for fiscal years 2007
                                                 through 2010—estimated at $137.8 million—will be funded by the NASA
                                                 Shared Services Center (NSSC), a planned initiative to consolidate various
                                                 agency services such as purchasing and human resources. (See table 3.) As
                                                 a result, the fiscal year 2004 budget for the IFMP direct program portion of
                                                 implementing the system is $497.5 million.

Table 3: IFMP’s and NSSC’s Share of Life-Cycle Costs in the May 2003 Budget for Fiscal Years 1999-2010

                                            Development and implementation               Operations and maintenance
 IFMP estimate component                        (IFMP’s share, FY 1999-2006)            (NSSC’s share, FY 2007-2010)                      Total
 Direct program                                                            $497.5                                 $137.8                $635.3
 Enterprise support                                                         133.2                                   56.2                 189.4
 Civil Service salaries/benefits                                            125.3                                   32.7a                158.0
 Total life-cycle cost                                                     $756.0                                 $226.7                $982.7
Source: NASA.
                                                 a
                                                  This number is not final and is still being reviewed by NASA.


                                                 In March 2003 an independent cost estimate team concluded that there is
                                                 an 85 percent confidence level that the direct program portion can be
                                                 successfully completed with the available funding of $497.5 million.
                                                 However, the direct program portion represents only about half of the
                                                 total life-cycle cost estimate. In addition, the team’s conclusion was
                                                 contingent on two optimistic assumptions: that there would be no
                                                 schedule disruptions and no increase in requirements.




                                                 Page 9                                                      GAO-04-118 Business Modernization
Disciplined Processes       Reflecting OMB guidance16 and the best practices of government and
Required by NASA Were       industry leaders, NASA requires that life-cycle cost estimates be prepared
Not Used in Preparing       on a full-cost basis, that estimates be summarized according to the current
                            breakdown of work to be performed, and that major changes be tracked to
IFMP’s Cost Estimates       the life-cycle cost. OMB guidance calls for a disciplined budget process to
                            ensure that performance goals are met with the least risk and the lowest
                            life-cycle cost, which includes direct and indirect costs, operations and
                            maintenance, and disposal. The Software Engineering Institute (SEI)17
                            echoes the need for reliable cost-estimating processes in managing
                            software implementations—identifying tasks to be estimated, mapping the
                            estimates to the breakdown of work to be performed, and having a clear
                            audit trail are among SEI’s requisites for producing reliable cost estimates.

                            Despite NASA requirements and OMB and SEI guidance, IFMP did not
                            prepare a full life-cycle cost estimate—that is, all direct and indirect costs
                            for planning, procurement, operations and maintenance, and disposal
                            were not included. For example, the life-cycle cost estimate does not
                            include the following:

                        •   the cost to operate and maintain the system beyond 2010;18
                        •   the cost of retiring the system;
                        •   enterprise travel costs, which are provided monthly by the NASA centers;
                            and
                        •   the cost of nonleased NASA facilities for housing IFMP.


                            In addition, IFMP did not prepare WBS estimates for active modules—that
                            is, those currently being implemented. According to NASA guidance,
                            breaking down work into smaller units helps facilitate cost estimating
                            and project and contract management, and helps ensure that relevant
                            costs are not omitted. The guidance also states that the WBS should
                            encompass both in-house and contractor efforts. According to the IFMP
                            Deputy Program Director, WBS estimates are not prepared for active
                            modules because information such as contract task orders can be used to


                            16
                             Planning, Budgeting, Acquisition, and Management of Capital Assets, OMB Circular
                            A-11, Part 7 (June 2002).
                            17
                             SEI is a government-funded research organization that is widely considered an authority
                            on software implementations.
                            18
                             NASA assumed a 10-year life cycle beginning in fiscal year 2001, but the actual retirement
                            date for the system is unknown, according to the Deputy Program Director.




                            Page 10                                               GAO-04-118 Business Modernization
                    prepare the cost estimates. However, there is not one overriding contract
                    where each module is considered a deliverable at a fixed price. Rather,
                    numerous contracts at both the project and center level for a module’s
                    implementation—many of which can be awarded for a level of effort at
                    agreed-upon fixed rates at various phases in the implementation. Without
                    a WBS estimate for the project as a whole, NASA cannot ensure that all
                    relevant contractor costs are included in the cost estimate. In addition,
                    using contract task orders to prepare the cost estimate would not ensure
                    that government in-house costs are included in the life-cycle cost estimate.

                    Finally, for modules in the planning phase, the program utilized NASA’s
                    subject matter experts and professional cost estimators to prepare
                    business case analyses. However, although these analyses contained WBS
                    cost estimates, the audit trail from the WBS estimate to the program’s life-
                    cycle cost estimate was not always clear. Without a clear audit trail, it is
                    difficult to determine whether the differences between the detailed
                    WBS estimates and the official program cost estimate are appropriate.
                    The lack of a clear audit trail has been a weakness since the inception of
                    the program. For example, IFMP was unable to provide us with traceable
                    support for its baseline cost estimate for direct program costs.

                    NASA has made some improvements that should help the program prepare
                    better cost estimates. In May 2002 the NASA Administrator appointed an
                    executive to provide leadership and accountability in the direction and
                    operation of the system. The NASA headquarters program office also
                    hired a business manager to oversee and provide consistency for the
                    cost-estimating process and provide an analyst to review enterprise
                    support costs.


                    Although NASA guidance requires sufficient program schedule margins
Program Schedule    to manage risks, efforts to complete the integrated system as quickly as
May Be Optimistic   possible might have resulted in a schedule that is too compressed to
                    accommodate program challenges, such as personnel shortages and
                    uncertainties about software’s availability. If the program schedule margin
                    is too compressed, the program could incur additional risks, including
                    added cost growth as well as failure to meet IFMP’s schedule objectives.
                    OMB’s e-Government initiatives—which aim to streamline agency
                    business processes and eliminate redundant systems governmentwide—
                    could also provide challenges for NASA’s IFMP planning. As a result, the
                    program schedule may be optimistic.




                    Page 11                                      GAO-04-118 Business Modernization
While implementing the Core Financial module (see table 1), IFMP has
faced human resource challenges, and the program continues to face these
challenges with other modules. For example, personnel shortages at
Marshall Space Flight Center for several months affected the Core
Financial project and other projects. In this case, a schedule slip was
avoided, but during fiscal year 2002, the shortages resulted in nearly
$400,000 for extra hours worked by center employees. Human resource
challenges are also affecting the Budget Formulation module. The
simultaneous implementation of this module with the Core Financial
module—an action advised against by a contractor conducting a
lessons-learned study—placed heavy demand on already scarce resources
and added complexity to the program. As a result the schedule for
implementing the Budget Formulation module has already slipped.
Sometimes, relying more on contractor personnel can alleviate
shortfalls in civil service personnel, but a recent Budget Formulation
project status report indicated that the implementation contractor might
also have difficulties acquiring and/or retaining qualified personnel. The
implementation schedules for the remaining modules overlap, putting the
program at further risk of schedule slippages.

Uncertainty regarding software availability also puts the program at risk
for completing the integrated system on schedule. For example, complete
software solutions and requirements for IFMP’s Contract Administration
module have not yet been determined. Although contract-document-
generation software is available and tailored to meet the unique interface
and reporting requirements of the federal government,19 the “best of suite”
software solution—SAP—does not currently meet these requirements.
NASA faces the same challenge with IFMP’s Human Resources
Management module. NASA’s monthly status reports show that the
program is working with SAP to develop a software solution for the
Human Resources Management module that will meet federal government
requirements, but the outcome is uncertain. In addition, the program could
adopt an e-Government solution for its Human Resources Management
module rather than the SAP solution.




19
  For example, according to the current business case analysis, NASA’s document-
generation system would have to meet several federal requirements, including providing
the General Services Administration’s Federal Procurement Data System and the National
Science Foundation’s Federal Assistance Awards Data System with data, along with
reports to the Department of Labor and the Small Business Administration.




Page 12                                            GAO-04-118 Business Modernization
                            Inserting e-Government solutions into IFMP planning—which calls
                            for using “best of suite” software—could create more difficult interface
                            development and a less-integrated system, thus interrupting the program’s
                            cost and schedule. E-Government initiatives are already affecting
                            NASA’s planning for the payroll, procurement, and travel modules in
                            the integrated system. For example, the payroll function, which was
                            once part of the Human Resources Management module, will likely
                            become a separate module under e-Government. Similarly, the Contract
                            Administration module has been split into two components: one for
                            procurement document generation, for which software is available
                            although requirements are not finalized, and one for the remainder of
                            NASA’s Contract Administration requirements, for which requirements
                            and software are currently unknown. Furthermore, e-Travel could replace
                            the Travel Management module, which has already been implemented.

                            According to the program’s fiscal year 2002 Independent Annual Review,
                            e-Government initiatives are forcing the program into a reactionary mode,
                            thus increasing risk to the program’s success. The review specifically
                            noted that (1) the benefits of a fully integrated system could be lost under
                            e-Government, (2) the scope of IFMP and timing of future projects’
                            implementation have become uncertain, and (3) cost increases and
                            schedule slippage to accommodate directives may occur.


                            In addition to the uncertain reliability of IFMP’s life-cycle cost estimates
Processes Insufficient      and optimistic schedules, NASA cannot ensure that the funding set aside
to Ensure Adequate          for program contingencies is sufficient because the program did not
                            consistently perform in-depth analyses of the potential cost impact of risks
Funding Set Aside           and unknowns specific to IFMP, as required by NASA guidance. Moreover,
for Contingencies           the program did not quantify the cost impact of identified risks, link its
                            risks to funding reserves, or consistently set aside cost contingencies for
                            these risks.


In-Depth Analysis Not       NASA guidance stipulates that programs incorporate financial reserves,
Performed in Establishing   schedule margins, and technical performance margins to provide the
Cost Contingencies          flexibility needed to manage risks. According to the guidance, financial
                            reserves are to be established and maintained commensurate with
                            programmatic, technical, cost, and schedule risks. In other words, cost
                            contingencies should be tailored to the specific risks associated with a




                            Page 13                                      GAO-04-118 Business Modernization
particular program or project. In addition, NASA guidance suggests that
tools such as Probabilistic Risk Assessment20 can help in analyzing risk.

Although NASA’s business case analyses include a risk assessment and
recommended reserve levels, we found no evidence that these
recommended levels were used in establishing the actual reserve levels for
the IFMP module projects. Regardless, the actual levels established did not
match the recommended levels in the business case analyses in most
cases. We found that reserves for some IFMP modules—both in the
planning and active phase—were based not on IFMP-specific risks but on
reserve levels for other high-risk NASA programs. For example, for a
number of IFMP modules, reserves were set at levels used for spacecraft
implementations—typically about 30 percent—because industry
experience showed that large cost overruns in system implementations
such as IFMP are common. Yet it is unclear whether this reserve margin
is adequate for IFMP because the effect of IFMP-specific risks and
assumptions—such as uncertainties relating to software, schedule, and
OMB’s e-Government initiatives—were not analyzed. In addition, some
of the enterprises supporting the module projects described their method
of establishing funding reserves as a combination of rules of thumb
and guesswork.

The Budget Formulation module has already experienced shortfalls in
its reserves, and project officials expressed concerns that the module’s
functionality may have to be reduced. As of April 2003, the module
had expended its baseline reserves, which were established at about
20 percent on the basis of the level of risk for space flight missions—not
on the risks specific to the module. Although the project was able to bring
its budget back into balance by obtaining an agreement with SAP to limit
overtime pay to time in excess of 50 hours per week, its remaining
reserves total only $83,000 to cover all contingencies—including those that
could require changes to the Budget Formulation module.




20
  Probabilistic risk assessment is a method of systematically examining complex
technical systems to measure both the likelihood that an undesired event will occur
and the consequences that will result.




Page 14                                              GAO-04-118 Business Modernization
Cost Impact of                NASA requires programs to quantify the cost impact of high-criticality
Identified Risks Not          risks21 and to determine to what extent reserves may be exhausted,
Quantified or Linked to       should the risks become reality. According to SEI, estimating the potential
                              cost and schedule impact for all identified risks is an element of good
Cost Contingencies            estimating practice. Quantifying the cost impact of identified risks and
                              clearly and consistently linking the risk database to funding reserves helps
                              programs develop realistic budget estimates.

                              While IFMP identifies program risks, analyzes their severity, and plans
                              mitigation actions, the program typically does not prepare a cost impact
                              analysis for identified risks nor does it consistently link identified risks to
                              funding reserves to ensure that funds are available, should the risk occur.
                              For example, in February 2003, the Travel Management Project found that
                              some components of the Travel Management module might not satisfy
                              individual centers, be funded, or be technically feasible. However, the
                              cost impact of this risk, as well as others, was not quantified. Similarly,
                              in June 2003, the Budget Formulation module did not quantify the
                              cost impact of a number of identified risks.22 Without estimating the
                              potential cost impact of these risks, NASA cannot determine whether
                              it has sufficient reserves to cover the risks—which is particularly
                              problematic for Budget Formulation, since virtually no reserves remain
                              for this module.

                              Furthermore, in its July 2003 monthly status report, the IFMP headquarters
                              office identified three high-criticality risks that could have a cost impact
                              on the overall program; however, no liens23 were set aside against reserves
                              for these risks:

                          •   Reductions to out-year budgets could affect the implementation of future
                              integrated modules or the ongoing evolution of existing modules.




                              21
                                Risk criticality is a function of the likelihood that an event will occur and the severity
                              of the consequences if the event does occur. The criticality of each risk will be identified
                              as low, medium, or high. Risks with high criticality are also known as primary risks and
                              typically have a high likelihood of occurrence and a high magnitude of impact.
                              22
                                A mitigation plan for the Budget Formulation module indicated that the project was in the
                              process of assessing the potential cost impact of four of its six high-criticality risks, which
                              the project manager confirmed.
                              23
                                A lien is a potential cost to a project, direct or indirect, which may or may not come to
                              fruition, for which a portion of funding reserves is set aside.




                              Page 15                                                 GAO-04-118 Business Modernization
                  •   An e-Government solution may be adopted for human resources
                      management rather than the IFMP solution, resulting in more difficult
                      interface development and a less-than-integrated solution.
                  •   E-Government initiatives and policy decisions could disrupt IFMP
                      modules, resulting in delays or additional resource impacts.

                      An independent cost estimate team identified and quantified the impact of
                      two IFMP program risks, indicating that the cost and schedule impact of a
                      risk on a program or project can be sizeable. First, the team identified a
                      high-probability risk that NASA’s “full cost requirement”—in which all
                      direct and indirect agency costs, including civil service personnel costs,
                      are tied to individual programs and projects—could affect the Budget
                      Formulation module.24 The team estimated this risk at $2 million to
                      $3 million, with a potential schedule slip of 3 to 6 months. The Budget
                      Formulation Project is currently trying to determine what impact it may
                      have. The second risk identified by the independent cost review team—
                      that the Core Financial module may be transitioned to operations
                      before all integration points are addressed—could be more costly.
                      The team estimated this risk at $10.5 million to $20 million, also with a
                      potential 3- to 6-month schedule slip. However, the team considered this
                      risk as having a low probability of occurrence.

                      NASA is at a critical juncture and faces major challenges in improving
Conclusion            contract management and controlling costs. These challenges seriously
                      affect the agency’s ability to effectively manage its largest and most
                      costly programs. A modern integrated financial management system, as
                      envisioned in IFMP, is critical to ensuring that NASA has accurate and
                      reliable information to successfully meet these challenges. NASA has
                      made some improvements during the past year, such as hiring personnel
                      to provide the cost-estimating process with oversight and consistency.
                      However, if IFMP continues to ignore disciplined processes in estimating
                      program costs and impacts, it is unlikely that the program will meet
                      its goals.


                      To ensure that IFMP’s life-cycle cost estimate conforms to NASA guidance
Recommendations for   and best practices, we recommend that the NASA Administrator direct
Executive Action      IFMP to do the following:




                      24
                       NASA was to implement its full cost initiative October 1, 2003.




                      Page 16                                               GAO-04-118 Business Modernization
                     •   Prepare cost estimates by the current Work Breakdown Structure for the
                         remaining modules.
                     •   Provide a clear audit trail between detailed WBS estimates and the
                         program’s cost estimate for the remaining modules.
                     •   Prepare a full life-cycle cost estimate for the entire IFMP that meets
                         NASA’s life-cycle cost and full cost guidance.

                         To ensure that contingencies are funded in accordance with NASA
                         guidance and best practices, we recommend that the NASA Administrator
                         direct IFMP to do the following:

                     •   Utilize a systematic, logical, and comprehensive tool, such as Probabilistic
                         Risk Assessment, in establishing the level of financial reserves for the
                         remaining module projects and tailor the analysis to risks specific to IFMP.
                     •   Quantify the cost impact of at least all risks with a high likelihood of
                         occurrence and a high magnitude of impact to facilitate the continuing
                         analysis necessary to maintain adequate reserve levels.
                     •   Establish a clear link between the program’s risk database and financial
                         reserves.


                         Although NASA concurred with our recommendations for corrective
Agency Comments          action, NASA indicated that its current processes are adequate for (1)
and Our Evaluation       preparing WBS cost estimates, (2) estimating life-cycle costs, and (3)
                         establishing reserves on the basis of IFMP-specific risks. The agency cited
                         its business case analyses as the methodology through which it is
                         accomplishing these tasks.

                         We disagree that NASA’s current processes are adequate, and our
                         recommendations are aimed at improving these processes. As discussed in
                         this report, while NASA prepares WBS cost estimates for IFMP modules in
                         the planning phases by using business case analyses, it does not prepare
                         WBS cost estimates for active modules. And although IFMP indicates that
                         preparing cost estimates by using contract task orders is an appropriate
                         methodology, this approach will not ensure that all relevant costs,
                         including both contractor and government in-house costs, are included in
                         the life-cycle cost estimate. Regarding contract costs, there is not one
                         overriding contract where each module is considered a deliverable at a
                         fixed price. Rather, there are numerous contracts at both the project and
                         center level for implementing modules—many of which can be awarded
                         for a level of effort at agreed-upon fixed rates at various phases in the
                         implementation. Without a WBS estimate for the project as a whole, NASA
                         cannot ensure that all relevant contractor costs are included in the cost



                         Page 17                                     GAO-04-118 Business Modernization
              estimate. In addition, using contract task orders to prepare the cost
              estimate would not ensure that government in-house costs are included in
              the life-cycle cost estimate.

              According to NASA, IFMP will improve its business case analyses by
              providing better estimates of operational costs through the expected life of
              the module, retirement costs, and other full life-cycle costs. However, as
              discussed in this report, an audit trail is needed between the detailed
              estimates contained in the business case analyses and the program’s life-
              cycle cost estimate to ensure that these improvements are reflected in the
              program’s official cost estimate.

              Finally, as discussed in this report, although NASA’s business case
              analyses include recommended reserve levels, we found no evidence that
              these recommended levels were used in establishing the actual reserve
              levels for the IFMP module projects. Regardless, the actual levels
              established did not match the recommended levels in most cases. We
              found that the program established funding reserves on the basis of
              reserve levels set by other high-risk NASA programs, rather than on IFMP-
              specific risks as required by NASA guidance.


              To assess the reliability of NASA’s methodology for preparing the
Scope and     current cost estimate for IFMP, we reviewed program and project-level
Methodology   documentation to obtain an understanding of NASA’s current cost
              estimate and its major components and the methodology used to develop
              the estimate. We also interviewed program and project officials to clarify
              our understanding of the cost estimate and how NASA derived it. In
              addition, we compared the program’s cost-estimating methodology with
              SEI best practices, OMB requirements, and NASA’s own procedures and
              guidance. Finally, we reviewed internal and independent analyses of the
              cost estimate. We did not attempt to validate NASA’s estimate; rather, we
              reviewed NASA’s processes for preparing its estimate.

              To determine whether NASA’s current schedule is reasonable in terms
              of progress to date and available resources, we reviewed the program’s
              schedule objectives and NASA’s policies for managing program and
              project schedules. We monitored the schedule and risks to the schedule
              through our review of the program’s monthly status reports and internal
              NASA briefings. We interviewed program and project officials to ascertain
              NASA’s progress against the schedule.




              Page 18                                     GAO-04-118 Business Modernization
To evaluate NASA’s processes for ensuring the adequacy of cost
contingencies to mitigate the potential impact of identified program risks
and unknowns, we reviewed governmentwide and NASA policies and
SEI best practices for managing risk and establishing cost contingencies.
We also interviewed program officials at NASA headquarters and project
managers to obtain an understanding of how reserve levels were
established and maintained for the program. We then compared IFMP’s
processes for ensuring adequate cost contingencies with processes
dictated by OMB and NASA guidance and by best practices.

To accomplish our work, we visited NASA headquarters, Washington,
D.C.; Marshall Space Flight Center, Alabama; and Goddard Space
Flight Center, Maryland. We also contacted officials at Glenn Research
Center, Ohio.

We performed our review from April through September 2003 in
accordance with generally accepted government auditing standards.


As agreed with your offices, unless you announce its contents earlier,
we will not distribute this report further until 30 days from its date. At
that time, we will send copies to interested congressional committees;
the NASA Administrator; and the Director, Office of Management and
Budget. We will make copies available to others upon request. In addition,
the report will be available at no charge on the GAO Web site
at http://www.gao.gov.

If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841 or lia@gao.gov. Key contributors to this
report are acknowledged in appendix I.




Allen Li
Director
Acquisition and Sourcing Management




Page 19                                     GAO-04-118 Business Modernization
              Appendix I: Comments from the National Aeronautics and Space Administration
Appendix I: Comments from the National
Aeronautics and Space Administration




              Page 20                                           GAO-04-118 Business Modernization
Appendix I: Comments from the National Aeronautics and Space Administration




Page 21                                           GAO-04-118 Business Modernization
Appendix I: Comments from the National Aeronautics and Space Administration




Page 22                                           GAO-04-118 Business Modernization
                             Appendix II: GAO Contact and Staff Acknowledgments
Appendix II: GAO Contact and Staff
Acknowledgments

                  Allen Li (202) 512-3600
GAO Contact
                  Staff making key contributions to this report were Jerry Herley, Erin
Acknowledgments   Schoening, LaTonya Miller, and Karen Sloan.




(120244)

                  Page 23                                         GAO-04-118 Business Modernization
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail
                         alerts” under the “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548