oversight

Business Modernization: Improvements Needed in Management of NASA's Integrated Financial Management Program

Published by the Government Accountability Office on 2003-04-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to the Committee on Commerce,
             Science, and Transportation, U.S. Senate,
             and the Committee on Science, House of
             Representatives

April 2003
             BUSINESS
             MODERNIZATION
             Improvements Needed
             in Management of
             NASA’s Integrated
             Financial Management
             Program




GAO-03-507
             a
                                               April 2003


                                               BUSINESS MODERNIZATION

                                               Improvements Needed in Management of
Highlights of GAO-03-507, a report to the      NASA's Integrated Financial Management
Committee on Commerce, Science, and
Transportation, U.S. Senate, and the           Program
Committee on Science, House of
Representatives




 The National Aeronautics and
 Space Administration’s (NASA)
                                               The core financial module, if implemented as planned, may provide some
 nonintegrated financial                       improvement to NASA’s accounting system environment. However, NASA is
 management systems have                       not following key best practices for acquiring and implementing IFMP. In
 weakened its ability to oversee its           acquiring IFMP components, NASA is facing risks in understanding
 contractors, and its contract                 dependencies among commercial components. NASA has not analyzed the
 management has been on GAO’s                  interdependencies among selected and proposed IFMP components, and it
 high-risk list since 1990. In April           does not have a methodology for doing so. For programs like IFMP, which
 2000, NASA began its Integrated               involve building a system from commercial components, it is essential to
 Financial Management Program                  understand the characteristics and credentials of each component to select
 (IFMP), its third attempt in recent           ones that are compatible and can be integrated without having to build and
 years at modernizing financial                maintain expensive interfaces. By acquiring IFMP components without first
 management processes and
 systems. GAO was asked to review
                                               understanding system component relationships, NASA has increased its
 whether NASA was following key                risks of implementing a system that will not optimize mission performance
 best practices in acquiring IFMP              and will cost more and take longer to implement than necessary.
 components and implementing one
 of the first components—the core              In implementing the core financial module, NASA is facing risks in two
 financial module.                             additional areas:

                                               •   User needs. NASA did not consider the information needs of key
                                                   system users and deferred addressing the requirements of program
 GAO is recommending that NASA                     managers, cost estimators, and the Congress. Although this module
 develop and implement (1) a short-                should eliminate NASA’s separate, incompatible accounting systems,
 term plan to identify and mitigate
                                                   little has been done to reengineer acquisition management processes.
 the risks currently associated with
 relying on already deployed IFMP                  Program managers and cost estimators indicated that they will continue
 commercial components and (2) a                   to rely on other means to capture the data needed to manage programs
 longer term strategy for acquiring                such as the International Space Station.
 additional IFMP components that               •   Requirements management. NASA is relying on a requirements
 includes implementing a                           management process that does not require documentation of detailed
 methodology for commercial                        system requirements prior to system implementation and testing. Over
 system component dependency                       80 percent of the requirements GAO reviewed lacked specificity, and
 analysis. NASA agreed with GAO’s                  several could not be traced among various documents. These defects
 recommendation related to a short-                also significantly impaired the testing phase of the system
 term plan but disagreed with many                 implementation effort. Further, NASA has not implemented metrics to
 of the findings related to user
 needs and requirements
                                                   help gauge the effectiveness of its requirements management process.
 management. NASA also agreed                      NASA’s approach will likely result in increasing amounts of time spent
 with the importance of having an                  on costly rework and reduced progress.
 approach for acquiring additional
 IFMP components, but stated that              Unless these issues are successfully addressed, NASA is at increased risk of
 it already has an effective strategy          having IFMP become its third unsuccessful attempt to transform its financial
 in place. GAO reaffirms its                   management and business operations.
 recommendations.

www.gao.gov/cgi-bin/getrpt?GAO-03-507.

To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Gregory D.
Kutz at (202) 512-9095 or kutzg@gao.gov.
Contents



Letter                                                                                                  1
                             Results in Brief                                                           3
                             Background                                                                 6
                             NASA’s Acquisition Management Strategy Does Not Include
                               Analyzing Component Interdependencies                                    9
                             Core Financial Module Does Not Fully Address Key User
                               Information Requirements                                                11
                             NASA’s Requirements Management Process for the Core Financial
                               Module Is Ineffective                                                   21
                             Conclusions                                                               36
                             Recommendations for Executive Action                                      37
                             Agency Comments and Our Evaluation                                        38


Appendixes
              Appendix I:    Objectives, Scope, and Methodology                                        47
             Appendix II:    Comments from the National Aeronautics and Space
                             Administration                                                            51
                             GAO Comments                                                              70
             Appendix III:   GAO Contacts and Staff Acknowledgments                                    73
                             GAO Contacts                                                              73
                             Acknowledgments                                                           73


Tables                       Table 1: Five Contractors and Their Responsibilities                       8


Figures                      Figure 1: Space Shuttle Flight Operations Contract                        17
                             Figure 2: Example of Level of Detail Reported versus That
                                       Required by Cost Estimators                                     19
                             Figure 3: System Requirements for the “Manage Accounts Payable”
                                       Process                                                         27
                             Figure 4: Requirements for “Validate Payment” Subprocess                  28
                             Figure 5: Relationship between Requirements Development and
                                       Testing                                                         30




                             Page i                                                 GAO-03-507 NASA's IFMP
Contents




Abbreviations

CSC          Computer Services Corporation
ERP          Enterprise Resource Planning
FFMIA        Federal Financial Management Improvement Act
IBM          International Business Machines
IEEE         Institute of Electrical and Electronics Engineers
IFMP         Integrated Financial Management Program
IMCE         International Space Station Management and Cost Evaluation
ISS          International Space Station
NASA         National Aeronautics and Space Administration
OMB          Office of Management and Budget
SEI          Software Engineering Institute


 This is a work of the U.S. Government and is not subject to copyright protection in the
 United States. It may be reproduced and distributed in its entirety without further
 permission from GAO. It may contain copyrighted graphics, images or other materials.
 Permission from the copyright holder may be necessary should you wish to reproduce
 copyrighted materials separately from GAO’s product.




Page ii                                                          GAO-03-507 NASA's IFMP
A
United States General Accounting Office
Washington, D.C. 20548



                                    April 30, 2003                                                                                 Leter




                                    The Honorable John McCain
                                    Chairman
                                    The Honorable Ernest Hollings
                                    Ranking Minority Member
                                    Committee on Commerce, Science,
                                     and Transportation
                                    United States Senate

                                    The Honorable Sherwood L. Boehlert
                                    Chairman
                                    The Honorable Ralph Hall
                                    Ranking Minority Member
                                    Committee on Science
                                    House of Representatives

                                    Much of the National Aeronautics and Space Administration’s (NASA)
                                    success depends on the work of its contractors—on which it spends
                                    $12.7 billion, or 90 percent of its annual budget. For many years, NASA has
                                    not effectively overseen its contracts, principally because it has lacked
                                    accurate and reliable information on contract spending and performance
                                    and it has placed insufficient emphasis on end results, product
                                    performance, and cost control. Since 1990 we have identified NASA’s
                                    contract management function as an area at high risk.1 NASA’s ability to
                                    collect, maintain, and analyze cost and performance data has been
                                    weakened by nonintegrated, incompatible financial management systems
                                    and processes, and uneven and nonstandard cost-reporting capabilities.
                                    NASA made two efforts in the past to improve its financial management
                                    processes and develop a supporting system intended to produce the kind of
                                    accurate and reliable information needed to manage its contracts
                                    effectively, but both of these efforts were eventually abandoned after a
                                    total of 12 years and a reported $180 million in spending.



                                    1
                                      At that time, we began a special effort to review and report on the federal program areas
                                    that our work had identified as high risk because of vulnerabilities to waste, fraud, abuse,
                                    and mismanagement. We first issued our High-Risk Series in December 1992 and have
                                    continued to include NASA’s contract management as an area of high risk since. See U.S.
                                    General Accounting Office, High-Risk Series: NASA Contract Management, GAO/HR-93-11
                                    (Washington, D.C.: December 1992) and High-Risk Series: NASA Contract Management,
                                    GAO-03-119 (Washington, D.C.: January 2003).




                                    Page 1                                                            GAO-03-507 NASA's IFMP
In April 2000, NASA began its third attempt at modernizing its financial
management processes and systems. NASA has estimated the life cycle
cost of this effort through 2008 to be $861 million.2 This effort, known as
the Integrated Financial Management Program (IFMP), is expected to
produce an integrated, NASA-wide financial management system through
the acquisition and incremental implementation of commercial software
packages and related hardware and software components.3 Through the
proven business processes and centralized data management capabilities
embedded in these commercial components, NASA intends to reengineer
its management operations to “do business the way business does
business.” The core financial management module, which NASA considers
to be the backbone of IFMP, is currently operating at NASA headquarters
and 6 of NASA’s 10 centers4 and is expected to be fully operational in June
2003. According to NASA’s business case analysis for the system, the core
financial module will provide NASA’s financial and program managers with
timely, consistent, and reliable cost and performance information for
management decisions.

Given the importance of IFMP to NASA’s mission performance, you asked
us to review the program. The purpose of this report is to alert you now to
concerns we have based on our work to date and to provide NASA
management with constructive recommendations for improvement that it
can initiate as soon as possible. We are continuing our work and plan to
fully respond to your request later this year.

Our work to date has focused on whether NASA has management
processes in place for effective system acquisition and implementation.
This report addresses three issues concerning the acquisition of IFMP
components and the implementation of one of the first components—the
core financial module. Specifically, we determined whether NASA (1) was


2
  For this estimate, NASA has defined life cycle costs to include implementation efforts
through fiscal year 2008 and major upgrades, plus operation and support costs for each
system module for the first 2 years after the module goes live.
3
 The system is to consist of nine modules: core financial management, resume management,
travel management, position description management, human resource management,
payroll, budget formulation, contract administration, and asset management.
4
 NASA is comprised of its headquarters offices, nine Centers located throughout the
country, and the Jet Propulsion Laboratory. The Jet Propulsion Laboratory is operated by
the California Institute of Technology, but for purposes of this report, we treat the Jet
Propulsion Laboratory as a center.




Page 2                                                            GAO-03-507 NASA's IFMP
                   effectively evaluating the relationship among commercial systems
                   component options before acquiring them, (2) had adequately considered
                   the information needs of key users in implementing the core financial
                   module, and (3) had established and implemented an effective
                   requirements management process to support implementation of the core
                   financial module.

                   We performed our work from April 2002 through February 2003 in
                   accordance with generally accepted government auditing standards. We
                   had intended to include our assessment of a key element of NASA’s
                   acquisition strategy—whether NASA was acquiring IFMP components in
                   the context of an agencywide blueprint, commonly called an enterprise
                   architecture—in this report. However, because NASA did not provide the
                   data needed to complete our assessment until after the conclusion of our
                   fieldwork, we plan to address NASA’s enterprise architecture in a future
                   product. Details on our objectives, scope, and methodology are in
                   appendix I.



Results in Brief   If implemented as planned, IFMP may provide some improvement to
                   NASA’s current accounting system environment because it should eliminate
                   the separate, incompatible systems that have previously been used at each
                   of NASA’s 10 centers and should result in standardized accounting data.
                   However, NASA is not following key best practices for acquiring and
                   implementing IFMP. Specifically, NASA has not established an analytical
                   capability to guide and constrain its acquisition of IFMP commercial
                   components. Further, in implementing the core financial module
                   component, NASA has deferred addressing the needs of key system users
                   and has not properly developed detailed system requirements.
                   Consequently, the agency is at risk of making a substantial investment in a
                   system that will fall far short of its stated goal of providing meaningful and
                   reliable information to support effective program management and
                   congressional oversight.

                   NASA has not analyzed the interdependencies among selected and
                   proposed IFMP components, and it does not have a methodology for doing
                   so. For programs like IFMP, which involve building a system from
                   commercial components, it is essential to understand the characteristics
                   and credentials of each component in order to select ones that are
                   compatible and can be integrated without having to build and maintain
                   expensive interfaces. The alternative to such a structured and disciplined
                   approach to building a commercial component-based system is trial and



                   Page 3                                                  GAO-03-507 NASA's IFMP
error, which is fraught with risk. Although NASA has already acquired the
core financial module and three other IFMP commercial components, the
agency has not performed the analysis necessary to understand the logical
and physical relationships among the component parts it has acquired. By
acquiring these IFMP components without first understanding system
component relationships, NASA has increased its risks of implementing a
system that will not optimize mission performance, and will cost more and
take longer to implement than necessary.

For the core financial module, NASA did not consider the information
needs of key system users and deferred addressing the requirements of
program managers, cost estimators, and the Congress. Since 1990, we have
identified NASA’s contract management function as an area at high risk, in
part because of the lack of effective systems and processes for managing
and overseeing its procurement dollars, producing credible cost estimates,
and providing the Congress with appropriate visibility over its large,
complex programs. However, despite these previous problems, program
managers, cost estimators, and congressional staffs were not included in
defining system requirements for NASA’s core financial module. Instead,
NASA’s financial managers and accountants have primary responsibility for
this process. In addition, little has been done to reengineer acquisition
management processes, particularly with respect to the consistency and
detail of budget and actual cost data provided by contractors. Although
capable of accepting the data needed to satisfy the information needs of
these key users, NASA’s new core financial module is not being
implemented to accommodate this information. According to IFMP
program officials, they chose to defer certain system capabilities and
related user requirements in order to expedite implementation of the core
financial module. As a result, program managers and cost estimators told
us that they will not rely on the core financial module and instead will
continue to rely on other systems or use other labor-intensive means to
capture the data they need to manage programs such as the International
Space Station (ISS).

Further, NASA did not have an effective requirements management process
to support the implementation of the core financial module. Specifically,
NASA was relying on a systems requirements management process that did
not require documentation of detailed system requirements prior to system
implementation and testing. Although industry best practices and NASA’s
own system planning documents indicate that detailed requirements are
needed to serve as the basis for effective system testing, NASA’s approach
instead relied on certain subject matter experts’ knowledge of the detailed



Page 4                                                GAO-03-507 NASA's IFMP
requirements necessary to evaluate the functionality actually provided. As
a result of this approach, our review of the core financial module
requirements found that, for many of them, (1) the functionality to be
delivered was not adequately described or stated in a manner that allowed
for quantitative evaluation and (2) the traceability between the various
requirement documents was not maintained. Accordingly, the potential for
these requirements defects to result in costly rework is significant and
increases the risk that the project will not meet its cost, schedule, and
performance objectives. Because of the direct relationship between
requirements and testing, the lack of complete and unambiguous
requirements also significantly impairs the testing phase of the system
implementation effort. For example, the core financial module could not
process vendor invoices that contained over 200 line items—a common
occurrence on NASA’s large contracts—because NASA did not design an
appropriate test case. If NASA had documented its requirements, it would
have recognized that a properly designed test case had not been developed
to cover this necessary functionality. Furthermore, NASA has not
effectively implemented the types of metrics that can help the organization
understand the effectiveness of its requirements management process,
such as identifying and quantifying any weaknesses and then developing
the corrective actions needed.

We are making recommendations that address the need for NASA to
(1) develop and implement a short-term plan to identify and mitigate the
risks currently associated with relying on already deployed IFMP
commercial components and to expeditiously stabilize these components’
operation capability and performance, (2) as part of the short-term plan,
develop and properly document requirements, reengineer acquisition
management processes, and fully engage stakeholders—including program
managers, cost estimators, and the Congress—in the development of user
requirements, and (3) develop a longer term strategy for acquiring
additional IFMP components that includes implementing a methodology
for commercial system component dependency analysis.

In written comments, which are reprinted in appendix II, NASA concurred
with the need for a short-term plan but disagreed with most of our findings
related to user needs and requirements and testing. We remain convinced
that, as we have stated, NASA needs to (1) reengineer its acquisition
management processes to ensure that program and financial managers as
well as the Congress have needed budget and actual cost and schedule data
and (2) document detailed requirements to reduce, to acceptable levels, the
risks in implementing the selected processes. NASA also agreed with the



Page 5                                                GAO-03-507 NASA's IFMP
             importance of having an approach for acquiring additional IFMP
             components, but stated that it already has an effective strategy in place. We
             did not find convincing evidence to support NASA’s contention that it is
             using methodologically based dependency analysis—a best practice for
             implementing commercial component-based systems—in acquiring IFMP.



Background   NASA has a long and well-documented history of problems overseeing its
             procurement dollars, producing credible cost estimates, and providing the
             Congress with appropriate visibility over its large, complex programs. We
             first identified NASA’s contract management as an area at high risk in 1990
             because NASA lacked effective systems and processes for overseeing
             contractor activities. Over the past decade, other GAO, Inspector General,
             and task force reports have shown that NASA’s cost estimates lack
             credibility, in part because NASA does not collect the historical cost data
             needed to accurately project future costs or assess the validity of past
             estimates. Finally, because NASA had not provided the Congress with
             adequate visibility over the ISS program, the Congress had little advance
             warning when NASA reported that the estimated cost to complete the ISS
             had grown by about $5 billion in 1 year.

             Since we first identified NASA’s contract management as an area of high
             risk, we have reported that one of NASA’s most formidable barriers to
             sound contract management is the lack of a modern, integrated financial
             management system. NASA’s ability to collect, maintain, and analyze cost
             and performance data has been weakened by nonintegrated, incompatible
             accounting systems and processes, and uneven and nonstandard cost-
             reporting capabilities. The weaknesses in NASA’s financial management
             systems also caused its independent auditor, PricewaterhouseCoopers, to
             conclude for fiscal years 2001 and 2002 that NASA’s financial management
             systems do not substantially comply with the requirements of the Federal
             Financial Management Improvement Act (FFMIA). FFMIA builds on
             previous financial management reform legislation by emphasizing the need
             for agencies to have systems that can generate timely, accurate, and useful
             information with which to make informed decisions and to ensure
             accountability on an ongoing basis.

             While NASA’s efforts to design and implement a new financial management
             system certainly move NASA forward in this area, technology alone will not
             solve NASA’s problems. Our reviews, as well as NASA’s, show that finance
             is not viewed as an integral part of NASA’s program management decision
             process. Moreover, an independent task force created by NASA to review



             Page 6                                                 GAO-03-507 NASA's IFMP
and assess ISS costs, budget, and management reached a similar
conclusion. In its November 1, 2001, report, the International Space Station
Management and Cost Evaluation (IMCE) Task Force found that the ISS
program office does not collect the historical cost data needed to project
future costs accurately and thus perform major program-level financial
forecasting and strategic planning. The task force also reported that NASA’s
ability to forecast and plan is weakened by diverse and often incompatible
center-level accounting systems and uneven and non-standard cost
reporting capabilities. The IMCE Task Force also concluded that the
current weaknesses in financial reporting are a symptom, not a cause, of
the problem and that enhanced reporting capabilities, by way of a new
integrated financial management system, will not thoroughly solve the
problem. The root of the problem, according to the task force, is that
finance is not viewed as intrinsic to NASA’s program management decision
process.

NASA’s IFMP includes nine module projects supporting a range of financial,
administrative, and functional areas. According to NASA officials, of the
nine module projects, two are in operation, three are currently in
implementation, and four are future modules. The two projects in
operation are resume management and position description management;
the three projects in implementation are travel management, core financial,
and budget formulation; and the four future module projects are human
resources, payroll, asset management, and contract administration.

The core financial module project, which utilizes the SAP R/3 system, is the
backbone of IFMP and will become NASA’s standard, integrated accounting
system used agencywide. The other IFMP module projects will be
integrated/interfaced with the core financial module, where applicable. The
scope of the core financial module, when fully implemented, includes:
standard general ledger, budget execution, purchasing, accounts
receivable, accounts payable, and cost management. NASA plans to
implement the core financial module at all 10 NASA centers by June 2003.
The pilot for the core financial module—conducted at Marshall Space
Flight Center—was implemented in October 2002. NASA is rolling out or
deploying the core financial module at the other nine NASA centers and
headquarters in three waves. The first wave, which consisted of Glenn
Research Center, rolled out in October 2002. The second wave, which
consisted of NASA headquarters, Johnson Space Center, Kennedy Space
Center, and the Jet Propulsion Laboratory rolled out in February 2003.
Ames Research Center, considered a second wave center, rolled out in
April 2003. Finally, NASA plans to roll out the third wave, which consists of



Page 7                                                 GAO-03-507 NASA's IFMP
                       Dryden Flight Research Center, Goddard Space Flight Center, Langley
                       Research Center, and Stennis Space Center, in June 2003.



IFMP Acquisition       NASA is contracting with multiple companies to assist in the acquisition
Management Structure   management, integration, and implementation of its IFMP “system of
                       system components.” As shown in table 1, five contractors are assisting in
                       the integration and implementation of the core financial module. However,
                       none of these five contractors is responsible and accountable for
                       successfully implementing the entire IFMP system. Instead, NASA has
                       structured its IFMP acquisition so that NASA is the system integrator,
                       meaning that NASA is responsible for integrating multiple commercial
                       components and ensuring that they collectively perform in a manner that
                       meets the defined requirements.



                       Table 1: Five Contractors and Their Responsibilities

                       Entity                   Responsibility/function
                       Accenture                Implement the core financial module in accordance with agency
                                                requirements, including interfacing it with NASA’s existing systems
                                                environment.a
                       CSC (Computer            Support the operations, maintenance, and administration of the
                       Services                 new module, including integration efforts.
                       Corporation)
                       IBM (International       Develop training and user procedures and perform security and
                       Business Machines)       internal control reviews to ensure that the core financial module
                                                complies with accounting and financial reporting standards.
                       SAP                      Provide technical implementation support and training on NASA’s
                                                implementation of the core financial module.b
                       Titan Systems            Perform independent verification and validation of requirements
                       Corporation-Civil        and testing processes and results, such as tracing requirements to
                       Government               test cases.
                       Services Group
                       Source: NASA.
                       a
                       NASA plans to solicit additional contracts for implementation of other selected and acquired modules.
                       b
                        NASA has acquired SAP’s enterprise resource planning package and has thus far planned to
                       implement the core financial and budget formulation modules. NASA has also acquired three other
                       commercial software products—Travel Manager, Resumix, and Position Description Management.




                       Page 8                                                                  GAO-03-507 NASA's IFMP
NASA’s Acquisition    A key to effectively acquiring commercial component-based systems that
                      are intended to support agencywide business needs, like IFMP, is
Management Strategy   employing recognized acquisition management controls. One such control
Does Not Include      is to acquire system components only after deliberate and comprehensive
                      analysis and understanding of the components’ interdependencies.
Analyzing Component   Although NASA has already acquired the core financial module and three
Interdependencies     other IFMP commercial components, the agency has not performed the
                      analysis necessary to understand the logical and physical relationships
                      among the component parts it has acquired. By acquiring these IFMP
                      components without first understanding system component relationships,
                      NASA has increased its risks of implementing a system that will not
                      optimize mission performance and will cost more and take longer to
                      implement than necessary.

                      When acquiring a commercial component-based system or system of
                      systems, such as IFMP, industry best practices5 recognize the critical
                      importance of understanding the logical and physical relationships among
                      the component parts. To provide for this understanding, these practices
                      advocate that the system integrator, which in the case of IFMP is NASA,
                      employ an explicit methodology, including a risk-based process for
                      deciding among product alternatives, that collects and verifies information
                      about each component’s characteristics and credentials, evaluates the
                      dependencies and constraints among these components, and permits
                      informed decisions about which products to acquire and how to implement
                      them. This is necessary because commercial products are built around
                      each vendor’s functional and architectural assumptions and paradigms,
                      such as approaches to error handling and data access, and these
                      assumptions and paradigms are likely to be different among products from
                      different sources. Such differences complicate product integration.
                      Further, some commercial products have built-in dependencies with other
                      products that if not known can further complicate integration. For these
                      reasons, a structured and disciplined approach to systematically evaluating
                      product to product relationships is critical. The alternative to such a
                      structured and disciplined approach to building a commercial component-
                      based system is trial and error, which is fraught with risk.




                      5
                        See for example, Tricia Oberndorf, Lisa Brownsword, and Carol A. Sledge, Ph.D., An
                      Activity Framework for COTS-Based Systems, Technical Report CMU/SEI-2000-TR-010
                      (Pittsburgh, Pa.: Software Engineering Institute, Carnegie Mellon University, October 2000).




                      Page 9                                                            GAO-03-507 NASA's IFMP
In acquiring its IFMP components to date, NASA has not performed the
above-cited dependency analysis, and it does not have a methodology for
doing so. Despite this, NASA has acquired and is in the process of
implementing a commercial product (SAP’s R/3 core financial module) to
meet its needs in one business area (financial management), and it has
acquired three additional commercial products from three separate
vendors that are intended to meet its needs in other business areas (travel
management, resume management, and human capital position description
needs).6 Beyond the four products that it has already acquired, NASA plans
to acquire an unspecified number of additional commercial components
that are intended to meet its needs in other business areas. To integrate
those separate commercial products into a “system of system
components,” NASA has executed several contracts and plans to execute
more to build interfaces (hardware and software) to permit the
components to interoperate. For example, a contractor is currently
building an interface between the core financial module of the SAP product
and the travel manager product.

When acquiring and implementing commercial hardware and software
solutions, organizations can generally pursue one of two basic courses of
action. That is, an organization can opt for a single package of already
integrated software components, which is referred to as the “best of suite”
approach, or it can opt for different software components from different
vendors, which is referred to as the “best of breed” approach. NASA is
currently following the “best of breed” approach. According to the
Integration Office Deputy Program Manager, NASA has not performed
dependency analyses among the various components acquired to date, and
those being considered for later acquisition, because NASA’s initial
acquisition strategy was to acquire a single commercial solution (i.e., “best
of suite”) and thus it did not consider product interoperability to be a
concern. While NASA has since adopted a “best of breed” approach, the
Integration Office Deputy Program Manager stated that it still does not plan
to perform these analyses in the future because NASA will rely upon
commercial tools that support the development of interfaces between
commercial products, which the Integration Office Deputy Program
Manager claimed will make integration easy and relatively inexpensive and
negate the need for proactive dependency analysis. However, best


6
  NASA has acquired the following commercial products: (1) SAP AG’s R/3, version 4.62,
(2) Gelco’s Travel Manager, version 8.0, (3) Resumix, version 6, and (4) Avue Digital
Services’ Position Description Management, which is a subscription service.




Page 10                                                          GAO-03-507 NASA's IFMP
                        practices advocate that proactive dependency analysis and evaluation are
                        necessary for informed decision making regardless of whether integration
                        tools will be used, and particularly when a “best of breed” approach is
                        employed.

                        What this means is that NASA is implementing its “best of breed” approach
                        using trial and error. This reactive method does not allow for adequate
                        understanding of commercial product dependencies until the only
                        alternative to integrating them is building and maintaining complex
                        interfaces, which unnecessarily increase system acquisition and
                        maintenance costs, delay promised capabilities and benefits, and do not
                        optimize agency performance. The results of a recent study7 commissioned
                        by NASA recognize the added risk associated with the “best of breed”
                        approach, and thus the importance of proactive dependency analysis and
                        evaluation to minimize this risk. Specifically, the study states that NASA’s
                        “best of breed” approach will result in a higher total cost of ownership
                        because the agency will need to (1) acquire and maintain multiple software
                        licenses, (2) hire and maintain technical staff knowledgeable about each
                        commercial product, (3) build and maintain interfaces to integrate the
                        various products, and (4) provide training to system users on each
                        commercial product.



Core Financial Module   If implemented as planned, the core financial module may improve NASA’s
                        current system environment by eliminating the separate, incompatible
Does Not Fully          accounting systems that have been used at each of NASA’s 10 centers
Address Key User        previously. However, the core financial module currently being
                        implemented does not fully address the information requirements of key
Information             users, such as program managers, cost estimators, or the Congress. Our
Requirements            previous work at leading public and private sector organizations8 has
                        shown that user involvement and effectively reengineering business
                        processes are major factors in successfully implementing financial
                        management systems. In contrast, at NASA, key users such as program
                        managers and cost estimators were not involved in defining or
                        implementing NASA’s system requirements and have played a limited role


                        7
                        Gartner, Inc., A Report for NASA: IFMP Lessons Learned and Key Considerations for
                        Future Module Projects, January 20, 2003.
                        8
                        U.S. General Accounting Office, Executive Guide: Creating Value Through World-class
                        Financial Management, GAO/AIMD-00-134 (Washington, D.C.: April 2000).




                        Page 11                                                       GAO-03-507 NASA's IFMP
                             in all aspects of the implementation of the core financial module. Instead,
                             NASA’s financial managers and accountants have primary responsibility for
                             this process. Consequently, NASA has not effectively used this opportunity
                             to reengineer the way it does business and implement a financial
                             management system that addresses many of its most significant
                             management challenges, including improving contract management,
                             producing credible cost estimates, and providing the Congress with
                             appropriate visibility over its large, complex programs. According to IFMP
                             officials, they chose to forgo certain system capabilities to expedite
                             implementation of the core financial module and have stated that these
                             capabilities can be added at a later date. In the meantime, program
                             managers and cost estimators will continue to rely on other nonintegrated
                             systems outside of IFMP and use other labor-intensive means to capture
                             the data they need to manage programs such as the ISS.



If Implemented as Planned,   The core financial module, if implemented as planned, may provide some
Core Financial Module May    improvement to NASA’s current accounting system environment.
                             According to IFMP planning documents, the core financial module should
Provide Some
                             (1) eliminate much of the inconsistent data and lack of standardization,
Improvements                 (2) collect agency costs and allocate those costs to cost centers, including
                             civil service personnel costs, and (3) maintain a standard general ledger to
                             provide control over financial transactions, resource balances, and assets
                             and liabilities. If NASA is successful, the core financial module could
                             reduce the extensive amount of time and resources currently required to
                             consolidate NASA’s 10 different reporting entities and close the books each
                             accounting period. However, as discussed later, our findings relating to
                             NASA’s requirements management and testing processes may affect NASA’s
                             ability to achieve these improvements.



Key Users Were Not           The IFMP core financial module, although technologically capable of
Involved in the              meeting the needs of program managers, cost estimators, and the
                             Congress, is not being configured to do so because these key users have
Implementation of the Core   not been actively involved in the implementation of the module. Our
Financial Module             previous work at leading public and private sector organizations has shown
                             that user involvement in reengineering business processes and establishing
                             and implementing system requirements are major factors in successfully




                             Page 12                                                GAO-03-507 NASA's IFMP
                            implementing financial management systems.9 In fact, at these leading
                            organizations, not only did program and business managers participate in
                            the design and implementation of financial management systems, they
                            typically were responsible for driving the effort and played a key role in
                            reengineering core business processes. In contrast, at NASA, financial
                            managers and accountants have had primary responsibility for the
                            implementation of the core financial module, while the other key users
                            mentioned above have been largely excluded.

                            According to IFMP planning documents, financial managers and
                            accountants are considered direct customers responsible for the
                            administrative processes that will be reengineered and automated.
                            Therefore, these individuals, to date, have been engaged in defining system
                            requirements and priorities. On the other hand, stakeholders—including
                            program mangers, cost estimators, and the Congress—are described in
                            NASA documents as the ultimate beneficiaries of system improvements but
                            are not expected to be actively involved in the system’s implementation.
                            While NASA has formed teams to reengineer portions of the agency’s
                            administrative process, these teams primarily consisted of financial
                            managers. As a result, NASA has neither reengineered its core business
                            processes nor established adequate requirements of the system to address
                            many of its most significant management challenges, including improving
                            contract management, producing credible cost estimates, and providing the
                            Congress with appropriate visibility over its large, complex programs.



The Core Financial Module   The core financial module is not being implemented to provide program
Will Not Provide the        managers with the information they need to fully monitor the work being
                            performed by contractors. Based on our review of NASA’s three largest
Information Needed to
Manage Contracts




                            9
                             GAO/AIMD-00-134.




                            Page 13                                               GAO-03-507 NASA's IFMP
                                 space flight programs—Space Launch Initiative,10 ISS, and Space Shuttle—
                                 we found that the core financial module, as currently planned, will not
                                 accommodate much of the information provided by NASA’s contractors
                                 and needed by program managers to monitor contractor performance.
                                 Specifically, the core financial module is not being implemented to
                                 (1) accommodate the contract schedule information received from
                                 contractors and needed by program managers to monitor contractor
                                 performance and (2) maintain cost data at a sufficient level of detail for
                                 certain contracts.

Core Financial Module Does Not   To adequately oversee NASA’s largest contracts, program managers need
Integrate Cost and Schedule      reliable contract cost data—both budgeted and actual—and the ability to
Data Needed by Program           integrate these data with contract schedule information to monitor
Managers                         progress on the contract. However, because program managers were not
                                 involved in defining system requirements or reengineering business
                                 processes, the core financial module is not being designed to integrate cost
                                 and schedule data needed by program managers. As a result, program
                                 managers are resorting to using other systems that will result in additional
                                 cost over and above IFMP costs.

                                 The primary source of contract schedule information used by program
                                 managers comes directly from NASA’s contractors in the form of monthly
                                 hard copy or electronic cost and schedule performance reports. NASA
                                 tracks contract schedule status by comparing the budgeted and actual cost
                                 of work planned with budgeted and actual cost of work completed for
                                 specific time periods. The term “schedule” incorporates both the concept
                                 of status of work and whether a project or task is being completed within
                                 planned time frames. Depending on the nature of the work being
                                 performed, the method of measuring work progress varies. Work is
                                 measured in terms of tasks when a specific end product or end result is


                                 10
                                   During the time of our review, NASA was pursuing a program—known as the Space
                                 Launch Initiative—to build a new generation of space vehicles to replace its aging space
                                 shuttle. This was part of NASA’s broader plan for the future of space travel—known as
                                 NASA’s Integrated Space Transportation Plan. On October 21, 2002, NASA postponed further
                                 implementation of the program to focus on defining the Department of Defense’s role,
                                 determining future requirements of the ISS, and establishing the agency’s future space
                                 transportation needs. In November 2002, the administration submitted to the Congress an
                                 amendment to NASA’s fiscal year 2003 budget request to implement a new Integrated Space
                                 Transportation Plan. The new plan makes investments to extend the space shuttle’s
                                 operational life and refocuses the Space Launch Initiative program on developing an
                                 orbital space plane—which provides crew transfer capability to and from the space
                                 station—and next generation launch technology.




                                 Page 14                                                        GAO-03-507 NASA's IFMP
produced. But when work does not produce a specific end product or
result, level-of-effort or a more time-oriented method of measurement is
used. The type of information, level of detail, and reporting format
provided by contractors are determined during the contract negotiation
process and vary from contract to contract depending on the size,
complexity, and duration of the contract. In general, however, these reports
show contractor progress against cost and schedule targets set by the
program manager and against which contractor performance can be
measured. Contractors also report any significant variances from the
targets and explain how they will be mitigated.

NASA’s program managers need this contractor information to plan and
manage their programs effectively. However, the information from cost and
schedule performance reports is not recorded in the core financial module.
Instead, NASA uses only data from monthly contractor financial
management reports, commonly referred to as NASA form 533 reports, to
update the core financial module. NASA form 533 reports contain
estimated and actual contractor cost data but, according to NASA program
managers, do not contain the data needed to adequately assess schedule
performance. According to IFMP officials, the information needed to
perform cost and schedule analysis by program managers is outside the
scope of the core financial module and IFMP. IFMP program officials told
us that they chose to forgo certain system capabilities to expedite
implementation of the core financial module and have stated that these
capabilities can be added at a later date. However, NASA does not currently
have a plan for maintaining the data contained in cost and schedule
performance reports in the core financial module or IFMP.

Because contract schedule information is not currently maintained through
the core financial module, program managers will continue to rely on hard
copy reports, electronic spreadsheets, or other means to monitor
contractor performance. Several of NASA’s programs, including the Space
Launch Initiative and the ISS, are currently using other systems to monitor
contract cost and schedule performance, but these are stand-alone efforts
and have not been part of a coordinated NASA plan. Officials at Marshall
Space Flight Center have recognized the importance of maintaining
common cost and schedule performance data in a single integrated system
that is available to all NASA managers at all locations. As such, these
officials have proposed that NASA establish the system they currently use
as a NASA-wide standard.




Page 15                                                GAO-03-507 NASA's IFMP
                                  NASA has stated that the core financial module is expected to result in a
                                  single, integrated financial management system that is intended to serve
                                  the needs of its program managers. By not including the cost and schedule
                                  information needed by program managers in the core financial module,
                                  NASA risks operating with two sets of books—one that is used to report to
                                  management and the Congress and another that is used to manage NASA’s
                                  programs.

Core Financial Module Will Rely   Because NASA has not fundamentally changed the way it operates by
on Legacy Coding Structure        involving key users in business process reengineering efforts, the core
                                  financial module is not being implemented to capture cost information at
                                  the same level of detail that it is received from NASA’s contractors. Instead
                                  of implementing an accounting code structure that would meet the
                                  information needs of program managers, NASA has embedded the same
                                  accounting code structure that it uses in its legacy reporting system in the
                                  core financial module. As a result, the availability of detailed cost data is
                                  dependent on the adequacy of NASA’s legacy coding structure. In some
                                  cases, the cost information received by NASA on monthly contractor
                                  reports must be aggregated to a higher, less detailed level before it is
                                  posted against the old accounting code structure. For example, as shown in
                                  figure 1, program managers for the Space Shuttle receive monthly
                                  contractor reports on the space flight operations contract that track costs
                                  related to friction stir weld and propulsion safety upgrades separately.




                                  Page 16                                                GAO-03-507 NASA's IFMP
Figure 1: Space Shuttle Flight Operations Contract




                                           Flight                Space Shuttle
                                          Hardware              Flight Operations
                                            $100                       $100




   Level captured                    Flight hardware        Solid rocket
     in the core                         upgrades            booster
  financial module                          $40                 $60            Data to
                                                                                IFMP




    Level received             Flight hardware         Friction stir weld           Propulsion
                               safety upgrades             upgrades                 upgrades
  from contractors
                                      $40                     $10                      $10



Source: GAO analysis of NASA cost data.


Note: Amounts shown are for illustrative purposes only.


However, because the NASA legacy accounting code structure embedded
in the core financial module only tracks the cost of space shuttle flight
hardware upgrades, the more detailed costs that program managers need,
such as friction stir weld and propulsion safety upgrades, are not available
through the core financial module. According to NASA officials, the core
financial module is capable of capturing this more detailed cost data;
however, due to the complexity associated with converting detailed data
from the centers’ legacy systems, NASA has deferred this capability. While
this information is available to program managers from the contractor, it is
not available through the core financial module. In fact, on this particular
contract, program managers have access to the contractor’s system and,
therefore, have access to an even greater level of detail than that reported
by the contractor on hard copy reports.

On the other hand, in cases where the legacy coding structure adequately
reflects the programs’ information needs, the cost data received from



Page 17                                                                     GAO-03-507 NASA's IFMP
                              contractors do not have to be aggregated prior to posting. For example,
                              program officials with the ISS program recently redesigned the program’s
                              cost coding structure in order to more precisely identify the cost of specific
                              work. This was not done as part of an IFMP reengineering effort, but in
                              response to external criticisms of the program’s failure to manage its costs.
                              Regardless of the reason, the program’s reengineering effort has to some
                              extent improved the usefulness of the cost data being entered into the core
                              financial module.



Core Financial Module Will    The core financial module, as currently planned, will not provide
Not Provide the Information   sufficiently detailed data for cost estimators. Although the core financial
                              module is technologically capable of maintaining the detailed data required
Needed to Prepare Credible
                              by cost estimators, cost estimators were not involved in defining the
Cost Estimates                system requirements or reengineering business processes. As a result,
                              NASA has not determined the most cost-effective way to satisfy the
                              information needs of its cost estimators nor reengineered its business
                              process to ensure that their needs are met.

                              According to members of NASA’s cost estimating community, they typically
                              need cost data at an even greater level of detail than that currently being
                              provided by NASA’s contractors. The cost estimators we spoke with told us
                              that their requests for more detailed cost data are often not satisfied
                              through the contract negotiation process. For example, as shown in figure
                              2, while program managers may want—and contractors may provide—the
                              cost of an engine fan, cost estimators need to know more detailed
                              information, including the cost of the various tasks needed to make a rotor
                              assembly, which ultimately becomes part of the fan.




                              Page 18                                                 GAO-03-507 NASA's IFMP
Figure 2: Example of Level of Detail Reported versus That Required by Cost Estimators



                                                                                      Contract




                                                                                                 Auxiliary
                                                                   Engine
                                                                                                  power




                                                         Fan                   Compressor            Turbine




          Level of detail
                                                                                                                   Dual spool
          often provided                 Fan assembly                    Full scale fan rig      Minor fan rig
                                                                                                                 compressor rig
          by contractors



                                         Case assembly                    Rotor assembly




          Level of detail
         required by cost                                           Work packages                  Task 1
            estimators                                                                             Task 2
                                                                                                   Task 3



Source: NASA Work Breakdown Structure Reference Guide.



                                                               The lack of sufficiently detailed information for cost estimators is due to
                                                               NASA’s lack of reengineering efforts for the acquisition management
                                                               process, which should have been done prior to implementing the core
                                                               financial module. Because the core financial module will not contain
                                                               sufficiently detailed historical cost data necessary for projecting future
                                                               costs, cost estimators will continue to rely on labor-intensive data
                                                               collection efforts after a program is completed. These efforts involve
                                                               searching through old hard copy and electronic contractor reports to
                                                               extract all relevant data. NASA pays its contractors extra to provide data
                                                               required but not contained in these reports, usually at a later point in time.



                                                               Page 19                                                      GAO-03-507 NASA's IFMP
                            Data collection after the fact is expensive but, according to some NASA
                            officials, is more cost effective than requiring contractors to provide
                            detailed cost data throughout the course of the contract. However, NASA
                            has not done the analysis needed to determine the appropriate mix of
                            routinely requiring contractors to provide detailed cost data and capturing
                            that data in the core financial module versus purchasing the data after a
                            contract is complete.



Core Financial Module May   NASA has identified the Congress as a key stakeholder and ultimate
Not Provide Better          beneficiary of system improvements. However, based on our discussions
                            with congressional staffs from NASA’s authorizing committees, the agency
Information for
                            did not consult with them regarding their information needs. Consequently,
Congressional Oversight     NASA cannot be sure that it is implementing a system that will provide the
                            Congress with the information it needs for oversight. As discussed
                            previously, according to IFMP planning documents, financial managers and
                            accountants are considered direct customers and are responsible for
                            defining system requirements and priorities. On the other hand, NASA
                            considered the Congress a stakeholder and, therefore, did not seek input
                            from congressional staffs in defining system requirements.

                            Similar to the problems faced by program managers and cost estimators,
                            the core financial module may not address many of the information needs
                            of the Congress. To properly assess the agency’s annual budget submission
                            and make funding decisions, the Congress needs timely, reliable cost and
                            schedule information on the status of large, high-risk programs, such as the
                            ISS. As previously described, the module will not provide the type of cost
                            and schedule information that program managers need to adequately
                            monitor the status of NASA’s major programs and may not maintain
                            sufficient information to readily address any special congressional needs
                            that arise.

                            Nevertheless, the Congress should be able to receive somewhat better
                            information about NASA’s finances than it has in the past because, as
                            previously described, the core financial module may improve some aspects
                            of NASA’s ability to produce reliable financial information. For example,
                            the use of a standard general ledger will provide more standardized
                            accounting data and general ledger controls. As a result, the core financial
                            module should enable NASA to provide timelier, more reliable high-level
                            cost information to the Congress on some issues, such as annual spending
                            limits.




                            Page 20                                                GAO-03-507 NASA's IFMP
NASA’s Requirements      NASA has not effectively implemented a requirements management
                         process11 to support the implementation of the core financial module and
Management Process       therefore has increased the risk that the agency will not be able to
for the Core Financial   effectively identify and manage the detailed system requirements that
                         system developers and program managers use to acquire, implement, and
Module Is Ineffective    test a system. Specifically, based on discussions with IFMP officials and a
                         review of the process documents related to the core financial module, we
                         found that NASA was relying on a requirements management process that
                         did not require detailed documentation of system requirements prior to
                         system testing. Industry best practices, as well as NASA’s own system
                         planning documents, indicate that detailed system requirements should be
                         documented to serve as the basis for effective system testing. Instead,
                         NASA’s approach relied on the expertise of certain subject matter experts
                         to remember the detailed requirements necessary to evaluate the
                         functionality actually provided.

                         As a result of this approach, we found that (1) for over 80 percent of the 132
                         core financial module requirements we reviewed, the functionality to be
                         delivered was not adequately described or stated in a manner that allowed
                         for quantitative evaluation and (2) the traceability among the various
                         requirement documents was not maintained. Accordingly, the potential for
                         these requirements defects to result in costly rework is significant and
                         increases the risk that the core financial module will not meet its cost,
                         schedule, and performance objectives. Because of the direct relationship
                         between requirements and testing, the lack of complete and unambiguous
                         requirements also significantly impairs the testing phase. Furthermore,
                         NASA has not effectively implemented the types of metrics that can help it
                         understand the effectiveness of its requirements management process,
                         such as identifying and quantifying any weaknesses in its process and then
                         developing the corrective actions needed.




                         11
                          According to the Software Engineering Institute, requirements management is a process
                         that establishes a common understanding between the customer and the software project
                         manager regarding the customer’s business needs that will be addressed by a project. A
                         critical part of this process is to ensure that the requirements development portion of the
                         effort documents, at a sufficient level of detail, the problems that need to be solved and the
                         objectives that need to be achieved.




                         Page 21                                                             GAO-03-507 NASA's IFMP
NASA Requirements         Requirements are the specifications that system developers and program
Management Process Was    managers use to acquire, implement, and test a system. Requirements
                          should be consistent with one another, verifiable, and directly traceable to
Not Designed to Provide   higher-level business or functional requirements. It is critical that
Detailed System           requirements be carefully defined and that they flow directly from the
Requirements              organization’s concept of operations (how the organization’s day-to-day
                          operations are or will be carried out to meet mission needs).12 Improperly
                          defined or incomplete requirements have been commonly identified as a
                          root cause of system failure and systems that do not meet their cost,
                          schedule, or performance goals. Without adequately defined requirements
                          that have been properly reviewed and validated, a significant risk exists
                          that the system will need extensive and costly changes before it will meet
                          NASA’s needs.

                          As discussed previously, NASA is designing and fielding the core financial
                          module without having determined the specific information needs of its
                          key stakeholders, including program managers, cost estimators, and the
                          Congress. The omission of this critical step increased the risk that the
                          project would not effectively include all the detailed system requirements
                          that were needed to achieve management’s vision of a core financial
                          management module that provides timely, consistent, and reliable cost and
                          performance information for management decisions.

                          IFMP officials stated that their basic approach to developing the core
                          financial module system requirements was (1) defining high-level
                          requirements that could be used for making a software selection,
                          (2) defining the business processes that the core financial module needed
                          to address, (3) linking the requirements that were originally defined for the
                          software selection to those business processes, and (4) using subject
                          matter experts to determine whether the application met the business
                          processes envisioned by the users during their discussions of the needed
                          functionality. A key feature of the NASA approach is that the detailed
                          requirements covered in the discussion of the business processes are not


                          12
                            According to Institue of Electrical and Electronics Engineers Standard 1362-1998, a
                          concept of operations document is normally one of the first documents that is produced
                          during a disciplined development effort since it describes system characteristics for a
                          proposed system from the user's viewpoint. This is important since a good concept of
                          operations document can be used to communicate overall quantitative and qualitative
                          system characteristics to the user, developer, and other organizational elements. This allows
                          the reader to understand the user organizations, missions, and organizational objectives
                          from an integrated systems point of view.




                          Page 22                                                            GAO-03-507 NASA's IFMP
required to be documented prior to testing. Rather, NASA depends on
subject matter experts, who are assigned to ensure that the core financial
module has the needed functionality, to know the detailed requirements
necessary to evaluate the functionality actually provided. Such an
approach relies on the subject matter expert being available throughout the
process and on the expert remembering the undocumented requirements
completely and consistently. Specifically, an individual assigned to develop
a test case13 is relied on to understand the detailed requirements associated
with all facets of that test case and then to ensure that the test will provide
the information needed to understand whether the functionality was
actually provided.

IFMP officials also stated that the current approach was based on
discussions with their contractors and eliminated the need for detailed
documented requirements normally associated with efforts such as IFMP.
They also recognized that this approach was somewhat inconsistent with
their own Requirements Management Framework, issued in October 2000,
which stated that “[i]n order to test the software, a more detailed statement
of a requirement or process may be required to insure [sic] the successful
completion of a test.” This document also recognized that these detailed
requirements would be needed for “a more refined testable set of
requirements . . . and needed to serve as a basis for the testing that will
occur . . .” In a January 2003 report14 by a contractor on the lessons learned
on the IFMP effort, the contractor noted that NASA would need to develop
a set of requirements and design specifications that had been validated by
the individuals responsible for managing each process. The contractor also
noted that although such an approach delays the first phase of the project
design, it reduces the overall implementation time.

As a result of NASA’s stated approach to requirements management, our
review of NASA’s system requirements related to the process documents
for the core financial module found that key attributes of effective
requirements were missing for many requirements. According to the
Institute of Electrical and Electronics Engineers (IEEE)—a leading source



13
 A test case is a series of actions, performed serially, in parallel, or in some combination,
that creates the desired test conditions. Rex Black, Managing the Testing Process:
Practical Tools and Techniques For Managing Hardware and Software Testing (Redmond,
Wash.: Microsoft Press, 1999).
14
     Gartner, Inc.




Page 23                                                            GAO-03-507 NASA's IFMP
                                 for defining the best practices for efforts such as this—good requirements
                                 have several characteristics, including the following:15

                                 • The requirements document contains all the requirements identified by
                                   the customer, as well as those needed for the definition of the system.

                                 • The requirements fully describe the software functionality to be
                                   delivered. Functionality is a defined objective or characteristic action of
                                   a system or component. For example, a system may have inventory
                                   control as its primary functionality.

                                 • The requirements are stated in clear terms that allow for quantitative
                                   evaluation. Specifically, all readers of a requirement should arrive at a
                                   single, consistent interpretation of it.

                                 • Traceability among various requirement documents is maintained.
                                   Requirements for projects such as IFMP can be expressed at various
                                   levels depending on user needs. They range from agencywide business
                                   requirements to increasingly detailed functional requirements that
                                   eventually permit the software project managers and other technicians
                                   to design and build the required functionality in the new system.
                                   Adequate traceability ensures that a requirement in one document is
                                   consistent with and linked to applicable requirements in another
                                   document.

                                 NASA established about 590 requirements for the core financial module.16
                                 We reviewed in detail one business process area of this module—the
                                 “Manage Accounts Payable” process—that included 132 of these
                                 requirements. We found that (1) for over 80 percent of the 132 requirements
                                 the functionality to be delivered was not adequately described or stated in a
                                 manner that allowed for quantitative evaluation and (2) the traceability
                                 between the various requirement documents was not maintained.

Requirements Were Not Specific   For over 80 percent of the 132 “Manage Accounts Payable” requirements,
                                 the process documents lacked the specific information necessary to
                                 understand the required functionality that should be provided and how to


                                 15
                                      IEEE 830-1998.
                                 16
                                  NASA originally identified about 590 requirements. However, 51 of these were deleted and
                                 86 were deferred.




                                 Page 24                                                         GAO-03-507 NASA's IFMP
                                  determine quantitatively, through testing or other analysis, whether the
                                  system will meet NASA's needs. The following are examples of core
                                  financial module requirements that lacked the necessary specificity.

                                  • One requirement stated that the system must “[a]llow the information
                                    contained in the system to be queried to present detailed data as
                                    requested (such as payee information). The capability to perform a Print
                                    Screen must be available to all user screens.” This requirement did not
                                    clearly state such items as (1) the data elements that must be supported,
                                    (2) how the user would obtain the data definitions for the data elements
                                    that could be used, (3) the tool or process that would be used to perform
                                    these queries, and (4) the relationship between the ability to perform
                                    such queries and the requirement to be able to print the screen.

                                  • The core financial module was required to support “multiple payment
                                    addresses and/or bank information for a single payee.” This requirement
                                    did not clearly state the maximum number of payment address and bank
                                    information entries that should be allowed.

                                  • Several requirements called for the core financial module to make
                                    accounting entries; however, these requirements did not define the
                                    specific accounting entries that should be made.

                                  The lack of documented requirements that are complete and unambiguous
                                  not only increases the risk that the project's functionality goals will not be
                                  met, but also significantly impairs the testing phase of the system
                                  implementation efforts, as discussed later in this report.

Traceability Was Not Maintained   NASA has adopted a four-level approach to defining its requirements—
                                  processes, subprocesses, activities, and tasks, with processes stating high-
                                  level requirements and tasks providing the most detailed level. In reviewing
                                  the various requirement documents, we found that (1) traceability was not
                                  always maintained through the various documents and (2) the level of
                                  detail did not provide additional specificity for a given requirement as it
                                  progressed through the hierarchy.

                                  Traceability allows the user to follow the life of the requirement both
                                  forward and backward through these documents and from origin through
                                  implementation. Traceability is also critical to understanding the
                                  parentage, interconnections, and dependencies among the individual
                                  requirements. This information in turn is critical to understanding the
                                  impact when a requirement is changed or deleted. Without an effective



                                  Page 25                                                 GAO-03-507 NASA's IFMP
traceability approach, it is very difficult to perform such actions as
(1) accurately determining the impact of changes and making value-based
decisions when considering requirement changes, (2) maintaining the
system once it goes into production, (3) tracking the project's progress, and
(4) understanding the impact of a defect discovered during testing.

To illustrate these issues, we attempted to follow the hierarchy of
requirements for one of the core financial module's seven processes
through the four levels of requirements utilized by NASA for this project.
As shown in figure 3, the “Manage Accounts Payable” process area has 132
requirements associated with nine subprocesses. However, one of the 132
requirements, "Multiple User Access," contained in the “Manage Accounts
Payable” process, was not shown in any of the subprocesses, and it was
unclear where this requirement would be further defined.




Page 26                                                GAO-03-507 NASA's IFMP
Figure 3: System Requirements for the “Manage Accounts Payable” Process



                          Report on payment activities (14)                                   Recertify payment activities (8)




                Execute and manage payments (49)                                                        Process IPAC payments (20)


                                                                Manage accounts
                                                                    payable

                   Maintain AP master data (11)                                                               Process HHS (9)




                           Enter invoice (41)                                                             Manage travel (33)



                                                                Validate payment (17)

Source: NASA.

                                                    Note: The total number of requirements shown for the subprocesses (202) exceeds the number of
                                                    requirements shown for the “Manage Accounts Payable” process (132) because some requirements
                                                    apply to more than one activity.


                                                    Our review of the nine subprocesses found that 5 of the remaining 131
                                                    requirements contained in the subprocesses were not linked to any activity.
                                                    For example, a requirement that applies to all federal agencies and is
                                                    designed to ensure compliance with the Internal Revenue Service's 1099
                                                    reporting requirements was not included in any of the activities. Therefore,
                                                    the individuals responsible for implementing the requirements contained in
                                                    the activities would not have the full universe of requirements they must
                                                    address. Conversely, as shown in figure 4, several of the activities for the
                                                    “Validate Payment” subprocess did not contain any related requirements.
                                                    Therefore, it was unclear whether these activities should have been
                                                    associated with this subprocess. For example, the “clear advance” and
                                                    “adjust invoice” activities did not include any requirements related to
                                                    validating payments. Further, the lack of requirements for these activities
                                                    may cause confusion for the individuals assigned to test the functionality



                                                    Page 27                                                               GAO-03-507 NASA's IFMP
                                                  associated with this subprocess. We were also unable to trace the
                                                  requirements from activities to tasks, which should be the most detailed
                                                  level of requirements because, for the activities associated with the
                                                  “Validate Payment” subprocess, NASA used the same information for the
                                                  activities and tasks. In other words, the requirements for the tasks were
                                                  identical to those listed for the activities and therefore did not provide any
                                                  additional details.



Figure 4: Requirements for “Validate Payment” Subprocess



                                                     Adjust payment amount as neccessary (2)



                            Compute holdback amount (1)                                      Determine holdback status (1)




                 Check for other deductions (1)                                                     Determine other adjustments (2)


                                                              Validate Payment
                Check for proper approvals (1)                       (17)                                    Post invoice (11)




                        Clear advance (0)                                                                      Set flag (2)




                           Adjust invoice (1)                                                          Validate payment (0)




Source: NASA.

                                                  Note: The total number of requirements shown for the activities (22) exceeds the number of
                                                  requirements shown for the “Validate Payment” subprocess (17) because some requirements apply to
                                                  more than one activity.


                                                  As can been seen in this example, NASA was unable to maintain adequate
                                                  traceability of the requirements for this subprocess as it progressed
                                                  through the hierarchy. More important, the level of specificity associated
                                                  with these requirements did not change. Based on our review, we generally
                                                  found that the wording of a given requirement was identical regardless of



                                                  Page 28                                                               GAO-03-507 NASA's IFMP
                              the requirement document reviewed. For example, if a requirement was
                              listed in a subprocess area and flowed to an activity, the same wording was
                              used and the needed level of specificity to help ensure proper
                              implementation was not available. Therefore, although NASA appeared to
                              have adopted a requirements hierarchy that would facilitate the needed
                              specificity as the requirements flowed from subprocesses to tasks and
                              activities, the implementation of this approach did not address the
                              specificity problems discussed earlier. Accordingly, this is another factor
                              that increases the risk that this project will not meet its schedule, cost, and
                              functionality goals. A NASA contractor hired to help evaluate the
                              implementation of the core financial module had similar findings. For
                              example, the contractor found that NASA had not developed
                              documentation that explicitly details the relationship between lower-level
                              requirements and requirements of the next level.



Requirements Defects          Because requirements provide the foundation for system testing, the
Adversely Affect Testing of   specificity and traceability defects in the system requirements preclude
                              NASA from implementing a disciplined testing process. That is,
the Core Financial Module
                              requirements must be complete, clear, and well documented to design and
                              implement an effective testing program. Consequently, NASA is taking a
                              significant risk that its testing efforts will not detect significant defects
                              until after the system is placed into production. Industry best practices
                              indicate that the sooner a defect is recognized and corrected, the cheaper it
                              is to fix. This is especially true since NASA is depending on the subject
                              matter experts’ knowledge, rather than documented requirements, to
                              ensure that the application does not have any significant defects before the
                              system is placed into production.

                              As shown in figure 5, there is a direct relationship between requirements
                              and testing.




                              Page 29                                                  GAO-03-507 NASA's IFMP
Figure 5: Relationship between Requirements Development and Testing


                       Stages of system development                                              Stages of testing

                        Concept of operations                                              User acceptance testing
                        Specifies how system is used in                                    Verifies that system operates
                        operation                                                          correctly with operational hardware
                                                                                           and meets users' needs




                              Functional requirements                                  System acceptance testing
                              Specifies the high-level functions                       Verifies that the complete system
                              of the system                                            satisfies functional requirements




                                  Design requirements                              Integration testing
                                  Specifies the tasks each software                Verifies that units of software, when
                                  component must perform                           combined, work together as
                                                                                   intended
         For projects such
         as IFMP, these
         items are normally
         handled by the
         commercial
         software vendor
                                      Detailed design and coding               Unit testing
                                      Specifies the detailed steps for         Verifies that each component of the
                                      each software component and              software faithfully implements the
                                      implements those steps                   detailed design




Source: GAO.



                                                       Although the actual testing activities occur late in the development cycle,
                                                       test planning can help disciplined activities reduce requirements-related
                                                       defects. For example, developing conceptual test cases based on the
                                                       requirements derived from the concept of operations and functional
                                                       requirements stages can identify errors, omissions, and ambiguities long



                                                       Page 30                                                             GAO-03-507 NASA's IFMP
                                  before any code is written or a system is configured. Disciplined
                                  organizations also recognize that planning testing activities in coordination
                                  with the requirements development process has major benefits.

                                  We have identified several indications that NASA’s testing program has
                                  been adversely affected by the lack of complete and specific requirements.
                                  Although we plan to continue our review of NASA’s testing plan for IFMP
                                  implementation, we noted (1) significant defects that appeared to be
                                  related to requirements occurred in the application after it was placed into
                                  production and (2) several cases where NASA did not ensure that
                                  modifications made to the application did not cause unintended effects and
                                  that the system or component still complied with its specified requirements
                                  after the change.

Significant Defects Appeared in   Our review of the system test defect reports for the core financial module
Production System                 disclosed that several defects considered by NASA to have an initial
                                  severity rating of critical17 or high18 had been identified after the system
                                  was placed into production at Marshall Space Flight Center and Glenn
                                  Research Center. Detecting such problems after the system goes into
                                  production may lead to costly rework due to factors such as having to
                                  reenter transactions and adjust reports manually. Furthermore, the manual
                                  processes required to make these adjustments may introduce data integrity
                                  errors. Our preliminary review indicated that the root cause of many of
                                  these defects could be linked to the lack of complete requirements. For
                                  example, see the following:

                                  • Shortly after the system was placed into production, NASA found that
                                    the core financial module was not properly executing certain business
                                    rules. An emergency fix was developed, and the defect report noted that
                                    a long-term solution and requirements would need to be developed. It
                                    was unclear why the subject matter experts did not include the business
                                    rules in the test cases used to evaluate the functionality of the


                                  17
                                   NASA defines critical defects as those that “impact the ability to move forward or
                                  complete an entire business function or task, and impacts multiple business functions,
                                  multiple users and/or locations. [It] presents a failure that has no workaround or
                                  alternative.”
                                  18
                                    NASA defines high defects as those that have “a significant impact on the completion of a
                                  business function or task, however, activities can continue as far as the next function. A
                                  limited number of business functions, business users, or locations are impacted, and may be
                                  an impact to only one.”




                                  Page 31                                                          GAO-03-507 NASA's IFMP
                                    application. However, we believe one cause is the lack of documented
                                    requirements that the testers could use to develop effective test cases.

                                 • NASA was unable to process vendor invoices that contained over 200
                                   line items, which, according to NASA officials, is a common occurrence
                                   on NASA’s large contracts. It was unclear why the subject matter experts
                                   responsible for testing this functionality had not developed the test
                                   cases to ensure that large invoices were properly processed before the
                                   system was placed into production. IFMP officials recognized that this
                                   was an oversight in their testing process. If NASA had documented its
                                   requirements, it would have recognized that a properly developed test
                                   case had not been designed to cover this necessary functionality.

                                 • About 3 months after the core financial module was placed into
                                   production at one center, it was found that when the system produced
                                   multiple bills for the same customer, only the first bill was sure to have
                                   the proper account classification code printed. The remaining bills often
                                   contained incorrect values since the program improperly assumed that
                                   the account classification code would not change until the customer
                                   changed. Since the account classification code is critical for these types
                                   of bills, the center was required to manually make the necessary
                                   corrections on the bills.

Adequate Regression Testing Is   Efforts such as the core financial module undergo change constantly at this
Not Being Performed              stage of their development as functionality is being added and defects are
                                 being corrected. However, before the revised application is released,
                                 testing needs to be performed to ensure that any modifications have not
                                 caused unintended effects and that the system or component still complies
                                 with its specified requirements. This practice is commonly referred to as
                                 regression testing. An effective regression testing program is critical for
                                 ensuring that the functionality associated with requirements that has been
                                 validated during previous testing efforts has not been impaired by
                                 subsequent changes in the application.

                                 Although NASA officials stated that they require regression testing before
                                 deploying any changes, we found that they do not have an effective method
                                 to ensure that adequate regression testing is being performed or that a
                                 consistent approach is being taken in performing such testing. According
                                 to NASA officials, the individual identifying a defect is responsible for
                                 ensuring that the defect is corrected and for determining the amount of
                                 testing necessary to ensure that the defect has been corrected. For
                                 example, if a defect is identified when executing a test case, the tester may



                                 Page 32                                                GAO-03-507 NASA's IFMP
only test the section of the case where the error was originally identified
rather than performing all the steps in the test case. This approach
increases the risks that defects introduced into the application during
enhancements or defect corrections will not be detected until after the
application is deployed, which results in costly rework. We noted several
examples where NASA appeared to perform inadequate regression testing.
These include the following:

• After adding an interface, it was found that an application screen for
  recording advances did not operate as it had before the change was
  made.

• A process for recording transactions provided by the Department of the
  Treasury failed after an update to the application program.

• One center was testing a certain type of invoice and received an error
  message. This error was attributed to a system patch that had been
  applied to the application.

IFMP officials agreed that they did not have a comprehensive regression
testing program with a consistent approach. However, they told us that
they believed that any defects would be detected by the centers as the
application progresses through its releases because they encourage each
center to completely retest the application before placing the application
into production. This approach is particularly risky in light of the
requirements defects discussed previously, which substantially increase
the risk that the testing conducted by the centers not yet operational may
not detect any negative impacts associated with a system change. However,
IFMP officials recognized that, after all centers are in production, a
regression testing program will be needed.

A NASA contractor monitoring this project also identified potential
problems relating to regression testing. According to the contractor
performing this work, NASA has not implemented the testing tools
necessary to adequately perform the regression testing to provide NASA
reasonable assurance that changes made in a given software release do not
have any adverse consequences for future releases.




Page 33                                               GAO-03-507 NASA's IFMP
Performance Metrics Could     Without a well-documented set of requirements, it is impossible to place an
Be Used to Assess Potential   error in context and understand the cause of the defect—for example,
                              determining whether the error was caused by the underlying requirements
Risks of Identified           or by some other process failure, such as inadequate testing or inadequate
Weaknesses                    controls over system configuration. NASA has not effectively captured the
                              types of metrics that can help the organization understand the
                              effectiveness of its management processes, such as identifying and
                              quantifying any weaknesses in its requirements management process.
                              Accordingly, NASA is unable to implement a metrics measurement process
                              that allows it to understand (1) its capabilities to manage the IFMP effort,
                              (2) how its process problems will affect its cost, schedule, and
                              performance objectives, and (3) the corrective actions needed to reduce
                              the risks associated with the problems identified.

                              The Software Engineering Institute (SEI) has found that metrics identifying
                              important events and trends are invaluable in guiding software
                              organizations to informed decisions. Key SEI findings relating to metrics
                              include the following:

                              • The success of any software organization depends on its ability to make
                                predictions and commitments relative to the products it produces.

                              • Effective measurement processes help software groups succeed by
                                enabling them to understand their capabilities, so that they can develop
                                achievable plans for producing and delivering products and services.

                              • Measurements enable people to detect trends and to anticipate
                                problems, thus providing better control of costs, reducing risks,
                                improving quality, and ensuring that business objectives are achieved.19

                              A critical element in helping to ensure that a project meets its cost,
                              schedule, and performance goals is to ensure that defects are minimized
                              and corrected as early in the process as possible. Although NASA has a
                              system that captures the defects that have been identified during testing,
                              we found that the agency did not analyze its identified defects to determine
                              their root causes. Understanding the root cause of a defect is critical to
                              evaluating the effectiveness of a process. For example, if a significant

                              19
                               William A. Florac, Robert E. Park, and Anita D. Carleton, Practical Software
                              Measurement: Measuring for Process Management and Improvement (Pittsburgh, Pa.:
                              Software Engineering Institute, Carnegie Mellon University, 1997).




                              Page 34                                                    GAO-03-507 NASA's IFMP
number of defects are caused by inadequate requirements definition, then
the organization knows that the requirements management process it has
adopted is not effectively reducing risks to acceptable levels. IFMP officials
stated that they do not capture the root causes of their defects.

Our initial observations identified that the root cause of many defects
appeared to relate directly to the requirements management process. For
example, see the following:

• About a week after the system was placed into production at a center,
  NASA found that it was making payments to its vendors 1 day earlier
  than required by Treasury regulations. This occurred because NASA
  thought that Treasury would warehouse its payments. If NASA had
  researched and documented the requirements associated with payment
  warehousing for cash management purposes, it would have known that
  Treasury does not warehouse payments such as these.

• About 3 weeks after the system went into production, NASA found that
  one of the payment processing tools was not working as required. It was
  unclear whether this was caused by a requirements defect or failure to
  properly test the functionality. A review of the requirements documents
  relating to this functionality provided a different description of the
  requirement than that included in the defect report.

• In early October 2002, NASA found that the accounting entries for
  certain advance transactions were incorrect. Properly documenting the
  requirements, developing a test case that ensured the requirements were
  met by the application, and executing that test case after the change was
  made should have detected this problem.

• After a patch was applied to the system, it was found that some code
  was duplicated, which caused an error. The apparent reason for the
  duplication of code was that manual adjustments were made to the code
  after the patch had been applied.

By analyzing the root causes of its identified defects, NASA could
determine whether the requirements management approach it has adopted
sufficiently reduces its risks of the system not meeting cost, schedule, and
functionality goals to acceptable levels. Root cause analysis would also
help to quantify the risks inherent in the testing process that NASA has
selected for the core financial module. Because, as discussed previously, its
approach in both these areas includes elements that are not considered



Page 35                                                 GAO-03-507 NASA's IFMP
              industry best practices, such metrics would be particularly important to
              NASA’s being reasonably assured that its processes will result in a system
              that meets its business needs.



Conclusions   NASA has established the right goal for IFMP, and its ongoing
              implementation of several already-acquired system components,
              particularly the core financial module, may provide some improvements to
              NASA’s accounting data. However, implementation of these components
              will only partially address NASA’s information needs related to its complex
              space programs and contracts because NASA has deferred implementation
              of the system capabilities needed to provide this information and has not
              reengineered key business processes such as acquisition management.
              NASA’s long-standing weaknesses in this area have been central to our
              designation of NASA contract management as high risk. Moreover, NASA’s
              approach to acquiring and implementing IFMP components has and will
              continue to introduce risk and increase the chances that the agency will fall
              short of meeting its IFMP goal.

              NASA faces serious near-term risks in implementing the commercial
              components that it has already acquired, including the core financial
              module. However, it is too far along in deploying these components to its
              centers, and relying on them to support operations, to stop and first acquire
              and then implement them properly. Instead, NASA will be forced to make
              the best of what it has acquired and implemented, meaning that NASA will
              have to stabilize the components while they are operational by identifying
              and correcting requirements defects and adequately testing the
              components to ensure that completed requirements are met. Such rework
              of already-deployed system components is a much more costly approach to
              implementing systems than adequately defining requirements and
              effectively testing system capabilities before they are deployed. However,
              NASA has left itself no other viable option.

              In the longer term, NASA has an opportunity to avoid the mistakes it has
              made to date in acquiring system components, such as the core financial
              module, by first determining whether proposed components are the best
              solutions to meeting the agency’s corporate needs before it acquires them.
              It is critically important that NASA’s acquisition management strategy for
              future components includes a well-defined, risk-based methodology for
              understanding the dependencies among commercial component options
              before it acquires any additional components. Once these components are
              acquired, it is also critically important that NASA employ effective



              Page 36                                                GAO-03-507 NASA's IFMP
                      requirements management, testing, and performance metrics practices in
                      implementing the components. To do less will increase the risk of IFMP
                      becoming NASA’s third unsuccessful attempt to transform its financial
                      management and business operations.



Recommendations for   Given that NASA has already largely deployed and placed into production
                      the IFMP commercial components acquired to date, we recommend that
Executive Action      the NASA Administrator direct the Program Executive Officer for IFMP to
                      focus near-term attention on stabilizing the operational effectiveness of
                      these deployed commercial components.

                      Specifically, to mitigate the risks associated with relying on already-
                      deployed IFMP commercial components and to expeditiously stabilize
                      these components’ operational capability and performance, we
                      recommend that the Administrator direct the Program Executive Officer
                      for IFMP to develop and implement a corrective action plan. At a minimum,
                      this plan should provide for

                      • identifying known and potential risks,

                      • assessing the severity of the risks on the basis of probability and impact,

                      • developing risk mitigation strategies,

                      • assigning accountability and responsibility for implementing these
                        strategies,

                      • tracking progress in implementing these strategies, and

                      • reporting progress regularly and frequently to relevant congressional
                        committees.

                      Additionally, this plan should provide for

                      • developing and properly documenting requirements that are consistent,
                        verifiable, and traceable, and that contain the necessary specificity to
                        minimize requirement-related defects;

                      • conducting thorough regression testing before placing modified
                        components into production;




                      Page 37                                                GAO-03-507 NASA's IFMP
                      • implementing a metrics program that will identify and address the root
                        causes of system defects;

                      • reengineering acquisition management processes, particularly with
                        respect to the consistency and detail of budget and actual cost and
                        schedule data provided by contractors; and

                      • engaging stakeholders—including program managers, cost estimators,
                        and the Congress—in developing a complete and correct set of user
                        requirements.

                      To mitigate future risks, we further recommend that the Administrator
                      require the Program Executive Officer for IFMP to complete the following
                      actions before the acquisition of any additional IFMP components:

                      • establish and implement a methodology for commercial system
                        component dependency analysis and decision making, and

                      • evaluate the suitability of already acquired, but not yet implemented,
                        IFMP component products within the context of a component
                        dependency analysis methodology.



Agency Comments and   In its written comments on a draft of this report, NASA stated that it
                      recognized and was addressing several of the concerns we raised and had
Our Evaluation        already implemented some of the recommendations. NASA also stated that
                      it disagreed with some of the issues in the report. NASA’s comments on our
                      recommendations included the following:

                      • With regard to our recommendation to establish and implement a
                        methodology for IFMP commercial system component dependency
                        analysis and decision-making, NASA stated that it agreed with the
                        importance of having an approach for acquiring additional IFMP
                        components and believes that it has an effective strategy already in
                        place. We disagree that NASA has an effective strategy because it did not
                        provide convincing evidence to support its position that it is using
                        methodologically based dependency analysis—a best practice for
                        implementing commercial component-based systems—in acquiring
                        IFMP.

                      • Although NASA concurred with our recommendation regarding the
                        need for a short-term plan to mitigate the risks currently associated with



                      Page 38                                                GAO-03-507 NASA's IFMP
                         relying on already-deployed IFMP commercial components, it disagreed
                         with many of our findings in the areas of (1) its efforts to involve users
                         and reengineer its business process to ensure that the core financial
                         module would meet the needs of program managers and cost estimators
                         and (2) the need for detailed system requirements. We continue to
                         believe that any effort that falls short of end-to-end business process
                         reengineering will not result in a system that substantially improves the
                         data available for contract oversight and decision-making and that
                         documented, detailed requirements are necessary to reduce the risks of
                         implementing the selected processes to acceptable levels.

                      Overall, NASA disagreed with our findings related to three issues—
                      dependency analysis, user needs, and requirements and testing—which are
                      addressed in the following sections. NASA also included several technical
                      comments, which we have addressed as appropriate throughout the report.



Dependency Analysis   In its written comments on our recommendation to establish and
                      implement a methodology for IFMP commercial component dependency
                      analysis and decision making, NASA stated that it agreed with the
                      importance of having an approach for acquiring additional IFMP
                      components but disagreed with our finding that it has not performed such
                      dependency analysis to date in acquiring four IFMP commercial
                      components. It also disagreed that it lacked a methodology to guide its
                      analysis, and subsequent decision making, for future IFMP component
                      acquisitions. According to NASA’s comments, the agency already has an
                      effective strategy in place and it has followed this strategy to date in
                      acquiring four IFMP commercial components. NASA described this
                      strategy as consisting of two factors: following an enterprise resource
                      planning (ERP) suite integration strategy (i.e, “best of suite” approach) and
                      using an enterprise application integration framework and associated tool
                      set for integrating current and future IFMP components.

                      NASA said that prior to receiving our draft report, it had provided us
                      detailed documentation describing how it began performing its component
                      dependency analysis before selecting the SAP ERP product for IFMP’s core
                      financial module. The agency also noted that in a meeting following its
                      receipt of our draft report, it had provided us with clear evidence that the
                      program began performing dependency analysis before selecting the SAP
                      product. Further, NASA’s comments stated that it was using an enterprise
                      application integration tool to facilitate product integration and ease the
                      associated complexities of integrating multiple products. NASA added that



                      Page 39                                                GAO-03-507 NASA's IFMP
there were “perhaps miscommunications” and “some misunderstanding” of
its approach, and opined that much of our concern about component
dependency stems from our belief that NASA is not following a “best of
suite” approach but rather a “best of breed” approach. To support its view
that it is following a “best of suite” approach, NASA offered several
statements, including that it is (1) on record in presentations and letters,
including responses to congressional inquiries, that it is following “best of
suite,” (2) developing business cases before implementing an IFMP
module, (3) working with SAP to extend its ERP product, and (4) following
a prioritization process when considering how to introduce functionality
that the SAP ERP product does not provide.

We agree with several of NASA’s comments and disagree with others.
Collectively, NASA’s comments do not change our finding and
recommendation. Specifically, we do not question that NASA is using an
enterprise application integration product and that this product facilitates
integration of system components, both commercial and legacy
components. Further, we do not challenge NASA’s statements regarding its
representation in presentations and briefings that it is following a “best of
suite” approach, its development of business cases, its interactions with
SAP, and its use of a prioritization process.

However, we do challenge NASA’s assertion that much of our concern is
based on our belief that it is following a “best of breed” approach. On the
contrary, our finding and recommendation do not hinge on the distinctions
between “best of breed” versus “best of suite,” despite evidence supporting
our statements in the report that NASA is indeed following a best of breed
approach. Such evidence includes (1) a report from a NASA contractor
hired to provide an independent evaluation of IFMP stating that NASA is
following a “best of breed” approach, (2) NASA’s acquisition of four
separate commercial products from four vendors to satisfy the first five of
nine planned IFMP system modules, and (3) NASA’s statement in its
comments that additional products may be selected in the future. We fully
appreciate that when implementing an ERP solution, other vendor
products will likely be needed to fill gaps between agency requirements
and the ERP product’s capabilities. Accordingly, we state in our report that
proactive, methodologically based dependency analysis and evaluation is
needed regardless of whether an agency is following a “best of breed” or a
“best of suite” approach, although we appropriately recognize that this
analysis and evaluation is more vital in a “best of breed” effort.




Page 40                                                GAO-03-507 NASA's IFMP
             Instead, our finding and recommendation is based on whether NASA is
             following, and plans to follow, methodologically based dependency
             analysis—a best practice for implementing commercial component-based
             systems—in acquiring IFMP. In this regard, documentation that NASA
             provided us during the course of our review, and that it provided following
             its receipt of our draft report, both of which NASA cited in its comments,
             does not offer convincing evidence that NASA is following this best
             practice. For example, the documentation lacked product descriptions and
             comparisons as well as any analysis of integration requirements. Moreover,
             the Deputy Program Manager responsible for IFMP integration told us
             during the course of our review that proactive analysis of prospective IFMP
             components’ dependencies had not been performed and was not planned,
             and that NASA did not have a methodology for doing such analysis. The
             Deputy Program Manager for IFMP integration added, similar to NASA’s
             comments on a draft of this report, that the agency’s use of an enterprise
             application integration product and its associated tools will make
             integration easy and will negate the need for proactive dependency
             analysis. As noted above, we recognize that this product and tool set
             facilitates integration of multiple system components. However, it does not
             negate the need for dependency analysis and understanding to support
             informed decision-making before integration begins. As we state in our
             report, without such a proactive approach to acquiring system
             components, the risk of component product incompatibilities increases, as
             do the challenges and complexities that integration products and tools
             must overcome in integrating the products.



User Needs   NASA agreed that deployed and in-deployment modules do not yet meet all
             the needs of program managers. NASA indicated that this was the result of
             its “step-wise” approach in implementing the core financial module first
             and then integrating follow-on modules at a later date. As noted in our
             report, however, the deferral of many basic management functions has
             resulted in critical NASA programs, such as the ISS, using other systems to
             monitor contract cost and schedule performance. By not including the cost
             and schedule information needed by program managers in the core
             financial module, NASA risks operating with two sets of books—one that is
             used to report to management and the Congress and another that is used to
             manage NASA’s programs. NASA disagreed with our specific findings
             related to user needs in three key areas:




             Page 41                                               GAO-03-507 NASA's IFMP
• NASA believes that we have understated its accomplishments and the
  significance of the current capabilities delivered by the core financial
  module.

• NASA took issue with our assessment of the level of detail maintained in
  the core financial module, but did not comment specifically on our
  recommendation that the agency reengineer its acquisition management
  processes, particularly with respect to the consistency and detail of
  budgeted and actual cost and schedule data provided by contractors.

• NASA disagrees with our characterization that key users were not
  actively involved in the implementation of the core financial module or
  defining system requirements, although NASA indicates that better
  coordination was needed between program managers and the financial
  management community.

First, we acknowledge again the significant effort that NASA has put into
this project. Moving from 10 separate, incompatible systems to a single
integrated financial management system is a major, complex undertaking.
However, as we discussed previously in this report, the core financial
module falls short of NASA’s own representation of the module’s
capabilities, which was to provide program managers with the information
required for day-to-day decision making. Specifically, it does not provide
integrated cost and schedule performance information needed by program
managers to oversee many of NASA’s largest and most complex contracts.
In commenting on a draft of this report, NASA officials stated that it was
never NASA’s intent to integrate schedule data with the initial core financial
module implementation. However, IFMP planning documents (including its
program plan and business case analysis), congressional testimony by
NASA’s Administrator, and NASA’s own press releases clearly established
an expectation that the core financial module would remedy many of
NASA’s long-standing management challenges by providing program
managers and other users with integrated financial and performance
information. For example, according to his testimony before the House of
Representatives Committee on Science on February 27, 2002, the NASA
Administrator stated that while all components of IFMP are important, the
successful completion of the core financial project will satisfy the Office of
Management and Budget requirement that the financial and performance
management systems supporting day-to-day operations are fully




Page 42                                                 GAO-03-507 NASA's IFMP
integrated.20 NASA responded that it is currently in the process of engaging
program managers and defining specific requirements related to needed
cost and schedule performance data.

Second, we recognize that the commercial components NASA has selected
for its core financial module are technologically capable of capturing and
maintaining the detailed cost data required by program managers and cost
estimators. However, the level of detailed cost data currently maintained in
the core financial module depends on the level of detail provided by NASA’s
contractors and the coding structure embedded in the core financial
module. With respect to the level of detail provided by contractors, we
reported that NASA has not reengineered its acquisition management
processes to ensure that contractors are consistently providing the detailed
cost data needed by program managers and cost estimators and
recommended that NASA do so.

NASA did not specifically address our recommendation but stated that it is
incumbent upon program managers and cost estimators to learn and
understand the capabilities of the new module and take advantage of them
for their specific purposes. NASA’s comments also indicate that the data
structure in the core financial module would be extended beyond the
current legacy capabilities (i.e., the module will be able to record a greater
level of detail) in fiscal year 2004. However, increasing the module’s
capacity to store greater detail will not ensure that the information needed
by program managers and cost estimators is requested and received from
contractors and subsequently updated in the module. Although NASA
commented that it would review its current project management process to
ensure that its contractors provide the appropriate levels of cost data, we
continue to believe that any effort that falls short of end-to-end business
process reengineering will not result in a system that substantially
improves the data available for contract oversight and decision making.

Third, we acknowledge that the IFMP implementation team made an effort
to include resource management staff from program management offices in
various process teams. However, as we discussed previously in this report,
no effort was made to include the cost estimating community in these

20
 The Chief Financial Officers Act of 1990 and subsequent related financial management
reform legislation, among other things, set expectations for agencies to develop and deploy
more modern financial management systems, produce sound cost and operating
performance information, and design results oriented reports on the government’s financial
condition by integrating budget, accounting, and program information.




Page 43                                                          GAO-03-507 NASA's IFMP
                           efforts. While program management office staff did participate, these
                           efforts did not address the program cost and schedule needs of program
                           managers or cost estimators. For example, the program management staff
                           with whom we spoke, who worked on three of NASA’s largest programs
                           (ISS, Space Shuttle, and Space Launch Initiative), viewed the core financial
                           module as an “accounting system” that would be used by the accountants
                           but was not necessarily going to change the way that program managers
                           manage their programs. With this understanding, it is not surprising that
                           the core financial module does not meet the needs of program managers or
                           cost estimators. Implementing an integrated financial management system
                           that is intended to change the way an organization does business is
                           extremely complex and involves cultural, organizational, and process
                           improvements. It also means making financial management an agency-wide
                           priority. Our work at leading public and private sector organizations has
                           shown that implementing a financial management system that meets the
                           organization’s business needs takes more than just placing a handful of
                           business or line management representatives on the implementation team.
                           NASA’s approach has resulted in a core financial module that will be of
                           limited value to program managers and cost estimators, who will continue
                           to rely on other systems or ad hoc processes to get the data they need. As
                           such, implementation of the core financial module to date continues to
                           foster the concern that, at NASA, finance is not viewed as an integral part
                           of NASA’s program management decision process.



Requirements and Testing   NASA generally agreed that improvements were needed in its requirements
                           management and testing processes and has stated that it has already begun
                           to make improvements. For example, NASA recognized the need to
                           implement a more rigorous regression testing methodology and stated that
                           by October 2003 it would have an improved regression testing program.
                           NASA also recognized that its process for tracing requirements and testing
                           needed improvement and stated that it planned to implement improved
                           capability and functionality for traceability over the next few months.

                           However, according to NASA officials, they are following best practices for
                           implementing an ERP solution and have “defined and implemented
                           rigorous, closed-loop requirements and testing processes.” Further,
                           regarding the applicability of requirements management standards, NASA
                           did not agree that IEEE 830-1998 was applicable to the IFMP since it was an
                           ERP implementation effort. NASA stated that specifying detailed
                           requirements for already-developed software is high risk and that other
                           leading industry experts have told them that NASA needed to change its



                           Page 44                                                GAO-03-507 NASA's IFMP
processes to conform to the capabilities of the commercial software
selected rather than attempt to change the software to conform to the
existing NASA processes. We agree with NASA’s position that it needs to
change its business processes to conform to the software; however, we do
not agree with the agency’s position that detailed requirements are not
needed. We continue to believe that NASA needs to properly configure the
software based on detailed requirements in a manner that supports the
business processes that have been adopted from the selected ERP solution.
Because it has not done so, we continue to believe that NASA has not
effectively implemented the types of disciplined processes necessary to
reduce this project’s risks to acceptable levels. Acceptable levels refer to
the fact that any systems acquisition effort, such as that being undertaken
by NASA, will have some requirements-related defects. However, the goal
is to reduce the risks and prevent significant requirements defects in order
to limit the negative impact of these defects on cost, timeliness, and
performance of the project.

During our review, we discussed with IFMP officials our concerns about
the lack of documented, detailed system requirements for implementing
the core financial module. In those meetings, we recognized that NASA’s
approach for developing requirements was based on a business process
model and did not disagree that this approach could be used to define how
NASA would implement the necessary functionality. However, we continue
to believe that once the business processes are defined and selected,
documented, detailed requirements are necessary to reduce the risks of
implementing the selected processes to acceptable levels. As NASA noted
in its comments, its consultants also recommended that NASA needed to
“determine the requirements while putting together the design process.”
Therefore, guidance provided by the IEEE standard is applicable to the
successful configuration and implementation of commercial software
packages and is useful to help gauge the effectiveness of those efforts.


As agreed with your offices, unless you announce its contents earlier, we
will not distribute this report further until 30 days from its date. At that
time, we will send copies to interested congressional committees, the
NASA Administrator, and the Director of the Office of Management and
Budget. We will make copies available to others upon request. In addition,
the report will be available at no charge on the GAO Web site at
http://www.gao.gov.




Page 45                                                GAO-03-507 NASA's IFMP
If you or your staffs have any questions concerning this report, please
contact Gregory D. Kutz at (202) 512-9505 or kutzg@gao.gov, Randolph C.
Hite at (202) 512-6256 or hiter@gao.gov, Allen Li at (202) 512-4841 or
lia@gao.gov, or Keith A. Rhodes at (202) 512-6412 or rhodesk@gao.gov. Key
contributors to this report are acknowledged in appendix III.




Gregory D. Kutz
Director
Financial Management and Assurance




Allen Li
Director
Acquisition and Sourcing Management




Randolph C. Hite
Director
Information Technology Architecture
 and Systems Issues




Keith A. Rhodes
Chief Technologist
Applied Research and Methods




Page 46                                             GAO-03-507 NASA's IFMP
Appendix I

Objectives, Scope, and Methodology                                                           AA
                                                                                              ppp
                                                                                                ep
                                                                                                 ned
                                                                                                   n
                                                                                                   x
                                                                                                   id
                                                                                                    e
                                                                                                    x
                                                                                                    Iis




              To determine whether the National Aeronautics and Space Administration
              (NASA) is effectively managing the Integrated Financial Management
              Program (IFMP) acquisition, we reviewed relevant program-level
              acquisition management documentation to obtain an understanding of
              NASA’s plans and strategy, including the program overview, program- and
              project-level management plans, the acquisition strategy, implementation
              and integration plans, briefing materials on the agency’s plans to develop
              an information architecture, and a report on IFMP lessons learned by
              NASA’s consultant, Gartner, Inc. We also interviewed various program
              officials, including the Program Executive Officer for IFMP, the IFMP
              Program Director, the IFMP Deputy Program Director, the Core Financial
              Project Manager, the Integration Office Deputy Program Manager, and the
              Chief Information Officer to clarify our understanding of the agency’s
              strategy and obtain current information on the status of the agency’s
              efforts. Specifically, we inquired as to NASA’s basis for selecting already-
              acquired commercial products and its plans for selecting future modules.
              We then compared NASA’s plans and activities to relevant best practices,
              Office of Management and Budget (OMB) requirements, federal guidance,
              and NASA procedures and guidance.

              We had also intended to include our assessment of a key element of NASA’s
              acquisition strategy—whether NASA was acquiring IFMP components in
              the context of an enterprise architecture—in this report. However, because
              NASA did not provide the data needed to complete our assessment until
              after the conclusion of our fieldwork, we plan to address NASA’s enterprise
              architecture in a future product.

              To determine whether NASA had adequately considered the information
              needs of key users in implementing the core financial module of IFMP, we
              reviewed IFMP documents discussing the business case and properties of
              the core financial module and spoke with IFMP implementers at Marshall
              Space Flight Center—the lead center on this project—and NASA
              headquarters. To determine whether the data requirements, as established
              by the IFMP implementers, would address NASA’s known problems with
              cost control and cost tracking, we spoke with program managers involved
              in three of NASA’s largest programs and other NASA program and business
              management staff at three centers—Marshall Space Flight Center, Johnson
              Space Center, and Glenn Research Center. We also reviewed prior work on
              NASA’s cost problems, including the report by the International Space
              Station Management and Cost Evaluation Task Force, which reviewed the
              recent cost growth in that program and identified causes and necessary
              actions.



              Page 47                                                GAO-03-507 NASA's IFMP
Appendix I
Objectives, Scope, and Methodology




In addition to speaking with program managers and their staffs, we spoke
with cost estimators at the three centers mentioned above as well as
Langley Research Center and NASA headquarters. We also spoke with
center staff who oversee and support earned value management for
programs that use that tool, and with the congressional staffs of NASA’s
authorization committees. We asked them about the extent to which they
had been asked by IFMP implementers for input on their data needs, the
extent to which they had been involved in IFMP’s design and
implementation, and whether they had been briefed by IFMP implementers
on the capabilities of the core financial module.

To determine what kind of cost information program managers use to
oversee their programs, we reviewed selected large, cost-type contracts for
NASA’s three largest space flight programs—the International Space
Station, the Space Shuttle, and the Space Launch Initiative project
(intended to develop technologies for the next generation replacement for
the Space Shuttle.) For all three of these programs, cost control and cost
tracking have been issues of concern. The three programs together involve
most of NASA’s work in the human space flight area, which accounts for
most of the agency’s spending. These programs range from relatively new
(Space Launch Initiative) to quite mature (Space Shuttle) programs and
require the procurement of a wide range of goods and services. Each of
these programs is being run at multiple centers, involves the work of
multiple contractors, and uses cost-type contracts1 that run for multiple
years.

For the contracts we selected, we spoke to responsible personnel about
how costs are tracked and monitored, including the level of detail provided
by contractors, the format in which cost data are available, and how
contract cost data reporting requirements are developed. We also obtained
and reviewed copies of contractor financial management reports and cost
and schedule performance reports that we compared with contract or
program work breakdown structures, as well as contract cost data
reporting requirements and statements of work. We analyzed and discussed
with agency officials how all these documents and reports related to each
other and to the work breakdown structure. We also discussed how the


1
 Cost-reimbursement contracts are often the most appropriate type for developmental items
or those for which the exact price of the goods or services being purchased cannot be
definitely known prior to contract award. This type of procurement instrument places
greater risk on the government than contracts based on firm fixed prices.




Page 48                                                         GAO-03-507 NASA's IFMP
Appendix I
Objectives, Scope, and Methodology




reported information was used by the programs and the extent to which
that information would be included in the core financial module. We did
not, however, evaluate whether the information currently received from
contractors, or represented as needed by program managers and cost
estimators, was adequate for management purposes.

IFMP’s core financial module is intended to address known problems with
NASA’s program cost accounting and with its financial reporting. We did
not, however, review how the core financial module will address the
agency’s financial reporting issues, including property accounting and
budgetary information, and whether the module will reduce the time and
resources needed to close the books each accounting period and reduce
the number of postclosing adjusting entries. We plan to review and report
on these issues at a later date.

To assess whether NASA had established and implemented an effective
requirements management process to support implementation of the core
financial module, we wanted to determine whether NASA had effectively
implemented (1) the disciplined processes that can reduce project risks to
acceptable levels for its requirements management process and (2) the
types of metrics to identify and quantify any weaknesses in its
requirements management process. To accomplish these objectives, we

• reviewed various requirements documents produced for the core
  financial module project, including the over 500 contract requirements
  used to acquire the SAP software;

• performed an in-depth review and analysis of the 132 requirements,
  which represent about 22 percent of the contract requirements,
  developed for the “Managing Accounts Payable” process to determine
  whether they had the attributes normally associated with good
  requirements and whether these requirements traced between the
  various requirements documents;

• reviewed NASA’s procedures for defining its requirements management
  framework and compared these procedures to its current practices;

• reviewed business processes, problem defect reports, test conditions,
  test cases, and test execution logs contained in Accenture’s Method
  Delivery Management system—NASA’s project management tool; and




Page 49                                              GAO-03-507 NASA's IFMP
Appendix I
Objectives, Scope, and Methodology




• reviewed guidance published by the Institute of Electrical and
  Electronics Engineers (IEEE), Software Engineering Institute (SEI), and
  publications by experts to determine the attributes that should be used
  for developing good requirements and for identifying and quantifying
  performance metrics.

To augment these document reviews and analyses, we interviewed officials
from NASA headquarters, Marshall Space Flight Center, and NASA’s
independent verification and validation contractor—Titan Systems
Corporation. In addition, we discussed with NASA officials the processes
they used to measure the effectiveness of their requirements management
process and compared NASA’s process to those used by disciplined
organizations. In order to determine the processes that can be used to help
an organization understand the effectiveness of its processes, we used
information from IEEE, SEI, and subject matter experts.

We conducted our work at NASA headquarters in Washington, D.C.;
Marshall Space Flight Center in Huntsville, Alabama; Glenn Research
Center in Cleveland, Ohio; Johnson Space Center in Houston, Texas;
Langley Research Center in Hampton Roads, Virginia; and Goddard Space
Flight Center in Greenbelt, Maryland. We received written comments on a
draft of this report from the NASA Deputy Administrator. These comments
are addressed in the “Agency Comments and Our Evaluation” section of
this report and are reprinted in appendix II. We performed our work from
April 2002 through February 2003 in accordance with generally accepted
government auditing standards.




Page 50                                               GAO-03-507 NASA's IFMP
Appendix II

Comments from the National Aeronautics and
Space Administration                                           Appendx
                                                                     Ii




Note: GAO comments
supplementing those in the
report text appear at the end
of this appendix.




See comment 1.




                                Page 51   GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




See comment 1.




                 Page 52                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




See comment 1.




                 Page 53                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 2.




See comment 3.




                 Page 54                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 4.




                 Page 55                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




                 Page 56                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




See comment 1.




                 Page 57                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 5.




See comment 1.




                 Page 58                                  GAO-03-507 NASA's IFMP
Appendix II
Comments from the National Aeronautics
and Space Administration




Page 59                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




See comment 1.




                 Page 60                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




                 Page 61                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




                 Page 62                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




                 Page 63                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 6.




                 Page 64                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 7.




                 Page 65                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 8.




                 Page 66                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 1.




                 Page 67                                  GAO-03-507 NASA's IFMP
                 Appendix II
                 Comments from the National Aeronautics
                 and Space Administration




See comment 9.




                 Page 68                                  GAO-03-507 NASA's IFMP
Appendix II
Comments from the National Aeronautics
and Space Administration




Page 69                                  GAO-03-507 NASA's IFMP
               Appendix II
               Comments from the National Aeronautics
               and Space Administration




               The following are GAO’s comments on the National Aeronautics and Space
               Administration’s (NASA) letter dated March 25, 2003.



GAO Comments   1. See the “Agency Comments and Our Evaluation” section of this report.

               2. Although NASA indicates that it has extended the SAP suite to include
                  Business Warehouse (for reporting) and Asset Management, it did not
                  provide us any documentation to support these selections during the
                  course of our fieldwork.

               3. We did not assess the deployment and operation of the three modules
                  to which NASA referred. We understand that the NASA Inspector
                  General has recently begun a review of the Travel Management module.

               4. The scope of our work did not include a review of the Integrated
                  Financial Management Program (IFMP) budget or schedule. We plan to
                  address these issues in a future product.

               5. As stated in this report, although the core financial module will provide
                  some improvement to NASA’s current accounting system environment,
                  certain system capabilities have been deferred and will not be available
                  when the system becomes an agencywide operation in June. Without an
                  effort to reengineer NASA’s acquisition management processes, it is
                  unlikely that detailed cost information will be available to meet the
                  needs of program managers, cost estimators, and the Congress. Thus,
                  NASA’s assertion that it will be able to transition to “full cost
                  management practices” in June of 2003 is questionable.

               6. NASA’s comments refer to the “Accounts Payable” process illustrated in
                  figure 3 of the report. While we did not verify or evaluate the extent to
                  which the additional requirements to which NASA refers in its response
                  were established or validated, the accounts payable requirements, as
                  described, do not provide for quantitative evaluation to determine
                  whether the system meets NASA’s needs. Furthermore, the additional
                  requirements did not provide the needed clarification for the
                  requirement cited in our report related to the ability to query
                  information. Moreover, given that NASA added new requirements for
                  reporting, it is unclear whether the existing accounts payable
                  requirement was to provide some other query functionality not
                  included in the other general reporting requirements.




               Page 70                                                GAO-03-507 NASA's IFMP
Appendix II
Comments from the National Aeronautics
and Space Administration




7. As noted in our report, requirements provide the foundation for system
   testing. In meetings with NASA, we acknowledged that the “process-
   centric” approach that the agency adopted was an acceptable
   methodology for understanding how the processes supported by the
   selected enterprise resource planning (ERP) solution could be
   implemented at NASA. However, we believe that this approach still
   requires the development and documentation of the necessary
   requirements to fully understand the functionality to be provided by a
   given process. Without such requirements, a disciplined testing process
   is very difficult to implement since requirements are a fundamental
   attribute of an effective testing process. As discussed in our report, we
   continue to believe that the lack of an effective requirements
   management process hampered NASA’s testing efforts since significant
   defects in the production system should have been detected before
   system implementation.

    Although NASA stated that it will repeat its testing efforts at each
    center implementing the system, without adequately documenting its
    requirements and ensuring that the testing process adequately tests
    those requirements, it does not have reasonable assurance that the
    testing process will identify significant defects before a center is
    converted to the production system. For example, NASA stated that it
    had developed over 1,700 test conditions. However, it was not until the
    system was placed into production that NASA identified several
    significant weaknesses, as discussed in our report. We continue to
    believe that NASA will not have reasonable assurance that it has
    adequately tested the system until it (1) documents its requirements
    and (2) develops test conditions that fully test those requirements.

8. As noted in our report, discussions with IFMP officials recognized that
   a test case was not properly developed to test large contracts that
   contained over 200 line items—a common occurrence according to
   IFMP officials—and that this was an oversight in their testing process.
   Had NASA developed and documented a detailed requirement for this
   functionality and then mapped its test conditions against those
   requirements, it would have recognized that it had not developed a test
   condition to properly demonstrate and test the functionality prior to
   the system going into production. Properly processing these types of
   payments may have enabled NASA to reduce the impact of the payment
   backlog.




Page 71                                                GAO-03-507 NASA's IFMP
Appendix II
Comments from the National Aeronautics
and Space Administration




9. As noted in our report, NASA does not have metrics that properly
   analyze the cause of the defects so that it can improve its processes.
   For example, although NASA was able to show the number of defects
   that were related to subsequent implementation, referred to as the
   second wave, it did not have information that could be used to analyze
   whether these defects were caused by, for example, requirements or
   testing problems or by not adequately correcting prior defects.
   Therefore, although NASA states that it has a structured testing and
   problem analysis process in place, we continue to believe that the
   examples provided in NASA’s comments do not provide the data
   necessary to identify the causes of defects or assess the effectiveness
   of processes such as the requirements management and testing
   processes. As noted in our report, these types of data can be used to
   prevent or anticipate problems before they occur, resulting in less
   rework.




Page 72                                              GAO-03-507 NASA's IFMP
Appendix III

GAO Contacts and Staff Acknowledgments                                                        Appendx
                                                                                                    iI




GAO Contacts      Diane Handley, (404) 679-1986
                  Cynthia Jackson, (202) 512-5086
                  Chris Martin, (202) 512-9481



Acknowledgments   Staff members who made key contributions to this report were Erin Baker,
                  Molly Boyle, Felicia Brooks, Francine DelVecchio, Michael Giannone,
                  Jamie Haynes, Richard J. Herley, Kristi Karls, LaTonya Miller, Maria Storts,
                  Eric Trout, Teresa Tucker, Carrie Wilson, and Jenniffer Wilson.




(120148)          Page 73                                                GAO-03-507 NASA's IFMP
GAO’s Mission            The General Accounting Office, the audit, evaluation and investigative arm of
                         Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal government
                         for the American people. GAO examines the use of public funds; evaluates federal
                         programs and policies; and provides analyses, recommendations, and other
                         assistance to help Congress make informed oversight, policy, and funding
                         decisions. GAO’s commitment to good government is reflected in its core values of
                         accountability, integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO
                         Mailing Lists” under “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Service Requested