oversight

Defense Financial Management: Immature Software Development Processes at Indianapolis Increase Risk

Published by the Government Accountability Office on 1997-06-06.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to the Under Secretary of
                 Defense (Comptroller)



June 1997
                 DEFENSE FINANCIAL
                 MANAGEMENT
                 Immature Software
                 Development
                 Processes at
                 Indianapolis Increase
                 Risk




GAO/AIMD-97-41
             United States
GAO          General Accounting Office
             Washington, D.C. 20548

             Accounting and Information
             Management Division

             B-276764

             June 6, 1997

             The Honorable John J. Hamre
             The Under Secretary of Defense (Comptroller)

             Dear Mr. Hamre:

             In conjunction with our responsibilities to audit the U.S. government’s
             financial statements, we are reviewing the Department of Defense (DOD)
             financial management systems. As you know, Defense’s ability to produce
             accurate, auditable financial statements and other reliable management
             reports as required by the Chief Financial Officers (CFO) Act of 1990, as
             expanded upon by the Government Management Reform Act of 1994, has
             been hampered by inadequate financial systems. This report discusses the
             results of our evaluation of the Defense Finance and Accounting Service
             (DFAS) Financial Systems Activity (FSA)-Indianapolis’ capability for
             developing and maintaining software for its information systems. Our
             objective was to evaluate software development processes used at
             FSA-Indianapolis. The four projects we reviewed were selected by the FSA
             director as those best representing its software development processes
             and practices.


             DFAS was created in 1991 from the financial centers of the military
Background   departments as the executive agent responsible for finance and accounting
             functions within DOD. Through consolidation, DFAS acquired the
             responsibility for more than 200 existing “legacy” finance and accounting
             systems, commonly referred to as automated information systems. Some
             are being eliminated and others are being further consolidated into a
             smaller number of “migratory” and “interim migratory” systems. In
             October 1993, the newly formed DFAS organization, called the Financial
             Systems Organization (FSO), was created to provide traditional central
             design activity services, as well as technical support, on a fee-for-service
             basis. FSO headquarters staff were in Indianapolis with additional
             personnel at six geographically dispersed FSAs having the primary mission
             to develop, modify, and maintain DFAS’ automated information systems and
             secondarily to provide technical support in a number of systems-related
             areas.

             Just prior to this reorganization, in June 1993, the Indianapolis center
             completed an assessment of the software engineering processes
             associated with its role as a central design activity. This internal software




             Page 1                               GAO/AIMD-97-41 Defense Financial Management
                   B-276764




                   process assessment concluded that the overall software engineering
                   process practiced at Indianapolis was consistent with level 1 of the
                   Software Engineering Institute’s (SEI) capability maturity model.1 SEI
                   characterizes a Level 1 software process, which is the initial and most
                   basic of the five levels, as ad hoc, and occasionally even chaotic, with few
                   processes defined, and success depending on individual effort. Table 1
                   describes each level of this model.

                   Today, the FSAs are mainly concerned with maintaining and modifying the
                   109 existing automated systems, with software development, modification,
                   and maintenance of DFAS’ mission-oriented and support systems
                   consuming more than a reported 80 percent of the FSO’s fiscal year 1995
                   budget. Recognizing this role, in 1994, FSO initiated a major effort to
                   improve its underlying software processes. In March of that year, the
                   original FSO-developed software process improvement (SPI) strategic
                   action plan was approved.

                   Since 1994, FSO has been implementing its long-term SPI plan to improve
                   and standardize maintenance and modification processes. The plan
                   includes the implementation of a system modification scenario and
                   achievement of a level 2 software engineering capability according to the
                   criteria of SEI’s model. (See appendix II for a list of the 10 major objectives
                   in the strategic plan, and the names and locations of other activities and
                   systems under the SPI umbrella.)


                   Although FSA-Indianapolis does not yet satisfy the criteria for a level 2 (i.e.,
Results in Brief   repeatable) software development capability on any of the four projects
                   we reviewed, the two projects under its SPI program showed strengths and
                   improvement activities in many of the key process areas (KPAs). For
                   example, projects under the SPI program generally kept software-related
                   work products consistent with requirements. In contrast, projects not
                   under the SPI program had few such identifiable strengths or improvement
                   activities.

                   While the SPI program is making progress in ensuring that its projects
                   implement defined and documented processes, many of its processes were

                   1
                    The Software Engineering Institute (SEI) is a nationally recognized, federally funded research and
                   development center established at Carnegie Mellon University in Pittsburgh, Pennsylvania, to address
                   software development issues. In the late 1980s, with assistance from the Mitre Corporation, SEI
                   developed a process maturity framework designed to assist organizations in improving their software
                   processes. In general, software process maturity serves as an indicator of the likely range of cost,
                   schedule, and quality of results that can be expected to be achieved by projects within a software
                   organization.



                   Page 2                                          GAO/AIMD-97-41 Defense Financial Management
              B-276764




              not yet institutionalized.2 For example, many policies were still in draft
              form or were in the planning phase, and therefore were not yet an ongoing
              way of doing business. In addition, software quality assurance activities,
              such as audits, were not used to ensure that defined software processes
              and standards were being followed. Such deficiencies pose unnecessary
              risks to the success of the software project until they are addressed.

              By more rigorously implementing its project management processes
              among its SPI projects, FSA-Indianapolis could accelerate progress toward
              reaching the level 2 capability. This would enhance its ability to repeat
              individual project successes within similar application areas.


              To evaluate FSA-Indianapolis’ software development capability, version 3.0
Scope and     of the Software Engineering Institute’s (SEI) software capability evaluation
Methodology   (SCE) method3 was used by an SEI-trained team of GAO specialists, including
              an authorized lead evaluator trained in this evaluation technique by SEI.
              The evaluation is a method of assessing agencies’ and contractors’
              software development processes against industry-accepted criteria in SEI’s
              five-level software capability maturity model (CMM), as shown in table 1.
              These levels and the key process areas described within each level define
              an organization’s ability to develop software, and can be used to improve
              its software development processes. The findings generated from an SCE
              identify (1) process strengths that mitigate risks, (2) process weaknesses
              that increase risks, and (3) improvement activities that indicate potential
              mitigation of risks.




              2
               The SEI defines institutionalization as the building of an infrastructure and corporate culture that
              suggest methods, practices, and procedures so that they become the ongoing way of doing business,
              even after those who originally defined them are gone. Software Capability Evaluation, Version 3.0,
              Method Description (CMU/SEI-96-TR-002, April 1996).
              3
               Version 3.0 of the SCE method is based on SEI’s capability maturity model, version 1.1.



              Page 3                                           GAO/AIMD-97-41 Defense Financial Management
                                           B-276764




Table 1: CMM Levels and Descriptions
                                           Level                     Name                     Description
                                           5                         Optimizing               Continuous process improvement is
                                                                                              enabled by quantitative feedback from the
                                                                                              process and from piloting innovative ideas
                                                                                              and technologies.
                                           4                         Managed                  Detailed measures of the software process
                                                                                              and product quality are collected. Both the
                                                                                              software process and products are
                                                                                              quantitatively understood and controlled.
                                           3                         Defined                  The software process for both management
                                                                                              and engineering activities is documented,
                                                                                              standardized, and integrated into a
                                                                                              standard software process for the
                                                                                              organization. All projects use an approved,
                                                                                              tailored version of the organization’s
                                                                                              standard software process for developing
                                                                                              and maintaining software.
                                           2                         Repeatable               Basic project management processes are
                                                                                              established to track cost, schedule, and
                                                                                              functionality. The necessary process
                                                                                              discipline is in place to repeat earlier
                                                                                              successes on projects with similar
                                                                                              applications.
                                           1                         Initial                  The software process is characterized as
                                                                                              ad hoc, and occasionally even chaotic.
                                                                                              Few processes are defined, and success
                                                                                              depends on individual effort.
                                           Note: According to an SEI study (Moving on Up: Data and Experience Doing CMM-Based
                                           Process Improvement, Technical Report CMU/SEI-95-TR-008, August 1995) of 48 organizations
                                           that implemented software process improvement programs, the time required to increase process
                                           maturity from level 1 to level 2 took an average of 30 months, with a range of 11 months to 58
                                           months.

                                           Source: Capability Maturity Model for Software, Version 1.1, (Technical Report
                                           CMU/SEI-93-TR-24, February 1993).



                                           We requested that FSA-Indianapolis identify for our evaluation those
                                           projects that best represented their software development processes
                                           implemented at Indianapolis. The Director, FSA-Indianapolis identified two
                                           SPI projects and two non-SPI projects, as follows:



SPI Projects                           •   Defense Transportation Payment System (DTRS)
                                       •   Standard Army Financial Inventory Accounting and Reporting System
                                           Modernization (STARFIARS-MOD)




                                           Page 4                                         GAO/AIMD-97-41 Defense Financial Management
                       B-276764




Non-SPI Projects   •   Corps of Engineers Financial Management System (CEFMS)4
                   •   Standard Finance System (STANFINS)

                       We evaluated the software development processes used on these projects,
                       focusing on the key process areas necessary to achieve a repeatable
                       capability. In particular, the team evaluated the degree of implementation
                       and institutionalization of all KPA goals in accordance with the SCE
                       methodology. Accordingly, rating judgments were made at the goal level. A
                       goal is satisfied if the associated findings indicate that this goal is
                       implemented and institutionalized either as defined in CMM, with no
                       significant weaknesses, or that an adequate alternative exists.

                       Organizations that have a repeatable software development process—one
                       that can be counted on to render the same results if the same processes
                       are followed—have been able to significantly improve their productivity
                       and return on investment. According to SEI,5 processes for a repeatable
                       capability (CMM level 2) are considered the most basic in establishing
                       discipline and control in software development and are crucial steps for
                       any project to mitigate risks associated with cost, schedule, and quality. As
                       shown in table 2, these processes include (1) requirements management,
                       (2) software project planning, (3) software project tracking and oversight,
                       (4) software subcontract management, (5) software quality assurance, and
                       (6) software configuration management.




                       4
                        CEFMS represents FSA-Indianapolis’ attempt to adapt the Corps of Engineers’ Financial Management
                       System (CEFMS) for Army posts, camps, and stations.
                       5
                        Software Capability Evaluation, Version 3.0, Method Description (CMU/SEI-96-TR-002, April 1996).



                       Page 5                                         GAO/AIMD-97-41 Defense Financial Management
                                    B-276764




Table 2: CMM Level 2 “Repeatable”
Key Process Area Descriptions       CMM Level 2 KPAs                             Description
                                    Requirements management                      Defining, validating, and prioritizing
                                                                                 requirements, such as functions,
                                                                                 performance, and delivery dates.
                                    Software project planning                    Developing estimates for the work to be
                                                                                 performed, establishing the necessary
                                                                                 commitments, and defining the plan to
                                                                                 perform the work.
                                    Software project tracking and oversight      Tracking and reviewing software
                                                                                 accomplishments and results against
                                                                                 documented estimates, commitments, and
                                                                                 plans and adjusting these based on the
                                                                                 actual accomplishments and results.
                                    Software subcontract management              Selecting qualified contractors and
                                                                                 managing them effectively.
                                    Software quality assurance                   Reviewing and auditing the software
                                                                                 products and activities to ensure that they
                                                                                 comply with the applicable processes,
                                                                                 standards, and procedures and providing
                                                                                 the staff and managers with the results of
                                                                                 their reviews and audits.
                                    Software configuration management            Selecting project baseline items, such as
                                                                                 specifications; systematically controlling
                                                                                 these items and changes to them; and
                                                                                 recording and reporting status and change
                                                                                 activity for these items.

                                    The Department of Defense provided written comments on a draft of this
                                    report. These comments are presented and evaluated at the end of this
                                    report and are reprinted in appendix I. We conducted our review from
                                    August 1996 through February 1997 in accordance with generally accepted
                                    government auditing standards.


                                    In order for FSA-Indianapolis to be rated at CMM level 2, all evaluated
FSA-Indianapolis                    projects would have to pass every level 2 KPA. As shown in appendix III,
Software                            this is not the case. No project passed every KPA, nor was there a single KPA
Development                         that was passed by every project. Therefore, we conclude that
                                    FSA-Indianapolis as an organization remains a long way from achieving the
Processes Are                       repeatable level of maturity (level 2).
Immature
                                    Organizations that have not developed the process discipline necessary to
                                    better manage and control their projects at the repeatable level incur
                                    greater risk of schedule delay, cost overruns, and poor quality software.
                                    To mitigate this, such organizations typically rely upon the variable




                                    Page 6                                    GAO/AIMD-97-41 Defense Financial Management
    B-276764




    capabilities of individuals, rather than on institutionalized processes
    considered basic to software development.

    Highlights of our evaluation of the four projects follow.

•   Requirements Management. The purpose of requirements management is
    to establish a common understanding between the customer and the
    software project of the customer’s requirements that will be addressed by
    the software project.

    DTRS was the only project that met all of the goals for requirements
    management. Specifically, for this project, functional and performance
    requirements and delivery dates were defined, validated, and prioritized.

    Other than DTRS, the projects’ functional requirements were not adequately
    reviewed at the early stages of the software development life cycle.
    Specifically, the configuration control boards (CCBs) used in
    FSA-Indianapolis were responsible primarily for funding decisions but not
    the review and authorization of the establishment of and changes to
    software baselines. This situation can lead to wasted effort developing
    requirements which may be technically infeasible.

    In addition, the CEFMS project and its prime contractor in FSA-Indianapolis
    depended on a subcontractor to perform the requirements management
    function, but the subcontractor did not satisfy any of the goals within the
    requirements management KPA. Specifically, although the contractor
    reviewed software change requests before they were incorporated into the
    CEFMS project, a baseline of requirements was not established. Without a
    baseline, it is difficult to manage changes to the project and maintain the
    stability of the software produced from release to release.




    Page 7                              GAO/AIMD-97-41 Defense Financial Management
                                                 B-276764




Table 3: Results for the Requirements Management Key Process Area
                                                                                                  Project
Requirements management goal                         STANFINS                DTRS                      STARFIARS-MOD             CEFMS
System requirements allocated to software Unsatisfied                        Satisfied                 Unsatisfied               Unsatisfied
are controlled to establish a baseline for
software engineering and management use.
Activity:

—The software engineering group reviews the
allocated requirements before they are
incorporated into the software project.
Software plans, products, and activities are         Unsatisfied             Satisfied                 Satisfied                 Partially satisfied
kept consistent with the system
requirements allocated to software.
Activities:

—The software engineering group uses the
allocated requirements as the basis for
software plans, work products, and activities.

—Changes to the allocated requirements are
reviewed and incorporated into the software
project.
                                                 Satisfied - Practices that achieve the intent of the goal were implemented.

                                                 Unsatisfied - Weaknesses that significantly impact the goal exist.

                                                 Partially satisfied - Practices that achieve the intent of the goal were implemented but not
                                                 institutionalized.



                                            •    Software Project Planning. The purpose of software project planning is to
                                                 establish reasonable plans for performing the software engineering and for
                                                 managing the software project.

                                                 DTRS  and STARFIARS-MOD had software development plans and documented
                                                 software estimates (e.g., effort, cost, and schedule) for the project.
                                                 Further, STARFIARS-MOD had recently implemented function point analysis6
                                                 to estimate software size. On the other hand, software risks associated
                                                 with the cost, resource, schedule, and technical aspects of the projects
                                                 were not adequately identified, assessed, or documented for any of the
                                                 four projects evaluated. Without risk assessment, the reliability of
                                                 estimates is questionable, and the ability of a project to meet its schedule
                                                 is reduced.

                                                 6
                                                  Function points are derived using an empirical relationship based on countable measures (e.g.,
                                                 number of user inputs, number of user outputs, number of user inquiries, number of files, and number
                                                 of external interfaces) of software’s information domain and assessments of software complexity.



                                                 Page 8                                           GAO/AIMD-97-41 Defense Financial Management
                                               B-276764




Table 4: Results for the Software Project Planning Key Process Area
                                                                                  Project
Software project planning goal                   STANFINS        DTRS                  STARFIARS-MOD         CEFMS
Software estimates are documented for use        Unsatisfied     Partially satisfied   Partially satisfied   Unsatisfied
in planning and tracking the software
project.
Activities:

—Estimates for the size of the software work
products (or changes to the size of software
work products) are derived according to a
documented procedure.

—Estimates for the software project’s effort
and costs are derived according to a
documented procedure.

—Estimates for project’s critical computer
resources are derived according to a
documented procedure.

—The project’s software schedule is derived
according to a documented procedure.

—Software planning data are recorded.
                                                                                                                     (continued)




                                               Page 9                              GAO/AIMD-97-41 Defense Financial Management
                                                 B-276764




                                                                                 Project
Software project planning goal                       STANFINS   DTRS                  STARFIARS-MOD         CEFMS
Software project activities and         Unsatisfied             Partially satisfied   Partially satisfied   Unsatisfied
commitments are planned and documented.
Activities:

—Software project planning is initiated in the
early stages of, and in parallel with, the overall
project planning.

—A software life cycle with predefined stages
of manageable size is identified or defined.

—The project’s software development plan is
developed according to a documented
procedure.

—The plan for the software project is
documented.

—Software work products that are needed to
establish and maintain control of the software
project are identified.

—The software risks associated with the cost,
resource, schedule, and technical aspects of
the project are identified, assessed, and
documented.

—Plans for the project’s software engineering
facilities and support tools are prepared.
                                                                                                                    (continued)




                                                 Page 10                          GAO/AIMD-97-41 Defense Financial Management
                                                B-276764




                                                                                                 Project
Software project planning goal                      STANFINS                DTRS                      STARFIARS-MOD             CEFMS
Affected groups and individuals agree to            Unsatisfied             Satisfied                 Partially satisfied       Unsatisfied
their commitments related to the software
project.
Activities:

—The software engineering group participates
on the project proposal team.

—The software engineering group participates
with other affected groups in the overall project
planning throughout the project’s life.

—Software project commitments made to
individuals and groups external to the
organization are reviewed with senior
management according to a documented
procedure.

                                                Satisfied - Practices that achieve the intent of the goal were implemented.

                                                Unsatisfied - Weaknesses that significantly impact the goal exist.

                                                Partially satisfied - Practices that achieve the intent of the goal were implemented but not
                                                institutionalized.



                                            •   Software Project Tracking and Oversight. The purpose of software project
                                                tracking and oversight is to provide adequate visibility into actual progress
                                                so that management can take effective actions when the software project’s
                                                performance deviates significantly from software plans.

                                                FSA-Indianapolis   projects that were evaluated underwent periodic status
                                                reviews at meetings with key personnel present, and changes to
                                                commitments were generally agreed to by the affected groups and
                                                individuals. However, software risks associated with cost, resource,
                                                schedule, and technical aspects of the projects were not tracked.
                                                Moreover, although the SPI projects tracked performance and actual
                                                results, a mechanism for making corrections if and when projects failed to
                                                meet estimates was not in place. As a result of these weaknesses, the
                                                projects reviewed are more likely to be affected by unplanned events and
                                                are less likely to meet schedule and cost commitments.




                                                Page 11                                          GAO/AIMD-97-41 Defense Financial Management
                                               B-276764




Table 5: Results for the Software Project Tracking and Oversight Key Process Area
Software project tracking and oversight                                          Project
goal                                             STANFINS        DTRS                  STARFIARS-MOD         CEFMS
Actual results and performances are              Unsatisfied     Partially satisfied   Partially satisfied   Unsatisfied
tracked against the software plans.
Activities:

—A documented software development
plan is used for tracking the software
activities and communicating status.

—The size of the software work products (or
size of the changes to the software work
products) is tracked, and corrective actions
are taken as necessary.

—The project’s software effort and costs
are tracked, and corrective actions are
taken as necessary.

—The project’s critical computer resources
are tracked, and corrective actions are
taken as necessary.

—The project’s software schedule is
tracked, and corrective actions are taken as
necessary.

—Software engineering technical activities
are tracked, and corrective actions are
taken as necessary.

—The software risks associated with cost,
resource, schedule, and technical aspects
of the project are tracked.

—Actual measurement data and replanning
data for the software project are recorded.

—The software engineering group conducts
periodic internal reviews to track technical
progress, plan, performance, and issues
against the software development plan.

—Formal reviews to address the
accomplishments and results of the
software project are conducted at selected
project milestones according to a
documented procedure.
                                                                                                                     (continued)




                                               Page 12                             GAO/AIMD-97-41 Defense Financial Management
                                               B-276764




Software project tracking and oversight                                         Project
goal                                             STANFINS      DTRS                  STARFIARS-MOD         CEFMS
Corrective actions are taken and                 Unsatisfied   Partially satisfied   Partially satisfied   Unsatisfied
managed to closure when actual results
and performance deviate significantly
from the software plans.
Activities:

—The project’s software development plan
is revised according to a documented
procedure.

—The size of the software work products (or
size of the changes to the software work
products) is tracked, and corrective actions
are taken as necessary.

—The project’s software effort and costs
are tracked, and corrective actions are
taken as necessary.

—The project’s critical computer resources
are tracked, and corrective actions are
taken as necessary.

—The project’s software schedule is
tracked, and corrective actions are taken as
necessary.

—Software engineering technical activities
are tracked, and corrective actions are
taken as necessary.

—Actual measurement data and replanning
data for the software project are recorded.
                                                                                                                   (continued)




                                               Page 13                           GAO/AIMD-97-41 Defense Financial Management
                                              B-276764




Software project tracking and oversight                                                        Project
goal                                            STANFINS                  DTRS                      STARFIARS-MOD             CEFMS
Changes to software commitments are             Unsatisfied               Satisfied                 Satisfied                 Partially satisfied
agreed to by the affected groups and
individuals.
Activities:

—Software project commitments and
changes to commitments made to
individuals and groups external to the
organization are reviewed with senior
management according to a documented
procedure.

—Approved changes to commitments that
affect the software project are
communicated to the members of the
software engineering group and other
software-related groups.

                                              Satisfied - Practices that achieve the intent of the goal were implemented.

                                              Unsatisfied - Weaknesses that significantly impact the goal exist.

                                              Partially satisfied - Practices that achieve the intent of the goal were implemented but not
                                              institutionalized.



                                          •   Software Subcontract Management. The purpose of software subcontract
                                              management is to select qualified software subcontractors and manage
                                              them effectively.

                                              A contractual agreement between the government and the software
                                              contractors was used as a basis for managing the contracts and
                                              contractors. The projects had also designated a contracting officer
                                              representative to be responsible for establishing and managing software
                                              task orders. However, FSA-Indianapolis was unable to produce a written
                                              organizational policy describing the process for managing software
                                              contracts. Also, the contractor-developed software plans (e.g., software
                                              development plan, configuration management plan, and quality assurance
                                              plan), which the government would use to track contractors’ progress,
                                              either (1) did not exist or (2) were not specific to the particular project
                                              under contract. The lack of an approved organizational policy removes an
                                              important source of guidance for project personnel; hence, there is a
                                              higher risk that individual projects will manage software contractors
                                              inconsistently with wide-ranging results. Accordingly, it would be prudent
                                              for FSA-Indianapolis to ensure that software contractors have a CMM rating
                                              of at least level 2. Neither the DTRS nor the STARFIARS-MOD project had
                                              contractor support, and therefore were not evaluated against this KPA.


                                              Page 14                                          GAO/AIMD-97-41 Defense Financial Management
                                                  B-276764




Table 6: Results for the Software Subcontract Management Key Process Area
                                                                                  Project
Software subcontract management goal                STANFINS      DTRS                STARFIARS-MOD       CEFMS
The organization selects qualified software         Unsatisfied   Not evaluated       Not evaluated       Unsatisfied
subcontractors.
Activities:

—The work to be subcontracted is defined and
planned according to a documented
procedure.

—The software subcontractor is selected,
based on an evaluation of the subcontract
bidders’ ability to perform the work, according
to a documented procedure.
The organization and the software                   Unsatisfied   Not evaluated       Not evaluated       Partially satisfied
subcontractor agree to their commitments
to each other.
Activities:

—The contractual agreement between the
prime contractor and the software
subcontractor is used as the basis for
managing the subcontract.

—A documented subcontractor’s software
development plan is reviewed and approved
by the prime contractor.

—Changes to the software subcontractor’s
statement of work, subcontract terms and
conditions, and other commitments are
resolved according to a documented
procedure.
                                                                                                                   (continued)




                                                  Page 15                         GAO/AIMD-97-41 Defense Financial Management
                                             B-276764




                                                                               Project
Software subcontract management goal             STANFINS      DTRS                STARFIARS-MOD       CEFMS
The organization and the software                Unsatisfied   Not evaluated       Not evaluated       Partially satisfied
subcontractor maintain ongoing
communications.
Activities:

—The prime contractor’s management
conducts periodic status/coordination reviews
with the software subcontractor’s management.

—Periodic technical reviews and interchanges
are held with the software subcontractor.

—Formal reviews to address the
subcontractor’s software engineering
accomplishments and results are conducted at
selected milestones according to a
documented procedure.

—The software subcontractor’s performance is
evaluated on a periodic basis, and the
evaluation is reviewed with the subcontractor.
                                                                                                                (continued)




                                             Page 16                           GAO/AIMD-97-41 Defense Financial Management
                                                B-276764




                                                                                Project
Software subcontract management goal              STANFINS      DTRS                STARFIARS-MOD       CEFMS
The organization tracks the software              Unsatisfied   Not evaluated       Not evaluated       Unsatisfied
subcontractors’ actual results and
performance against its commitments.
Activities:

—The contractual agreement between the
prime contractor and the software
subcontractor is used as the basis for
managing the subcontract.

—A documented and approved
subcontractor’s software development plan is
used for tracking the software activities and
communicating status.

—The prime contractor’s management
conducts periodic status/coordination reviews
with the software subcontractor’s management.

—Formal reviews to address the
subcontractor’s software engineering
accomplishments and results are conducted at
selected milestones according to a
documented procedure.

—The prime contractor’s software quality
assurance group monitors the subcontractor’s
software quality assurance activities according
to a documented procedure.

—The prime contractor’s software
configuration management group monitors the
subcontractor’s activities for software
configuration management according to a
documented procedure.

—The prime contractor conducts acceptance
testing as part of the delivery of the
subcontractor’s software products according
to a documented procedure.

—The software subcontractor’s performance is
evaluated on a periodic basis, and the
evaluation is reviewed with the subcontractor.

                                                                                                     (Table notes on next page)




                                                Page 17                         GAO/AIMD-97-41 Defense Financial Management
    B-276764




    Satisfied - Practices that achieve the intent of the goal were implemented.

    Unsatisfied - Weaknesses that significantly impact the goal exist.

    Partially satisfied - Practices that achieve the intent of the goal were implemented but not
    institutionalized.

    Not evaluated - Goal did not apply in the organization’s environment or insufficient evidence to
    rate the goal.



•   Software Quality Assurance. The purpose of software quality assurance is
    to provide management with appropriate visibility into the process being
    used by the software project and of the products being built. SQA involves
    reviewing and auditing the software products and activities to verify that
    they comply with applicable procedures and standards.

    FSA-Indianapolis had a software quality assurance group, but the
    STARFIARS-MOD  and CEFMS projects did not perform SQA activities during our
    evaluation, nor were their SQA personnel available for interviews.
    Therefore, these projects were not evaluated against this KPA. Moreover,
    although DTRS performed SQA activities, these were focused on the
    individual product, rather than on the overall process.7 This is important
    because while focusing on the product can improve that particular item,
    focusing on the process ensures the consistent quality of all products.
    Further, there was no evidence of verification of processes and standards
    by the project SQA staff. Without process-focused SQA, FSA-Indianapolis
    cannot be certain that (1) its established software development processes
    are being followed as intended and (2) deviations from software standards
    and procedures are identified. Unless these basic issues are addressed,
    FSA-Indianapolis will have difficulty improving its processes.




    7
     According to SEI, the process used for developing products should be defined, understood, measured,
    and progressively improved. As process quality increases, management also has greater insight,
    understanding, and control of risks. (See Software Capability Evaluation (SCE) Version 2.0,
    Implementation Guide, CMU/SEI-94-TR-5, February 1994).



    Page 18                                          GAO/AIMD-97-41 Defense Financial Management
                                                 B-276764




Table 7: Results for the Software Quality Assurance Key Process Area
                                                                                    Project
Software quality assurance goal                      STANFINS      DTRS                  STARFIARS-MOD       CEFMS
Software quality assurance activities are            Unsatisfied   Partially satisfied   Not evaluated       Not evaluated
planned.
Activities:

—An SQA plan is prepared for the software
project according to a documented procedure.

—The SQA group’s activities are performed in
accordance with the SQA plan.
Adherence of software products and                   Unsatisfied   Partially satisfied   Not evaluated       Not evaluated
activities to the applicable standards,
procedures, and requirements is verified
objectively.
Activities:

—The SQA group’s activities are performed in
accordance with the SQA plan.

—The SQA group participates in the
preparation and review of the project’s
software development plan, standards, and
procedures.

—The SQA group reviews the software
engineering activities to verify compliance.

—The SQA group audits designated software
work products to verify compliance.
Affected groups and individuals are                  Unsatisfied   Partially satisfied   Not evaluated       Not evaluated
informed of software quality assurance
activities and results.
Activities:

—The SQA group periodically reports the
results of its activities to the software
engineering group.

—Deviations identified in the software activities
and software work products are documented
and handled according to a documented
procedure.

—The SQA group conducts periodic reviews
of its activities and findings with the customer’s
SQA personnel, as appropriate.
                                                                                                                     (continued)




                                                 Page 19                             GAO/AIMD-97-41 Defense Financial Management
                                                 B-276764




                                                                                                  Project
Software quality assurance goal                     STANFINS                 DTRS                      STARFIARS-MOD             CEFMS
Noncompliance issues that cannot be                 Unsatisfied              Not evaluated             Not evaluated             Not evaluated
resolved within the software project are
addressed by senior management.
Activity:

—Deviations identified in the software activities
and software work products are documented
and handled according to a documented
procedure.

                                                 Satisfied - Practices that achieve the intent of the goal were implemented.

                                                 Unsatisfied - Weaknesses that significantly impact the goal exist.

                                                 Partially satisfied - Practices that achieve the intent of the goal were implemented but not
                                                 institutionalized.

                                                 Not evaluated - Goal did not apply in the organization’s environment or insufficient evidence to
                                                 rate the goal.



                                             •   Software Configuration Management - The purpose of software
                                                 configuration management is to establish and maintain the integrity of
                                                 products of the software project throughout the project’s software
                                                 lifecycle.

                                                 STARFIARS-MOD  had a configuration management plan and DTRS had a draft
                                                 plan. In addition, both projects identified the software work products to be
                                                 placed under configuration management. However, no software
                                                 configuration control board (SCCB) existed for any of the projects to
                                                 authorize the establishment of a software baseline and the identification of
                                                 configuration items. Moreover, (1) the SCM procedures were inconsistent
                                                 within projects and these projects contained multiple library systems,
                                                 (2) some of the SCM systems were poorly documented, and (3) some SCM
                                                 staff were inexperienced and had inadequate training. For example, SCM
                                                 staff from the STANFINS and CEFMS projects had no formal training in
                                                 configuration management. These weaknesses increase the risk of being
                                                 unable to control software integrity uniformly within various projects,
                                                 thus potentially increasing software development time and costs, or
                                                 decreasing software product quality.




                                                 Page 20                                          GAO/AIMD-97-41 Defense Financial Management
                                                  B-276764




Table 8: Results for the Software Configuration Management Key Process Area
                                                                                   Project
Software configuration management goal              STANFINS      DTRS                  STARFIARS-MOD         CEFMS
Software configuration management                   Unsatisfied   Partially satisfied   Partially satisfied   Unsatisfied
activities are planned.
Activities:

—An SCM plan is prepared for each software
project according to a documented procedure.

—A documented and approved SCM plan is
used as the basis for performing the SCM
activities.
Selected software work products are                 Unsatisfied   Partially satisfied   Partially satisfied   Unsatisfied
identified, controlled, and available.
Activities:

—A documented and approved SCM plan is
used as the basis for performing the SCM
activities.

—A configuration management library system
is established as a repository for the software
baseline.

—The software work products to be placed
under configuration management are identified.

—Products from the software baseline library
are created and their release is controlled
according to a documented procedure.
Changes to identified software work                 Unsatisfied   Satisfied             Satisfied             Partially satisfied
products are controlled.
Activities:

—Change requests and problem reports for all
configuration items/units are initiated,
recorded, reviewed, approved, and tracked
according to a documented procedure.

—Changes to baselines are controlled
according to a documented procedure.
                                                                                                                       (continued)




                                                  Page 21                           GAO/AIMD-97-41 Defense Financial Management
                                            B-276764




                                                                                             Project
Software configuration management goal         STANFINS                 DTRS                      STARFIARS-MOD             CEFMS
Affected groups and individuals are            Unsatisfied              Partially satisfied       Satisfied                 Unsatisfied
informed of the status and content of
software baselines.
Activities:

—The status of configuration items/units is
recorded according to a documented
procedure.
—Standard reports documenting the SCM
activities and the contents of the software
baseline are developed and made available to
affected groups and individuals.
—Software baseline audits are conducted
according to a documented procedure.

                                            Satisfied - Practices that achieve the intent of the goal were implemented.

                                            Unsatisfied - Weaknesses that significantly impact the goal exist.

                                            Partially satisfied - Practices that achieve the intent of the goal were implemented but not
                                            institutionalized.




                                            FSA-Indianapolis has begun, through its software process improvement
Conclusions                                 program, to improve its software process for a limited number of projects.
                                            This is a significant positive step, but it has yet to satisfy all of the
                                            requirements for a level 2 capability. A significant amount of effort
                                            remains until the entire organization can demonstrate that it meets level 2
                                            criteria. Until then, significant risks remain that investments made in new
                                            software development will not achieve their operational improvement
                                            objectives and that software will not be delivered consistent with cost and
                                            schedule estimates and needs.


                                            To better position FSA-Indianapolis to develop and maintain its software
Recommendations to                          successfully and to protect its software investments, we recommend that
the Under Secretary                         you direct the Director of the Defense Finance and Accounting Service to:
of Defense
                                        •   Ensure that software configuration control functions are performed for
(Comptroller)                               each development project. The configuration control board that carries
                                            out these functions should include software engineering specialists, and
                                            authorize the software baseline, configuration items, and other relevant
                                            software products.




                                            Page 22                                          GAO/AIMD-97-41 Defense Financial Management
                         B-276764




                     •   Ensure that any future contracts or contract modifications for software
                         development include as part of the evaluation criteria that the
                         contractor(s) (1) have an independently assessed software development
                         capability of at least CMM level 2, (2) develop project specific software
                         plans, and (3) perform software quality assurance and software
                         configuration management activities.
                     •   Require that projects develop, document, and periodically update a risk
                         management plan that identifies and assesses risks to cost, schedule, and
                         quality goals. The plan should also outline strategies for mitigating the
                         risks, including mechanisms for corrective action when projects exceed
                         established thresholds.
                     •   Require that each development project perform both product- and
                         process-focused software quality assurance activities throughout the
                         system life cycle.
                     •   Ensure that each development project (1) prepares a software
                         configuration management plan that addresses all work products to be
                         placed under configuration management and (2) follows a documented
                         procedure.
                     •   Expedite the promulgation of FSA-Indianapolis policies and procedures for
                         software development.
                     •   Delay any major investment in software development for projects at
                         FSA-Indianapolis beyond that needed to sustain critical day-to-day
                         operations until the repeatable level of process maturity (level 2) is
                         attained and validated through an independent performance audit or, at a
                         minimum, until the above recommendations are fully implemented.


                         In commenting on a draft of this report, DOD officials generally agreed with
Agency Comments          the intent of our recommendations, with one exception. DOD disagreed
and Our Evaluation       with our recommendation to delay major investments in software
                         development for projects at FSA-Indianapolis beyond that needed to
                         sustain critical day-to-day operations until it satisfies a level 2 capability.
                         Moreover, they expressed concern relative to either the scope of the
                         recommendations or how the recommendations should be implemented.

                         DOD  disagreed with our assessment that FSA-Indianapolis does not yet
                         satisfy the criteria for a level 2 (i.e., repeatable) software development
                         capability for any of the four projects we reviewed, stating that the
                         Defense Transportation Payment System (DTRS) achieved a level 2
                         software development capability in November 1996. As discussed in this
                         report, in September 1996 when we evaluated the four projects chosen for
                         the SCE, none of the projects satisfied all six of the key process areas to



                         Page 23                              GAO/AIMD-97-41 Defense Financial Management
B-276764




qualify as level 2. However, one project, DTRS, showed more progress than
the others by satisfying the criteria in one key process area. Further, as
noted in our report, DTRS was rated as either partially or fully satisfying
most of the goals within the remaining five key process areas. Accordingly,
given the time lapse between our evaluation and the issuance of this
report, it is possible that the internal SCE performed by DFAS could have
resulted in a level 2 rating for this one project. The three other systems
(i.e., the Defense Business Management System, the Marine Corps Total
Force System, and the Defense Civilian Pay System) that DOD asserted
were level 2 systems were not developed by FSA-Indianapolis, and
therefore are not relevant to the FSA-Indianapolis software development
capability rating. However, in our view, the major issue is not whether
FSA-Indianapolis has any supportable level 2 projects but instead when will
FSA-Indianapolis become a level 2 organization.


DOD  partially concurred with our recommendations that, for each project,
DOD  should establish a Software Configuration Control Board that
collaborates with the functionally-oriented Configuration Control Board.
DOD agreed that the functions specified for an SCCB should be performed
but stated that it would rather place the functions within the existing CCB
as opposed to establishing another board. Our major concern was to
ensure that software configuration control functions are performed for
each project, and that a configuration control board consisting of software
engineering specialists controlled this activity. Accordingly, the
DOD-suggested alignment would be appropriate if the existing CCB
membership includes appropriate technical representation.

DOD partially concurred with our recommendation that future contract
modifications for software development require the contractor to have an
independently assessed software development capability of at least CMM
level 2. While agreeing in principle that CMM level 2 is desirable, DOD stated
that “. . .to use the CMM level 2 measurement as a sole discriminator in
future contractor awards would preclude the use of a much needed quality
assurance service available from other contractors. On future projects
under the subcontract management Key Process Area, an evaluation
criteria will be included in contracts for consideration on contractors
having level 2 or higher capability.” We did not mean to imply that the CMM
level 2 criteria should be used as the sole determinant in evaluating
software development contractors; however, it should be a significant
criterion. Accordingly, FSA-Indianapolis should use the level 2 requirement
as an important criterion in software development procurements to reduce




Page 24                              GAO/AIMD-97-41 Defense Financial Management
B-276764




the risk of schedule slippage, cost overruns, and poor software quality. We
modified our recommendation to clarify this point.

DOD partially concurred with our recommendation that projects develop,
document, and periodically update a risk management plan that identifies
and assesses risks to cost, schedule, and quality goals. DOD agreed with the
function and actions included in our recommendation, but stated that
these are not required because a plan to manage risk is not a requirement
at CMM level 2. Within the Software Project Planning (SPP) key process
area for CMM level 2, our SCE team evaluated FSA-Indianapolis against
activity 13 which states, “The software risks associated with the cost,
resource, schedule, and technical aspects of the project are identified,
assessed, and documented” as well as the other activities required to
satisfy this key process area. Thus, contrary to DOD’s assertion that this
activity is not required until level 3, the CMM cites this activity as one of the
specific requirements for satisfying the level 2 Software Project Planning
key process area. During the SCE, no project, including DTRS, was able to
demonstrate that it was identifying, assessing, and documenting the
software risks associated with cost, schedule, resource, and technical
aspects of the project. Further, the level 3 requirement referred to by DOD
builds upon the level 2 requirement cited above.

DOD  partially concurred with our recommendation to require that each
project perform both product- and process-focused software quality
assurance activities throughout the system life cycle. However, the
Department disagreed that these functions should be performed on each
project managed by FSA-Indianapolis stating that “The Department believes
that the appropriateness of these activities should be determined on a
project-by-project basis that considers cost versus the limited life cycle of
many of the legacy and interim migratory systems.” Similarly, DOD partially
concurred with our recommendation to require that each project
(1) prepare a configuration management plan that addresses all work
products to be placed under configuration management and (2) follow a
documented procedure stating that implementation of this
recommendation should also be made on a project-by-project basis. We
agree that these activities should not apply to systems that (1) are
currently operating as legacy systems, (2) have a short (i.e., 1- or 2-year)
life cycle, and (3) are due for replacement within 1 or 2 years. However,
because the CMM level 2 rating is dependent on satisfying the requirement
for all projects, FSA-Indianapolis cannot reach level 2 as an organization
until all the key process areas, including software quality assurance and




Page 25                               GAO/AIMD-97-41 Defense Financial Management
B-276764




software configuration management, are satisfied. We modified these
recommendations to clarify this provision.

Finally, DOD disagreed with our recommendation that FSA-Indianapolis
delay any major investment in software development beyond that needed
to sustain critical day-to-day operations until a repeatable level of maturity
is reached. DOD indicated that delaying major development efforts until all
projects achieve a level 2 would result in significant impacts on their
system development initiatives and schedules. We disagree with Defense’s
contention. The recommendation does limit DOD’s ability to take on the
development of major automated information systems; however, it still
permits FSA-Indianapolis to (1) maintain and modify existing operational
systems and (2) develop prototypes and proof of concept systems. As a
level 1 organization, FSA-Indianapolis will still be taking on the risk of
schedule slippage, cost overruns, and poor quality software. DOD will have
to be selective in its choice of development initiatives so as not to repeat
the failed startups of the past. For example, in its attempt at the Military
Pay Redesign, the U.S. Army Finance and Accounting Center (now known
as FSA-Indianapolis) experienced schedule slippage from September 1984
to September 1993 and a cost overrun from $14.6 million to $82 million. In
our report, Army Decision to Use Air Force Military Pay System Appears
Advantageous (GAO/IMTEC-89-28, March 1989) we found that there was a lack
of planning and documentation of risks. Furthermore, in its report on the
Corps of Engineers Financial Management System (Report No. 97-051,
December 18, 1996), the DOD’s Office of the Inspector General stated that
the DFAS Indianapolis Center “had not developed detailed plans for
reducing the risks of achieving the expected performance of CEFMS.” We
believe that this was instrumental in the decision to classify CEFMS as a
special interest program subject to the Major Automated Information
Systems Review Council review.


We are sending copies of this report to the Chairmen and Ranking
Minority Members of the House Committee on Government Reform and
Oversight, Subcommittee on Government Management, Information and
Technology; and the Senate Committee on Governmental Affairs. We are
also sending copies to the Director of the Office of Management and
Budget and the Secretary of Defense. Copies will also be made




Page 26                              GAO/AIMD-97-41 Defense Financial Management
B-276764




available to others upon request. If you have any questions or wish to
discuss the issues in this letter, please contact me at (202) 512-6234. Major
contributors to this report are listed in appendix IV.

Sincerely yours,




William S. Franklin
Director, Information Systems Methodology & Support




Page 27                             GAO/AIMD-97-41 Defense Financial Management
Contents



Letter                                                                                               1


Appendix I                                                                                          30

Comments From the
Department of
Defense
Appendix II                                                                                         36

Software Process
Improvement
Program Financial
Systems Activities and
Automated
Information Systems
Appendix III                                                                                        38

Summary of
FSA-Indianapolis SCE
Results
Appendix IV                                                                                         39

Major Contributors to
This Report
Tables                   Table 1: CMM Levels and Descriptions                                        4
                         Table 2: CMM Level 2 “Repeatable” Key Process Area                          6
                           Descriptions
                         Table 3: Results for the Requirements Management Key Process                8
                           Area
                         Table 4: Results for the Software Project Planning Key Process              9
                           Area
                         Table 5: Results for the Software Project Tracking and Oversight           12
                           Key Process Area




                         Page 28                            GAO/AIMD-97-41 Defense Financial Management
Contents




Table 6: Results for the Software Subcontract Management Key              15
  Process Area
Table 7: Results for the Software Quality Assurance Key Process           19
  Area
Table 8: Results for the Software Configuration Management Key            21
  Process Area




Abbreviations

AIS                automated information system
ASQC               American Society for Quality Control
CCB                configuration control board
CDA                central design activity
CEFMS              Corps of Engineers Financial Management System
CFO                Chief Financial Officer
CMM                capability maturity model
DFAS               Defense Finance and Accounting Service
DOD                Department of Defense
DTRS               Defense Transportation Payment System
FSA                financial systems activity
FSO                financial systems organization
KPA                key process area
SCCB               software configuration control board
SCE                software capability evaluation
SCM                software configuration management
SDS                system development scenario
SEI                Software Engineering Institute
SMS                system modification scenario
SPI                software process improvement
SQA                software quality assurance
STANFINS           Standard Finance System
STARFIARS-MOD      Standard Army Financial Inventory Accounting
                        Reporting System Modernization


Page 29                           GAO/AIMD-97-41 Defense Financial Management
Appendix I

Comments From the Department of Defense




             Page 30      GAO/AIMD-97-41 Defense Financial Management
Appendix I
Comments From the Department of Defense




Page 31                               GAO/AIMD-97-41 Defense Financial Management
Appendix I
Comments From the Department of Defense




Page 32                               GAO/AIMD-97-41 Defense Financial Management
Appendix I
Comments From the Department of Defense




Page 33                               GAO/AIMD-97-41 Defense Financial Management
Appendix I
Comments From the Department of Defense




Page 34                               GAO/AIMD-97-41 Defense Financial Management
Appendix I
Comments From the Department of Defense




Page 35                               GAO/AIMD-97-41 Defense Financial Management
Appendix II

Software Process Improvement Program
Financial Systems Activities and Automated
Information Systems
                       In August 1995, the FSO developed a strategic plan identifying the
                       movement to a standard process as a key element in improving the
                       efficiency of the FSO and in achieving its productivity goals. The plan’s 10
                       major objectives were to

                   •   achieve CMM level 2 for an initial FSO system (FY 96),
                   •   identify and implement productivity metrics for SMS
                       (FY 96),
                   •   define system development scenario (SDS) rapid prototyping (FY 96),
                   •   identify and implement productivity metrics for SDS rapid prototyping
                       (FY 97),
                   •   define SDS (FY 97),
                   •   identify and integrate standard function point analysis tools (FY 97),
                   •   achieve CMM level 2 for all migratory systems (FY 97),
                   •   identify and implement productivity metrics for SDS (FY 98),
                   •   complete full Ada development and maintenance capability (FY 98), and
                   •   achieve CMM level 3 for first migratory system (FY 99).

                       In an August 1996 briefing to the GAO team of SEI-trained specialists
                       conducting the software capability evaluations at FSA-Indianapolis, the SPI
                       program director stated that SPI officially began in November 1993, with its
                       objectives being to (1) create a single, standard set of development
                       processes, (2) improve and standardize development, modification, and
                       reengineering processes, and (3) establish a basis for measuring
                       performance. The activities and systems under SPI were:


FSA-Cleveland      •   Defense Retiree and Annuitant Pay System.

FSA-Columbus       •   Defense Business Management System.

FSA-Denver         •   Defense Debt Management System.
                   •   Defense Joint Military Pay System.
                   •   Defense Retiree and Annuitant Pay System.


FSA-Indianapolis   •   Defense Transportation Payment System.
                   •   Standard Army Financial Inventory Accounting and Reporting
                       System-Modernization.


FSA-Kansas City    •   Marine Corps Total Force System.



                       Page 36                              GAO/AIMD-97-41 Defense Financial Management
                    Appendix II
                    Software Process Improvement Program
                    Financial Systems Activities and Automated
                    Information Systems




FSA-Pensacola   •   Defense Civilian Pay System.
                •   Fund Administration and Standardized Document Automation System.

                    According to the FSO director, a software capability evaluation of the DTRS
                    project, which was performed in December 1996, found that it satisfied all
                    level 2 KPAs. He further asserted that the FSA-Kansas City-based Marine
                    Corps Total Force System and the FSA-Pensacola-based Defense Civilian
                    Pay System projects were also rated as level 2 projects. These software
                    capability evaluations were performed by FSO staff.




                    Page 37                                  GAO/AIMD-97-41 Defense Financial Management
Appendix III

Summary of FSA-Indianapolis SCE Results



                                                                     Project
               KPA            STANFINS         DTRS                    STARFIARS-MOD CEFMS
               RM             Fail             Pass                    Fail                   Fail
               SPP            Fail             Fail                    Fail                   Fail
               SPTO           Fail             Fail                    Fail                   Fail
               SSM            Fail             Not evaluated           Not evaluated          Fail
               SQA            Fail             Fail                    Not evaluated          Not evaluated
               SCM            Fail             Fail                    Fail                   Fail
               Pass - All goals corresponding to the KPA were satisfied.

               Fail - One or more goals corresponding to the KPA was not satisfied.

               Not evaluated - KPA did not apply in the organization’s environment or insufficient evidence to
               rate the KPA.




               Page 38                                         GAO/AIMD-97-41 Defense Financial Management
Appendix IV

Major Contributors to This Report


                       David Chao, ASQC Certified Software Quality Engineer, SEI Authorized Lead
Accounting and           Evaluator
Information            Gary R. Austin, SCE Team Member
Management Division,   K. Alan Merrill, SCE Team Member
                       Prithviraj Mukherji, Ph.D., SCE Team Member
Washington, D.C.       Frank W. Reilly, SCE Team Member
                       Paul Silverman, ASQC Certified Software Quality Engineer, SCE Team
                         Member
                       Nancy M. Donnellan, SCE Team Apprentice




(511502)               Page 39                           GAO/AIMD-97-41 Defense Financial Management
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested