oversight

Defense IRM: Poor Implementation of Management Controls Has Put Migration Strategy at Risk

Published by the Government Accountability Office on 1997-10-20.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to the Ranking Minority Member,
                Committee on Governmental Affairs,
                U.S. Senate


October 1997
                DEFENSE IRM
                Poor Implementation
                of Management
                Controls Has Put
                Migration Strategy at
                Risk




GAO/AIMD-98-5
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Accounting and Information
      Management Division

      B-275670

      October 20, 1997

      The Honorable John Glenn
      Ranking Minority Member
      Committee on Governmental Affairs
      United States Senate

      Dear Senator Glenn:

      As you know, in 1989, the Department of Defense started its Corporate Information
      Management (CIM) initiative in an effort to save billions of dollars by streamlining operations
      and deploying standard information systems to support common business operations. A key
      part of this initiative is Defense’s migration effort, which involves replacing its functionally
      duplicative and inefficient automated information systems with the Department’s best existing
      information systems. Concerned that the billions of dollars projected to be spent on CIM-related
      technology efforts are at high risk, you asked that we provide information on the status and
      progress of Defense’s migration effort and assess whether Defense has effective controls in
      place to manage and oversee the initiative. This report also highlights the implications of our
      findings for Defense as it responds to investment management requirements of the
      Clinger-Cohen Act of 1996.

      We are sending copies of this report to the Chairman of the Senate Committee on Governmental
      Affairs and to the Chairmen and Ranking Minority Members of the Senate Committee on Armed
      Services; Subcommittee on Defense, Senate Committee on Appropriations; House Committee
      on National Security; Subcommittee on National Security, House Committee on Appropriations;
      and Senate and House Committees on the Budget; the Secretary of Defense; the Acting
      Assistant Secretary of Defense for Command, Control, Communications and Intelligence; the
      Acting Under Secretary of Defense (Comptroller); and the Director, Office of Management and
      Budget. Copies will also be made available to others upon request.

      If you have any questions about this report, please call Mickey McDermott, Assistant Director,
      at (202) 512-6240. Other major contributors to this report are listed in appendix V.

      Sincerely yours,




      Jack L. Brock, Jr.
      Director, Defense Information and
        Financial Management Systems
Executive Summary


             In 1989, the Department of Defense started its Corporate Information
Purpose      Management (CIM) initiative in an effort to save billions of dollars by
             streamlining operations and deploying standard information systems to
             support common business operations. However, 8 years after beginning
             CIM and making substantial investments, Defense has not met its savings
             goal because it has not fully implemented sound management practices to
             carry out this initiative. GAO placed the effort on its high-risk list in
             February 1995, in part because it found that CIM-related technology
             investments, which are expected to total billions of dollars each year, are
             vulnerable to waste and mismanagement.

             A key part of CIM is Defense’s migration effort, which involves replacing its
             functionally duplicative and inefficient automated information systems
             with the best existing systems. Defense believes that migration can cut
             costs associated with developing and maintaining disparate systems
             supporting the same functions. It also believes that if done properly,
             migration can help standardize business processes. Concerned that
             billions of dollars projected to be spent on CIM-related technology efforts
             are at high risk, the Ranking Minority Member of the Senate Committee on
             Governmental Affairs asked GAO to (1) provide information on the number
             and cost of systems designated for migration, the number of legacy
             systems already terminated or scheduled for termination, and savings
             resulting from terminations of legacy systems and (2) determine whether
             Defense’s management control and oversight processes for migration
             systems are ensuring that the investments are economically sound and in
             compliance with its technical and data standards.


             When it initiated CIM, Defense believed that the thousands of automated
Background   information systems supporting its business operations—which include
             such functions as logistics, communications, personnel, health affairs, and
             finance—were redundant and inefficient. These operations were
             traditionally carried out in isolation by the individual military services and
             Defense agencies. As such, the information systems supporting these
             discrete operations were developed independently even though they may
             well have served similar purposes.

             DOD’s CIMinitiative included several aspects: (1) corporate policy/planning,
             (2) process and data modeling, (3) process improvement, (4) performance
             measurement, (5) standard information systems, and (6) computing and
             communications infrastructure. As CIM was implemented, Defense
             emphasized two ways of achieving process improvements and addressing



             Page 2                                   GAO/AIMD-98-5 Defense Migration Strategy
                   Executive Summary




                   problems associated with its disparate and stovepiped information
                   technology environment: (1) reengineering business processes first and
                   then applying technology to the new processes and (2) selecting the best
                   DOD information systems from pools of existing, or legacy, systems that
                   provide the same automated support services and eventually replacing the
                   duplicative systems with the best systems. The second approach is known
                   as migration.

                   In October 1993, Defense embarked on an “accelerated” migration
                   strategy, which placed more emphasis on the second improvement
                   approach. As part of this strategy, it asked its managers to select migration
                   systems in 6 months and develop and deploy them departmentwide in the
                   following 3 years. Defense believed that instilling pressure to select and
                   deploy migration systems would reap savings much quicker than
                   streamlining or reengineering the business processes and acquiring
                   systems after those processes were reengineered.

                   The consequence of the increased emphasis on migration was that the
                   dramatic gains that could be achieved through reengineering would be
                   postponed. This impaired the chances of CIM achieving its objectives for
                   dramatic improvements and cost savings in several respects. First, it kept
                   Defense from focusing on redesigning core business processes, which
                   promised order-of-magnitude improvements. Second, it increased the risk
                   that bad business processes would be perpetuated. Third, it would make
                   future reengineering efforts more difficult by entrenching inefficient and
                   ineffective work processes. As a result, reengineering never took hold in
                   many of DOD’s functional areas. The migration arm of CIM, however, is still
                   active.


                   From fiscal years 1995 through 2000, Defense plans to spend at least
Results in Brief   $18 billion on migration. It has selected 363 migration systems and has
                   targeted 1,938 legacy systems for potential termination. Despite this
                   substantial investment, Defense did not adhere to decision-making and
                   oversight processes it established to ensure that the economical and
                   technical risks associated with migration projects have been mitigated.

                   Defense’s primary corporate-level control mechanism for ensuring that
                   sound management and development practices were followed for each
                   migration investment has not been effective. This control requires that all
                   migration system selections be approved by the Assistant Secretary of
                   Defense for Command, Control, Communications and Intelligence



                   Page 3                                  GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary




(ASD C3I). Despite this requirement, about 67 percent of migration
selections were never submitted for approval, and many of those that were
submitted were approved in the absence of critical technical and
programmatic support. Consequently, Defense does not have assurance
that the migration systems being developed will help achieve its
technology goals and that sound business decisions were made in
selecting the systems.

Because this control mechanism did not support the migration effort, GAO
also assessed whether Defense’s traditional acquisition oversight
processes, which are designed to help ensure that individual major
information system investments are economically and technically sound,
were helping to ensure that the risks associated with migration were
mitigated. However, these processes also did not support the migration
effort because they have not fully ensured that economic analyses for
migration projects are prepared and reviewed and that the systems comply
with technical and data standards. For example, economic analyses for 12
of 43 major migration systems—for which Defense has invested hundreds
of millions of dollars in total—have not yet been submitted for
independent review. Delaying the preparation of an economic analysis to
the later stages of development defeats the purpose of the analysis—to
demonstrate that a proposal to invest in a new system is valid before that
investment is made.

Had there been more rigorous attention to established oversight
procedures and sound business practices, Defense might well have
avoided the migration problems GAO identified in previous reviews, which
unnecessarily cost the Department hundreds of millions of dollars. One
functional area, for example, embarked on and later abandoned a
substantially flawed effort to develop a standard suite of migration
systems for materiel management after spending over $700 million without
strong oversight. In addition, some functional areas did not account for
various categories of significant costs when making their migration
decisions or adequately consider alternatives to developing systems
in-house.

In implementing the migration strategy, Defense did not ensure that it had
adequate departmentwide visibility over status, costs, and progress. As a
result, it could not provide accurate and reliable information, as requested,
on the number and cost of systems designated for migration, and the
numbers of systems terminated and scheduled to be terminated.
Furthermore, the Department has not been able to convincingly
demonstrate whether the migration strategy has been successful or not,


Page 4                                  GAO/AIMD-98-5 Defense Migration Strategy
                            Executive Summary




                            and it has not been able to provide the Congress with accurate and reliable
                            information needed for decision-making purposes.

                            Defense has taken a positive step to turn around its information
                            technology investment process as part of its implementation of the
                            Clinger-Cohen Act of 1996 (Division E of Public Law 104-106). This
                            legislation requires federal agencies to have processes and information in
                            place to help ensure that information technology projects (1) are being
                            implemented at acceptable costs, within reasonable and expected time
                            frames, and (2) are contributing to tangible, observable improvements in
                            mission performance. In June 1997, the Secretary of Defense outlined his
                            expectations for improvements in management processes and information
                            resources related to information technology. These expectations
                            incorporate some of the most important elements of the Clinger-Cohen Act
                            and are an excellent starting point for bringing meaningful change to the
                            current information technology management process. Additionally, the
                            Department’s Task Force on Defense Reform is currently examining the
                            current structure of the Chief Information Officer (CIO) position to ensure
                            that the CIO can devote full attention to reforming information resources
                            management within the Department.

                            In implementing the Clinger-Cohen Act, the Department faces a formidable
                            challenge in successfully implementing real change across an
                            organizational structure that has clearly defined roles and responsibilities
                            for the three individual services to execute national defense policy and
                            objectives. The separation of budget authority, program execution, and
                            functional authority have all contributed to an environment that has
                            fostered stovepipe systems within each service and has made
                            departmentwide oversight difficult. For Clinger-Cohen implementation to
                            make a difference, any new processes or requirements must be
                            successfully accomplished within this environment.



Principal Findings

Management Control          Defense’s primary corporate-level control mechanism for ensuring that
Processes Over Individual   sound management and development practices are followed for each
Migration Efforts Have      migration investment has not been effective. This control requires the
                            functional area manager, known as the Principal Staff Assistant1 (PSA), to
Broken Down
                            1
                             The PSA is the senior executive-level manager who is responsible for the management of a defined
                            function or functions within the DOD.



                            Page 5                                               GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary




submit all systems selected for migration for approval by the Assistant
Secretary of Defense for Command, Control, Communications and
Intelligence. This approval is important because the Assistant Secretary, as
the Department’s Chief Information Officer, needs to ensure that DOD’s
technology-related goals are met.

However, this control mechanism has broken down in two respects. First,
even though all system selections should have been submitted for review
and approval by the Assistant Secretary, 245 of the 363 selections were
never submitted for this oversight. Second, most of the systems that were
submitted for this review were approved in the absence of critical
technical and programmatic support. Specifically, about 19 percent were
approved without supporting technical justification and about 54 percent
were approved without documents showing that the functional area had
evaluated different options for improving a business area, such as
reengineering. Because all of the migration selections submitted for this
oversight have been approved—whether adequately supported with
technical and programmatic justification or not—this control has
essentially become meaningless.

Because the oversight from the ASD C3I that was specifically established
for migration selections has been marginal in trying to ensure that sound
management and development practices are followed for migration
selection, GAO assessed whether the Department’s traditional major
information systems acquisition oversight processes were doing so for 43
of the major migration systems.2 Major systems are Defense’s most
expensive and critical information systems and can cost up to hundreds of
millions of dollars to develop. These review processes are designed to
assess whether projects are affordable and financial and operational risks
have been minimized.

However, these processes did not provide sufficient assurance that the
systems (1) were economically justified or (2) complied with Defense’s
technical and data standards—which are intended to help pave the way
toward an interoperable systems environment. For example, economic
analyses or updates of previously prepared economic analyses for 12 of
the 43 systems had not been submitted for independent review. These 12
systems are under development or modernization and Defense has
invested hundreds of millions of dollars in them. In addition, even though
the Department recognizes that economic analyses play a critical role in

2
 There are 49 major migration systems; however, GAO’s review focused on the 43 major migration
systems under the oversight of the Major Automated Information System Review Council. The
remaining six systems are under the oversight of the Defense Acquisition Board.



Page 6                                              GAO/AIMD-98-5 Defense Migration Strategy
                         Executive Summary




                         assessing whether system development efforts will be cost-effective and
                         beneficial, it has not yet published official guidance to standardize the
                         methodology and analytic techniques for preparing economic analyses.

                         GAO’s previous reviews of migration systems identified a number of
                         problems that could have been prevented had there been better oversight
                         by the Major Automated Information System Review Council (MAISRC) and
                         the ASD C3I. For example, GAO’s review of the materiel management
                         migration strategy showed that a functional area was able to embark and
                         spend over $700 million pursuing a substantially flawed effort—which was
                         later abandoned—without rigorous department-level oversight. In
                         addition, in previous reviews of migration efforts in depot maintenance,
                         transportation, and finance, GAO found that functional areas failed to
                         account for various categories of significant costs when making their
                         migration decisions—including costs related to interfacing migration
                         systems with other systems, project management, and the system
                         selection process. More rigorous oversight and guidance could have
                         helped ensure that these costs were included.

                         GAO  also found a lack of good departmentwide visibility over the migration
                         effort in terms of project costs and progress. For example, DOD has not
                         been tracking key performance issues, such as cost savings resulting from
                         migration or management and staff productivity improvements. Further, it
                         does not have a complete picture of the costs of all migration projects and
                         its scheduling information is inaccurate and unreliable. As a result, the
                         Department has not provided its own department-level decisionmakers
                         with information needed to manage the effort, and it has not provided the
                         Congress with complete and accurate information on migration. Moreover,
                         in the absence of good performance measures for migration, the
                         Department has not been able to show whether the overall strategy has
                         been successful or not.


Reforming DOD’s          Defense recognizes the need for a better information technology
Information Technology   investment environment and has taken steps to implement the
Investment Process       Clinger-Cohen Act of 1996. The Congress passed the act in an effort to put
                         an end to poorly managed and wasteful information technology projects.
Requires Effective       Among other things, it requires agencies to adopt an investment process
Implementation of the    that provides for the continual identification, selection, control, life-cycle
Clinger-Cohen Act        management, and evaluation of information technology projects. As a first
                         step in implementing the act, the Secretary of Defense directed the ASD
                         C3I, as the Department’s Chief Information Officer, to take the lead in




                         Page 7                                   GAO/AIMD-98-5 Defense Migration Strategy
                  Executive Summary




                  implementing an investment process as well as a performance- and
                  results-based management strategy for information technology.

                  This is an important move toward bringing meaningful change to the
                  current decision-making and oversight environment for migration.
                  However, the current structure of the CIO position in the Department
                  seriously limits the CIO’s ability to effectively serve as a bridge between top
                  management, line management, and information management support
                  officials and to identify opportunities to use information technology to
                  enhance performance. At present, the ASD C3I also serves as the
                  Department’s CIO. Asking the same individual to serve in both capacities
                  prevents the CIO from devoting full attention to reforming information
                  resources management within the Department. It also means that the ASD
                  C3I will continue to be responsible for providing oversight for the same
                  systems that he is responsible for selecting, developing, and implementing
                  in his capacity as a PSA. This shortchanges much-needed independent
                  oversight for about 45 percent of the Department’s migration system
                  investment.

                  Finally, effective implementation of the Clinger-Cohen Act will not occur
                  unless Defense’s information and system investment control processes
                  successfully address the challenge posed by the prevailing organizational
                  structure and culture found throughout the Department. This condition
                  has promoted stovepipe systems solutions in each component agency and
                  has made it difficult to implement departmentwide oversight or visibility
                  over information resources. This same condition has contributed to the
                  difficulty that has limited the Department in modernizing business
                  processes and implementing corporate information systems across service
                  and agency lines. This is most evident in the perceived failure of the CIM
                  initiative, which was intended to reengineer business processes
                  throughout the Department. By doing so, the Department expected to save
                  billions by having more efficient, effective business processes running
                  across service and component lines. However, these benefits have yet to
                  be widely achieved after 8 years of effort. Without the Secretary’s strong
                  and continued support for management processes and controls designed
                  to improve information management initiatives, Clinger-Cohen
                  implementation could well suffer similar results.


                  To ensure that continued investment in migration systems provides
Recommendations   measurable improvements in mission-related and administrative
                  processes, GAO is recommending that the Secretary of Defense require



                  Page 8                                   GAO/AIMD-98-5 Defense Migration Strategy
                      Executive Summary




                      Defense components to rank development/modernization systems
                      justifications and complete them on an expedited basis. GAO is further
                      recommending that the Chief Information Officer certify that these
                      justifications include the following:

                  •   An analysis of operational alternatives that clearly demonstrate that
                      continuing with migration is the best solution for improving performance
                      and reducing costs in the functional area.
                  •   An economic analysis showing a return on investment or other mission
                      benefits that justify further investment.
                  •   Documentation showing that the system currently complies with
                      applicable Defense technical standards and uses standard data.

                      GAO’s recommendations are also aimed at (1) correcting weaknesses
                      within the current life-cycle management environment and (2) ensuring
                      the successful reengineering of processes and implementing of corporate
                      information systems for functional areas.


                      The Acting Assistant Secretary of Defense for Command, Control,
Agency Comments       Communications and Intelligence provided written comments on a draft of
                      this report. Defense concurred with five recommendations and partially
                      concurred with two recommendations. Defense did not concur with the
                      remaining five recommendations and expressed concerns about certain
                      aspects of the report. Generally, the Department concurred with GAO’s
                      recommendations that it revise or develop internal policies and
                      procedures to conform to the Clinger-Cohen Act, develop and implement
                      performance measures, and improve the performance of internal tracking
                      systems.

                      The Department did not agree to a proposal in the draft report that it limit
                      further investments in ongoing migration systems to those that meet
                      critical needs until they are independently determined to be economically,
                      functionally, and technically justified. Defense believed that such a
                      limitation would not only adversely affect military readiness, system
                      development, and contract obligations, but also increase system
                      obsolescence. GAO understands DOD’s concerns regarding readiness and
                      contract obligations. That is why the recommendation in the draft report
                      recognized that critical needs still need to be met. GAO’s point is that
                      greater management attention needs to be placed on the migration
                      decision-making process for approving and funding the development and
                      modernization of migration systems to achieve intended benefits and



                      Page 9                                  GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary




minimize unnecessary costs. In order to better clarify this position, GAO
reworded its recommendation to focus on strengthening the controls for
the decision-making process so that these investments can be better
considered in the Department’s budget process.

Defense also did not concur with a proposal in the draft report that it
consider separating the CIO and the ASD C3I positions so that CIO can devote
full attention to departmentwide information resource management issues
and provide independent oversight. It noted, however, that all ASD C3I
functions are being reviewed by the Department’s Task Force on Defense
Reform as part of its review of the Office of the Secretary of Defense’s
organizational structure. GAO withdrew the proposal because DOD is now
considering this matter. Nevertheless, the concern that the current
structure of the CIO position does not allow the CIO to devote full attention
to critical IRM issues—such as computer security, the Year 2000 problem,
and the need to develop and implement an integrated information
technology architecture—remains valid.

Defense also stated that the report negatively portrays Defense’s efforts to
streamline its acquisition processes. GAO disagrees. Acquisition
streamlining allows the Department to greatly simplify its acquisition
processes for information technology purchases. However, acquisition
streamlining was not intended to excuse DOD from exercising management
controls to ensure that those purchases and related system development
efforts made good business sense. Clinger-Cohen implementation should
allow the Department an opportunity to develop more appropriate
controls.

Defense’s comments are discussed in chapter 4 and reprinted in
appendix I.




Page 10                                 GAO/AIMD-98-5 Defense Migration Strategy
Page 11   GAO/AIMD-98-5 Defense Migration Strategy
Contents



Executive Summary                                                                                    2


Chapter 1                                                                                           16
                        Background                                                                  16
Introduction            Migration Has Become the Focal Point for CIM                                18
                        Objectives, Scope, and Methodology                                          24

Chapter 2                                                                                           28
                        Most Systems Not Approved by Defense’s Senior Technology                    29
Management Control        Manager
Processes Over          Acquisition Oversight Process Did Not Ensure System Selections              31
                          Were Economically and Technically Sound
Individual Migration    Better Oversight Could Have Helped to Prevent Problems                      37
Efforts Have Broken       Identified in Previous GAO Reviews
Down                    Defense Lacks Visibility Over Migration Effort                              39

Chapter 3                                                                                           43
                        The Clinger-Cohen Act                                                       43
Reforming DOD’s         DOD Implementation of the Clinger-Cohen Act                                 45
Information             Challenges Confronting DOD in Implementing Clinger-Cohen                    46
Technology
Investment Process
Requires Effective
Implementation of the
Clinger-Cohen Act
Chapter 4                                                                                           50
                        Recommendations                                                             51
Conclusions,            Agency Comments and Our Evaluation                                          54
Recommendations,
and Agency
Comments
Appendixes              Appendix I: Comments From the Department of Defense                         58
                        Appendix II: Description of the Migration System Decision                   77
                          Process
                        Appendix III: Detailed Analysis of Migration Cost Information               81




                        Page 12                                GAO/AIMD-98-5 Defense Migration Strategy
                       Contents




                       Appendix IV: Detailed Analysis of Database Used to Collect                  84
                         Migration Information
                       Appendix V: Major Contributors to This Report                               88

Related GAO Products                                                                               89


Tables                 Table 1.1: Status Reported to GAO on Defense Migration Systems              21
                         Selections and Costs
                       Table 1.2: Major Migration and Interim Systems/Applications                 22
                         Selected for Each Functional Area
                       Table III.1: Costs Reported to the Congress Compared to Costs               82
                         Reported to GAO for Clinical Health, Civilian Personnel, and
                         Transportation
                       Table IV.1: Levels of Incompleteness in DIST in April 1996 and              84
                         February 1997

Figures                Figure 1.1: An Example of Migration in Practice                             18
                       Figure II.1: Migration Selection and Oversight Processes                    77
                       Figure IV.1: Number of Incorrect and Correct Migration and                  85
                         Legacy System Dates for Clinical Health, Civilian Personnel, and
                         Transportation




                       Page 13                                GAO/AIMD-98-5 Defense Migration Strategy
Contents




Abbreviations

ASD C3I    Assistant Secretary of Defense for Command, Control,
                Communications and Intelligence
CIO        Chief Information Officer
CIM        Corporate Information Management initiative
DOD        Department of Defense
DAB        Defense Acquisition Board
DII COE    Defense Information Infrastructure Common Operating
                Environment
DISA       Defense Information Systems Agency
DIST       Defense Integration Support Tools
FY         fiscal year
IRM        information resources management
JTA        Joint Technical Architecture
MAISRC     Major Automated Information System Review Council
OMB        Office of Management and Budget
OSD        Office of the Secretary of Defense
PSA        Principal Staff Assistant
PA&E       Program Analysis and Evaluation
SFFAS      Statements of Federal Financial Accounting Standards
STARS      Standard Accounting and Reporting System
TAFIM      Technical Architecture Framework for Information
                Management


Page 14                              GAO/AIMD-98-5 Defense Migration Strategy
Page 15   GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1

Introduction


                   The Department of Defense (DOD) began its Corporate Information
Background         Management initiative (CIM) in October 1989 to help meet the challenge of
                   effectively managing its diverse operations as it downsized its forces and
                   activities. At the time, the Department believed that the thousands of
                   systems and numerous administrative and mission-related processes1
                   supporting DOD functions were redundant and inefficient and that they
                   should be standardized and made more efficient. To address these
                   problems, Defense planned to

               •   simplify and improve business processes;
               •   centralize responsibility and authority in functional areas, such as finance,
                   personnel, communications, health affairs, logistics, command and
                   control, and intelligence; and
               •   develop an integrated communications and data processing infrastructure
                   based on departmentwide standards.

                   From the outset, DOD recognized that implementing CIM would be difficult
                   because the basic tenet of the initiative—managing and implementing
                   business improvements and building corporate systems along functional
                   lines—represented a major shift in the way Defense traditionally did
                   business. Whereas each military and Defense agency had historically
                   managed its own business functions and information systems, CIM called
                   on senior functional officials, known as Principal Staff Assistants (PSAs),2
                   together with their Defense component counterparts to be responsible for
                   implementing improvements and information systems within the
                   Department’s business functions across service and agency lines.

                   As CIM was implemented, Defense emphasized two ways of achieving
                   process improvements and addressing problems associated with its
                   disparate and stovepiped information technology environment.3 The first
                   was to improve, or reengineer, business processes first and then apply
                   technology to the new processes. The second involved selecting the best

                   1
                    Also known as business processes.
                   2
                    PSAs are the Under Secretaries of Defense, the Director of Defense Research and Engineering, the
                   Assistant Secretaries of Defense, the Director of Operational Test and Evaluation, the General Counsel
                   of DOD, the Inspector General, the Assistants to the Secretary of Defense and the Office of the
                   Secretary of Defense (OSD) Directors or equivalents who report directly to the Secretary or the
                   Deputy Secretary of Defense. In essence, they are top executives at the department level who, together
                   with their Defense component counterparts, are charged with developing corporate systems and
                   reengineering business processes within their respective functional areas.
                   3
                    The CIM initiative included several aspects, such as (1) corporate policy/planning, (2) process and
                   data modeling, (3) process improvement, (4) performance measurement, (5) standard information
                   systems, and (6) computing and communications infrastructure. DOD considered all the aspects of
                   CIM to be important to effectively field information systems that support its mission.



                   Page 16                                               GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1
Introduction




DOD information systems from pools of existing, or legacy, systems that
provide similar automated support services and eventually replacing the
duplicative systems with the best systems. A few years into the CIM effort,
this became known as migration.4 Defense believed that if done properly,
migration could cut costs associated with developing and maintaining
disparate systems supporting the same functions. It also believed that
migration would help to standardize business processes and allow Defense
to achieve savings that could be put to better use in advancing warfighting
capabilities. Figure 1.1 illustrates how migration would work in one
functional area.




4
 In DOD, migration can also involve developing or acquiring a new system, rather than choosing from
existing systems. However, most migration efforts have involved choosing from existing systems.



Page 17                                              GAO/AIMD-98-5 Defense Migration Strategy
                                         Chapter 1
                                         Introduction




Figure 1.1: An Example of Migration in
Practice

                                            For its finance and accounting functions, Defense's goal is to reduce the
                                            number of systems from the 324 that the services and agencies operated
                                            in 1991 to 32 systems.

                                            As of the end of fiscal year 1996, the Defense Finance and Accounting
                                            Service's strategic plan stated it had reduced the number of finance and
                                            accounting systems to 217.

                                            The migration strategy for accounting and finance is to

                                            (1)    identify a single migration system for each of the Department's
                                                   finance and accounting functions, such as civilian pay,

                                            (2)    standardize the procedures and practices used to perform these
                                                   functions, and

                                            (3)    migrate from existing service-unique systems to the migration
                                                   systems.

                                            For some of these functions, such as general fund accounting, Defense
                                            established a two-pronged approach:

                                            (1)    migrate initially to a single interim system or set of systems within
                                                   each service and

                                            (2)    migrate from the service systems to a departmentwide system or
                                                   set of systems.




                                         While Defense began CIM emphasizing the need to improve, or reengineer,
Migration Has                            processes before applying technology, it ended up placing more priority on
Become the Focal                         obtaining quick savings through an accelerated migration strategy, which
Point for CIM                            began in October 1993. This strategy called for selection of migration
                                         systems in 6 months and departmentwide transition to selected systems in
                                         the following 3 years. Defense believed that setting tight time frames for
                                         migration with some potential slippage would allow it to “harvest the




                                         Page 18                                      GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1
Introduction




low-hanging fruit” of potential savings before reengineering. By default,
this meant that the increased emphasis on migration would postpone the
dramatic gains that could be achieved through reengineering.5

The risks involved with postponing reengineering efforts were significant.
Reengineering identifies, analyzes, and redesigns an organization’s core
business processes, aiming to achieve dramatic improvements in critical
areas of performance, such as cost, quality, service, and speed. It focuses
on redesigning the business process as a whole in order to achieve the
greatest possible benefits to an organization and its customers. Migration,
on the other hand, focuses on standardizing existing processes and
information systems. This can yield some benefits, such as reducing the
need to maintain disparate systems that support the same functions, but
seldom yields dramatic improvements. In addition, if the process is
inefficient or outmoded, migration only serves to perpetuate a bad
process. Finally, choosing migration before reengineering may cause
future reengineering efforts to be more difficult by entrenching inefficient
and ineffective work processes.

We raised these concerns both before6 and after7 Defense decided to
embark on the accelerated migration strategy. We believed that the shift in
emphasis toward migration seriously endangered CIM’s chances for
success. For this reason, and because CIM-related technology investments,
which are expected to total billions of dollars each year, are vulnerable to
waste and mismanagement, we designated CIM as a high-risk government
information technology initiative.8 Nevertheless, Defense proceeded to
concentrate on migration even in areas where it had recognized that there
was a significant need to change business processes. One such area was
transportation. We reported in 1996 that even though Defense recognized
that its military transportation processes were fragmented, outdated,

5
 In an October 1993 memorandum, the Deputy Secretary of Defense announced a near-term strategy
that focused on migration and data standardization. The memorandum stated that completion of other
initiatives were not to be prerequisites of implementation of migration systems and data standards.
While some functional areas may have continued process reengineering initiatives, our work indicates
that many others gave priority to the migration strategy.
6
 Defense ADP: Corporate Information Management Must Overcome Major Problems
(GAO/IMTEC-92-77, September 14, 1992).
7
 Defense Management: Stronger Support Needed for Corporate Information Management Initiative to
Succeed (GAO/AIMD/NSIAD-94-101, April 12, 1994) and Defense Management: Impediments
Jeopardize Logistics Corporate Information Management (GAO/NSIAD-95-28, October 21, 1994).
8
 Our high-risk effort, which began in 1990, identifies those federal program areas we consider high risk
because they are especially vulnerable to waste, fraud, abuse, and mismanagement. See High-Risk
Series: Information Management and Technology (GAO/HR-97-9, February 1997) and High-Risk Series:
An Overview (GAO/HR-95-1, February 1995) for a discussion on CIM.



Page 19                                               GAO/AIMD-98-5 Defense Migration Strategy
                          Chapter 1
                          Introduction




                          inefficient, and costly, it focused on technology solutions rather than the
                          need to identify and correct the root causes of its transportation problems.9

                          Because reengineering took a backseat to migration in other business
                          areas as well, CIM has never been able to achieve the level of cost savings
                          and process improvements originally intended. A recent report10
                          conducted by RAND, a consulting organization, notes that “The CIM effort
                          is today widely viewed as a failure in most quarters of the DOD. It has not
                          resulted in either significant process reengineering or visible savings in the
                          hardware and software required to support all the varied information
                          systems in the Defense infrastructure.”

                          Defense is still planning to invest at least $18 billion in its migration effort
                          from fiscal years 1995 through 2000. Because this investment is significant,
                          we were asked for information on the status and progress of migration and
                          whether Defense has significant controls in place to manage and oversee
                          the effort.


Status of the Migration   Defense reported that it has selected 363 systems for migration. Forty-nine
Effort                    of the 36311 migration systems are large-scale, or major, systems12 that are
                          expected to cost at least $13 billion—or about 72 percent of the total
                          $18 billion cost estimate—to develop, deploy, and maintain over fiscal
                          years 1995 through 2000. Defense has identified 1,938 legacy systems for
                          potential termination. As of April 1996, its Defense Integration Support
                          Tools (DIST) database showed that 281 had already been terminated, 886
                          were scheduled for termination by the year 2000, and 771 more were
                          identified for potential elimination after the year 2000.

                          Table 1.1 provides the number of systems selected by each functional area,
                          the number of legacy systems identified for potential termination, and
                          DOD’s estimated costs of developing, deploying, and maintaining the
                          migration systems. We excluded costs for 27 systems for which DOD

                          9
                          Defense Transportation: Migration Systems Selected Without Adequate Analysis (GAO/AIMD-96-81,
                          August 29, 1996).
                          10
                           Strategic Appraisal 1997: Strategy and Defense Planning for the 21st Century, February 1997, RAND’s
                          Project Air Force; edited by Zalmay M.Khalilzad and David A Ochmanek.
                          11
                            The numbers of migration systems selected and legacy systems identified for potential termination
                          are subject to change as PSAs make additional selections and reevaluate previous ones. Also, the PSAs
                          classified 138 of the 363 migration systems as interim systems.
                          12
                            In DOD, major information systems projects are those that (1) have estimated program costs in
                          excess of $30 million in any 1 year, (2) have estimated program costs of over $120 million in total,
                          (3) have total life-cycle costs of over $360 million, or (4) are designated as being of special interest.
                          DOD periodically revises these dollar thresholds.


                          Page 20                                                  GAO/AIMD-98-5 Defense Migration Strategy
                                         Chapter 1
                                         Introduction




                                         collected costs because the systems were classified. And costs for the
                                         remaining 121 of 363 systems were not included in the table because the
                                         Office of the ASD C3I had not collected the costs for these systems. Table
                                         1.2 provides the name, status, and cost for each of the 49 major systems.


Table 1.1: Status Reported to GAO on Defense Migration Systems Selections and Costs
Dollars in millions
                                  Number of legacy
                                 systems identified      Number of migration/            Number of migration/         Total cost reported to
                                      for potential         interim systems                 interim systems            GAO for fiscal years
Functional area                        termination                  selected                 reporting costs             1995 through 2000
Command and Control                                67                            35                             34                      4,493.2
Intelligence                                    304                              69                             46                        572.6a
Mission Support
  Atomic Energy                                     7                             8                              2                         12.7
  Communications                                   17                             2                              2                      3,483.2
  Environmental Security                        286                              11                              6                         33.1
  Financeb                                      267                              52                             31                      1,417.1
  Health                                           62                            55                             49                      2,533.8
  Human Resources                               198                              20                             10                        933.1
  Information Management                           21                             5                              0                              0.0
  Inspector General                                 8                             2                              1                         15.8
  Logistics                                     496                              63                             53                      4,459.1
  Meteorology and
  Oceanography                                     33                            33                              0                              0.0
  Policy                                            3                             7                              7                         73.9
  Procurement                                      67                             1                              1                        282.1
  Science and Technology                        102c                              0c                             0c                               c

Total                                         1,938                            363                            242                    $18,309.7
                                         Note: We did not independently verify information provided in this table.
                                         a
                                          Costs were not included for 27 systems for which DOD collected cost data because the systems
                                         were classified.
                                         b
                                          The information presented for finance is not consistent with information presented in figure 1.1 of
                                         this chapter because the counts of systems occurred at different points in time.
                                         c
                                          The science and technology functional area has not yet selected migration systems and has not
                                         specified which, if any, of its legacy systems may be terminated.

                                         Source: Information obtained from the DIST, DOD’s April 1996 Report to the Congress,
                                         discussions with Defense functional area managers, and the ASD C3I. Cost data were provided as
                                         of December 1996.




                                         Page 21                                               GAO/AIMD-98-5 Defense Migration Strategy
                                                  Chapter 1
                                                  Introduction




Table 1.2: Major Migration and Interim Systems/Applications Selected for Each Functional Area
Dollars in millions
                                                                                                               Total cost data
                                                                                                         reported to GAO for
                                                                                            Current         fiscal years 1995
         System name                                 Acronym                             milestonea             through 2000
Systems being reviewed by the Defense Acquisition Board (six systems):
     Command and Control systems
1        Advanced Field Artillery Tactical Data
         System                                      AFATDS                                       3                   $ 320.6
2        Combat Service Support Control System CSSCS                                              2                     104.9
3        Forward Area Air Defense C2
         Intelligence System                         FAADC2I                                      3                     216.4
4        Maneuver Control System                     MCS                                          3                     253.7
     Intelligence systems
5        All Source Analysis System                  ASAS                                         3                     347.9
                                                                                                                             b
6        Joint Service Imagery Processing System JSIPS                                            2
Systems being reviewed by the Major Automated Information System Review Council (43 systems):
     Command and Control systems
1        Air Force Mission Support System            AFMSS                                        3                     351.9
2        Army Global Command and Control
         System                                      AGCCS                                        3                     254.6
3        Counter Narcotics Command
         Management Control System                   CN/CMS                                       3                     112.2
4        Global Command and Control System           GCCS                                         3                     422.1
5        Naval Aviation Logistics Command
         Management Information System               NALCOMIS                                     3                      93.5
6        Naval Tactical Command System - Afloat NTCS-A                                            3                     314.1
7        Operations Support System                   OSS                                          3                     174.1
8        Shipboard Non-Tactical ADP Program III SNAP III                                          3                     401.4
9        Strategic War Planning System               SWPS                                         3                     343.4
10       Tactical Support Center                     TSC                                          3                     154.3
     Intelligence systems
                                                                                                                             b
11       Exploitation Support System                 ESS                                          0
12       High Performance Computing
                                                                                                                             b
         Modernization Program                       HPCMP                                        2
     Mission support systems
       Communications
13       Defense Information System Network          DISN                                         0                   2,556.2
14       Defense Message System                      DMS                                          2                     927.0
                                                                                                                  (continued)




                                                  Page 22                            GAO/AIMD-98-5 Defense Migration Strategy
                                               Chapter 1
                                               Introduction




Dollars in millions
                                                                                         Total cost data
                                                                                   reported to GAO for
                                                                      Current         fiscal years 1995
        System name                                Acronym         milestonea             through 2000
     Finance
15      Defense Joint Military Pay System          DJMS                   1/2                     114.9
16      Defense Procurement Payment System         DPPS                   1/2                      68.5
17      Standard Accounting And Reporting
        System                                     STARS                  2/3                     263.7
     Health
18      Ambulatory Data System                     ADS                      3                      90.3
19      Composite Health Care System               CHCS                     3                     785.6
20      Composite Health Care System II            CHCS II                  0                       1.4
21      Corporate Executive Information System     CEIS                 0/1/2                      18.2
     Human resources
22      Defense Civilian Personnel Data System     DCPDS                    1                      58.5
23      Defense Commissary Information System DCIS                          2                     120.8
24      Defense Commissary Point of Sales
        System                                     POS                      3                     262.4
25      Joint Recruiting Information Support
        System                                     JRISS                    0                     207.0
26      Navy Standard Integrated Personnel
                                                                                                       b
        System                                     NSIPS                    0
                                                                                                       b
27      Reserve Component Automation System RCAS                          2/3
28      Standard Installation/Division Personnel
                                                                                                       b
        System - 3                                 SIDPERS-3                2
     Logistics
29      Ammunition Management Standard
        System                                     AMSS                     0                      55.7
30      Command and Control Information
        Processing System                          C2IPS                    3                     311.8
31      Defense Automatic Addressing System        DAAS                     3                      12.7
32      Defense Medical Logistics Standard
        Support System                             DMLSS                    2                     129.0
33      Department of Army Movements
        Management System - Redesigned             DAMMS-R                  3                      43.9
34      Depot Maintenance System                   DMS                      0c                    438.6
35      Distribution Standard System               DSS                      3                     270.2
36      Global Transportation Network              GTN                      2                     179.5
37      Joint Computer-Aided Acquisition and
        Logistics System                           JCALS                    2                     789.0
38      Joint Engineering Data Management
        Information and Control System             JEDMICS                  3                     243.7
                                                                                            (continued)


                                               Page 23         GAO/AIMD-98-5 Defense Migration Strategy
                                              Chapter 1
                                              Introduction




Dollars in millions
                                                                                                                                  Total cost data
                                                                                                                            reported to GAO for
                                                                                                            Current            fiscal years 1995
        System name                               Acronym                                                milestonea                through 2000
39      Materiel Management System                MMS                                                               0c                      1,149.2
40      Transportation Coordinators’ Automated
        Information for Movement System II        TC-AIMS II                                                        0                             15.1
41      Transportation Operational Personal
        Property System                           TOPS                                                              3                             95.4
     Meteorology & Oceanography
42      Primary Oceanographic Prediction
                                                                                                                                                     b
        System                                    POPS                                                              3
     Procurement
43      DOD Standard Procurement System           SPS                                                               1                            282.1
        Total                                                                                                                            $13,355.5

                                              Note: We did not verify the information provided in this table. As discussed in chapter 2, these
                                              costs may be inaccurate.
                                              a
                                               There are four milestones in the major system acquisition review process. These are milestone
                                              0—approval to conduct concept studies; milestone 1—approval to begin a new acquisition
                                              program; milestone 2—approval to enter engineering and manufacturing development; and
                                              milestone 3—approval to produce, field, or deploy the system. Multiple milestones may be shown
                                              for some projects because the projects’ subsystems or applications are in different phases of the
                                              acquisition process.
                                              b
                                              The Office of the ASD C3I did not collect costs for this system.
                                              c
                                               Both of these programs were redirected and are no longer targeted to be standard migration
                                              systems.

                                              Source: Information DOD reported to the Congress in April 1996 and to GAO in the course of this
                                              audit.




                                              We were asked (1) for information on the number and cost of systems
Objectives, Scope,                            designated for migration, the number of legacy systems already terminated
and Methodology                               or scheduled for termination, and savings resulting from terminations of
                                              legacy systems and (2) whether Defense’s management control and
                                              oversight processes for migration systems are ensuring that the
                                              investments are economically sound and in compliance with Defense’s
                                              technical and data standards.

                                              To assess the status of Defense’s overall migration strategy and obtain
                                              available information on the number and cost of systems designated for
                                              migration, the number of legacy systems already terminated and scheduled




                                              Page 24                                               GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1
Introduction




for future termination, and savings resulting from terminations of legacy
systems, we analyzed a Defense April 1996 report to the Congress
containing information on the schedule, cost, and status of the migration
systems. Defense prepared this report13 in response to a requirement in
Section 366 of the National Defense Authorization Act for fiscal year 1996.
We also obtained and analyzed a copy of the DIST database as of April 1996
that Defense used to develop the report to the Congress, as well as to
provide its own senior managers information on migration system
selections and legacy system terminations. Additionally, we reviewed—but
did not independently verify—cost information that Defense reported to
Congress as part of the April 1996 Section 366 report. We also interviewed
senior DOD officials to determine if additional cost and performance
information on migration systems existed at the Office of the Secretary of
Defense (OSD) level.

We assessed the reliability of the data presented to the Congress in two
ways. First, we identified missing and conflicting information in the
April 1996 Section 366 report to the Congress and requested that the ASD
C3I staff to clarify conflicting information. Second, we compared the data
reported to the Congress for three functional activities (business
areas)—Transportation, Civilian Personnel, and Clinical Health—against
data provided to us by the functional area managers at these activities. We
then established and analyzed a database containing schedule and
available cost information for the migration and legacy systems and
modified the database to reflect updated schedule, costs, and other
descriptive data Defense provided to us during the course of the audit.

To assess Defense’s management control and oversight processes for
migration systems to determine whether the investments are economically
justified and comply with Defense technical and data standards, we
reviewed the department-level management and oversight processes for
approving the PSAs’ migration system selections and for overseeing major
migration systems’ acquisitions by the Major Automated Information
System Review Council (MAISRC).14 Our review did not focus on the process
by which the PSAs select, develop, and manage their migration systems.



13
  As noted in this chapter, this report did not include costs for 121 information systems.
14
 MAISRC provides acquisition oversight for information systems that (1) are anticipated to cost
$30 million or more a year, (2) have estimated program costs in excess of $120 million, (3) have
estimated life-cycle costs of more than $360 million, or (4) are designated for review by the ASD C3I,
who also chairs the MAISRC. MAISRC reviews such matters as whether a proposed system is being
developed in accordance with Defense policies, procedures, and regulations and whether system
managers took steps to minimize the cost of a new system.



Page 25                                                 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1
Introduction




Nor did we examine acquisition review processes within the individual
functional areas employed for nonmajor system projects.

To analyze Defense’s oversight processes for reviewing and approving the
PSAs’ migration system selections, we reviewed department-level approval
of the PSAs’ selections. We obtained a list of migration systems that PSAs
had selected for their functional areas. We then visited the Office of the
ASD C3I and the Defense Information Systems Agency (DISA) and reviewed
the business case analysis documentation, technical analysis
documentation, and other documentation provided to those offices for
department-level approval of the PSA’s migration and interim system
selections. We also interviewed officials from the Office of the ASD C3I,
DISA, and the offices of selected PSAs regarding the documentation
supporting the selections.

To review Defense’s acquisition oversight processes for major migration
systems, we first identified the major migration systems, as defined by
Defense regulations. We then obtained and analyzed the progress reports
and other documentation provided to MAISRC for those systems. We also
interviewed Defense representatives and officials responsible for
information systems oversight in the Office of the ASD C3I and offices of
DISA, the Defense Acquisition Board (DAB), MAISRC, and Program Analysis
and Evaluation (PA&E). We also interviewed representatives of the program
offices or oversight offices for selected systems for which oversight had
been delegated by MAISRC. Through document reviews and interviews, we
determined whether economic analyses for major migration systems had
been independently reviewed and if so, whether the reviews had identified
problems in the analyses. We also determined whether DOD guidance
existed on preparing economic analyses for information systems. A GAO
economist met with PA&E analysts, reviewed problems that were identified
with economic analyses, and verified the validity of the problems reported.

Additionally, we determined whether MAISRC had information on
alternative analyses performed for major migration system selections.
Lastly, we determined whether MAISRC has sufficient oversight information
to ensure that major migration systems are complying with applicable
technical and data standards that are necessary to achieve an
interoperable systems environment.

We reviewed Defense’s policies and guidance for migration systems to
ensure that information technology is acquired, managed, and used in the
most efficient and effective manner. Our assessment included analyzing



Page 26                                GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1
Introduction




the National Defense Authorization Acts, Committee reports, and
Conference reports for fiscal years 1993 through 1997; Defense Office of
Inspector General reports; our prior studies of CIM and the migration
system strategy, other available evaluations of the CIM and the migration
strategy, and existing legislation affecting these efforts. The major studies
reviewed included studies by RAND and the Defense Science Board.15
Pertinent existing legislation includes the Clinger-Cohen Act, the
Government Performance and Results Act, the Paperwork Reduction Act,
and the Chief Financial Officers Act.

We conducted our review from August 1996 through July 1997 in
accordance with generally accepted government auditing standards. We
requested comments on a draft of this report from the Department of
Defense. The Acting Secretary for Command, Control, Communications
and Intelligence provided us with written comments. These comments are
discussed in chapter 4 and reprinted in appendix I.




15
 Achieving an Innovative Support Structure for 21st Century Military Superiority: Higher Performance
at Lower Costs, 1996 Summer Study, Defense Science Board, Department of Defense. The Defense
Science Board is a Federal Advisory Committee established to provide independent advice to the
Secretary of Defense. Statements, opinions, conclusions, and recommendations in their reports do not
necessarily represent the official position of DOD.



Page 27                                             GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2

Management Control Processes Over
Individual Migration Efforts Have Broken
Down
               Embarking on the migration strategy was an extremely risky endeavor for
               the Department of Defense. First, in developing standard systems within
               functional areas, the Department had to carefully consider complex
               technical issues, such as interfacing migration systems with other systems
               and contending with nonstandard data formats and definitions. For
               Defense, such complexities were compounded by its sheer size and the
               numbers of disparate systems. We believe an even tougher obstacle facing
               Defense, however, was the prevailing culture, which was based on
               decentralized department organizational structure, nonstandard processes
               and procedures. In general, each military service and Defense agency has
               historically managed its own business functions and information
               technology projects, whereas CIM and migration required business
               improvements and information systems to be managed on a
               Departmentwide, or corporate basis.

               Therefore, to ensure the strategy’s success, it was vital for Defense to
               establish an effective decision-making and oversight environment for
               making migration investments. It would need controls and processes that
               ensured inefficient administrative and mission-related work processes
               were modernized before significant technology investments were made to
               support them. It would also need controls that ensured that the migration
               projects themselves were effectively managed as investments so that the
               Department could target resources and attention to priority areas and stop
               those projects that failed to meet their goals. Finally, Defense needed life
               cycle management controls that ensured, on a system-by-system basis,
               that sound management and development practices were followed.

               However, Defense’s primary corporate-level control mechanism for
               ensuring that sound management and development practices were
               followed for each migration investment has not been effective. This
               control requires that all selections to be submitted for approval by the
               Assistant Secretary of Defense for Command, Control, Communications
               and Intelligence (ASD C3I), who is also the Department’s Chief Information
               Officer (CIO). In approving the systems, the Assistant Secretary is to review
               data, technical, and programmatic factors relating to the selection. We
               found, however, that most migration selections never came in for this
               approval and those that did were approved in the absence of critical
               technical and programmatic supporting documents. (A description of the
               migration decision process is provided in appendix II.)

               Because this control was playing a marginal role in ensuring the success of
               migration, we assessed whether Defense’s acquisition oversight



               Page 28                                 GAO/AIMD-98-5 Defense Migration Strategy
                     Chapter 2
                     Management Control Processes Over
                     Individual Migration Efforts Have Broken
                     Down




                     processes—which are designed to augment management oversight for the
                     Department’s most expensive information systems—were helping to
                     ensure that the major migration investments were economically and
                     technically sound. These processes have also broken down where
                     migration is concerned. Had there been more rigorous attention to
                     oversight in both areas, Defense may well have avoided the migration
                     problems we identified in previous reviews that cost the Department
                     hundreds of millions of dollars.

                     In implementing the migration strategy, Defense did not ensure that it had
                     adequate visibility over status, costs, and progress. As a result, it has not
                     been able to (1) demonstrate whether the migration strategy has been
                     successful, (2) provide the Congress with accurate and reliable
                     information needed for oversight purposes, and (3) provide its own
                     decisionmakers at the headquarters level with information needed to
                     oversee the strategy.


                     After selecting a migration system, the Principal Staff Assistant (PSA)1
Most Systems Not     responsible for a functional area is to submit the system for approval by
Approved by          the Assistant Secretary of Defense for Command, Control,
Defense’s Senior     Communications and Intelligence—the Senior Information Management
                     Official and Chief Information Officer in DOD. In his memorandum2 setting
Technology Manager   forth the requirements for selecting and reviewing migration systems, the
                     ASD C3I stated that in approving the selections, he would consider data,
                     technical, and programmatic factors. For example, the ASD C3I review
                     would include such issues as whether the migration systems will lend
                     themselves to data sharing and whether they conform to DOD’s technical
                     standards—which act as a set of “building codes” for constructing systems
                     to ensure they will be compatible with its information infrastructure and




                     1
                      The PSA is the senior executive-level manager who is responsible for the management of a defined
                     function or functions within DOD.
                     2
                      Requirements for selecting and reviewing migration systems under DOD’s accelerated migration
                     strategy are described in two memorandums. The first memorandum was issued by the Deputy
                     Secretary of Defense in October 1993, and the second was issued by the Assistant Secretary of Defense
                     for Command, Control, Communications and Intelligence in November 1993.



                     Page 29                                              GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2
Management Control Processes Over
Individual Migration Efforts Have Broken
Down




technically interoperable with each other.3 This approval is important
because, as the Department’s Chief Information Officer, the Assistant
Secretary needs to ensure that the migration systems help facilitate the
sharing of information across service and agency lines.

However, both the PSAs and the ASD C3I did not adhere to this oversight
process. First, 245 selections, or about 67 percent, were never submitted
for this oversight even though all 363 system selections should have
been.4, 5 Of the 49 major, or large scale, migration systems, only 16 were
submitted for this review. Thus, for the bulk of migration systems, the
Assistant Secretary was unable to ensure that Defense technical standards
would be met and that the best system development practices were being
followed.

Second, most of the systems that were submitted for this review were
approved in the absence of critical technical and programmatic business
case support. For example, 23 of the 118 selections that were submitted
for this oversight, about 19 percent, were approved without documents
stating that the functional area complied or planned to comply with
technical standards. In addition, 64 of the 118 selections, about 54 percent,
were approved without documents showing that the PSA responsible for
the functional area performed some type of functional economic analysis
or other similar business case analysis. This analysis is the means by
which Defense managers are supposed to evaluate different options for
improving business areas, such as outsourcing, reengineering, or
migration. Because all of the migration selections submitted for this

3
 Defense’s technical and data standards are designed to enable systems to easily interoperate and
transfer information. Its standard definitions for data elements are intended to ensure that users of all
Defense systems define the same data in the same way and have a common understanding of their
meaning. Defense has developed or is in the process of defining technical standards in the Technical
Architecture Framework for Information Management (TAFIM), the Joint Technical Architecture
(JTA), and the Defense Information Infrastructure Common Operating Environment (DII COE). The
Defense Information Systems Agency (DISA) is responsible for developing, obtaining from commercial
sources, and maintaining the compilation of Defense Information Infrastructure technical standards,
and it is responsible for maintaining a Defense data dictionary system as a repository of data
requirements and for facilitating the cross-functional coordination and approval of standard formats,
definitions, etc. PSAs, the military services, Defense agencies, and Joint Chiefs of Staff are responsible
for reaching agreement on the standards and approving them as DOD standard data elements. DISA is
then responsible for disseminating the approved standard data elements for use throughout the
Department.
4
 Nine of the systems that were submitted for approval were grandfathered into approval because
development for them was well underway at the time the accelerated migration system strategy began
in 1993.
5
 Forty-two of the systems that were approved by the ASD C3I as migration systems did not follow
DOD’s migration system approval process described in appendix II. Instead, the ASD C3I approved
these 42 Command and Control systems based on reviews by the Military Communications Electronics
Board.



Page 30                                                GAO/AIMD-98-5 Defense Migration Strategy
                          Chapter 2
                          Management Control Processes Over
                          Individual Migration Efforts Have Broken
                          Down




                          oversight have been approved—whether adequately supported by
                          technical and business case justification or not—the ASD C3I oversight has
                          essentially become a meaningless process.


                          Because the ASD C3I oversight control over the migration strategy played a
Acquisition Oversight     marginal role in ensuring that sound management and development
Process Did Not           practices are followed, we assessed whether Defense’s traditional major
Ensure System             information systems acquisition oversight processes were doing so for 43
                          of the major migration systems.6 These review processes are designed to
Selections Were           assess whether projects are affordable and financial and operational risks
Economically and          have been minimized.
Technically Sound         Once under acquisition oversight, systems are normally reviewed and
                          approved at each of four milestones.7 The reviews involve assessing such
                          matters as whether the proposed system is being developed in accordance
                          with Defense policies, procedures, and regulations; whether the systems’
                          program managers took steps to minimize the cost of a new system by
                          ensuring full and open competition; and whether the program managers
                          will effectively use advanced system design and software engineering
                          technology to minimize software and maintenance costs.

                          However, these reviews have not been effective for the migration effort.
                          As discussed in the following section, MAISRC could not provide sufficient
                          assurance that the major migration systems are economically justified and
                          comply with Defense’s technical and data standards.


Ineffective Review and    A principal tool of acquisition oversight is the economic analysis because
Preparation of Economic   it helps to ensure that the system chosen for development is cost-effective.
Analyses                  If done properly, it can enable reviewers to determine whether DOD has

                          6
                           As noted earlier, there are 49 major migration systems. Six of these are under the oversight of the
                          Defense Acquisition Board (DAB). This is Defense’s senior-level forum for advising the Under
                          Secretary of Defense (Acquisition and Technology) on critical decisions concerning major weapon
                          systems and large-scale information systems, principally in the communication, command, control,
                          and intelligence functional areas. Criteria for DAB oversight are (1) estimated expenditures for
                          research, development, test, and evaluation of more than $355 million or (2) estimated procurement
                          costs of more than $2.135 billion. One difference between the DAB and MAISRC oversight processes is
                          that systems reviewed by DAB are required to have independent cost estimates rather than economic
                          analyses. According to the DAB analyst responsible for coordinating oversight of these systems,
                          Program Analysis and Evaluation (PA&E) analysts performed independent cost estimates for all six
                          migration systems under DAB oversight.
                          7
                           The four milestones are: milestone 0, approval to conduct concept studies; milestone 1, approval to
                          begin a new acquisition program; milestone 2, approval to enter engineering and manufacturing
                          development; and milestone 3, approval to produce, field, or deploy the system.



                          Page 31                                               GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2
Management Control Processes Over
Individual Migration Efforts Have Broken
Down




sufficient funds in its budget to pay for the system, that is, whether the
system is affordable. It can also reveal potential conflicts between
available funding and the planned schedule for deployments. To arrive at
these conclusions, the economic analysis establishes baseline life-cycle
costs and benefit estimates for the project and calculates the project’s
return on investment.

The importance of developing complete and accurate economic analyses
is underscored by several governmentwide requirements. For example, the
Office of Management and Budget’s (OMB) Circular A-130, Management of
Federal Information Resources, calls on agencies “to conduct benefit-cost
analyses to support ongoing management oversight processes that
maximize return on investment and minimize financial and operational
risks for investments in major information systems on an agencywide
basis.” Likewise, OMB’s Circular A-11, Part 3, Planning Budgeting and
Acquisition of Fixed Assets, (July 16, 1996), and its Bulletin No. 95-03,
Planning and Budgeting for the Acquisition of Fixed Assets, state that “the
planning for fixed asset acquisitions should be based on a systematic
analysis of expected benefits and costs.”

However, even though economic analyses play a critical role in assessing
whether system development efforts will be cost-effective and beneficial,
Defense has not yet established a set of minimum standards that an
economic analysis must meet to be considered valid. In addition, it has not
published official guidance for preparing an economic analysis. PA&E
developed and published an unofficial economic analysis guide that
recommended, but did not require, standard methods and formats for
preparing an economic analysis for an information system
development/modernization project. According to a PA&E representative,
this unofficial guide is no longer adequate because it does not require the
degree of standardization in methodology and analytic techniques needed
to support a portfolio management approach for managing information
technology investments, as required by the Clinger-Cohen Act of 1996. For
example, the guide does not require that returns on investment for systems
development/modernization projects be calculated in a standard manner
using a standard definition. This lack of standardization results in
economic analyses for different systems that are not comparable enough
for DOD managers to have a good basis for deciding which systems offer
the highest payoffs. The PA&E official stated that in conjunction with DOD’s
implementation of a portfolio management approach, it plans to officially
publish appropriate documents and provide minimum standards and
guidance for the economic analyses process.



Page 32                                    GAO/AIMD-98-5 Defense Migration Strategy
    Chapter 2
    Management Control Processes Over
    Individual Migration Efforts Have Broken
    Down




    Our review also found that DOD decisionmakers do not view economic
    analyses as key tools for deciding whether to invest in an information
    system development or modernization project. As a result, DOD often lacks
    complete and accurate information on system development/modernization
    projects’ estimated costs and benefits at the time decisions are made to
    invest in the projects. Specifically, as the following examples show, we
    found that (1) economic analyses for many systems have not been
    submitted for independent review and (2) a significant portion of those
    that were submitted were inadequately prepared. Thus, many of the
    benefits that could be derived from this tool have not been realized.

•   Twelve of the 43 major migration systems have not yet submitted an
    economic analysis, or an update of a previously prepared economic
    analysis, for independent review.8 These systems were under development
    or modernization and DOD had invested hundreds of millions of dollars in
    them in total. These included 5 systems that were under direct MAISRC
    oversight and 7 systems for which MAISRC delegated oversight
    responsibility to a service or agency. These 12 systems, or components of
    them, are all in the second development phase or beyond.9 Delaying the
    preparation or updating of a previously prepared economic analysis to the
    later stages of development for these 12 systems defeats the purpose of
    the economic analysis, which is to demonstrate that a proposal to invest in
    a new system is valid before that investment is made.
•   Four systems that were under direct MAISRC oversight, were in the first
    development phase—concept exploration—and were, therefore, not yet
    required by DOD’s acquisition regulations to submit an economic analysis
    or an update of a previously prepared economic analysis for independent
    review. Although these four systems were technically in compliance with
    Defense’s acquisition regulations, DOD had already made major
    investments in them without the benefit of knowing whether returns on
    investment are going to be acceptable.
•   Even though Defense has not established standards for the required
    analyses, Program Analysis and Evaluation (PA&E) staff who review the
    economic analyses that are under direct MAISRC oversight told us that 10 of




    8
     Of the 43 major migration systems under MAISRC review, 27 were under direct review by MAISRC
    and 16 were delegated to other Defense oversight organizations, with MAISRC retaining responsibility
    for ensuring adequate oversight.
    9
     The responsibility for independently reviewing economic analyses for the five systems that were
    under direct MAISRC oversight rests with DOD’s Program Analysis and Evaluation staff, while the
    responsibility for reviewing economic analyses for the seven delegated systems rests with the various
    Defense organizations to which MAISRC delegated oversight responsibility.



    Page 33                                               GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2
Management Control Processes Over
Individual Migration Efforts Have Broken
Down




the 19 economic analyses reviewed had problems.10 These problems
included the following:
• Understating or omitting the costs of standardizing data, implementing
  the standard data in the systems, and developing system interfaces.
• Failing to estimate the amount of savings expected for terminating
  duplicative legacy systems.
• Relying on professional judgment to make unsupported assumptions
  rather than making objective analyses to estimate the value of benefits
  and costs. For example, in one case, the economic analysis estimated a
  cost avoidance associated with replacing an old system by assuming
  that both the old and the replacement systems’ software maintenance
  costs could be accurately estimated using two different rates per line of
  code, but the analysis provided no data supporting the validity of either
  rate.

After identifying problems with these 10 economic analyses, PA&E staff
worked with MAISRC analysts and the systems’ program managers to
address the problems. However, DOD continued to develop the systems in
spite of the fact that the investments had not been justified by complete
and accurate economic analyses.

PA&E  analysts told us that there were other problems that impeded their
review and verification of the economic analyses. For example, economic
analyses are often not updated and provided to PA&E for review after major
changes occur in the project, such as significant cost growth or redirection
of the project. A second problem is that economic analyses are often not
supported by analyses of alternatives that weigh the cost and benefits of
various technical options, such as whether to buy commercial off-the-shelf
software or develop a system in-house.11 This analysis would help Defense
decisionmakers make sound decisions on whether the proposed
alternatives offer sufficient military or economic benefits to be worth their
cost, and to determine which alternative is the best approach. It would
also identify alternatives that DOD may want to reconsider at a later time if
the selected approach runs into difficulties.




10
 We did not identify concerns or problems with the economic analyses for the remaining 8 of the 43
major migration systems that were under direct MAISRC oversight for which MAISRC had delegated
oversight responsibility to another DOD organization.
11
 The analysis of technical alternatives would normally precede the economic analysis. In Defense, an
economic analysis weighs the costs, benefits, and risks associated with maintaining the status quo
versus the chosen technical solution.



Page 34                                              GAO/AIMD-98-5 Defense Migration Strategy
                            Chapter 2
                            Management Control Processes Over
                            Individual Migration Efforts Have Broken
                            Down




DOD’s Acquisition           Defense has established several sets of standards that are designed to
Oversight Organizations     ensure that systems developed are compatible with its communications
Lack Assurance That Major   and computing infrastructure and that they are technically interoperable
                            with each other. Some Defense leaders consider systems interoperability
Systems Comply With         and the ability to exchange data across functional lines to be the most
Technical and Data          important consideration in migration system development, transcending
Standards                   economic benefits. These standards include the Technical Architecture
                            Framework for Information Management (TAFIM), Defense Information
                            Infrastructure Common Operating Environment (DII COE), and DOD
                            standard data.12 DOD system acquisition directives call on MAISRC and DAB to
                            ensure that program managers comply with the Department’s policies and
                            procedures and use best practices in developing and modernizing
                            individual information systems. These best practices include building
                            systems and databases that comply with applicable technical standards
                            and use DOD standard data.

                            However, MAISRC and DAB do not have adequate assurance that the major
                            migration systems are complying with applicable technical standards13 and
                            are using standard data. For example, out of 43 major systems under
                            MAISRC oversight, program managers reported to MAISRC that (1) only 19
                            were in compliance with the TAFIM standards or had plans to comply with
                            the TAFIM standards and (2) 9 systems were using DOD standard data or had
                            plans to use standard data.14 We found similar results for compliance with
                            DII COE standards. Specifically, program managers reported that only 25
                            systems were in compliance or had plans to be in compliance with DII COE.
                            These self-reports by program managers are questionable because they are
                            not independently verified.

                            In addition to obtaining information on technical and data standards
                            directly from program managers, MAISRC and DAB also obtain such
                            information from the Defense Information Systems Agency (DISA). DISA
                            supports MAISRC and DAB oversight of major systems by performing
                            technical reviews of system documentation to determine if it indicates that
                            interoperability issues are being addressed or will be addressed as the

                            12
                              See footnote 3 for more information on technical and data standards.
                            13
                              All DOD technical standards may not apply to all systems and initiatives. For example, those
                            standards that apply to application software and data would not apply to a system or initiative that
                            included only computer equipment or hardware. To illustrate, Defense’s High Performance Computing
                            Modernization Program initiative largely involves the purchase of computer equipment and not the
                            development of software. Therefore, application software-related standards may not apply.
                            14
                              In addition, information available to DAB indicated that all six migration systems under DAB
                            program managers have already prepared plans to bring their systems into compliance with DOD
                            technical standards.



                            Page 35                                              GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2
Management Control Processes Over
Individual Migration Efforts Have Broken
Down




system is developed. According to a DISA representative, DISA reviewed
system documentation that had been prepared for 37 of the 43 MAISRC
migration systems and all 6 of the DAB systems and found that the
documentation indicated interoperability issues were planned to be
addressed for each of them.

DISA also provides feedback, advice, and assistance on interoperability and
related issues during integrated product team meetings with MAISRC, DAB,
and system program managers. During these meetings and on other
occasions when it is asked to do so, DISA provides MAISRC and DAB general
information on compliance with standards, such as whether the major
systems’ program managers have contacted DISA and appear to be making
reasonable attempts to bring their systems into compliance with
applicable standards. DISA also assists program managers in developing
strategies and cost estimates for achieving compliance with standards.

However, DISA does not regularly report detailed information to MAISRC and
DAB that would enable these oversight organizations to ensure that each
major system is adequately complying with applicable technical and data
standards.15 For example, DISA does not regularly report to MAISRC and DAB
such detailed information as (1) each system’s current compliance with
applicable standards—that is, each system’s current status relative to the
eight levels that DOD has defined for the DII COE, and each system’s number
and percentage of data elements that have been approved as DOD data
standards in the Defense Data Dictionary System, (2) whether each
program office prepared sound strategies, schedules, and cost estimates
for achieving compliance with applicable standards, (3) whether each
system’s compliance strategy and cost estimate were approved by DISA,
(4) each system’s current schedule and cost status for achieving
compliance compared with its baseline schedule and cost estimate, and
(5) whether each system has been independently certified by DISA’s Joint
Interoperability Testing Command to be in compliance with the technical
and data standards. MAISRC analysts responsible for overseeing the major
migration systems confirmed that they did not know many of the systems’
current status regarding compliance with applicable technical standards
and use of DOD standard data, or whether the program managers had

15
  A DISA representative stated that program managers are supposed to input data into DOD’s DIST
database that could be used by MAISRC and DAB to track each individual system’s progress in
complying with DII COE. For example, the representative noted that program managers are supposed to
input compliance data into DIST, including (1) their systems’ current level of compliance with DII COE,
(2) information on their strategies for achieving compliance, and (3) the expected date and cost to
achieve compliance. However, as discussed in appendix IV, our analysis of the data in the DIST
database showed they are incomplete and inaccurate. Until DOD corrects these deficiencies, MAISRC
and DAB cannot rely on DIST for tracking compliance with standards for individual systems.



Page 36                                               GAO/AIMD-98-5 Defense Migration Strategy
                             Chapter 2
                             Management Control Processes Over
                             Individual Migration Efforts Have Broken
                             Down




                             developed and were following sound strategies for bringing the systems
                             into compliance.

                             The technical and data standards are supposed to help pave the way to an
                             interoperable systems environment. However, without complete and
                             accurate data on individual systems’ compliance with standards, MAISRC
                             and DAB cannot assure Defense’s Chief Information Officer that the
                             Department’s major information systems are complying with the
                             standards. Without this information, as well as standards compliance
                             information from the managers of DOD’s nonmajor systems, the CIO cannot
                             gauge the Department’s progress toward achieving its goal of an
                             interoperable systems environment. Additionally, this information is
                             critical if the CIO is to successfully implement an integrated technical
                             architecture for the Department as required by the Clinger-Cohen Act of
                             1996.


                             The lack of rigorous oversight for the migration strategy has increased the
Better Oversight             risk that Defense will pursue flawed system strategies. In fact, our
Could Have Helped to         previous reviews of migration systems identified a number of problems
Prevent Problems             that could have been prevented had there been better oversight by MAISRC
                             and the ASD C3I. For example, in several reviews, we found that functional
Identified in Previous       areas did not account for various categories of significant costs when
GAO Reviews                  making their migration decisions. These findings are highlighted as
                             follows.

                         •   In reviewing the depot maintenance standard system migration effort, we
                             found that Defense did not address the full costs of developing interfaces
                             needed to allow system components to exchange data with information
                             systems currently used by the services to accomplish their missions. One
                             official estimated that this represented $70 million in costs.16
                         •   In analyzing what it would cost to develop its Standard Accounting and
                             Reporting System (STARS), we reported that Defense neglected to consider
                             internal project management costs and costs to enhance all of the STARS
                             components to bring them into compliance with DOD’s standard general
                             ledger, key accounting requirements, and the standard budget and
                             accounting classification code.17



                             16
                              Defense Management: Selection of Depot Maintenance Standard System Not Based on Sufficient
                             Analyses (GAO/AIMD-95-110, July 13, 1995).
                             17
                              DOD Accounting Systems: Efforts to Improve System for Navy Need Overall Structure
                             (GAO/AIMD-96-99, September 30, 1996).



                             Page 37                                           GAO/AIMD-98-5 Defense Migration Strategy
    Chapter 2
    Management Control Processes Over
    Individual Migration Efforts Have Broken
    Down




•   In reviewing the transportation migration effort, we found that Defense
    did not include all costs associated with its evaluation of in-house systems
    when analyzing costs and savings for its 28 migration systems. These
    included $16 million for its analysis of candidate migration systems and
    $2 million for maintaining migration system hardware. We also found that
    if these costs were included in its systems selection analyses, Defense
    would have found that the overall return on investment would have
    decreased and that it may actually lose money on its investment.18

    In previous reviews, we also found that Defense functional areas did not
    adequately consider other alternatives to developing systems in-house. For
    example, while the transportation area reviewed commercial off-the-shelf
    transportation software projects for some transportation business areas,
    this review was inadequate because it did not (1) analyze the degree to
    which unmodified software could meet unique Defense requirements,
    (2) identify the expected cost to make necessary software modifications,
    (3) determine the time required to make the modifications, and (4) provide
    for a hands-on view of the software in operation. In addition, Defense
    concluded that software packages that could provide some degree of
    transportation functionality would require modifications that were too
    costly. However, Defense could not provide documented analysis to
    support this conclusion. Further, Defense planned on making $13 million
    worth of software modifications to just five of its in-house selections. We
    believe better oversight by the CIO (then referred to as the Senior
    Information Management Official) may well have forced greater
    consideration of commercial packages in the transportation area.

    Finally, our review19 of Defense’s effort to develop a standard suite of
    migration systems for materiel management showed that the Department
    spent hundreds of millions of dollars without achieving the expected
    benefits because it did not adequately anticipate and mitigate risks. From
    1992 to late 1995, Defense spent about $714 million developing standard
    systems with minimal results. During that time, there were dramatic
    changes in the goals and expectations for the program and only one
    application was partially deployed. Because of changes in objectives and
    scheduling and problems in development, prospects for achieving the
    original objective of implementing a standard suite of integrated materiel
    management systems appeared dim.

    18
     Defense Transportation: Migration Systems Selected Without Adequate Analysis (GAO/AIMD-96-81,
    August 29, 1996).
    19
     Defense IRM: Critical Risks Facing New Materiel Management Strategy (GAO/AIMD-96-109,
    September 6, 1996).



    Page 38                                           GAO/AIMD-98-5 Defense Migration Strategy
                         Chapter 2
                         Management Control Processes Over
                         Individual Migration Efforts Have Broken
                         Down




                         In 1996, Defense finally abandoned its strategy to develop a standard suite
                         of materiel management systems because of funding cuts, cost overruns,
                         and schedule delays and embarked on a new strategy that involved
                         individual deployment of nine system applications at selected sites as the
                         applications were developed. However, the decision to drastically change
                         the course of the strategy was initiated without first conducting critical
                         economic and risk assessments that would estimate the costs, benefits,
                         and risks of alternative strategies and having the analyses independently
                         reviewed by MAISRC and PA&E.20 The need for department-level review of
                         analyses supporting this decision was important, given the fact that the
                         new strategy represented a departure from Defense’s goal of eliminating
                         redundant legacy systems and varied business processes.21


                         A successful information technology investment process cannot operate
Defense Lacks            without accurate, reliable, and up-to-date data on project costs, benefits,
Visibility Over          and risks. It is the basis for informed decision-making. However, in
Migration Effort         implementing the migration strategy, Defense neglected to provide
                         visibility over progress and costs. The absence of reliable and accurate
                         performance, cost, and schedule information on the migration effort has
                         been debilitating in several respects. First, it has impaired the
                         Department’s ability to demonstrate whether the migration strategy has
                         been successful. Second, it has kept Defense from providing the Congress22
                         with accurate and reliable information needed for oversight purposes.
                         Third, it has prevented Defense from providing its own senior managers
                         with information needed to oversee the migration effort.


Key Performance Issues   Since it embarked on the migration strategy, Defense has not tracked
Have Not Been Tracked    overall departmentwide savings or validated improvements to operations
                         resulting from the migration strategy. In particular, it has not been
                         systematically tracking such key performance issues as (1) administrative
                         and operational cost savings resulting from elimination of redundant

                         20
                          The materiel management standard system was originally projected to cost $5.3 billion to develop
                         and deploy. The threshold for major information system projects is $360 million in total life-cycle
                         costs.
                         21
                           Under the revised strategy, the military services would be allowed to keep their legacy systems
                         longer than anticipated, and some legacy systems would not be shut down.
                         22
                           For example, Defense reported to the Congress that it had selected 363 migration and interim
                         systems, using DIST as the source of the data. Defense also provided cost data that were collected by
                         the Office of the ASD C3I based on a different list of 245 migration and interim systems. Since the ASD
                         C3I’s staff relied largely on an official list of 245 systems that ASD C3I issued in July 1995 to provide
                         cost information to the Congress, that information was incomplete. In addition, the ASD C3I’s staff did
                         not reconcile the two lists of systems in its report to the Congress.



                         Page 39                                                GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2
Management Control Processes Over
Individual Migration Efforts Have Broken
Down




systems, (2) cost reductions resulting from improved information systems
support to functional areas, (3) management or staff productivity
improvements, and (4) benefits accruing to mission effectiveness that are
attributable to information technology support. Having this type of
information is the only means of ensuring that the billions of dollars being
spent on the migration are producing sufficient returns and achieving
departmentwide progress.23 Without it, Defense cannot justify whether the
migration strategy has been a worthwhile investment or support the need
to continue pursuing migration.

At one point in the migration effort, as directed by the Congress,24 Defense
developed performance measures. However, it did not regularly collect
data on all the measures, and the accuracy of much of the information it
collected was questionable because of the data integrity problems
plaguing the database Defense relies on for migration inventory and
schedule-related information. In 1995, the House Committee on National
Security directed Defense25 to reevaluate its measures. In March of 1997, a
Defense working group proposed a revised set of performance measures
relating to migration systems and other information technology issues
within the Department. However, senior Defense management did not
approve the measures and instead tasked the working group to redo them
to ensure that they are linked to the Department’s strategic information
technology management goals. The working group has not established a
schedule for completing this effort and finalizing the measures.

Having performance measures will help to validate individual reports of
migration successes, such as the migration systems characterized as
successful in Defense’s report to the Congress pursuant to Section 381 of
the National Defense Authorization Act for Fiscal Year 1995. These are the
Standard Procurement System, the Distribution Standard System, the
Defense Civilian Personnel Data System, and the Defense Medical
Logistics Standard Support System.




23
 According to a DOD representative, the Department performed intense reviews of migration systems
during the last three cycles of its planning, programming, and budgeting process. During these reviews,
DOD asked program managers and other senior managers to provide performance information on their
systems. However, the representative stated that the performance information provided was limited
because DOD does not routinely track this information for its information systems.
24
  National Defense Authorization Act for Fiscal Year 1995.
25
  H. Rep. No. 104-131, p. 161.



Page 40                                               GAO/AIMD-98-5 Defense Migration Strategy
                            Chapter 2
                            Management Control Processes Over
                            Individual Migration Efforts Have Broken
                            Down




Cost Information Is         We could not determine the full cost of the migration effort because
Incomplete and May Be       Defense’s migration cost information is incomplete and may be
Significantly Understated   significantly understated. For example, when Defense provided migration
                            cost data to us in December 1996, it could only provide costs for 242 of 363
                            migration systems. We did not include costs in this report for 27 of 242
                            systems for which DOD collected costs because the systems were
                            classified. However, the remaining 121 systems were not included because
                            the Office of the ASD C3I had not collected the costs for these systems.
                            When we obtained cost information directly from three of the functional
                            areas, we found that at least $1.6 billion in costs had been excluded from
                            the approximately $18 billion total. This included about $1.4 billion for
                            clinical health systems, $56 million for civilian personnel systems, and
                            $111 million for transportation systems.

                            We also found that the $18 billion total cost estimate does not account for
                            some very important costs related to developing, deploying, and
                            maintaining migration systems both before and after a project has been
                            initiated. For example, the transportation functional area has an office
                            designated to oversee the development and deployment of its migration
                            systems. But, when accounting for costs for the systems, DOD does not
                            factor in the cost to maintain this oversight responsibility. In addition, our
                            previous reviews of finance and logistics migration efforts have found that
                            DOD did not account for costs for these projects relating to such activities
                            as project management and developing interfaces with other systems. Our
                            detailed analysis of DOD’s migration cost information is provided in
                            appendix III.


Schedule Information Is     We could not accurately determine how many legacy systems have been
Unreliable                  terminated and how many are scheduled for termination because the
                            database Defense uses to track information systems is plagued with data
                            integrity problems.

                            Defense’s own analyses of the database have shown that it cannot readily
                            provide simple descriptive information for many systems. For example, a
                            February 1997 DOD analysis of DIST showed that 55 percent of the migration
                            systems in the database had incomplete information on interfaces with
                            other systems and 77 percent had incomplete data on installations where
                            the systems operated.

                            Our analysis of DIST found that 61 percent the migration system
                            implementation dates and 32 percent of legacy system termination dates



                            Page 41                                    GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2
Management Control Processes Over
Individual Migration Efforts Have Broken
Down




were questionable. In addition, when we compared DIST scheduling
information to information maintained by three functional areas we
reviewed, we found that most of the migration systems implementation
dates and legacy system termination dates in DIST were incorrect. For
example, DIST showed that 92 legacy systems were terminated by
April 1996 in the clinical health, civilian personnel, and transportation
areas, while functional area managers told us that only 43 had actually
been terminated. Also, for the three functional areas, DIST showed that 53
legacy systems were scheduled for future termination while functional
area managers told us that 91 were slated for future termination.

We also found that Defense has not ensured that the data definitions and
formats used in DIST are fully compatible with data maintained in other
Defense information systems that track and report on systems. While DOD
is attempting to address this issue, DISA provided us information showing
that only 66 of the 218 data elements contained in DIST have been approved
as standard data elements in the Defense Data Dictionary System. Without
standard definitions and data formats, data cannot be easily transferred to
DIST from the other systems that may be used by functional area managers
and other decisionmakers.

Defense officials acknowledge that DIST is incomplete and inaccurate, and
the Department has begun efforts to make the database more accurate and
more user friendly. However, Defense officials also stated that they still
used DIST to generate reports to the Congress because it was the only
departmentwide database containing schedule and other descriptive data
on all Defense migration and legacy systems. A detailed analysis of DIST’s
integrity problems is provided in appendix IV.




Page 42                                    GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3

Reforming DOD’s Information Technology
Investment Process Requires Effective
Implementation of the Clinger-Cohen Act
                    Defense has begun implementing Division E of the Clinger-Cohen Act of
                    1996 (Public Law 104-106), which the Congress passed in an effort to put
                    an end to problems associated with the development and implementation
                    of government information systems—such as those evident in the
                    migration effort. Under this legislation, Defense is required to establish
                    additional controls to ensure that more attention is devoted to the
                    selection, control, and evaluation of information technology projects and
                    that the Department’s technology investments are managed from a
                    portfolio perspective.

                    Implementing the Clinger-Cohen Act in DOD can bring meaningful reform
                    to the management of migration and other information technology
                    investments. However, the current structure of the Chief Information
                    Officer (CIO) position in the Department will not permit the CIO to devote
                    full attention to reforming information resources management within DOD.
                    And it will continue to impose responsibilities on the CIO that conflict with
                    the CIO’s obligation to provide independent oversight over the development
                    and implementation of information systems.


                    The Clinger-Cohen Act recognizes that the key to successfully managing
The Clinger-Cohen   information technology projects is ensuring that investment processes
Act                 provide for the continued identification, selection, control, life-cycle
                    management, and evaluation of information technology investments. Best
                    practices on which the act is based have shown that to ensure an
                    investment process is successful, top executives need to periodically
                    assess all major projects—proposed, under development, and
                    operational—then prioritize them and make funding decisions based on
                    factors such as cost, risk, return on investment, and support of
                    mission-related outcomes. Once projects are selected for funding,
                    executives need to monitor them continually, taking quick actions to
                    resolve development problems and mitigate risks. After a project is
                    implemented, executives should evaluate actual versus expected results
                    and revise their investment management process based on lessons
                    learned.

                    As our guide, Assessing Risks and Returns: A Guide for Evaluating Federal
                    Agencies’ Information Technology Investment Decision-making,1 points
                    out, a key to success in this type of management is considering all the
                    major technology investments that are vying for funding at a designated
                    level (departmental, functional area, or service/agency level) as a total

                    1
                     GAO/AIMD-10.1.13 February 1997, Version 1.



                    Page 43                                       GAO/AIMD-98-5 Defense Migration Strategy
    Chapter 3
    Reforming DOD’s Information Technology
    Investment Process Requires Effective
    Implementation of the Clinger-Cohen Act




    package, or portfolio, of possible technology investments. Once this
    perspective is adopted, an organization can focus scarce information
    technology resources on the projects with the greatest impact on mission
    and concentrate management attention on those high-impact projects that
    become troubled projects. It can also establish performance goals and
    stop or replace those systems or initiatives that fail to meet those goals.

    This type of decision-making process can be applied to almost any
    organization, even one that is as highly decentralized as DOD. According to
    our investment guide, separate information technology investment
    decision-making processes can exist at various levels. In DOD, such
    processes could exist at the departmental level and the functional area or
    service/agency level, provided that the Department can identify which
    major projects and initiatives should be managed at each level. Criteria for
    determining projects that should be managed at the various levels may
    include the dollar amount of the investment, the degree of risk associated
    with the project, and whether the system is to be shared across functional
    area lines or service/agency lines.

    The Clinger-Cohen Act contains the following additional requirements that
    are important for DOD to implement in order to address problems evident
    in its migration strategy.

•   Agencies are to determine whether their administrative and
    mission-related business processes should be improved before investing in
    major information systems to support them.
•   The investment process is to provide a means for senior management to
    obtain timely information regarding progress (at established milestones) in
    terms of cost, capability of the system to meet requirements, timeliness,
    and quality. It should also provide for the evaluation of the results of
    information technology investments.
•   Performance measures are to be prescribed for information technology
    used by or to be acquired for the agency.
•   The CIO is to monitor the performance of information technology
    programs; evaluate the performance of those programs on the basis of
    applicable performance measures; and advise the agency head regarding
    whether to continue, modify, or terminate the program or project.
•   The CIO is to be responsible for providing advice and other assistance to
    agency heads and senior managers to ensure that information technology
    is acquired and information resources are managed for the agency in a
    manner that implements the policies and procedures of the act and the
    priorities of the agency head.



    Page 44                                   GAO/AIMD-98-5 Defense Migration Strategy
                           Chapter 3
                           Reforming DOD’s Information Technology
                           Investment Process Requires Effective
                           Implementation of the Clinger-Cohen Act




                       •   The CIO is to develop, maintain, and facilitate the implementation of a
                           sound and integrated information technology architecture for the agency.
                           The architecture is an integrated framework for evolving or maintaining
                           existing information technology and acquiring new information technology
                           to achieve the agency’s strategic and information resources management
                           (IRM) goals.


                           As a first step in implementing the Clinger-Cohen Act, on June 2, 1997, the
DOD Implementation         Secretary of Defense outlined his expectations for improvements in
of the Clinger-Cohen       information technology-related management processes and information
Act                        resources.2 We believe the expectations identified by the Secretary are an
                           excellent starting point for implementing the Clinger-Cohen Act and
                           bringing meaningful change to the current information technology
                           management process. For example, the Secretary has called on the CIO to
                           design and implement a process for maximizing the value and assessing
                           and managing the risks of DOD information technology acquisitions. This
                           process is to

                       •   provide for the selection of information technology investments to be
                           made by the Department, the management of such investments, and the
                           evaluation of the results of such investments;
                       •   be integrated with processes for making budget, financial, and program
                           management decisions;
                       •   include minimum criteria to be applied in considering whether to
                           undertake a particular investment in information systems, including
                           criteria related to the quantitatively expressed projected net, risk-adjusted
                           return on investment, and specific quantitative and qualitative criteria for
                           comparing and prioritizing alternative information system investment
                           projects;
                       •   identify, for each proposed investment, quantifiable measurements for
                           determining the net benefits and risks of the investment; and
                       •   provide the means for senior managers to be able to obtain timely
                           information regarding the progress of an investment in an information
                           system, including the milestones for measuring progress on an
                           independently verifiable basis, in terms of cost, capability of the system to
                           meet specified requirements, timeliness, and quality.

                           In addition, the Secretary has called on the CIO to institutionalize
                           performance-based and results-based management for information

                           2
                            Memorandum from the Secretary of Defense to the Secretaries of the Military Departments, Chairman
                           of the Joint Chiefs of Staff, Under Secretaries of Defense, and others on the Implementation of
                           Subdivision E of the Clinger-Cohen Act of 1996, dated June 2, 1997.



                           Page 45                                            GAO/AIMD-98-5 Defense Migration Strategy
                         Chapter 3
                         Reforming DOD’s Information Technology
                         Investment Process Requires Effective
                         Implementation of the Clinger-Cohen Act




                         technology. In doing so, the CIO is to work with the Chief Financial Officer,
                         the Principal Staff Assistants (PSAs), and the Defense components. The CIO
                         is to establish goals for improving the efficiency and effectiveness of DOD
                         operations and issue instructions to functional areas on performance
                         measurements. The CIO is also to monitor the performance of information
                         technology programs, evaluate the performance of those programs on the
                         basis of applicable performance measurements, and advise the Secretary
                         of Defense regarding whether to continue, modify, or terminate programs
                         or projects. Additionally, the Secretary established a Chief Information
                         Officer Council for DOD to serve as the principal forum for discussing
                         improvements in DOD practices for the management of information
                         technology.

                         In outlining his expectations, the Secretary noted that the act poses
                         questions that should be answered before investing in information
                         technology, including:

                     •   What functions are we performing and are they consistent with the
                         mission?
                     •   If we should be performing particular functions, could they be performed
                         more effectively and at lower cost by the private sector?

                         The Secretary further stated that if a function should indeed be performed
                         by the Department, the law requires that the function be examined and
                         redesigned or reengineered before applying new technology.


                         Proper implementation of the Clinger-Cohen Act would help to address a
Challenges               number of weaknesses that are currently standing in the way of Defense’s
Confronting DOD in       ability to provide a good decision-making and oversight environment for
Implementing             information technology projects. For example, with full implementation of
                         the act, Defense could begin considering information technology
Clinger-Cohen            investments as a total package of possible projects so that it can target
                         resources to those projects having the greatest impact on mission and
                         concentrate management attention on troubled areas. It could also
                         strengthen visibility over project performance, costs, and schedules so
                         that senior managers can begin comparing the results being achieved
                         against projected, costs, benefits, and risks and to identify actual or
                         potential managerial, organizational, or technical problems.

                         Moreover, in implementing the act, Defense has an opportunity to
                         (1) begin enforcing compliance with data and technical standards to



                         Page 46                                   GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3
Reforming DOD’s Information Technology
Investment Process Requires Effective
Implementation of the Clinger-Cohen Act




ensure that DOD’s goals for interoperability and the sharing of information
are met and (2) increase oversight for functional area assessments of
whether to reengineer, migrate, or undertake some other path toward
improvement.

However, the current structure of the CIO position in Defense will not
permit the CIO to effectively serve as a bridge between top management,
line management, and information management support officials and
identify opportunities to use information technology to enhance
performance.

The Clinger-Cohen Act requires that information resources management
be the CIO’s primary responsibility and that the CIO be involved in key
decisions regarding the application of information technology in support
of the agency’s missions. Currently, the Assistant Secretary of Defense for
Command, Control, Communications and Intelligence (ASD C3I) also acts
as the CIO. By asking the CIO to also shoulder a heavy load of programmatic
responsibility, Defense has made it difficult, if not impossible, for the CIO
to devote full attention to IRM issues. These issues go well beyond the
development and modernization of information systems.

For example, Defense needs its CIO to be heavily involved with
implementing a more aggressive and proactive computer security
program. As we reported in May 1996,3 attackers have seized control of
entire Defense systems, many of which support critical functions, such as
weapons systems research and development, logistics, and finance.
Attackers have also stolen, modified, and destroyed data and software. In
addition, Defense needs its CIO to help ensure that Year 20004 corrections
are made to all of DOD’s information systems. If systems are not corrected
on time, the impact on Defense operations could be widespread, costly,
and debilitating to important warfighting and administrative operations.
While DOD has delegated year 2000 responsibility to its components, it still
needs to ensure, at the department level, that sufficient priority and




3
 Information Security: Computer Attacks at Department of Defense Pose Increasing Risks
(GAO/AIMD-96-84, May 22, 1996).
4
 The Year 2000 problem is rooted in the way dates are recorded and computed in automated
information systems. For the past several decades, systems have typically used two digits to represent
the year in order to conserve on electronic data storage and reduce operating costs. With this two-digit
format, however, the year 2000 is indistinguishable from 1900, 2001 from 1901, etc. As a result of this
ambiguity, system or application programs that use dates to perform calculations, comparisons, or
sorting may generate incorrect results when working with years after 1999.



Page 47                                               GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3
Reforming DOD’s Information Technology
Investment Process Requires Effective
Implementation of the Clinger-Cohen Act




resources are being devoted to the problem and that all systems have been
identified and corrected.5

There is also a direct conflict of responsibilites between the oversight and
programmatic obligations associated with the two positions. The ASD C3I
serves as the Principal Staff Assistant for command, control,
communications, and intelligence systems. These systems represent about
45 percent of the migration system investment—about $8.5 billion of the
total $18 billion migration investment. This poses a conflict for both
control mechanisms we assessed. For the first control—the ASD C3I
approval process—the same individual is responsible for both selecting a
system and approving the selection. For the second control—acquisition
oversight—Defense’s Acquisition Executive is also responsible for
developing 14 of the 43 major automated information systems under the
Major Automated Information System Review Council (MAISRC).6

A second challenge confronting DOD in implementing the Clinger-Cohen
Act is the prevailing organizational structure and embedded culture found
throughout the Department. Specifically, the three military services have
clearly defined roles and responsibilities and separate budget authority,
program execution, and functional authority for the enforcement of
national defense policy and objectives. As we have reported7 throughout
the CIM initiative, this environment has promoted stovepipe systems
solutions in each component agency and has made it difficult to
implement departmentwide oversight or visibility over information
resources. This same condition has contributed to the difficulty that has
limited the Department in modernizing business processes and
implementing corporate information systems across service and agency
lines. This is most evident in the perceived failure of the Corporate
Information Management initiative (CIM), which was intended to
reengineer business processes throughout the Department. In doing so,
the Department expected to save billions by having more efficient,
effective business processes running across service and component lines.
However, these benefits have yet to be widely achieved after 8 years of

5
 We have been assessing Year 2000 efforts at DOD and have issued reports on Year 2000 work at
individual DOD components. We also plan to report on DOD’s overall response to the Year 2000 issue.
6
 As the Department’s CIO, the ASD C3I is the Acquisition Executive for DOD’s major information
systems. In this capacity, the staff who oversee major information systems acquisition for MAISRC
report to the ASD C3I.
7
 Defense ADP: Corporate Information Management Initiative Faces Significant Challenges
(GAO/IMTEC-91-35, April 22, 1991); Defense ADP: Corporate Information Management Initiative Must
Overcome Major Problems (GAO/IMTEC-92-77 September 14, 1992); and Defense Management:
Stronger Support Needed for Corporate Information Management Initiative to Succeed
(GAO/AIMD/NSIAD-94-101, April 12, 1994).



Page 48                                              GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3
Reforming DOD’s Information Technology
Investment Process Requires Effective
Implementation of the Clinger-Cohen Act




effort. Without the Secretary’s strong and continued support for
management processes and controls designed to improve information
management initiatives, Clinger-Cohen Act implementation could well
suffer similar results.




Page 49                                   GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4

Conclusions, Recommendations, and
Agency Comments

                 In taking its initial steps to implement the Clinger-Cohen Act, Defense has
                 recognized that it needs a better information technology investment
                 environment. However, implementing the act will not be easy given
                 weaknesses pervading the current decision-making and oversight
                 environment. Examples included the following:

             •   DOD’s  management and oversight processes over information technology
                 projects are not effectively integrated to enable the Department to manage
                 from a portfolio perspective. Considering investments as a total package
                 of possible projects is important because it forces an agency to decide
                 which projects are the most critical to meeting mission needs and thus
                 should receive the most resources and attention. Life-cycle management
                 controls that provide oversight on a system-by-system basis can
                 supplement and enhance this process, but they cannot be a substitute for
                 it.
             •   DOD still has not established information technology performance
                 measures. These need to be in place before a new investment process can
                 begin so that senior managers can begin comparing results being achieved
                 against projected costs, benefits, and risks.
             •   As far as visibility over costs and schedule are concerned, DOD has
                 fragments of a mechanism for collecting and maintaining all project
                 information. However this mechanism—DIST—does not maintain cost
                 information and its accuracy and reliability are questionable. Further, as
                 we have recently reported to the Director of the Defense Information
                 Systems Agency,1 efforts to improve the DIST have been slow-moving.
             •   While the Clinger-Cohen Act requires agencies to conduct
                 post-implementation reviews, DOD’s oversight processes have concentrated
                 on projects prior to their implementation. Thus, in addition to focusing on
                 the pre-implementation stages of system life-cycles, DOD will have to
                 ensure that more attention is given to whether implemented systems are
                 achieving the forecasted benefits and continuing to meet mission needs
                 after they are implemented.
             •   Currently, DOD does not have an effective mechanism in place to ensure
                 that the CIO has complete and accurate information on DOD-wide
                 compliance with technical and data standards. Such a mechanism is


                 1
                  We recently reported in Defense Computers: DOD’s Inventory of Automated Information Systems
                 Needs to Be Improved to Successfully Address Year 2000 Problems (GAO/AIMD-97-112, August 13,
                 1997) that Defense is planning to increase the accuracy of the tool by developing a purging
                 methodology to validate the data in DIST. At the end of January 1997, DIST officials told us that it
                 would take 90 days to determine the methodology. DOD reported that the methodology was finally
                 developed and implemented by the end of August 1997. DOD further reported that it is now checking
                 DIST data on a system-by-system basis and is developing parametric and consistency checks for the
                 data elements to ensure that the required data elements are fully and accurately populated by the
                 users.



                 Page 50                                              GAO/AIMD-98-5 Defense Migration Strategy
                      Chapter 4
                      Conclusions, Recommendations, and
                      Agency Comments




                      necessary to enable the CIO to achieve the Department’s goals for systems
                      interoperability, effectively implement an integrated technical architecture
                      for the entire Department, and ensure the efficient exchange of standard
                      data among systems.

                      Moreover, our review of the migration strategy suggests that a real threat
                      to successful implementation of the Clinger-Cohen Act is the Department’s
                      consistent lack of adherence to sound decision-making and oversight
                      processes. For example, at the oversight level, systems that clearly lacked
                      key pieces of technical, programmatic, and economic justification were
                      allowed to go forward. At the decision-making level, many functional areas
                      did not even bother to submit their systems or analyses that supported
                      their decisions to department-level oversight controls. If they were
                      effectively followed, the life-cycle management controls would have
                      helped ensure that sound business and development practices were
                      followed for the migration systems.

                      Perhaps, the greatest challenge to successful operation, however, will be
                      operating in an organizational environment that has resisted
                      departmentwide oversight and visibility over information resources. For
                      example, the Department’s CIM effort largely failed in meeting its original
                      goals of bringing widespread efficiencies to its business processes. The
                      Department was unable to create meaningful change across organizational
                      lines and management support was insufficient to overcome initial
                      resistance to new ways of doing business. Efforts to implement the
                      Clinger-Cohen Act will require unwavering top-level commitment to
                      overcome both organizational resistance and to institute meaningful
                      controls.


                      To ensure that DOD’s continued investment in migration systems provides
Recommendations       measurable improvements in mission-related and administrative
                      processes, we recommend that the Secretary of Defense require the
                      Defense components to rank development/modernization migration
                      systems justifications and complete them on an expedited basis. Further,
                      the Secretary should require the Chief Information Officer to review these
                      justifications and certify that they include the following:

                  •   A business case of operational alternatives for each functional area that
                      clearly demonstrates that continued development and deployment of the
                      migration system is the best solution for improving performance and
                      reducing costs in the functional area it serves, when compared to other



                      Page 51                                 GAO/AIMD-98-5 Defense Migration Strategy
    Chapter 4
    Conclusions, Recommendations, and
    Agency Comments




    available alternatives. The alternatives analyzed should include
    reengineering the functional area’s processes before making investments
    in information systems and using, when appropriate, the private sector to
    perform major functions now performed by government personnel and
    information systems.
•   An economic analysis showing a return on investment or other
    results-based benefits to the Department that justify further investment in
    the migration system.
•   Current compliance with applicable Defense technical standards and uses
    standard data, or a schedule and plan for bringing the system into
    compliance with these standards.

    Lastly, the Secretary of Defense should require the Chief Information
    Officer to establish routine procedures for reporting on the status of
    reviews of migration system justifications to the Deputy Secretary of
    Defense so the information can be used in the Department’s planning,
    programming, and budgeting system. Any exception to the
    accomplishment of these reviews should be approved by the Deputy
    Secretary of Defense.

    Further, we recommend that the Department’s Chief Information Officer
    revise Defense’s policies, practices, and procedures to institutionalize the
    management of information systems and technology expenditures as
    investments and ensure that these investments provide measurable
    improvements in mission performance. In the course of taking action on
    these matters, we recommend that the Secretary direct that the Chief
    Information Officer, in coordination with the Chief Financial Officer and
    other appropriate Defense officials take the following actions:

•   Ensure that Defense’s strategic information technology planning and
    investment control policies, practices, and procedures include
    requirements that Principal Staff Assistants document that they and the
    key stakeholders for their functional areas (including the services,
    agencies, and major commands) have (1) conducted thorough economic
    and risk analyses of alternative operational approaches (business case
    analyses) for accomplishing the mission of the functional area,
    (2) examined trade-offs among the competing proposals, and
    (3) prioritized the alternative proposals based on mission impact, risk, and
    return.
•   Finalize and issue guidance for developing and using analyses of
    alternatives and economic analyses for information system
    decision-making. Once this guidance is issued, the CIO should require that



    Page 52                                 GAO/AIMD-98-5 Defense Migration Strategy
    Chapter 4
    Conclusions, Recommendations, and
    Agency Comments




    all Defense program managers and Defense managers, as appropriate, be
    trained in using the guidance. Also, the CIO should ensure that the guidance
    defines a standard return on investment definition for Defense information
    systems and require that this definition be used to calculate all returns on
    investment for information systems efforts. Additionally, the CIO should
    require that all major migration and other information systems under
    direct review by the Major Automated Information System Review Council
    (MAISRC), and those delegated by MAISRC to other oversight organizations
    for review, have their alternatives analyses and economic analyses
    independently verified by Defense’s Program Analysis and Evaluation
    (PA&E) office, or another qualified independent review organization, in
    accordance with this guidance before major investments are authorized
    for system development/modernization.
•   Require post-implementation reviews of migration and other information
    systems and ensure that these reviews are designed to compare actual
    systems’ costs, benefits, risks, and returns against the original baseline
    estimates/projections and determine the causes of any differences
    between planned and actual results.
•   Expedite the definition, coordination, testing, and implementation of
    information management performance measures in the Department and
    establish milestones for evaluating progress in implementing these
    performance measures.
•   Modify the DIST system or acquire/develop a new Defense-wide
    management information system or systems for tracking and reporting key
    schedule, progress, and performance information on migration systems
    and other Defense information systems and ensure the system or systems
    contain complete, current, and accurate
    • schedule data necessary to track the progress of each migration
       system’s development/deployment and each legacy system’s
       termination;
    • budgeted and actual cost data on each system for which the Department
       maintains such data (an alternative to putting budget and cost data in
       DIST is to establish the capability to directly interface with other systems
       in the Under Secretary of Defense (Comptroller’s) Office or other
       Defense organizations containing systems’ budget and/or cost data);
    • data necessary to track the progress of each migration system in
       complying with applicable Defense technical and data standards,
       including whether each system has been independently certified to be in
       compliance with applicable technical and data standards;
    • data for tracking progress in accomplishing the mission-based
       performance goals and information management performance goals for




    Page 53                                  GAO/AIMD-98-5 Defense Migration Strategy
                         Chapter 4
                         Conclusions, Recommendations, and
                         Agency Comments




                           the functional areas supported by each system once these data have
                           been identified;
                         • information determined to be needed for oversight by the Defense
                           Acquisition Board (DAB), MAISRC, and PA&E; and
                         • other information determined to be needed for management and
                           oversight by the Defense CIO, the CIO Council, other Defense senior
                           managers, and the Congress.
                     •   Develop and implement management controls and a quality assurance
                         program to ensure the DIST data’s accuracy and completeness since these
                         data are used to track and report to senior Defense managers and the
                         Congress on the overall status of the migration initiative and on other
                         Defense information management initiatives.


                         The Department of Defense provided written comments on a draft of this
Agency Comments          report, which are reprinted in appendix I. The Acting Assistant Secretary
and Our Evaluation       of Defense for Command, Control, Communications and Intelligence
                         concurred with five of our recommendations and partially concurred with
                         two recommendations. Defense did not concur with the remaining five
                         recommendations and expressed concerns about certain aspects of our
                         report. Defense’s concerns are highlighted and discussed below. Appendix
                         I also provides detailed responses to DOD’s views on our recommendations
                         and to other specific comments on our findings.

                         Generally, the Department concurred with our recommendations that the
                         Department revise or develop internal policies and procedures to conform
                         to the Clinger-Cohen Act and to improve the performance of information
                         systems management. Defense also noted that the issues we reported are
                         in the process of being or will be addressed as it implements the
                         Clinger-Cohen Act, the Federal Acquisition Streamlining Act, the
                         Government Performance and Results Act, and the Paperwork Reduction
                         Act.

                         However, it did not agree to a recommendation in our draft that it limit
                         further investments to expenditures that meet mission critical needs until
                         the investments are economically, functionally, and technically justified
                         and such justifications are independently reviewed. In DOD’s view, limiting
                         migration system spending would adversely affect military readiness,
                         system development, and government contract obligations and increase
                         system obsolescence. We understand DOD’s concerns regarding readiness
                         and contract obligations. That is why our recommendation in the draft
                         report recognized that critical needs still need to be met. Our point is that



                         Page 54                                  GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4
Conclusions, Recommendations, and
Agency Comments




greater management attention needs to be placed on the decision-making
process for approving and funding the development and modernization of
migration systems to achieve intended benefits and minimize unnecessary
costs. In order to clarify this position, we modified our recommendation to
focus on strengthening the controls for the migration system
decision-making process so that these investments can be better
considered in the Department’s budget process.

In addition, Defense did not concur with a proposal in the draft report that
it consider separating the CIO and the ASD C3I positions so that the CIO can
devote full attention to departmentwide information resource
management issues and provide independent oversight. It noted, however,
that all ASD C3I functions are being reviewed by the Department’s Task
Force on Defense Reform as part of its review of the Office of the
Secretary of Defense’s organizational structure. We withdrew the proposal
because DOD is now considering this matter. Nevertheless, our concern
that the current structure of the CIO position does not allow the CIO to
devote full attention to critical IRM issues—such as computer security, the
Year 2000 problem, and the need to develop and implement an integrated
information technology architecture—remains valid.

Defense also stated that our report overlooked one of the key cost
reduction aspects of the migration effort—that is, modernizing multiple
systems within each service and Defense agency is more expensive than
simply modernizing one or a few departmentwide migration systems.
While these cost reductions may be possible, Defense has not yet
demonstrated that it has achieved such savings. Further, our review
determined that Defense does not know, or track, cost reductions that
result from its migration efforts. Finally, the achievements that Defense
refers to in its comment letter are mostly the results of process
reengineering, not the migration of automated systems.

Defense also stated that our report negatively portrays the role and
benefits of acquisition streamlining within the Department. Defense
reported that it uses a “best-value” approach to modernizing its systems
that balances the functional and technical capabilities while still
considering cost. We disagree with Defense’s characterization of our
report. We fully support acquisition streamlining efforts in DOD because
they promote economy, efficiency, and effectiveness in the procurement
of property and services. However, these three elements must be carefully
balanced in order to achieve the results intended by the Division D of the
Clinger-Cohen Act of 1996 which covers federal acquisition reform. Such a



Page 55                                 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4
Conclusions, Recommendations, and
Agency Comments




balance is particularly important to achieve prudent decisions for an
investment as substantial as the multibillion dollar migration initiative, and
it is contemplated by the Clinger-Cohen Act of 1996 to properly control
these investments.

In our view, for the migration effort, it appears that expediency was
achieved at the expense of economy and effectiveness. In conducting our
review, we sought to examine whether the Department of Defense was
following its own policy for controlling migration investments. When we
found that these policies were not executed as intended, we turned to the
Department’s acquisition review process, which should have
supplemented oversight for the major migration systems. However, even
here, we found that DOD for the most part, was not providing sufficient
assurance that migration investments were worthwhile. Specifically, we
found that controls that Defense built into this process would not ensure
success.

In addition, the migration strategy is a high-risk endeavor because it
requires Defense to carefully consider complex technical issues that are
compounded by the sheer size of the Department, the number of disparate
systems, and the prevailing culture, which promotes stovepiped systems
solutions and hampers departmentwide oversight or visibility over
information resources. As such, the success of the strategy hinges on
whether practical and sound management practices—such as conducting
economic, risk, and alternative analyses and providing rigorous oversight
over the selection process—are followed. As we note in the report, these
practices are required by executive branch policies; congressional reform
initiatives, including the Clinger-Cohen Act, which took effect August 8,
1996; and/or Defense regulations. Under the Clinger-Cohen Act, for
example, Defense is required to implement a process for selecting
information technology investments using specific criteria for comparing
and ranking alternative information system projects. The best practices on
which the act is based have shown that to ensure the success of an
investment process, top executives need to make funding decisions based
on factors such as cost, risk, return on investment, and support of
mission-related outcomes. Thus, unless Defense promptly embraces the
need to develop and review economic, risk, and alternative analyses, it
stands little chance of successfully implementing the Clinger-Cohen Act.

In its comments, Defense also expressed concern that the scope of our
report was unnecessarily broader than it was when we began our review.
We were originally asked for information on the status and cost of the



Page 56                                  GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4
Conclusions, Recommendations, and
Agency Comments




migration effort. However, when we found that Defense could not provide
accurate and reliable information on migration or convincingly
demonstrate whether the strategy has been successful, our congressional
requester asked that we review whether Defense’s management control
and oversight processes for migration were ensuring that the investments
are economically sound and complied with technical and data standards.
We reviewed the reliability of the acquisition process controls over
migration investments because they are the leading Defense-recognized
controls over major investments. Our congressional requester also asked
that we focus on DOD’s implementation of the Clinger-Cohen Act because it
offered a chance for DOD to bring meaningful reform to the management of
migration and other information technology investments. Many
information technology investments for systems in the Department are, in
fact, migration systems.

Finally, Defense’s comments on our report identify a number of other
disagreements with our findings and conclusions. We reviewed these
comments, incorporated them into our report where appropriate, and
followed up with Defense on additional information it provided after
submitting its response to us.




Page 57                               GAO/AIMD-98-5 Defense Migration Strategy
Appendix I

Comments From the Department of Defense


Note: GAO comments
supplementing those in the
report text appear at the
end of this appendix.




See comment 1.




See comment 2.




                             Page 58   GAO/AIMD-98-5 Defense Migration Strategy
                 Appendix I
                 Comments From the Department of Defense




See comment 3.




See comment 4.




                 Page 59                                   GAO/AIMD-98-5 Defense Migration Strategy
                 Appendix I
                 Comments From the Department of Defense




See comment 5.




See comment 6.




                 Page 60                                   GAO/AIMD-98-5 Defense Migration Strategy
                    Appendix I
                    Comments From the Department of Defense




Now on p. 51.



See comment 7.




Now on pp. 51-52.




                    Page 61                                   GAO/AIMD-98-5 Defense Migration Strategy
                 Appendix I
                 Comments From the Department of Defense




See comment 7.




Now on p. 52.


See comment 7.




Now on p. 52.




Now on p. 52.



See comment 8.




                 Page 62                                   GAO/AIMD-98-5 Defense Migration Strategy
                 Appendix I
                 Comments From the Department of Defense




Now on p. 52.



See comment 9.




                 Page 63                                   GAO/AIMD-98-5 Defense Migration Strategy
                  Appendix I
                  Comments From the Department of Defense




Now on p. 53.




See comment 10.




                  Page 64                                   GAO/AIMD-98-5 Defense Migration Strategy
                Appendix I
                Comments From the Department of Defense




Now on p. 53.




Now on p. 53.




Now on p. 53.




                Page 65                                   GAO/AIMD-98-5 Defense Migration Strategy
                Appendix I
                Comments From the Department of Defense




Now on p. 54.




                Page 66                                   GAO/AIMD-98-5 Defense Migration Strategy
                  Appendix I
                  Comments From the Department of Defense




See comment 11.




                  Page 67                                   GAO/AIMD-98-5 Defense Migration Strategy
                Appendix I
                Comments From the Department of Defense




Now on p. 33.




Now on p. 32.




                Page 68                                   GAO/AIMD-98-5 Defense Migration Strategy
                Appendix I
                Comments From the Department of Defense




Now on p. 33.




                Page 69                                   GAO/AIMD-98-5 Defense Migration Strategy
               Appendix I
               Comments From the Department of Defense




               The following are GAO’s comments on the Department of Defense’s letter
               dated September 12, 1997.


               1. The scope of our work was directed by the Congress. Defense is correct
GAO Comments   that the initial scope was expanded, at the direction of our congressional
               requester because Defense was unable to provide complete and accurate
               information to answer the initial questions. These questions were (1) What
               are the reported costs for migration systems and the legacy systems
               Defense expects to terminate for fiscal years 1995, 1996, and 1997?
               (2) How many and what type of migration systems did Defense designate,
               how many legacy systems did Defense terminate, and how many are
               scheduled for future termination? and (3) What economic studies did
               Defense prepare and use to justify its migration actions? Because Defense
               was unable to provide satisfactory answers to these initial questions, our
               congressional requester directed us to expand our work to evaluate
               Defense’s management controls over decision-making for selecting and
               implementing migration systems. When we found that Defense was not
               following the Department-level management controls that had been
               established for overseeing the selection and implementation of migration
               systems, we looked for other Department-level management controls over
               migration systems. We found that Defense’s oversight of major systems
               acquisitions by MAISRC and DAB was one of the few Department-level
               management controls over migration systems. However, because these
               management controls were limited to the major migration systems, we
               looked for Defense actions that would increase Department-level
               management control over all migration systems. This led us to examine
               Defense’s actions to implement the Clinger-Cohen Act of 1996. Throughout
               our review, we kept our congressional requester and our Defense point of
               contact informed about changes in the scope of our work.

               2. We reviewed the accuracy of all data in the report, especially data that
               Defense questioned in tables, figures, and the text regarding (1) the
               number of migration systems selected by the functional managers and
               approved at the Department level, (2) the costs Defense reported to us for
               migration systems, and (3) our findings related to the preparation and
               independent review of economic analyses for major systems under review
               by MAISRC. We contacted the appropriate Defense officials and discussed
               each instance in which Defense questioned the accuracy of data. We also
               requested that these officials provide us additional documentation
               pertaining to the data in question. Based on these discussions and our
               review of the additional documentation, we made changes where



               Page 70                                   GAO/AIMD-98-5 Defense Migration Strategy
Appendix I
Comments From the Department of Defense




appropriate. In the few cases in which changes were warranted, we
changed the table, figure, or text.

3. We agree that CIM consists of several aspects, and we stated this in a
footnote in the report. However, our work showed that as CIM was
implemented, Defense emphasized two ways of achieving process
improvements and addressing problems associated with its disparate and
stovepiped information technology environment: (1) business process
reengineering and (2) migration.

4. As we state in our report, the $18 billion dollar estimate for developing,
maintaining, and deploying migration systems for fiscal years 1995 through
2000 is based on cost information provided to us by Defense. Defense
provided essentially the same information to the Congress in April 1996.
We agree it is incomplete and inaccurate, and we stated so in our report.

In addition, we agree that migration selections should be based on a
balanced assessment of functional, technical, and cost factors. However,
we found that Defense’s selections placed very little emphasis on costs
and related economic factors. Our work showed that in Defense, the
preparation of an economic analysis is often a paper exercise to produce a
document that decisionmakers do not use when deciding whether to
invest in an information system development or modernization project.
Additionally, we found that in many cases, Defense placed little
importance on having complete and accurate information on system
development/modernization projects’ estimated costs and benefits at the
time decisions were made to invest in the projects.

Further, we agree with Defense’s contention that modernizing multiple
systems would normally be expected to be more expensive than
modernizing a smaller number of migration systems. In fact, our report
does not take issue with the migration strategy’s potential to reduce costs.
Rather, it shows that Defense’s poor management controls over migration
systems may have kept Defense from achieving expected cost reductions
or, worse yet, resulted in wasting hundreds of millions of dollars on failed
efforts such as the materiel management migration initiative. Also,
because Defense has not tracked the cost reductions and other
measurable improvements that may have been achieved through
migration, Defense cannot demonstrate the extent that savings are actually
occurring.




Page 71                                   GAO/AIMD-98-5 Defense Migration Strategy
Appendix I
Comments From the Department of Defense




5. We fully support acquisition streamlining initiatives and encourage
improvements. However, a balance between function, technology, and
cost is as necessary as a balance between acquisition streamlining and
effective management controls to reduce risk. We found that Defense’s
streamlined approach to acquisition oversight and its revisions of its
acquisition policy in March 1996 resulted in inadequate management
control over major systems acquisition projects. Specifically, acquisition
processes did not fully ensure that economic analyses for migration
projects were prepared and reviewed and that systems complied with
technical and data standards. This increased the risk that functional areas
would develop faulty cost estimates and risk assessments and pursue
flawed efforts without rigorous department-level oversight.

While we do not support a one-size-fits-all approach to acquisition
oversight, we believe that Defense’s existing acquisition oversight
processes do not require the degree of standardization in methodology and
analytic techniques to ensure the preparation of the economic analyses
necessary to support a portfolio management approach for information
technology investments, as required by the Clinger-Cohen Act of 1996.
Further, under OMB’s Capital Programming Guide, which was issued in
July 1997, Defense will be required to perform an economic analysis, along
with other extensive planning steps, before submitting a budget proposal
for major information technology projects.

If Defense is to effectively implement the Clinger-Cohen Act of 1996 and
integrate its requirements with those of the Government Performance and
Results Act, the Paperwork Reduction Act, and the Chief Financial
Officers Act, changes will be necessary not only in Defense’s acquisition
oversight processes but also in its other information technology
investment and management control processes. Further, to effectively
implement the requirements specified in these laws, the Chief Financial
Officer, Defense Acquisition Executive, and leaders of the services,
agencies, and major commands must work cooperatively with the CIO.

6. We recognize that DISA reviewed system documentation for many of the
major migration systems to determine if it indicated that interoperability
issues were being addressed or were intended to be addressed as the
system was developed. However, our work showed that DISA does not
regularly report detailed information to MAISRC and DAB that would enable
these oversight organizations to ensure that each major system is
adequately complying with applicable technical and data standards. For




Page 72                                   GAO/AIMD-98-5 Defense Migration Strategy
    Appendix I
    Comments From the Department of Defense




    example, DISA does not regularly report to MAISRC and DAB such detailed
    information as

•   each system’s current compliance with applicable standards—that is, each
    system’s current status relative to the eight levels that Defense has defined
    for the Defense Information Infrastructure Common Operating
    Environment and each system’s number and percentage of data elements
    that have been approved as Defense data standards in the Defense Data
    Dictionary System;
•   whether each program office prepared sound strategies, schedules, and
    cost estimates for achieving compliance with applicable standards;
•   whether each system’s compliance strategy and cost estimate were
    approved by DISA;
•   each system’s current schedule and cost status for achieving compliance
    compared against its baseline schedule and cost estimate; and
•   whether each system has been independently certified by DISA’s Joint
    Interoperability Testing Command to be in compliance with the technical
    and data standards.

    MAISRC analysts responsible for overseeing the major migration systems
    confirmed that they did not know (1) many of the systems’ current status
    regarding compliance with applicable technical standards and use of
    Defense standard data, or (2) whether the program managers had
    developed and were following sound strategies for bringing the systems
    into compliance.

    As noted in our report, Defense’s technical and data standards are
    supposed to help pave the way to an interoperable systems environment.
    However, without complete and accurate data on individual systems’
    compliance with standards, MAISRC and DAB or any other DOD manager
    cannot assure the CIO that Defense’s major information systems are
    complying with the standards. Without this information, as well as
    standards compliance information from the managers of Defense’s
    nonmajor systems, Defense’s CIO cannot gauge the Department’s progress
    toward achieving its goal of an interoperable systems environment.

    7. As noted in the “Agency Comments and Our Evaluation” section of our
    report, we modified our recommendation to focus on strengthening the
    controls for the migration system decision-making process so that these
    investments can be better considered in the Department’s budget process.
    Nevertheless, we remain concerned that if Defense proceeds with its
    decisions to develop/modernize major migration systems without ensuring



    Page 73                                   GAO/AIMD-98-5 Defense Migration Strategy
    Appendix I
    Comments From the Department of Defense




    that they are based on sound investment decision-making practices, it will
    risk some portion of these investments not providing measurable
    improvements in mission performance for the warfighter. Defense will
    also risk the systems’ not operating as intended and not being
    interoperable with other systems.

    We have reported on numerous occasions that poorly planned and
    managed systems development/modernization efforts often fail and must
    be restarted, delaying the delivery of systems that can effectively meet the
    warfighter’s needs. This wastes the limited funds Defense has available for
    providing information technology support to the warfighter and
    unnecessarily diverts funds from other needed investments. For example,
    our review of Defense’s materiel management migration strategy showed
    that Defense spent over $700 million pursuing a substantially flawed effort
    that was later abandoned. We found that Defense’s planning and
    decision-making for this strategy were not sound: Defense did not
    complete an economic analysis until nearly 3 years after the effort began,
    and the analysis was not updated even though Defense dramatically
    changed the course of the systems development. We are concerned that
    Defense may repeat this experience unless it ensures that the remaining
    migration system investment decisions are based on sound planning.

    8. Defense’s response does not clearly specify why it partially concurred
    rather than fully concurred. Nevertheless, it is critical that in revising its
    directives and instructions, DOD require that the PSAs document that they
    and the key stakeholders for their functional areas (including the services,
    agencies, and major commands)

•   have conducted thorough economic and risk analyses of alternative
    operational approaches (business case analyses) for accomplishing the
    mission of the functional area and
•   have examined trade-offs among the completing proposals and ranked the
    alternative proposals based on mission impact, risk, and return.

    Undertaking such reviews is a sound business practice that is consistent
    with the planning and management practices called for in the
    Clinger-Cohen Act of 1996.

    9. We disagree with DOD’s position that DODI 7041.3 provides adequate
    guidance for preparing of an economic analysis for an information system
    development/modernization effort. The existing DODI 7041.3 regulation,
    dated November 7, 1995 , is very general. Defense itself recognized that the



    Page 74                                   GAO/AIMD-98-5 Defense Migration Strategy
Appendix I
Comments From the Department of Defense




regulation was insufficient guidance for preparing an economic analysis
because it developed and published an unofficial economic analysis guide
to supplement the regulation. That unofficial guide recommended, but did
not require, standard methods and formats for preparing an economic
analysis for an information system development/modernization project.
According to a PA&E representative, this unofficial guide is no longer
adequate because it does not require the degree of standardization in
methodology and analytic techniques needed to support a portfolio
management approach for information technology investments, as
required by the Clinger-Cohen Act of 1996. For example, the guide does
not require that returns on investment for systems
development/modernization projects be calculated in a standard manner
using a standard definition. This lack of standardization results in
economic analyses for different systems that are not comparable enough
for DOD managers to have a good basis for deciding which systems offer
the highest payoffs. Defense stated that in conjunction with its
implementation of a portfolio management approach, it plans to officially
publish appropriate documents and provide minimum standards and
guidance for the economic analysis process. However, Defense has not
established a time frame for completing and publishing the guidance. An
expedited time frame is needed so that this effort can be given adequate
priority for resources.

10. Although DOD’s existing acquisition regulations do not require
independent review and verification of alternatives analyses for major
information technology investments, conducting independent reviews of
key decision-making documents is a good business practice consistent
with the type of sound information technology investment planning
promoted by the Clinger-Cohen Act of 1996. An independent review of an
alternatives analysis helps ensure that the methodology used to conduct
the analysis is sound and that the information is accurate and complete
and can be relied on for choosing among competing investment options.
This helps to ensure that the alternative analysis will serve its purpose: to
provide a reliable, detailed, systematic evaluation of alternative solutions
in terms of costs and accompanying benefits so that decisionmakers can
judge whether the proposed alternatives offer sufficient military or
economic benefits to be worth their cost, and to determine which
alternative is the best approach. By not requiring independent review of
alternatives analyses, Defense increases the risk of making poor decisions
to pursue particular options for developing or acquiring migration
systems. It also risks sending a message throughout the Department that it
is acceptable to decided to invest in major information technology



Page 75                                   GAO/AIMD-98-5 Defense Migration Strategy
Appendix I
Comments From the Department of Defense




projects without the benefit of complete and accurate information on the
estimated costs and benefits of alternative investments.

Where economic analyses are concerned, Defense is correct in stating that
its existing acquisition regulations do not require that economic analyses
for delegated systems be independently reviewed by PA&E. Our
recommendation has been revised to enable verification by other qualified
reviewers. However, our recommendation is still valid since our review
revealed that Defense does not ensure that economic analyses for
delegated systems are independently reviewed by any qualified reviewer.

We identified seven major delegated migration systems for which
economic analyses had not been prepared or updated and independently
reviewed even though substantial investments had already been made in
them. The failure to obtain independent review for these systems and to
fully address concerns raised by the reviewer is unacceptable since it
increases the chances that inaccurate, incomplete, and unreliable cost and
benefit information will be used to justify a costly system project.
Assuming this risk is hardly a wise choice given that as a result, Defense
may end up using scarce resources to build information systems that offer
less payback than available alternatives. In addition, we believe that the
failure to obtain an independent review of an economic analysis is certain
to decrease the value of the analysis as a management tool for such
activities as Defense’s formulation of the Program Objective
Memorandum, budget justifications, ranking of investment alternatives,
laying a baseline for tracking performance improvements, coordination of
project development schedules and costs, sensitivity (“what if”) analyses,
and identification of project components that are unfunded.

Furthermore, even though DOD’s acquisition regulations do not require that
PA&E be the economic analysis reviewer for delegated systems, because
PA&E has the necessary expertise, it makes sense that DOD use PA&E to
conduct these reviews. Also, the reviews should be more consistent if PA&E
performs all the economic analysis reviews for major systems. Regardless
of whether PA&E or another qualified independent organization performs
the independent reviews, it is critical that an independent review be made
of the economic analyses for major information technology investments
and that problems identified in the analyses are addressed before
investment decisions are made.

11. See “Agency Comments and Our Evaluation” section in chapter 4.




Page 76                                   GAO/AIMD-98-5 Defense Migration Strategy
Appendix II

Description of the Migration System
Decision Process

                                       As shown in figure II.1, Defense’s migration system selection process is
                                       principally the responsibility of the managers of Defense functional areas
                                       (such as logistics, finance, human resources, and health affairs) with
                                       assistance by the technical staff who report to and advise the ASD C3I.


Figure II.1: Migration Selection and
Oversight Processes
                                        PSA selects a
                                                                 all system selections      ASD C3I approves
                                        migration
                                                                ma                          selections
                                        system                    jor
                                                                      sy                    ASD C3I is to
                                                                        s te
                                       PSA selects a                        m               approve selections       Defense
                                                                              se
                                       migration system for a                    lec        after review of data,    Information
                                                                                    tio
                                       functional area, such                           ns   technical, and           Systems Agency
                                       as logistics, finance,                               programmatic issues.
                                       human resources, and
                                       health affairs.                                                               Program
                                                                                            Defense Acquisition      Analysis and
                                       All system selections                                Board or Major           Evaluation
                                       are to go to the ASD                                 Automated                DISA and PA&E
                                       C3I for approval.                                                             provide advice and
                                                                                            Information System
                                                                                            Review Council           assistance.
                                       Major system
                                                                                            oversees major
                                       selections also go to
                                       one of Defense's                                     system acquisitions
                                       acquisition review                                   These boards are to
                                       boards for acquisition
                                                                                            ensure compliance with
                                       oversight.                                           Defense policies,
                                                                                            procedures, and
                                                                                            regulations.




                                       Source: GAO review of Defense regulations, memorandums, related documents, and interviews
                                       with Defense officials.




                                       The specific steps involved with the process illustrated above are
                                       explained in more detail in the following sections.


Principal Staff Assistant              PSAs are the senior Defense managers for the functional areas. They
(PSA) Selects the                      include Under Secretaries, Assistant Secretaries, the Inspector General,
Migration System                       the Director of Operational Test and Evaluation, the General Counsel, and
                                       a number of other top DOD officials. PSAs are responsible for choosing
                                       migration systems from their respective functional area legacy systems or
                                       for developing new systems.

                                       An important first step involved with deciding whether to migrate and
                                       determining the migration system(s) that will best support the functional



                                       Page 77                                                    GAO/AIMD-98-5 Defense Migration Strategy
Appendix II
Description of the Migration System
Decision Process




area is assessing the functional area’s business processes and deciding if
changes are needed. For example, a functional area may be better off
outsourcing a process rather than attempting to bring improvements
through internal information system modifications. Or, it could decide that
it should fundamentally change its business processes and acquire
different information systems or significantly modify existing systems to
better support the new processes. In DOD, the functional area goes about
making this broad evaluation of paths toward improvement and making a
“business case” for the selected path through the preparation of a
functional economic analysis. In most cases, functional areas have
concluded that migration is a viable strategy for improvement.

When it embarked on its accelerated migration strategy in 1993, Defense
allowed PSAs to tailor the functional economic analysis, as appropriate, to
expedite the migration system selections. At the same time, however, PSAs
and program managers who manage the individual acquisition projects
were told that they should still follow requirements already established for
developing and maintaining automated systems (life-cycle management
processes). For example, managers would still need to perform an
economic analysis for migration systems to establish a baseline life-cycle
cost and benefit estimate for the projects.1 This analysis follows the
decision to migrate has been made and should be based on an analysis of
alternative life-cycle costs, benefits, and risks involved in feasible
approaches for the project.




1
 Defense regulations require that an economic analysis and an alternatives analysis be prepared to
support all major information systems investments that meet the MAISRC review criteria. However,
for those major systems that meet criteria for oversight by the DAB, Defense regulations require an
independent cost estimate, rather than an economic analysis. For nonmajor information system
investments that do not meet either MAISRC or DAB review criteria, OMB and Defense guidance, as
well as best industry and government practices, recommend that an economic analysis and an
evaluation of alternatives be prepared to help decisionmakers estimate the costs and benefits for the
project and choose the most cost-effective approach among competing information technology
investments.



Page 78                                               GAO/AIMD-98-5 Defense Migration Strategy
                               Appendix II
                               Description of the Migration System
                               Decision Process




The ASD C3I Approves the        After the PSAs select the migration systems for their functional activities,
Systems Selected for           the selections are to be sent to the ASD C3I for approval. In deciding
Migration2                     whether to approve the selections, the ASD C3I reviews data, technical, and
                               programmatic factors. These include:

                           •   Whether the migration systems will lend themselves to data-sharing within
                               their design. A key step in the process of ensuring this ability to share data
                               is known as data standardization. This involves reaching agreement on
                               standard definitions that are to be used for the data elements’ names,
                               meanings, and other characteristics so that all users of the data have a
                               common, shared understanding of it.
                           •   Whether the migration systems currently conform to or can evolve to
                               conform with Defense’s technical standards. These technical standards
                               are a set of “building codes” for constructing systems to ensure that they
                               will be compatible with Defense’s communications and computing
                               infrastructure and technically interoperable with each other.3
                           •   Whether the system selections are supported by a full or abbreviated
                               functional economic analysis or other similar justification.

                               The Defense Information Systems Agency (DISA) assists in the review of
                               these factors and passes on any concerns it has to the ASD C3I.


Major Migration Systems        In addition to the management and oversight processes established
Also Undergo DOD’s Major       specifically for the accelerated migration strategy, the major migration
Systems Acquisition            systems are also subject to Defense’s traditional major information
                               systems acquisition oversight processes by either the Defense Acquisition
Review Process                 Board (DAB) or the Major Automated Information System Review Council
                               (MAISRC). DAB generally provides oversight for weapon systems while
                               MAISRC provides oversight for management information systems.




                               2
                                To foster an Enterprise Integration approach and augment the approval process for initial migration
                               selections, Defense established a top management oversight structure to help resolve issues affecting
                               multiple functional areas and to promote cross-functional planning and information-sharing. This
                               structure consisted of two committees: (1) the Enterprise Integration Executive Board, which was
                               chaired by the Deputy Secretary of Defense and included the most senior executive members in the
                               Department, and (2) the Enterprise Integration Corporate Management Council, which served as a
                               supporting committee and was co-chaired by the Principal Deputy Under Secretary of Defense for
                               Acquisition and Technology and the ASD C3I. However, these committees were deactivated in the
                               summer of 1995—about 1 year after they were established—after the Deputy Secretary decided they
                               were not working as intended.
                               3
                                Defense’s technical standards for constructing information systems include, among others, the
                               Technical Architecture Framework for Information Management (TAFIM) and the Defense
                               Information Infrastructure Common Operating Environment (DII COE).



                               Page 79                                               GAO/AIMD-98-5 Defense Migration Strategy
Appendix II
Description of the Migration System
Decision Process




Specifically, DAB reviews those systems that (1) are estimated to eventually
cost over $355 million to research, develop, test, and evaluate, (2) have
estimated procurement costs of over $2.135 billion, or (3) are designated
for DAB review by the Under Secretary of Defense for Acquisition and
Technology, who also chairs DAB. MAISRC reviews those systems that (1) are
anticipated to cost $30 million or more a year, (2) have estimated program
costs in excess of $120 million, (3) have estimated life-cycle cost of more
than $360 million, or (4) are designated for MAISRC review by the ASD C3I,
who also chairs the MAISRC.4

DAB and MAISRC reviews involve assessing such matters as whether the
proposed system is being developed in accordance with Defense policies,
procedures, and regulations; the systems’ program managers took steps to
minimize the cost of a new system by ensuring full and open competition;
and the program managers will effectively use advanced system design
and software engineering technology to minimize software development
and maintenance costs.

When the DAB or MAISRC staffs review a migration system, they obtain
assistance from the Program Analysis and Evaluation (PA&E) staff to
review the system’s economic analysis or cost estimate. This staff is
responsible for reviewing economic analyses for major information
systems and determining whether cost and benefit estimates are accurate
and complete. PA&E analysts are recognized as DOD’s experts on economic
and cost analyses. Organizationally, PA&E reports to the Under Secretary of
Defense (Comptroller). MAISRC requests that DISA provide technical reviews
of the plans and other key documents prepared to support the system’s
development. MAISRC also obtains assistance from other Defense
organizations in reviewing the systems.

Program managers for the systems reviewed by MAISRC provide periodic
progress information to MAISRC regarding such issues as cost, schedule,
return on investment, and compliance with technical and data standards.
Generally, this information is provided in written quarterly reports, which
may be tailored to meet the needs of both the program manager and
MAISRC. In addition, MAISRC conducts in-process reviews of the systems by
establishing “integrated product teams” that provide advice and assistance
to the systems’ program managers on various subjects as needed to
progress through the acquisition process.



4
 These dollar amounts are in fiscal year 1996 constant dollars (dollar amounts after they have been
adjusted for the effect of inflation).



Page 80                                               GAO/AIMD-98-5 Defense Migration Strategy
Appendix III

Detailed Analysis of Migration Cost
Information

                      Having accurate and reliable cost and inventory information is critical for
                      decision-making purposes. It helps Defense managers to determine
                      whether migration system investments will be cost-effective and whether
                      enough funds will be available for developing and deploying the system.
                      However, Defense’s migration cost information is incomplete and may be
                      significantly understated.

                      Collecting cost information for migration systems in itself is a challenging
                      task because Defense does not maintain complete cost information on all
                      of the migration systems in the Defense Integration Support Tools (DIST)
                      database—its primary information system tracking tool. Thus, to report to
                      the Congress and others on migration costs, Defense must turn to
                      numerous sources. For example, in providing information on costs to the
                      Congress in April 1996, Defense asked individual functional area managers
                      to provide total estimated cost data for each of their migration systems for
                      fiscal years 1995 through 2000.

                      Defense officials stated that cost data are not maintained in DIST because
                      functional area managers expressed concern that controls were not
                      adequate to ensure that only authorized personnel were allowed access to
                      the data. They also noted that Defense does not have budget information
                      for all the migration systems because it does not separately budget or
                      collect information for systems with annual cost of less than $2 million. In
                      fiscal year 1996, DOD’s Office of the Under Secretary of Defense
                      (Comptroller) maintained budget information on the systems that cost
                      $2 million or more and is currently working on including all migration
                      systems and other information system costs (for systems with annual
                      costs of more than $2 million) within one database.


                      In December 1996, Defense provided us with cost information on the
Cost Information Is   migration effort, but these data provided costs for only 242 of the 363
Incomplete            migration systems. We did not include costs in this report for 27 of 242
                      systems for which DOD collected costs because the systems were
                      classified. Defense did not provide us costs for the remaining 121 systems
                      because the Office of the ASD C3I had not collected costs for these
                      systems.

                      When we reviewed the cost data reported for the clinical health, civilian
                      personnel, and transportation areas, as part of our review of migration




                      Page 81                                 GAO/AIMD-98-5 Defense Migration Strategy
                                         Appendix III
                                         Detailed Analysis of Migration Cost
                                         Information




                                         systems’ cost reported to the Congress in April 1996,1 we identified three
                                         reasons why the functional areas did not report costs for some systems.
                                         First, for some of the systems, these costs were not available at the time
                                         the report was made to the Congress. Second, for other systems, the
                                         functional area managers were not aware that the report was to include
                                         interim migration systems. Third, for some systems, functional area
                                         approval as a migration system may not have been completed at the time
                                         cost data were gathered for the report to Congress. As table III.1
                                         illustrates, the costs not reported represented a substantial portion of the
                                         total being spent on migration for the three functional areas.

Table III.1: Costs Reported to the
Congress Compared to Costs               Dollars in millions
Reported to GAO for Clinical Health,                                                   Total costs                                Difference
Civilian Personnel, and Transportation                                                   reported          Total costs          (amount not
                                                                                             to the          reported        reported to the
                                         Functional area                                Congress              to GAO              Congress)
                                         Clinical
                                         Health                                           $1,014.0             $2,451.3              $1,437.3
                                         Civilian
                                         Personnel                                             67.4                123.9                  56.5
                                         Transportation                                     1,005.2             1,116.6                  111.4
                                         Total                                            $2,086.6             $3,691.8              $1,605.2
                                         Source: GAO’s analysis of information obtained from the April 1996 Report to the Congress,
                                         Defense functional area managers, and the ASD C3I.




                                         We also found that Defense does not account for some very important
Significant Costs Not                    costs related to developing, deploying, and maintaining migration systems
Tracked                                  both before and after a project has been initiated. It is extremely important
                                         to have accurate estimates up front on direct, indirect, and support costs
                                         associated with a system development or acquisition project and to be
                                         able to track these costs throughout the life-cycle. At the beginning of a
                                         system development project, a complete cost picture provides an
                                         organization with a realistic picture of what it will take to develop, deploy,
                                         and maintain a system over its life-cycle. It also enables an organization to
                                         make sound trade-off decisions among competing investments.
                                         Continually tracking direct, indirect, and support costs helps managers
                                         account for their past activities, manage current operations, and assess
                                         progress toward planned objectives.


                                         1
                                         As stated in chapter 1, this report was made pursuant to a requirement in Section 366 of the National
                                         Defense Authorization Act for Fiscal Year 1996.



                                         Page 82                                               GAO/AIMD-98-5 Defense Migration Strategy
Appendix III
Detailed Analysis of Migration Cost
Information




As noted in this report, when we conducted detailed reviews of selected
finance and logistics migration efforts, we found that DOD excluded
significant categories of costs from its initial life-cycle cost estimates.
Furthermore, our work in the clinical health, civilian personnel, and
transportation functional areas showed that DOD also does not capture and
report the total actual costs for its systems as they are developed,
deployed, and maintained. Instead, DOD decisionmakers employ cost
accounting systems that omit relevant project costs, such as those
associated with project management and oversight. The transportation
functional area, for example, has an office designated to oversee the
development and deployment of its migration systems. But, when
accounting for costs for the systems, DOD does not factor in the cost to
maintain this oversight responsibility. As a result, DOD cannot reliably
report migration system projects’ actual costs and compare them against
established baselines, and it cannot reliably use information relating to
projects’ actual costs to improve future cost estimating efforts.

Beginning October 1, 1996, a new federal accounting standard requires the
Department of Defense and other federal agencies to begin accounting for
all identifiable direct, indirect, and support costs of its outputs. This would
provide a mechanism for developing total costs of capital items, such as
information systems. This requirement is defined in the Statement of
Federal Financial Accounting Standards, (SFFAS) No. 4, Managerial Cost
Accounting Concepts and Standards for the Federal Government.




Page 83                                  GAO/AIMD-98-5 Defense Migration Strategy
Appendix IV

Detailed Analysis of Database Used to
Collect Migration Information

                                              There are a number of data integrity problems associated with the DIST
                                              database, which is used to keep information on Defense’s migration
                                              systems. These problems directly affect Defense’s ability to accurately
                                              determine how many legacy systems were terminated and how many are
                                              scheduled for termination. Defense officials acknowledge that the DIST
                                              database is incomplete and inaccurate, but stated they used it to generate
                                              reports to the Congress because it was the only departmentwide database
                                              containing schedule and other descriptive data on all of Defense’s
                                              migration and legacy systems.

                                              When we obtained a copy of the DIST database for analysis in April 1996,
                                              Defense’s own analysis of the DIST database confirmed that it had
                                              incomplete data in several categories of key information. A more recent
                                              Defense analysis of migration systems showed that DIST continues to have
                                              high levels of incomplete data. Table IV.1 illustrates how information was
                                              incomplete in several categories of information.

Table IV.1: Levels of Incompleteness in
DIST in April 1996 and February 1997                                                                             Percent of systems
                                                                                                                 with incomplete data
                                                                                                                      April   February
                                              Data category                                                           1996        1997
                                              Interfaces with other systems                                             88           55
                                              Computer installations where the system operated                          79           77
                                              Computer hardware on which the system operated                            75           68
                                              System software                                                           66           61
                                              Organization responsible for the system                                   32           26
                                              Source: GAO analysis of DIST Data Status Working Papers.



                                              Our analysis supports Defense’s concerns as still relevant and identified
                                              the following additional data integrity problems.

                                          •   Large numbers of migration system implementation dates (220, or
                                              61 percent) of the migration systems’ implementation dates and legacy
                                              system termination dates (629, or 32 percent) shown in DIST were in the
                                              year 2010. According to system documentation and our discussions with
                                              officials responsible to operate DIST, the presence of the year 2010 for
                                              these dates indicates the system owners responsible for the migration
                                              systems did not provide a date, resulting in the DIST system automatically
                                              inserting the default date of 2010.




                                              Page 84                                           GAO/AIMD-98-5 Defense Migration Strategy
                                           Appendix IV
                                           Detailed Analysis of Database Used to
                                           Collect Migration Information




                                       •   As figure IV.1 illustrates, when we compared the DIST schedule data for
                                           clinical health, civilian personnel, and transportation to the actual
                                           schedule maintained for these areas, we found that incorrect data for
                                           migration implementation and legacy system termination outweighed
                                           correct data.


Figure IV.1: Number of Incorrect and
Correct Migration and Legacy System
                                           120
Dates for Clinical Health, Civilian
Personnel, and Transportation
                                                                                                                       101
                                           100



                                            80



                                            60



                                            40                                                                33
                                                                           30
                                                                 23

                                            20



                                             0
                                                               Migration system                               Legacy system
                                                               implementation                                 termination dates
                                                               dates


                                                                            Correct               Incorrect




                                           Sources: GAO’s analysis of information obtained from the DIST, April 1996 Report to the
                                           Congress, Defense functional area managers, and ASD C3I.




                                           Functional area managers in these three areas identified additional
                                           inaccuracies. For example, managers in the transportation area stated that




                                           Page 85                                             GAO/AIMD-98-5 Defense Migration Strategy
Appendix IV
Detailed Analysis of Database Used to
Collect Migration Information




the area had implemented six migration systems rather than none as
indicated in DIST. For the three functional areas, DIST showed that 92 legacy
systems were terminated by April 1996, while functional managers told us
that only 43 had actually been terminated. Also, for the three functional
areas, DIST showed that 53 legacy systems were scheduled for future
termination, but functional managers told us that 91 were slated for future
termination.

We also learned that DIST has other problems that hamper its use. One
problem is that Defense has not ensured that the data definitions and
formats used in DIST are fully compatible with data maintained in other
Defense information systems that track and report on systems. While DOD
is attempting to address this issue, DISA provided us information showing
that only 66 of the 218 data elements contained in the DIST have been
approved as standard data elements in the Defense Data Dictionary
System. Without standard definitions and data formats, data cannot be
easily transferred to DIST from other systems that may be used by PSAs,
program managers, and other decisionmakers.

In addition, Defense assigns unique numbers to each system in DIST, but
these numbers are not universally used throughout the Department. The
different number schemes make it more difficult to facilitate the exchange
of information between these systems. Use of the same identifying number
for each system throughout DOD is also important because Defense often
changes the names of its information systems over their life. Without
consistent identification numbers, tracking an information system in the
face of such name changes is more arduous.

Defense recognizes that it needs better information to deal with migration
and other technology issues for which DIST is relied on, such as the Year
2000 problem.1 In November 1996, the Under Secretary for Defense
(Comptroller) and the ASD C3I issued a joint memorandum to senior
Defense managers stating that DIST was the backbone tool for managing
the Department’s information technology investment and that to be
effective, it must contain accurate data on all the Department’s
information systems. The memorandum also stated that registration of
information systems in the DIST was mandatory. However, the minimum
set of data that is required under this joint memorandum focused on
information needed to address Year 2000 issues and did not include


1
 The Year 2000 problem refers to the possibility that many computer systems will fail on New Year’s
morning in the year 2000 because they were not designed to accommodate the change of date to the
new millennium.



Page 86                                              GAO/AIMD-98-5 Defense Migration Strategy
Appendix IV
Detailed Analysis of Database Used to
Collect Migration Information




critical information on migration implementation dates, cost, savings, and
performance measures.1




1
 See chapter 4, footnote 1, for recent Defense developments regarding DIST.



Page 87                                              GAO/AIMD-98-5 Defense Migration Strategy
Appendix V

Major Contributors to This Report


                       Cristina T. Chaplain, Communications Analyst
Accounting and         Danny R. Latta, Assistant Director, Policies and Issues
Information            Harold J. Brumm, Jr., Economist
Management Division,
Washington, D.C.
                       George L. Jones, Evaluator-in-Charge
Kansas City Field      David R. Solenberger, Senior Evaluator
Office                 Karen S. Sifford, Staff Evaluator
                       John G. Snavely, Staff Evaluator
                       Michael W. Buell, Staff Evaluator
                       Karl G. Neybert, Staff Evaluator




                       Page 88                                  GAO/AIMD-98-5 Defense Migration Strategy
Related GAO Products


              High-Risk Series: Information Management and Technology (GAO/HR-97-9,
              February 1997).

              Air Traffic Control: Improved Cost Information Needed to Make Billion
              Dollar Modernization Investment Decisions (GAO/AIMD-97-20, January 22,
              1997).

              Defense Communications: Performance Measures Needed to Ensure DISN
              Program Success (GAO/AIMD-97-9, November 27, 1996).

              Defense IRM: Strategy Needed for Logistics Information Technology
              Improvement Efforts (GAO/AIMD-97-6, November 14, 1996).

              Information Technology Investment: Agencies Can Improve Performance,
              Reduce Costs, and Minimize Risks (GAO/AIMD-96-64, September 30, 1996).

              DOD Accounting Systems: Efforts to Improve System for Navy Need Overall
              Structure (GAO/AIMD-96-99, September 30, 1996).

              Defense IRM: Critical Risks Facing New Materiel Management Strategy
              (GAO/AIMD-96-109, September 6, 1996).

              Defense Transportation: Migration Systems Selected Without Adequate
              Analysis (GAO/AIMD-96-81, August 29, 1996).

              Information Management Reform: Effective Implementation Is Essential
              for Improving Federal Performance (GAO/T-AIMD-96-132, July 17, 1996).

              Acquisition Reform: Regulatory Implementation of the Federal Acquisition
              Streamlining Act of 1994 (GAO/NSIAD-96-139, June 28, 1996).

              Medical ADP Systems: Defense Achieves Worldwide Deployment of
              Composite Health Care System (GAO/AIMD-96-39, April 5, 1996).

              Best Management Practices: Reengineering the Air Force’s Logistics
              System Can Yield Substantial Savings (GAO/NSIAD-96-5, February 21, 1996).

              Inventory Management: DOD Can Build on Progress in Using Best Practices
              to Achieve Substantial Savings (GAO/NSIAD-95-142, August 4, 1995).

              Information Technology Investment: A Governmentwide Overview
              (GAO/AIMD-95-208, July 31, 1995).



              Page 89                                GAO/AIMD-98-5 Defense Migration Strategy
Related GAO Products




Defense Management: Selection of Depot Maintenance Standard System
Not Based on Sufficient Analyses (GAO/AIMD-95-110, July 13, 1995).

Defense Communications: Management Problems Jeopardize DISN
Implementation (GAO/AIMD-95-136, July 13, 1995).

Best Practices Methodology: A New Approach for Improving Government
Operations (GAO/NSIAD-95-154, May 1995).

Government Reform: Goal-Setting and Performance (GAO/AIMD/GGD-95-130R,
March 27, 1995).

Defense Infrastructure: Enhancing Performance Through Better Business
Practices (GAO/T-NSIAD/AIMD-95-126, March 23, 1995).

Paperwork Reduction Act: Reauthorization Can Strengthen Government’s
Management of Information and Technology (GAO/T-AIMD/GGD-95-80,
February 7, 1995).

Government Reform: Using Reengineering and Technology to Improve
Government Performance (GAO/T-OCG-95-2, February 2, 1995).

High-Risk Series: An Overview (GAO/HR-95-1, February 1995).

Reengineering: Opportunities to Improve (GAO/AIMD-95-67R, January 6, 1995).

Reengineering Organizations: Results of a GAO Symposium (GAO/NSIAD-95-34,
December 13, 1994).

Defense Management: Impediments Jeopardize Logistics Corporate
Information Management (GAO/NSIAD-95-28, October 21, 1994).

DOD’s CALS   Initiative (GAO/AIMD-94-197R, September 30, 1994).

Defense Management Initiatives: Limited Progress in Implementing
Management Initiatives (GAO/T-AIMD-94-105, April 14, 1994).

Financial Management: Financial Control and System Weaknesses
Continue to Waste DOD Resources and Undermine Operations
(GAO/T-AIMD/NSIAD-94-154, April 12, 1994).




Page 90                                    GAO/AIMD-98-5 Defense Migration Strategy
Related GAO Products




Defense Management: Stronger Support Needed for Corporate
Information Management Initiative to Succeed (GAO/AIMD/NSIAD-94-101,
April 12, 1994).

Defense IRM: Management Commitment Needed to Achieve Defense Data
Administration Goals (GAO/AIMD-94-14, January 21, 1994).

Defense IRM: Business Strategy Needed for Electronic Data Interchange
Program (GAO/AIMD-94-17, December 9, 1993).

Financial Management: Reliability of Weapon System Cost Reports Is
Highly Questionable (GAO/AIMD-94-10, October 28, 1993).

Defense: Health Care (GAO/IMTEC-93-29R, June 3, 1993).

Mission-Critical Systems: Defense Attempting to Address Major Software
Challenges (GAO/IMTEC-93-13, December 24, 1992).

Defense ADP: Corporate Information Management Must Overcome Major
Problems (GAO/IMTEC-92-77, September 14, 1992).

Air Force ADP: Status of Logistics Modernization Projects and CIM Impacts
(GAO/IMTEC-92-66, July 30, 1992).

Defense Logistics: DOD Initiatives Related to Cutting Costs
(GAO/T-NSIAD-92-24, March 26, 1992).

Defense ADP: Lessons Learned From Development of Defense Distribution
System (GAO/IMTEC-92-25, March 20, 1992).

Automated Information Systems: Defense Should Stop Further
Development of Duplicative Recruiting Systems (GAO/IMTEC-92-15,
February 27, 1992).

Defense ADP: A Coordinated Strategy Is Needed to Implement the CALS
Initiative (GAO/IMTEC-91-54, September 13, 1991).

Defense Distribution System Illustrates Problems Facing Corporate
Information Management Interim Standard Systems (GAO/T-IMTEC-91-12,
April 23, 1991).




Page 91                                 GAO/AIMD-98-5 Defense Migration Strategy
           Related GAO Products




           Defense ADP: Corporate Information Management Initiative Faces
           Significant Challenges (GAO/IMTEC-91-35, April 22, 1991).

           Defense ADP: Corporate Information Management Savings Estimates Are
           Not Supported (GAO/IMTEC-91-18, February 22, 1991).

           Army Automation: Decisions Needed on SIDPERS-3 Before Further
           Development (GAO/IMTEC-90-66, September 5, 1990).

           Automated Information Systems: Defense’s Oversight Process Should Be
           Improved (GAO/IMTEC-90-36, April 16, 1990).

           Schedule Delays and Cost Overruns Plague DOD Automated Information
           Systems (GAO/T-IMTEC-89-8, May 18, 1989).

           Automated Information Systems: Schedule Delays and Cost Overruns
           Plague DOD Systems (GAO/IMTEC-89-36, May 10, 1989).




(511355)   Page 92                              GAO/AIMD-98-5 Defense Migration Strategy
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested