oversight

Automated Information Systems: Defense's Oversight Process Should Be Improved

Published by the Government Accountability Office on 1990-04-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

Defense’s Oversight Process
Should l3e Improved

                --..-
                                 !I



                        141220
Information    Management   and
Technology    Division

B-238775

April 16,199O

The Honorable John Conyers
Chairman, Legislation and National
  Security Subcommittee
Committee on Government Operations
House of Representatives

The Honorable Frank Horton
Ranking Minority Member, Legislation
  and National Security Subcommittee
Committee on Government Operations
House of Representatives

As requested in your May 18, 1989, letter, we have assessed the process used by the Office of
the Secretary of Defense to oversee the development of major automated information
systems. This report describes Defense policies on the development and oversight of
automated information systems, explains how those policies have been implemented, and
provides our observations on recent oversight reviews conducted by Defense’s Major
Automated Information System Review Committee.

As agreed with your office, unless you publicly announce its contents earlier, we plan no
further distribution of this report until 30 days from the date of this letter. At that time, we
will send copies to interested parties and make copies available to others upon request.

This report was prepared under the direction of Samuel W. Bowlin, Director, Defense and
Security Information Systems, who may be reached at (202) 275-4649. Other major
contributors are listed in the appendix.



LLLkQ~&
Ralph V. Carlone
Assistant Comptroller General
Executive Summary


                   The Department of Defense spends about $9 billion annually to acquire,
Purpose            operate, and maintain general purpose automated information systems.
                   For more than 10 years, Defense policy for developing and overseeing
                   systems has required a structured process which stresses sound finan-
                   cial management and continuing mission evaluation. However, because
                   Defense has problems developing systems on time and within budget,
                   the Chairman and Ranking Minority Member, Legislation and National
                   Security Subcommittee, House Committee on Government Operations,
                   asked GAO whether Defense is effectively controlling the acquisition of
                   major automated information systems.


                   Defense’s policy for the life cycle management of major automated
Background         information systems requires thorough and effective oversight generally
                   commensurate with the anticipated investment. The greater the invest-
                   ment, the higher the level of oversight. The Office of the Secretary of
                   Defense (OSD) is required to oversee systems when development and
                   deployment costs are estimated to exceed $25 million in 1 year, $100
                   million total, or if the system is of special interest. In the late 1970s the
                   Major Automated Information System Review Council (now Committee)
                   was established by O!3Dto review systems requiring its oversight.

                   Representing the Secretary of Defense, the MAISRC reviews systems at
                   established milestones during the development cycle to determine
                   whether they should be continued, redirected, or terminated. The MAISRC
                   members are senior-level Defense officials and the chairperson is the
                   Assistant Secretary of Defense (Comptroller). Staff from various OSD
                   offices assist the MAISRC members in reviewing major systems.


                   OSD is not effectively enforcing established policies for controlling the
Results in Brief   acquisition of major automated information systems. The military ser-
                   vices are responsible for managing automated information system devel-
                   opment, and they are required to provide effective oversight even when
                   the systems will be reviewed by the MAISRC.

                   GAO believes that the repeated identification of life cycle management
                   deficiencies indicates that OSDneeds to be more aggressive in holding the
                   services accountable for compliance with Defense policies. For almost 10
                   years, OSD has been directing the services to correct non-compliance defi-
                   ciencies on individual systems, and emphasizing the services’ oversight
                   responsibilities. While OSD'S authority to redirect or terminate systems
                   may be its most effective tool for holding the services accountable, GAO'S


                   Page 2                                GAO/iMTEC90-36 Defense’sOversight Process
                         Executive Summary




                         work indicates that OSD is not effectively using its authority when the
                         circumstances indicate that such action is warranted.



Principal Findings

OSDHas Not Been          Defense services and agencies have been criticized repeatedly by GAO,
Effective in Resolving   congressional committees, and others for developing systems that run
                         over budget, behind schedule, and which do not operate as intended.
Development Problems     Recent reviews by GAO, the Defense Inspector General, and the MAISRC
                         blame many of these deficiencies on the services’ failure to fully comply
                         with established life cycle management policies and procedures.

                         For example, a February 1989 Defense Inspector General report’ on
                         information resources management within the three military services
                         cited 240 instances of non-compliance with regulations. The report
                         stated that the services encountered development problems because pro-
                         gram managers did not follow established guidelines and regulations. In
                         May 1989, GAO reported2 on eight Defense systems which had expe-
                         rienced significant cost growth and schedule delays. GAO noted that ser-
                         vice officials attributed the problems to underestimating the systems’
                         original costs, design failures, program redirection, and enhancements to
                         the original project scope. In October 1989, OSDconcluded that the signif-
                         icant cost growth and schedule delays were generally due to weaknesses
                         in (1) executing life cycle management policies, (2) the quality of pro-
                         gram management decisions, (3) the effectiveness of service oversight,
                         and (4) the currency and accuracy of oversight information provided to
                         OSD.

                         OSDhas taken a number of actions intended to enhance life cycle man-
                         agement policies and improve compliance by the services. For almost 10
                         years, OSD has emphasized the requirement that the services maintain
                         their own review process. Although each of the services has imple-
                         mented an oversight process, deficiencies persist in system development,
                         particularly non-compliance with life cycle management policies and
                         procedures.

                         ‘Defenue Inspector General, Report                                       (Report No.
                         89-03), February 17,1989.

                         ‘Automated Information Systems:ScheduleDelaysand Costoverruns PlagueDODSystems(GAO/
                         IIciITFJc-89-36,
                                       May 10,1989).



                         Page 3                                    GAO/lMTEGW36 Defense’sOversight Process
                         Executlvb fhmmnry




OSDHas Not Effectively   The MAISRC'S reviews of system development efforts have generally
Used Its Authority to    raised a number of good questions and the MAISRC has directed the ser-
                         vices to correct identified problems. At the same time, however, GAO'S
Redirect or Terminate    work shows that 06~ has not made the tough decisions to redirect or
Systems                  terminate system development efforts when the circumstances indicate
                         that it should.

                         GAO's analysis of nine systems reviewed by the MAISRC during the last
                         few years showed that MAISRC staff-including   analysts in OSD'S Pro-
                         gram Analysis and Evaluation and the Director of Operational Test and
                         Evaluation offices-consistently     identified deficiencies in major systems
                         being developed by all three military services. Many of the deficiencies
                         related to the services’ inappropriate life cycle management task execu-
                         tion, and the services’ failure to carry out their oversight responsibili-
                         ties. For example, the staff identified deficiencies such as unjustified
                         costs and benefits, unacceptable test plans, inadequate analysis of
                         potential risks, ineffective program management, and incomplete system
                         design. The MAWC members were routinely made aware of these defi-
                         ciencies and almost always brought the deficiencies to the services’
                         attention.

                         GAO's analysis of the nine systems, as well as its own reviews of system
                         development efforts, also shows that @SDhas allowed system develop-
                         ment efforts to continue despite development problems or the services’
                         failure to comply with Defense policies. For example, OSDhas allowed
                         system development to continue when the MAISRC identified problems in
                         the services’ development approach, and when previous MAISRC guidance
                         was not implemented.


                         GAO recommends that the Secretary of Defense make greater use of the
Recommendationsto        authority to redirect or terminate system development efforts when the
the Secretary of         results of MAISRC reviews indicate that such action is warranted. To
Defense                  accomplish this, the Secretary should direct the MAISRC to (1) withhold
                         milestone approval and prohibit further development on any systems
                         that do not comply with Defense policies and (2) ensure that its deci-
                         sions are reflected in the services’ budgets.

                         To achieve a long-term solution to the problem of the services’ non-com-
                         pliance with Defense policies, the Secretary should direct the MAISRC to
                         implement a separate procedure to periodically assess the adequacy of
                         the services’ oversight processes, and recommend corrective action
                         when it determines that the processes are deficient. Such a procedure


                         Page 4                                GAO/IMTJ3G9O36Defense’sOversight Process
                  Jbcutive Summary




                  would give the MAISRC a basis for assessing the risk of non-compliance in
                  individual systems and for holding the services accountable for failing
                  to implement ark acceptable oversight process.


                  GAO did not request official agency comments on a draft of this report.
Agency Comments   However, GAO discussed the report’s findings with agency officials and
                  incorporated their comments where appropriate.




                  Page6                               GA0/IMTEC9036Defenae’s OversightProcese
Contents


Executive Summary
Chapter 1
Background                Life Cycle Management Is Fundamental to Defense’s
                               Oversight of Systems
                          The MAISRC Is Defense’s Oversight Authority                                9
                          Defense Is Reassessing the Acquisition and Oversight                      11
                               Process
                          Objectives, Scope, and Methodology                                        12

Chapter 2                                                                                           14
Defense Is Not            Systems Developed by the Services Have Contained
                              Numerous Deficiencies
                                                                                                    14
Effectively Controlling   OSD Has Directed the Services to Improve Compliance                       15
System Development            and Oversight
                          The MAISRC Continues to Identify Deficiencies and                         16
                              Recommend Corrective Action
                          OSD Has Not Effectively Used Its Authority to Redirect                    18
                              or Terminate Systems

Chapter 3                                                                                           21
Conclusions and           Recommendations to the Secretary of Defense                               22
Recommendations
Appendix                  Appendix I: Major Contributors to This Report                             24

Related GAO Products                                                                               28

Figure                    Figure 1.1: Defense’s Life Cycle Management Phases and                     9
                               Milestones

Table                     Table 2.1: Deficiencies Identified by the MAISRC Support
                              Staff


            Y




                          Page 6                              GAO/IMTEG!lO-36Defense’sOversight Process
    Abbreviations

    DOD       Department of Defense
    GAO       General Accounting Office
    IMTJX     Information Management and Technology Division
    MAISRC    Major Automated Information System Review Committee
Y
    OSD       Office of the Secretary of Defense
    DUNE      Director of Operational Test and Evaluation
    PA&E      Program Analysis and Evaluation


    Page7
Chapter 1

Background


                        The Department of Defense spends about $9 billion annually on auto-
                        mated information systems used to manage billions of dollars in logis-
                        tics, personnel, and financial resources critical to its mission, Since the
                        late 197Os, Defense has required a structured process called life cycle
                        management for developing or modernizing major automated informa-
                        tion systems. The process emphasizes the need to develop systems that
                        will meet Defense requirements and stresses sound financial manage-
                        ment and continuing mission evaluation. Life cycle management also
                        requires Defense management to oversee systems and determine
                        whether they should be continued, redirected, or terminated.

                        The level of oversight required by life cycle management generally
                        depends on a system’s cost-the greater the cost, the higher the level of           ’
                        oversight, The Office of the Secretary of Defense (OSD) established the
                        Major Automated Information System Review Committee (MAISRC) to
                        oversee the development of systems when cost estimates exceed $25
                        million for 1 year, $100 million total, or when the system is of special
                        interest.


                        Life cycle management is intended to ensure that Defense management
Life Cycle              is accountable for the success or failure of systems. Life cycle manage-
Management Is           ment is also intended to give Defense program managers a structured
Fundamental to          approach for developing automated information systems. Defense’s
                        guidelines for life cycle management define six development phases and
Defense’sOversight of   six decision points (called milestones) where system progress is assessed
Systems                 and documented. Figure 1.1 shows the six life cycle management phases,
                        the corresponding milestones, and the questions which must be affirma-
                        tively answered before a system can proceed to its next development
                        phase.




                        Page 8                                GAO/IMTJIC90-36 Defense’sOversight Process
                                             Chapter 1
                                             Background




Figure 1.l : Defense’s Life Cycle Management Phases and Milestones

   Phase 0          Phase 1                Phase 2            Phase 3              Phase 4           Phase 5




                                                                                                              A




                                                                                                                       -l
                                                                                                                       Time to

                                                                                                                       modernize
                                                                                                                       or replace?
                                                                        Does the                                       How best

                                                                        system                                         can this




                                                                        3
                                                                        work?                                          be done?




                                                                Milestones
                                                          ”


                                             The decision to allow a system to proceed from one phase to the next is
                                             based, in part, on management’s analysis of system documentation.
                                             Throughout development, the program manager is expected to maintain
                                             documentation that demonstrates the level of analysis and planning put
                                             into the system, The documentation is expected to be kept current and
                                             updated any time a change has been approved.


                                             As Defense’s senior information resource management official, the
The MAISRC Is                                Assistant Secretary of Defense (Comptroller) is the Chairperson of the
Defense’sOversight                           MAISRC, and is responsible for conducting oversight reviews of major
Authority                                    automated information systems, In addition to the Comptroller, the
                                             MAISRC is made up of technical, functional, and system sponsor repre-
                                             sentatives such as
                     Y
                                       l     the Assistant Secretary of Defense for Command, Control, Communica-
                                             tions and Intelligence, who provides insight on communications and
                                             security issues;


                                             Page 9                                          GAO/IMTEG90-36       Defense’s   Oversight   Process
        Chapter 1
        Background




    l the Assistant Secretary of Defense, Program Analysis and Evaluation
      (PA&E), who provides expert advice on program cost estimating;
    l the Director, Operational Test and Evaluation (DOTBtE), who provides
      expert advice on testing;
    l the OSD functional sponsor, who validates the requirements; and
    . a senior representative from the military service or Defense agency that
      is going to use the system.

        MAISRC technical and functional support staff analyze and give an opin-
        ion on the quality and depth of each system’s development plans. Prior
        to formal MAISRC oversight reviews, the support staff analyze a system’s
        status and prepare summary documents for the MAISRC members. For
        example, PA&E analysts assess the adequacy of the service’s cost/benefit
        and economic analyses and DOLYZEanalysts assess the service’s test plans
        and test results.

        During the oversight review, the MAISRC members determine, from an OSD
        perspective, whether the system is being developed in accordance with
        Defense policies, procedures, and regulations. Among other things, the
        MAISRC should analyze whether the service has taken steps to

l       reduce the time and cost to develop and field automated information
        systems by maximizing the use of commercial products;
l       avoid duplication and unnecessary expenditures on new systems by
        effectively using existing systems;
l       minimize the cost of new systems by ensuring full and open competition;
        and
l       effectively use advanced system design and software engineering tech-
        nology to minimize software development and maintenance costs.

        The MAISRC'S review is intended to determine whether development
        should be continued, redirected, or terminated. The MAISRC'S decision
        and any recommendations are documented in a System Decision Memo-
        randum. When necessary, the memorandum is supposed to provide spe-
        cific guidance to the military service on correcting problems.

        The military services are responsible for establishing their own similar
        oversight and milestone approval processes for systems being developed
        or modernized. Among other things, the services are supposed to ensure
        that their systems follow life cycle management principles and adhere to
        program schedule and cost goals. The systems are supposed to be
        reviewed by senior-level service officials even if an OSD MAISRC review is
        to be held.


        Page 10                              GAO/IMTEC9636   Defense’s   Oversight   Process
                          Chapter1
                          Background




The MAISRC Has            Defense tries to minimize the layers of oversight by permitting the
                          MAISRC to delegate its review and milestone approval authority to the
Delegated Oversight for   services. The MAISRC normally maintains approval authority for the first
SomeSystems to the        two milestones, which focus on early decisions such as need justifica-
Services                  tion, alternative concept analysis, and acquisition strategy. The MAISRC'S
                          attention to these early milestones is intended to ensure that early deci-
                          sions establish a solid foundation for later development.

                          Before delegating approval authority for a system, the MAISRC is sup-
                          posed to be assured that the service has an effective review and mile-
                          stone approval process in place. At the time of our review, milestone
                          approval for over half of Defense’s 52 major automated information sys-
                          tems had been delegated to the services. When OSDdelegates oversight
                          authority to a service, the MAISRC is expected to monitor the system’s
                          progress to ensure its successful development is not jeopardized. The
                          MAISRC has established the following criteria to determine whether dele-
                          gation should be revoked:

                          cost growth of 25 percent or more for the overall program,
                          schedule slippage of 6 months or more,
                          inadequate service oversight, or
                          significant problems in the system’s development.


                          In October 1989, the Deputy Secretary of Defense established the
Defense Is Reassessing    Department of Defense Corporate Information Management initiative.
the Acquisition and       This initiative is intended to ensure the standardization, quality, and
Oversight Process         consistency of data from Defense’s multiple management information
                          systems, and to identify standard functional requirements for meeting
                          Defense’s management information needs. Industry leaders and senior-
                          level Defense officials will be asked to recommend an overall approach
                          and action plan for a Corporate Information Management program, eval-
                          uate the current oversight process, and recommend corrective actions.
                          This group will also review the procedures of Defense groups, analyzing
                          Defense’s various functional areas including financial management,
                          civilian personnel, and materiel management.

                          In addition, experts in Defense policy and information systems will
                          develop concepts for improving Defense business practices. These
                          experts will identify and develop standard functional requirements and
                          data formats for each of Defense’s functional areas. Recognizing that
                          one goal of the initiative is to develop standard, departmentwide sys-
                          tems, Defense’s fiscal year 1991 appropriations request reflected a $288


                          Page 11                              GAO/IMTEG96-36   Defense’s   Oversight   Process
                        Chapter1
                        lhekgmund




                        million reduction of the services’ and Defense agencies’ automated data
                        processing budgets.

                        In an effort to streamline Defense’s acquisition process, in October 1989
                        the Deputy Secretary of Defense designated the MAISRC as a committee of
                        the Defense Acquisition Board. The Comptroller will continue to serve
                        as the MAISRC chairperson, and life cycle management principles and
                        processes for major automated information systems are to remain in
                        effect. The MAISRC is to continue to review major systems, and telecom-
                        munications programs, prior to Defense Acquisition Board meetings.


                        The objective of this review was to determine whether Defense is effec-
Objectives, Scope,and   tively controlling the acquisition of major automated information sys-
Methodology             tems. This review was requested by the Chairman and Ranking Minority
                        Member, Legislation and National Security Subcommittee, House Com-
                        mittee on Government Operations. The May 18, 1989, request followed
                        the Subcommittee’s hearing on cost growth and schedule delays incurred
                        by Defense in its development of eight automated information systems.

                        We assessed the process used by the MAISRC to oversee major automated
                        information systems. This included an assessment of the input provided
                        by the Deputy Assistant Secretary of Defense for Information Resources
                        Management, the Assistant Secretary of Defense for Program Analysis
                        and Evaluation, and the Director, Operational Test and Evaluation.

                        We also identified the process used by the Air Force, Army, and Navy to
                        oversee system development. We limited our review to the three military
                        services, because their systems represent most of Defense’s automated
                        data processing purchases.

                        To gain a thorough understanding of Defense’s oversight process, we
                        reviewed directives, instructions, and regulations pertaining to systems
                        development and oversight. We also interviewed OSD and service staff
                        who oversee the development of systems. For example, we interviewed
                        the Deputy Assistant Secretary of Defense for Information Resources
                        Management, and members of the Deputy’s staff; officials from PA&E and
                        DOT&E who participate in MAISRC deliberations; staff representing each of
                        the services’ senior information resources management officials; and
                        program managers responsible for developing selected Air Force, Army,
                        and Navy systems.




                        Page 12                             GAO/IMTEG90-36   Defense’s   Oversight   Process
Chapter 1
Background




To assess the results of the oversight process, we reviewed nine systems
which had been subjected to 11 MAISRC reviews from April 1987 through
July 1989. The systems were the

Air Force Depot Maintenance Management Information System,
Air Force Personnel Concept III Program,
Army Integrated Procurement System,
Army Civilian Personnel System,
Army Supercomputer Program,
Navy Stock Point ADP Replacement Project,
Navy Engineering Data Management Information and Control System,
Navy Integrated Disbursing and Accounting Financial Information
Processing System, and
Naval Aviation Logistics Command Management Information System.

Before reviewing the systems, we discussed our selections with the Dep
uty Assistant Secretary of Defense for Information Resources Manage-
ment. The Deputy agreed that the systems we selected would provide an
accurate illustration of the MAISRC review process. However, because our
selection of the nine systems was not a random sample, our audit find-
ings cannot be scientifically projected to the entire universe of major
systems being developed by Defense.

We conducted our audit work from June 1989 to March 1990, primarily
within OSD and at the Departments of the Air Force, Army, and Navy
headquarters at the Pentagon, Washington, D.C. As requested, we did
not obtain official agency comments; however, we did discuss the con-
tents of this report with OSDofficials, and have incorporated their com-
ments where appropriate. Our work was performed in accordance with
generally accepted government auditing standards.




Page 13                             GAO/IMTEGBO-36Defense’sOverdgkt Proceee
Chapter 2

DefenseIs Not Effectively Controlling
      Development

                       In recent years the Department of Defense has been criticized by a
                       number of sources for its failure to deliver automated information sys-
                       tems on time and within budget, which operate as intended. Many of the
                       systems, problems have been caused by the military services’ failure to
                       fully comply with regulations, policies, and procedures, including life
                       cycle management requirements. Although OSD has taken actions
                       intended to improve the services’ compliance with requirements and
                       oversight of systems, problems still exist.

                       Specifically, our assessment of selected oversight reviews conducted by
                       the MAISRC within the last few years has shown that the services’ failure
                       to fully comply with life cycle management continues to be a problem.
                       We also found that OSD has not made the tough decisions to redirect or
                       terminate system development efforts when the circumstances indicate
                       that such actions may be warranted.


                       Defense’s problems in developing major automated information systems
Systems Developed by   are well documented. Our work, reviews conducted by other audit orga-
the Services Have      nizations, and congressional inquiries have identified hundreds of
Contained Numerous     instances in which the services have not complied with regulations, poli-
                       ties, and procedures when developing automated information systems.
Deficiencies           Specifically, systems experienced cost growth and schedule delays due
                       to poor needs identification, inadequate investigation of alternative
                       solutions, limited oversight, and incomplete justification of costs and
                       benefits.

                       A February 1989 report’ by the Defense Inspector General on informa-
                       tion resources management concluded that the services were not rigor-
                       ously complying with Defense’s life cycle management policies and
                       procedures. The report cited 240 instances of non-compliance with
                       established regulations, and noted that most of the instances occurred
                       during the initial two life cycle management phases-mission analysis/
                       project initiation and concept development. The report also stated that
                       the services encountered problems developing automated information
                       systems because program managers did not follow established guide-
                       lines and regulations, not because they lacked guidance.




                       ‘Defense Inspector General, Report on Information Resources Management Within DOD (Report No.
                       89-INS-O3),February 17,1989.



                       Page 14                                        GAO/IMTEG90-36     Defense’s   Oversight   Process
     .



                      chapter2
                      Defense Is Not Effbctively   Controlling
                      System Development




                      In a May 1989 report? on eight Defense systems, we noted that the esti-
                      mate to develop and deploy the systems had almost doubled-from
                      about $1 billion to about $2 billion. In addition, the development of two
                      of the systems had been abandoned after spending about $237 million,
                      and the completion dates for all but one of the remaining six systems
                      had been delayed by 3 to 7 years. Moreover, all eight systems expe-
                      rienced problems to varying degrees regardless of whether they had
                      been reviewed by the MAISRC. Our report noted that service officials
                      attributed the problems to underestimating the systems’ original costs,
                      design failures, program redirection, and enhancements to the original
                      project scope.
                                                                                                                         .
                      In response to congressional direction, OSD reviewed the history of the
                      eight automated information systems to identify the causes of the prob-
                      lems. It concluded,that the significant cost growth and schedule delays
                      were generally due to weaknesses in (1) executing life cycle manage-
                      ment policies, (2) the quality of program management decisions, (3) the
                      effectiveness of service oversight, and (4) the currency and accuracy of
                      oversight information provided to OSD. More specifically, OSDfound that
                      development problems were caused by the services’ insufficient require-
                      ments analysis, inaccurate cost and schedule estimates, inappropriate
                      development approaches, limited assessment of alternatives, and unreli-
                      able data provided to OSD.


                      OSD has attempted to strengthen the services’ oversight of automated
OSDHas Directed the   information systems by emphasizing the services’ responsibility to
Services to Improve   establish their own review processes similar to the MAISRC. In August
Compliance and        1981, for example, the Comptroller gave the services explicit guidance
                      for monitoring system development in accordance with life cycle man-
Oversight             agement policies and procedures. The Comptroller directed each service
                      to establish a senior management review group to function as a decen-
                      tralized version of the MAISRC. The management review groups were
                      directed to provide overall policy direction and reinforce both accounta-
                      bility and primary management controls.

                      In March 1986, as part of an effort to clarify and streamline automated
                      information system development, OSDemphasized the importance of life
                      cycle management and system oversight. Specifically, the Comptroller
                      stated that each service should have an accountable, executive-level

                      2Aut.omated Information Systems: Schedule Delays and Cost Overruns Plague DOD Systems (GAO/
                      IMTEC-89-36,May 10,1989).



                      Page 16                                       GAO/KM’IEG90-!36   Defense’s   Oversight   Process
                        Chapter 2
                        Defense Is Not Effectively   Controlling
                        System Development




                        review process based on the principles of life cycle management. The
                        Comptroller also stated that effective service oversight processes are
                        essential to achieve further streamlining by increasing the number of
                        delegated systems.

                        Although all three services have senior oversight groups to review auto-
                        mated information systems, deficiencies in systems’ development activi-
                        ties, particularly non-compliance with life cycle management policies
                        and procedures, have persisted. As a result of OSD’S in-depth analysis of
                        the eight systems on which we reported, the need to improve the ser-
                        vices’ oversight of systems and compliance with established require-
                        ments was again emphasized. In October 1989, OsD directed the services
                        to initiate action to ensure that the underlying weaknesses in life cycle
                        management task execution, program management, and service over-
                        sight processes are corrected. Although the services submitted action
                        plans in January 1990, OSD told the services that they were not specific
                        enough and asked that they be improved.


                        From April 1987 to July 1989, the MAISRC conducted 35 reviews of 22
The MAISRC              systems being developed by the services and Defense agencies. We ana-
Continues to Identify   lyzed the process the MAISRC staff used to review nine of the systems
Deficiencies and        being developed by the services, in order to determine the kinds of defi-
                        ciencies it identified, whether they were brought to the attention of the
Recommend               MAISRC members, and if corrective action was recommended.
Corrective Action
                        As shown in table 2.1, the MAISRC support staff identified one or more
                        deficiencies in all but one of the systems.




                        Page 16                                    GAO/IM’IEC9O-26   Defense’s   Oversight   Process
                                                      Chapter2
                                                      Defense Is Not Eiyectively   Cbntrolling
                                                      System Development




Table 2.1: Deficlencles Identified by the MAISRC Support Staff
                                                                                   Deficiencies Identified
                                                                                                                       Configuration!
                                                                                   Risk             Program                  capacity           Development
System                                    Testing       Cost/benefit        management           management             management                  strategy
.._._...._
         _ _.-~ .._._
                    .._..
                       -..--..--- -----
Air Force Depot Maintenance
   Management Information
   System                                        X                                         X
Air Force Personnel Concept Ill
   Program                                                                                                        X
                                                                                                                                                   ---
A;mylntegrated     Procurement
                                                                       X
   System                                   --___-X                                                                                                      ___..
Army &han Personnel System                       X                     X                                          X                                      __-
Ar&‘Suoercomwter        Proaram
Navy Stock Point ADP
   Replacement Project                           X                     X                                                                                         -
Navy Englneenng Data
   Management lnformatlon and
   Control System                                X
__.__II_._-......_
Navy Integrated Disbursing and
   Accounhq Flnanclal
   lnformatlon Processing
   Svstem                                        X                     X                  X                       X                      X                         X
Naval Awation Logistics
   Command Management
   lnformatlon System                            X                                        X                                              X


                                                    Many of the identified deficiencies occurred because the services were
                                                    not strictly following life cycle management policies and procedures.
                                                    These deficiencies were almost always reflected in the System Decision
                                                    Memoranda provided to the services after the MAISRC reviews. We also
                                                    found that PA&E and DOI%@consistently provided input to the oversight
                                                    reviews of the nine systems.

                                                    For seven of the nine systems, the MAISRC support staff noted deficien-
                                                    cies in the test plans submitted by the services. For example, in prepar-
                                                    ing for a June 1988 milestone I review of the Army’s Integrated
                                                    Procurement System, DOME determined that the Army needed to
                                                    develop a test and evaluation master plan for the system. Although the
                                                    MAISRC granted milestone approval, its System Decision Memorandum
                                                    directed the Army to submit a test and evaluation plan within 6 months.
                                                    According to a DCYIXEanalyst, the test plan was submitted in September
                          Y
                                                    1989-15 months after the milestone review-and was subsequently
                                                    approved by DOME.




                                                    Page 17                                                 GAO/lMTECXO-36   Defense’s       Overnight   Process




                                                                                                   ‘8 ,!!     ” “,#
                        chapter2
                        Defense Is Not Effectively   Controlling
                        System Development




                        Similarly, for four of the nine systems, the MAISRC support staff noted
                        deficiencies in the services’ estimates of system costs and benefits. For
                        example, PA&E found that the Navy submitted an incomplete indepen-
                        dent cost estimate and benefit analysis for the Integrated Disbursing
                        and Accounting Financial Information Processing System. PA&E noted
                        that the Navy, in estimating the system’s cost, did not include all sunk
                        costs, costs for government-furnished equipment and services, and oper-
                        ator expenses, The MAISRC System Decision Memorandum included a
                        requirement that the Navy obtain PA&E validation of the system’s life
                        cycle cost estimate, independent cost estimate, and program benefits
                        estimate prior to milestone approval.

                        Concerns about the adequacy of the services’ risk analysis, program
                        management, configuration and capacity management, or overall devel-
                        opment strategy were also raised by the support staff for five of the
                        nine systems we reviewed. For example, in preparing for a June 1989
                        review of the Air Force Depot Maintenance Management Information
                        System, the MAISRC staff noted that the Air Force did not appear to have
                        an active risk management program that would identify risks, develop
                        plans to address those risks, and closely monitor the program for indica-
                        tions that problems are occurring.

                        In its System Decision Memorandum for this system, the MAISRC deferred
                        approval of milestone II. The memorandum directed the Air Force to
                        develop and document a risk management program that would consider
                        risks related to cost, schedule, and technical implementation, According
                        to an OSDofficial, the Air Force submitted its plans to address risk in
                        October 1989, and the plans will be assessed before the next milestone II
                        review.


                        While the MAISRC reviews we assessed have found deficiencies and rec-
OSDHas Not              ommended corrective actions, our work also shows that other problems
Effectively Used Its    have not been addressed. Specifically, the MAISRC did not take corrective
Authority to Redirect   action to ensure the services complied with its previous guidance, and
                        failed to redirect or recommend terminating systems that had questiona-
or Terminate Systems    ble acquisition strategies.

                        For example, during the fiscal year 1986 appropriations process, the
                        Conference Committee on Appropriations expressed concern about the
          Y
                        progress of the Naval Aviation Logistics Command Management Infor-
                        mation System and directed OSDto conduct a MAISRC review. At that
                        time, the system had been in development 9 years, full deployment was


                        Page 18                                    GAO/IMTEC%O36   Defense’s   Oversight   Process
Chapter2
Defense Is Not Effectively   Controlling
System Development




4 years behind schedule, and oversight authority for the system was
delegated to the Navy. The MAISRC reviewed the system in July 1986,
and expressed concern that the Navy’s plans to complete deployment of
the system in 1996 would delay operational benefits. As a result, the
MAERC directed the Navy to complete deployment of the system by 1993.
It also allowed the Navy to retain direct oversight responsibility for the
system.

In a July 1989 review, the MAISRC staff noted that the Navy had not
followed the 1986 guidance, that the estimate for full deployment had
slipped to 1999, and that the Navy had spent $233 million on the sys-
tem. Nevertheless, the MAISRC granted conditional milestone approval for
limited deployment of the system. The System Decision Memorandum
directed the Navy to submit another assessment of alternatives for
speeding up full deployment.

During a recent review of the Naval Aviation Logistics Command Man-
agement Information System,3 we identified concerns about equipment
acquisition and the sufficiency of stress testing and operational testing.
During the fiscal year 1990 appropriations process, the Congress stated
that it expects the MAISRC to ensure that the Navy addresses these con-
cerns and achieves deployment of the system by 1996 as promised to the
Congress.

During a recent review of an Air Force system, the MAISRC failed to iden-
tify a number of development problems. Following a June 1989 mile-
stone II review of the Air Force Personnel Concept III system, the MAISRC
commended the Air Force for having done an excellent job planning and
executing the program. The MAISRC delegated future milestone approval
and oversight authority for the system to the Air Force, and the System
Decision Memorandum provided minimal guidance.

However, in a separate report4 on this $660 million personnel system, we
found that the Air Force planned to deploy the system to 126 bases even
though it had not (1) fully developed and tested the system, (2) fully
analyzed requirements or alternatives for the hardware design, or (3)
adequately supported personnel savings. As a result of our work, the
Congress withheld further funding for the system until the Air Force

        ter Acquisition: Navy’s Aviation Logistics System Not Ready for Deployment (GAO/
        90-11, Feb. 9, 1990).

“Air Force ADP: The Personnel Concept III System Is Not Ready for Deployment (GAO/
Im-90-22,     Feb. 27,199O).



Page 19                                          GAO/IMTJXCXM%M     Defense’s   Oversight   Process
    Chapter2
    Defense Is Not JWfectively   Controlling
    System Development




    demonstrates that the proposed hardware configuration is the most
    cost-effective alternative.

    As a result of its review of the Navy’s Integrated Disbursing and
    Accounting Financial Information Processing System, the MAISRC con-
    cluded that the system’s overall program planning and design were not
    sufficient to grant milestone approval. However, despite the system’s
    history of development failures, cost growth, and schedule delays, the
    MAISRC did not use its authority to recommend terminating the system.

    When the MAISRC reviewed this system for the first time in February
    1989, the system had been in development for more than 10 years, and
    the Navy had spent about $94 million of the system’s estimated $591
    million life cycle cost. The MAISRC found that

l   the overall program planning and design of the system were not com-
    plete and did not meet life cycle management policy requirements,
l   substantial development efforts had been implemented without the com-
    pleted system design and without required senior Navy management
    oversight reviews,
l   the development methodology, after two prior failed attempts, was
    unorthodox, and
l   test plans were incomplete.

    The MAISRC directed the Navy to take specific action to correct the
    numerous deficiencies identified during the review before further devel-
    opment of the system could proceed. For example, the Navy was
    directed to complete milestone II planning and analysis, complete the
    system’s design efforts, and evaluate the risks associated with the unor-
    thodox development approach. The corrective action was to be reported
    to the MAISRC within 6 months. However, the Congress subsequently
    denied fiscal year 1990 funds for this Navy system.




    Page 20                                    GAO/IMTEC%O-36   Defense’s   Oversight   Process
Chapter 3

Conclusions md Recommendations


             Our review has shown that OSDis not effectively enforcing Defense poli-
             cies for controlling the acquisition of major automated information sys-
             tems. Problems exist in two areas. First, OSDhas not been effective in
             getting the services to develop automated information systems that com-
             ply with the requirements of life cycle management. Secondly, OSD has
             not terminated or redirected system development efforts when the ser-
             vices have not complied with life cycle management or other Defense I
             policies.

             Defense policies and procedures for automated information system
             development call for thorough and effective oversight commensurate
             with the anticipated investment. The military services are responsible
             for managing automated information system development, and they are
             required to provide effective oversight even when the systems will be
             reviewed by the MAISRC. MAISRC oversight is not intended to be a
             substitute.

             Yet we found that OSD'S oversight process is not holding the services
             accountable for compliance with Defense policies. It seems to us that
             OSD'S authority to redirect or terminate systems when the hws~c recom-
             mends withholding milestone approval may be the most effective tool
             OSD has to hold the services accountable for complying with Defense pol-
             icies. Our work indicates, however, that rather than having the MAISRC
             withhold approval and reduce the services’ budget requests for individ-
             ual systems, OSDhas tried to fix the problem by directing the services to
             correct non-compliance deficiencies on individual systems, and empha-
             sizing the services’ oversight responsibilities.

             It is interesting to note that OSDrecently used the budget process to push
             the services toward working with each other to develop standard sys-
             tems. As part of its Corporate Information Management initiative, OSD
             reduced each of the services’ fiscal year 1991 budget requests to reflect
             anticipated savings from the initiative.

             We believe that the repeated identification of life cycle management
             deficiencies by the MAISRC indicates that OSDneeds to be more aggressive
             in holding the services accountable for complying with Defense policies.
             We do not believe that OSDshould rely on the MAISRC to routinely identify
             the services’ non-compliance deficiencies during individual system
             reviews. Moreover, OSD should not be funding systems that are not in
             full compliance with established policies.




             Page21                               GAO/IMTJZG9O-%Jhfense'aOveraightProceas
                     Chapter3
                     Conclusions   and Recommendations




                    , Some of the systems covered in this review, as well as others we
                      recently reviewed, also indicate that OSDis not making the tough deci-
                      sions to terminate or redirect development efforts when they need to be
                      made. For example, the MAISRC has allowed systems to continue even
                      though its previous guidance had not been implemented in a timely man-
                      ner or when it has identified concerns about the adequacy of the ser-
                      vice’s development approach. On a number of occasions, the Congress
                      has stepped in and provided specific guidance or denied funding when it
                      has found that the MAISRC has allowed a system to proceed despite non-
                      compliance or other development problems.


                     In May 1989, we recommended that the Secretary of Defense review and
Recommendationsto    revise, as appropriate, the management control and decision-making
the Secretary of     process for major automated information systems development. We
Defense              understand that the Secretary of Defense is reviewing the role of the
                     MABRC and that any necessary changes will be made based upon the out-
                     come of the review. We support the MAISRC oversight process and recom-
                     mend that the Secretary of Defense be more rigorous in enforcing
                     policies for controlling the acquisition of major automated information
                     systems.

                     Specifically, we recommend that the Secretary of Defense make greater
                     use of the authority to redirect or terminate system development efforts
                     when the results of MAISRC reviews indicate that such action is war-
                     ranted. To accomplish this, the Secretary should direct the MAISRC to
                     withhold milestone approval authority and prohibit further develop-
                     ment when the services submit a system for review that does not com-
                     ply with Defense policies. The Secretary should also direct the MAISRC to
                     ensure that its decisions are reflected in the services’ budgets.

                     In order to achieve a long-term solution to the problem of non-compli-
                     ance, the Secretary should direct the MAISRC to implement a separate
                     procedure to periodically assess the adequacy of the services’ oversight
                     processes, and recommend corrective action when it determines that the
                     processes are deficient. Such a procedure would give the MAISRC a basis
                     for assessing the risk of non-compliance in individual systems and for
                     holding the services accountable for failing to implement acceptable
                     oversight.




                     Page 22                              GAO/IMTEG36-36   Defense’s   Oversight   Process
l




    Y




        Page 23   GAO/IMTEG90-36   Defense’s   Oversight   Process
Appendix I

Major Contributors to This Report


                        Thomas J. Howard, Assistant Director
Information             Kenneth W. Huber, Evaluator-in-Charge
Management and          Gwendolyn Dittmer, Evaluator
                        Sabine Richard, Evaluator
Technology Division,
Washington, D.C.




                        Page 2i                            GAO/IMTEG9036   Defense’s   Oversight   Process



             ..I.   -
.




    Page 25   GAO/lMTEG90-26   Defense’s   oversight   Process
Y




    Page 26   GAO/IMlEC9O-36   Defense’s   Overdght   Process
Y




    Page 27   GAO/IMTEG90-36   Defense’s   Oversight   Process
Related GAO ProdWts


             Schedule Delays and Cost Overruns Plague DOD Automated Information
             Systems (GAO/T-IMTEC-89-8, May 18,198Q).

             Automated Information Systems: Schedule Delays and Cost Overruns
             Plague DOD Systems (GAO/IMTEC-89-36, May 10,198Q).

             ADP Acquisition: Air Force Logistics Modernization Projects (GAO/
             IMTEC-89-42, Apr. 21,198Q).

             ADP Acquisition: Defense Logistics Services Center Modernization Pro-
             gram (GAO/IMTEC-8932, Mar. 20,198Q).

             ADP Acquisition: Army Civilian Personnel System (GAO/IMTEC-89-22FS,
             Mar. 3,198Q).

             ADP Acquisition: Naval Aviation Logistics Command Management
             Information System (GAO~MTFC-SQ-WS, Feb. 23, 1989).

             ADP Acquisition: Navy’s Efforts to Develop an Integrated Dispersing
             and Accounting System (GAO/IMTECSQ-20FS, Feb. 8, 1989).

             Computer Procurement: Decision Needed on Navy’s Standard Auto-
             mated Financial SyS&!m(GAO/IMTEC-8%47,Sept. 13, 1988).

             ADP Systems: Army Decision to Use Air Force Military Pay System
             Appears Advantageous (GAO~IMTEC-89-28,Mar. 1, 1989).

             Computer Systems: Navy Needs to Assess Less Costly Ways to Imple-
             ment Its Stock Point System (GAOIIMTEC-89-2, Dec. 14, 1988).

             Computer Procurement: Navy CAD/CAM Acquisition Has Merit but Man-
             agement Improvements Needed (GA~/IMTEC-~~-~~, May 11, 1988).

             ADP Procurement: Army Accounting System Modernization (GAO/
             IMTEC-ss-14B~, Dec. 28, 1987).

             Computer Systems: Navy Stock Point ADP Replacement Program Needs
             Better Management Controls (GAO/IMTJZC-W-30,Sept. 17,1987).




(51044@)     Page 28                            GAO/IMTEG903t3   Defense’s   Oversight   Process
1~c~c~uc~s1.sfor   twpiw   of’(;izo   rt~por1.s   sl~otiltl 1~ stbllt to:




‘I’t~ltq)llolIt~ 202-275-624 1

‘1’1~ first five twpiw     of t~atcll report. artA free, Adtlit,itmal       twpiw   art*
$2.00 twh
.