Defense’s Oversight Process Should l3e Improved --..- !I 141220 Information Management and Technology Division B-238775 April 16,199O The Honorable John Conyers Chairman, Legislation and National Security Subcommittee Committee on Government Operations House of Representatives The Honorable Frank Horton Ranking Minority Member, Legislation and National Security Subcommittee Committee on Government Operations House of Representatives As requested in your May 18, 1989, letter, we have assessed the process used by the Office of the Secretary of Defense to oversee the development of major automated information systems. This report describes Defense policies on the development and oversight of automated information systems, explains how those policies have been implemented, and provides our observations on recent oversight reviews conducted by Defense’s Major Automated Information System Review Committee. As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies to interested parties and make copies available to others upon request. This report was prepared under the direction of Samuel W. Bowlin, Director, Defense and Security Information Systems, who may be reached at (202) 275-4649. Other major contributors are listed in the appendix. LLLkQ~& Ralph V. Carlone Assistant Comptroller General Executive Summary The Department of Defense spends about $9 billion annually to acquire, Purpose operate, and maintain general purpose automated information systems. For more than 10 years, Defense policy for developing and overseeing systems has required a structured process which stresses sound finan- cial management and continuing mission evaluation. However, because Defense has problems developing systems on time and within budget, the Chairman and Ranking Minority Member, Legislation and National Security Subcommittee, House Committee on Government Operations, asked GAO whether Defense is effectively controlling the acquisition of major automated information systems. Defense’s policy for the life cycle management of major automated Background information systems requires thorough and effective oversight generally commensurate with the anticipated investment. The greater the invest- ment, the higher the level of oversight. The Office of the Secretary of Defense (OSD) is required to oversee systems when development and deployment costs are estimated to exceed $25 million in 1 year, $100 million total, or if the system is of special interest. In the late 1970s the Major Automated Information System Review Council (now Committee) was established by O!3Dto review systems requiring its oversight. Representing the Secretary of Defense, the MAISRC reviews systems at established milestones during the development cycle to determine whether they should be continued, redirected, or terminated. The MAISRC members are senior-level Defense officials and the chairperson is the Assistant Secretary of Defense (Comptroller). Staff from various OSD offices assist the MAISRC members in reviewing major systems. OSD is not effectively enforcing established policies for controlling the Results in Brief acquisition of major automated information systems. The military ser- vices are responsible for managing automated information system devel- opment, and they are required to provide effective oversight even when the systems will be reviewed by the MAISRC. GAO believes that the repeated identification of life cycle management deficiencies indicates that OSDneeds to be more aggressive in holding the services accountable for compliance with Defense policies. For almost 10 years, OSD has been directing the services to correct non-compliance defi- ciencies on individual systems, and emphasizing the services’ oversight responsibilities. While OSD'S authority to redirect or terminate systems may be its most effective tool for holding the services accountable, GAO'S Page 2 GAO/iMTEC90-36 Defense’sOversight Process Executive Summary work indicates that OSD is not effectively using its authority when the circumstances indicate that such action is warranted. Principal Findings OSDHas Not Been Defense services and agencies have been criticized repeatedly by GAO, Effective in Resolving congressional committees, and others for developing systems that run over budget, behind schedule, and which do not operate as intended. Development Problems Recent reviews by GAO, the Defense Inspector General, and the MAISRC blame many of these deficiencies on the services’ failure to fully comply with established life cycle management policies and procedures. For example, a February 1989 Defense Inspector General report’ on information resources management within the three military services cited 240 instances of non-compliance with regulations. The report stated that the services encountered development problems because pro- gram managers did not follow established guidelines and regulations. In May 1989, GAO reported2 on eight Defense systems which had expe- rienced significant cost growth and schedule delays. GAO noted that ser- vice officials attributed the problems to underestimating the systems’ original costs, design failures, program redirection, and enhancements to the original project scope. In October 1989, OSDconcluded that the signif- icant cost growth and schedule delays were generally due to weaknesses in (1) executing life cycle management policies, (2) the quality of pro- gram management decisions, (3) the effectiveness of service oversight, and (4) the currency and accuracy of oversight information provided to OSD. OSDhas taken a number of actions intended to enhance life cycle man- agement policies and improve compliance by the services. For almost 10 years, OSD has emphasized the requirement that the services maintain their own review process. Although each of the services has imple- mented an oversight process, deficiencies persist in system development, particularly non-compliance with life cycle management policies and procedures. ‘Defenue Inspector General, Report (Report No. 89-03), February 17,1989. ‘Automated Information Systems:ScheduleDelaysand Costoverruns PlagueDODSystems(GAO/ IIciITFJc-89-36, May 10,1989). Page 3 GAO/lMTEGW36 Defense’sOversight Process Executlvb fhmmnry OSDHas Not Effectively The MAISRC'S reviews of system development efforts have generally Used Its Authority to raised a number of good questions and the MAISRC has directed the ser- vices to correct identified problems. At the same time, however, GAO'S Redirect or Terminate work shows that 06~ has not made the tough decisions to redirect or Systems terminate system development efforts when the circumstances indicate that it should. GAO's analysis of nine systems reviewed by the MAISRC during the last few years showed that MAISRC staff-including analysts in OSD'S Pro- gram Analysis and Evaluation and the Director of Operational Test and Evaluation offices-consistently identified deficiencies in major systems being developed by all three military services. Many of the deficiencies related to the services’ inappropriate life cycle management task execu- tion, and the services’ failure to carry out their oversight responsibili- ties. For example, the staff identified deficiencies such as unjustified costs and benefits, unacceptable test plans, inadequate analysis of potential risks, ineffective program management, and incomplete system design. The MAWC members were routinely made aware of these defi- ciencies and almost always brought the deficiencies to the services’ attention. GAO's analysis of the nine systems, as well as its own reviews of system development efforts, also shows that @SDhas allowed system develop- ment efforts to continue despite development problems or the services’ failure to comply with Defense policies. For example, OSDhas allowed system development to continue when the MAISRC identified problems in the services’ development approach, and when previous MAISRC guidance was not implemented. GAO recommends that the Secretary of Defense make greater use of the Recommendationsto authority to redirect or terminate system development efforts when the the Secretary of results of MAISRC reviews indicate that such action is warranted. To Defense accomplish this, the Secretary should direct the MAISRC to (1) withhold milestone approval and prohibit further development on any systems that do not comply with Defense policies and (2) ensure that its deci- sions are reflected in the services’ budgets. To achieve a long-term solution to the problem of the services’ non-com- pliance with Defense policies, the Secretary should direct the MAISRC to implement a separate procedure to periodically assess the adequacy of the services’ oversight processes, and recommend corrective action when it determines that the processes are deficient. Such a procedure Page 4 GAO/IMTJ3G9O36Defense’sOversight Process Jbcutive Summary would give the MAISRC a basis for assessing the risk of non-compliance in individual systems and for holding the services accountable for failing to implement ark acceptable oversight process. GAO did not request official agency comments on a draft of this report. Agency Comments However, GAO discussed the report’s findings with agency officials and incorporated their comments where appropriate. Page6 GA0/IMTEC9036Defenae’s OversightProcese Contents Executive Summary Chapter 1 Background Life Cycle Management Is Fundamental to Defense’s Oversight of Systems The MAISRC Is Defense’s Oversight Authority 9 Defense Is Reassessing the Acquisition and Oversight 11 Process Objectives, Scope, and Methodology 12 Chapter 2 14 Defense Is Not Systems Developed by the Services Have Contained Numerous Deficiencies 14 Effectively Controlling OSD Has Directed the Services to Improve Compliance 15 System Development and Oversight The MAISRC Continues to Identify Deficiencies and 16 Recommend Corrective Action OSD Has Not Effectively Used Its Authority to Redirect 18 or Terminate Systems Chapter 3 21 Conclusions and Recommendations to the Secretary of Defense 22 Recommendations Appendix Appendix I: Major Contributors to This Report 24 Related GAO Products 28 Figure Figure 1.1: Defense’s Life Cycle Management Phases and 9 Milestones Table Table 2.1: Deficiencies Identified by the MAISRC Support Staff Y Page 6 GAO/IMTEG!lO-36Defense’sOversight Process Abbreviations DOD Department of Defense GAO General Accounting Office IMTJX Information Management and Technology Division MAISRC Major Automated Information System Review Committee Y OSD Office of the Secretary of Defense DUNE Director of Operational Test and Evaluation PA&E Program Analysis and Evaluation Page7 Chapter 1 Background The Department of Defense spends about $9 billion annually on auto- mated information systems used to manage billions of dollars in logis- tics, personnel, and financial resources critical to its mission, Since the late 197Os, Defense has required a structured process called life cycle management for developing or modernizing major automated informa- tion systems. The process emphasizes the need to develop systems that will meet Defense requirements and stresses sound financial manage- ment and continuing mission evaluation. Life cycle management also requires Defense management to oversee systems and determine whether they should be continued, redirected, or terminated. The level of oversight required by life cycle management generally depends on a system’s cost-the greater the cost, the higher the level of ’ oversight, The Office of the Secretary of Defense (OSD) established the Major Automated Information System Review Committee (MAISRC) to oversee the development of systems when cost estimates exceed $25 million for 1 year, $100 million total, or when the system is of special interest. Life cycle management is intended to ensure that Defense management Life Cycle is accountable for the success or failure of systems. Life cycle manage- Management Is ment is also intended to give Defense program managers a structured Fundamental to approach for developing automated information systems. Defense’s guidelines for life cycle management define six development phases and Defense’sOversight of six decision points (called milestones) where system progress is assessed Systems and documented. Figure 1.1 shows the six life cycle management phases, the corresponding milestones, and the questions which must be affirma- tively answered before a system can proceed to its next development phase. Page 8 GAO/IMTJIC90-36 Defense’sOversight Process Chapter 1 Background Figure 1.l : Defense’s Life Cycle Management Phases and Milestones Phase 0 Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 A -l Time to modernize or replace? Does the How best system can this 3 work? be done? Milestones ” The decision to allow a system to proceed from one phase to the next is based, in part, on management’s analysis of system documentation. Throughout development, the program manager is expected to maintain documentation that demonstrates the level of analysis and planning put into the system, The documentation is expected to be kept current and updated any time a change has been approved. As Defense’s senior information resource management official, the The MAISRC Is Assistant Secretary of Defense (Comptroller) is the Chairperson of the Defense’sOversight MAISRC, and is responsible for conducting oversight reviews of major Authority automated information systems, In addition to the Comptroller, the MAISRC is made up of technical, functional, and system sponsor repre- sentatives such as Y l the Assistant Secretary of Defense for Command, Control, Communica- tions and Intelligence, who provides insight on communications and security issues; Page 9 GAO/IMTEG90-36 Defense’s Oversight Process Chapter 1 Background l the Assistant Secretary of Defense, Program Analysis and Evaluation (PA&E), who provides expert advice on program cost estimating; l the Director, Operational Test and Evaluation (DOTBtE), who provides expert advice on testing; l the OSD functional sponsor, who validates the requirements; and . a senior representative from the military service or Defense agency that is going to use the system. MAISRC technical and functional support staff analyze and give an opin- ion on the quality and depth of each system’s development plans. Prior to formal MAISRC oversight reviews, the support staff analyze a system’s status and prepare summary documents for the MAISRC members. For example, PA&E analysts assess the adequacy of the service’s cost/benefit and economic analyses and DOLYZEanalysts assess the service’s test plans and test results. During the oversight review, the MAISRC members determine, from an OSD perspective, whether the system is being developed in accordance with Defense policies, procedures, and regulations. Among other things, the MAISRC should analyze whether the service has taken steps to l reduce the time and cost to develop and field automated information systems by maximizing the use of commercial products; l avoid duplication and unnecessary expenditures on new systems by effectively using existing systems; l minimize the cost of new systems by ensuring full and open competition; and l effectively use advanced system design and software engineering tech- nology to minimize software development and maintenance costs. The MAISRC'S review is intended to determine whether development should be continued, redirected, or terminated. The MAISRC'S decision and any recommendations are documented in a System Decision Memo- randum. When necessary, the memorandum is supposed to provide spe- cific guidance to the military service on correcting problems. The military services are responsible for establishing their own similar oversight and milestone approval processes for systems being developed or modernized. Among other things, the services are supposed to ensure that their systems follow life cycle management principles and adhere to program schedule and cost goals. The systems are supposed to be reviewed by senior-level service officials even if an OSD MAISRC review is to be held. Page 10 GAO/IMTEC9636 Defense’s Oversight Process Chapter1 Background The MAISRC Has Defense tries to minimize the layers of oversight by permitting the MAISRC to delegate its review and milestone approval authority to the Delegated Oversight for services. The MAISRC normally maintains approval authority for the first SomeSystems to the two milestones, which focus on early decisions such as need justifica- Services tion, alternative concept analysis, and acquisition strategy. The MAISRC'S attention to these early milestones is intended to ensure that early deci- sions establish a solid foundation for later development. Before delegating approval authority for a system, the MAISRC is sup- posed to be assured that the service has an effective review and mile- stone approval process in place. At the time of our review, milestone approval for over half of Defense’s 52 major automated information sys- tems had been delegated to the services. When OSDdelegates oversight authority to a service, the MAISRC is expected to monitor the system’s progress to ensure its successful development is not jeopardized. The MAISRC has established the following criteria to determine whether dele- gation should be revoked: cost growth of 25 percent or more for the overall program, schedule slippage of 6 months or more, inadequate service oversight, or significant problems in the system’s development. In October 1989, the Deputy Secretary of Defense established the Defense Is Reassessing Department of Defense Corporate Information Management initiative. the Acquisition and This initiative is intended to ensure the standardization, quality, and Oversight Process consistency of data from Defense’s multiple management information systems, and to identify standard functional requirements for meeting Defense’s management information needs. Industry leaders and senior- level Defense officials will be asked to recommend an overall approach and action plan for a Corporate Information Management program, eval- uate the current oversight process, and recommend corrective actions. This group will also review the procedures of Defense groups, analyzing Defense’s various functional areas including financial management, civilian personnel, and materiel management. In addition, experts in Defense policy and information systems will develop concepts for improving Defense business practices. These experts will identify and develop standard functional requirements and data formats for each of Defense’s functional areas. Recognizing that one goal of the initiative is to develop standard, departmentwide sys- tems, Defense’s fiscal year 1991 appropriations request reflected a $288 Page 11 GAO/IMTEG96-36 Defense’s Oversight Process Chapter1 lhekgmund million reduction of the services’ and Defense agencies’ automated data processing budgets. In an effort to streamline Defense’s acquisition process, in October 1989 the Deputy Secretary of Defense designated the MAISRC as a committee of the Defense Acquisition Board. The Comptroller will continue to serve as the MAISRC chairperson, and life cycle management principles and processes for major automated information systems are to remain in effect. The MAISRC is to continue to review major systems, and telecom- munications programs, prior to Defense Acquisition Board meetings. The objective of this review was to determine whether Defense is effec- Objectives, Scope,and tively controlling the acquisition of major automated information sys- Methodology tems. This review was requested by the Chairman and Ranking Minority Member, Legislation and National Security Subcommittee, House Com- mittee on Government Operations. The May 18, 1989, request followed the Subcommittee’s hearing on cost growth and schedule delays incurred by Defense in its development of eight automated information systems. We assessed the process used by the MAISRC to oversee major automated information systems. This included an assessment of the input provided by the Deputy Assistant Secretary of Defense for Information Resources Management, the Assistant Secretary of Defense for Program Analysis and Evaluation, and the Director, Operational Test and Evaluation. We also identified the process used by the Air Force, Army, and Navy to oversee system development. We limited our review to the three military services, because their systems represent most of Defense’s automated data processing purchases. To gain a thorough understanding of Defense’s oversight process, we reviewed directives, instructions, and regulations pertaining to systems development and oversight. We also interviewed OSD and service staff who oversee the development of systems. For example, we interviewed the Deputy Assistant Secretary of Defense for Information Resources Management, and members of the Deputy’s staff; officials from PA&E and DOT&E who participate in MAISRC deliberations; staff representing each of the services’ senior information resources management officials; and program managers responsible for developing selected Air Force, Army, and Navy systems. Page 12 GAO/IMTEG90-36 Defense’s Oversight Process Chapter 1 Background To assess the results of the oversight process, we reviewed nine systems which had been subjected to 11 MAISRC reviews from April 1987 through July 1989. The systems were the Air Force Depot Maintenance Management Information System, Air Force Personnel Concept III Program, Army Integrated Procurement System, Army Civilian Personnel System, Army Supercomputer Program, Navy Stock Point ADP Replacement Project, Navy Engineering Data Management Information and Control System, Navy Integrated Disbursing and Accounting Financial Information Processing System, and Naval Aviation Logistics Command Management Information System. Before reviewing the systems, we discussed our selections with the Dep uty Assistant Secretary of Defense for Information Resources Manage- ment. The Deputy agreed that the systems we selected would provide an accurate illustration of the MAISRC review process. However, because our selection of the nine systems was not a random sample, our audit find- ings cannot be scientifically projected to the entire universe of major systems being developed by Defense. We conducted our audit work from June 1989 to March 1990, primarily within OSD and at the Departments of the Air Force, Army, and Navy headquarters at the Pentagon, Washington, D.C. As requested, we did not obtain official agency comments; however, we did discuss the con- tents of this report with OSDofficials, and have incorporated their com- ments where appropriate. Our work was performed in accordance with generally accepted government auditing standards. Page 13 GAO/IMTEGBO-36Defense’sOverdgkt Proceee Chapter 2 DefenseIs Not Effectively Controlling Development In recent years the Department of Defense has been criticized by a number of sources for its failure to deliver automated information sys- tems on time and within budget, which operate as intended. Many of the systems, problems have been caused by the military services’ failure to fully comply with regulations, policies, and procedures, including life cycle management requirements. Although OSD has taken actions intended to improve the services’ compliance with requirements and oversight of systems, problems still exist. Specifically, our assessment of selected oversight reviews conducted by the MAISRC within the last few years has shown that the services’ failure to fully comply with life cycle management continues to be a problem. We also found that OSD has not made the tough decisions to redirect or terminate system development efforts when the circumstances indicate that such actions may be warranted. Defense’s problems in developing major automated information systems Systems Developed by are well documented. Our work, reviews conducted by other audit orga- the Services Have nizations, and congressional inquiries have identified hundreds of Contained Numerous instances in which the services have not complied with regulations, poli- ties, and procedures when developing automated information systems. Deficiencies Specifically, systems experienced cost growth and schedule delays due to poor needs identification, inadequate investigation of alternative solutions, limited oversight, and incomplete justification of costs and benefits. A February 1989 report’ by the Defense Inspector General on informa- tion resources management concluded that the services were not rigor- ously complying with Defense’s life cycle management policies and procedures. The report cited 240 instances of non-compliance with established regulations, and noted that most of the instances occurred during the initial two life cycle management phases-mission analysis/ project initiation and concept development. The report also stated that the services encountered problems developing automated information systems because program managers did not follow established guide- lines and regulations, not because they lacked guidance. ‘Defense Inspector General, Report on Information Resources Management Within DOD (Report No. 89-INS-O3),February 17,1989. Page 14 GAO/IMTEG90-36 Defense’s Oversight Process . chapter2 Defense Is Not Effbctively Controlling System Development In a May 1989 report? on eight Defense systems, we noted that the esti- mate to develop and deploy the systems had almost doubled-from about $1 billion to about $2 billion. In addition, the development of two of the systems had been abandoned after spending about $237 million, and the completion dates for all but one of the remaining six systems had been delayed by 3 to 7 years. Moreover, all eight systems expe- rienced problems to varying degrees regardless of whether they had been reviewed by the MAISRC. Our report noted that service officials attributed the problems to underestimating the systems’ original costs, design failures, program redirection, and enhancements to the original project scope. . In response to congressional direction, OSD reviewed the history of the eight automated information systems to identify the causes of the prob- lems. It concluded,that the significant cost growth and schedule delays were generally due to weaknesses in (1) executing life cycle manage- ment policies, (2) the quality of program management decisions, (3) the effectiveness of service oversight, and (4) the currency and accuracy of oversight information provided to OSD. More specifically, OSDfound that development problems were caused by the services’ insufficient require- ments analysis, inaccurate cost and schedule estimates, inappropriate development approaches, limited assessment of alternatives, and unreli- able data provided to OSD. OSD has attempted to strengthen the services’ oversight of automated OSDHas Directed the information systems by emphasizing the services’ responsibility to Services to Improve establish their own review processes similar to the MAISRC. In August Compliance and 1981, for example, the Comptroller gave the services explicit guidance for monitoring system development in accordance with life cycle man- Oversight agement policies and procedures. The Comptroller directed each service to establish a senior management review group to function as a decen- tralized version of the MAISRC. The management review groups were directed to provide overall policy direction and reinforce both accounta- bility and primary management controls. In March 1986, as part of an effort to clarify and streamline automated information system development, OSDemphasized the importance of life cycle management and system oversight. Specifically, the Comptroller stated that each service should have an accountable, executive-level 2Aut.omated Information Systems: Schedule Delays and Cost Overruns Plague DOD Systems (GAO/ IMTEC-89-36,May 10,1989). Page 16 GAO/KM’IEG90-!36 Defense’s Oversight Process Chapter 2 Defense Is Not Effectively Controlling System Development review process based on the principles of life cycle management. The Comptroller also stated that effective service oversight processes are essential to achieve further streamlining by increasing the number of delegated systems. Although all three services have senior oversight groups to review auto- mated information systems, deficiencies in systems’ development activi- ties, particularly non-compliance with life cycle management policies and procedures, have persisted. As a result of OSD’S in-depth analysis of the eight systems on which we reported, the need to improve the ser- vices’ oversight of systems and compliance with established require- ments was again emphasized. In October 1989, OsD directed the services to initiate action to ensure that the underlying weaknesses in life cycle management task execution, program management, and service over- sight processes are corrected. Although the services submitted action plans in January 1990, OSD told the services that they were not specific enough and asked that they be improved. From April 1987 to July 1989, the MAISRC conducted 35 reviews of 22 The MAISRC systems being developed by the services and Defense agencies. We ana- Continues to Identify lyzed the process the MAISRC staff used to review nine of the systems Deficiencies and being developed by the services, in order to determine the kinds of defi- ciencies it identified, whether they were brought to the attention of the Recommend MAISRC members, and if corrective action was recommended. Corrective Action As shown in table 2.1, the MAISRC support staff identified one or more deficiencies in all but one of the systems. Page 16 GAO/IM’IEC9O-26 Defense’s Oversight Process Chapter2 Defense Is Not Eiyectively Cbntrolling System Development Table 2.1: Deficlencles Identified by the MAISRC Support Staff Deficiencies Identified Configuration! Risk Program capacity Development System Testing Cost/benefit management management management strategy .._._...._ _ _.-~ .._._ .._.. -..--..--- ----- Air Force Depot Maintenance Management Information System X X Air Force Personnel Concept Ill Program X --- A;mylntegrated Procurement X System --___-X ___.. Army &han Personnel System X X X __- Ar&‘Suoercomwter Proaram Navy Stock Point ADP Replacement Project X X - Navy Englneenng Data Management lnformatlon and Control System X __.__II_._-......_ Navy Integrated Disbursing and Accounhq Flnanclal lnformatlon Processing Svstem X X X X X X Naval Awation Logistics Command Management lnformatlon System X X X Many of the identified deficiencies occurred because the services were not strictly following life cycle management policies and procedures. These deficiencies were almost always reflected in the System Decision Memoranda provided to the services after the MAISRC reviews. We also found that PA&E and DOI%@consistently provided input to the oversight reviews of the nine systems. For seven of the nine systems, the MAISRC support staff noted deficien- cies in the test plans submitted by the services. For example, in prepar- ing for a June 1988 milestone I review of the Army’s Integrated Procurement System, DOME determined that the Army needed to develop a test and evaluation master plan for the system. Although the MAISRC granted milestone approval, its System Decision Memorandum directed the Army to submit a test and evaluation plan within 6 months. According to a DCYIXEanalyst, the test plan was submitted in September Y 1989-15 months after the milestone review-and was subsequently approved by DOME. Page 17 GAO/lMTECXO-36 Defense’s Overnight Process ‘8 ,!! ” “,# chapter2 Defense Is Not Effectively Controlling System Development Similarly, for four of the nine systems, the MAISRC support staff noted deficiencies in the services’ estimates of system costs and benefits. For example, PA&E found that the Navy submitted an incomplete indepen- dent cost estimate and benefit analysis for the Integrated Disbursing and Accounting Financial Information Processing System. PA&E noted that the Navy, in estimating the system’s cost, did not include all sunk costs, costs for government-furnished equipment and services, and oper- ator expenses, The MAISRC System Decision Memorandum included a requirement that the Navy obtain PA&E validation of the system’s life cycle cost estimate, independent cost estimate, and program benefits estimate prior to milestone approval. Concerns about the adequacy of the services’ risk analysis, program management, configuration and capacity management, or overall devel- opment strategy were also raised by the support staff for five of the nine systems we reviewed. For example, in preparing for a June 1989 review of the Air Force Depot Maintenance Management Information System, the MAISRC staff noted that the Air Force did not appear to have an active risk management program that would identify risks, develop plans to address those risks, and closely monitor the program for indica- tions that problems are occurring. In its System Decision Memorandum for this system, the MAISRC deferred approval of milestone II. The memorandum directed the Air Force to develop and document a risk management program that would consider risks related to cost, schedule, and technical implementation, According to an OSDofficial, the Air Force submitted its plans to address risk in October 1989, and the plans will be assessed before the next milestone II review. While the MAISRC reviews we assessed have found deficiencies and rec- OSDHas Not ommended corrective actions, our work also shows that other problems Effectively Used Its have not been addressed. Specifically, the MAISRC did not take corrective Authority to Redirect action to ensure the services complied with its previous guidance, and failed to redirect or recommend terminating systems that had questiona- or Terminate Systems ble acquisition strategies. For example, during the fiscal year 1986 appropriations process, the Conference Committee on Appropriations expressed concern about the Y progress of the Naval Aviation Logistics Command Management Infor- mation System and directed OSDto conduct a MAISRC review. At that time, the system had been in development 9 years, full deployment was Page 18 GAO/IMTEC%O36 Defense’s Oversight Process Chapter2 Defense Is Not Effectively Controlling System Development 4 years behind schedule, and oversight authority for the system was delegated to the Navy. The MAISRC reviewed the system in July 1986, and expressed concern that the Navy’s plans to complete deployment of the system in 1996 would delay operational benefits. As a result, the MAERC directed the Navy to complete deployment of the system by 1993. It also allowed the Navy to retain direct oversight responsibility for the system. In a July 1989 review, the MAISRC staff noted that the Navy had not followed the 1986 guidance, that the estimate for full deployment had slipped to 1999, and that the Navy had spent $233 million on the sys- tem. Nevertheless, the MAISRC granted conditional milestone approval for limited deployment of the system. The System Decision Memorandum directed the Navy to submit another assessment of alternatives for speeding up full deployment. During a recent review of the Naval Aviation Logistics Command Man- agement Information System,3 we identified concerns about equipment acquisition and the sufficiency of stress testing and operational testing. During the fiscal year 1990 appropriations process, the Congress stated that it expects the MAISRC to ensure that the Navy addresses these con- cerns and achieves deployment of the system by 1996 as promised to the Congress. During a recent review of an Air Force system, the MAISRC failed to iden- tify a number of development problems. Following a June 1989 mile- stone II review of the Air Force Personnel Concept III system, the MAISRC commended the Air Force for having done an excellent job planning and executing the program. The MAISRC delegated future milestone approval and oversight authority for the system to the Air Force, and the System Decision Memorandum provided minimal guidance. However, in a separate report4 on this $660 million personnel system, we found that the Air Force planned to deploy the system to 126 bases even though it had not (1) fully developed and tested the system, (2) fully analyzed requirements or alternatives for the hardware design, or (3) adequately supported personnel savings. As a result of our work, the Congress withheld further funding for the system until the Air Force ter Acquisition: Navy’s Aviation Logistics System Not Ready for Deployment (GAO/ 90-11, Feb. 9, 1990). “Air Force ADP: The Personnel Concept III System Is Not Ready for Deployment (GAO/ Im-90-22, Feb. 27,199O). Page 19 GAO/IMTJXCXM%M Defense’s Oversight Process Chapter2 Defense Is Not JWfectively Controlling System Development demonstrates that the proposed hardware configuration is the most cost-effective alternative. As a result of its review of the Navy’s Integrated Disbursing and Accounting Financial Information Processing System, the MAISRC con- cluded that the system’s overall program planning and design were not sufficient to grant milestone approval. However, despite the system’s history of development failures, cost growth, and schedule delays, the MAISRC did not use its authority to recommend terminating the system. When the MAISRC reviewed this system for the first time in February 1989, the system had been in development for more than 10 years, and the Navy had spent about $94 million of the system’s estimated $591 million life cycle cost. The MAISRC found that l the overall program planning and design of the system were not com- plete and did not meet life cycle management policy requirements, l substantial development efforts had been implemented without the com- pleted system design and without required senior Navy management oversight reviews, l the development methodology, after two prior failed attempts, was unorthodox, and l test plans were incomplete. The MAISRC directed the Navy to take specific action to correct the numerous deficiencies identified during the review before further devel- opment of the system could proceed. For example, the Navy was directed to complete milestone II planning and analysis, complete the system’s design efforts, and evaluate the risks associated with the unor- thodox development approach. The corrective action was to be reported to the MAISRC within 6 months. However, the Congress subsequently denied fiscal year 1990 funds for this Navy system. Page 20 GAO/IMTEC%O-36 Defense’s Oversight Process Chapter 3 Conclusions md Recommendations Our review has shown that OSDis not effectively enforcing Defense poli- cies for controlling the acquisition of major automated information sys- tems. Problems exist in two areas. First, OSDhas not been effective in getting the services to develop automated information systems that com- ply with the requirements of life cycle management. Secondly, OSD has not terminated or redirected system development efforts when the ser- vices have not complied with life cycle management or other Defense I policies. Defense policies and procedures for automated information system development call for thorough and effective oversight commensurate with the anticipated investment. The military services are responsible for managing automated information system development, and they are required to provide effective oversight even when the systems will be reviewed by the MAISRC. MAISRC oversight is not intended to be a substitute. Yet we found that OSD'S oversight process is not holding the services accountable for compliance with Defense policies. It seems to us that OSD'S authority to redirect or terminate systems when the hws~c recom- mends withholding milestone approval may be the most effective tool OSD has to hold the services accountable for complying with Defense pol- icies. Our work indicates, however, that rather than having the MAISRC withhold approval and reduce the services’ budget requests for individ- ual systems, OSDhas tried to fix the problem by directing the services to correct non-compliance deficiencies on individual systems, and empha- sizing the services’ oversight responsibilities. It is interesting to note that OSDrecently used the budget process to push the services toward working with each other to develop standard sys- tems. As part of its Corporate Information Management initiative, OSD reduced each of the services’ fiscal year 1991 budget requests to reflect anticipated savings from the initiative. We believe that the repeated identification of life cycle management deficiencies by the MAISRC indicates that OSDneeds to be more aggressive in holding the services accountable for complying with Defense policies. We do not believe that OSDshould rely on the MAISRC to routinely identify the services’ non-compliance deficiencies during individual system reviews. Moreover, OSD should not be funding systems that are not in full compliance with established policies. Page21 GAO/IMTJZG9O-%Jhfense'aOveraightProceas Chapter3 Conclusions and Recommendations , Some of the systems covered in this review, as well as others we recently reviewed, also indicate that OSDis not making the tough deci- sions to terminate or redirect development efforts when they need to be made. For example, the MAISRC has allowed systems to continue even though its previous guidance had not been implemented in a timely man- ner or when it has identified concerns about the adequacy of the ser- vice’s development approach. On a number of occasions, the Congress has stepped in and provided specific guidance or denied funding when it has found that the MAISRC has allowed a system to proceed despite non- compliance or other development problems. In May 1989, we recommended that the Secretary of Defense review and Recommendationsto revise, as appropriate, the management control and decision-making the Secretary of process for major automated information systems development. We Defense understand that the Secretary of Defense is reviewing the role of the MABRC and that any necessary changes will be made based upon the out- come of the review. We support the MAISRC oversight process and recom- mend that the Secretary of Defense be more rigorous in enforcing policies for controlling the acquisition of major automated information systems. Specifically, we recommend that the Secretary of Defense make greater use of the authority to redirect or terminate system development efforts when the results of MAISRC reviews indicate that such action is war- ranted. To accomplish this, the Secretary should direct the MAISRC to withhold milestone approval authority and prohibit further develop- ment when the services submit a system for review that does not com- ply with Defense policies. The Secretary should also direct the MAISRC to ensure that its decisions are reflected in the services’ budgets. In order to achieve a long-term solution to the problem of non-compli- ance, the Secretary should direct the MAISRC to implement a separate procedure to periodically assess the adequacy of the services’ oversight processes, and recommend corrective action when it determines that the processes are deficient. Such a procedure would give the MAISRC a basis for assessing the risk of non-compliance in individual systems and for holding the services accountable for failing to implement acceptable oversight. Page 22 GAO/IMTEG36-36 Defense’s Oversight Process l Y Page 23 GAO/IMTEG90-36 Defense’s Oversight Process Appendix I Major Contributors to This Report Thomas J. Howard, Assistant Director Information Kenneth W. Huber, Evaluator-in-Charge Management and Gwendolyn Dittmer, Evaluator Sabine Richard, Evaluator Technology Division, Washington, D.C. Page 2i GAO/IMTEG9036 Defense’s Oversight Process ..I. - . Page 25 GAO/lMTEG90-26 Defense’s oversight Process Y Page 26 GAO/IMlEC9O-36 Defense’s Overdght Process Y Page 27 GAO/IMTEG90-36 Defense’s Oversight Process Related GAO ProdWts Schedule Delays and Cost Overruns Plague DOD Automated Information Systems (GAO/T-IMTEC-89-8, May 18,198Q). Automated Information Systems: Schedule Delays and Cost Overruns Plague DOD Systems (GAO/IMTEC-89-36, May 10,198Q). ADP Acquisition: Air Force Logistics Modernization Projects (GAO/ IMTEC-89-42, Apr. 21,198Q). ADP Acquisition: Defense Logistics Services Center Modernization Pro- gram (GAO/IMTEC-8932, Mar. 20,198Q). ADP Acquisition: Army Civilian Personnel System (GAO/IMTEC-89-22FS, Mar. 3,198Q). ADP Acquisition: Naval Aviation Logistics Command Management Information System (GAO~MTFC-SQ-WS, Feb. 23, 1989). ADP Acquisition: Navy’s Efforts to Develop an Integrated Dispersing and Accounting System (GAO/IMTECSQ-20FS, Feb. 8, 1989). Computer Procurement: Decision Needed on Navy’s Standard Auto- mated Financial SyS&!m(GAO/IMTEC-8%47,Sept. 13, 1988). ADP Systems: Army Decision to Use Air Force Military Pay System Appears Advantageous (GAO~IMTEC-89-28,Mar. 1, 1989). Computer Systems: Navy Needs to Assess Less Costly Ways to Imple- ment Its Stock Point System (GAOIIMTEC-89-2, Dec. 14, 1988). Computer Procurement: Navy CAD/CAM Acquisition Has Merit but Man- agement Improvements Needed (GA~/IMTEC-~~-~~, May 11, 1988). ADP Procurement: Army Accounting System Modernization (GAO/ IMTEC-ss-14B~, Dec. 28, 1987). Computer Systems: Navy Stock Point ADP Replacement Program Needs Better Management Controls (GAO/IMTJZC-W-30,Sept. 17,1987). (51044@) Page 28 GAO/IMTEG903t3 Defense’s Oversight Process 1~c~c~uc~s1.sfor twpiw of’(;izo rt~por1.s sl~otiltl 1~ stbllt to: ‘I’t~ltq)llolIt~ 202-275-624 1 ‘1’1~ first five twpiw of t~atcll report. artA free, Adtlit,itmal twpiw art* $2.00 twh .
Automated Information Systems: Defense's Oversight Process Should Be Improved
Published by the Government Accountability Office on 1990-04-16.
Below is a raw (and likely hideous) rendition of the original report. (PDF)