: United States General Accounting Office GAO Report to the Chairman, Subcommittee on Readiness, Committee on Armed . Services, House of Representatives May 1996 AIR FORCE ADP Depot Maintenance System Development Risks’Are High GAO/IMTEC-9046 GAO United States General Accounting Office Washington, D.C. 20548 Information Management and Technology Division B-220195.9 May 25,199O The Honorable Earl Hutto Chairman, Subcommittee on Readiness Committee on Armed Services House of Representatives Dear Mr. Chairman: In response to your concerns about development problems, rising cost, and schedule delays, we reviewed the Air Force Logistics Command’s attempt to develop the Depot Maintenance Management Information System (DMMIS). This new automated system is intended to modernize and improve data processing support at the Command’s six maintenance centers. In 1984, the Command estimated the system would cost about $85 million and be fully operational in February 1989. The Command now estimates it will cost $242.4 million and be fully operational in Sep- tember 1993. To develop DMMIS, the Command plans to purchase hardware and adapt commercial, off-the-shelf software called Manufacturing Resource Plan- ning (MRP II)’ for each of 19 product divisions at its six centers. Our spe- cific objectives were to determine if (1) the adaptation of MRP II software will meet the needs of the Command’s depot operations, (2) the acquisi- tion cost estimates are reasonable, and (3) the development schedule can be met. Appendix I provides details on our objectives, scope, and meth- odology. Appendix II provides additional information on Air Force depot maintenance operations and the 19 product divisions at the six depot maintenance centers. software will be more difficult to adapt to the depot environment Results in Brief MRP II than the Command had expected. Originally expected to meet 90-95 per- cent of DMMIS requirements, the Command currently estimates that MRP II will meet only 65 to 70 percent of the system requirements-however, one contractor study concluded that 51 percent is more realistic. This means that the Command must significantly change the MRP II software, which increases the risks of not staying on schedule and within budget. ‘MRP II IS a widely accepted industry approach to integrated manufacturing planrung and control. It includes software. procedures. and controls to coordinate. manage, and operate a complex manufac- tunng or repair facility by following a single master production plan. Page 1 GAO/IMTJZSO46 Air Force Depot Maintenance System B-220196.9 Further adding to risks is the amount of work that could be needed to prepare depots for DMMIS. At one location, over 4 years and 184,000 staff hours were needed primarily to clean up the existing data bases that DMMIS will use, and the Air Force expects a similar effort at each of the other five locations. In June 1989, the Command recognized that without a change in devel- opment strategy, DMMIS could not be completed for $242.4 million and by September 1993. As a result, the Command is currently negotiating changes with Grumman Data Systems, Incorporated, the primary con- tractor, that will significantly alter the DMMIS scope and implementation approach. Under the revised approach, the Command no longer plans to complete a prototype at 1 of the 19 product divisions before beginning work at other divisions. Instead it will begin adapting the MRP II software at five product divisions simultaneously. Because of the significant modifications to MRP II software that are nec- essary, the new approach adds risk to an already risky program. By waiting until February 1991-the scheduled completion date of the pro- totype-the Command would know whether the MRP II concept will meet its needs for the DMMIS program. Considering the relatively short time until the prototype is to be completed, we believe the Command should complete the DMMIS prototype before beginning development efforts at other product divisions. depot-level maintenance to keep Air Force units and weapons systems in a state of readiness. The Command relies on computer technology to provide the enormous amount of information needed to accomplish its mission. Many of the Command’s computer systems originated in the 1950s and 1960s and, like other systems that date back to this era, have not kept pace with advances in computer technology. In 1984, the Command began the DMMIS program to improve the overall efficiency and effectiveness of its depot maintenance operations. It expects DMMIS to provide repair depots with on-line capability to fore- cast work loads; schedule repair activities; track and control inventories; program manpower, materials, and other resources; and track and man- age production costs, Currently, these activities are done either manu- ally or by more than 50 existing systems, 29 of which the Command plans to replace with DMMIS. (To do all this, the DMMIS system will include about 2.5 million lines of code.) Page 2 GAO/IMTJZG!#O46 Air Force Depot Maintenance System ~- .-- B-220195.9 This is an enormous undertaking considering the Command plans to implement DMMIS at all five of its Air Logistics Centers and the Aero- space Guidance and Metrology Center. The Air Force Logistics Com- mand operates a depot maintenance industrial complex with annual work valued at $2.5 billion. This maintenance community employs over 39,000 persons, and has invested $4.5 billion in facilities and equipment. Air Force depot maintenance is responsible for the repair and rebuilding of 1,200 aircraft, 1,200 missiles, 6,400 engines, and 1.1 million other repairable items. Compounding the number of items for which the depots are responsible is the fact that many repairs are extremely labor intensive. For example, the repair of one aircraft, such as the B-52, can require 50,000 hours. Annually, over 44 million hours are expended by the depots to perform maintenance. The depot maintenance process can be very complex. For example, a sin- gle engine can comprise 1,800 parts or subcomponents, require 12 departments to become involved in its maintenance, spread out over five to six subassembly areas before the engine parts are sent to a final assembly area. Periodically, each engine is completely dismantled and every part examined, See appendix II for further information on the depot maintenance process. The original DMMIS acquisition strategy called for development of three distinct systems-production, financial, and resource management. Because of technological difficulties and high risks in integrating these three systems, the Command abandoned this strategy in 1985, and later restructured the DMMIS development strategy into a single system design using commercial MRP II software. Recognizing the complexities of the proposed system and the depot maintenance environment, the Command plans to develop DMMIS in three phases. l Phase I involved developing an interim system called the Exchangeable Production System to improve controls over new parts managed at the Command’s Air Logistic Centers. m Phase II called for the completion of a system prototype at the Ogden Air Logistics Center Industrial Products and Landing Gear Division, 1 of 19 product divisions scheduled for DMMIS automation. The Command intended to test and evaluate DMMIS in an operational environment before implementing it at the other product divisions. Phase III involved full-scale development and system implementation. l Page 3 GAO/lMTEG9046 Air Force Depot Maintenance System B-220195.9 The Command completed Phase I implementation at all five Air Logis- tics Centers in July 1988 at an estimated cost of $28.1 million. In Janu- ary 1988, the Command awarded an $84 million contract to Grumman Data Systems to design, develop, implement, and maintain phases II and III of DMMIS. Other program costs-totaling over $130 million-will include (1) software and hardware maintenance for phase I, (2) com- puter terminals and other hardware used by maintenance workers on the shop floor, (3) contractor technical support to help the Command manage the program and oversee system testing, and (4) several other miscellaneous items. According to the Command, successful completion of the DMMIS program could save as much as $706 million over the 8-year useful life of the system. Most of these benefits are attributable to improvements in pro- ductivity and reduced inventory costs. DMMIS Is a High Risk government, consider DMMIS a very complex and high-risk undertaking. Program The Command’s ability to adapt MRP II software to meet DMMIS require- ments is clearly the most critical issue yet to be resolved. Other issues affecting the program’s success include the size and complexity of the program development effort and the lack of management continuity. MRP II May Not Meet As discussed earlier, in designing the DMMIS development strategy in 1984, the Command intended to develop three separate systems which DMMIS Requirements would interface with one another. When this proved technically infeasi- ble, the Command hired Deloitte Haskins & Sells to evaluate alternatives and recommend a workable solution. In its May 1985 report, Deloitte recommended that the Command use a commercially available software package-called MRP n-in lieu of developing a completely new system. Deloitte estimated MRP II would satisfy from 90-95 percent of DMMIS requirements. A subsequent Air Force Audit Agency assessment of Deloitte’s projections found that more realistically, MRP II would satisfy about 65-70 percent of DMMIS requirements. In January 1988, the Com- mand awarded the DMMIS development contract to Grumman Data Sys- tems, Incorporated, expecting that 65-70 percent of the requirements would be met by MRP II. In May 1988, Entek, Inc., (a contractor hired by the Command to pro- vide technical support) reported that the MRP II software would meet only 5 1 percent of DMMIS requirements. Consequently, Entek concluded Page 4 GAO/IhlTEG90-46 Air Force Depot Maintenance System B220196.9 that more extensive modifications-as well as a significant amount of newly developed software-were necessary to meet DMMIS requirements. Although these would probably raise costs and delay the schedule, neither the Command nor Grumman tried to make new estimates. About the same time, Synergy, Incorporated, (another independent con- tractor) assessed the potential operational impacts of DMMIS on aircraft readiness. Synergy concluded that DMMIS could, if properly developed and implemented, increase the number of aircraft available during both peace and wartime situations. Synergy raised concerns, however, about adapting MRP II to the complex depot repair environment: MRP II had worked in manufacturing processes (where the tasks are more predict- able and routine), but adapting it to the depot repair environment was risky. Synergy had surveyed several private sector companies and found limited success in modifying MRP II to repair or overhaul-type operations. However, the Command believed adapting MRP II would be easier than starting from scratch. Additional Risks Could The time and effort needed to prepare each location for DMMIS adds to the risk that the program may not be completed on schedule. Site prepa- Affect, Cost and Schedule ration activities? for the first implementation site in Ogden took 4 years and 184,000 hours to complete. While it is difficult to judge how long it will take to prepare the other locations, the Command expects a similar level of effort will be needed. Another concern was raised by the Air Force’s Cost Analysis Improvement Group, which concluded that to meet the September 1993 target completion date, DMMIS would likely need an additional $32.4 million to cover site-unique requirements not previously identified. The Secretary of Defense’s Major Automated Information System Review Committee found that DMMIS risk management needed strength- ening. In June 1989, this committee directed the Command to develop a plan to identify and address DMMIS developmental risks. According to that plan, using MRP II software is a high-risk issue involving cost and schedule estimates and the success of DMMIS itself. The Command originally underestimated DMMIS complexity. The Vice Commander for the Logistics Management Systems Center told us that ‘Site preparation activities include taking a physical mventorv of all parts and materials, cleaning up existing data bases that will be integrated into DMMIS, gather&g new data that will be entered into DMMIS. and mstalling power lines. Page 5 GAO/IMTEG904 Air Force Depot Maintenance System B220196.9 the DMMIS program has changed so dramatically since its inception in 1984 that it no longer resembles the program initially approved by the Air Force. From time to time, the Command reduced the program’s scope to stay within cost and schedule estimates. Among other things, the Command has reduced the number of systems DMMIS will replace from 43 to 29 and decided to prototype the system at one rather than two product divisions. These changes as well as the program’s cost and schedule growth prompted the Major Automated System Review Com- mittee to assume oversight of the program in September 1988. Previ- ously, the Air Force’s Automated Information System Acquisition Review Council had that responsibility. Lack of Management The lack of management continuity has also increased the program’s development risks. The Command has changed program managers three Continuity Increases times since the DMMIS program began in 1984. In addition, the current DMMIS Development Risks program manager, who arrived in June 1989, anticipates retiring and leaving the program in June 1991. Without trying to determine why these changes occurred or the specific impact each had on DMMIS'S devel- opment, this rapid turnover of top management is not conducive to pro- gram stability, particularly for a program of this size and complexity. According to a recent General Services Administration study of 18 fed- eral computer modernization programs, the program manager’s ability was the single most important factor in successful system development efforts. While we are not questioning the ability of the various DMMIS program managers, the lack of managerial continuity, in our view, clearly puts DMMIS at risk. In June 1989, the Command revised its development strategy for DMMIS The Command Has to stay within cost and schedule estimates. It adopted a Deloitte Haskins Revised Development & Sells recommendation to implement a technique-called Conference Strategy but Risks Room Pilot-which allows the system users to work directly with the system development contractor to identify requirements unique to their Remain working environment, develop software, and resolve potential operating problems. This early and direct involvement of the users, in theory, allows developers to compress the development schedule. Another part of the Command’s revised strategy, however, may actually increase risks. ~Previously, the Command intended to complete installa- tion and testing of DMMIS for one product division-the Industrial Prod- ucts and Landing Gear Division-at the Ogden site (currently scheduled for February 1991) before beginning work at additional sites. But now it Page 6 GAO/IMTJ3G9O46 Air Force Depot Maintenance System ‘\\‘\\ B-220196.9 has begun work at two other product divisions at the Ogden site and plans to begin system development and installation at two other loca- tions in July 1990. Therefore, the Command will be developing DMMIS fo five product divisions at three locations at the same time, before the system has been tested and proven to work anywhere. Recognizing the risk entailed by starting work at other product divisions before the Ogden prototype is completed, DMMIS'S program manager believes this strategy is necessary if the Command is to finish within current cost and schedule estimates. Even with this revised structure, however, recent information from the DMMIS program office shows that the schedule is beginning to slip. Under the restructured approach, for example, development work at the two engine divisions was scheduled to start in February 1990. The latest program office informal estimate is July 1990. Nevertheless, the Com- mand believes it can still meet its overall September 1993 target comple- tion date. Clearly, DMMIS is one of the largest and most complicated development Conclusions and efforts undertaken by the Air Force Logistics Command. This in itself is Recommendations enough to make the program a high-risk venture. There are three other issues, however, which tend to increase the developmental risks. First, the MRP II software will now require more extensive software changes than originally planned, and until the Command completes the proto- type system in February 1991 I it is uncertain if MRP II will effectively work in the depot environment. Yet, under the Command’s recent pro- gram restructure, it plans to begin adapting the MRP II software at two other product divisions before the prototype system is complete. This, we believe, is adding an unnecessary risk to an already risky program. Second, a huge effort may be needed to prepare each location for DMMIS. Since site preparation activities, which mainly involve cleaning up the data bases, took over 4 years to complete at the first location, a similar effort at the other locations could have an impact on the system schedule. Third, since its inception, the DMMIS program has had four program man- agers. In addition, the current program manager, who has been with the program since June 1989, may be leaving next year. This lack of man- agement continuity, in our view, also adds risk to the program. Page7 - E220195.9 To address these issues, we recommend that the Secretary of Defense direct the Secretary of the Air Force to: l complete the prototype system at the Ogden Industrial Products and Landing Gear Division before committing resources to develop DMMIS at other product divisions, l restructure the DMMIS schedule and cost estimates to reflect this change in development strategy, and l ensure management continuity for the DMMIS program. In accordance with your wishes, we did not obtain official agency com- ments on this report. We did, however, discuss its contents with Depart- ment of Defense and Air Force officials and have included their comments where appropriate. In a recent meeting, the program manager told us he concurs in principle with our recommendations and will reconsider his position on the concurrent system development at the two engine divisions. As agreed with your office, unless you publicly announce the report’s contents earlier, we plan no further distribution of it until 30 days from the date of this letter. At that time, we will provide copies of this report to the Secretary of Defense and the Secretary of the Air Force. We will also make copies available to other interested parties upon request. This work was performed under the direction of Samuel W. Bowlin, Director, Defense and Security Information Systems, who can be reached at (202) 275-4649. Other major contributors are listed in appendix III. Sincerely yours, Ralph V. Carlone Assistant Comptroller General Page 8 GAO/IMTJ3G90-46 Air Force Depot Maintenance System Page 9 GAO/lMTEC9046 Air Force Depot Maintenance System Contents Letter Appendix I Objectives, Scope, and Methodology Appendix II 14 An Overview of Depot Maintenance Appendix III 16 Major Contributors to This Report Table II. 1: Location of Air Force’s 19 Product Divisions 14 Abbreviations ADP automated data processing DMMIS Depot Maintenance Management Information System GAO General Accounting Office IMTEC Information Management and Technology Division MRPII Manufacturing Resource Planning Page 10 GAO/lMTJ3GSO-46 Air Force Depot Maintenance System Page 11 GAO/lMTEG90-46 Air Force Depot Maintenance System Appendix I Objectives, Scope, and Methodology Concerns over past system development problems and significant cost and schedule growth prompted the Chairman, Subcommittee on Readi- ness, House Committee on Armed Services, to ask us to review the sta- tus of the Air Force’s Depot Maintenance Management Information System (DMMIS). After further discussions with his office, we agreed to assess whether the DMMIS development approach will result in a system that meets requirements within the expected cost and schedule. Specifi- cally, our objectives were to determine whether (1) the commercial MRP II software will satisfy DMMIS functional requirements, (2) the acquisition cost estimates are reasonable, and (3) the development schedule can be met. To address the first objective, we focused on the (1) feasibility of apply- ing commercial MRP II software in a repair environment, and (2) method- ology used by the development contractor and the Air Force Audit Agency to estimate the expected effectiveness of the MRP II software in meeting DMMIS requirements. We obtained and reviewed industry trade journals and published periodicals on MRP II software. We analyzed sup- porting Command and contractor documentation as well as independent assessments. To understand how well MRP II software works in a repair environment, we spoke with private industry and Defense officials who have worked with or are now implementing an MKP II system. We also visited the DMMIS test site located at the Air Logistic Center at Ogden, Utah, and spoke with system users on the pros and cons of implement- ing MRP II in a repair environment and the level of pre-implementation activities required. To determine whether the approved DMMIS acquisition cost and schedule estimates can reasonably be met, we updated the information presented in a prior GAO report] on cost and schedule growth. We obtained and analyzed budgetary and contract cost data, reviewed current project sta- tus reports and pertinent program documents such as the program office estimate, cost/benefit analysis, and program decision papers. We also tracked key program milestones and spoke with Command and program officials regarding the accuracy and reasonableness of these milestones. Our review was conducted from January 1989 to February 1990, prima- rily at the DMMIS Program Management Office in Fairborn, Ohio; the Directorate of Maintenance at the Air Force Logistics Command at ‘Air Force ADP: Logistics Systems Modemizatlon Costs Continue to Increase IGAO/IMTEC-89-7I3, Dec. 28. 1988). Page 12 GAO/IMTFX-90-46 Air Force Depot Maintenance System Appendix I Objectives, Scope, and Methodology Wright-Patterson Air Force Base, Ohio; and the DMMIS test site at the Air Logistics Center in Ogden, Utah. In accordance with the requester’s wishes, we did not obtain official agency comments on this report. However, we discussed the contents with Department of Defense and Air Force officials and have included their comments where appropriate. We performed our work in accor- dance with generally accepted government auditing standards. Page 13 GAO/IMTEG90-46 Air Force Depot Maintenance System An Overview of Depot Maintenance The Air Force Depot Maintenance community consists of 19 product divisions that perform repair, modifications, and manufacturing in sup- port of Air Force weapons systems. Each division specializes in a certain type of work. These product divisions are located at the five Air Logis- tics Centers] and the Aerospace Guidance and Metrology Center, Ohio. Each operating site has three product divisions, except for one, which has four. Each product division is comprised of a Production Branch, Engineering and Planning Branch, and Scheduling and Inventory Con- trol Branch which makes them, to a great extent, independent. Each product division operates independently, and can be considered a sepa- rate business enterprise. Product divisions located at the same centers perform work for each other in a manner similar to that of a manufacturer who sends work out to a specialty shop. When such work is completed, it is returned to the originating division for assembly into a finished product. Table II. 1 below shows the 19 different product divisions and the center where they are located. Table 11.1:Location of Air Force’s 19 Product Divisions Center Product division Aerospace Gurdance and Metrology Center 1 Aircraft products 2. Mrssrle products 3. Support equipment Oklahoma 0ty Air Logtstrcs Center 4. Aircraft (B-52, E-3) 5. Propulsion 6 Accessories Ogden Air Logrstrcs Center 7 Arrcraft (F-16, F-4) 8 Mwle and aircraft systems 9 Industrial products and landtng gear San Antonio Atr Logisttcs Center IO Aircraft (C5A, C-130) 11 Engines 12 Technology repair Sacramento Arr Logrstlcs Center 13. Aircraft (A-10, F-l 11) 14 Flight instruments and pneudraulics 15 Communtcatrons and electronics 16. Industrial products and electronrc components Warner Robrns Arr Logrstrcs Center 17 Aircraft (F-15, C-130) 18. Airborne electronics 19 Industrial products ‘These centers are located at Hill Air Force Base. Utah; Tinker Aw Force I&e. Oklahoma; McClellan Air Force Base. Califonua: Kelly Air Force Base, Texas; and Robins .4ir Force Base, Georgia. Page 14 GAO/IMTEG9046 Air Force Depot Maintenance System Appendix II An Overview of Depot Maintenance There are two basic categories of work at the depots-permanent and temporary. Permanent work involves repetitive maintenance require- ments that are anticipated in advance of actual accomplishment. Having been anticipated, they are managed on a programmed basis. Permanent work loads include aircraft, missiles, and engines. In contrast, tempo- rary work loads have the characteristics of generating spontaneously and being non-repetitive, such as the manufacturing of new products. For permanent work loads, the customers provide work load projections covering several years. Prior to the beginning of each fiscal year, air- craft, engines, and other items are negotiated for the year. During the negotiations, there is a need to project the work load against available resources (i.e., staffing, equipment, facilities, materials). There are sev- eral work loads that are negotiated and managed on other than a yearly basis. For example, depot-level aircraft engine maintenance facilities only exist at San Antonio and Oklahoma City Air Logistics Centers. Engine work is negotiated on a quarterly basis and includes several repair concepts. One of these involves conplete engine disassembly, inspection of all components, routing of repairable components to the appropriate repair shop, obtaining new or repaired components for engine assembly, and engine testing. The other concept is on-conditions maintenance. This is a repair program whereby each engine or engine module is disassembled only to the extent necessary to make repairs and/or replace worn, damaged, and life-limited parts. Page 16 GAO/IMTEG9046 Air Force Depot Maintenance System Appwdix III Major Contributors to This Report John B. Stephenson, Assistant Director Information Suzanne M. Burns, Assignment Manager Management and Sanford F. Reigle, Adviser Technology Division, Washington, D.C. Daniel V. Loesch, Regional Management Representative Cincinnati Regional Steven M. Hunter, Evaluator-in-Charge Office Barbara L. Centers, Evaluator (5103il) Page 16 GAO/IMTEGSO46 Air Force Depot Maintenance System Requests for copies of GAO reports should be sent to: U.S. General Accounting Office Post Office Box 6015 Gaithersburg, Maryland 20877 Telephone 202-275-6241 The first five copies of each report are free. Additional copies are $2.00 each. There is a 25% discount on orders for 100 or more copies mailed to a single address. Orders must be prepaid by cash or by check or money order made out to the Superintendent of Documents.
Air Force ADP: Depot Maintenance System Development Risks Are High
Published by the Government Accountability Office on 1990-05-25.
Below is a raw (and likely hideous) rendition of the original report. (PDF)