I . REPOiT ‘i’0 THE* COMMITTEE ON APPROPRIATIONS HOUSE OF REPRESENTATIVES Problems OnImplementing The Defense Supply Agency’s Standard Automated Materiel Management System B-163074 BY THE COMPTROLLER GENERAL OF THE UNITED STATES JUNE 4,15’-7=I COMPTROLLER GENERAL OF THE UNITED STATES WASHlbIGS‘ON. D.C. 20548 B- 163074 Dear Mr. Chairman: In House Report 91- 1570 your Committee on Appropria- tions requested the General Accounting Office to immediately commence a comprehensive review of the Defense Supply Agency’s Standard Automated Materiel Management System \ in line with the Committee’s directive of September 24, 1969. The observations and conclusions in this report were discussed with officials of the Department of Defense, but we did not request formal agency comments from the Department. \ As previously agreed with members of your staff, copies of our report are being sent to the Secretary of Defense and the Director, Defense Supply Agency. We plan no further distribu- tion of this report unless copies are specifically requested, and then we shall distribute copies only after your agreement has been obtained or public announcement has been made by you concerning the contents of the report. Sincerely yours, Comptroller General of the United States The Honorable George H. Mahon, Chairman Committee a> Appropriations House of Representatives - -.-,. 5QTH ANNIVERSARY 192l-1971~------- COMPTROLLER GENERAL'S REPORT PROBLEMS IN IMPLEMENTING THE TO COMMITTEE ONAPPROPRIATIONS DEFENSE SUPPLY AGENCY'S HOUSEOF REP~SENTATIVES STANDARD AUTOMATED MATERIEL MANAGEMENT SYSTEM B-163074 DIGEST -----_ WHYTHEREVIEWWASMADE The House Commttee on Appropriations' report on the 1971 Department of Defense Appropriation Bill (H. Report 91-1570) contained a re- quest that the General Accounting Office (GAO) immediately begin a comprehensive review of the Defense Supply Agency's Standard Auto- mated Materiel Management System in line with the Committee's direc- tive of September 24, 1969. This directive requested the GAO to re- v-y-c%"tiiiui‘ng basis the development, installation, and opera- tion of automatic data processing-systems. ,.-_. Baekgromd The Standard Automated Materiel Management System is a uniform data system for the Agency's five supply centers which will automate in- ventory control point functions in the areas of distribution, re- quirements and supply control, financial management, procurement and production, cataloging, and provisioning. The prototype for the sys- tem was installed at the Defense Construction Supply Center located in Columbus, Ohio. The system became operational there in September 1969. One of the primary concerns recognized upon the establishment of the Defense Supply Agency in 1962 was a need to accomplish the uniformity of several supply systems which lacked standardization in policies and procedures and types of automatic data processing systems. GAO recognizes the importance of uniformity of policies and procedures and automatic data processing systems at all supply centers and the cost savings and other benefits that generally can be expected from stan- dardization. FINDINGSANDCONCLUSIONS Serious problems have plagued the development and implementation of the system. It was originally planned that the system would be opera- tional at the five supply centers by early 1967, but as of January 1971 the Defense Supply Agency could not estimate when this would oc- cur. Tear Sheet JUWE 419-P GAO believes that many of the problems associated with the system oc- curred because, at key decision points in the development of the system, the Agency placed greatest emphasis on getting hardware installed and operating as quickly as possible and was not willing to accept any de- lays in order to correct known problems. Some of the problems are: --Authority and responsibility for the planning and implementation of the system has been fragmented. The program lacked an overall project manager with sufficient authority to make necessary deci- sions or run the program. (See pa 7.) --An in-depth study was not made of the Defense Supply Agency's re- quirements for frequency of processing. (See p. 9.) --Although there was a significant increase in the work load at the supply centers, the Agency did not make a critical reappraisal of its original approach to the implementation of the system. (See p* 10.) --There has been little effort made to redesign the system even though the computer hardware selected was not the same as was con- sidered when the system was originally designed. (See p. 12.) --Significant problems noted during the testing of the system were not resolved prior to implementation of the prototype. (See p. 17.) --An in-depth study of the estimated costs and savings for the sys- tem has not been performed. (See p* 20.) --Standards have not been developed for measuring whether the pro- totype system has resulted in improved performance in supply operations. (See p. 23.) Although continued efforts by the Agency have produced substantial improvements in the system, the prototype still is not performing all of its assigned tasks as frequently as originally planned. Plans to obtain eqtipmen~&reater -..-. -Impam2 From experience with the prototype, the Defense Supply Agency has con- cluded that the system cannot be extended successfully to all other supply centers using the IBM 360/50 ::quiprilent. Current plans call for another competition to select equipment with greater capacity, test the system at one location, extend the system to the other supply cen- ters, and redesign the system at a later time to take advantage of the capabilities of newly ar:quirpd h,drJware and software. In GAO's opin- ion, after the equipment is ,elected, the system should be redesigned to take advantage of the I:,l,)abilities of the equipment at one location prior to installation at all supply centers. (See pp. 12, 15, and 23.) Interim extension p2cm.s Pending acquisition of new equipment, the Agency wants to extend the current system to the Defense General Supply Center and Defense Per- sonnel Support Center. The Agency has concluded that the work loads at these centers are such that the. IBM 360/50 equipment can do the job. In GAO's opinion, the Agency has not demonstrated an urgent need to extend the system or that substantial benefits would accrue by an interim extension of the IBM 360/50's to these centers. (See p. 21.) MATTERS FORCONSIDERATION OF THE COMMITTEE The Committee may wish to have the Defense Supply Agency prepare a cur- rent cost-benefit analysis prior to proceeding with another competition. The Committee may also wish to discuss with officials of the Agency --steps being taken to improve management of the program, --whether the Agency intends to make a study of its requirements for frequency of processing, --whether the system should be redesigned before installation at all five supply centers, --steps being taken to ensure that possible future problems noted during the testing phase are resolved prior to implementation of the system, --need for development of quantitative standards for measuring im- provements in supply performance at the prototype to tell if the system has improved supply performance, and --whether the system should be extended to the Defense General Supply Center and Defense Personnel Support Center with the IBM 360/50 equipment. Tear Sheet -~~. -.-__ Contents Page DIGEST 1 CHAPTER 1 INTRODUCTION 4 2 MANAGEMENTOF ACQUISITION PROGRAM 7 Lack of an overall systems manager 7 Failure to study requirements for fre- quency of processing 9 Increase in work-load requirements 10 Change in computer hardware 12 Request for larger computers 15 Matters for consideration by the Commit- tee 16 3 SYSTEMTESTING AND COMPATIBILITY 17 System testing 17 Compatibility 19 Matters for consideration by the Commit- tee 19 4 COSTS, PLANS TO EXTEND, AND INABILITY TO MEA- SURE SYSTEM BENEFITS 20 Total estimated costs and savings 20 Interim plans to extend SAMMSto other supply centers 21 Plans to obtain equipment of greater ca- pacity 23 Inability to measure benefits of SAMMS 23 Matters for consideration by the Commit- tee 25 ABBREVIATIONS DCSC Defense Construction Supply Center DSA Defense Supply Agency GAO General Accounting Office IBM International Business Machines RCA Radio Corporation of America SAMMS Standard Automated Materiel Management System COMPTROLLERGENE-@AL'SREPORT PROBLEMS IN IMPLEMENTING THE TO COMPlITTEEON A.Z?ROPRIATIQl'S DEFENSE SUPPLY AGENCY'S HOUSEOF REPR?SFNTATII/%S STANDARD AUTOMATED MATERIEL MANAGEMENT SYSTEM B-163074 DIGEST ------ WHY TUE RE?IEW WAS MADE The House Committee on Appropriations' report on the 1971 Department of Defense Appropriation Bill (H. Report 91-1570) contained a re- quest that the General Accounting Office (GAO) immediately begin a comprehensive review of the Defense Supply Agency's Standard Auto- mated Materiel Management System in line with the Committee's direc- tive of September 24, 1969. This directive requested the GAO to re- view on a continuing basis the development, installation, and opera- tion of automatic data processing systems. Background The Standard Automated Materiel Management System is a uniform data system for the Agency's five supply centers which will automate in- ventory control point functions in the areas of distribution, re- quirements and supply control, financial management, procurement and production, cataloging, and provisioning. The prototype for the sys- tem was installed at the Defense Construction Supply Center located in Columbus, Ohio. The system became operational there in September 1969. One of the primary concerns recognized upon the establishment of the Defense Supply Agency in 1962 was a need to accomplish the uniformity of several supply systems which lacked standardization in policies and procedures and types of automatic data processing systems. GAO recognizes the importance of unif0rmit.y of policies and procedures and automatic data processing systems at all supply centers and the cost savings and other benefits that generally can be expected from stan- dardization, FINDINGS AND CONCLUSIONS Serious problems have plagued the development and implementation of the system. Ii; was originally planned that the system would be opera- tional at the five supply centers by early 1967, but as of January 1971 the Defense Supply Agency could not estimate when this would oc- cur. I GAO believes that many of the problems associated with the system oc- curred because9 at key decision points in the development of the system, the Agency placed greatest emphasis on getting hardware installed and operating as quickly as possible and was not willing to accept any de- lays in order to correct known problems. Some of the problems are: --Authority and responsibility for the planning and implementation of the system has been fragmented. The program lacked an overall project manager with sufficient authority to make necessary deci- sions or run the program. (See p. 7.) --An in-depth study was not made of the Defense Supply Agency's re- quirements for frequency of processing. (See p0 9.) --Although there was a significant increase in the work load at the supply centers, the Agency did not make a critical reappraisal of its original approach to the implementation of the system. (See p. 10.) --There has been little effort made to redesign the system even though the computer hardware selected was not the same as was con- sidered when the system was originally designed. (See p. 12.) --Significant problems noted during the testing of the system were not resolved prior to implementation of the prototype. (See P. 17.) --An in-depth study of the estimated costs and savings for the sys- tem has not been performed. (See p. 20.) --Standards have not been developed for measuring whether the pro- totype system has resulted in improved performance in supply operations. (See p. 23.) Although continued efforts by the Agency have produced substantial improvements in the system, the prototype still is not performing all of its assigned tasks as frequently as originally planned. Plans to obtain equipment of greater capacity From experience with the prototype, the Defense Supply Agency has con- cluded that the system cannot be extended successfully to all other supply centers using the IBM 360/50 equipment. Current plans call for another competition to select equipment with greater capacity, test the system at one location, extend the system to the other supply cen- ters, and redesign the system at a later time to take advantage of the capabilities of newly acquired hardware and software. In GAO's opin- ion, after the equipment is selected, the system should be redesigned to take advantage of the capabilities of the equipment at one location prior to installation at all supply centers. (See pp. 12, 15, and 23.) In.terim extension p&z8 Pending acquisition of new equipment, the Agency wants to extend the current system to the Defense General Supply Center and Defense Per- sonnel Support Center. Tk,e Agency has concluded that the work 'loads at these centers are such that the IBM 360/50 equipment can do the job. In GAO's opinion, the Agency has not demonstrated an urgent need to extend the system or that substantial benefits would accrue by an interim extension of the IBM 360/50's to these centers. (See p. 21.) MA!l'TERS FORCONSIDERATION OF THE COMMITTEE The Committee may wish to have the Defense Supply Agency prepare a cur- rent cost-benefit analysis prior to proceeding with another competition. The Committee may also wish to discuss with officials of the Agency --steps being taken to improve management of the program, --whether the Agency intends to make a study of its requirements for frequency of processing, --whether the system should be redesigned before installation at all five supply centers9 --steps being taken to ensure that possible future problems noted during the testing phase are resolved prior to implementation of the system, --need for development of quantitative standards for measuring im- provements in supply performance at the prototype to tell if the system has improved supply performance9 and --whether the system should be extended to the Defense General Supply Center and Defense Personnel Support Center with the IBM 360/50 equipment. CIIAPTER 1 INTRODUCTION The Defense Supply Agency (DSA) began operations in January 1962 and was given the responsibili.ty for central- ized management of several supply systems which had been independently developed. These systems lacked standardiza- tion in policies and procedures and in the types of auto- matic data processing equipment in use. Study groups re- viewed each of the functional areas such as procurement and cataloging and prepared studies in 1962 and 1963 on proce- dures followed at each of the supply centers. These studies identified inconsistencies and recommended procedures to be adopted. The next step toward achieving uniformity of systems encompassing the major functions associated with materiel management began in April 1963, and the data system design concept for a Standard Automated Materiel Management Sys- tem (SAKMS) was approved in June 1964. SAMMS is a uniform data system which automates inventory control point func- tions for distribution, requirements and supply control, financial management, procurement and production, catalog- ing , and provisioning. The stated objectives of SAMRIS were (1) to adopt for all supply centers the best features from each of the sev- eral systems that were inherited by DSA, (2) to improve ex- isting operating techniques by developing new applications, (3) to improve data processing to take advantage of the capabilities inherent in modern data processing equipment, (4) to standardize data processing equipment, and (5) to exploit modern data processing equipment to provide effec- tive and economical performance of logistics operations in peacetime and to facilitate adjustment to the demands of mobilization or war. The initial plans developed in 1964 called for a com- petitive selection of computer hardware, various tests to be made of the hardware and software, installation of SAMMS at a prototype installation, and subsequent extension of the system to DSA’s four other supply centers. It was 4 originally planned to have SAMMS installed and operating at all locations by early 1967. The Radio Corporation of America (RCA) 3301 computer was initially selected for SAMMS in May 1965. Of the two bids received, the RCA bid was lower. Before DSA could proceed with acquisition of the computer, RCA had to perform a successful benchmark test. The test was to demonstrate the ability of the proposed system to perform specified data processing functions within stated work load and time constraints and the availability and operability of the proposed equipment. RCA subsequently failed, on two occa- sions, to pass the required benchmark test. Negotiations with RCA were terminated in April 1966 and DSA began nego- tiations with the International Business Machines (IBM) Corporation, the only contractor other than RCA to submit a bid for the SAMMS hardware procurement. The IBM 360/50 computer was subsequently selected for SAMMS and passed the benchmark test in May 1967. The pro- totype was at the Defense Construction Supply Center (DCSC in Columbus, Ohio. IBM equipment was installed at DCSC in July and October 1968 and, after various tests of the sys- tem, SAMMS became operational at DCSC on September 11, 1969, SAMMS was one of five computerized management systems discussed in a prior GAO report to the Chairman, House Committee on Appropriations, entitled “Inquiry Into Prac- tices Followed by the Department of Defense Components in Acquiring and Installing New Automatic Data Processing Equipment for Use in Computerized Management Systems” (B-163074, dated March 13, 1968). This report concluded that the design and installation of new management systems or major revisions of existing management systems should be undertaken only after a thorough evaluation of requirements and a thorough study of the operating function involved are made to reassess the basic objectives and to identify known and potential problem areas which should be corrected. The report pointed out that from standardization it can generally be expected that cost savings and other bene- fits will result, Further , three of the five systems, in- c lud ing SAMLIS, were designed primarily to achieve standard- ization and modernization of existing data processing 5 systems and equipment D However, adequate studies of the operating function had not been undertaken prior to proceed- ing with these projects. In our current review of SAMM, we were concerned with factors relevant to the decision in April 1966 to implement SAM&K with the IBM 360/50 computer and subsequent events. We evaluated the reasonableness of planning and cost and savings estimates for the program, adequacy of the testing of the system prior to installation at the first supply center) and compatibility of SAMMS with other automatic data processing systems. During our examination we reviewed pertinent records, correspondence, studies, and plans and interviewed offi- cials of the following offices or agencies. Assistant Secretaries of Defense (Comptroller) and (Installations and Logistics), The Pentagon Headquarters, Defense Supply Agency, Cameron Station, Virginia Defense Construction Supply Center, Columbus, Ohio Data Systems Automation Office, Columbus, Ohio Fieldwork was performed during the period September 1970 through January 1971. CHAPTER 2 ! MANAGEMENT OF ACQU.ISITION PROGRAM Since its inception, SAMMS has been plagued with seri- ous problems. These problems were caused, in part, by the absence of an overall systems manager for the program and the lack of adequate studies of DSA’s requirements at the inception of the program. To a large extent, we believe that these problems occurred because, at key decision points in the development of the system, DSA placed the greatest emphasis on getting hardware installed and operating as quickly as possible and was not willing to accept any de- lays in order to correct known problems. When there was an increase in work-load requirements, DSA was not willing to take the time necessary to give appropriate consideration to this fact. When there was a change in computer hard- ware, DSA was not willing to take the time necessary to re- design the system. Although substantial improvements have been made by DSA and the installation of a larger computer at the proto- type installation has alleviated some of the problems, the system is still not performing as originally planned. LACK OF AN OVERALL SYSTEMS MANAGER Authority and responsibility for the planning and im- plementation of SAMMS have been fragmented. No one organi- zation or individual was given the appropriate authority and responsibility to plan, direct, and exercise control. We believe that the lack of a strong single manager for SAMMS contributed significantly to many of the problems that have been experienced in implementing the system. DSA designated a SAMMS project officer in 1965 but he was not given the overall authority of decisionmaking. The project officer’s job was primarily one of coordination, and any problems that arose had to be resolved through “persua- s ion” with other functional levels within DSA. For example, the project officer informed us that the requirements for daily processing set by the functional di- rectorates in 1964 were too high to meet their needs and 7 that the equipment available at the time was not cap:~ble of processing these requirements. The project officer tried to "persuade" the functional directorates to rzducc daily processing requirements to reflect more realistic needs, but he was not successful. As a result, the unrealistic requirements were incorporated in the request for proposal. It was not until June 1970 that, after the prototype instal- lation proved that these requirements could not be met, DSA took action to reduce them. In April 1970, DSA established a team to thoroughly re- view SAMMS. The review team recommended 85 improvements in operations. One recommendation was that a SAM&ISprogram di- rector be appointed to implement the team's recommendations. The review team's report summarized the effect of the lack of a program director for SAM% as follows: "The development and installation of SAMIIS has been and is plagued by conflict of interests at every turn. A SAM% Program Director with power of decision and necessary resources to direct the developmental, functional and operational elements at all levels toward the accomplishment of what is best for DSA as a whole is a must." As a result of the review team's recommendation, the Director of DSA, in June 1970, designated a SAblMSprogram director for a period of 90 days and gave him wide author- ity to act on behalf of the Director of DSA. During this period, the program director was to accomplish the imple- mentation of the review team's recommendation at the Proto- type. We believe that the SAMMS program director was suc- cessful in effecting numerous system improvements in SAbIMS, but we are concerned that the program director was appointed only for a go-day period. The Assistant Secretary of Defense (Comptroller) also has expressed concern over the need for a permanent SAMMS program director. In a letter to DSA dated November 6, 1970, the Assistant Secretary remarked that a strong pro- gram manager would provide intensified SAMMSsystems man- agement for the following problems. 8 1. Need for improved control of changes to ensure com- patibility of programs, functional documentation, and employee training. 2. Need for establishment of a mechanism at DCSC for the prompt and effective solution of future SAMMS problems, particularly those which cut across sev- eral functional Headquarters, DSA, directorates. 3. Need for review and direct control of another hard-, ware competition and installation of SAMMS at all supply centers) as well as the redesign and imple- mentation of an improved SAMMS. At the completion of our fieldwork, DSA had not made any plans for the designation of a permanent SAWS program di- rector. FAILURE TO STUDY REQUIREMENTS FOR FREQUENCY OF PROCESSING Systems planning should include sufficient study to determine optimum frequency for data processing. Process- ing too often can be more costly than is warranted by the value of information produced. On the other hand, infre- quent processing may result in additional costs and inade- quate supply support because management decisions will be made without current information. DSA did not make an in-depth study to determine ex- actly what frequency of processing was needed for supply centers to perform their mission most efficiently. As originally planned, frequency of processing appar- ently was based on what the DSA functional directorates would have liked to get from the system. It was not until it became evident that these plans were unrealistic that DSA took action to reduce them. The planned frequencies of processing established for each SAMMS subsystem in 1964 were the objectives for the system until August 1968. From August 1968 through Decem- ber 1970 numerous changes were made to the frequency of processing. A comparison of the original and revised weekly processing schedule as of December 1970 is shown below. 9 Frequency of weekly processing Approved Original schedule Subsystem schedule December 1970 Distribution: Multidaily 42 14 Daily 7 7 Requirements 7 2 Procurement 7 3 Financial 7 3 Cataloging 7 2 Provisioning 7 2 The actual frequency of processing at the prototype has never equaled that originally planned. For the months of November and December 1970, however, the actual fre- quency was very close to the approved frequency. We discussed the frequency of subsystem processing with DSA personnel to determine what effect the reduction in frequency had on the accomplishment of their mission. We were advised that the current frequency at the proto- type was adequate and that higher frequency would not be economical. In view of the numerous changes that have been made to the approved processing frequency, we believe that an in- depth study should be made to determine exactly what fre- quency of processing is needed for the supply centers to perform their mission most effectively. INCREASE IN WORK-LOAD REQUIREMENTS Analyses were made in June 1964 by DSA of current work loads at the supply centers. Anticipated increases in work load were projected over a S-year period when SAMMS’ was scheduled for implementation in 1967. This work load was then included in the request for proposal, and vendors were required to demonstrate the ability of their equip- ment to process the projected work load for a peak day within 10 hours. 10 When IBfiZ was selected in May 1967, the actual volumes of work load at the supply centers had increased an average of 62 percent over the work load projected in the request for proposal, largely due to the buildup in Southeast Asia. However, DSA did not give appropriate consideration to the fact that the original projected volumes of work load were no longer valid and that, if its original approach was con- tinued, it was likely that a capacity problem would be en- countered. We discussed the increased work load with DSA offi- cials. They felt that, if additional capacity was needed, they would merely upgrade the system to a larger computer ( i.e., IBlrl 360/65). DSA officials also stated that to have given recognition to the increased work load would have meant that DSA would have to revise the request for proposal which, in turn, would mean another competition for the SAMMS procurement. This alternative was not acceptable to DSA because of the delay that it would have caused to the im- plementation of SAMMS. 11 CJIAN'X IK COFfPUTER I~lARDWARE After selection of thz RCA 3301 computer in May 1965, DSA began designing SAMMS around the RCA equipment. By the time it was decided, in April 1966, that the RCA equipment was not acceptable p DSA had already expended substantial design effort on the system. When the decision was made to switch from RCA to the IBM 360/50 computer, DSA directed that redesign of the system be held to a minimum to keep costs down. We believe that many of the problems experi- enced with SAMMS can be attributed, in part, to the fact that little effort was made to redesign tI,e system even though there was a change in equipment. SAMMS is designed as a sequential system which pro- cesses only one functional program at a time. In general, each process is dependent upon an action or process that immediately precedes it in the processing system. OSD of- ficials told us that, because SAMMS had been designed as a sequential system, the amount of multiprogramming possible was limited and thus precluded the full exploitation of the computer capabilities. They said that SAM;hllSwould have to be redesigned to remove the constraints of the present se- quential system. Multiprogramming is a capability which allows several programs to be run concurrently on the same computer. DSA uses the term “multi-batch processing” to define this type of operation. The use of multiprogramming greatly improves computer productivity because it reduces the amount of time that the central processing unit is idle while awaiting an input or output. The currently installed SAMNS operates in a very limited multiprogramming mode. Because the system, as initially designed, is essentially sequential and pro- cess dependent, extensive use of multiprogramming is virtu- ally impossible. DSA recognized potential increased use of multiprogram- ming in 1966 but believed that the state of the art at the time was limited. Therefore DSA proceeded with its sequential-processing design. This decision was based also on considerations that (1) inventory control point process- ing was primarily sequential in nature and (2) DSA did not have in-house expertise to design and run a multiprogrammed system. 12 A DSA Analysis of ADP Requirements for SAMMS, dated .June 12 9 1969, pointed out that increased use of multipro- gramming was more feasible; but, to use this capability, the current system would have to be redesigned. An analy- sis of the increased productivity that would result from this approach was never made. DSA estimated that it would take from l-l/‘2 to 2 years to redesign the system. Since its goal was to extend SAM&IS to all centers as quickly as possible, DSA concluded that system redesign would not of- fer a short-term solution. Moreover, DSA believes that the current system as designed can meet its needs. The significance of the system design problem was brought out in a report dated May 26, 1970, by a DSA System Review Team. “We believe that the major cause of the current problems with SAME extends back to the beginning of the *** project. A major error occurred when equipment was selected using specifications which were drastically different from those which were used to program the current system. Once the equipment was selected, requirements should have been developed in accordance with the capability of the selected equipment or new equipment should have been selected based on the new requirements. It appears that no effort was made to do either.” In a memorandum dated July 10, 1969, the Assistant Sec- retary of Defense (Comptroller) instructed DSA to immedi- ately initiate plans to redesign SAMVS to take full advan- tage of available hardware and software capabilities and start another competition for SAMMS when the design was com- pleted. The Assistant Secretary subsequently agreed with DSA’s approach of deferring system redesign until after new hardware was selected and the system was extended to all supply centers s The rationale given for agreeing with DSA’s approach at that time was that this approach was the quickest and most practical means of getting SAMI% installed at all DSA supply centers. DSA offici::ls told us that, even if the system was re- designed, the present equipment could not accommodate the work load at all centers. They informed us that, because of 13 t-he increase in work load and anti.cipatyd increases in the future, an upgrade in equipment was needed. Although the above approach may be the quickest means of installing SAMMS at all supply centers, we are not con- vinced that this is the best day to proceed. In our opin- ion, after the equipment is selected, SAMMS should be rede- signed to take advantage of the capabilities of the equip- ment at one location prior to extending the system to all supply centers. Although we recognize that this will re- quire considerable effort on the part of DSA and will delay extension of the system, DSA has not demonstrated a critical need to extend the system as quickly as possible and its approach is the most cost beneficial. Benchmark - test IBM successfully passed the benchmark test by using multiprogramming even though this feature was not included in the actual system software of SAMMS. The benchmark test was made to demonstrate (1) the ability of the proposed system to perform specified data processing functions within stated work load and time con- straints and (2) the availability and operability of the vendor’s proposed equipment. The test was conducted using actual data and performing functions typical of center’s day-to-day needs within a prescribed amount of time. One of the mandatory requirements established by DSA to evaluate the benchmark test results was that computer processing hours be equal to, or less than, 10 hours. Using the multiprogramming feature, IEV completed processing in about 9 hours and thus passed the benchmark test. 1Zt our request, a DSA official prepared an analysis of what the timings would have been had IBM not used the multi- programming feature. This analysis showed that the timing would have been at least 12 hours and, consequently, IBM would not have passed the test. We asked DSA officials why IBM was allowed to use multi- programming if this feature was not to be included in their actual system design. We were told that specifications 14 included in the request for proposal provided guidance as to overall system requirements and the integrated manner in which the system was to operate. Within these guide- lines p vendors were not restricted as to the method they proposed to process the work load. Since IBM’s proposal met the requirements of the request for proposal, DSA was obligated to test what was proposed. Although the IBM proposal may have met requirements stated in the request for proposal, we believe DSA should have recognized that the multiprogramming capability was not applicable since this capability was not included in the SAMMS system design. In our opinion, the benchmark test was not a true measurement of the system timings that could be expected when SAMMS was implemented. DSA officials informed us that, if IBM had not passed the benchmark test, another competition for the SAWS hardware selection would have been necessary. This would have delayed the implemen- tation of the system. Again, DSA’s desire to proceed as quickly as possible seems to have been the governing factor. REQUEST FOR LARGER COMPUTERS As early as June 1969, DSA recognized that the planned computer configuration would be inadequate to support cur- rent and future work-load requirements at the supply cen- ters, In a memorandum to the Assistant Secretary of Defense (Comptroller) dated June 13, 1969, DSA requested approval to extend SAMMS on a sole-source basis by replacing six IBM 360/50 computers with the larger IBM 360/65 computers. DSA felt that the larger computers were necessary to effec- tively implement the system. DSA’s request was denied on the basis that (1) the system could be redesigned to take full advantage of IBM 360/50 hardware and software capabil- ities and (2) if greater capacity proved to be essential after redesign, competitive selection of equipment 1~0~14 be more economical to the Government. DSA was directed to rc- design the system and prepare a new request for proposal for another competition to select hardware. In a subset-uent memorandum to the Assistant Secretary of Defense (Comptroller) dated January 28, 1970, DSA stated that its efforts to improve efficiency at the first instal- lation (DCSC) had not resulted in enough additional computer time to do all the tasks intended. DSA requested permission 15 to install one IBM 360/65 at DCSC, on an interim basis, to obtain the needed additional computer capacity. This re- quest was approved on the basis that additional capacity was needed for DCSC to perform its mission. In March 1970 an IBM 360/65 was installed at DCSC and one of the IBM 360/50's was removed. Although the larger computer has provided some relief in computer processing time, SAMMSis still not performing all tasks as frequently as originally planned. MATTERSFOR CONSIDERATION BY THE COMMITTEE DSA is in the process of another competition for the SAMMShardware. Plans call for redesign of the system at a later date to take advantage of the capabilities of hard- ware and software selected. Whether DSA should install SAMMSat all five supply centers prior to redesigning the system is a matter that the Committee may wish to discuss with agency officials. The Committee also may wish to discuss what actions have been taken to improve the systems management of SAMMS and whether DSA intends to make a study of its requirements for frequency of processing. 16 CHAPTER 3 SYSTEM TESTING AND COMPATIBILITY SYSTEM TESTING One of the most important phases in the development of any system is complete testing required prior to implemen- tation. Sound management dictates the resolution of sig- nificant problems arising during the testing phase of a system before it becomes operational. SAMMS was placed in operation at DCSC in September 1969, even though the sys- tem testing objectives were never fully achieved. In our opinion, failure to ensure that test objectives were ac- complished contributed to many of the problems encountered after the system was implemented. DSA instructions required threemajor tests of SWIMS prior to system implementation--the controlled functional test, the volume test, and the environmental test. Each of these tests is discussed below. Controlled functional test The primary objectives of this test were to assure the DSA headquarters functional managers that (1) the data sys- tem would perform according to logic contained in the sys- tem’s requirements, (2) the sequence of processing was as prescribed by the requirements, (3) the data system would produce the required outputs and (4) the functional operat- ing manuals were fully compatible with th% data system de- sign. The test provided machine output which Gould be mea- sured against predetermined results. If program errors were identified, the program could be corrected and then retested before proceeding with testing of the next segment. In the case of SAMMS, all program segments were not tested and problems resolved before proceeding to the next segment. In some cases, program segments failed to pass the test two or three times and problems that were categorized as re- solved were subjected to little, if any, retesting. In other cases, program segments were not tested at all before proceeding to the next segment. 17 Al though a M&INS pal i cy axi yy1id? r7i:e uiemorandum stip- ulated that the controll.ed fur :ctional iezct must be corn- pleted prior to moving into the environmental test 9 these two tests in fact overlapped. This resulted in pushing de- ficiencies forward and the problems became more complex and difficult to solve. Volume -- test The volume test. originally was scheduled to follow the controlled functional test; however, the entire volume test was completed on February 18, 1969, whereas the controlled functional test was not completed until June 1969. The purpose of the volume test was to satisfy DSA that the programs and hardware could handle a normal work load and an overload within established time frames. A final report on the volume test pointed out that ac- tual timings did not meet objectives, and recommendations were made to improve the timing. Environmental test The primary objectives of the environmental test were to validate the total system and enable operating personnel to become familiar with it. A midpoint evaluation of the environmental test showed that machine time was about 18 hours & day, whereas the re- vised objective was 12 to 14 hours. In addition, in his final progress report on the environmental test, the Com- mander at the prototype expressed his concern about the ca- pability to process the procurement, requirements, catalog- ing 9 and financial segments of the system only once a week; wherea.s the original plans called for running these seg- ments daily. Although the Director of DSA called for efforts to re- solve these problems, SAMMS was implemented prior to their resolution. As late as Cctober 1970, a review team set up by the Office of the Assistant Secretary of Defense (Comp- troller) noted that a large number of errors could be at- tributed to a lack of system testing. BW COMPATIBILITY We were informed that SAMMSwould have to be compat- ible to and interface with the following major computer sys- tems. 1. The Defense Automatic Addressing System. 2. Mechanization of Contract Administration Services System. 3. Mechanization of Warehouse and Shipping Procedures System. 4. Defense Logistics Services Center System. DSA stated that compatibility requirements between SAMMSand the first three systems listed above were pro- vided for through Department of Defense military standards which prescribed the method of communication between the systems. With respect to the fourth system, compatibility re- quirements are prescribed in publications (cataloging man- uals) which dictate how to do business with the Defense Logistics Services Center. DSA has stated that this system will be replaced at some time in the future by the Defense Integrated Data System and that, when this new system be- comes operational, SAMMSwill have to be redesigned to achieve the required interface with the system. MATTERSFOR CONSIDERATION BY THE COMMITTEE Since DSA is in the process of another competition for the SAMMShardware, the Committee may wish to discuss with DSA officials their plans for testing of the system and what steps will be taken to ensure that test objectives are accomplished prior to implementation of the system. 19 CHAPTER 4 COSTS, PLANS TO EXTEND, AND INABILITY TO MEASURE SYSTEM BENEFITS Through fiscal year 1971, DSA estimates that $28.8 mil- lion will have been expended for the rental oi equipment and the development, testing, and implementation of SAMMS. No procurement funds have been expended, although in fiscal year 1970 $5 million in funds were appropriated for pro- curement of equipment a Due to the delay in implementing SAMMS, these funds were reprogrammed to other projects as shown below s Amount (millions) Mechanization of Contract Administration Services Sys tern $2.8 Other (for the purchase of computers and/or components for ADP systems at various DSA locations) 2.2 Total $5. TOT_4L ESTIMATED COSTS -- AI’JD SAVINGS -- DSA prepared an analysis of total SAMMS costs and sav- ings in #July 1966. The estimates in this analysis were ex- tensively qualified, as indicated by the following quote H from the analysis concerning the validity of the estimated savings, “I must stress again the dangers in use of these estimates in any Headquarters consolidation of . projected SAMhIS savings developed in this manner. C)nly if used in the broadest ‘order of magnitude’ sense should such estimates be considered.” The 1966 analysis pointed out that the estimates should .. be used as ball-park figures only and that an in-depth study 20 of SAMMScosts should be undertaken. We were told that this study was never made. As of January 1971, no further stud- ies concerning total SAm4S costs and savings have been pre- pared. The Office of the Assistant Secretary of Defense (Comptroller) has requested updated estimates of total SAMEIScosts but DSA has not provided any additional infor- mation. During the course of our review, we were informed that a new analysis was in process and would be completed in December 1970. We were subsequently advised that this analysis would not be completed until late spring of 1971. INTERIM PLANS TO EXTEND SAIWS TO OTHER SUPPLY CENTERS The Office of the Assistant Secretary of Defense (Comptroller) stipulated that DSA could extend SAMMSto ad- ditional supply centers only if it was able to use the IBM 360/50 computer. DSA concluded that the work load at the Defense General Supply Center and Defense Personnel Support Center was such that the 360/50 could do the job. An eco- nomic analysis of extending the system to the Defense Gen- eral Supply Center showed total estimated costs of $1.5 mil- lion offset by estimated benefits of $2.4 million for a net savings of about $900,000. In reviewing this analysis, the Office of the Assis- tant Secretary of Defense (Comptroller) concluded, in part: "The analysis does not adequately demonstrate that SAMI%, based on prototype experience will result in improved functional performance at DGSC [Defense General Supply Center]. Broad claims of improve- ments are made but they are not substantiated by actual achievements at DCSC extrapolated to DGSC. DSA has made no statement to the effect that SAMMS extension is urgently required for mission perfor- mance at DGSC, Other problems with the analysis are that alternatives to the proposed action are neither defined nor evaluated; some benefits that are claimed are not acceptable while others are doubtful." In a memorandum dated January 29, 1971, the Assistant Sec- retary of Defense (Comptroller) advised the Director, DSA, 21 that the eCtiIlcJmic analysis did not proliide a strong basis far exporting S4:\MS to the Defense General Supply Center and that a decision would be deferred until he had an oppor- tunity to review GAO’s report on SAM&IS to the House Appro- priations Commll ttee a DS4 officials told us that, in the fiscal year 1972 budget 5 they included about $1.4 million for extension of SAMMS on IBM 360/50 computers to the Defense General Supply Center and the Defense Personnel Support Center. DSA’s reasons for extension are to reduce the lead time essential for file conversion, training of personnel, and testing and to achieve benefits of having a standard system at three locations. Since DSA has decided that SAWS cannot be fully ex- tended with this equipment and since plans are to replace this equipment as soon as the new competition is completed, we do not believe that it is appropriate to install addi- tional interim systems at the supply centers in the absence of an urgent need or demonstration of significant cost re- ductions and improved supply responsiveness to be achieved at the centers. 22 PLANS TO OBTAIN EQUIPMENT OF GREATER CAPACITY As of January 1, 1971, DSA.was in the process of pre- paring a new request for proposal for another competition for the SAMMS hardware. We were told that the key mile- stone dates for future implementation of SAMMS would vary according to which manufacturer won the competition. DSA’s target dates for installing SAMMS are represented as fol- lows : Milestone dates If IBM is If other vendor Event selected is selected Release of request for proposal Mar. 1971 Mar. 1971 Deadline for submitting proposal June ” June ” Selection of vendor Oct. ” Oct. ” Installation of equipment at prototype and testing July 1972 Aug. 1973a Final approval of system at prototype installation Jan. 1973 Feb. 1974 Completion of installation No estimate No estimate at DSA supply centers available available aIf a vendor other than IBM is selected, additional time will be required to modify the software which has been tailored to IBM equipment to that of another vendor and train personnel in the use of another vendor’s equipment. INABILITY TO MEASURE BENEFITS OF SAMMS Before undertaking the installation of a new system, it is essential to establish criteria against which to mea- sure results. Some yardsticks are needed to be able to tell if the system is doing what it was acquired to do. DSA personnel expressed the opinion that SAMMS had im- proved supply operations at DCSC, but they were unable to provide us with any quantitative evidence to show what bene- fits had been realized. Further, DSA was unable to identify 23 any criteria for measurement of the success or failure of SAmS relative to the mission of DCSC. We compared various statistics showing supply perfor- mance at DCSC before the installation of SAMMSwith like statistics after SAMMSwas implemented. Although we noted significant differences in some of these statistics, we were advised by DSA personnel that these differences could not be attributed solely to SAMMS. Two examples follow. Stock availability rate A report prepared by the SAMMSprogram director pointed out that the stock availability rate at DCSC had im- proved since the implementation of SAMMS. The rate of stock availability increased from 79.2 percent in Au- gust 1969 to an average of 82.3 percent for fiscal year 1970. We were informed, however, that the increase in the stock availability rate could not be attributed to SAMMSbut was a result of DSA's increasing DCSC's stock fund by about $20 million. The additional money was put into the fund in April 1969; but the first effects did not begin to show up until November 1969, after implementation of SAMMS. Rates for requisitions filled on time Statistics for rates of requisitions filled on time for fiscal years 1969 and 1970 are shown below. Requisition Percent filled _--__---- on time issue priority Fiscal year Fiscal year group --1969 1970 I 54.4 31.7 II 70.0 57.3 All groups 63.7 55.9 We could not determine if the reduction in the number of requisitions filled on time was attributable to SAMMS. As shown by the above examples, it was not possible to idcbntify any qilantitative improvements or degradation in the prototype operations as a result of SAMMS. We believe 24 that DSA should establish standards for measuring the bene- fits in supply performance to tell if the system is doing what it was acquired to do. MATTERS FOR CONSIDERATION BY THE COMMITTEE The Committee may wish to have DSA prepare a current cost-benefit study for SAMMSprior to proceeding with an- other planned competition of the SAMMShardware. The Committee also may wish to discuss DSA's plans to extend SAMMSto the Defense General Supply Center and the Defense Personnel Support Center. Further, the Committee may wish to discuss with DSA the development of quantitative standards for measuring improvements in supply performance at DCSC to tell if SAMMShas resulted in improvements. U.S. GAO Wash.. me. 25
Problems in Implementing the Defense Supply Agency's Standard Automated Materiel Management System
Published by the Government Accountability Office on 1971-06-04.
Below is a raw (and likely hideous) rendition of the original report. (PDF)