DOCUMENT ESUME 7-17 02815 - [A2073168] Us-srirtj- Veterans' Administration Justification of Costs and Benefits of Proposed Computer System (Target). HRD-77-98; B-114859. Jujy 20, 1977. 51 pp. * 4 appendices (24 pp.). Report to Rep. John E. Moss; Rep. Charles Rose; by Elmer B. Staats, Comptroller General. Issue Area: Facilities and Material Management (700). Contact: Human Resources Div. Budget Function: Miscellaneous: Automatic Data Processing (1001); Miscellaneous: Financial Management and Information Systems (1002). Organization Concerned: Veterans Administration. Congressional Relevance: Rep. John E. Moss; Rep. Charles Rose. The Veterans' Administration's (VA's) proposed computer system, the Target System, is intended to modernize benefit claims processing and to improve service to veterans. The VA estimates that the system will cost over $100 million to develop and over $12 million each year to operate. Findings/Conclusions: The VA has proceeded with the development of the Target System without economic analyses of the proposed system or of alternative ways of meeting its benefit claims processing needs. The VA had not established measurable goals or standards for improved service to veterans, and, therefore, it had no basis for comparing alternative approaches or for determining whether systems less complex and expensive than Target could do the job. Although major cost savings are possible, Target's costs could exceed monetary benefits as compared with the current benefit claims processing system. Recommendations: The Administrator of Veterans Affairs should establish, in measurable terms, goals and standards for improving benefit claims services to veteians. The Congress should not permit the VA to award a contract for the Target Sistem until it has set up the measurable goals and standards so that the most cost-effective way of processing benefit claims can be determined. If Congress determines that the VA has satisfactorily resolved these issues, the VA should not be allowed to award a contract until it has resolved the uncertainties about progress of development and the cost and implementation schedule of the system. The VA needs a definitive and complete implementation plaD to accomplish this. (Author/SC) ITRIIICTaD w- Net be j e.idi We vGeneral ah aarhbY He f*ic 'f G -s,* n a Xlete, - ' .... z REPORT OF THE - COMPTROLLER GENERAL i a ,i<,. .,' OF THE UNITED STATES Veterans Administration Justification Of Costs And Benefits Of Proposed Computer System (Target) The Veterans Administration's proposed com- puter system, thu Target System, is intended to modernize benefit claims processing and improve service to veterans. VA estimates that the system will cost over $00 million to de- velop and over $12 million each year to operate. VA needs to convincingly demonstrate that the Target System is the most cost-effective system, before awarding a contract to procure the system's equipment. HRD-77-98 JULY 20, 1977 COMPTROLLER GENERAL OF THE UNITED STATES W HINGTOt4, D.C. 1A14 B-114859 The Hnorable John E. Moss The Honorable Charles Rose House of Representatives Your December 4, 1975, letter requested an investigation of the cost-benefit justification of the Target System, a new computer system being developed by the Veterans Administration to modernize the agency's benefit claims processing and im- prove service to veterans. This report contains our findings concerning the cost- benefit justification for this proposed system and our recom- mendations to the Congress on page 49. The report also contains a recommendation to the Adminis- trator of Veterans Affairs on page 49. As you know, section 236 of the Legislative Reorganization Act of 1970 requires the head of a Federal agency to submit a written statement on actions taken on our recommendations to the House Committee on Government Operations and the Senate Committee on Govern- mental Affairs not later than 60 days after the date of the report and to the House and Senate Committees on Appropria- tions with the agency's first request for appropriations made more than 60 days after the date of the report. As you are aware, the Subcommittees on UD-Independent Agencies, House and Senate Committees on Appropriations, and the House and Senate Committees on Veterans' Affairs would like to receive copies of this report. We also believe that the Offii:e of Management and Budget and the General Services Administration should receive copies. We will be in touch with your office in the near future to arrange for release of the report so that the requirements of section 236 can be set in motion. Comptroller General of the United States REPORT OF THE VETERANS ADMINISTRATION COMPTROLLER GENERAL JUSTIFICATION OF COSTS OF THE UNITED STATES AND BENEFITS OF PROPOSED COMPUTER SYSTEM (TARGET) DIGEST The Veterans Administration (VA) is developing a computer system, the Target System, to modernize VA's benefit claims processing and improve service to veterans. VA estimates that the system will cost $100.9 million to develop and $12.5 million to operate each year. The costs and benefits of the proposed system and alternatives to it were not thoroughly evaluated. They should be. VA has proceeded with development of the Tar- get System without economic analyres of alter- sative ways of meeting its bnefit claims- processing needs. (See ch. 2.) Proponents of the Target System argue that the benefit claims system is inefficient and that service to veterans is suffering. Yet, VA had not established measurable goals or standards for improved service. In other words, VA had no basis for comparing alternative approaches and for determining whether systems less com- plex and less expensive than Target could do the job. (See ch. 2.) VA cost-benefit analyses comparing the current system to the Target System have not conclu- sively demonstrated that Target will result in major savings, compared to current processing methods, or that Target is the most cost- effective way to process benefit claims. (See ch. 3.) Target is still a risky venture. Although major cost savings are possible, Target's costs could exceed monetary benefits, com- pared with current benefit claims-processing methods. (See ch. 3.) 'oarh o. Upon nmoval, the report Cow date should b noted hereon.i HRD-77-98 RECOMMENDATIONS The Administrator of Veterans Affairs should establish, in measurable terms, goal; and Atandards for improving benefit claims serv- ices to veterans. The Congress should not permit VA to award a contract for the Target System until VA has set up the measurable gals and standards, so the most cost-effective way of processing benefit claims can be determined. If the Congress determines that VA has satis- factorily resolved the above ssues, VA should not be allowed to award a contract until it has resolved the uncertainties about progress of development and the cost and implementation schedule of the system. VA needs a definitive and cumplete implementation plan to accomplish this, AGENCY- COMMENTS VA, in commenting on the report, cited recent actions to assess the system. These include a modified implementation schedule for Target and an updated cost-benefit analysis. The Administrator of Veterans Affairs believes that these actions should resolve any un- certainty about the progress of development, costs, and implementation. (See ch. 5.) However, GAO believes that the progress of Target's development, the cost estimates, and the feasibility of VA's plan and schedule for putting the system into peration are still uncertain. VA needs a complete plan for getting Target working. (See chs. 3 and 5.) ii Contents Page DIGEST i CHAPTER 1 INTRODU"CTION 1 Compensation, pension, and education system Target System 2 Economic analysis and cost-benefit/ cost-effectivenesi methodology 6 2 VA HAS PROCEEDED WITH DEVELOPMENT OF THE TARGET SYSTEM WITHOUT ECONOMIC ANALYSES OF ALTERNATIVE APPROACHES 8 Need to improve the CP&E benefit claims system 9 VA designs the Target System 12 3 VA HAS NOT CONCLUSIVELY DEMONSTRATED THAT THE TARGET SYSTEM IS ECONOMICALLY JUSTIFIED 14 VA's recognition of need to improve cost-benefit methodology 14 Results of aggregate analysis of ex- pected cost savings from Target System 16 July 1974 cost-benefit analysis 17 February and September 1976 cost-benefit analyses 20 February 1977 cost-benefit analysis 28 GAO sensitivity analysis 33 Developments since the February 1977 cost-benefit analysis cast more uncer- tainty on the cost effectiveness of Target 33 4 STATUS OF THE CURRENT CP&E SYSTEM 37 5 CONCLUSIONS, AGENCY COMMENTS AND OUR EVALUA- TION, AND RECOMMENDATIONS 38 Conclusions 38 VA comments and our evaluation 39 Recommendation to the Administrator of Veterans Affairs 49 Recommendations to the Congress 49 Page CF-PTER 6 SCOPE OF REVIEW 51 APPENDIX I Letter dated November 23, 1976, from the Administrator of Veterans Affairs 52 II GAO sensitivity analysis of VA's February 1977 cost-benefit study 66 III Letter dated May 19, 1977, from the Administrator of Veterans Affairs 71 IV Principal VA officials responsible for administering activities discussed in this report 75 ABBREVIATIONS BIRLS Beneficiary Identification and Records Locator System CP&E Compensation, pension, and education DDM Department of Data Management GAO General Accounting Office TMI Technology Management, Incorporated VA Veterans Administration CHAPTER 1 INTRODUCTION In response to a request from Congressmen John E. Moss and Charles Rose in December 1975, we reviewed the Veterans Administration's (VA's) proposed Target System--a computer system intended to modernize VA's benefit claims-processing system. One facet of this effort, dealt with in this report, is a review of the cost-benefit justification for the system. In addition to the above-mentioned request, the House Committee on Appropriations expressed a specific interest in the Target System's potential costs and benefits in its re- port 1/ on the fiscal year 1977 Housing and Urban Development and Independent Agencies appropriation b -he Committee said that Target should b justified principally on a emon- stration that it would measurably improve VA's internal effi- ciency over the best performance obtainable under the current system. The Committee emphasized that Target must be measur- ably cost effective compared to an efficient manual system. The Chairman of the Subcommittee on HUD-Independent Agencies reiterated the Committee's position in a January 24, 1977, letter to the Administrator of Veterans Affairs, stat- ing the need for VA to insure that the Target System must be measurably cost effective compared to an efficient manual or evolutionary-developed system. COMPENSATION, PENSION, AND EDUCATION SYSTEM VA's Department of Veterans Benefits administers non- medical benefits and services through 59 major field stations (58 regional offices and a records processing center) within and outside the continental United States. These benefits and services include compensation for service-connected disabilities; pensions for aged, needy, and unemployable veterans; vocational rehabilitation, education, and training assistance; and information and assistance through personal contact. In fiscal year 1977, VA will pay about $9 billion in compensation and pension benefits to 4.9 million veterans and survivors, and about $4.3 billion in education and train- ing benefits to 2.2 million veterans. l/"Report Together With Supplemental, Separate, and Dissent- ing Views on Department of Housing and Urban Development- Independent Agencies Appropriation Bill, 1977" (HR 14233) 94-1220, June 8 1976, page 40. 1 The present compensation, pension, and education (CP&E) delivery system was designed and installed in the late 1950s. It is primarily a manual system; only the claims payment process is automated. Claims processing and financial activi- ties are performed in all regional offices. These activities include awarding or disallowing claims for benefits; int-ract- ing with veterans and beneficiaries; and maintaining veterans' and beneficiaries' claim folders. The regional offices pre- pare input for a computer center in Hines, Illinois, on vari- ous devices; accumulate the input into batches; and send it to the computer center, generally by mail. At the computer center these transactions are written on magnetic tape and processed sequentially through a series of computer cycles to (1) update the CP&E master files, (2) generate payment notices to the Treasury Department, which prints benefits checks, (3) generate various operational and management re- ports, and (4) perform various accounting functions. VA characterizes the system as technically limited, labor intensive, and paper bound and has proposed a new sys- tem to improve efficiency and take advantage of new automatic data processing technology. TARGET SYSTEM The new computer system being developed by VA is called the Target System. It is intended to modernize VA's benefit claims-processing system and improve services to veterans. Benefits expected by VA from the new system include much faster development of claims, more timely delivery of initial benefit :hecks to veterans, faster responses to veterans' in- quiries, and major savings from workload reductions in the regional offices as a result of more efficient procedures and workflow. According to April 1977 VA estimates, total one-time development costs of the Target System, including the acauisi- tion of data processing and telecommunications equipment, will be about $100.9 million. Recurring annual operating costs of the system are estimated at $12.5 million. In January 1976 the General Services Administration issued a request for proposal to prospective vendors for (1) automatic data processing systems for the Target System computer centers, (2) terminal systems for VA regional voffices, and (3) related software and services. After inquiries by the House Appropriations Committee about the justification for the number of computer centers selected for the system, the request for proposal was 2 suspended. On June 15, 1976, the General Services.Administra- tion issued an amended request for proposal prescribing fewer computer centers. In response to the request for proposal, prospective vendors submitted proposals on December 1, 1976, and VA anti- cipates awarding the contract in July 1977. Operational fatures The Target System will use computers in three regional computer centers to _vide i data entry and automated claims- processing capabilities to the VA regional offices. The system will have a central computer facility--Target Central System--for maintenance of master records, centralized re- porting and accounting functions, and generation of payment notices to the Treasurv Department, which prints the benefit checks. The stated k , operational feacures of the Target System rare: -- Computerized processing and control of claims in the regional offices, including automatic calculation of benefit awards, control of pending claims, and work- load reporting. -- Immediate response to veteran inquiries concerning (1) status of claims in process, (2) satus and amounts of award checks, and (3) information in the master record. -- Automated printing of awards, acknowledgements, and other routine letters. -- Ready access to information for reporting. Terminals, installed in 56 of the 58 regional offices and the records processing center, will be connected to the three regional computers by telecommunications lines. Input data will be transmitted from the regional offices to the regional computers, each of which will maintain on-line work-in-process control files on pending claims within its region. These files will be updated automatically as a byproduct of claims processing. The regional offices will be able to obtain (1) information concerning pending claims from the regional computer files and (2) information from the master files at the central computer facility through the regional computers. In addition, the regional computers will link the regional offices to a claimant locator system at Austin, Texas. The following charts depicit the operational features of the current CP&E system and the Target System. 3 THE PRESENT CP&E SYSTEM IHN, LIOIls REGIONAL EO REPORT TO OFFICE M YMCENTRAL. MASTER OFFICE AUN, PAYMENT ORDOERS REGIO EI OFFICE(58 REGIONAL OFFICES) WLsfl 1 i TREA"UP' CECKS TO AUSTN, TEXAS 1 BIRLS (BENEFICIARY IDENTIFICATION AND RECORDS LOCATOR SYSTEM) IS A COMPUTERIZED CLAIMANT LOCATOR SYSTEM 4 THE TARGET SYSTEM HINES, ILLINOIS CENTRAL COMPUTER FILE CENTER REGIONAL COMPUTER RANDOM CENTER MASTER FILE BIRLS INDEX * 3 REGIONAL COMPUTER CENTERS * EACH REGIONAL COMPUTER CENTER SUPPORTS UP TO 20 REGIONAL OFFICES AND IS LINKED TO THE REGIONAL OFFICES, BIRLS INDEX. AND CENTRAL COMPUTER CENTER e6Y TELECOMMUNICATIONS LINES. 5 Pilot program In September 1974 VA began a pilot test of the Target System processing concepts in Philadelphia and Baltimore. Terminals in selected units of the Philadelphia and Baltimore regional offices were linked with a data processing center in Philadelphia, which serves as a regional computer center. The Philadelphia center was also linked with the computerized claimant locator system in Austin and the centralized master files of the benefit claims system at Hines. In the summer of 1975 the pilot test was expanded to three other regional offices--in New York, Washington, and Los Angeles. In Novem- ber 1976 the pilot test was again expanded to place the entire Philadelphia regional office operation under pilot operating conditions. ECONOMIC ANALYSIS AND COST-BENEFIT/COST-EFFECTIVENESS METHODOLOGY Economic analysis concerns the basic problem of economic choice (value received for value invested). The general for- mat for an economic analysis involves determining the cost and benefit of each future course of action. Through a cost- benefit or cost-effectiveness analysis, the decisionmaker can use a set of facts and figures to select a preferred means of achieving certain goals. The cost-benefit approach focuses on maximizing dollar benefits minus cost; the cost-effectiveness approach focuses on achieving a given performance objective at a minimum cost. The most important step in economic analysis is defining the objective of the program or system being analyzed. The objective must be some fixed standard of accomplishment. Once the objective has been established, all feasible means or ways of meeting the objective are determined. All feasible alternatives should be studied so that management can be pre- sented with those alternatives that are most cost effective. The assumptions and constraints surrounding the alternatives selected for comparison should be explicitly stated so that management can evaluate them. Once the costs and benefits of relevant alternatives have been determined, they should be ranked according to one of the following general criteria: (1) least cost for a given level of performance, (2) bst performance for a given cost level, or (3) largest ratio of benefits to cost. Since most important decisions involve elements of uncertainty, the typi- cal economic analysis should address those elements to test 6 whether the ranking of alternatives is sensitive to variations in any of the elements used to measure costs and benefits. This testing of the effects of uncertainty might help to as- sure management that one alternative is clearly preferable; however, it might also indicate that an alternative that seems preferable might prove a poor choice if some basic assumptions leading to its selection prove unreasonable. 7 CHAPTER-2 VA HAS PROCEEDED WITH DEVELOPMENT OF THE TARGET SYSTEMWITHOUT ECONOMIC ANALYSES-OF ALTERNATIVE APPROACHES VA has proceeded with development of the Target System without economic analyses of alternative system approaches for meeting its CP&E processing needs. Since the inception of the Target System approach in 1972, VA has cited the deficiencies of the existing CP&E system, which have resulted in poor service to the veterans, as a principal argument suppcrting development of Target. This argument implies that veterans were being so poorly served by the current system that it should be replaced by Target. Yet an April 1972 VA Task Force report, which served as the primary basis for specifically adopting the Target System as an agency policy goal, acknowledged that the exist- ing CP&E automatic data processing system serviced 95 percent of the veterans satisfactorily. In addition, VA has not established quantifiable goals or standards for improved service by either the present sys- tem or Target against which to consider the extent of system improvement or redesign necessary to improve service to veterans determined to be poorly served by the current sys- tem. Without quantifiable goals or standards, VA had no basis to compare alternative approaches or to determine whether systems less complex and less expensive than Target could effectively meet its CP&E processing needs. Since VA did not make economic analyses of alternative approaches--a major element in cost-benefit/cost-effectiveness methodology--its economic justification for Target can address only the cost effectiveness of the system compared to current processing proced ves. VA established the Target System as a policy goal although total ystem costs were not addressed. Accordingly, VA proceeded witl the design and development of the system without giving itself the flexibility of considering viable alternative approaches should the cost of the system not justiiy the benefits received. NEED-TO IMPROVE-THE CP&E-BENEFIT CLAIM SYSTEM In May 1971. VA contracted with Technology Mana9gment, Incorporated (TMI), to investigate the efficiency and effec- tiveness of VA data processing. In October 1971, TMI re- ported on the status of data processing in VA and presented recommendations for improvement. With regard to the CP&E benefits delivery system, TMI concluded that the system had provided VA with significant cost and service benefits. However, the data processing functions were not adequately integrated with the manual efforts they supported; as a result, service to veterans was suffering. TMI recommended that a special task force be established at the Administrator's level to analyze and re- design the CP&E system. At the time of the TMI study, VA's Department of Data Management (DDM) had Fplans which called for -- edesign and recoding of major computer processing operations of the CP&E system to take advantage of advanced computer technology, -- placement of tape iles on faster access devices, and -- ultimate conversion of the system to a operation permitting transactions to be processed when recorded rather than be accumulated into batches for processing and permitting direct access to each record on the computer file. DDM recommended a ph&sed schedule, or a gradual approach, for design, development, and implementation. This would allow VA to (1) concentrate on the most beneficial areas, (2) more easily manage development, and (3) eliminate the need for concurrent old ad new systems. One phase of the DDM plan considered inputting transac- tions from regional offices by optical scanning devices to a central data processing center, which would process transac- tions and provide master record inquiry capability to regional offices. The D study, dated September 1971, partially addressed manual operations in the VA regional offices, but TMI pointed out that both the data processing and manual functions of the CP&E delivery system were complex and had to be studied to 9 allow them to be adequately integrated. Accordingly, TMI recommended that the redesign of the CP&E system be studied and that emphasis be placed on integrating the two functions. VA's decision-to develop-and implement the-TargetSstem On February 2, 1972, the Administrator of Veterans Af- fairs designated a task force to (1) determine the feasibility and merit of a comprehensive versus a gradual approach to re- designing the system, (2) prepare a detailed plan and time schedule for the total system redesign, if appropriate, with rough estimates of cost, and (3) specify cost evaluation criteria for the new system. In March 1972 the task force submitted an interim report proposing short-range solutions to the CP&E processing prob- lems. In April the task force submitted to the Administrator its final report, which discussed long-range objectives re- garding the CP&E system. This task force report said that about 95 percent of the CPE claimants and beneficiaries were serviced satisfactorily by the existing CP&E system. However, the report noted that the small percentage of cases not effectively serviced by the compuiter system disproportionately impaired the system image. The task force concluded that service to veterans was being impeded by file folder control poblems, slow access to status information on claimants, error rates and repeat processing of cases, and delays in obtaining data from sources outside VA. The task force devised and recommended an "ideal" or "target" system, which could serve as a design goal for VA. The system was intended to reduce or minimize costs of providing CP&E services, improve service to benefi- ciaries, and have the flexibility to incorporate system changes and advances in computer technology. The task force envisioned the use of terminals with cathode ray tube display for inquiry and data input at the VA regional offices. The task force also recommended organi- zational changes in the regional offices; minicomputers to edit input at the field level before large-scale processing; and immediate processing of transactions to completion by a central or combination regional/central computer system. In addition, the task force recommended that a comprehensive re- design study team be designated to complete a comprehensive system redesign within 18 months. This study would seek to 10 automate all work processes except those requiring human judgment or intervention; send claims folders to storage for historical purposes; and generate all possible reports as a byproduct of computer processing. The task force considered a DDM feasibility study com- pleted in November 1971 for the use of optical scanning devices for inputting CP&E transactions--one phase of DDM's plan to improve the CP&E system. (See p. 9.) In separate appraisals made in April 1972, both the task force and TMI concluded that optical scanning device technology was way behind that of key entry processing methods and did not provide for an incremental growth to a future on-line data-base-oriented system. The task force also criticized the methodology used to prepare the cost justification in the optical scanning device feasibility study. (See ch. 3.) The task force also concluded that, if the Target System concept were adopted by VA, interim changes to the CP&E system should be carefully evaluated in terms of their consistency with Target unless they provided overwhelming short-term advantages. Although the task force recommended a comprehensive re- design of the CP&E system, it did not say why this approach, instead of a gradual redesign approach, was chosen. The task force did not estimate the total cost of this system. It said that the total cost could not be determined without a feasibility study of the system's full potential impact on existing operations. VA Administrator-establishes-Target System as agency policy goal In flay 1972 te Administrator solicited comments from the Director of Veterans Benefits, the Chief Data Management Director, the Controller, and the Assistant Administrator for Planning and Evaluation about the task force recommendations. These officials generally agreed with the task force recom- mendations to establish the Target System as a policy goal. The Chief Data Managt:ment Director did not agree that optical scanning input devices should be excluded from tb Target System for input where an overnight response, rather than immediate processing of transactions, was adequate. He also commented that the task force's and TMI's appraisal of the optical scanning device feasibility study erroneously treated the proposal as if it were the only action planned 11 to improve service or reduce costs, rather than only one of several steps eventually leading to an on-line direct access system for CP&E processing. The Controller and the Assistant Administrator for Plan- ning and Evaluation concurred in the task force recommenda- tions but cautioned that careful justification was needed. The Controller suggested that VA undertake an indepth cost- benefit feasibility study before the proposal was adopted even as a policy oal. After evaluating the task force's report and the agency officials' comments, the Administrator in July 1972 adopted the Target System as a policy goal and designated a design team to complete a comprehensive system redesign by Decem- ber 31, 1973. The Ad:ainistrator said that the Target System would remain the agency policy goal until full implementation was achieved, unless factors emerging from design or cost/ performance studies revealed the need for modification or a different approach. VA DESIGNS THE TARGET SYSTEM In October 1973 the VA study team, with substantial tech- nical assistance from TMI, completed its conceptual design of the Target System. The effort was originally to be completed in December 1973 but was accelerated at the request of the Office of Management and Budget. The design study addressed the objectives of the system, the field organizational en- vironment in which it would operate, the operation and limita- tions of the current CP&E system, the flow of work processes in the field offices, general equipment and programing re- quirements, and a schedule for system development and imple- mentation. The proposed design fit in with VA's plan to reorganize activities at the VA regional offices and establish Federal Regional Centers, in which certain field activities would be centralized and which would link VA's central office and regional offices. The design study team envisioned placing computers in each Federal Regional Center to interact with the computers at Hines and provide support to terminals in VA regional offices and Federal Regional Centers. The desiin also included specifications and an implemen- tation schedule for a pilot test of the Target System. The major objectives of the pilot test, to begin in September 1974, 12 were (1) to test the feasibility of man-machine interaction in claims handling and other support areas, (2) to test and evaluate the responsiveness, timeliness, and accuracy of benefits derived through automation of the processing and control of claims in the field, (3) to provide, as a test vehicle, the basis for the ultimate procurement of equipment and supporting programs to satisfy local computer processing requirements in support of the Federal Regional Center con-- cept, and (4) to provide the means for assessing the impact of the Target System on the Hines and Austin data processing centers. The study team defined numerous specific objectives for the Target System, including (1) reducing the duplication of effort in data input generation and output report production which existed in field operations, (2) enhancing control over claims and other work in process, (3) providing immediate access to master records to support veteran inquiries and adjudicative actions, thus reducing the need to obtain infor- mation from veterans' claims folders in the regional office files, (4) reducing the overall processing cycle of claims, (5) developing a flexible system design to be responsive to VA's efforts to reorganize field operations, (6) increasing the amount of automatic letter preparation, (7) providing for an improved nonstandard reporting capability, and (8) improving the control of claim folder movement within VA regional offices. The design study report did not address alternative approaches to achieving these objectives, other than the system that it proposed. The design study team assumed that the computer systems at the Hines Data Processing Center (two International Busi- ness Machines 360/65 computers), with certain equipment im- provements and additions, were sufficient to accommodate the Target System workload. However, the study team cautioned that this assumption would have to be validated by further analysis. It should be noted that the Target System objectives were generally stated in the task force and the design study reports. Many of the objectives focus on faster processing of claims, reduction of errors, and improved services, but they were not expressed in quantifiable terms. Accordingly, without quantifiable goals VA had no basis for comparing alternative approaches which might have improved service and speeded up processing within reasonable ranges with a smaller capital investment. 13 CHAPTER-3 VA HAS NOT CONCLUSIVELY DEMONSTRATED THAT THE TARGET SYSTEM IS ECONOMICALLY-JUSTIFIED After VA established the Target System as a policy goal, an aggregate analysis 1/ of gross cost savings expected to result from the Target System and four detailed cost-benefit analyses were made. These analyses 'nmpared only the Target System with the current CP&E system. he validity of the results of the aggregate analysis a the first cost-benefit study, both of which showed that Target would result in major cost savings, were questioned by the VA Controller's office. Later escalation of cost estimates cast further doubt on whether savings would be achieved. ater VA studies showed areas of uncertainty which suggest that VA has not conclusively demonstrated that the Target System will result in major cost savings over current CP&E processing methods or that it is the most cost-effective approach to meeting its CP&E processing needs. Our analysis o Vis cost--benefit study completed in February 1977 shows that major cost savings might be real- ized. However, Target is a risky venture and its cost could exceed its monetary benefits. Further uncertainty is indicated oy the fact that VA has yet to develop a total implementation plan identifying specifically tasks to be accomplished and their schedule, an:. associated resource requirements and costs. VA'S RECOGNITION OF-NEED-TO IMPROVE COST-BENEFIT METHODOLOGY When TMI made its 1971 study of the effectiveness and ef- ficiency of VA's data processing (see p. 9), it criticized VA's use of cost-benefit studies of automatic data processing proposals. TMI said these studies suffered from a lack of credibility. TMI found that such studies frequently overstated projected savings, understated projected costs, and failed to 1/An aggregate analysis deals with gross costs of the present and proposed systems to provide management with a reasonable basis for comparison. It does not deal with specific information as to where these costs will accrue. 14 explore alternative systems. TMI noted that in many cases the studies tended to be perfunctory exercises, while in other cases they were used as sales devices by proponents of the system. Accordingly, TMI recommended that VA develop suitable cost-benefit criteria for assessing automatic data processing system proposals. In 1972 the CP&E redesign task force (see p. 10) confirmed TKI's findings that VA cost-benefit studies were deficient and lacked cred.ility, and cited the need for VA to establish suitable cost-evaluation criteria for assessing all systems proposals. This finding was based on the task force's review of a cost-benefit analysis of a January 1972 DDM feasibility study proposing the expanded use of optical scanning devices to replace outmoded flexowriter equipment 1/ for inputting CP&E transactions. The task force considered this study to be typical of VA cost-benefit analyses at that time and cited such deficiencies as overstated savings, understated costs, failure to independently verify costs, and failure to explore and compare alternatives. As a result, the task force recommended that future cost-benefit analyses -- be completed independently of any organization having a vested interest in the proposed system; -- be made by teams supplied with the expertise needed to independently and objectively evaluate and compute basic data; -- seek alternatives which will identify maximum results at minimum cost; -- start from the present level of staff, equipment, and other resources to properly compare present and pro- posed systems; and -- identify the effect of proposed systems upon pre- determined objectives. The extent to which these guidelines were followed by TMI and VA in their cost-benefit analyses of the Target System are discussed below. l/Flexowriter equipment produces paper tape for inputting transactions to the CP&E system. Until recently, paper tape was the primary input medium for the system. 15 RESULTS OF AGGREGATE ANALYSIS OF EXPECTED COST SAVINGS FROM TARGET SYSTEM In conjunction with the Target design study mentioned in chapter 2, in July 1973 TMI prepared an aggregate analysis analysis of cost savings expected from the Target System. This focused on gross comparisons of the costs of the CP&E system save and the Target System to estimate how much Target might over current processing pro- in annual CPSE operating costs cedures. TMI estimated that tne Target System would involve might save a capital investment of $41 million and that it estimated annually, with an pay- a net of about $16.7 million back period of about 2.5 years. The results of the aggregate analysis were presented in the Target design study report. was The report cautioned that a more detailed cost comparison costs necessary to ascertain more specific information on where scope will accrue and that, for a system of the magnitude and cost had to of Target, expected operational performance and be continually reexamined. u VA Controller's office q estioned validity of aggregate analysis of- The Director of Budget Service, in the Controller's his staff's fice, reported to the Control'er on the results of review of TMI's aggregate cost analysis of the Target System 1973. Based on its review, the Budget Service in November TMI staff listed major Target System costs not considered by used as the basis for and certain questionable assumptions savings estimates. The Budget Service's findings suggested that the expected higher Target System capital investment would be $5.2 million as than indicated by TMI, annual savings might be as much would $10.4 million lower, and TMI's estimated payback period be extended from 2.5 to 7.3 years. In addition, the Budget Service Director noted that TMI had failed to apply guidelines required by Office of Management and Budget Circular A-94, which prescribe that costs and/or benefits should be dis- value counted 1/ to their present value to recognize the time that l/The purpose of discounting is to account for the fact at the future costs and benefits become less significant present time the further into the future that such costs and benefits are projected. Therefore, money the Government to expends now or in the immediate future is of more concern the taxpayer than money which is to be spent several years from now. 16 I of money in its overall evaluation. The Budget Service Dir- ector said that applying such present value techniques would probably push Target's payback period further beyond the Service's revised estimate of 7.3 years. The Budget Service Director added that, since his staif's findings could reduce the net savings estimate for the Target System, it would be helpful if part of the justification for Target were based upon solid improvements in service to veterans. He advised that quantified measurements of such improvements be properly documented and used t help sell the Target System. JULY 1974 COST-BENEFIT-ANALYSIS In July 1974, TMI completed the first detailed cost- benefit analysis of the Target System. TMI determined and compared projected Target costs and savings for fiscal years 1974-84 in order to evaluate the economic impact of the sys- tem's development and implementation. Although TMI corcluded that substantial savings above the current CP&E processing would accrue over the life of the Target System, it did not consistently measure costs and benefits over the projected life of the system. As a result, the VA Controller questioned whether significant savings would occur. Methodology- and- findings TMI analyzed costs and benefits of four different regional computer center configurations of the basic Target System design in order to establish the fundamental cost relation- ships among the system's various site, communications, and hardware capacity configurations. The Target System config- urations analyzed by TMI were made up of one, two, four, and eight regional computer centers. The TMI study did not address alternative approaches to automating the functions in VA's field offices. TMI determined that projected savings from the Target System would accrue primarily from reductions in personnel resulting from revised and eliminated operations associated with VA's CP&E field operations. All personnel savings would be achieved by (1) a decreased need for certain categories of staff and (2) personnel attrition over a period of 2 to 5 years after Target implementation. TMT calculated total one-time Target System investment costs for the four configurations, but made no specific recommencation about the preferred Target System configura- tion. 17 Since costs and benefits would be timephased over the life of the Target System, TMI considered the time value of money associated with each possible investment decision. TMI broke out the costs and benefits associated with each Target configuration and the current CP&E system on a year-by-year basis for fiscal years 1974-84 and applied a discount factor to these projections to translate expected costs and benefits into their present value. TMI's analysis for four configurations of regional computer centers is shown below: 1 2 4 8 center centers centers centers -(millions) One-time invest- ment cost $40.6 $42.4 $50.3 $62.4 Net present value benefit or deficit (-) 19.0 17.5 6.2 -8.0 Inconsistent treatment of costs and benefits In estimating present value net benefits from the Target System, TMI assumed that wage rates would increase at an average rate of 6.5 percent per year over the period covered by the analysis. Using this assumption, TMI estimated unit and hourly costs for fiscal years 1976 and 1980 wh_ ' were applied to the volume and staff-hours for those years to de- velop total costs. The costs for those 2 years were used to establish a trend line, from which cost estimates for other years were determined. Consequently, the figures for present value net benefits in the TMI analysis include a factor for inflation in the costs of CP&E operations that Target would eliminate. This could be a legitimate approach to calculating benefits only if the Target System costs were also inflated before making any comparisons. However, TMI did not apply any- inflation factor in calculating Target costs and thus significantly distorted the benefit-cost ratios for the Target System. This distortion is important because over 50 percent of the savings TMI calculated were directly attributed to in- flation factors rather than operational changes or improvements expected to result from the Target System. 18 Tn August 1974 the VA Controller brought TMI's incon- sistent treatment of costs and benefits to the attention of the General Manager, Target System Project Management Office. The Controller noted that the savings from CP&E costs to be eliminated by Target were inflated at a 6.5-percent annual rate to reflect salary increases, but that annual Target System costs through fiscal year 1984 wre estimated in constant dollars with no adjustments for inflation. The Controller had determined that consistent measurement of costs and benefits for a two regional center Target System configuration and inclusion of more recent estimates of telecommunications costs would have shown a present value net deficit of $19.9 million over the planned period of Target development and operation. The de- ficit for configurations of four and eight regional centers would have been even greater. In August 1974 the General Manager, Target System Project Management Office, responded that his office had revised the TMI cost-benefit analysis to bring it more in line with accepted practice regarding present value net benefits. The costs were also revised to resolve questions regarding expected telecom- iounications costs and to reflect early purchase of all Target hardware rather than long-term rental, which had been included in the initial TMI analysis. As a result of these modifications of the TMI analysis agreed upon by the Controller's office and the Target System Project Management Office, the TMI present value net benefits for a two regional computer center configuration were reduced from $17.5 million to $2.3 million for the period covered by -he analysis. This resulted in a benefit-cost ratio of 1.04_ to 1, barely over the break-even point. Thus, with just two regional centers, it became apparent that the expected present value net benefits resulting from implementation of the Target System would not be great and that any later escala- tion of projected Target costs might eliminate the readjusted Target savings. Significant escalation-of projected Target System costs In January 1975 the Federal Computer Performance Evaluation and Simulation Center, under an interagency agree- ment with VA, completed a computer sizing study 1/ as a basis 1/A sizing study uses computer programs that compare simulated processing requirements against models of selected computers. These models contain hardware and software characteristics of available vendors' equipment. 19 for estimating the size and cost of systems available from various vencors to handle the volume and type of processing required by the Target System. The sizing study indicated that more powerful equipmernt than tha assumed by the design study team would be required to proce3i the Target workload. Based on the results of the Simulation Center sizing study and more current data on projected CP&E workload require- ments, VA decided that Target should consist of four regional comp:lter centers and a central computer facility, designated as the Target Central System. As a result, in about March 1975 VA reestimated the cost of a four regional computer center configuration at $94.9 million, which represented almost a 90-percent escalation of one-time investment costs from the July 1974 TMI estimate of $50.3 million. The VA Controller's office reviewed these escalated Target System costs and analyzed their impact on projected savings expected from Target. The res' lts of this analysis were documented in an April 4, 1975, memorandum from the Con- troller to VA's Associate Deputy Administrator. The Con- troller's office had recomputed the projected Target System net benefits by applying the TMI cost-benefit model to a four regional computer center configuration and incorporated the March 1975 escalated Target cost estimates. This resulted in a present value net deficit of $39.7 million over the - projected period of system development and operation. The Controller concluded that (1) justification for further de- velopment of Target should be tied to expected service im- provements rather than a favorable benefit-cost ratio and (2) if the viability of Target was questionable, cost-effective alternatives must be developed for improving the present CP&E system. As a result of the Controller's memorandum, an executive review committee responsible for overseeing the Target System development discussed whether the system would produce a sav- ings or a deficit. As a result of this concern, it was decided in April 1975 that VA's Office of Planning and Evaluation would make a cost-effectiveness study analyzing the partially opera- tional pilot system to validate and firm up expected Target savings. FEBRUARY AND SEPTEMBER 1976 COST-BENEFIT ANALYSES In February 1976 the Management Engineering Service of VA's Office of Planning and Evaluation completed a cost- benefit analysis of a Target System configuration of four regional computer centers. VA had established that four centers 20 was the optimum configuration primarily because (1) more vendors would be able to compete for he system and (2) four centers would supply superior reliability, which would result in the lowest overall cost. In July 1976 we reported to the Chairman, Subcommittee on HUD-Independent Agencies, House Committee on Appropria- tions, / that VA did not have a sufficient basis for con- cluding that four regional computer centers was the most cost-effective configuration. In June 1976 the Target System request for proposal was amended to provide for three regional computer centers. In September the Engineering Service updated the February study to reflect revisions in costs resulting from the reduction of regional centers from four to three and a 6-month delay in operation of the system. The Engineering Service reported the following cost-savings elements: February 1976 September 1976 (4 regional (3 regional computer-centers) computer centers) One-time develop- ment cost $81,691,000 $77,336,000 Gross annual savings $32,554,841 $33,097,310 Present value net benefit $42,685,225 $41,448,470 Benefit-cost ratio 1.45 1.47 In making these studies the Engineering Service was hindered by time constraints, the limited number of opera- tional applications in the pilot system, and the limited in- formation available at the time of the studies. As a result, the Service had to base much of its analysis on assumptions, technical estimates, and limited measurements of pilot opera- tions. Assumptions, technical estimates, and limited measurements are typical constraints that cost-benefit analyses must contend with. Because such analyses are subject to error, the effect of these potential errors should be tested through such 1/"VA Justification for Establishing Four Regional Computer Centers for its Proposed Tr.rget System" (HRD-76-145, July 13, 1976). 21 techniques as sensitivity analysis (see p. 26) to determine their impact on decisions reached. The potential effect of such errors was never addressed by the Engineering Service in the February and September 1976 studies. Our analysis of these studies showed several limitations that indicated considerable uncertainty as to the cost-benefit posture of the Target System. Measurements-between-Target-and-the current- CP&E -system were- not -comparable In measuring cost savings attributed to Target, the Engineering Service used various estimating techniques, including actual measurements of pilot operations in Phila- delphia and Baltimore. About 30 percent of the estimated annual Target savings were based on the measurements obtained from the pilot operation; the other savings were based on technical estimates. When the Engineering Service did make actual measure- ments, it compared its measurements to CP&E work rate staziards for regional offices reorganized to process claims in unitized or team approach in the current system, and calculated cst savings for those tasks where pilot improved operations. How-- ever, the Engineering Service's measurements of pilot operations did not encompass all elements used to develop the work rate standards for reorganized regional offices. A consistent measurement comparing a current work rate standard for claims processing against the pilot operation of that processing would require that the current standard and the pilot measure- ment cover the same work elements. Since this was not the case, either the current standard or the pilot measurement should have been adjusted to allow a valid comparison. Such an adjustment would reduce the reported Target cost savings. For example, almost 30 percent of the current work rate standard for education claims processing pro ides for such activities as (1) reviews and cosignatures of adjudicators' disposition of cases by senior adjudicators and (2) miscel- laneous claims actions and correspondence actions associated with claims processing. The Engineering Service measured only one specific type of claim under pilot--original education claims involving institutions of higher learning--and these claims neither required reviews by senior adjudicators nor included the variety of miscellaneous claims actions or correspondence actions provided for in the current work rate standard for education claims processing. 22 The Engineering Service assumed that Target would eliminate most miscellaneous claims and correspondence actions because they would be automated and that the time involved to handle those items not eliminated would be inconsequential. Considering the volume of claims that will be processed under Target, comparing current standards for processing, which include all those items, with pilot measurements not adjusted for even a small percentage of these itelis would overstate estimated savings. Measurements were-based on-limited-observations About 30 percent of the estimated annual savings for Target were based on measurements of the pilot operation. The measurements, however, were based on such a limited number of observations that their representation of the ef- fects of the Target System was questionable. For example, the Engineering Service based its estimates of time to process original education claims in pilot on only five measurements of one operator in Baltimore and one measure- ment of a hypothetical case constructed for one operator in Philadelphia. Such a small sample was hardly sufficient to provide enough insight into the broad range of claims actions that the Target System will face. However, even these few observations, together with a self-maintained log kept by one operator in Baltimore for about 8 months, indicated that the actual time it would take to process a claim under the Target System would vary greatly, even when only one operator was being measured. There would undoubtedly be even greater variability if more than one operator were measured. Further, the measurements also indicated that the processing times under the Target System could actually exceed the times the Engineering Service used in its analysis. Three of the five actual observations and the overall average time developed from the self-maintained log exceeded the average time the Service used in its analysis. Also of concern to us was the manner in which savings were determined for claims other than education claims. Although only one type of education claim was measured, the Engineering Service study team assumed that the Target System would achieve the same savings for all 21 categories of CP&E claims. In addition, the team calculated savings for machine processing of calculations now performed manually. To the extent that processing procedures for other claims differ from the one type of education claim measured by the 23 Service, the calculated savings for these claims would be erroneous. This was in fact the case, because in a cost- benefit analysis of Target completed in February 1977, 15 of the 21 categories of CP&E claims were sufficiently developed in pilot to provide a substantially broadened base for the Service to measure claims processing. In 9 of the 15 categor- ies of claims measured, the actual processing times exceed those assumed in the February and September 1976 studies. Costs-were not-independently verified or-analyzed One pitfall to avoid in making cost-benefit studies is allowing some organizational bias to enter into the procedures used to estimate the costs of alternative systems. The Engin- eering Service was not careful enough in this respect, for it accepted, without independent verification, cost estimates prepared by the Target development staff and did not make any independent analyses. In the February and September 1976 studies by the Engineering Service, one-time costs for the Target System were presented as follows: February 1976 September 1976 Personal service $18,867,600 $16,076,000 Travel 1,362,000 935,300 Contractual service 2,172,000 2,574,000 Communications 12,200,000 6,827,000 Site preparation 3,203,000 3,878,000 Computer rental 10,365,000 8,921,000 Computer purchases 32,717,000 37,505,000 All other ' -805,000 - 620,000 $81,691,000 $77;336,000 A particular concern was with one-time costs associated with the categories of personal service, travel, and contractual services, which totaled $22.4 million and $19.6 million, respectively, in the February and September reports. These costs included tasks associated with system design, programing, testing, installation, and training necessary to familiarize employees with Target operations. Costs associated with such tasks can be expressed as software costs. Department of Vete- rans Benefits officials did not provide an itemized breakdown of these costs and explained that they were developed on the basis of past experience with systems development. We could 24 not address the various elements that make up these software costs, but our experience and that of other Government agen- cies in systems development indicated that these costs may have been substantially understated. Recent computer industry studies and the experiences of other Government agencies indicate that the cost to develop software exceeds the cost of computer equipment. One study shows that equipment costs represent less than 25 percent of the total costs of designing, developing, and operating a new system. A Rand Corporation study indicates that by 1985 equipment costs will represent less than 5 percent of total system development costs. In addition, the General Services Administration's Automatic Data Processing Strategy Study-- completed in March 1975 by ncisions and Designs Incorporated-- indicates (1) that developing application software has become the largest cost item of automated data processing and (2) that if trends continue, software is expected to account for 90 percent of all costs by 1980. Further, a 1974 study of all Department of Defense data processing costs by the Institute of Defense Analysis indicates an average software to equipment cost ratio of 1.7 to 1. The results of these studies are borne out by our experience with other Federal agencies. The experience of some agencies shows that the ratio of software costs to equipment costs ranges from 1.7 to 1 to 3.5 to 1. The VA estimates were about 0.5 to 1 in both studies, based on estimated automated data pro- cessing equipment purchase costs of $41.7 million and $42.8 million, 1/ respectively. We recognize that the circumstances of Target development may differ from general experience. However, March 1977 VA estimates show that the software costs were substantially understated in the 1976 studies. l/The purchase costs shown in the February and September cost- benefit analyses were $32.7 million and $37.5 million, respectively. These costs represent purchase prices less a 50-percent rental credit on the equipment VA plans to lease for awhile before purchase. The other 50 percent of the rentals are shown in equipment rental costs. Thus, total cash outlays will be the same as expected purchase prices. 25 Lack-of-sensitivity-analysis Sin ost important decisions involve elements of uncer- tainty, cosL-benefit analyses should explicitly address this uncertainty. One method of addressing uncertainty is sensi- tivity analysis, wherein the analyst tests the effect of changes in values of key uncertain factors on the ranking of alternatives. Several savings elements in the cost-benefit analysis were based on technical estimates and were therefore subject to measurement error and management bias. Where measurement samples of pilot operations were taken to deter- mine estimated savings, problems of sample representation and sampling error existed. As previously discussed, there also was some uncertainty about the cost of the proposed system. These uncertain elements should have been subjected to sensi- tivity tests to determine whether Target's apparent cost-benefit was solidly based. We raised our concern about sensitivity analysis with the Engineering Service study team. The team members said that the February and September 1976 studies did not include any sensitivity analysis because they believed that their estimates of Target savings were very conservative. Therefore, they saw no need to subject their results to sensitivity tests. However, the need for sensitivity analysis has since been borne out by the results of the Service's third cost-benefit analysis in February 1977 and later events which have shown several shifts in VA's pro' ,ns of Target costs and savings. In its February i cost-benefit analysis, the Engineering Service included a sensitivity analysis to acknowledge uncer- tainty in projected Target costs and savings based on estimates. However, as discussed on p. 31, we found this sensitivity analysis to be incomplete because it did not fully probe all major areas of uncertainty that might substantially affect the estimate of Target costs and savings. COn October 28, 1976, we met with the Administrator of Veterans Affairs and several other VA officials to discuss our observations on the cost-benefit justification of the Target System. We noted that: -- VA did not fully explore alternatives to Target for improving or redesigning the current CP&E system from a cost-benefit point of view. 26 -- VA needed to define in quantifiably measurable terms its objective of improving service to the veteran and to examine alternative approaches for accomplishing this objective. -- Our analyses of VA cost-benefit studies cast doubt on the cost effectiveness of the Target System. The Administrator and the other VA officials present did not agree with our observations and reiterated the agency's commitment to proceed with Target System procurement with- out delay. In a letter dated November 23, 1976 (see app. I), the Administrator of Veterans Affairs responded to our findings as follows: -- VA has considered systems alternatives both before and after it adopted the Target System as an agency policy goal. -- VA will be able to extensively improve service to the veteran through Target. -- VA's cost-benefit analysis is fully supported by a valid set of assumptions and remains the agency's clear posi- tion until it is updated in the future. To emphasize the agency's position, the Administrator enclosed documents to refute our observations about VA's failure to study alternatives and the need for VA to define in quantifi- ably measurable terms its objective of improving service to veterans and to explore and examine alternative means of accom- plishing this objective. A VA document entitled "Target System Alternative Con- siderations" (see app. I), cited numerous items as alterna- tives considered since 1971. Three of the items have been discussed--a September 1971 paper on redesign of the CP&E system (see p. 9), consideration of a Target System con- figuration composed of eight regional computer centers (see p. 17), and reduction of the configuration from four to three centers (see p. 21). A fourth item, mentioned on page 59, does address another approach for processing CP&E benefits in VA. This approach, presented orally by a vendor to the VA pilot/target project management staff in April 1975, envisioned placing terminal controllers in each VA regional office with an inquiry 27 capability to a central data base. VA ruled out this approach, concluding that (1) the terminal controllers ere for a special purpose and would have no value after the regional offices were closed each day and (2) it needed larger, general-purpose computers in its various data processing centers to replace obsolete equipment. This argument implies that the Target System computer will be used for functions other than CP&E benefits claims processing. However, VA's cost-benefit justification and the request for proposal address only the CP&E system, and VA does not have any definitive plans for other processing functions for Target. The other items cited by VA on pages 55 to 60 as system alternatives do not directly address our findings that VA did not fully explore alternatives to the Target System for improv- ing or redesigning the current CP&E system from a cost-benefit point of view. None of the items are substantive CP&E system alternatives to Target, and VA apparently made no cost- effectiveness analyses for any of these items. Another VA document, "Non-Monetary Savings from the Target System" (see app. I), that the Administrator enclosed provides a summary of the material contained in the February 1976 Management Engineering Service cost-benefit analysis. This document cited nonmonetary benefits to veterans, VA, and others as a result of what VA terms improvements in service expected through the Target System. However, VA has not stated which of the service improvements and other nonmonetary benefits cited are exclusive to Target and which might be provided by other approaches and at what comparative cost to the taxpayer. This document does not directly address our finding that VA needs to define in quantifiably measurable terms its objective of improving service to the veteran. FEBRUARY- 1977 COST-BENEFITANALYSIS In February 1977 the Engineering Service reported the results of its third cost-benefit analysis of Target based on data collected in January 1977. The Service reported the following cost savings. 28 One-time development cost $ 81,941,000 Net recurring savings over total system life a/$211,619,000 Present value net benefit $ 55,93.4,599 Benefit-cost ratio 1.46 a/These savings represent estimated recurring savings, once Target is implemented, less recurring operating costs. One- time system development costs were not reflected in this figure. The present value net benefit and the benefit-cost ratio do include consideration of the one-time development costs expected to be incurred after October 1i,1976. COne- time costs of $12,407,000, incurred before October 1, 1976, were considered sunk costs and were not included in either analysis. As in the February and September 1976 studies, the Engineering Service operated under tight time constraints. Because of this, the estimates of expected savings to be achieved from Target are ased largely on 1 week's data collection efforts, including work sampling at Philadelphia's pilot operation and the collection and analysis of cost data at VA regional offices in Chicago and Philadelphia and a com- puter center in Hines, Illinois. Costs and manpower savings associated with regional office operations in Target were calculated for seven areas of activity: authorization, rat- ing, input, files, finance, veterans service, and administra- tion. We made a sensitivity analysis that addressed only the February 1977 report. After the release of the February report, VA made changes in its estimates of costs and benefits which acknowledged some of the observations of our sensitivity analysis. Because VA was still assessing Target costs and implementation schedules, we did not redo our analysis. Target^is-still-a-risky-venture in-terms-oftcoat-effectiveness Our analysis of the February 1977 study disclosed limi- tations which show that there is considerable uncertainty about the costs and benefits of the Target System and that it is still a risky venture in terms of cost effectiveness. This uncertainty is based on the following: -- The Engineering Service's work sampling of pilot opera- tions did not include an allowance for nonwork actions (personal, fatigue, and delay time) on some of the 29 activities. These activities were so numerous that the exclusion of this allowance greatly overstated the savings projected for the area of authorization. The Service acknowledged this error and amended its savings estimates. (See p. 34.) -- A large part of the manpower savings attributable to Target was based on work sampling measurements of one adjudication unit in Philadelphia whose personnel did superior quality work. Accordingly, this unit's claims-processing times may not be representative of similar units in Philadelphia or in other VA regional offices. (See below.) -- The projected savings could be overstated because of errors inherent in the sampling procedures and technical estimates used to determine savings and costs. These errors were not fully considered by the Service's sensitivity tests. (See p. 31.) -- One time costs shown in the report were substantially understated. (See p. 32.) -- The personnel savings projected for the early years of Target did not consider the VA policy on computer installations, which provides that personnel will not be terminated because their jobs are automated. Since personnel savings are more likely to be attained pri- marily through attrition, the Service's estimates of savings in the early years of Target were questionable. In March 1977 the savings projected for fiscal years 1979 and 1980 were adjusted by the Service to reflect that policy. (See p. 34.) Basis-for-a-large-part-of savings may-not-be-representative The Engineering Service did work sampling and made other technical measurements of the pilot operations in Philadelphia to estimate projected savings from the Target System. About $60 million of the projected net savings of $211 million were based on measurements of one adjudication unit processing claims under pilot. According to Philadelphia regional office officials, personnel of this unit, one of nine adjudication units in Philadelphia, did higher quality work than personnel in the other units. Accordingly, the claims-processing ef- ficiency of this unit is not representative of all units in the Philadelphia regional office and may not be typical of other adjudication units throughout the country. 30 The Engineering Service team claimed that the sampling of this superior unit is compensated for by the fact that the pilot system was not operating efficiently during the time of measurements. Therefore, the system's performance did not approach the expected minimum level of performance for Target. Thus, they believed that pilot represents a "worse case" Target situation. The pilot operation has been functioning since September 1974, yet after more than 2 years VA is clearly still having trouble demonstrating the achievement o the level of re- sponsiveness it expects from the Target System. Accordingly the validity of the Service's assumption will depend on the ability of the winning vendor to demonstrate that Target will be as responsive as expected. Estimating-errors inherent -in proected-savings - not fully considered -by Engineering Service-sensitivity- analysis The greatest proportion of the savings projected by the Engineering Service to be achievable from Target were based on work sampling and technical estimates. Since no estimating procedures can precisely estimate savings to be achieved from a project like Target, errors are inevitable. Although the Service included a sensitivity analysis in the February 1977 study to acknowledge the impact of estimating errors, it did not show the possible negative impact on savings in all cases in which sampling and technical estimates were used. The negative impact of estimating errors should be addressed to fully assess the degree of risk in basing decisions on uncer- tain values. The sensitivity analysis did not acknowledge the possible negative effect of inherent errors in work sampling. The work sampling was done for 15 different types of claims processing in the authorization area. The size of the samples of many types of claims was not sufficient to provide precise estimates; therefore, the range of sampling error is extremely broad. Other parts of the Engineering Service analysis involving sampling erro':s are the wage rates used to measure cost savings in the authorization and files areas in the VA regional offices. The Engineering Service used average wage rates to compute the estimated staff-years saved. Implicit in this approach is the assumption that staff-years saved will be distributed pro rata throughout the various grade levels of the total work 31 force affected. However, Target development personnel told us that savings would be concentrated in lower grade positions. Finally, the Engineering Service sensitivity analysis did not show the impact of potential error on the present value of proceeding with the Target System. Since the benefit- cost ratio of 1.46 was calculated from the present values of costs and benefits, the effects of uncertainties should have been treated on the same basis to provide the decisionmaker with a consistent basis for comparing values affecting his decision. One-time- costs-of-Target-were substantially- nderstated The Engineering Service reported one-time development costs of Target at $81.9 million. As in the 1976 cost-benefit studies, our particular concern was with personal services, travel, and contractual services associated with system design, programing, testing, installation, and training necessary to familiarize employees with Target operations (software costs). These costs totaled $24.3 million. Omitted from these costs were costs of programers involved with redesigning the cur- rent CP&E system, which will serve as an interim Target Cen- tral System until the redesign is completed. The Service classified these costs as recurring costs over the 10-year life of the system because plans were incomplete and the dura- tion of the effort was undetermined. These costs totaled about $15.4 million. Considering that VA has acknowledged the current CP&E system to be cumbersome and difficult to maintain, we questioned the openendedness of the redesign ef- fort. VA advised us in March 1977 that the redesign costs had been reclassified from recurring costs to the more appropriate category of one-time development costs. As a result the one-time costs estimated by VA in March 1977 were increased to $97.3 million. The classification of redesign costs as one-time develop- ment costs does affect Target's cost-benefit posture from an investment decision poirt of view, in which the time value of money is considered. bnowing these costs early in the sys- tem life would have a greater impact on present value costs, because the earlier these costs are incurred, the less they are discounted. 32 GAOSENSITIVITY -ANALYSIS We made a sensitivity analysis to test the effects of the uncertainties resulting from the limitations of the Engineering Service cost-benefit analysis. (See app. II.) Our analysis was directed at determining pessimistic estimates of costs and savings associated with the uncertain- ties of the Target System. This was done to show the degree of risk being undertaken by VA in proceeding with Target. We found that, because of the uncertainties, as much as $47.7 million in present value savings could be questioned and additional present value costs of as much as $8.9 million could be incurred. The Engineering Service study showed a Target S =em present value net bnefit of $55.9 million, with a benefit- cost ratio of 1.46. Our sensitivity analysis showed that these uncertain savings and costs could result in a present value net deficit of as much as $700,000. Our analysis did not consider possible slippages in implementation. If implementation schedules are not met, development costs will escalate further and place Target in an even worse benefit- cost posture. We recognize that the net savings from Target could fall between our pessimistic estimate and a level even greater than that shown by the Engineering Service study. There is no way of knowing the probability of the savings falling at a given point in this range. However, later VA revisions to the February 1977 study (see pp. 34 and 43) already sug- gest that savings will be less than projected by the Engin- eering Service and these revisions address only some of the savings we are questioning. VA may disagree with our estimates of the potential effect of uncertainties--because we also made assumptions. Our anal- ysis was made merely to show that there is considerable uncer- tainty about the cost effectiveness of the Target System. DEVELOPMENTS-SINCE-THE-FEBRUARY 1977 COST-BENEFIT-ANALYSIS- CAST-MORE UNCERTAINTY ON THE - COST-EFFECTIVENESS - OF - TARGET After the February study, the Engineering Service amended its report in March 1977 to correct a work sampling measurement error and to present a more gradual initiation of savings in the early system life. The result has been a 33 significant reduction in Target's benefit-cost ratio. Further VA was still uncertain about the progress of Target development, cost estimates, and the feasibility of the plan and schedule for implementing Target. After we raised ques- tions on VA's plan for converting to Target, VA made an as- sessment of the above matters in April 1977. The VA assess- ment resulted in a revised implementation schedule for Target and a related update to the cost-benefit analysis. The up- dated cost-benefit analysis further reduced Target's benefit- cost ratio. Despite VA's efforts to revise its implementation schedule and related cost-benefit analysis, uncertainty still remains. Results-of-amendments-to the February977-study In March 1977 the Engineering Service amended its February 1977 cost-benefit analysis to correct a measurement error in its work sampling of pilot operations that w had noted. (See p. 29.) The report was also amended to reflect the impact on savings of a more gradual phasing in of regional offices to Target operations than that assumed in the February report. VA's analysis now provides for gradual personnel reductions attained primarily through attrition and recognizes that not all regional offices serviced by each regional computer center can become fully operational simultaneously, as was assumed in the Engineering Service's February 1977 study. In March the Service reported the following cost savings. One-time development cost $81,941,000 Present value net benefit $25,571,816 Benefit-cost ratio 1.21 Events-leading-to VA's April-1977 assessment of-the-Target-situation In its February 1977 study, the Engineering Service projected costs and savings based on equipment installation dates and schedules contained in a Target Central System conversion plan that VA provided to the House Committee on Appropriations in NovemDer 1976. The Service cautioned in its report that the installation and operational dates shown in the plan were critical and that any slippages would affect total costs and savings. The objective of the conversion plan was to provide VA regional offices with all the claims-processing functions that have been implemented and demonstrated by the pilot 34 system as early as possible. The plan presented staffing requirements, cost estimates, and schedules associated with (1) translating programs of the current CP&E system so that they can be used on the equipment of the winning Target vendor and serve as an interim Target Central System until a new cen- tral system is developed and (2) continuing development of pilot programs and converting these programs so they can be used on the new equipment in the regional computer centers. The plan also addressed the effort that would be required to redesign the interim Target Central System, although no schedule was set up for its completion. The redesign would provide for more efficient operation of the Target Central System, allow for easier and cheaper maintenance, and permit direct access to any record in the system. We found the conversion plan to be incomplete in many respects. We advised VA that a more comprehensive and de- finitive identification of conversion tasks and associated resources was needed to give VA a basis for realistically estimating conversion and development costs and to provide assurance that the schedule was realistic. Also, the plan did not address how and by whom the conversion would be man- aged. Adequate monitoring and control of the implementation of a system as complex as Target is essential to insure that schedules will be met, that costs will not escalate substan- tially, and that savings expected from Target can be initiated when desired. We briefed VA and various congressional offices on our observations about this plan in March 1977. VA acknowledged that further work was needed and said that two contracts had been awarded to address incomplete elements of the plan. After we briefed VA, the Administrator of Veterans Affairs instructed the Chief Benefits Director and the Chief Data Management Director to formulate a plan by April 1 for trans- ferring responsibility for Target development from the Depart- ment of Veterans Benefits to the Department of Data Management. A Target Implementation Planning Team was formed, composed primarily of Department of Data Management personnel from the various VA computer centers and including Target user represen- tatives from the Department of Veterans Benefits. The team's mission was to assess the Target implementation plan to deter- mine what has been accomplished and what needs to be done, potential problem areas, cost estimates, schedule feasibility, and milestones to be used for management control. The team's work was completed in April 1977 when the team reported its findings to VA management. 35 The team concluded that, in view of the status of the development of Target, VA's implementation plan posed sub- stantial risk to both data processing and regional office operations. The team developed a revised implementation schedule which will reduce the benefit-cost rat p of Target further to 1.18 but is a less demanding schedule aimed at reducing the risks of disruption of regional office and data processing operations. A detailed discussion of the revised implementation schedule and related cost-benefit analysis is contained in chapter 5. To date, VA has still not developed a comprehensive plan addressing specifically the tasks to be accomplished in implementation, their schedule, and associated resources and costs. Accordingly, uncertainty still remains regarding costs, progress of development, and the feasibility of VA's schedule for implementation of the system. 36 CHAPTER 4 STATUS OF THE CURRENT CP&E SYSTEM As VA noted (see app. I), when the Target System was conceived, the computer systems at Hines were overtaxed. The situation grew progressively worse as a result of vari- ous factors, such as increasing workload and failure to make needed system improvements for more efficient equipment operation. In May 1976 VA installed a large-scale computer system which has greatly alleviated the problems at Hines. VA reported to various congressional parties in November 1976 that, as a result of the installation of this computer, bene- fit payment cycles have increased to 8 to 10 per month for Compensation and Pension and 7 for Education; in January 1976 the Compensation and Pension payment cycles were down to 4. Further, VA said that the Hines Data Processing Center had reassumed responsibility for computer processing which it had earlier been forced to contract for with private vendors. Other improvements have been made in the CP&E system. From November 1974 to February 1976, a firm under contract to VA (1) translated several major CP&E programs to a pro- graming language compatible with the equipment's technology to facilitate maintenance of the programs and (2) improved these programs to increase their efficiency. The flexowriter machines used for input to the CP&E system presented increasingly serious maintenance problems. These machines were no longer being manufactured and parts were difficult to obtain. They are being replaced by more modern input equipment, which should reduce errors in the input process and alleviate problems caused by malfunction- ing equipment. In addition, VA budget projections indicate that the CP&E workload has leveled off and will decrease in the future, primarily because of the expiration of a 10-year delimiting period for education benefits for nearly 3.7 million veterans. These changes suggest that the Hines Data Processing Center is no longer in the crisis situation it was in up to 1976. As a result, Hines apparently has the capability to process the CP&E workloads should Target implementation be delayed until the uncertainties regarding its cos t effec- tiveness compared to viable alternatives are resolve 37 CHAPTER 5 CONCLUSIONS; AGENCY COMMENTS AND OUR EVALUATION, AND RECOMMENDATIONS CONCLUSIONS VA has proceeded with the development of the Target System without cost-effectiveness studies of alternative approaches for making improvements to meet its CP&E system processing needs. Throughout Target's development, the system's proponents have argued that the CP&E system is inefficient and that serv- ice to the veterans is suffering. However, VA had not estab- lished quantifiable goals or standards for improved service. Without such goals or standards, VA had no basis for comparing alternative approaches and for determining whether there were systems less complex and less expensive than Target that could effectively meet its CP&E processing needs. Since VA has not made economic analyses of alternative approaches--a major element in cost-benefit/cost-effectiveness methodology--its economic justification for Target can address the cost effectiveness of the system only in relation to cur- rent processing procedures. VA established the Target System as a policy goal although its total system costs were not addressed. Thus, VA has proceeded with the design and development of the system without giving itself the flexibility of considering viable alternative approaches should the cost not justify the benefits received. Several cost-benefit analyses and related updates of the Target System were made. Our review of these analyses shows that VA has not conclusively demonstrated that the Target System will result in major cost savings over current CP&E processing methods or that Target is the most cost-effective approach to meeting CP&E processing needs. As a result, VA cannot assure the Subcommittee on HUD-Independent Agencies that t- system will be measurably cost effective compared to an efficient manual or evolutionary-developed system. Target is a risky venture in terms of cost effectiveness. Our sensitivity analysis of VA's February 1977 cost-benefit study shows considerable uncertainty, which could result in Target's cost exceeding its monetary benefits. 38 There is still uncertainty about the progress of develop- ment, cost estimates, and the feasibility of VA's plan and schedule for implementing Target. A definitive and complete implementation plan is needed to resolve these uncertainties. The contract for Target's data processing equipment, currently scheduled to be awarded in July 1977, should not be awarded until these matters are resolved. VA COMMENTS AND OUR-EVALUATION By letter dated May 19, 1977, the Administrator sent us VA's comments on our report. The comments consisted of a c ver letter, a revised implementation schedule and support- iri documentation, statements of VA positions relating to various parts of our report, and minor editorial suggestions. The documentation accompanying the Administrator's letter totaled some 70 pages. Much of this material is detailed and extraneous information not germane to the issues discussed in our report. All pertinent information has been appropri- ately considered and incorporated in this report. Accord- ingly, we have included as appendix III only the Administra- tor's cover letter and have summarized the pertinent contents of VA's revised implementation schedule and related cost- benefit analysis, and the relevant parts of the statements of VA positions. Cover-letter The Administrator stated that he concurred in our pro- posal to establish, in measurable terms, goals and standards for improving benefit claims services to veterans. He said he will identify goals, objectives, and performance standards for the Target System which will be quantified before in- stallation and against which the system benefit improvements will be evaluated. We believe that the Administrator's action is a necessary step to gage the system's success if Target is to be imple- mented, but it is not the action that we were proposing VA take. What we suggested, and have been suggesting is that VA establish, in measurable terms, goals all along, and standards for improving benefit claims services to veterans before it can adequately determine what type of system it ne-edsto meet those goals and standards at the least cost. As noted on page 8, VA has never adequately defined, in measurable terms, goals or standards for improved service for either the present 39 system or Target against which to consider the extent of system improvement or redesign necessary to meet its proc- essing needs. The Administrator also said that, in his judgment, the outstanding questions regarding Target have been satisfac- torily resolved and that VA should proceed immediately to make Target a reality as soon as possible. He said that, since he became Administrator on March 2, 1977, he and key VA officials have given top priority to a thorough study of the Target System. He stated that this involved a detailed examination of all aspects of the Target implementation sched- ule, the projected cost benefits, and all unresolved manage- ment questions raised by the Subcommittee on HUD-Independent Agencies, House Committee on Appropriations. 1/ The Administrator said that VA officials presented to the Subcommittee staff on April 25, 1977, a revised Target implementation schedule and supporting documentation for its information and analysis. We discuss the revised schedule on page 41 and the related cost-benefit analysis on page 43. The Administrator claimed that the revised schedule is achievable with very low risk of disrupted service to the veteran population and that VA's analysis of the schedule demonstrates conclusively that Target will be cost beneficial. He stated further that the revised schedule and related cbst- benefit analysis should resolve any uncertainties about the progress of development, costs, and implementation cited in our report. We do not believe that these uncertainties or the un- certainty about the cost effectiveness of Target cited in our l/On January 24, 1977, the Subcommittee Chairman wrote to the Administrator raising several questions concerning Target. Points that the Chairman said had to be resolved by VA before awarding a contract for Target included (1) the need to insure that Target is measurably cost effective compared to an efficient manual or evolutionary-developed system and to develop a sound practical program to achieve the savings claimed, (2) the need for a comprehensive list of quantifi- able service improvements that will be made possible by Target and based to the extent possible on VA's experience with the pilot program, and (3) the need for a complete conversion plan for Target, including the redesign of the current CP&E system, related cost estimates, and a comprehensive time-phased schedule. 40 report are resolved. Our reasons are set forth in the following sections dealing with the revised implementation schedule and related cost-benefit analysis, and the state- ments of VA's positions on the matters discussed in this report. Revised target-implementation schedule As part of its reponse, VA provided a modified implemen- tation schedule and a cost-benefit analysis associated with this schedule. The pertinent updated cost and benefit esti- mates included with this revised schedule have been in- corporated in this report. This implementation schedule does not directly address the primary focus of our report, which questions the economic justification for the proposed Target System. Instead, this schedule presents a general strategy for Target implementation and the costs associated with that strategy. The schedule contains a lot of material about the general strategy for the Target implementation and lists the various steps to be performed and the projected dates for their accom- plishment. Since most of this material is irrelevant to the subject of our report, we are summarizing only the pertinent parts. Basically the schedule calls for installation of the three regional computer cnt rs and the interim Target Central System in fiscal year 1978. Operations would begin in the Target Central System and the first regional center in April 1978, in the second regional center in June 1978, and in the third in July 1978. VA will implement basically an inquiry capability, which will allow regional offices to access the CP&E master files and the records locator system file through the terminals installed in those offices. Cer- tain other functions which are primarily parts of other Target subsystems will also be implemented. Other Target subsystems will be implemented in phases through fiscal year 1979 after prototype testing at the Philadelphia pilot station. The current CP&E system will serve as an interim Target Central System until a new central system is developed. (See p. 35.) The previous schedule called for implementation of all developed Target functions at the first regional computer center in April 1978. The Implementation Planning Team con- cluded in April 1977 that, in view of the status of system development, this approach posed substantial risk to both data processing and regional office operations. The team said that 41 regional office operations could be disrupted by implementing all functions simultaneously without adequate leadtime for training and organizational preparation. Additional risk would be incurred by installing in those regional offices not familiar with the system, functions that were not fully field tested in the prototype regional office. The Administrator pointed out that he was convinced that the new implementation schedule was achievable with very low risk of disrupted service to veterans. We agree that the revised schedule is less demanding han the original schedule and is directed at reducing the risks of disruption of regional office and data processing opera- tions during Target implementation. We cannot assess the feasibility of this implementation schedule because, at the time the report went to printing, VA had not developed a total implementation plan containing a detailed identification of project tasks, the scheduling of these tasks, and associated resource requirements and costs. Further, VA acknowledged that a contractor was still working to estimate more accurately the amount of work re- quired to develop, convert, and redesign the Target System. We believe that, in the absence of a total implementation plan, uncertainty still remains with regard to the cost and status of Target development and implementation. Statements of-VA positions A VA document entitled "Statements of VA Positions," attached to the Administrator's May 19 letter, contained 23 pages of statements of VA position~ on 11 major .reas of VA concern about matters discussed ir our report. Some of the material was not responsive to the points we raised. Thus, rather than include all the documentation, we have summarized and commented below only on VA statements per- tinent to the issues we raised. Cost-benefit analysis On page 14 we stated that VA has not conclusively demon- strated that the Target System will result in major cost savings. VA responded that no cost-benefit study can "conclusively demonstrate" what the savings will be and that such is not the purpose of a cost-benefit study. Rather, it is an analysis of whether savings to be achieved justify the costs. 42 Although we agree'that a cost-benefit analysis cannot conclusively provide a precise projection of savings, we believe that it can conclusively demonstrate through adequate sensitivity analysis that in the most pessimistic case the system will be cost beneficial within a range of magnitude. Our sensitivity analysis of VA's February 1977 cost- benefit study shows a degree of uncertainty about the savings to be achieved and that Target might well result in a deficit. Revised cost and benefit estimates developed by the Engineering Service in April 1977 in accord with the new implementation schedule were as follows: One-time development cost $100,875,970 Net recurring savings over total system life a/$173,506,609 Present value net benefit $21,613,933 Benefit-cost ratio 1.18 a/These savings represent estimated recurring savings, once Target is implemented, less recurring operating costs. One-time system development costs were not reflected in this figure. The present value ..et benefit and the benefit- cost ratio do include consideration of one-time development costs expected to be incurred after March 1, 1977. One-time costs of $14,626,76) incurred before March 1, 1977, were considered sunk costs and were not included in either analysis. VA maintains that its April 1977 update demonstrates as conclusively as any cost-benefit study can that Target will be cost beneficial. VA provided a sensitivity analysis of its April 1977 update, but this analysis had the same short- comings as the February 1977 study in that it showed neither the possible negative impact on savings in all cases where sampling and technical estimates were used nor the impact of potential error on the present value of the option of proceed- ing with Target. VA noted our sensitivity analysis of the February 1977 cost-benefit study and said that the probability that our most pessimistic estimate will become a reality is one chance in 1024. But further discussion with VA personnel disclosed that this statement is misleading. The computed factor of 1024 represents the probability that Target's cost- benefit position will be worse than the pessimistic case 43 postulated by our sensitivity analysis. As we stated on page 33, we acknowledged that the net savings from Target could range between our pessimistic estimate to a level even greater than that shown by the February 1977 study but said that the probability of the savings falling at any given point in this range cannot be determined. - It should be noted that in two later updates of this study VA has greatly reduced the estimated present value net benefit for the system from $55.9 million in February 1977 to $21.6 million in April 1977. VA cited several categories of possible additional Target savings not addressed in the February 1977 study and later updates. These additional savings are not quantified and are not supported by any indepth VA analysis. Further, VA does not say which of these additional savings are unique to Target or which could be achieved by alternative approaches. Alternative approaches On page 8 we said that VA did not make economic analyses of alternative system approaches to Target for meeting its CP&E processing needs. In response, VA cited three alternative schedules for Target implementation which it deemed of sufficient merit to prepare separate cost-benefit analyses. However, VA's eco- nomic analysis of implementation schedules does not directly address our point that VA did not study and make economic analyses of alternative system approaches for improving or redesigning the current CP&E system. VA contends that the Target approach is cost effective and that any alternative which changes the basic nature of the Target approach would never offset Target sunk costs and the ensuing years of delay. Because VA has not made economic analyses of alternative approaches, it has no way of knowing to what extent Target sunk costs might be recoverable through an alternative system approach. In March 28, 1977, testimony before the Subcommit- tee on HUD-Independent Agencies, House Committee on Appropria- tions, VA officials said that VA has not determined how much of the sunk costs would be applicable if it are to develop alternative approaches. 44 Standards-and definitions On page 8 we said that V has not defined, in measurable terms, goals and standards for improving benefit claims serv- ices to veterans. VA responded that standards for measuring both processing timeliness and pending workloads for the current system have been in effect for over 10 years and that a well-defined work measurement system has also been operating. VA then provided five pages of information on its standards for its existing mode of processing CP&E benefit claims. Our report does not dispute the existence or use of standards for the current CP&E system. Our report simply points out that standards and goals for improved services that VA seeks to attain were not established to provide a basis for determining the most cost-efective improvements to the current system. In fact, VA acknowledges in its statements on this point that overall timeliness goals, which VA considered to be important for Target, have not been established. Target-versus-efficient manual system On page 1 we cited the position of the House Committee on Appropriations that Target should be justified principally on a demonstration that i would measurably improve VA's internal efficiency over the best performance obtainable under the current system. VA, in response, cited a comparative study of its Roanoke regional office operation and the Philadelphia prototype Tar- get regional office. Based on its observations VA concluded in this study that certain nonmonetary benefits would be achieved from Target (for example, under Target CP&E benefi- ciaries would receive initial payments and adjustments to their awards faster and faster responses to their inquiries about their claims). It should be noted that the Roanoke regional office is only a part of the current total P&E claims-processing system, which encompasses both manual operations at the re- gional offices and their interaction with and reliance on the computer operations at Hines. In separate statements regard- ing computer support of its current CP&E system, VA acknowl- edges that the computer operations at Hines use computer 45 programs and operational techniques conceived largely in 1959 and 1960, and cites weaknesses of the current computer system that can be associated with this 18-year-old design. We stated on pace 37 that the difficulties at Hines were caused in part by VA's failure to make needed system improvements for more efficient operation of the equipment. In view of this, we question that the operation of the Roanoke regional office, which relies on the support of the Hines cputer system, represents the best-performance obtainable under he current system. Existing-CP&E system service VA made note of our statement on page 10 that an April 1972 task force report, which served as the primary basis for specifically adopting the Target System as a policy goal, acknowledged that the exi ting CP&E automatic data processing system serviced 95 percent of the veterans satisfactorily. In response VA stated that, even though 95 percent of the claimants were being serviced satisfactorily under standards effective at that time, VA data processing prob- lems were becoming severe and service was deteriorating. VA cited an increase in compensation workloads and a dramatic increase in education workload which peaked in fiscal year 1976. We acknowledged on page 20 that workload projections changed and that this contributed to VA's deter- mination in 1975 that more powerful equipment would be re- quired to process the Target workload. Target-cost-escalation In responding to a statement on page 20 of our report, VA noted t.iat in March 1975 Target's one-time development costs were estimated at $94.9 million for a four regional center configuration and have escalated only $5.9 million in 2 years. VA stated that, considering inflation and changes in the implementation plan, the escalation is reasonable and attests to the relative accuracy of the present cost esti- mates of $100.9 million. VA seeks to convey a picture of stability in cost estimating, but in fact cost estimates have fluctuated considerably over the past 2 years. In Febru:ry 1976 VA estimated one-time costs for a four regional center cnfiguration at $81.7 million. This esti- mate was reduced in September 1976 to $77.3 million, reflect- ing a three regional center configuration. VA reestimated 46 its one-time costs in March 1977 at $97.3 million and again in April 1977 at $100.9 million. Thus, VA's one-time cost estimates have fluctuated dramatically over the 2-year period. VA Controller's cost statement On pages 19 and 20 we discuss the VA Controller's office findings relating to TMI's cost-benefit analysis of the Target System completed in July 1974. VA responded that after the TMI study the scope of Target was greatly increased and that in March 1975 cost estimates were developed reflecting that increased scope. VA said that, since there were no benefit figures reflecting the increased scope, the Controller used the benefits reported in the TMI study when it considered the escalated costs of Target in April 1975 and concluded that Target would result in a deficit. Thus, a February 1976 study by VA's Engineering Service reflecting the increased scope of Target showed sub- stantial savings. We discussed this statement of position with VA person- nel, who advised us that the scope of Target was not increased other than the equipment expansion required partly because of increased workload which we described on page 20. Rather than the scope of Target being changed, the methodology em- ployed in the later cost-benefit analysis differed from that used in the TMI analysis. Regional data processing center configaration We said on page 21 that VA had established that four regional centers was the optimum configuration for Target but that VA did not have sufficient basis for concluding that this was the most cost-effective configuration. VA replied that in early 1976 it began experiencing a declining workload for CP&E claims. This decline allowed VA to reduce the workload projections for Target, and studies made reflecting the reduced workload verified the feasibility of reducing the number of regional computer centers from four to three. Accordingly, the Target request for proposal was modified to reflect the reduced number of centers. The additional studies referred to by VA were made after inquiries by the House Committee on Appropriations regarding VA's justification for establishing the four regional computer center configuration for Target. Further, the studies referred 47 to by VA as reflecting the decreased workload and verifying the feasibility of three regional centers did not in fact reflect the decreased workload. In our report on VA's jus- tification for establishing four regional computer centers for Target (see p. 21), we questioned the workload projec- tions used in these studies and recommended in July 1976 VA reappraise its workload requirements. VA agreed with that the recommendation. Other processing functions for the Target System On page 28 we stated that VA rled out a possible alter- native system approach to Target presented orally by a vendor in April 1975 because the type of equipment the vendor sug- gested was special-purpose equipment that had no value to VA for applications other than CP&E benefit claims processing. We pointed out that VA has no definitive plans for other processing functions for the Target System. VA repeated our statement that the request for proposal and cost-benefit justification do not include any processing requirements other than for CP&E benefit claims and alluded to fiscal year 1977 House Appropriations Committee hearings, at which maximizing equipment use was discussed. VA did not refute our statement that it lacks any defini- tive plans for other processing functions for the Target System. Distribution of personnel savings We state on page 3 that, on the basis of our sensitivity analysis, wage rates associated with cost savings in the au- thorization and files areas in the VA regional offices were subject to sampling error because they were based on the as- sumption that staff-years saved will be distributed pro rata throughout the various grade levels of the total work force affected. VA contended that its assumptions are valid but acknowledges tnat a precise average wage can be determined only after a job reclassification. We state that such a dis- tribution was not likely to occur because we were told by Target development personnel that savings would be concen- trated in lower grade positions. If, in fact, staff reduc- tions do occur at the lower grade levels, VA cost savings based on average annual salaries could be overstated. 48 Status of the current CP&E system VA agreed that the crisis situation at Hines has been contained. (See p. 37.) VA pointed out, however, that the underlying causes of its past problems at Hines have not been corrected. It cited the awkwardness of the current computer system that resists changes and deficiencies related to com- puter controls as two of the problems VA must contend with. VA said that service delays experienced by the current system because of its slow responsiveness to inquiries and correction of rejected transactions will be shortened con- siderably by Target. We acknowledge that Target will cut down response times between the VA regional offices and the central system at Hines. However, we must also point out that for several years operations will rely on the interim Target Central System, which will basically be the current Hines computer system operating on different equipment. Many design prob- lems which VA acknowledges result in an awkward system resis- tant to change will remain until the Target Central System is redesigned and installed, a task presently scheduled for com- pletion in September 1983. VA contends that Target should not be delayed because there is no apparent pending disaster. We are not suggesting that this is the only issue involved. What we are suggesting is that VA is in a position to buy time until the uncertain- ties cited in our report are resolved. RECOMMENDATION TO THE ADMINISTRATOR OF VETERANS AFFAIRS We recommend that the Administrator establish, in measurable terms, goals and standards for improving benefit claim services to veterans. RECOMMENDATIONS TO THE CONGRESS We recommend that the Congress not permit VA to award a contract for the Target System until VA has established, in measurable terms, goals and standards for improving bene- fit claims services to veterans as a basis for determining the most cost-effective approach for meeting benefit claims processing needs. 49 If the Congress determines that VA has satisfactorily resolved the above issue, we further recommend that VA not be allowed to award a contract until it has resolved the un- certainties about progress of development and the cost ana implementation schedule of the system. To resolve these un- certainties VA needs a definitive and complete implementa- tion plan, which it did not have at the time this report went to printing. 50 CHAPTER 6 SCOPE OF REVIEW Our review was limited to examining VA actions to explore alternatives to the Target System and analyzing the adequacy of cost-benefit studies to justify the system. In meeting these objectives, we reviewed Office of Man- agement and Budget, General Services Administration, Depart- ment of Defense, and VA documents concerning the nature and conduct of cost-benefit analyses; the design, development, implementation, and operation of automatic data processing systems; and the acquisition of automated data processing and related equipment. We concentrated on VA policies and procedures regarding the design of the Target System. We interviewed various officials and staff members with responsibility for the design and development of the Target System and cost-benefit justification for the system. We reviewed pertinent VA documentation for the cost-benefit analyses of Target made by VA and Technology Management, Incorporated. 51 APPENDIX I APPENDIX I VETERANS ADMINISTRATION r OFFInc o TH ADMINISTRATOr OF VTRANs AFFAIRS WASHINGTON, D.C. 204 . November 23, 1976 Mr. Gregory J. Ahart Director of Human Resources Division U.S. General Accounting Office 441 G Street, N.W., Room 6864 Washington, D.C. 20548 Dear Mr. Ahart: Several days ago, during the course of your briefing about the TARGET System, I promised you that I would provide you with a response to two of the key points contained in your briefing. Enclosed are three documenus which I believe should have a great bearing on the content and context of your forth- coming report. I. A paper "TARGET SYSTEM ALTERNATIVE CONSIDERATIONS" summarizes material illustrating that serious and significant effort has been expended to consider systems alternatives prior to the adoption of the TARGET System as an agency policy goal. Subsequent to the adoption of the goal, other alternatives have also been considered. II. A paper "NON-MONETARY SAVINGS FROM THE TARGET SYSTEM" provides an executive summary of the material contained in the February 1976 Cost/ Benefit Analysis, Chapter V. It also contains other more recent observations which indicate that VA will be able to improve service exten- sively through the installation of the TARGET System. Especially noteworthy is our assumption that overpayments can be reduced through the faster interaction of employees with the master records. We expect to be able to interrupt benefits quicker when schooling is dropped or questions arise. 52 APPENDIX I APPENDIX I III. The COST/BENEFIT ANALYSIS (Feb. 1976) has often been referred to in our conversations. I ask that you refer specifically to Chapter V. Note on page 54 the lapsed time savings for various claims processing circumstances. Incidentally, I want to reiterate that the entire Cost/Benefit Analysis is fully supported by a very valid set of assumptions. It remains the clear position of the agency until it is updated at some future time. Upon reviewing this material I am sure you will conclude, as I have, that the TARGET System will save the agency hard dollars as well as provide an improved level of service. Sincere a X &IIoate Da~miy AdmjiiWa RICHARD L. UDEBUSH Administrator Enclosure 53 APPENDIX I APPENDIX I TARGET SYSTEM ALTERNATIVE CONSIDERATIONS The question of the consideration of alternatives to the VA TARGET SYSTEM has been raised on many occasions. Recently we have had a briefing by GAO (10-28-76) which indicated to the Administrator and his Staff that a GAO Report would soon be released indicating that alternatives to the TARGET SYSTEM have not been considered in the light of emerging factors. A significant part of the monologue is concerned with challeng- ing past cost-benefit studies. In order to set the stage for consideration of the issue of alternatives there needs to be a review made of the Agency relative to benefit delivery throughout the period of the past several years. This period has been characterized by: A. The Viet Nam Era G.I. Bill (1966) B. Expanded VN Era G.I. Bill (1967) C. Expanded Compensation Programs (1967-76) D. Expanded Pension Programs (1956-76) The period was also marked by increasingly difficult legis- lation, requiring additional computer capacity. The pension programs were characterized by the need for incremental benefit adjustments based on changed income reports; new classes and rates of burial benefits have been provided; clothing allowances have been authorized. The G.I. Bill Education programs were revised markedly in 1972 to provide for prepayment of benefits and for advanced payments at the beginning of each term. Reporting fees for institutions were added and were subsequently modified; additional classes of eligibles have been added to the G.I. Bill Education program. Entitlement periods for G.I. Bill benefits were extended several times. Until this year, FY (1977), workloads have generally increased markedly as well, creating an environment of much more work with the work being generally more difficult to administer. During the period the workforces have been increased as well. In 1973 a whole new group of employees was created to handle the G.I. Bill problems on the college campuses. These Veterans 54 APPENDIX I APPENDIX I Representatives on Campus have performed a liaison function at the schools, advising veterans of their rights and serving to assist the veterans with any problems they have concerning their G.I. Benefits. About 1200 of these positions are now in existence. Other employees have been added throughout the regional offices to handle the lerger workloads. In the ADP environment, the Education Programs were imple- mented without significant equipment enhancements (1966). Not until the turn of the decade was a serious effort undertaken to make a general systems improvement for the Benefit Delivery Systems. In March 1971, an analysis was published of the prospects for a distributed system using OCR processes and local printers. Later, in November 1971, the feasibility study was completed with the recommendation that the Administrator approve the use of remote OCR terminals which would completely change the input structure as well as provide other changes which would result in major changes in ADP in support of the CP and E programs. Concurrently with the OCR studies, a review was conducted resulting in a published statement dated September 21, 1971, providing for proposed changes to the CP&E systems which would (1) combine all files, (2) accelerate communications from regional offices to the Hines DPC, (3) migrate master files to direct access storage and (4) control pending issues and mnage- ment reporting. This alternative plan was not implemented since consultant advice was accepted to use state-of-the-art technology. The CP and E systems were greatly overtaxed during the 1969-70-71 period. A Task Force was appointed to study the problem and make recommendations for solution. Their March 1972 report recommended significant staffing increases at Hines DPC as well as the addition of another large scale computer. The processing equipment at Hines after the addition of the newly authorized one was: one IBM 7080 computer two IBM 360/65 computers It was understood that the IBM 7080 system would be phased out since it was generally agreed that it had reached obsolescence. In July 1972, the Administrator formally disapproved the OCR System proposed in 1971. It was suggested that an interim CRT-type system for input should replace our old flexowriter system. This has been done, with a Sycor process, competitively procured, having been adopted in the past several months. 55 APPENDIX I APPENDIX I The level of dissatisfaction with service to VA benefi- ciaries was very pronounced. The Task Force, assembled from a cross section of the Agency, and utilizing consulting services as well, ade three long range recommendations: 1. A Dt i interactive on-line terminal target '.,l a policy goal. 2. . ~r: a d:- .gn team represented by all elements i t Ager, '. 3. Comptete redesign study in 18 months. These missions were carried out as planned resulting in a comprehensive set of published .design documents in October 1973. The documents were submitted to he Administrator who approved the Target System as conceptualized. Briefings were immediately conducted for OMB, for Congressional Committees and for veteran organizations. The Agency utilized consultant services (TMI) in the develop- ment of the TARGET plan. Concurrently with the TARGET SYSTEM planning, the Agency was involved with an internal evaluation of the Department of veterans Benefits, especially the methodology of and organization for delivery of benefits. Internal plans were made to reorganize several of the ROs along a functional, less compartmentalized basis. The test reorganization began in late fall, 1972. The Hay Associates, a well-regarded consultant service company, was retained to review the test work which had been done and to perform an independent "Organization Study of the Regional Office Structure." Their study in 1973 confirmed the advisability of restructuring the regional office organization to a unitized or team approach. Their report also recommended a long-term attention to integrate new technology into the benefit delivery process, and further emphasis be placed on factors which would prepare the regional offices for the Target System. During this period, workloads were steadily increasing and dissatisfaction with ADP performance and service also increased. In an effort to ameliorate the difficulties with punched paper tape input, interim alternative approaches were either imple- mented or tried: 1. OCR input processes were expanded, especially for the Education System. The experience was not good. 56 APPENDIX I APPENDIX I 2. Electronic transmission of paper tape data was tried. It failed in a test environment although a modicum of advantage was gained when the paper-tape reader test equipment at Hines DPC proved to be better than the ones previously in use. 3. Electronic transmission of reject messages was attempted, but failed primarily because of volume and the difficulty of separating priority messages produced by the systems. [It should be noted that the Pilot System has performed very well in achieving the results sought in paragraphs (2) and (3) above.] Workloads continued to grow faster than budget forecasts indicated they would. During the time the Target System Design Study was under development much conceptual material was considered in the process and alternatives were constantly considered, although certainly the policy had been established and the direction was clear. Distributed mini-computer systems were frequently dis- cussed but were rejected as being largely unprogrammable and unavailable for other general purposes of the agency. In shot., the computer resource would have been so fragmented as to have little residual value (after 5 P.M. and on weekends) for the Agency. At this point in time, the Agency included the Target System as a key policy goal in its array of Presidential Manageme c Objectives. It was approved. As part of the Agency's process of providing ADP long range plans, an alternative was proposed which would have altered the course of development of the entire ADP process and organi- zation in the VA. A long range plan was proposed which would have resulted in the primary ADP applications: PAID, AMIS, LOAN GUARANTY, Compensation and Pension, Education, etc., being distributed to the several regional DP facilities in a decentralized mode. This alternative to the conceptual Target System was rejected because it was seen as being fragmentary and pushing the state of the art then emerging regarding distributed data bases. As part of the Design Study Team proposal, the inclusion of a prototype Target System appeared for the first time. The purpose of the prototype, which we called the PILOT TEST System, was (1) to test VA employee reaction to an on-line ADP System, 57 APPENDIX I APPENDIX I (2) test the claims processing concept, (3) to provide a better basis for procurement. This pilot system itself provided for testing alternatives and design modifications. Ten months after approval, the PILOT System (September 1974) was operating in an Inquiry Mode exclusively in Philadelphia VAC and in Baltimore RO. See Attachment 1 for the continuing historical growth of the Pilot System functions which has been leading to a fully function- ing regional data processing support function. The Pilot System at first was planned for an IBM 360/40 Computer application. ther alternatives had to be considered when the VA-owned 360/40 was not made available. Alternatives considered were a leased 360/40, leased 360/50, leased370/145, and a VA-owned Honeywell H-250 computer installed in the Los Angeles DPC. The alternative chosen was the 370/145. Alternative PILOT sites were also considered: Boston, New York, Philadelphia, and Atlanta. Philadelphia was selected. The Inquiry System performed so well, and the workload and service-to-veterans problem was so severe, that the alternative was pursued to add 7 more regional offices to the Inquiry facility of the PILOT System. The seven offices considered were: New York, Washington, D.C., Cleveland, St. Petersburg, Los Angeles, San Francisco and San Diego. It was decided to provide inquiry processing for New York, Washington, and Los Angeles. This was accomplished in July/August 1975. In the Hay Report referenced earlier in this paper, there were recommendations, which were not adopted, intended to reorganize the DVB field activities into Federal Regional Center groupings. These large FRCs would have had significant concen- trated workloads, mostly clerical and/or routine in nature, that would have modified the Target System concept had the recommenda- tions been followed. In the early planning for an on-line inter- active system the alternatives of having ten (or eight) data processing facilities (RDPCs) were carefully considered. The number of RDPCs were reduced to four by the VA after third-party studies were conducted. This also provided for a consistent service/management approach since the benefits programs are administered in a four area geographical partition. Further alternatives were considered much later (January 1976) when the COMTEN Studies (Consultant Contractor) were performed indicating three RDPCs could handle the workload; sizing studies for a configuration of 2 RDPCs were also pursued. In 1973, our design assumptions provided for a full data base management systems support structure. There was a simulation test performed in 1974, which indicated multi- computers of the class of IBM MP 370/168 would be required in the regional data processing centers. Alternatives were immediately reviewed which led to the restructuring of the design assumption to provide for a mid-level DBM structure only; 58 APPENDIX I APPENDIX I this resulted in a change in computer class requiremen.s from 370/168 MP to 370/158 MP, with a resultant reduction in estimated cost of 10-12 million dollars. During this time ADP service to the CP&E programs continued to be unreliable even with the operation of the additional 360/65 computer system. The equipment enhancement process to provide an interim capacity for CP&E work was not adequate. Concurrently with the installation - the second IBM 360/o5 computer at Hines DPC the initial Target System planning assumption was formed that dual 60/6; systems could handle the workloa of the Target System, provided further significant enhancement of the hardware was made. An inter-agency working agreement had been established with FEDSIM early in the total Target effort. Sizing studies performed by FEDSIM indicated the Target System would require more power than dual 360/65 computers could provide. An alternative had to be developed which, when developed, indicated computers of the MP 370/168 class would be required. In April 1974, Control Data Representatives had proposed placing the entire VA file on random access files with an inquiry facility in all regional offices. The level of support for both hardware and especially software was so conditional that it was decided not to pursue this alternative. In late Spring 1974, the initial PILOT work indicated that inquiry capability would be brought on-line at Philadelphia and Baltimore about Septemb r 1974. IBM performed contractual services for the VA in order to assist us in getting the PILOT System demonstrated. IBM was paid to train and advise the programming staff on the techniques of top down structured programming and related documentation aids. IBM proposed to the VA that an inquiry network, using CRT input-output termin a.., be established to serve mary large regional offices with critical Educational System workloads. In discussions with IBM, their analysts would not commit themselves to an implementation schedule not later than October 1, 1974, so this alternative was not pursued. The agency has also met with several time-sharing and facilities management vendors in the past couple of years. Although we have expressed interest in such possible future arrangements, without definite commitments on the part of the VA, these possible suppliers do not closely pursue such prospects. In pril 1975, IBM proposed another alternative which provided for the regional office workloads tu be processed individually in each regional office using IBM 3790 technology. This proposal would have placed mini-computers in each regional office with an inquiry capability to a central data base. Although the proposal had some merit and was widely considered in the VA, it was rejected for two basic reasons: 59 APPENDIT I APPENDIX I 1. The mini-computers were special purpose, with the resource greatly fragmented so that such equipment had little or no value to the VA after 5 P.M. 2. This alternative did nothing to resolve the recognized problem of a longer-range computer replacement program; we have obsolete operating equipmert in Philadelphia and equipment approaching obsolescence in other locations. 60 APPENDIX I APPENDIX I Pilot Processing Installation Function Date Inquiry Master Record (CP&E) Sept 1974 BIRLS Sept 1974 Pending Issue Apr 1975 Status July 1976 BIRLS Delayed Responses July 1976 Change of Address Jan 1975 Change of Name Jan 1975 Change of Fiduciary Jan 1975 Education Special Payments Mar 1975 Original Education Awards Institutions Higher Learning Apr 1975 Non-College Degree Apr 1975 Apprenticeship Apr 1975 On-Job Training Apr 1975 Fliglat Dec 1975 Correspondence Dec 1975 Stop Payment (CP&E) Apt 1975 Suspend Payment (CP&E) Apr 1975 Resume Payment (CP&E) Apr 1Y75 Nottce of Death Sept 1975 61 APPENDIX I APPENDIX I Function Date Burial wards Oct 1975 Work-In-Process Establish Pending Issue Apr 1975 Cancel Pending Issue June 1975 Timeliness Reports Feb 1976 Over Standard Listings Feb 1976 Clear Pending Issue Apr 1976 Change Pending Issue Apr 1976 Diaries June 1976 Statistical Quality Review Jan 1977* Reject Selection Reports/Listings Feb 1977* Letter Writing Acknowledgement Apr 1975 Development May 1975 Education Award May 1975 Compensation Award Feb 1976 Pension ward June 1976 Disallowance June 1976 Original C&P Awards Compensation Feb 1976 Pension June 1976 Disallowances C&P June 1976 Education Ju.se 1976 Burial Oct 1975 62 APPENDIX I APPENDIX I Function Date Security Command Authority Sept 1974 Jurisdictional Rastriction July 1975 Sensitive File Access Sept 1974 Violations Logging Feb 1976 Dynamic PLssword Generation Sept 1974 Review and Disclosure Corrections to C&P Personal Data Dec 1976* Corrections to Ed. Personal Data Dec 1976* Corrections to BIRLS Data June 1976 Supplemental wards Compensation & Pension Rating Mar 1977* Income Mar 1977* Dependency Mar 1977* Other Nov 1976* Education Nov 1976* Reject Tran-.ission Apr 1975 Miscellaneous Claims Processing Record Folder Receipt May 1975 Interim Disposition Apr 1975 Evidence Review Apr 1976 Veterans Assistance Letters June 1977* *Tentative Installation Dates 63 APPENDIX I APPENDIX I N 0 N - M O NE T A R Y S A V I N G S F RO M T HE TARGET SYSTEM In February 1976 a Cost-Benefit Analysis of the Target System was released by the VA Office of Planning and Evaluation. The analysis/report was distributed very widely throughout the VA as well as throughout the government. In FY 1977 Budget it was Hearings (March 1976) the report was discussed broadly; introduced into the Senate record. The primary thrust of the analysis was in the area of hard dollar savings. However, a significant portion of the study, Chapter V, Non-Monetary Benefit nalysis, describes the following benefits: A. Benefits to the Veteran 1. Faster Processing of Claims. 2. Faster Response to status inquiries. 3. Faster payment of retroactive benefits. 4. Decreased waiting time and duration of interviews. 5. Expedited Change of Address. B. Benefits to the VA 1. Improved public image because of better service; reduced potential fcr error. 2. Reduction in complaints and related complaint processing. 3. Elimination of reject processing. C. Benefits to Others 1. Service Organizations also share in the ability to provide improved service. 2. Treasury Department should have fewer returned checks. 64 APPENDIX I APPENDIX 3. Equipment requirements (office furniture, machinery) will be reduced. 4. Value to Congress, especially in the reduction of complaints regarding service. Currently, we have been adversely impacted by a large number of overpayments, especially in the Educational Program. It is believed that quicker interaction, promised by Target System, offers a good opportunity for reducing potential over- payments. In the case of a veteran who is overpaid for one month, faster system interaction can intercept additional pay- ments which may now be paid because of slower transmission of the interruption notice. Additionally, we see other improved service characteristics of the Target System: 1. Elimination of the need for mailing data to and from the Hines DPC. 2. Reduction in the number of checks (and other correspondence) returned or delayed because of forwarding. 3. A management information system for claims pro- cessing which is more responsive than the current system and which includes daily updating thus permitting earlier detection and resolution of problem areas. 4. A method of self-training (Computer Assisted In- struction) with computer controlled graduation. CAI permits greater standardization of training while permitting each student to progress at his or her own pace. CAI also provides faster re- sponses to students' answers thus improving the efficiency of the training. 5. Employees will be face to face with the computer. They will manipulate it and engage it in two-way exchange of information. The computer will be- come more of a tool and less of a tyrant. 65 APPENDIX II APPENDIX II GAO SENSITIVITY ANALYSIS OF VA'S FEBRUARY 1977 COST-BENEFIT STUDY Since most important decisions like the proposed Target procurement involve elements of uncertainty, cost-benefit analysis of such procurements should address this uncertainty explicitly. One method of addressing uncertainty is sensi- tivity analysis, wherein the analyst tests several values for key uncertain factors to see how sensitive the ranking of alternatives is to variations in these factors. Several savings and cost elements in the Engineering Service's February 1977 cosC-benefit analysis were based either on as- sumptions, sampling, or technical estimates and, consequently, are uncertain. These elements should be tested to determine their effect on the potential cost effectiveness of Target. The Service did some sensitivity testing in its February 1977 study, but it neither tested all elements involving sampling and technical estimates nor showed the effect of its tests on the present value benefit-cost ratio. We made a sensitivity analysis to test the effect of the uncertainties disclosed by our review of the February 1977 study. Our analysis was directed toward determining pessimistic estimates of costs and savings to show the degree of risk involved in basing estimates on uncertain factors. Our analysis indicated that the estimated gross present value savings in this study could be overstated by as much as $47.7 million and that estimated present value costs could, conservatively speaking, be understated by about $8.9 million. Therefore, procurement of Target could re- sult ir. a present value net deficit of as much as $700,000, instead of the $55.9 million present value net benefit re- ported by the Engineering Service in its February 1977 study. Further, if software development costs approach the ratio of software to equipment costs experienced by some other Government agencies, the Target System present value net deficit could be even greater. We recognize that the circumstances of Target development may differ from genezal experience. Accordingly, our test of possible understatement of software development costs is more subjective than tests in the other areas. Our estimates of potential understatements of cct s and overstatements of savings were based on the following ssump- tions: 66 APPENDIX II APPENDIX II -- Y, areas where statistical samples were used, the ertremes of the confidence limits would give a reasonable worst case estimate of savings. -- In areas where technical estimates were used, a reasonable percentage of potential error would give a worst case estimate of savings and costs. -- In the area of software development cost, experience of other Government agencies indicates the possibility that software development costs could escalate beyond those currently estimated by VA, although this judgment is subjective. Details of our analyses are discussed below. SAVINGS ESTIMATES ASED ON SAMPLING PROCEDURES The Engineering Service based about $33 million of its estimated present value savings from Target on work samples in the authorization area. But the Service did not include in its work sampling of some of the tasks measured in pilot operations an allow- ance for nonwork actions. Accordingly, we tested for poten- tial error in savings estimates or authorization that would result from this omission. Also, our analysis showed that the Service's sampling estiaates are not very precise and that the range of estimates is extremely broad. Because of the short time the Service had to collect the data, the sample sizes of many elements were too small to yield precise estimates, and the Service did not test the potential overstatement of savings that would result from this lack of precision in sam'ling. We calculated statistical confidence limits in each of the elements measured. Using the lower limits of statistical confidence, we made a worst case projection of the possible impact of sampling error. We estimated that, because of the Serv 4;e's omission of nonwork actions and the lack of preci- sion in sampling, present value savings in authorization could be overstated by as much as $34.8 million. We also tested the Service's assumption that staff-years saved in the autbhrization and files areas would be distri- buted pro rata through the various grade levels of the total work force affected. Target personnel told us that savings would be concentrated in lower grade positions. We calculated 67 APPENDIX II APPENDIX II sampling errors in wage rates from this pro rata distribution and estimated that present value savings for Target could be overstated by as much as $674,000 for authorization and $122,006 for the functional area of files. It should be stressed that our estimates of potential overstatement of savings from sampling errors are based on the Engineering Service's assumption that the sample results from one superior unit in Philadelphia can be extrapolated to all VA regional offices. Since such an assumption is probably not valid, our estimate of potential savings over- statement in authorization could be conservative. SAVINGS BASED ON TECHNICAL ESTIMATES The Engineering Service tested the uncertainty of projected savings in three functional areas--input, files, and administration--because they were based on technical estimates. However, savings for a fourth area--veterans services--were also derived from technical estimates, so we estimated the potential effect of errors in that area. Our estimates in these four areas reflected the potential overstatement of present value savings as follows: Savings that may Functional area not be realized (millions) Files $1.0 Administration 1.5 Input 1.8 Veterans services 1.9 Total $6.2 POTENTIAL SLIPPAGE OF BENEFITS Since VA policy does not permit displacing personnel because of computer installations, such personnel savings could be achieved only through normal attrition and reassign- ment. The Engineering Service study assumed that personnel savings would be achieved within 6 months after installa- tion of the first regional computer center in regional of- fices served by that center and within 3 months after the installation of the second and third centers for regional offices serviced by those centers. 68 APPENDIX II APPENDIX II In March 1977 VA submitted a plan to the Subcommittee on HUD-Independent Agencies, House Committee on Appropria- tions, providing for steady personnel reductions primarily through attrition. We used this plan to estimate the poten- tial overstatement of personnel savings in the Service study. Our analysis shows that, after the reported savings are re- duced because of the effects of sampling errors previously discussed, present value savings in fiscal year 1979 would be further reduced by as much as $5.9 million through the application of the attrition plan. RECTL.SIFICATION AND SHIFT OF REDESIGN COSTS After the Engineering Service study, VA reclassified $15,387,000 associated with redesign of the Target Central System from operating costs to one-time development costs. Since these costs are more likely to be incurred over the system's early life than over the entire duration of the system's life, as assumed by the Service, these costs would not be as heavily discounted. We assumed that redesign costr would be incurred in the first 5 years of the Target System': life and estimated that, as a result, present value costs would be increased by about $2,979,000. POTENTIAL INCREASE IN RECURRING COSTS The Engineering Service study acknowledged that annual recurring costs associated with computer center operations could vary by as much as 10 percent. We reduced the recurring costs of $11,487,000 shown in the cost-benefit study by $1,538,000 to reflect the shift in Target Central System re- design costs. We further assumed that if the remaining costs could be understated by 10 percent, the present value costs of Target could be understated by as much as $5.9 million. POTENTIAL INCREASES IN SOFTWARE COSTS In the Service study one-time costs for the Target System were stated as follows: Personal service $15,468,000 Travel 935,000 Contractual services 7,865,000 Communications 6,758,000 Site preparation 4,062,000 Automatic data processing rental 8,895,000 Automatic data processing purchases 37,505,000 All other 453,000 $81,941,000 69 APPENDIX I I APPENDIX II Of particular concern to us were the costs associated with personal services, travel, and contractual services, which totaled about $24.3 million. These are the costs associated with software development. In March 1977 VA revised these costs by shifting $15,387,000 for redesign of the Target Central System from recurring to one-time costs. This adjustment brought the total one-time software develop- ment costs estimate to about $39.7 million. These costs were further assessed by VA in April 1977. In view of the March 1977 VA estimates, the ratio of software costs to equipment costs became about 0.9 to 1. The March 1977 software development costs estimates represent a 103-percent increase over those estimated in the September 1976 cost-benefit study. If these costs should escalate further and VA experiences the same ratio of software development costs experienced by some other Government agencies, the present value net deficit of Target would be even greater than the $700,000 pessimistic estimate we made on page 66. The possible ratios and their impact on the total present value cost of Target are as follows: Ratio of software :ncrease in total to equipment costs (note a) present value costs (millions) 1.0 to 1 $ 2.1 1.1 to 1 5.1 1.25 to 1 9.5 1.7 to 1 22.8 a/Based on estimated automated data processing equipment pur- chase costs of $42.8 million. (See p. 25.) We recognize that the circumstances of Target development may produce a software-equipment ratio different from the ratios of 1.7 to 1 and 3.5 to 1 experienced by other Government agen- cies. (See p. 25.) VA's most recent assessment of software costs in April 1977 resulted in an estimate of $36.8 million, but VA acknowledged that a contractor was still working to estimate more accurately the magnitude of work required to develop, convert, and redesign the Target System. At this stage we are dealing with estimates which could change as Target development progresses, and estimates by nature are not precise. Accordingly, we are presenting the possible im- pact on the benefit-cost ratio should development costs es- calate. 70 APPENDIX III APPENDIX III VETERANS ADMINISTRATION OFFICE Or Tom ADMIIISTRATOR OF VtTE.ANS AFFAIRS WASHINGTON. D.C. 20420 ".e. May 19, 1977 Mr. Gregory J. Ahart Director, Human Resources Division U. S. General Accounting Office 441 G Street, N.W., Room 6864 Washington, D.C. 205S48 Dear Mr. Ahart: Thank you for giving me the opportunity to comment on the GAO draft report to Congressmen Charles Rose and John E. Moss in regard to 'he Veterans Administration's Target System. Since I became Administrator on March 2, 1977, a thorough study and analysis of the VA Target System by myself and the key knowledgeable officials of the Agency has had a top priority. Before commenting in detail on the GAO draft report, let me state my firm conviction that the current implementation plan for the Target System is achievable with very low risk in service to our veteran population. I further believe the system will prove cost effec- tive, and that it will vastly improve VA service to millions of veterans and dependents. As you know, many questions regarding the Target System have been asked in recent years by your office and by committees and individual members of Congress. In a venture of this magnitude and this importance to the veterans of our nation, I consider it not only proper but healthy that every possible question should be raised. Now, however, it is my best judgment that the outstanding questions have been satisfactorily resolved. I believe the VA should proceed immediately with the necessary steps to make the er System a reality at the earliest possible date. On March 2, 1977, I expressed to the HUD-Indepen- dent Agencies Subcommittee on Appropriations, House of Repre- sentatives, my need for an additional period of time to evaluate the status of the project. On April 1 the Target 71 APPENDIX III APPENDIX III System was reorganized to provide a centralized management organization charged with the responsibility of creating a total management plan by examining every aspect of the Target System and developing an appropriately controlled and well segmented project protocol and reporting system. I also formed an Executive Steering Group comprised of top level VA management to provide guidance and participate actively in the management and development of Target and a Target Implementation Planning Team comprised of leading A tech- nicians to assess the status of the effort and to provide me with a comprehensive report with recommendations. These groups were instructed to examine in detail all aspects of the implementation schedule, the projected cost benefits, and all unresolved Target management questions surfaced by the HUD-Independent Agencies Subcommittee. his eval- uation resulted in the preparation of a modified implemen- tation schedule, attainable at low risk, which I consider realistic and achievable. Basically, the plan calls for the installation of the three Regional Data Processing Centers and the interim Target Central System in FY 1978, with the implementation of nationwide inquiry for the Compeisation, Pension and Education (CP&E) master record and the Beneficiary Identification and Records Locator files. Subsequent subsystems of Target will be implemented on a phased basis through FY 1979 to insure full prototype testing at the Philadelphia VA Center prior to implementa- tion at other Regional Offices. This phased implementation allows ample lead time for Regional Office training prior to implementation. On April 25, 1977, members of my staff met with members f the HUD-Independent Agencies Subcommittee Staff and presented to them a revised implementation schedule and supporting documentation for their information and analysis. It is recommended that the GAO report be modi- fied to reflect the contents of this documentation, revised schedules and related cost estimates. (See Attachment A.) During the review of the GAO draft report, we isolated major areas of concern within the report for .iich we feel a statement of the VA's position is appropriate. These items are included in Attachment B. Attachment C presents a list of minor editorial changes which may be of assistance to you in the formulation of the final report. 72 APPENDIX III APPENDIX III The major thrust of the draft report is an examina- tion of the cost benefit and cost effectiveness methodologies used by the VA and the projected cost and benefits of the Target System. Considerable emphasis was placed on early efforts dating back to 1972. Our most recent analysis of the revised implementation schedule demonstrates conclusively that the Target System will be cost beneficial. It must be understood that a cost analysis of this magnitude will have certain assumptions that will be revised as more details are available. For example, once the award for the Target equip- ment is made, the cost will be revised to reflect the actual discount of the equipment rather than an estimated discount. Questions pertaining to the methodology of our cost analysis are addressed in Attachment B. I concur in and accept the recommendation to estab- lish, in measurable terms, goals and standards that the VA seeks to attain to improve benefits claims services to veterans. We will identify goals, objectives and performance standards for the Target System which will be quantified before installation, against which the system benefit improvements will be evaluated. The revised implementation schedule and the related cost analysis which are included should satisfy any uncertainties abcut the progress of development, costs and implementation as cited in the GAO draft rport to Congress. The GAO's draft report acknowledges that its own "pessimistic estimate" of the system's projected cost effectiveness is subjective and could be in error. Since February 1977 the VA has updated and revalidated its cost analysis to reflect modifications resulting from detailed analysis and related planning. Based on an intensive scrutiny of the Target development status, I believe that the current implementation plan is achievable with minimum risk and that the system will e cost effective. The Target System is essential if the VA is to provide improved services to veterans and dependents and is required for the reduction of administrative costs asso- 73 APPENDIX III APPENDIX III ciated with benefits delivery. In my judgment, the out- standin S questions have been satisfactorily answered and. it is my desire to proceed with implementation of the system. Sincerely, MAX CLELAND Administrator Enclosures 74 APPENDIX IV APPENDIX IV PRINCIPAL VA OFFICIALS RESPONSIBLE FOR ADMINISTERING ACTIVITIES DISCUSSED IN THIS REPORT Tenure of office From To ADMINISTRATOR OF VETERANS AFFAIRS: J. M. Cleland Mar. 1977 Present R. L. Roudebush Oct. 1974 Feb. 1977 R. L. Roudebush (acting) Sept. 1974 Oct. 1974 D. E. Johnson June 1969 Sept. 1974 DEPUTY ADMINISTRATOR OF VETERANS AFFAIRS: R. H. Wilson Mar. 1977 Present Vacant Jan. 1977 Mar. 1977 O. W. Vaughn Nov. 1974 Jan. 1977 Vacant Oct. 1974 Nov. 1974 R. L. Roudebush Jan. 1974 Oct. 1974 F. B. ,Rhodes May 1969 Jan. 1974 CHIEF BENEvITS DIRECTOR: D. St buck May 1977 Present A. J. Bochicchio (acting) Mar. 1977 May 1977 R. H. Wilson Jan. 1975 Mar. 1977 J. J. Mulone (acting) Nov. 1974 Jan. 1975 O. W. Vaughn Mar. 1973 Nov. 1974 O. B. Oien Feb. 1970 Mar, 1973 CHIEF DATA MA1NAGEMENT DIRECTOR: W. R. Martin Oct. 1975 Present W. R. Martin (acting) Aug. 1975 Oct. 1975 R. T. Brown Aug. 1974 July 1975 P. J. Budd July 1963 July 1974 75
Veterans' Administration Justification of Costs and Benefits of Proposed Computer System (Target)
Published by the Government Accountability Office on 1977-07-20.
Below is a raw (and likely hideous) rendition of the original report. (PDF)