2 ~JNITEDSTATESGENERAL WASHINGTON, D.C CIVIL DIVISIOM B-133044 Dear Mr. Johnson: The General Accounting Office reviewed the AutqmAted --- Management Information System (AMIS) of the Veterans.Awnis- : tration (VA) to determine if the system was providin.g_VA management with reliable and timely information. Our review wakmade at the Central Office, Washington, EC.; the re- gional offices in Houston and Waco, Texas; the Data Processing Center (DPC) in Austin, Texas; and the VA hospitals in Big Spring, Houston, and Waco, Texas, and in Washington, D.C. The VA established AMIS in 1964 because of VA's inability to manually gather reliable and timely information needed in administering and reporting on its various assistance programs for the Nation's growing veteran population (about 28 million at June 30, 1971). Prior to the establishment of AMIS, conso- lidated reports of comparative data by station, activity, and accounting period were prepared manually at the Central Office from reports submitted by VA hospitals, regional offices, and other reporting stations. VA officials have estimated that about 8,000 different AMIS reports--issued monthly, quarterly, semiannually and annually--will be processed in 1972. On the basis of information available for 1970, we have estimated that the annual operat- ing costs of AMIS are about $7 million. We believe that VA has done a commendable job in develop- ing a system for compiling and reporting information on its diversified programs and operations. The reports generated as a result of AMIS and the opportunity to have access to informa- tion in the AMIS data bank should be of considerable value to Central Office management. For several years VA has been aware that the usefulness of AMIS reports to Central Office managers has been impaired because of inaccurate data in the reports. Because of the continued unreliability of the AMIS reports, some Central Office managers, especially in the Department of Medicine and Surgery, were reviewing, verifying, and correcting AMIS report data before using it, thereby precluding its timely use. 50TH ANNIVERSARY 1921- 1971 I B-133044. , Although VA has initiated various actions to correct this situation, these actions have not been fully effective because: 1. Field station employees either have been care- less or have not been fully aware of what they are required to do with respect to transmitting data to DPC. 2. Field station employees have not been following the procedures established to detect and correct inaccurate data after its transmission to DPC. 3. VA has not been making periodic reviews to verify the implementation of its internal control procedures. 4. VA does not have a system for compiling statistics on the number of errors found in AMIS reports, thus VA management is deprived of perspective on the magnitude of the problem and the identification of the stations consis- tently reporting inaccurate data. INACCURATE DATA--A CONTINUING PROBLEM During the period January through April 1968, VA awarded contracts in amounts totaling $277,000 to a management con- sulting firm to obtain assistance in the study, design, and implementation of improvements in the reporting of AMIS data. VA records showed that a study was needed because AMIS reports for the Department of Medicine and Surgery were untimely, inaccurate, and incomplete. As a result of this study, certain changes in the reports for the Department of Medicine and Surgery were suggested. It appears that the study concentrated on the type of reporting desired by management and did not identify the specific causes for the unreliability of the data. In a report dated May 1969, the Central Office Internal Audit Service commented on the continuing problems concerning accuracy, timeliness, usefulness of data, excessive data, and flexibility of the system and concluded that the 2 ; B-133044' * validity of AMIS, overall, was dependent upon clearing up these problems. The Internal Audit Service recommended that high priority be given to improving and developing AMIS to the satisfaction of the users, In July 1969 the Chief Medical Director, in a memorandum to the Chief Data Management Director, stated that there was a continuing need for accurate and timely management reports and, in pointing out his dissatisfaction with the accuracy of the AMIS reports, commented as follows: "Currently, there are no meaningful *** controls which ensure input completeness or reasonableness. Manual checking and correcting of the reports after they arrive at Central Office is after the fact, is a monumental job, delays the timely use of these re- ports, and is never complete." As a result of several meetings between officials of the Departments of Data Management and Medicine and Surgery, actions were taken in September 1969 to improve the accuracy of the data in certain AMIS reports. These actions involved (1) additional use of the computers to identify the stations that were late in transmitting data for inclusion in the AMIS reports and (2) providing the Central Office with a consolidated error analysis report showing the errors made by each station in transmitting data to DPC. VA procedures provide that data input from field stations be edited (checked by the DPC computer for acceptability for input to the data bank) and that the computer audit the rea- sonableness of data before it is put in the data bank. They provide also that, when erroneous or questionable data is detec- ted, DPC is to send teletype messages (referred to as edit and audit messages) to the field stations requesting correction or clarification of the data. VA maintains statistics on the number of edit and audit messages sent to field stations. Another'procedure used by VA to detect and correct errors in data submitted by field stations, before the data is included in AMIS reports, provides for DPC to send all field stations a monthly printout of the data recorded in the data bank for their stations. Upon receipt of the printout, the station is required to verify the data and report any corrections that are needed. - . B-133044 Problems with the accuracy of the data in AMIS reports have continued, as evidenced by the following excerpts from an April 1970 VA teletype message from the office of the Controller to all Department of Medicine and Surgery field stations: "IN ADDITION TO THE RECEIPT OF THE MACHINE-GENERATED EDIT AND AUDIT MESSAGES, FOUR CENTRAL OFFICE EMPLOYEES SPENT THE MAJOR PART OF THE FIRST SIX WORKDAYS OF APRIL, 1970 MAKING TELEPHONE CALLS TO DM&S [DEPART- MENT OF MEDICINE AND SURGERY] STATIONS TO ADVISE OF IMBALANCES, MISSING SEGMENTS OR OUT-OF-LINE SITUATIONS: *** * * * * * "THE MISSING SEGMENTS AND OUT-OF-BALANCE ITEMS WILL AFFECT OVER 100 AMIS PRINTOUTS WHICH WILL BE RECEIVED IN CENTRAL OFFICE DURING APRIL, 1970; THE AFFECTED INDIVIDUAL STATION PRINTOUTS, REGION TOTALS AND NATION WIDE TOTALS WILL BE USELESS AS MANAGEMENT INFORMATION." (Underscoring supplied.) IMPROVEMENTS STILL NEEDED We discussed the problem of inaccurate data with Central Office officials in 16 services in the Departments of Veterans Benefits and Medicine and Surgery. About half of the officials, mostly in the Department of Medicine and Surgery, told us that inaccurate data was still a problem and that they found it necessary to review and often to verify information in AMIS re- ports before they could use the information to manage their operations. The extent of such review and verification varied among the services. The officials commented, however, that their reviews of the AMIS reports generally would disclose only gross inaccuracies and would not disclose all the inaccurate data. They stated that their verification frequently involved contacting the appropriate field station and asking it to resubmit the data to DPC. One Central Office official informed us that, after receipt of AMIS reports for the quarter ended March 31, 1970, letters were sent to $80 stations advising each of the errors in the data that it had reported and of the need for accuracy in the data to be submitted for the June 30, 1970, year-end reports. This official told us that the letters had accomplished little 4 B-133044 or nothing and that the year-end reports contained numerous errors. He said that correcting the inaccuracies required considerable time and contributed to delays in making manage- ment decisions and in formulating his service's part of the 1972 budget. The official pointed out several significant errors which, if undetected, could have distorted procurement requirements and appropriation requests. The continuing incidences of inaccurate data included in the AMIS reports indicate that field station employees respon- sible for preparing the basic data documents for submission to DPC either do not fully understand what they are to do or are careless in preparing the documents. A teletype message dated March 1, 1971, from the Central Office to Department of Medicine and Surgery field stations, illustrates the continuing problems with inaccurate data and identifies these two causes for the problem. Following are excerpts from that message. "WE ARE EXPERIENCING CONTINUING PROBLEMS WITH THE ACCURANCY OF REPORTING *** THESE PROBLEMS ARE DUE TO TWO FACTORS, I.E., THE EMPLOYEES PREPARING THE AMIS CODE SHEETS *** ARE MISINTERPRETING HEADING INSTRUCTIONS PRINTED ABOVE THE ENTRY BLOCKS ON THE CODE SHEETS AND ARE ALSO FAILING TO CHECK THE ENTRIES TO ASSURE THAT REPORTED FIGURES ARE MATH- EMATICALLY CORRECT. ****** "STATION MANAGEMENT WILL ASSURE THAT EMPLOYEES RESPONSIBLE FOR PREPARING THIS CODE SHEET FULLY UNDERSTAND THESE INSTRUCTIONS AND ALSO ASSURE THAT THESE REPORTS ARE MATHEMATICALLY ACCURATE PRIOR TO SUBMISSION." We found that in some cases employees responsible for implementing VA's procedures for ensuring the accuracy of data reported to DPC were not carrying out their prescribed functions. Each Department of Medicine and Surgery station has an AMIS reports coordinator who is responsible for en- suring that the monthly printouts of data, furnished by DPC, are reviewed for completeness and accuracy and that additional or corrected data is sent to DPC when omissions or errors are noted. The reports coordinator is also responsible for ensuring that appropriate action is t 39 p$lp#j \*E DPC edit and audit messages. 5 I ’ B-133044 - We found a number of errors in AMIS reports that should have been corrected by the reports coordinators at two hospitals. At one hospital, where we found 59 errors, the re- ports coordinator had not distributed the month-end printouts for review, because, in his opinion, such a review would have required too much time. The reports coordinator stated also that, in his opinion, some of the errors were not important. At the other hospital, where we found 48 errors, the reports coordinator had not taken action to ensure that appropriate responses were being made to all edit and audit messages sent from DPC. CONCLUSIONS We believe that the incidence of inaccurate data in re- ports furnished for management use should be infrequent enough to permit management to have reasonable confidence in the reliability and accuracy of the data and to find it not necessary to verify the accuracy of the data prior to using it. In our opinion, the basic problem facing VA is to eliminate, to the maximum extent possible, inaccurate data before it is reported by the field stations to DPC. We believe that all VA employees involved in accumulating, reporting, and verifying AMIS data should be made aware of the importance of accuracy in their work and the need for them to follow the prescribed procedures. Also we believe that VA should review periodically the implementation of the procedures for detecting and correcting inaccurate data to provide assurance that this aspect of internal control is functioning as intended. VA was compiling statistics on the number of errors detected in the data submitted by field stations to the data bank at DPC; however, these statistics do not show the full extent of the inaccurate data reported by stations to DPC and give no indication as to the extent to which the internal control procedures for identifying and correcting inaccurate data are ineffective. This fact is borne out by the number of errors found in the AMIS reports. 6 . " B-,133044 We believe that VA should establish a system for com- piling statistics on the inaccurate data found in AMIS re- ports, because it would give Central Office management a more complete picture of the reporting of inaccurate data by stations and a means of identifying weaknesses in the pro- cedures for detecting and correcting inaccurate data. RECOMMENDATIONS We recommend that,to improve the reliability of AMIS reports, you take action to: --Establish a program to educate and train VA employees involved in AMIS, so that each employee is aware of the importance of care and accuracy in following the prescribed procedures for accumulating, reporting, and verifying AMIS data and is sufficiently trained to perform his function. --Establish a system for compiling and report- ing statistics, by source, on the number of errors in AMIS reports detected after issuance, so that Central Office management will be in a better position to (1) identify those stations that consistently report inaccurate data and (2) evaluate the effective- ness of internal controls for identifying and correcting inaccurate data sent by stations to the data bank. --Periodically review the implementation of the procedures for detecting and correcting inaccurate data. Your attention is invited to section 236 of the Legisla- tive Reorganization Act of 1970 which requires that you sub- mit written statements of the action taken with respect to the above recommendations,, The statements are to be sent to the House and Senate Committees on Government Operations not' 1. > later than 60 days after the date of this report and to the i 0. .. House and Senate Committees on Appropriations in connection " -r with the first request for appropriations submitted by your agency more than 60 days after the date of this report. We shall appreciate being furnished with copies of your state- ments to the Committees. 7 A -( c " B-133044 Copies of this report are being sent to the Committees “> < -d<I _ mentioned above, the Senate and House Committees on Veterans' --". Affairs, and to the Director, Office of Management and Budget. We appreciate the cooperation and courtesy extended to us by VA employees during our review. Sincerely yours, Director, Civil Division The Honorable Donald E. Johnson Administrator of Veterans Affairs c 8
Review of the Automated Management Information System of the Veterans Administration
Published by the Government Accountability Office on 1971-09-14.
Below is a raw (and likely hideous) rendition of the original report. (PDF)