United States General Accounting Office GAO Report to the Chairman, Subcommittee on Oversight, Committee on Ways and Means, House of Representatives January 1997 TAX SYSTEMS MODERNIZATION Imaging System’s Performance Improving but Still Falls Short of Expectations GAO/GGD-97-29 United States GAO General Accounting Office Washington, D.C. 20548 General Government Division B-270263 January 16, 1997 The Honorable Nancy L. Johnson Chairman, Subcommittee on Oversight Committee on Ways and Means House of Representatives Dear Chairman Johnson: One of the major objectives of the Internal Revenue Service’s (IRS) modernization effort is to move away from a labor-intensive tax return processing system that relies on thousands of employees transcribing data from paper tax returns and move to an electronic system that reduces processing costs and eliminates transcription errors. One strategy for achieving that objective is to reduce the number of paper returns by increasing the number of returns filed electronically. For returns that will continue to be filed on paper, IRS plans to achieve its objective through document imaging and optical character recognition (OCR). The Service Center Recognition/Image Processing System (SCRIPS) is the first of what was to have been two document imaging and OCR systems.1 SCRIPS became operational in the latter part of 1994, and we reported that it experienced numerous performance problems during the 1995 filing season.2 This report responds to your request that we follow up on those problems and assess the performance of SCRIPS in 1996. Specifically, this report (1) identifies the primary causes for performance problems that occurred in 1995; (2) assesses whether SCRIPS performance improved in 1996 as of September 30, 1996; and (3) provides a status report on IRS’ future plans for SCRIPS. In July 1995, we reported on IRS’ progress in implementing some of the Background business and technological components of its modernization effort, known as Tax Systems Modernization (TSM).3 Although we said that IRS had made some progress, we also said that pervasive management and technical weaknesses existed that placed the modernization effort at risk. Among 1 The second system, the Document Processing System (DPS), was intended to replace SCRIPS and expand the imaging capability to more complex tax forms. However, on October 8, 1996, IRS announced that it was terminating that project. 2 The 1995 Tax Filing Season: IRS Performance Indicators Provide Incomplete Information About Some Problems (GAO/GGD-96-48, Dec. 29, 1995). 3 Tax Systems Modernization: Management and Technical Weaknesses Must Be Corrected If Modernization Is To Succeed (GAO/AIMD-95-156, July 26, 1995). Page 1 GAO/GGD-97-29 Performance of SCRIPS B-270263 other things, (1) IRS did not have a business strategy to maximize electronic filing, the result of which could slow the planned decrease in the workload of paper processing systems; and (2) IRS lacked the full range of managerial and technical foundations to realize its modernization objectives. Some of these key foundation components were a complete cost/benefit analysis of the overall modernization effort and thorough testing of individual systems before they were implemented. In September 1996, we reported on IRS’ progress in addressing the managerial and technical weaknesses we identified in July 1995.4 We concluded that although IRS is working to resolve these weaknesses, it had not fully satisfied any of our recommendations. SCRIPS was one of the systems that was designed under the conditions cited in our July 1995 report. It was intended to replace the aging OCR equipment that IRS had been using to process all of the paper Federal Tax Deposit (FTD) coupons; almost all of the paper information returns (e.g, Forms 1099); some of the individual income tax returns filed on Form 1040EZ; and some employment tax returns (Form 941). In addition, SCRIPS was expected to process Form 1040PC, a paper form that taxpayers can generate when they use computer software to prepare a tax return. Figure 1 shows the various SCRIPS components. Under the character recognition and image capture component of SCRIPS, scanners (1) read information from the document and convert the information to machine-readable format for later computer processing and (2) create an image of the document. In the event of recognition errors during document scanning, IRS staff can access an image of the tax return in lieu of having to locate the original paper tax return to make corrections. IRS expected that having an image of the tax return would improve the productivity of staff doing data validation. In addition, the images of FTD coupons and certain information return documents are stored on optical disk for later use. The older OCR systems used microfilming as the storage medium—essentially a manual process that required more physical storage space than optical disks. IRS continues to retain a paper copy of the Form 1040EZ rather than store the image because IRS has certain legal concerns.5 4 Internal Revenue Service: Business Operations Need Continued Improvement (GAO/AIMD/GGD-96-152, Sept. 9, 1996). 5 One concern is the validity of an imaged signature. Another concern is that SCRIPS images only one side of the two-sided Form 1040EZ. The second page includes a worksheet that taxpayers use for tax calculations, and IRS considers it part of the tax return. Page 2 GAO/GGD-97-29 Performance of SCRIPS B-270263 Page 3 GAO/GGD-97-29 Performance of SCRIPS B-270263 Figure 1: SCRIPS Components Character Recognition Image Processing Data Management & Image Capture The image controller helps The multiprocess server A scanner operator feeds a block ensure that each scanned maintains control over the tax of documents into the scanner. document image moves from data. All changes and updates The scanner numbers each the scanner to the server. It are processed through the document, takes an image, and also provides data backup server. makes the first OCR attempt. until images reach the server. Updates Final Output 3 Scanners with Monitor, Keyboards and Printer a Updates 3 Image Controllers 1 Multiprocessor Server Updates Page 4 GAO/GGD-97-29 Performance of SCRIPS B-270263 Cont. Secondary Character Outputs Recognition and Identification Secondary character recognition is used for those fields (1) where characters are not completely read by the scanner or (2) IRS requires secondary recognition. Tax data is read by the recognition engine, and the block reader reads name and address information. Updates Optical Disks Store FTDs and information return images. 5 Recognition Engines for Tax Data Final Output Updates 8 mm Tape Drive Stores tax data. 2 Name and Address Block Readers In those instances where the secondary recognition and name and address block reader cannot identify the correct characters, the file is sent to a workstation operator. That operator reviews the image (with the capability to magnify it) to determine what corrections are needed. Updated files are sent back to the file server. 9-Track Tape Drive Forwarded to the next stage with tax data for further processing. Updates 66 Workstations Source: GAO graphic based on IRS data. Page 5 GAO/GGD-97-29 Performance of SCRIPS B-270263 IRS expected that SCRIPS would provide faster, more accurate document processing. Specifically, IRS expected that SCRIPS would result in a 20 percent productivity increase over manual data entry and a 10 percent productivity increase over older OCR equipment. Other expected benefits included lower costs for system maintenance and storage of tax return data. Originally, IRS planned to implement SCRIPS in all 10 service centers where it currently processes paper returns. However, after the contract was awarded in February 1993, IRS decided to consolidate paper tax return processing in five centers. Accordingly, SCRIPS was tested in Cincinnati in the summer of 1994, and the other four SCRIPS centers began using SCRIPS between September and November 1994. SCRIPS experienced significant performance problems in 1995. Two Results in Brief problems were system downtime and slow processing rates. For example, although SCRIPS was expected to increase productivity by 20 percent over the manual data entry system, SCRIPS actually processed Forms 1040EZ in 1995 at a rate that was about 7 percent slower than manual data entry. Because of performance problems, two of the five centers stopped processing Forms 1040EZ on SCRIPS and reverted to using manual data entry. SCRIPS performance deficiencies in 1995 stemmed primarily from both hardware and software problems. Hardware problems occurred with the scanner that captures the image of the document. Software problems occurred with the operating system for the image controller that reconciles the number of documents scanned to the number of documents in the SCRIPS database. The hardware and software problems were not detected before the rollout of SCRIPS because testing in Cincinnati was incomplete. For example, SCRIPS was never tested at the peak volumes that IRS experiences during a filing season. Although IRS had expected that SCRIPS would be processing five document types—forms 1040EZ, 941, and 1040PC; FTD coupons; and information returns—IRS postponed plans to process forms 941 and 1040PC on SCRIPS. Of the three remaining document types, the Cincinnati test certified the software application only for FTD coupons. Therefore, the software applications for information returns and Form 1040EZ were not thoroughly tested before they were put into production. Page 6 GAO/GGD-97-29 Performance of SCRIPS B-270263 To improve the performance of SCRIPS for 1996, IRS made hardware and software modifications, some of which were made before the start of the 1996 filing season. As a result of some of these enhancements and more staff familiarity with SCRIPS, according to IRS officials in all five SCRIPS service centers, SCRIPS performed significantly better during the 1996 filing season than it did in 1995. For example, between April and June, unscheduled system downtime decreased from about 791 hours in 1995 to about 43 hours in 1996. Also, performance on another indicator—the number of documents processed per hour—improved for Forms 1040EZ and FTD coupons.6 Specifically, the number of Forms 1040EZ processed per hour in 1996 increased by four documents per hour (about 6 percent) and the number of FTD coupons increased by 3 documents per hour (less than 1 percent). However, the number of information returns processed per hour decreased by five documents per hour (about 3 percent). In addition to a slower processing rate for information returns, SCRIPS (1) is not processing all the forms that it was expected to process in 1996, (2) is expected to cost more than originally estimated, and (3) is expected to provide lower labor cost savings than IRS originally anticipated. IRS’ October 1994 business case for SCRIPS stated that in 1996 SCRIPS would be processing all FTD coupons and information returns, all Forms 1040EZ, 50 percent of the Forms 1040PC, and 93 percent of the Forms 941. However, during the first 9 months of 1996, SCRIPS processed only FTD coupons, 60 percent of the information returns, and about 50 percent of the Forms 1040EZ. This reduced level of performance stems in large part from (1) IRS’ decision, after the contract was awarded, to consolidate all paper processing in 5 service centers instead of 10; and (2) IRS’ decision to purchase only 5 systems when the contractor was planning to provide 10 systems—1 in each of the 10 service centers. Furthermore, IRS officials did not know the extent to which the hardware and software modifications that had already been made for 1996 and those that were planned for later in the year would increase SCRIPS’ capability for processing more 1040EZs and information returns. Therefore, IRS officials did not significantly increase the expectations for SCRIPS in 1996 over those that they had in 1995. The latest cost estimate for SCRIPS is $288 million—considerably more than previous cost estimates, which, according to IRS’ post-implementation 6 The number of documents processed per hour, commonly referred to as the composite rate, includes not only the time to scan the documents but also the time spent by work station operators validating and correcting tax return data. Validation and correction are necessary in those instances when the scanner and secondary recognition systems cannot read certain tax return data or the system detects an error made by a taxpayer. Page 7 GAO/GGD-97-29 Performance of SCRIPS B-270263 review report on SCRIPS, ranged from $133 million to $209 million. At least $20 million of that increase is attributed to maintenance costs that were not included in either the $133 million or $209 million estimates.7 Also, IRS estimated in October 1994 that SCRIPS would provide about $17 million in labor savings from fiscal years 1994 through 2000. In September 1995 IRS lowered that estimate to about $5 million. Furthermore, a September 1996 IRS investment evaluation of SCRIPS estimated that SCRIPS will yield a negative return on investment (i.e., SCRIPS’ costs will exceed its benefits) from 1991 to 2001. However, the evaluation noted that the estimate is based on performance in 1995 and does not reflect the software and hardware modifications that were made in 1996. In July 1995, IRS decided to terminate all software programming for the 1040PC because of SCRIPS’ instability, a lack of system capacity, and a number of software application problems. Also, IRS has decided not to use SCRIPS to process Forms 941 in fiscal year 1997. Decisions on using SCRIPS to process Forms 941 beyond fiscal year 1997 have not been finalized. In addition to funding constraints, according to IRS officials, a decision on using SCRIPS for Forms 941 will depend on (1) the results of a September 1996 capacity test of SCRIPS, which were not available when we completed our audit work; and (2) decisions by IRS’ Investment Review Board8 on recommendations made by a paper processing task team. Our objectives were to (1) determine the primary causes for SCRIPS Objectives, Scope, performance problems in 1995; (2) assess whether those problems were and Methodology corrected as of September 30, 1996; and (3) provide a status report on IRS’ future plans for SCRIPS. To accomplish these objectives, we did the following: • We interviewed National Office officials in the SCRIPS project office and the taxpayer service function, which has responsibility for those tax forms processed on SCRIPS, to obtain their views on the extent to which SCRIPS’ performance improved during fiscal year 1996. 7 The supporting documentation for IRS’ $288 million cost estimate showed a separate breakout for maintenance costs for only fiscal years 1995 and 1996. We were unable to determine what, if any, portion of the costs for fiscal years 1997 to 2000 was attributed to maintenance. According to IRS officials, maintenance costs may have been combined with other costs for those fiscal years. 8 In October 1995, IRS created an executive-level Investment Review Board, currently co-chaired by the Deputy Commissioner and the Chief Financial Officer, for selecting, controlling, and evaluating all of IRS’ information technology investments. Page 8 GAO/GGD-97-29 Performance of SCRIPS B-270263 • We interviewed IRS contracting office officials to determine what, if any, performance requirements the contractor was being held to in 1995 and 1996. • We interviewed officials at all five SCRIPS service centers to determine whether performance had improved in 1996 and to identify the performance indicators officials were using to evaluate SCRIPS performance. • We observed SCRIPS in operation at the service centers in Memphis and Cincinnati and at the program development site in Washington, D.C. • We interviewed contractor officials about the workload requirements that IRS had specified for SCRIPS. • We reviewed IRS business cases9 for SCRIPS that documented the objectives of the system, its expected benefits, and its estimated cost. IRS prepared business cases for SCRIPS in February 1992, April 1992, December 1993, October 1994, and December 1995. To assess SCRIPS, we used the performance expectations that were included in the October 1994 business case because it was the one that was in effect at the beginning of the 1995 filing season. • We reviewed IRS evaluations and reports on the implementation of SCRIPS, including an August 1995 performance evaluation report; a December 1995 post-implementation review report that assessed whether business goals were met; a May 1995 Internal Audit report on SCRIPS testing in Cincinnati; a February 1996 Internal Audit report on the rollout of SCRIPS; and a September 1996 report on the results of an investment evaluation of SCRIPS. • We compared available SCRIPS performance data for January through September 1995 and 1996. • We computed composite rates—the number of documents processed per hour—for Form 1040EZ, information returns, and FTD coupons. To compute those rates, we used IRS data on the number of hours spent for various aspects of tax return processing, including scanning, data correction, and data validation for January through September 1995 and 1996. Our composite rate calculations, similar to those done by IRS shortly after SCRIPS was implemented, did not include time for document preparation or another function that is referred to as code and edit.10 9 IRS prepares business cases to justify an information technology project and to demonstrate that it is cost beneficial. 10 For SCRIPS, the IRS data we received for computing composite rates showed the combined hours for both the code and edit function and the document preparation function. During code and edit, staff prepare returns for computer entry by, among other things, ensuring that all data are present and legible. For the manual entry system, the code and edit function was not combined with the document preparation function, and the hours associated with document preparation were not shown. Therefore, we did not include these aspects of processing in our composite rate comparison. Page 9 GAO/GGD-97-29 Performance of SCRIPS B-270263 We did our audit work between October 1995 and September 1996 and in accordance with generally accepted government auditing standards. Other than identifying obvious reporting errors, we did not verify the accuracy of IRS’ data on the number of hours spent for various aspects of tax return processing. We did attempt to detemine what elements were included in IRS’ cost estimates for SCRIPS, but we did not attempt to verify the accuracy of the costs for those elements. We requested comments on a draft of this report from the Commissioner of Internal Revenue or her designated representative. Responsible IRS officials, including the National Director for Submission Processing, the Assistant Commissioner for Forms and Submission Processing, and the SCRIPS project manager, provided IRS’ comments in a November 13, 1996, meeting. Their comments on our recommendation were reiterated in a November 19, 1996, memorandum from the Acting Chief of Taxpayer Service. IRS’ comments are summarized and evaluated on pages 22 and 23. SCRIPS performed well below expectations in 1995. Extensive, unscheduled SCRIPS Performed system downtime and slower than expected processing rates affected Substantially Below SCRIPS’ ability to meet the expectations that IRS had established before the Expectations in 1995 start of the 1995 filing season. Hardware problems with the scanner contributed to significant amounts of system downtime; various software problems contributed to slow processing rates. SCRIPS Processed Less In 1995, SCRIPS processed 19 percent more FTD coupons than IRS expected. Volume and at Slower However, as shown in table 1, SCRIPS did not meet IRS’ volume expectations Rates Than Expected for the three other document types scheduled for processing in 1995—information returns, Form 1040EZ, and Form 941 (IRS did not expect to start processing Form 1040PC on SCRIPS until 1996). Page 10 GAO/GGD-97-29 Performance of SCRIPS B-270263 Table 1: Number of Documents SCRIPS Expected to Process and Number IRS expected to Number actually Number Actually Processed in 1995 Document type process in 1995a processed in 1995 FTD coupons 76,422,540 90,630,944 Information returns 57,369,000 24,897,860 Form 1040EZ 8,594,500 4,873,419 Form 941 4,826,140 0b a IRS expectations as of October 1994. b Sometime between October and December 1994 IRS decided not to use SCRIPS to process Forms 941. Source: IRS data. SCRIPS processed 19 percent more FTD coupons than IRS had expected because more FTD coupons were filed than expected, and service center officials placed the highest priority on processing those forms. That priority stemmed from (1) IRS procedures that require that 90 percent of all FTD coupons be processed within 24 hours of receipt; and (2) the absence of a backup processing system, because IRS had cancelled its maintenance contract for the older OCR equipment that had been processing FTD coupons. IRS had backup systems for the other documents that SCRIPS was expected to process in 1995. The manual data entry system could be used for forms 1040EZ and 941, and IRS extended the maintenance contract for the older OCR equipment that was processing information returns. Not only did SCRIPS not process the number of forms expected, but the speed with which it did its processing was slower than expected. As shown in table 2, SCRIPS did not meet the processing rate expectations for any of the three document types it processed in 1995. Moreover, the actual processing rate for Forms 1040EZ in 1995 was about 7 percent less than the rate achieved in 1995 by manual data entry. Table 2: Expected and Actual Processing Rates for SCRIPS in 1995 Processing rate (documents per hour) in 1995 Compared to Rates for Systems Expected for Actual for Actual for older SCRIPS Was to Replace Document type SCRIPS SCRIPS systems 1040EZ 186 63 68 Information returns 178 147 131 FTD coupons 1,170 758 Not applicablea a IRS cancelled the maintenance contract for the OCR system that had been processing FTD coupons. SCRIPS was the only system used to process FTD coupons in 1995. Source: GAO computations using IRS data for January through December 1995. Page 11 GAO/GGD-97-29 Performance of SCRIPS B-270263 Because of performance problems, two of the five service centers stopped using SCRIPS to process Forms 1040EZ in 1995.11 Instead, they reverted to manual data entry, which required using more staff resources than planned, thus increasing processing costs. IRS had planned to use 25.6 staff years to process other-than-full-paid12 Forms 1040EZ during the 1995 filing season in the five SCRIPS centers but used 66.5 staff years instead.13 Hardware problems contributed to SCRIPS’ performance deficiencies in 1995. According to IRS’ August 1995 performance evaluation report, the SCRIPS scanner experienced the most hardware failures, which contributed to a “substantial amount of downtime” at the centers.14 According to IRS data, between April and June 1995, the five service centers experienced about 791 hours of unscheduled downtime. In addition, the scanner jammed when paper was extremely thin, which was sometimes the case with information returns. Software problems also occurred. According to IRS officials and evaluations of SCRIPS, the image controller did not operate fast enough to keep up with the scanner because the operating system software was inefficient. The image controller is to track each scanned document image file to ensure that it moves from the scanner and ultimately to the server where it is stored. This component helps reconcile the number of documents scanned to the number of documents in the database. IRS officials and evaluations of SCRIPS also indicated that SCRIPS did not provide accurate reports on this reconciliation—referred to as “run-to-run” balancing. The FTD coupon run-to-run balancing report, for example, did not provide total counts to help IRS staff confirm that all of the FTD coupons in a block that was scanned had in fact been processed. This balancing is particularly important for FTD coupons because IRS needs to accurately classify tax deposits to the appropriate Treasury account. At one center we visited, officials told us that they had to revert to manual counts for doing run-to-run balancing. 11 According to IRS officials, service center directors had discretion in managing the processing workload. Some chose to revert to the manual system; others did not. 12 Other-than-full-paid tax returns are those returns filed by taxpayers who either were due a refund or did not pay the full amount of tax owed at the time of filing. These returns were about 92 percent of the Forms 1040EZ filed at the five SCRIPS service centers during the 1995 filing season. 13 The number of staff years for Form 1040EZ includes those used on SCRIPS and manual data entry. 14 IRS’ evaluation of SCRIPS performance during the 1995 filing season noted that wide variations in scanner operator skills also affected the scanner’s performance. IRS officials also noted that even though the scanner may have been “down,” other SCRIPS components could have been operating. Page 12 GAO/GGD-97-29 Performance of SCRIPS B-270263 Performance Problems Not Many of the problems experienced with SCRIPS in 1995 might have been Thoroughly Identified anticipated if IRS had thoroughly tested SCRIPS before installing the system Before Rollout Due to in the other four service centers. The pilot test of SCRIPS was incomplete because it (1) did not certify all software applications that were to be used Incomplete Testing during 1995; and (2) did not test SCRIPS’ ability to handle peak processing volumes, such as those experienced in the tax return filing season. An organization tests a new or modified system to detect system design and development errors and to correct them before putting a system into operation. In our July 1995 report on TSM, we said that although IRS recognized the importance of testing, it had not yet developed a complete and comprehensive testing plan for TSM.15 We said that individual TSM systems were developing their own test plans, which IRS described as rudimentary and inadequate. This testing environment was in effect when IRS did a pilot test of SCRIPS at the Cincinnati Service Center in the summer of 1994. The purpose of a pilot test is to evaluate the performance of a system in one location before deciding whether to implement the system at other locations. IRS uses the pilot test to certify that the system is meeting its program or business objectives. During the pilot test, IRS is to collect data on the performance of the system and compare the data against established performance goals to certify that the system is performing as expected. Due to delays in receiving and testing software, only the FTD coupon application was certified as a result of the pilot test. The certifications for Forms 1040EZ and information returns were not done due to management concerns that conducting these certifications would delay the rollout of SCRIPS to five centers, which was scheduled for January 1995. As a result of incomplete testing, for example, IRS did not identify problems with the information returns software until SCRIPS was fully operational. Due to these software problems, a higher percentage of returns were sent to data validation than originally anticipated, thus decreasing system productivity. According to IRS officials, those problems were corrected before the 1996 filing season. The pilot test was also incomplete because it did not (1) provide IRS a clear indication of SCRIPS’ ability to perform under peak workload conditions and (2) show the impact of running multiple software applications. If the system had been tested at larger volumes, running multiple software 15 GAO/AIMD-95-156. Page 13 GAO/GGD-97-29 Performance of SCRIPS B-270263 applications, problems may have been identified earlier and resources could have been diverted to address performance problems before the system’s rollout. According to an Internal Audit report on the pilot test, the contract required that SCRIPS process 200,000 documents a day during peak periods. During the 1-day volume test in the pilot, only 45,000 documents were scanned into the database. The remaining components of the system processed an additional 142,000 documents that had been scanned before the test. Thus, the scanner, one of the primary sources of downtime during 1995, was not tested under levels IRS experiences in a production environment. Also, only 10,000 documents were processed completely—only 5 percent of the daily production workload specified in the contract. According to IRS officials, a full production test plan had been developed in December 1993 that would simulate peak filing season volumes. However, the plan was never implemented. One reason IRS officials provided for not implementing the test plan was that IRS never had full access to a SCRIPS system to do the test. The system that IRS was to use for the test was the same system that the contractor was using for systems development work. Another factor that IRS officials cited was a sense of urgency to roll out SCRIPS for FTD coupons because IRS had no backup processing system.16 IRS’December 1995 report on the post-implementation review of SCRIPS cited several problems with the testing of SCRIPS in addition to those that occurred in the pilot test. For example, although an equipment acceptance test was done, the test was not adequate to validate the system’s readiness for production. The report also stated that IRS waived an “operational capabilities demonstration” that would have included (1) tests for the readability of forms and (2) measures of the number of documents scanned per hour. The report said that such a test, done as part of the contract award process, may have shown that the system could not meet the minimum performance requirements. The post-implementation review team proposed that tests be done to verify that the system could handle the production workload. It also stated that if such a test could not be done, a simulated test should be done focusing on the performance of key areas of the system, particularly those that could lead to processing bottlenecks. It also proposed a 1-year pilot test for forms that may be added to SCRIPS in the future and proposed that the pilot test include the peak volumes of a filing season. 16 As previously stated, IRS had cancelled the maintenance contract for the OCR system that IRS was using to process FTD coupons. In addition, IRS had redistributed the workload of paper FTD coupons in 1995—consolidating the work of 10 service centers into 4 service centers. Page 14 GAO/GGD-97-29 Performance of SCRIPS B-270263 Officials in the five service centers that used SCRIPS in 1996 said that it is Performance performing significantly better than it did in 1995. IRS officials told us that Improved for 1996, the primary performance expectation for SCRIPS in 1996 was system but SCRIPS stabilization. One of the primary indicators these officials used in evaluating stabilization was the amount of downtime. According to IRS Performed Below service center officials and available IRS data, downtime decreased Original Expectations substantially in 1996. This reduction enabled SCRIPS to process more Forms 1040EZ and information returns in 1996 while continuing to process all FTD and Costs May Be coupons. In addition, the processing rates for SCRIPS (i.e., the number of Higher Than Expected documents processed per hour) improved for two document types—Form 1040EZ and FTD coupons. Despite these improvements, the system, as of September 30, 1996, (1) was not processing all the forms that the October 1994 business case said it would process, (2) was expected to cost more than original estimates, and (3) was not expected to provide the estimated labor savings that were cited in the October 1994 business case. SCRIPS Performance According to service center officials, SCRIPS performed “significantly Improved in 1996 better” in 1996 than it did in 1995 because the system experienced significantly less downtime than in 1995. IRS did not begin tracking downtime in 1995 until April. Comparable data for April through June 1995 and 1996 show that unscheduled downtime did decrease significantly—from about 791 hours to 43 hours. As shown in table 3, mostly because of less unscheduled system downtime, SCRIPS processed many more documents during the first 9 months of 1996 than it did during the first 9 months of 1995. Table 3: Number of Documents SCRIPS Processed in 1995 and 1996 Number processed as of Number processed as of as of September 30 Document type September 30, 1995 September 30, 1996 Form 1040EZ 4,870,493 8,309,384 Information returns 24,631,809 22,618,452 FTD coupons 67,247,198 70,857,389 Total 96,749,500 101,785,225 Source: IRS data. Also, according to IRS officials, new software was installed in November 1995 to correct the run-to-run balancing problems that we Page 15 GAO/GGD-97-29 Performance of SCRIPS B-270263 discussed earlier. According to these officials, since that software was installed, SCRIPS centers have not reported problems with tracking the number of documents scanned and processed. Also in 1996, as shown in table 4, processing rates increased for Forms 1040EZ and FTD coupons. Table 4: Actual Processing Rates by Document Type as of September 30, Processing rate (documents per hour) 1995, and 1996 Actual for older SCRIPS actual rate SCRIPS actual rate systems as of Form type as of Sept. 30, 1995 as of Sept. 30, 1996 Sept. 30, 1996 Form 1040EZ 63 67 60 Information returns 150 145 133 FTD coupons 719 722 Not applicablea a IRS cancelled the maintenance contract for the OCR system that had been processing FTD coupons. SCRIPS was the only system used to process paper FTD coupons in 1996. Source: GAO computations based on IRS data. SCRIPS Processed Many Despite the improved performance in 1996, SCRIPS is still doing much less Fewer Documents Than than expected. In addition, estimated costs have increased and estimated Originally Expected in labor cost savings have decreased. 1996 While Its Estimated Costs Have Increased SCRIPS Processed Fewer The October 1994 business case stated that in 1996 SCRIPS would be Documents Than Expected in processing (1) all Forms 1040EZ, (2) all FTD coupons, (3) all information 1996 returns, (4) 93 percent of the Forms 941, and (5) 50 percent of the Forms 1040PC. During the first 9 months of 1996, as in 1995, SCRIPS processed all the FTD coupons IRS had received but no forms 1040PC or 941. Also, although SCRIPS processed more Forms 1040EZ and information returns during the first 9 months of 1996 compared with the first 9 months of 1995, it still processed only about 50 percent of the Forms 1040EZ and about 60 percent of paper information returns. As discussed in the next few paragraphs, this reduced level of performance, compared to the October 1994 business case, stems in large part from a major change in IRS’ plans after the SCRIPS contract was awarded. Furthermore, IRS officials did not know the extent to which hardware and software modifications that had been made for 1996 and those that were planned for later in the year Page 16 GAO/GGD-97-29 Performance of SCRIPS B-270263 would affect SCRIPS’ ability to process more Forms 1040EZ and information returns. Therefore, IRS officials did not significantly increase the expectations for SCRIPS in 1996 over those that they had in 1995. As specified in the contract, SCRIPS was designed on the assumption that it would be installed in each of the 10 service centers that were then processing paper returns. However, in December 1993—10 months after the contract was awarded—IRS announced plans to consolidate the processing of paper tax returns in five centers. As a result, although the total workload for SCRIPS remained the same, the volume to be processed by any one of the five SCRIPS service centers on the average doubled. Thus, the system that the contractor had designed would not meet the workload requirements without further systems development work. In May 1994, IRS attempted to revise the original contract to meet the volume requirements at five service centers. According to IRS’ post-implementation review report, a statement of work was written to revise the contract requirements to accommodate the five service center scenario. In December 1994, at IRS’ request, the contractor proposed changes totaling about $21 million. According to contractor officials, that proposal represented an interim attempt to meet the new workload requirements under a five service center scenario. According to IRS officials, IRS could not afford all of the proposed modifications. Also, IRS officials did not believe that all the proposed changes were needed. As a result, negotiations on these proposals were never completed. Thus, IRS decided to purchase five systems, one for each of the five SCRIPS service centers. To compensate for having fewer systems than intended, IRS decided to (1) continue paper processing of Forms 1040EZ in the other five service centers and (2) extend the maintenance contract for the OCR equipment that was being used to process information returns. After the 1995 filing season, IRS issued a statement of work for system enhancements that would help stabilize SCRIPS performance in 1996. According to IRS officials, they purchased those enhancements that they believed offered the greatest potential for improving SCRIPS’ performance in fiscal year 1996. For example, IRS purchased a third scanner for each service center that could (1) be used as a backup if one of the two primary scanners failed and (2) provide the ability to scan documents ahead and have them wait in queue for further processing. Many of these Page 17 GAO/GGD-97-29 Performance of SCRIPS B-270263 enhancements were implemented late in the 1996 filing season, after most of the Forms 1040EZ and information returns had been processed. According to IRS’ Office of Assistant Chief Counsel, IRS cannot hold the contractor to any specific performance requirements for the number of documents that SCRIPS must process within a specific time period, (e.g., per week, per hour). When IRS modified the contract to reflect that SCRIPS would be put in 5 centers instead of 10, according to the Assistant Chief Counsel’s Office, it did not clearly establish throughput requirements—the number of documents to be scanned per hour. Thus, despite paying for enhancements for 1996, IRS has determined that it cannot currently hold the contractor to any specific performance requirements. In February 1996, IRS Internal Audit recommended that IRS finalize throughput requirements for SCRIPS and do a test to determine whether the contractor is meeting the requirements. IRS officials told us that they are examining options for incorporating throughput requirements into the contract. IRSofficials said that they would have a better foundation for establishing throughput requirements once IRS evaluates the impact of the enhancements on SCRIPS’ capacity. According to contractor officials, given the enhancements made in 1996, they believe SCRIPS is capable of processing more documents than it has. They pointed out that SCRIPS’ capability to process documents depends not only on the system’s design but also on IRS’ human resource decisions, such as the number of work stations that are staffed and employee incentives (or lack thereof). IRSofficials tested SCRIPS in late September 1996. According to the test plan, the purpose of the test was to determine (1) the maximum number of FTD coupons, Forms 1040EZ, and information returns that SCRIPS can process; (2) the amount of free time, if any, that will be available to process Forms 941; (3) any bottlenecks in the SCRIPS system that can be eliminated to increase system throughput; and (4) the performance thresholds that could be put into the SCRIPS contract. However, because the software application for Form 941 was not complete, it would have been difficult to fully assess SCRIPS’ processing capability. The results of the test were not available when we completed our audit work. Expected Costs Have Increased Although the workload being processed on SCRIPS continues to be less than and Anticipated Labor Cost expected, SCRIPS’ estimated costs have risen. Also, anticipated labor cost Savings Have Decreased savings have decreased. Page 18 GAO/GGD-97-29 Performance of SCRIPS B-270263 According to IRS’ post-implementation review report, previous cost estimates for SCRIPS have ranged from $133 million to $209 million. The estimate of $133 million was made in February 1992 and again in April 1992. That cost estimate, however, assumed that SCRIPS would be implemented in 1994 and did not include the cost of maintaining SCRIPS. The current life-cycle cost estimate for SCRIPS is $288 million, which includes at least $20 million for maintenance.17 That estimate was included in the Department of the Treasury’s May 6, 1996, report to Congress on IRS’ progress in responding to our recommendations on the managerial and technical weaknesses of TSM.18 We could not determine how much IRS has already spent on SCRIPS since its inception because IRS does not have an accurate cost accounting system. Using the latest life-cycle cost estimate, SCRIPS is estimated to have cost about $145 million from fiscal year 1989 through fiscal year 1996. In October 1994, IRS estimated that SCRIPS would provide about $17 million in labor cost savings from fiscal years 1994 through fiscal year 2000. In September 1995, IRS lowered that estimate to about $5 million. Also, IRS’ September 1996 investment evaluation report on SCRIPS concluded that the system will yield a negative return on investment (i.e., costs will exceed benefits) from 1991 to 2001. However, the evaluation report stated that its return on investment estimate does not fully capture the operational benefits of the hardware and software enhancements that were made since the end of fiscal year 1995. Therefore, the report concludes that the final judgment on SCRIPS’ performance cannot be made until after the 1997 filing season. As discussed previously, SCRIPS was not used to process forms 1040PC and Use of SCRIPS for 941 in 1995 or 1996. In July 1995, IRS decided to terminate all software Form 941 Is Uncertain programming for the 1040PC because of SCRIPS’ instability, a lack of system capacity, and a number of software application problems. It is uncertain when, if at all, SCRIPS will be used to process Forms 941. The President’s fiscal year 1997 budget request included $850 million for TSM, about $38 million of which was for SCRIPS. In a June 6, 1996, letter, Treasury submitted a revised TSM funding request of $664 million to the House Appropriations Committee, of which about $30 million was for 17 See footnote 7. 18 Report to House and Senate Appropriations Committees: Progress Report on IRS’ Management and Implementation of Tax Systems Modernization, May 6, 1996. Page 19 GAO/GGD-97-29 Performance of SCRIPS B-270263 SCRIPS. According to the letter, at this funding level, SCRIPS would not be used to process Forms 941 for fiscal year 1997. Congress subsequently appropriated $336 million for TSM for fiscal year 1997. About 60 percent of that appropriation is earmarked for operational TSM projects, such as SCRIPS, but it was unclear when we prepared this report how much IRS would allocate to SCRIPS. However, we would expect that IRS will continue funding SCRIPS because the project is operational and IRS has no backup system for FTD coupons. In addition to funding constraints, decisions on the use of SCRIPS beyond fiscal year 1997 for Forms 941, according to IRS officials, will hinge on: (1) the results of the September 1996 test of SCRIPS, which we discussed earlier; and (2) Investment Review Board actions on recommendations made by a paper processing task team. In March 1996, IRS convened a task team to develop a paper processing strategy due to delays in implementing various aspects of IRS’ tax return processing vision. Specifically, IRS had originally expected that (1) a significant number of returns would be received electronically, thereby reducing the need for some manual data entry; and (2) DPS would be positioned to begin processing the remaining paper tax returns. As we reported in October 1995, IRS’ electronic filing program is falling short of expectations.19 Current estimates indicate that IRS may receive only 33 million electronic returns in 2001 rather than the 80 million that IRS had set as its goal. The shortfall in meeting IRS’ 80-million goal stems from the lack of a comprehensive business strategy to attract taxpayers to electronic filing. IRS is currently trying to develop such a strategy. Also, IRS announced on October 8, 1996, that it was terminating the DPS project because of “revised priorities and budget realities for the next several years.” Among other things, the task team developed a paper processing strategy based on the assumption that the maximum number of electronic returns that can be expected in 2001 is about 33 million. In addition, the team evaluated the need for a replacement system for IRS’ current manual data entry system. According to IRS officials, one part of the paper processing strategy may be to contract out the processing of some documents. These documents could include information returns because the processing of those documents is not as time-sensitive as the processing of income tax returns. In addition, in the future, more FTD coupons are to be submitted electronically as called for in the North American Free Trade Agreement. 19 Tax Administration: Electronic Filing Falling Short of Expectations (GAO/GGD-96-12, Oct. 31, 1995). Page 20 GAO/GGD-97-29 Performance of SCRIPS B-270263 For example, IRS estimates that about 44 percent of the FTD coupons will be received electronically in fiscal year 1998, compared with only 1 percent in fiscal year 1996. Thus, given that some of SCRIPS’ existing workload could decrease or be contracted out, IRS could possibly increase the number of Forms 1040EZ that SCRIPS could process or add additional forms, such as Forms 941, without requiring any additional system capacity beyond what is currently available. Because SCRIPS was developed before IRS started taking actions to address Conclusions the managerial and technical weaknesses of TSM that we identified in July 1995, SCRIPS suffered from some of those weaknesses that we said would contribute to such things as cost overruns and failure to meet mission goals. SCRIPS is expected to cost more than originally expected and, according to IRS’ September 1995 estimates, could provide less than one-third of the originally expected labor cost savings. We recognize that IRS is now completely reliant on SCRIPS for processing paper FTD coupons. However, the decrease in expected labor cost savings and the increase in estimated costs raise questions about the cost-effectiveness of SCRIPS. In addition, one of the most critical weaknesses for SCRIPS was a lack of thorough and complete system testing before the system was rolled out to five service centers. Although IRS tested SCRIPS in September 1996 to determine the maximum number of Forms 1040EZ, information returns, and FTD coupons SCRIPS can process, that test does not substitute for a test of SCRIPS’ ability to process any new document types along with the existing ones under a production environment that replicates peak volume conditions. We recommend that before deciding to increase the percentage of Forms Recommendation to 1040EZ or information returns that SCRIPS processes or using SCRIPS to the Commissioner of process other tax forms, such as Forms 941, the Commissioner (1) do a Internal Revenue cost-benefit analysis that includes examining the costs and benefits of alternative ways for processing those forms, such as those developed by the paper processing task team; and (2) if the analysis shows that it is cost-effective to have SCRIPS process Forms 941, ensure that IRS tests SCRIPS’ ability to process the existing software applications along with any new software applications that may be added using peak volumes or volumes simulating peak workload conditions to ensure that SCRIPS can meet performance expectations. Page 21 GAO/GGD-97-29 Performance of SCRIPS B-270263 We requested comments on a draft of this report from the Commissioner Agency Comments of Internal Revenue or her designated representative. Responsible IRS and Our Evaluation officials, including the National Director for Submission Processing, the Assistant Commissioner for Forms and Submission Processing, and the SCRIPS project manager, provided IRS’ comments in a November 14, 1996, meeting. Their comments on our recommendation were reiterated in a November 19, 1996, memorandum from the Acting Chief of Taxpayer Service. Besides commenting on our recommendation, the Deputy Chief Information Officer, Systems Development, provided several factual clarifications that we incorporated in the report where appropriate. The officials also asked that we update our report to reflect processing rate information through September 30, 1996, and they gave us the relevant data. We made that update. IRS officials generally agreed with our recommendation. With respect to the first part of our recommendation, the officials generally agreed that IRS should do a cost-benefit analysis before deciding to increase the percentage of Forms 1040EZ or information returns that SCRIPS processes or to add additional forms, such as Forms 941. They said that this analysis would be done by the end of fiscal year 1997 and would take into account approved results from the paper processing task team. IRS officials said that in the event of workload imbalances during the 1997 tax return filing season, if necessary, they may decide to increase the workload for SCRIPS before the cost/benefit analysis is completed. We recognize that IRS’ priority must be to process tax returns during the filing season in a timely manner, and it needs to reserve the right to do so. With respect to the second part of our recommendation, IRS officials said that they plan to work with the contractor to identify and implement any necessary contract modifications that may be needed to ensure complete testing. They said the testing will ensure that SCRIPS can meet performance expectations in a peak production environment. IRSofficials said that a few changes are planned for the 1997 filing season that could help improve SCRIPS’ future performance. Specifically, they mentioned that IRS would be testing different incentive systems for SCRIPS operators to determine the extent to which incentives affect operator performance, which could also affect overall SCRIPS’ performance. They also mentioned that IRS is negotiating with the contractor to provide a new scanner feeder that should resolve some of the problems experienced when information returns are filed on extremely thin paper. Page 22 GAO/GGD-97-29 Performance of SCRIPS B-270263 IRS officials expressed concern about using $133 million as the baseline cost estimate for SCRIPS. That estimate was included in a February 1992 business case and again in a revision dated April 23, 1992. IRS officials said that we should have used an earlier cost estimate of $209 million that was included in IRS’ Information Systems Initiative Summary Database in August 1991. IRS’ post-implementation review report on SCRIPS also stated that the SCRIPS project office considered the $209 million as the baseline life-cycle cost estimate. However, the report also noted that between fiscal years 1993 and 1996, IRS cited four other different life-cycle cost estimates for budget purposes, including an estimate of $132 million. We revised the report to acknowledge that $209 million was one of the cost estimates for SCRIPS, but we believe that the business case estimate is an appropriate baseline because the business case was a major basis for decisions to go forward with SCRIPS. IRS officials said that IRS developed the SCRIPS business case before it began taking steps to manage information technology projects as investments. Since that time, for example, IRS has developed an investment justification handbook to help ensure that project cost and benefit analyses that are included in business cases are standardized and complete. We are sending copies of this report to the Subcommittee’s Ranking Minority Member, the Chairman and Ranking Minority Member of the House Committee on Ways and Means, the Chairman and Ranking Minority Member of the Senate Committee on Finance, various other congressional committees, the Secretary of the Treasury, the Commissioner of Internal Revenue, the Director of the Office of Management and Budget, and other interested parties. Major contributors to this report are listed in the appendix. Please contact me on (202) 512-9110 if you have any questions. Sincerely yours, Lynda D. Willis Director, Tax Policy and Administration Issues Page 23 GAO/GGD-97-29 Performance of SCRIPS Appendix I Major Contributors to This Report David Attianese, Assistant Director, Tax Policy and Administration Issues General Government Sherrie Russ, Evaluator-in-Charge Division, Washington, Christopher Hess, Evaluator D.C. Monika Gomez, Evaluator Cecelia Ball, Senior Evaluator Kansas City Field Marvin McGill, Evaluator Office (268708) Page 24 GAO/GGD-97-29 Performance of SCRIPS Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Tax Systems Modernization: Imaging System's Performance Improving but Still Falls Short of Expectations
Published by the Government Accountability Office on 1997-01-16.
Below is a raw (and likely hideous) rendition of the original report. (PDF)