oversight

Tax Systems Modernization: Imaging System's Performance Improving but Still Falls Short of Expectations

Published by the Government Accountability Office on 1997-01-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to the Chairman, Subcommittee
                on Oversight, Committee on Ways and
                Means, House of Representatives


January 1997
                TAX SYSTEMS
                MODERNIZATION
                Imaging System’s
                Performance
                Improving but Still
                Falls Short of
                Expectations




GAO/GGD-97-29
             United States
GAO          General Accounting Office
             Washington, D.C. 20548

             General Government Division

             B-270263

             January 16, 1997

             The Honorable Nancy L. Johnson
             Chairman, Subcommittee on Oversight
             Committee on Ways and Means
             House of Representatives

             Dear Chairman Johnson:

             One of the major objectives of the Internal Revenue Service’s (IRS)
             modernization effort is to move away from a labor-intensive tax return
             processing system that relies on thousands of employees transcribing data
             from paper tax returns and move to an electronic system that reduces
             processing costs and eliminates transcription errors. One strategy for
             achieving that objective is to reduce the number of paper returns by
             increasing the number of returns filed electronically. For returns that will
             continue to be filed on paper, IRS plans to achieve its objective through
             document imaging and optical character recognition (OCR). The Service
             Center Recognition/Image Processing System (SCRIPS) is the first of what
             was to have been two document imaging and OCR systems.1

             SCRIPS became operational in the latter part of 1994, and we reported that it
             experienced numerous performance problems during the 1995 filing
             season.2 This report responds to your request that we follow up on those
             problems and assess the performance of SCRIPS in 1996. Specifically, this
             report (1) identifies the primary causes for performance problems that
             occurred in 1995; (2) assesses whether SCRIPS performance improved in
             1996 as of September 30, 1996; and (3) provides a status report on IRS’
             future plans for SCRIPS.


             In July 1995, we reported on IRS’ progress in implementing some of the
Background   business and technological components of its modernization effort, known
             as Tax Systems Modernization (TSM).3 Although we said that IRS had made
             some progress, we also said that pervasive management and technical
             weaknesses existed that placed the modernization effort at risk. Among


             1
              The second system, the Document Processing System (DPS), was intended to replace SCRIPS and
             expand the imaging capability to more complex tax forms. However, on October 8, 1996, IRS
             announced that it was terminating that project.
             2
              The 1995 Tax Filing Season: IRS Performance Indicators Provide Incomplete Information About Some
             Problems (GAO/GGD-96-48, Dec. 29, 1995).
             3
             Tax Systems Modernization: Management and Technical Weaknesses Must Be Corrected If
             Modernization Is To Succeed (GAO/AIMD-95-156, July 26, 1995).



             Page 1                                                 GAO/GGD-97-29 Performance of SCRIPS
B-270263




other things, (1) IRS did not have a business strategy to maximize
electronic filing, the result of which could slow the planned decrease in
the workload of paper processing systems; and (2) IRS lacked the full range
of managerial and technical foundations to realize its modernization
objectives. Some of these key foundation components were a complete
cost/benefit analysis of the overall modernization effort and thorough
testing of individual systems before they were implemented. In September
1996, we reported on IRS’ progress in addressing the managerial and
technical weaknesses we identified in July 1995.4 We concluded that
although IRS is working to resolve these weaknesses, it had not fully
satisfied any of our recommendations.

SCRIPS was one of the systems that was designed under the conditions cited
in our July 1995 report. It was intended to replace the aging OCR equipment
that IRS had been using to process all of the paper Federal Tax Deposit
(FTD) coupons; almost all of the paper information returns (e.g, Forms
1099); some of the individual income tax returns filed on Form 1040EZ;
and some employment tax returns (Form 941). In addition, SCRIPS was
expected to process Form 1040PC, a paper form that taxpayers can
generate when they use computer software to prepare a tax return.

Figure 1 shows the various SCRIPS components. Under the character
recognition and image capture component of SCRIPS, scanners (1) read
information from the document and convert the information to
machine-readable format for later computer processing and (2) create an
image of the document. In the event of recognition errors during
document scanning, IRS staff can access an image of the tax return in lieu
of having to locate the original paper tax return to make corrections. IRS
expected that having an image of the tax return would improve the
productivity of staff doing data validation. In addition, the images of FTD
coupons and certain information return documents are stored on optical
disk for later use. The older OCR systems used microfilming as the storage
medium—essentially a manual process that required more physical
storage space than optical disks. IRS continues to retain a paper copy of the
Form 1040EZ rather than store the image because IRS has certain legal
concerns.5



4
 Internal Revenue Service: Business Operations Need Continued Improvement
(GAO/AIMD/GGD-96-152, Sept. 9, 1996).
5
 One concern is the validity of an imaged signature. Another concern is that SCRIPS images only one
side of the two-sided Form 1040EZ. The second page includes a worksheet that taxpayers use for tax
calculations, and IRS considers it part of the tax return.



Page 2                                                   GAO/GGD-97-29 Performance of SCRIPS
B-270263




Page 3     GAO/GGD-97-29 Performance of SCRIPS
                                          B-270263




Figure 1: SCRIPS Components




            Character Recognition                      Image Processing                 Data Management
              & Image Capture                        The image controller helps       The multiprocess server
           A scanner operator feeds a block          ensure that each scanned         maintains control over the tax
           of documents into the scanner.            document image moves from        data. All changes and updates
           The scanner numbers each                  the scanner to the server. It    are processed through the
           document, takes an image, and             also provides data backup        server.
           makes the first OCR attempt.              until images reach the server.


                                                                                                                 Updates



                                                                                                                 Final
                                                                                                                 Output


 3 Scanners with Monitor, Keyboards and Printer a                                                                Updates
                                                        3 Image Controllers

                                                                                       1 Multiprocessor
                                                                                         Server




                                                                                                                 Updates




                                          Page 4                                      GAO/GGD-97-29 Performance of SCRIPS
                                              B-270263




Cont.
                       Secondary Character                                               Outputs
                    Recognition and Identification
        Secondary character recognition is used for those fields (1) where
        characters are not completely read by the scanner or (2) IRS
        requires secondary recognition. Tax data is read by the recognition
        engine, and the block reader reads name and address information.




                      Updates
                                                                                        Optical Disks
                                                                                        Store FTDs
                                                                                        and information
                                                                                        return images.

                              5 Recognition Engines
                                for Tax Data
                                  Final Output


                 Updates
                                                                                        8 mm Tape Drive
                                                                                        Stores tax data.
                              2 Name and Address
                                Block Readers

        In those instances where the secondary recognition and name and
        address block reader cannot identify the correct characters, the file
        is sent to a workstation operator. That operator reviews the image
        (with the capability to magnify it) to determine what corrections are
        needed. Updated files are sent back to the file server.                        9-Track Tape Drive
                                                                                       Forwarded to the next
                                                                                       stage with tax data for
                                                                                       further processing.

                    Updates



                                66 Workstations

                                              Source: GAO graphic based on IRS data.




                                              Page 5                                   GAO/GGD-97-29 Performance of SCRIPS
                   B-270263




                   IRS expected that SCRIPS would provide faster, more accurate document
                   processing. Specifically, IRS expected that SCRIPS would result in a
                   20 percent productivity increase over manual data entry and a 10 percent
                   productivity increase over older OCR equipment. Other expected benefits
                   included lower costs for system maintenance and storage of tax return
                   data.

                   Originally, IRS planned to implement SCRIPS in all 10 service centers where
                   it currently processes paper returns. However, after the contract was
                   awarded in February 1993, IRS decided to consolidate paper tax return
                   processing in five centers. Accordingly, SCRIPS was tested in Cincinnati in
                   the summer of 1994, and the other four SCRIPS centers began using SCRIPS
                   between September and November 1994.


                   SCRIPS experienced significant performance problems in 1995. Two
Results in Brief   problems were system downtime and slow processing rates. For example,
                   although SCRIPS was expected to increase productivity by 20 percent over
                   the manual data entry system, SCRIPS actually processed Forms 1040EZ in
                   1995 at a rate that was about 7 percent slower than manual data entry.
                   Because of performance problems, two of the five centers stopped
                   processing Forms 1040EZ on SCRIPS and reverted to using manual data
                   entry.

                   SCRIPS performance deficiencies in 1995 stemmed primarily from both
                   hardware and software problems. Hardware problems occurred with the
                   scanner that captures the image of the document. Software problems
                   occurred with the operating system for the image controller that
                   reconciles the number of documents scanned to the number of documents
                   in the SCRIPS database. The hardware and software problems were not
                   detected before the rollout of SCRIPS because testing in Cincinnati was
                   incomplete. For example, SCRIPS was never tested at the peak volumes that
                   IRS experiences during a filing season.


                   Although IRS had expected that SCRIPS would be processing five document
                   types—forms 1040EZ, 941, and 1040PC; FTD coupons; and information
                   returns—IRS postponed plans to process forms 941 and 1040PC on SCRIPS.
                   Of the three remaining document types, the Cincinnati test certified the
                   software application only for FTD coupons. Therefore, the software
                   applications for information returns and Form 1040EZ were not
                   thoroughly tested before they were put into production.




                   Page 6                                     GAO/GGD-97-29 Performance of SCRIPS
B-270263




To improve the performance of SCRIPS for 1996, IRS made hardware and
software modifications, some of which were made before the start of the
1996 filing season. As a result of some of these enhancements and more
staff familiarity with SCRIPS, according to IRS officials in all five SCRIPS
service centers, SCRIPS performed significantly better during the 1996 filing
season than it did in 1995. For example, between April and June,
unscheduled system downtime decreased from about 791 hours in 1995 to
about 43 hours in 1996. Also, performance on another indicator—the
number of documents processed per hour—improved for Forms 1040EZ
and FTD coupons.6 Specifically, the number of Forms 1040EZ processed
per hour in 1996 increased by four documents per hour (about 6 percent)
and the number of FTD coupons increased by 3 documents per hour (less
than 1 percent). However, the number of information returns processed
per hour decreased by five documents per hour (about 3 percent).

In addition to a slower processing rate for information returns, SCRIPS (1) is
not processing all the forms that it was expected to process in 1996, (2) is
expected to cost more than originally estimated, and (3) is expected to
provide lower labor cost savings than IRS originally anticipated. IRS’
October 1994 business case for SCRIPS stated that in 1996 SCRIPS would be
processing all FTD coupons and information returns, all Forms 1040EZ,
50 percent of the Forms 1040PC, and 93 percent of the Forms 941.
However, during the first 9 months of 1996, SCRIPS processed only FTD
coupons, 60 percent of the information returns, and about 50 percent of
the Forms 1040EZ. This reduced level of performance stems in large part
from (1) IRS’ decision, after the contract was awarded, to consolidate all
paper processing in 5 service centers instead of 10; and (2) IRS’ decision to
purchase only 5 systems when the contractor was planning to provide 10
systems—1 in each of the 10 service centers. Furthermore, IRS officials did
not know the extent to which the hardware and software modifications
that had already been made for 1996 and those that were planned for later
in the year would increase SCRIPS’ capability for processing more 1040EZs
and information returns. Therefore, IRS officials did not significantly
increase the expectations for SCRIPS in 1996 over those that they had in
1995.

The latest cost estimate for SCRIPS is $288 million—considerably more than
previous cost estimates, which, according to IRS’ post-implementation

6
 The number of documents processed per hour, commonly referred to as the composite rate, includes
not only the time to scan the documents but also the time spent by work station operators validating
and correcting tax return data. Validation and correction are necessary in those instances when the
scanner and secondary recognition systems cannot read certain tax return data or the system detects
an error made by a taxpayer.



Page 7                                                   GAO/GGD-97-29 Performance of SCRIPS
                         B-270263




                         review report on SCRIPS, ranged from $133 million to $209 million. At least
                         $20 million of that increase is attributed to maintenance costs that were
                         not included in either the $133 million or $209 million estimates.7 Also, IRS
                         estimated in October 1994 that SCRIPS would provide about $17 million in
                         labor savings from fiscal years 1994 through 2000. In September 1995 IRS
                         lowered that estimate to about $5 million. Furthermore, a September 1996
                         IRS investment evaluation of SCRIPS estimated that SCRIPS will yield a
                         negative return on investment (i.e., SCRIPS’ costs will exceed its benefits)
                         from 1991 to 2001. However, the evaluation noted that the estimate is
                         based on performance in 1995 and does not reflect the software and
                         hardware modifications that were made in 1996.

                         In July 1995, IRS decided to terminate all software programming for the
                         1040PC because of SCRIPS’ instability, a lack of system capacity, and a
                         number of software application problems. Also, IRS has decided not to use
                         SCRIPS to process Forms 941 in fiscal year 1997. Decisions on using SCRIPS
                         to process Forms 941 beyond fiscal year 1997 have not been finalized. In
                         addition to funding constraints, according to IRS officials, a decision on
                         using SCRIPS for Forms 941 will depend on (1) the results of a
                         September 1996 capacity test of SCRIPS, which were not available when we
                         completed our audit work; and (2) decisions by IRS’ Investment Review
                         Board8 on recommendations made by a paper processing task team.


                         Our objectives were to (1) determine the primary causes for SCRIPS
Objectives, Scope,       performance problems in 1995; (2) assess whether those problems were
and Methodology          corrected as of September 30, 1996; and (3) provide a status report on IRS’
                         future plans for SCRIPS.

                         To accomplish these objectives, we did the following:

                     •   We interviewed National Office officials in the SCRIPS project office and the
                         taxpayer service function, which has responsibility for those tax forms
                         processed on SCRIPS, to obtain their views on the extent to which SCRIPS’
                         performance improved during fiscal year 1996.



                         7
                          The supporting documentation for IRS’ $288 million cost estimate showed a separate breakout for
                         maintenance costs for only fiscal years 1995 and 1996. We were unable to determine what, if any,
                         portion of the costs for fiscal years 1997 to 2000 was attributed to maintenance. According to IRS
                         officials, maintenance costs may have been combined with other costs for those fiscal years.
                         8
                          In October 1995, IRS created an executive-level Investment Review Board, currently co-chaired by the
                         Deputy Commissioner and the Chief Financial Officer, for selecting, controlling, and evaluating all of
                         IRS’ information technology investments.



                         Page 8                                                    GAO/GGD-97-29 Performance of SCRIPS
    B-270263




•   We interviewed IRS contracting office officials to determine what, if any,
    performance requirements the contractor was being held to in 1995 and
    1996.
•   We interviewed officials at all five SCRIPS service centers to determine
    whether performance had improved in 1996 and to identify the
    performance indicators officials were using to evaluate SCRIPS
    performance.
•   We observed SCRIPS in operation at the service centers in Memphis and
    Cincinnati and at the program development site in Washington, D.C.
•   We interviewed contractor officials about the workload requirements that
    IRS had specified for SCRIPS.
•   We reviewed IRS business cases9 for SCRIPS that documented the objectives
    of the system, its expected benefits, and its estimated cost. IRS prepared
    business cases for SCRIPS in February 1992, April 1992, December 1993,
    October 1994, and December 1995. To assess SCRIPS, we used the
    performance expectations that were included in the October 1994 business
    case because it was the one that was in effect at the beginning of the 1995
    filing season.
•   We reviewed IRS evaluations and reports on the implementation of SCRIPS,
    including an August 1995 performance evaluation report; a December 1995
    post-implementation review report that assessed whether business goals
    were met; a May 1995 Internal Audit report on SCRIPS testing in Cincinnati;
    a February 1996 Internal Audit report on the rollout of SCRIPS; and a
    September 1996 report on the results of an investment evaluation of SCRIPS.
•   We compared available SCRIPS performance data for January through
    September 1995 and 1996.
•   We computed composite rates—the number of documents processed per
    hour—for Form 1040EZ, information returns, and FTD coupons. To
    compute those rates, we used IRS data on the number of hours spent for
    various aspects of tax return processing, including scanning, data
    correction, and data validation for January through September 1995 and
    1996. Our composite rate calculations, similar to those done by IRS shortly
    after SCRIPS was implemented, did not include time for document
    preparation or another function that is referred to as code and edit.10



    9
     IRS prepares business cases to justify an information technology project and to demonstrate that it is
    cost beneficial.
    10
      For SCRIPS, the IRS data we received for computing composite rates showed the combined hours for
    both the code and edit function and the document preparation function. During code and edit, staff
    prepare returns for computer entry by, among other things, ensuring that all data are present and
    legible. For the manual entry system, the code and edit function was not combined with the document
    preparation function, and the hours associated with document preparation were not shown. Therefore,
    we did not include these aspects of processing in our composite rate comparison.



    Page 9                                                     GAO/GGD-97-29 Performance of SCRIPS
                        B-270263




                        We did our audit work between October 1995 and September 1996 and in
                        accordance with generally accepted government auditing standards. Other
                        than identifying obvious reporting errors, we did not verify the accuracy of
                        IRS’ data on the number of hours spent for various aspects of tax return
                        processing. We did attempt to detemine what elements were included in
                        IRS’ cost estimates for SCRIPS, but we did not attempt to verify the accuracy
                        of the costs for those elements. We requested comments on a draft of this
                        report from the Commissioner of Internal Revenue or her designated
                        representative. Responsible IRS officials, including the National Director
                        for Submission Processing, the Assistant Commissioner for Forms and
                        Submission Processing, and the SCRIPS project manager, provided IRS’
                        comments in a November 13, 1996, meeting. Their comments on our
                        recommendation were reiterated in a November 19, 1996, memorandum
                        from the Acting Chief of Taxpayer Service. IRS’ comments are summarized
                        and evaluated on pages 22 and 23.



                        SCRIPS performed well below expectations in 1995. Extensive, unscheduled
SCRIPS Performed        system downtime and slower than expected processing rates affected
Substantially Below     SCRIPS’ ability to meet the expectations that IRS had established before the

Expectations in 1995    start of the 1995 filing season. Hardware problems with the scanner
                        contributed to significant amounts of system downtime; various software
                        problems contributed to slow processing rates.


SCRIPS Processed Less   In 1995, SCRIPS processed 19 percent more FTD coupons than IRS expected.
Volume and at Slower    However, as shown in table 1, SCRIPS did not meet IRS’ volume expectations
Rates Than Expected     for the three other document types scheduled for processing in
                        1995—information returns, Form 1040EZ, and Form 941 (IRS did not
                        expect to start processing Form 1040PC on SCRIPS until 1996).




                        Page 10                                    GAO/GGD-97-29 Performance of SCRIPS
                                      B-270263




Table 1: Number of Documents
SCRIPS Expected to Process and                                               Number IRS expected to                Number actually
Number Actually Processed in 1995     Document type                                process in 1995a              processed in 1995
                                      FTD coupons                                         76,422,540                      90,630,944
                                      Information returns
                                                                                          57,369,000                      24,897,860
                                      Form 1040EZ                                          8,594,500                       4,873,419
                                      Form 941                                             4,826,140                                0b
                                      a
                                          IRS expectations as of October 1994.
                                      b
                                       Sometime between October and December 1994 IRS decided not to use SCRIPS to process
                                      Forms 941.

                                      Source: IRS data.



                                      SCRIPS processed 19 percent more FTD coupons than IRS had expected
                                      because more FTD coupons were filed than expected, and service center
                                      officials placed the highest priority on processing those forms. That
                                      priority stemmed from (1) IRS procedures that require that 90 percent of all
                                      FTD coupons be processed within 24 hours of receipt; and (2) the absence
                                      of a backup processing system, because IRS had cancelled its maintenance
                                      contract for the older OCR equipment that had been processing FTD
                                      coupons. IRS had backup systems for the other documents that SCRIPS was
                                      expected to process in 1995. The manual data entry system could be used
                                      for forms 1040EZ and 941, and IRS extended the maintenance contract for
                                      the older OCR equipment that was processing information returns.

                                      Not only did SCRIPS not process the number of forms expected, but the
                                      speed with which it did its processing was slower than expected. As
                                      shown in table 2, SCRIPS did not meet the processing rate expectations for
                                      any of the three document types it processed in 1995. Moreover, the actual
                                      processing rate for Forms 1040EZ in 1995 was about 7 percent less than
                                      the rate achieved in 1995 by manual data entry.

Table 2: Expected and Actual
Processing Rates for SCRIPS in 1995                                         Processing rate (documents per hour) in 1995
Compared to Rates for Systems                                              Expected for            Actual for    Actual for older
SCRIPS Was to Replace                 Document type                            SCRIPS               SCRIPS       systems
                                      1040EZ                                        186                    63    68
                                      Information returns                           178                   147    131
                                      FTD coupons                                 1,170                   758    Not applicablea
                                      a
                                        IRS cancelled the maintenance contract for the OCR system that had been processing FTD
                                      coupons. SCRIPS was the only system used to process FTD coupons in 1995.

                                      Source: GAO computations using IRS data for January through December 1995.



                                      Page 11                                               GAO/GGD-97-29 Performance of SCRIPS
B-270263




Because of performance problems, two of the five service centers stopped
using SCRIPS to process Forms 1040EZ in 1995.11 Instead, they reverted to
manual data entry, which required using more staff resources than
planned, thus increasing processing costs. IRS had planned to use 25.6 staff
years to process other-than-full-paid12 Forms 1040EZ during the 1995 filing
season in the five SCRIPS centers but used 66.5 staff years instead.13

Hardware problems contributed to SCRIPS’ performance deficiencies in
1995. According to IRS’ August 1995 performance evaluation report, the
SCRIPS scanner experienced the most hardware failures, which contributed
to a “substantial amount of downtime” at the centers.14 According to IRS
data, between April and June 1995, the five service centers experienced
about 791 hours of unscheduled downtime. In addition, the scanner
jammed when paper was extremely thin, which was sometimes the case
with information returns.

Software problems also occurred. According to IRS officials and
evaluations of SCRIPS, the image controller did not operate fast enough to
keep up with the scanner because the operating system software was
inefficient. The image controller is to track each scanned document image
file to ensure that it moves from the scanner and ultimately to the server
where it is stored. This component helps reconcile the number of
documents scanned to the number of documents in the database. IRS
officials and evaluations of SCRIPS also indicated that SCRIPS did not provide
accurate reports on this reconciliation—referred to as “run-to-run”
balancing. The FTD coupon run-to-run balancing report, for example, did
not provide total counts to help IRS staff confirm that all of the FTD
coupons in a block that was scanned had in fact been processed. This
balancing is particularly important for FTD coupons because IRS needs to
accurately classify tax deposits to the appropriate Treasury account. At
one center we visited, officials told us that they had to revert to manual
counts for doing run-to-run balancing.



11
 According to IRS officials, service center directors had discretion in managing the processing
workload. Some chose to revert to the manual system; others did not.
12
  Other-than-full-paid tax returns are those returns filed by taxpayers who either were due a refund or
did not pay the full amount of tax owed at the time of filing. These returns were about 92 percent of
the Forms 1040EZ filed at the five SCRIPS service centers during the 1995 filing season.
13
  The number of staff years for Form 1040EZ includes those used on SCRIPS and manual data entry.
14
  IRS’ evaluation of SCRIPS performance during the 1995 filing season noted that wide variations in
scanner operator skills also affected the scanner’s performance. IRS officials also noted that even
though the scanner may have been “down,” other SCRIPS components could have been operating.



Page 12                                                    GAO/GGD-97-29 Performance of SCRIPS
                           B-270263




Performance Problems Not   Many of the problems experienced with SCRIPS in 1995 might have been
Thoroughly Identified      anticipated if IRS had thoroughly tested SCRIPS before installing the system
Before Rollout Due to      in the other four service centers. The pilot test of SCRIPS was incomplete
                           because it (1) did not certify all software applications that were to be used
Incomplete Testing         during 1995; and (2) did not test SCRIPS’ ability to handle peak processing
                           volumes, such as those experienced in the tax return filing season.

                           An organization tests a new or modified system to detect system design
                           and development errors and to correct them before putting a system into
                           operation. In our July 1995 report on TSM, we said that although IRS
                           recognized the importance of testing, it had not yet developed a complete
                           and comprehensive testing plan for TSM.15 We said that individual TSM
                           systems were developing their own test plans, which IRS described as
                           rudimentary and inadequate. This testing environment was in effect when
                           IRS did a pilot test of SCRIPS at the Cincinnati Service Center in the summer
                           of 1994.

                           The purpose of a pilot test is to evaluate the performance of a system in
                           one location before deciding whether to implement the system at other
                           locations. IRS uses the pilot test to certify that the system is meeting its
                           program or business objectives. During the pilot test, IRS is to collect data
                           on the performance of the system and compare the data against
                           established performance goals to certify that the system is performing as
                           expected.

                           Due to delays in receiving and testing software, only the FTD coupon
                           application was certified as a result of the pilot test. The certifications for
                           Forms 1040EZ and information returns were not done due to management
                           concerns that conducting these certifications would delay the rollout of
                           SCRIPS to five centers, which was scheduled for January 1995. As a result of
                           incomplete testing, for example, IRS did not identify problems with the
                           information returns software until SCRIPS was fully operational. Due to
                           these software problems, a higher percentage of returns were sent to data
                           validation than originally anticipated, thus decreasing system productivity.
                           According to IRS officials, those problems were corrected before the 1996
                           filing season.

                           The pilot test was also incomplete because it did not (1) provide IRS a clear
                           indication of SCRIPS’ ability to perform under peak workload conditions
                           and (2) show the impact of running multiple software applications. If the
                           system had been tested at larger volumes, running multiple software

                           15
                             GAO/AIMD-95-156.



                           Page 13                                     GAO/GGD-97-29 Performance of SCRIPS
B-270263




applications, problems may have been identified earlier and resources
could have been diverted to address performance problems before the
system’s rollout. According to an Internal Audit report on the pilot test, the
contract required that SCRIPS process 200,000 documents a day during peak
periods. During the 1-day volume test in the pilot, only 45,000 documents
were scanned into the database. The remaining components of the system
processed an additional 142,000 documents that had been scanned before
the test. Thus, the scanner, one of the primary sources of downtime during
1995, was not tested under levels IRS experiences in a production
environment. Also, only 10,000 documents were processed
completely—only 5 percent of the daily production workload specified in
the contract.

According to IRS officials, a full production test plan had been developed in
December 1993 that would simulate peak filing season volumes. However,
the plan was never implemented. One reason IRS officials provided for not
implementing the test plan was that IRS never had full access to a SCRIPS
system to do the test. The system that IRS was to use for the test was the
same system that the contractor was using for systems development work.
Another factor that IRS officials cited was a sense of urgency to roll out
SCRIPS for FTD coupons because IRS had no backup processing system.16


IRS’December 1995 report on the post-implementation review of SCRIPS
cited several problems with the testing of SCRIPS in addition to those that
occurred in the pilot test. For example, although an equipment acceptance
test was done, the test was not adequate to validate the system’s readiness
for production. The report also stated that IRS waived an “operational
capabilities demonstration” that would have included (1) tests for the
readability of forms and (2) measures of the number of documents
scanned per hour. The report said that such a test, done as part of the
contract award process, may have shown that the system could not meet
the minimum performance requirements.

The post-implementation review team proposed that tests be done to
verify that the system could handle the production workload. It also stated
that if such a test could not be done, a simulated test should be done
focusing on the performance of key areas of the system, particularly those
that could lead to processing bottlenecks. It also proposed a 1-year pilot
test for forms that may be added to SCRIPS in the future and proposed that
the pilot test include the peak volumes of a filing season.

16
  As previously stated, IRS had cancelled the maintenance contract for the OCR system that IRS was
using to process FTD coupons. In addition, IRS had redistributed the workload of paper FTD coupons
in 1995—consolidating the work of 10 service centers into 4 service centers.



Page 14                                                 GAO/GGD-97-29 Performance of SCRIPS
                                    B-270263




                                    Officials in the five service centers that used SCRIPS in 1996 said that it is
Performance                         performing significantly better than it did in 1995. IRS officials told us that
Improved for 1996,                  the primary performance expectation for SCRIPS in 1996 was system
but SCRIPS                          stabilization. One of the primary indicators these officials used in
                                    evaluating stabilization was the amount of downtime. According to IRS
Performed Below                     service center officials and available IRS data, downtime decreased
Original Expectations               substantially in 1996. This reduction enabled SCRIPS to process more Forms
                                    1040EZ and information returns in 1996 while continuing to process all FTD
and Costs May Be                    coupons. In addition, the processing rates for SCRIPS (i.e., the number of
Higher Than Expected                documents processed per hour) improved for two document types—Form
                                    1040EZ and FTD coupons.

                                    Despite these improvements, the system, as of September 30, 1996, (1) was
                                    not processing all the forms that the October 1994 business case said it
                                    would process, (2) was expected to cost more than original estimates, and
                                    (3) was not expected to provide the estimated labor savings that were
                                    cited in the October 1994 business case.


SCRIPS Performance                  According to service center officials, SCRIPS performed “significantly
Improved in 1996                    better” in 1996 than it did in 1995 because the system experienced
                                    significantly less downtime than in 1995. IRS did not begin tracking
                                    downtime in 1995 until April. Comparable data for April through June 1995
                                    and 1996 show that unscheduled downtime did decrease
                                    significantly—from about 791 hours to 43 hours.

                                    As shown in table 3, mostly because of less unscheduled system
                                    downtime, SCRIPS processed many more documents during the first 9
                                    months of 1996 than it did during the first 9 months of 1995.

Table 3: Number of Documents
SCRIPS Processed in 1995 and 1996                                Number processed as of     Number processed as of
as of September 30                  Document type                   September 30, 1995         September 30, 1996
                                    Form 1040EZ                                 4,870,493                 8,309,384
                                    Information returns                       24,631,809                 22,618,452
                                    FTD coupons                               67,247,198                 70,857,389
                                    Total                                     96,749,500                101,785,225
                                    Source: IRS data.



                                    Also, according to IRS officials, new software was installed in
                                    November 1995 to correct the run-to-run balancing problems that we




                                    Page 15                                     GAO/GGD-97-29 Performance of SCRIPS
                                      B-270263




                                      discussed earlier. According to these officials, since that software was
                                      installed, SCRIPS centers have not reported problems with tracking the
                                      number of documents scanned and processed.

                                      Also in 1996, as shown in table 4, processing rates increased for Forms
                                      1040EZ and FTD coupons.

Table 4: Actual Processing Rates by
Document Type as of September 30,                                            Processing rate (documents per hour)
1995, and 1996                                                                                                   Actual for older
                                                                SCRIPS actual rate SCRIPS actual rate            systems as of
                                      Form type                 as of Sept. 30, 1995 as of Sept. 30, 1996        Sept. 30, 1996
                                      Form 1040EZ                                   63                     67    60
                                      Information returns                         150                     145    133
                                      FTD coupons                                 719                     722    Not applicablea
                                      a
                                        IRS cancelled the maintenance contract for the OCR system that had been processing FTD
                                      coupons. SCRIPS was the only system used to process paper FTD coupons in 1996.

                                      Source: GAO computations based on IRS data.




SCRIPS Processed Many                 Despite the improved performance in 1996, SCRIPS is still doing much less
Fewer Documents Than                  than expected. In addition, estimated costs have increased and estimated
Originally Expected in                labor cost savings have decreased.
1996 While Its Estimated
Costs Have Increased

SCRIPS Processed Fewer                The October 1994 business case stated that in 1996 SCRIPS would be
Documents Than Expected in            processing (1) all Forms 1040EZ, (2) all FTD coupons, (3) all information
1996                                  returns, (4) 93 percent of the Forms 941, and (5) 50 percent of the Forms
                                      1040PC. During the first 9 months of 1996, as in 1995, SCRIPS processed all
                                      the FTD coupons IRS had received but no forms 1040PC or 941. Also,
                                      although SCRIPS processed more Forms 1040EZ and information returns
                                      during the first 9 months of 1996 compared with the first 9 months of 1995,
                                      it still processed only about 50 percent of the Forms 1040EZ and about
                                      60 percent of paper information returns. As discussed in the next few
                                      paragraphs, this reduced level of performance, compared to the
                                      October 1994 business case, stems in large part from a major change in IRS’
                                      plans after the SCRIPS contract was awarded. Furthermore, IRS officials did
                                      not know the extent to which hardware and software modifications that
                                      had been made for 1996 and those that were planned for later in the year




                                      Page 16                                               GAO/GGD-97-29 Performance of SCRIPS
B-270263




would affect SCRIPS’ ability to process more Forms 1040EZ and information
returns. Therefore, IRS officials did not significantly increase the
expectations for SCRIPS in 1996 over those that they had in 1995.

As specified in the contract, SCRIPS was designed on the assumption that it
would be installed in each of the 10 service centers that were then
processing paper returns. However, in December 1993—10 months after
the contract was awarded—IRS announced plans to consolidate the
processing of paper tax returns in five centers. As a result, although the
total workload for SCRIPS remained the same, the volume to be processed
by any one of the five SCRIPS service centers on the average doubled. Thus,
the system that the contractor had designed would not meet the workload
requirements without further systems development work.

In May 1994, IRS attempted to revise the original contract to meet the
volume requirements at five service centers. According to IRS’
post-implementation review report, a statement of work was written to
revise the contract requirements to accommodate the five service center
scenario. In December 1994, at IRS’ request, the contractor proposed
changes totaling about $21 million. According to contractor officials, that
proposal represented an interim attempt to meet the new workload
requirements under a five service center scenario.

According to IRS officials, IRS could not afford all of the proposed
modifications. Also, IRS officials did not believe that all the proposed
changes were needed. As a result, negotiations on these proposals were
never completed. Thus, IRS decided to purchase five systems, one for each
of the five SCRIPS service centers. To compensate for having fewer systems
than intended, IRS decided to (1) continue paper processing of Forms
1040EZ in the other five service centers and (2) extend the maintenance
contract for the OCR equipment that was being used to process information
returns.

After the 1995 filing season, IRS issued a statement of work for system
enhancements that would help stabilize SCRIPS performance in 1996.
According to IRS officials, they purchased those enhancements that they
believed offered the greatest potential for improving SCRIPS’ performance
in fiscal year 1996. For example, IRS purchased a third scanner for each
service center that could (1) be used as a backup if one of the two primary
scanners failed and (2) provide the ability to scan documents ahead and
have them wait in queue for further processing. Many of these




Page 17                                    GAO/GGD-97-29 Performance of SCRIPS
                                B-270263




                                enhancements were implemented late in the 1996 filing season, after most
                                of the Forms 1040EZ and information returns had been processed.

                                According to IRS’ Office of Assistant Chief Counsel, IRS cannot hold the
                                contractor to any specific performance requirements for the number of
                                documents that SCRIPS must process within a specific time period, (e.g., per
                                week, per hour). When IRS modified the contract to reflect that SCRIPS
                                would be put in 5 centers instead of 10, according to the Assistant Chief
                                Counsel’s Office, it did not clearly establish throughput requirements—the
                                number of documents to be scanned per hour. Thus, despite paying for
                                enhancements for 1996, IRS has determined that it cannot currently hold
                                the contractor to any specific performance requirements. In
                                February 1996, IRS Internal Audit recommended that IRS finalize throughput
                                requirements for SCRIPS and do a test to determine whether the contractor
                                is meeting the requirements. IRS officials told us that they are examining
                                options for incorporating throughput requirements into the contract.

                                IRSofficials said that they would have a better foundation for establishing
                                throughput requirements once IRS evaluates the impact of the
                                enhancements on SCRIPS’ capacity. According to contractor officials, given
                                the enhancements made in 1996, they believe SCRIPS is capable of
                                processing more documents than it has. They pointed out that SCRIPS’
                                capability to process documents depends not only on the system’s design
                                but also on IRS’ human resource decisions, such as the number of work
                                stations that are staffed and employee incentives (or lack thereof).

                                IRSofficials tested SCRIPS in late September 1996. According to the test
                                plan, the purpose of the test was to determine (1) the maximum number of
                                FTD coupons, Forms 1040EZ, and information returns that SCRIPS can
                                process; (2) the amount of free time, if any, that will be available to
                                process Forms 941; (3) any bottlenecks in the SCRIPS system that can be
                                eliminated to increase system throughput; and (4) the performance
                                thresholds that could be put into the SCRIPS contract. However, because
                                the software application for Form 941 was not complete, it would have
                                been difficult to fully assess SCRIPS’ processing capability. The results of
                                the test were not available when we completed our audit work.

Expected Costs Have Increased   Although the workload being processed on SCRIPS continues to be less than
and Anticipated Labor Cost      expected, SCRIPS’ estimated costs have risen. Also, anticipated labor cost
Savings Have Decreased          savings have decreased.




                                Page 18                                    GAO/GGD-97-29 Performance of SCRIPS
                        B-270263




                        According to IRS’ post-implementation review report, previous cost
                        estimates for SCRIPS have ranged from $133 million to $209 million. The
                        estimate of $133 million was made in February 1992 and again in
                        April 1992. That cost estimate, however, assumed that SCRIPS would be
                        implemented in 1994 and did not include the cost of maintaining SCRIPS.
                        The current life-cycle cost estimate for SCRIPS is $288 million, which
                        includes at least $20 million for maintenance.17 That estimate was included
                        in the Department of the Treasury’s May 6, 1996, report to Congress on IRS’
                        progress in responding to our recommendations on the managerial and
                        technical weaknesses of TSM.18

                        We could not determine how much IRS has already spent on SCRIPS since its
                        inception because IRS does not have an accurate cost accounting system.
                        Using the latest life-cycle cost estimate, SCRIPS is estimated to have cost
                        about $145 million from fiscal year 1989 through fiscal year 1996.

                        In October 1994, IRS estimated that SCRIPS would provide about $17 million
                        in labor cost savings from fiscal years 1994 through fiscal year 2000. In
                        September 1995, IRS lowered that estimate to about $5 million. Also, IRS’
                        September 1996 investment evaluation report on SCRIPS concluded that the
                        system will yield a negative return on investment (i.e., costs will exceed
                        benefits) from 1991 to 2001. However, the evaluation report stated that its
                        return on investment estimate does not fully capture the operational
                        benefits of the hardware and software enhancements that were made
                        since the end of fiscal year 1995. Therefore, the report concludes that the
                        final judgment on SCRIPS’ performance cannot be made until after the 1997
                        filing season.


                        As discussed previously, SCRIPS was not used to process forms 1040PC and
Use of SCRIPS for       941 in 1995 or 1996. In July 1995, IRS decided to terminate all software
Form 941 Is Uncertain   programming for the 1040PC because of SCRIPS’ instability, a lack of system
                        capacity, and a number of software application problems. It is uncertain
                        when, if at all, SCRIPS will be used to process Forms 941.

                        The President’s fiscal year 1997 budget request included $850 million for
                        TSM, about $38 million of which was for SCRIPS. In a June 6, 1996, letter,
                        Treasury submitted a revised TSM funding request of $664 million to the
                        House Appropriations Committee, of which about $30 million was for

                        17
                          See footnote 7.
                        18
                         Report to House and Senate Appropriations Committees: Progress Report on IRS’ Management and
                        Implementation of Tax Systems Modernization, May 6, 1996.



                        Page 19                                              GAO/GGD-97-29 Performance of SCRIPS
B-270263




SCRIPS. According to the letter, at this funding level, SCRIPS would not be
used to process Forms 941 for fiscal year 1997. Congress subsequently
appropriated $336 million for TSM for fiscal year 1997. About 60 percent of
that appropriation is earmarked for operational TSM projects, such as
SCRIPS, but it was unclear when we prepared this report how much IRS
would allocate to SCRIPS. However, we would expect that IRS will continue
funding SCRIPS because the project is operational and IRS has no backup
system for FTD coupons.

In addition to funding constraints, decisions on the use of SCRIPS beyond
fiscal year 1997 for Forms 941, according to IRS officials, will hinge on:
(1) the results of the September 1996 test of SCRIPS, which we discussed
earlier; and (2) Investment Review Board actions on recommendations
made by a paper processing task team.

In March 1996, IRS convened a task team to develop a paper processing
strategy due to delays in implementing various aspects of IRS’ tax return
processing vision. Specifically, IRS had originally expected that (1) a
significant number of returns would be received electronically, thereby
reducing the need for some manual data entry; and (2) DPS would be
positioned to begin processing the remaining paper tax returns. As we
reported in October 1995, IRS’ electronic filing program is falling short of
expectations.19 Current estimates indicate that IRS may receive only
33 million electronic returns in 2001 rather than the 80 million that IRS had
set as its goal. The shortfall in meeting IRS’ 80-million goal stems from the
lack of a comprehensive business strategy to attract taxpayers to
electronic filing. IRS is currently trying to develop such a strategy. Also, IRS
announced on October 8, 1996, that it was terminating the DPS project
because of “revised priorities and budget realities for the next several
years.”

Among other things, the task team developed a paper processing strategy
based on the assumption that the maximum number of electronic returns
that can be expected in 2001 is about 33 million. In addition, the team
evaluated the need for a replacement system for IRS’ current manual data
entry system. According to IRS officials, one part of the paper processing
strategy may be to contract out the processing of some documents. These
documents could include information returns because the processing of
those documents is not as time-sensitive as the processing of income tax
returns. In addition, in the future, more FTD coupons are to be submitted
electronically as called for in the North American Free Trade Agreement.

19
  Tax Administration: Electronic Filing Falling Short of Expectations (GAO/GGD-96-12, Oct. 31, 1995).



Page 20                                                   GAO/GGD-97-29 Performance of SCRIPS
                      B-270263




                      For example, IRS estimates that about 44 percent of the FTD coupons will
                      be received electronically in fiscal year 1998, compared with only
                      1 percent in fiscal year 1996. Thus, given that some of SCRIPS’ existing
                      workload could decrease or be contracted out, IRS could possibly increase
                      the number of Forms 1040EZ that SCRIPS could process or add additional
                      forms, such as Forms 941, without requiring any additional system
                      capacity beyond what is currently available.


                      Because SCRIPS was developed before IRS started taking actions to address
Conclusions           the managerial and technical weaknesses of TSM that we identified in
                      July 1995, SCRIPS suffered from some of those weaknesses that we said
                      would contribute to such things as cost overruns and failure to meet
                      mission goals. SCRIPS is expected to cost more than originally expected
                      and, according to IRS’ September 1995 estimates, could provide less than
                      one-third of the originally expected labor cost savings. We recognize that
                      IRS is now completely reliant on SCRIPS for processing paper FTD coupons.
                      However, the decrease in expected labor cost savings and the increase in
                      estimated costs raise questions about the cost-effectiveness of SCRIPS.

                      In addition, one of the most critical weaknesses for SCRIPS was a lack of
                      thorough and complete system testing before the system was rolled out to
                      five service centers. Although IRS tested SCRIPS in September 1996 to
                      determine the maximum number of Forms 1040EZ, information returns,
                      and FTD coupons SCRIPS can process, that test does not substitute for a test
                      of SCRIPS’ ability to process any new document types along with the
                      existing ones under a production environment that replicates peak volume
                      conditions.


                      We recommend that before deciding to increase the percentage of Forms
Recommendation to     1040EZ or information returns that SCRIPS processes or using SCRIPS to
the Commissioner of   process other tax forms, such as Forms 941, the Commissioner (1) do a
Internal Revenue      cost-benefit analysis that includes examining the costs and benefits of
                      alternative ways for processing those forms, such as those developed by
                      the paper processing task team; and (2) if the analysis shows that it is
                      cost-effective to have SCRIPS process Forms 941, ensure that IRS tests
                      SCRIPS’ ability to process the existing software applications along with any
                      new software applications that may be added using peak volumes or
                      volumes simulating peak workload conditions to ensure that SCRIPS can
                      meet performance expectations.




                      Page 21                                    GAO/GGD-97-29 Performance of SCRIPS
                     B-270263




                     We requested comments on a draft of this report from the Commissioner
Agency Comments      of Internal Revenue or her designated representative. Responsible IRS
and Our Evaluation   officials, including the National Director for Submission Processing, the
                     Assistant Commissioner for Forms and Submission Processing, and the
                     SCRIPS project manager, provided IRS’ comments in a November 14, 1996,
                     meeting. Their comments on our recommendation were reiterated in a
                     November 19, 1996, memorandum from the Acting Chief of Taxpayer
                     Service. Besides commenting on our recommendation, the Deputy Chief
                     Information Officer, Systems Development, provided several factual
                     clarifications that we incorporated in the report where appropriate. The
                     officials also asked that we update our report to reflect processing rate
                     information through September 30, 1996, and they gave us the relevant
                     data. We made that update.

                     IRS officials generally agreed with our recommendation. With respect to
                     the first part of our recommendation, the officials generally agreed that IRS
                     should do a cost-benefit analysis before deciding to increase the
                     percentage of Forms 1040EZ or information returns that SCRIPS processes
                     or to add additional forms, such as Forms 941. They said that this analysis
                     would be done by the end of fiscal year 1997 and would take into account
                     approved results from the paper processing task team. IRS officials said
                     that in the event of workload imbalances during the 1997 tax return filing
                     season, if necessary, they may decide to increase the workload for SCRIPS
                     before the cost/benefit analysis is completed. We recognize that IRS’
                     priority must be to process tax returns during the filing season in a timely
                     manner, and it needs to reserve the right to do so.

                     With respect to the second part of our recommendation, IRS officials said
                     that they plan to work with the contractor to identify and implement any
                     necessary contract modifications that may be needed to ensure complete
                     testing. They said the testing will ensure that SCRIPS can meet performance
                     expectations in a peak production environment.

                     IRSofficials said that a few changes are planned for the 1997 filing season
                     that could help improve SCRIPS’ future performance. Specifically, they
                     mentioned that IRS would be testing different incentive systems for SCRIPS
                     operators to determine the extent to which incentives affect operator
                     performance, which could also affect overall SCRIPS’ performance. They
                     also mentioned that IRS is negotiating with the contractor to provide a new
                     scanner feeder that should resolve some of the problems experienced
                     when information returns are filed on extremely thin paper.




                     Page 22                                    GAO/GGD-97-29 Performance of SCRIPS
B-270263




IRS officials expressed concern about using $133 million as the baseline
cost estimate for SCRIPS. That estimate was included in a February 1992
business case and again in a revision dated April 23, 1992. IRS officials said
that we should have used an earlier cost estimate of $209 million that was
included in IRS’ Information Systems Initiative Summary Database in
August 1991. IRS’ post-implementation review report on SCRIPS also stated
that the SCRIPS project office considered the $209 million as the baseline
life-cycle cost estimate. However, the report also noted that between fiscal
years 1993 and 1996, IRS cited four other different life-cycle cost estimates
for budget purposes, including an estimate of $132 million. We revised the
report to acknowledge that $209 million was one of the cost estimates for
SCRIPS, but we believe that the business case estimate is an appropriate
baseline because the business case was a major basis for decisions to go
forward with SCRIPS. IRS officials said that IRS developed the SCRIPS business
case before it began taking steps to manage information technology
projects as investments. Since that time, for example, IRS has developed an
investment justification handbook to help ensure that project cost and
benefit analyses that are included in business cases are standardized and
complete.


We are sending copies of this report to the Subcommittee’s Ranking
Minority Member, the Chairman and Ranking Minority Member of the
House Committee on Ways and Means, the Chairman and Ranking
Minority Member of the Senate Committee on Finance, various other
congressional committees, the Secretary of the Treasury, the
Commissioner of Internal Revenue, the Director of the Office of
Management and Budget, and other interested parties.

Major contributors to this report are listed in the appendix. Please contact
me on (202) 512-9110 if you have any questions.

Sincerely yours,




Lynda D. Willis
Director, Tax Policy and
  Administration Issues




Page 23                                     GAO/GGD-97-29 Performance of SCRIPS
Appendix I

Major Contributors to This Report


                        David Attianese, Assistant Director, Tax Policy and Administration Issues
General Government      Sherrie Russ, Evaluator-in-Charge
Division, Washington,   Christopher Hess, Evaluator
D.C.                    Monika Gomez, Evaluator

                        Cecelia Ball, Senior Evaluator
Kansas City Field       Marvin McGill, Evaluator
Office




(268708)                Page 24                                   GAO/GGD-97-29 Performance of SCRIPS
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested