oversight

Tax Administration: IRS Should Reassess the Level of Resources for Testing Forms and Instructions

Published by the Government Accountability Office on 2003-04-11.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to the Chairman and Ranking
             Minority Member, Committee on
             Finance, U.S. Senate


April 2003
             TAX
             ADMINISTRATION
             IRS Should Reassess
             the Level of Resources
             for Testing Forms and
             Instructions




GAO-03-486
             a
                                               April 2003


                                               TAX ADMINISTRATION

                                               IRS Should Reassess the Level of
Highlights of GAO-03-486, a report to          Resources for Testing Forms and
Chairman and Ranking Member of the
Senate Committee on Finance                    Instructions



Taxpayers rated the Internal                   IRS used taxpayers and its employees to test revisions to five individual
Revenue Service’s (IRS) ability to             income tax forms and instructions from July 1997 through June 2002.
provide clear and easy-to-use forms            According to IRS officials, they revised about 450 tax forms and instructions
and instructions among the lowest              in 2001, many of which were for individual income tax returns.
of 27 indicators of service in 1993.
Due to continuing concerns about
unclear forms and instructions,
                                               Testing forms and instructions can help ensure their clarity and thereby
GAO was asked to determine                     benefit taxpayers and IRS by, for instance, reducing taxpayers’ time to
                                               understand and complete tax forms, reducing calls to IRS for assistance, and
•    whether and how often IRS                 reducing taxpayer errors. Due to similar benefits, federal agencies we
     tests the clarity of new and              contacted that routinely collect information from the public test their
     revised individual income tax             questionnaires. Quantifying benefits due to testing is difficult, but IRS’s
     forms and instructions;                   experience in revising and testing Earned Income Credit and Child Tax
•    the benefits, if any, of testing          Credit forms and instructions suggests that benefits of testing in some cases
     forms and instructions for                can considerably exceed the cost of testing. If taxpayers who did their own
     clarity prior to their use; and           tax returns needed 1 less minute to understand these two credits due to
•    whether any factors limit IRS’s           testing, their time saved, valued at the minimum wage, would be worth $1.2
     ability to do more tests and if
     so, how they can be addressed.
                                               million; IRS’s contracting cost for the two tests was $56,000.

                                               Although IRS officials recognized that testing could be beneficial, they cited
                                               tight time frames and constrained resources as limiting their ability to do
This report makes a series of                  more tests. While IRS faces time constraints when making some changes to
recommendations that, if fully                 forms and instructions due to the passage of new laws, not all changes are
implemented, would improve IRS’s               time constrained. IRS does not have procedures specifying which versions of
annual process for creating and                draft forms and instructions should be tested with taxpayers or when in its
revising individual income tax                 annual forms development process testing should occur. Resources
forms and instructions by helping              currently available for testing are limited but the office responsible for
to ensure that its scarce testing              testing has not developed data on missed testing opportunities and has
resources are consistently applied
                                               limited data on the benefits that have been realized when testing occurred.
to the highest priorities,
impediments to testing are                     IRS’s planning and budgeting process uses such data to support resource
lessened, and appropriate                      allocation decisions.
information is developed so IRS
management can better ensure that              Potential Benefits If Testing Helps Ensure Clarity of Tax Forms and Instructions
adequate resources are available to             IRS’s benefits                                   Taxpayers’ benefits
support testing in view of the                  •    Fewer errors needing correction             •   Reduction in time and expenses to
potential benefits to taxpayers and             •    Fewer audits due to clarity-related             prepare tax returns
IRS.                                                 taxpayer errors                             •   Reduced burden from not having to
                                                •    Less demand for assistance at local             deal with IRS notices, such as
IRS agreed with our                                  IRS offices may allow IRS to provide            reduced time to
recommendations and plans to                         better service to other taxpayers                -open and read IRS’s notices
                                                •    Less demand for IRS’s toll-free                  -decide what to do
implement all but one of them in                     telephone assistance may enable IRS
time for the 2004 forms                                                                               -research tax records
                                                     to answer some calls that currently go           -prepare response to IRS
development cycle.                                   unanswered
                                                                                                      -copy and mail response, if
www.gao.gov/cgi-bin/getrpt?GAO-03-486.          •    Knowledge gained from testing may
                                                                                                        necessary, to IRS
                                                     lead to clearer forms and instructions in
                                                     the future                                       -call IRS for assistance
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Michael          Source: GAO
Brostek at (202) 512-9110 or
brostekm@gao.gov.
Contents


Letter                                                                                        1
               Results in Brief                                                               2
               Background                                                                     4
               IRS Tested Five Tax Forms and Instructions Over 5 Years                        6
               Testing Written Documents Helps Ensure Clarity and Benefits
                 Taxpayers and IRS                                                           7
               IRS Has Not Addressed Constraints to Increased Testing                       20
               Conclusions                                                                  25
               Recommendations for Executive Action                                         26
               Agency Comments and Our Evaluation                                           26

Appendix I     Objectives, Scope, and Methodology                                           30



Appendix II    Use of Testing in NCHS, Census, and BLS                                      35



Appendix III   Comments from the Internal Revenue Service                                   37



Tables
               Table 1: Forms and Instructions IRS Tested Using Individual
                        Taxpayers Between July 1997 and June 2002                             6
               Table 2: Percentage Decrease in the Number of EIC and Child Tax
                        Credit Errors before and after Revising and Testing                   9
               Table 3: Potential Benefits if Testing Helps Ensure the Clarity of
                        Tax Forms and Instructions                                          11
               Table 4: Illustration of Potential Savings to IRS Alone in 2000 from
                        Testing EIC and Child Tax Credit Forms and Instructions             15
               Table 5: Illustration of Potential Savings to Taxpayers and IRS in
                        2000 from Error Reductions Alone from Testing EIC and
                        Child Tax Credit Forms and Instructions                             16
               Table 6: Illustration of Potential Benefit to Taxpayers in 2000 if
                        Testing Reduced the Time Needed to Understand and
                        Complete EIC and Child Tax Credit Forms                             17
               Table 7: Illustration of Potential Cost Savings to IRS in 2002 if the
                        Rate Reduction Credit Instructions Had Been Tested and
                        Errors Were Reduced                                                 18




               Page i                                 GAO-03-486 Testing Forms and Instructions
          Table 8: Illustration of Potential Savings to Taxpayers and IRS in
                   2002 from Potential Error Reductions if Testing the Rate
                   Reduction Credit Had Reduced Taxpayer Errors                                     19
          Table 9: Illustration of Potential Benefit to Taxpayers in 2002 if
                   Testing Reduced the Time Needed to Understand and
                   Complete the Rate Reduction Credit                                               19


Figures
          Figure 1: Simplified Overview of IRS’s Annual Tax Forms
                   Development Process                                                              5
          Figure 2: Simplified Overview of Where Testing Can Occur in IRS’s
                   Annual Tax Forms Development Process                                             22




          Abbreviations

          BLS               Bureau of Labor Statistics
          EIC               Earned Income Credit
          ERS               Error Resolution System
          FTE               full-time equivalent
          IRS               Internal Revenue Service
          NCHS              National Center for Health Statistics
          OMB               Office of Management and Budget
          TIGTA             Treasury Inspector General for Tax Administration
          W&I               Wage and Investment



          This is a work of the U.S. Government and is not subject to copyright protection in the
          United States. It may be reproduced and distributed in its entirety without further
          permission from GAO. It may contain copyrighted graphics, images or other materials.
          Permission from the copyright holder may be necessary should you wish to reproduce
          copyrighted materials separately from GAO’s product.




          Page ii                                       GAO-03-486 Testing Forms and Instructions
United States General Accounting Office
Washington, DC 20548




                                   April 11, 2003

                                   The Honorable Charles E. Grassley
                                   Chairman,
                                   Committee on Finance
                                   United States Senate

                                   The Honorable Max Baucus
                                   Ranking Minority Member,
                                   Committee on Finance
                                   United States Senate

                                   Internal Revenue Service (IRS) tax forms and instructions that are not
                                   clear and understandable have been a long-standing frustration of
                                   taxpayers.1 For example, respondents to a 1993 customer satisfaction
                                   survey rated IRS’s ability to provide clear and easy-to-use tax forms and
                                   instructions and to minimize taxpayer’s burden as the lowest of 27
                                   indicators of service.2 More recently, we cited the instructions for the tax
                                   year 2001 rate reduction credit as confusing and one factor contributing to
                                   millions of taxpayer errors.3

                                   To be fair, IRS faces a challenge in clearly communicating with taxpayers
                                   due to the complexity of the Internal Revenue Code. However the code’s
                                   complexity also underscores the importance of tax forms and instructions
                                   that are as clear and understandable as possible.

                                   One approach with recognized potential for aiding in the development of
                                   clearer forms and instructions is testing. By testing draft forms and
                                   instructions with taxpayers, IRS might be able to detect and revise
                                   language that is unnecessarily confusing. The Treasury Inspector General
                                   for Tax Administration (TIGTA) and we have both previously



                                   1
                                    Tax forms and instructions as referred to in this report also include related schedules and
                                   worksheets.
                                   2
                                    1993 IRS Customer Satisfaction Survey: Public Perceptions of IRS Service Quality,
                                   Contract number TIR 90-0002 (Aug. 27,1993).
                                   3
                                    U.S. General Accounting Office, Tax Administration: Advance Tax Refund Program Was
                                   a Major Accomplishment, but Not Problem Free, GAO-02-827 (Washington, D.C.: Aug. 2,
                                   2002).



                                   Page 1                                         GAO-03-486 Testing Forms and Instructions
                   recommended that IRS do more to, for instance, identify what individual
                   taxpayers find difficult to understand about tax forms or publications.4

                   Because of your interest in ensuring that taxpayers have clearer,
                   understandable tax forms and instructions and because of the potential for
                   testing to contributing to that end, you asked us to determine:

                   •   whether and how often IRS used taxpayers to test the clarity of new or
                       revised individual income tax forms and instructions;

                   •   the benefits, if any, of having taxpayers test forms and instructions for
                       clarity prior to their use by the public; and

                   •   whether any factors limit IRS’s ability to test more forms and
                       instructions for clarity and if so, how these factors can be addressed.

                   To address these objectives, we interviewed officials (1) in IRS’s Wage and
                   Investment (W&I) Division’s Tax Forms and Publications Division, (2)
                   from three federal agencies that use private citizens extensively in testing
                   research and data collection activities, and (3) a private research firm that
                   specializes in testing written documents, such as forms and surveys, for a
                   variety of public and private sector clients. We also developed illustrations
                   of the potential benefits to taxpayers and IRS of testing forms and
                   instructions for clarity. Our scope and methodology are discussed in
                   greater detail in appendix I to this report.


                   From July 1997 through June 2002, IRS used taxpayers or its employees,
Results in Brief   primarily in focus groups, to test 5 individual income tax forms or
                   instructions. According to IRS officials, they revised about 450 forms and
                   instructions in 2001, many of which were for individual income tax
                   returns.

                   Testing written documents, such as forms and instructions, prior to their
                   use helps ensure that they communicate clearly. Testing thereby helps (1)
                   lessen the burden to respondents of understanding and completing forms
                   and (2) reduce errors made by respondents. To the extent that taxpayers


                   4
                    The Process of Developing Tax Forms for Individual Taxpayers Should be Further
                   Improved, TIGTA 2000-40-060 (March 2000); U.S. General Accounting Office, Tax
                   Administration: IRS Efforts to Improve Forms and Publications, GAO/GGD-95-34
                   (Washington, D.C.: Dec. 7, 1994).




                   Page 2                                     GAO-03-486 Testing Forms and Instructions
find IRS’s tax forms and instructions that were tested easier to understand,
IRS is less likely to be contacted by taxpayers for assistance in fulfilling
their tax obligations and is likely to have fewer taxpayer errors to detect
and correct. Benefits like these may be difficult to quantify, but form the
basis for several agencies we contacted, as well as ourselves, to make
testing a standard practice when developing or modifying documents used
to collect information from the public. IRS’s recent, limited experience
with testing forms and instructions used by individuals suggests that
testing is beneficial. At least in some cases the potential benefits of testing
IRS’s forms and instructions are likely to be substantially greater than the
costs of testing to IRS.

Although IRS officials said that making greater use of testing to improve
clarity of forms and instructions could be beneficial, they cited tight time
frames for making changes and limited resources in the Tax Forms and
Publications Division as the primary factors that limit their ability to
conduct more tests. Although IRS does face time constraints when making
some changes, especially when changes are due to new tax laws, time
constraints do not apply to all changes IRS makes to forms and
instructions for individual taxpayers. Further, IRS does not have clear
procedures specifying which draft versions of forms and instructions
should be tested with taxpayers or when testing should occur during the
annual forms update process. Resources available within the division
responsible for developing forms and instructions are limited. For
example, one employee has responsibility for organizing testing efforts.
However, the division has not developed information that IRS uses when
making resource allocation decisions. The office has not developed data
on the universe of changes for which testing is most likely to yield
significant benefits to taxpayers and IRS and it has only limited data on the
benefits that have been realized when testing has occurred.

This report makes a series of recommendations to improve IRS’s annual
process for creating and revising individual income tax forms and
instructions through ensuring that (1) its scarce testing resources are
consistently applied to the highest priorities, (2) impediments to testing
are lessened, and (3) appropriate information is developed so IRS
managers can ensure that adequate resources are available for testing in
view of the potential benefits to taxpayers and IRS. IRS agreed with our
recommendations and plans to implement all but one of them in time for
the 2004 forms development cycle. The remaining recommendation—to
ensure that an appropriate range of evaluations is conducted of tests—
would take more time to put into practice.



Page 3                                 GAO-03-486 Testing Forms and Instructions
             Although most organizational components within IRS are involved, the
Background   Tax Forms and Publications Division within IRS’s Media and Publications
             Division of the W&I Division is primarily responsible for creating and
             improving tax forms, instructions, and other documents. One goal of the
             Tax Forms and Publications Division is to make tax forms and instructions
             as clear and understandable as possible. It is divided into three branches—
             Individual Forms and Publications, Business Forms and Publications, and
             Tax Exempt/Government Entities and Specialty Forms and Publications.
             As of late January 2003, 103 persons were assigned to the Tax Forms and
             Publication Division, including about 15 persons whose primary
             responsibility was creating, revising, and reviewing individual income tax
             forms and instructions.

             Many tax forms and instructions are revised annually, often with short
             turnaround times and in response to tax law changes. IRS also periodically
             reviews tax forms and, when appropriate, schedules them for revision.
             According to IRS’s estimate, it revised about 450 tax forms and
             instructions in 2001 that affected individual and business tax returns. In
             addition to tax law changes, revisions to tax forms and instructions
             generally reflected procedural changes, legal rulings, and feedback from
             internal and external stakeholders about the understandability of forms
             and instructions.

             As illustrated in figure 1, the annual tax forms development process
             generally starts with a review of the current tax forms. IRS’s tax law
             specialists review the existing forms and instructions to determine what
             changes, if any, may be needed to reflect tax law changes and other
             requirements. The tax law specialists consider comments from a variety of
             sources both within and outside IRS. For example, comments may be
             obtained from IRS customer service staff in toll-free call centers who
             answer calls from taxpayers and have firsthand knowledge of particular
             forms or instructions that were confusing to taxpayers. IRS’s Taxpayer
             Advocate Service staff may also provide comments useful to the tax law
             specialists. The Tax Forms Coordinating Committee, comprised of
             representatives from all of IRS’s key components, the Department of the
             Treasury, and IRS’s Chief Counsel, reviews draft forms to help ensure that
             they are not overly burdensome and that they conform to legal and
             technical requirements. Draft forms are generally posted to IRS’s external
             website so that external stakeholders and taxpayers may review and
             comment on them.

             The Office of Management and Budget (OMB) is responsible for approving
             each form once every 3 years. The purpose of OMB’s approval is to assess


             Page 4                               GAO-03-486 Testing Forms and Instructions
                                       IRS’s compliance with the Paperwork Reduction Act, which, among other
                                       things, requires agencies to assess the extent of burden the information
                                       they collect imposes on the public. Under the Paperwork Reduction Act,
                                       OMB must approve new forms and major revisions to existing ones. After
                                       OMB’s approval, the forms and instructions are sent to IRS’s vendors to be
                                       printed. Generally, IRS needs to have approved forms ready for printing by
                                       early October to ensure that they can be printed and distributed to the
                                       public when the tax filing season starts the following January.

Figure 1: Simplified Overview of IRS’s Annual Tax Forms Development Process




                                       Note: GAO analysis of IRS data.




                                       Page 5                                 GAO-03-486 Testing Forms and Instructions
                      As shown in table 1, IRS used taxpayers or IRS employees to test the
IRS Tested Five Tax   clarity of five individual income tax forms and instructions from July 1997
Forms and             through June 2002. 5 IRS relied primarily on focus groups to do the testing.
                      Contractors using private citizens did three of the tests (i.e., Earned
Instructions Over 5   Income instructions, Child Tax Credit instructions, and Schedule D) and
Years                 IRS did the other two using IRS employees.

                      Table 1: Forms and Instructions IRS Tested Using Individual Taxpayers Between
                      July 1997 and June 2002

                                                                Year          Number of
                          Forms and instructions                tested       participants         Testing method
                          Earned Income Credit schedule         1998                   50           5 focus groups
                          Earned Income Credit instructions     1999                  104          10 focus groups
                          Child Tax Credit instructions         1999                  104          10 focus groups
                                                                                                    6 focus groups
                          Schedule D Form-Capital Gains                                              8 one-on-one
                          and Losses                            2001                    48               interviews
                          Innocent Spouse Application form      2002                    52          6 focus groups
                      Source: IRS.

                      Note: Forms and instructions may also refer to schedules and worksheets.


                      The two testing methods that IRS used—focus groups and one-on-one
                      interviews—are among the commonly used methods to obtain data from
                      individuals on whether documents such as forms and instructions are
                      clear and understandable. Focus groups generally consist of a small
                      number of participants—about 8 to 12 persons—-and are usually selected
                      and organized around the focus group topic. Focus groups are a form of
                      group interviewing that relies on interaction within the group to obtain the
                      impressions of a group of people but not necessarily the impressions of
                      each participant. One-on-one interviews, which aim to obtain individual
                      attitudes, beliefs, and feelings, are used to probe individuals about specific
                      difficulties they may have with completing a form or reading instructions.
                      In some instances, these methods may be used in combination depending
                      upon the particular circumstances of the test. As noted in table 1, IRS used




                      5
                       Because our review was limited to forms and instructions used by individual taxpayers,
                      table 1 includes only tests IRS did on these documents during this period. IRS conducted
                      some other tests during this period. For example, IRS used focus groups to test 5
                      publications. Of the 5 publications, 3 related to individual taxpayers, 1 to business
                      taxpayers, and one related to both individual and business taxpayers. IRS estimates that it
                      will revise over 90 publications during 2003.




                      Page 6                                             GAO-03-486 Testing Forms and Instructions
                              both focus groups and one-on-one interviews in testing the Schedule D
                              form.


                              Testing written documents such as tax forms and instructions helps
Testing Written               ensure they are clear, thereby benefiting taxpayers and IRS. Researchers
Documents Helps               from three federal agencies and a private research firm said testing leads
                              to clearer documents as well as more accurate responses. Our guidance on
Ensure Clarity and            developing and using questionnaires also recommends testing prior to
Benefits Taxpayers            distribution.6 IRS’s experience indicates that testing likely improves the
                              clarity of tax forms and instructions and may therefore help reduce the
and IRS                       number of errors taxpayers make. Although limited data were available on
                              the costs and benefits of testing IRS forms and instructions, recent
                              changes to Earned Income Credit (EIC) and Child Tax Credit forms
                              illustrate the potential for benefits to significantly exceed the costs of
                              testing.


Experiences of Three          Researchers from the National Center for Health Statistics (NCHS), the
Federal Agencies, a Private   Census Bureau, and the Bureau of Labor Statistics (BLS) said that testing
Research Firm, and IRS        helps ensure that their documents are clear, and therefore users are more
                              likely to understand them and complete them correctly. Consequently,
Support Use of Testing        researchers from the agencies said they routinely test written documents,
                              such as forms or surveys, prior to public distribution. A representative of a
                              private research firm also said that testing ensures that documents
                              communicate clearly and that his firm and many others perform such
                              testing for a wide variety of private and public clients.

                              The researchers from the federal agencies told us that the benefits of
                              testing are difficult to quantify and, in some instances, may not be
                              quantifiable. NCHS officials stated that the experience they have gained
                              over time from testing and revising documents has helped them develop
                              clearer forms from the outset, a benefit that may not be quantifiable. A
                              Census researcher also agreed that testing is beneficial but difficult to
                              quantify. Nevertheless, the Census researcher said that testing documents
                              prior to public use helps ensure that they are clear and understandable,
                              which gives the agency a greater chance of receiving accurate responses
                              and which lessens the need for follow-up interviews. Similarly, a BLS



                              6
                               U.S. General Accounting Office, Developing and Using Questionnaires, GAO/PEMD-
                              10.1.7 (Washington, D.C.: October 1993).




                              Page 7                                    GAO-03-486 Testing Forms and Instructions
researcher noted that, though difficult to quantify, testing benefits the
agency by reducing errors made by respondents and those reductions can
result in savings of time, money, and effort for BLS. See appendix II for
additional information on the use of testing in these three agencies.

The representative of the private research firm also provided some
perspective of its experiences in conducting tests for public and private
sector customers. This firm, like many similar companies, provides a
variety of testing and data collection services to public and private sector
customers. The firm arranges and conducts focus groups, one-on-one
interviews, and other tests in order to ensure the clarity of forms,
instruction manuals, surveys, and Web sites among other things. Focus
groups generally involve 12 participants and cost around $6,700 to $7,600,
excluding costs to develop the item to be tested but including incentive
pay for participants that could range from $25 to $50 per participant.
These costs may be higher, the representative said, if the participants
come from special groups. For example, if the participants are medical
doctors, incentive pay could be as high as $250 per participant.

The researcher also said that private firms vary in how much they spend to
test their documents. He estimated that firms generally spend between
$300,000 and $500,000 to ensure that a form or document is clear and will
meet the firm’s needs. In some cases, firms spend more than $500,000 to
do a series of tests, making revisions between each test, before they arrive
at a final version of a form or document. He also added that most firms do
not spend at levels that would allow them to test all their forms and
documents.

While maintaining that testing is beneficial, researchers also stated that
testing is not fail-safe. It can help identify particular parts of a form that
are not clear, researchers said, but it cannot ensure that subsequent
changes to the form will entirely resolve the clarity issue. In addition,
testing may identify problems participants have in completing written
documents, but the participants’ problems may be related to other issues,
such as poor math skills, rather than confusing or unclear documents.

Testing questionnaires before distribution is also recommended as a
quality assurance measure in our guidance on developing and using
questionnaires. According to our guidance, testing questionnaires before
they are used is one of the best ways to ensure that the document actually
communicates what it was intended to communicate and that users will
uniformly interpret it. Testing increases the likelihood that respondents
will provide the information needed and helps to alleviate inaccurate


Page 8                                  GAO-03-486 Testing Forms and Instructions
responses. Our guidance is also consistent with professional literature on
survey design. According to professional literature, “reducing
measurement error through better question design is one of the least
costly ways to improve survey estimates. For any survey, it is important to
attend to careful question design and pretesting.”7

IRS’s recent limited experience with testing indicates that testing may help
ensure the clarity of tax forms and instructions. In 1999, IRS revised the
forms and instructions related to the EIC and the Child Tax Credit, tested
the revised forms and instructions, and revised them again based on the
test results. The following year, when the revised and tested forms and
instructions were used by taxpayers, the error rates for the EIC and the
Child Tax Credit decreased by 28 and 35 percent, respectively, as shown in
table 2.8

Table 2: Percentage Decrease in the Number of EIC and Child Tax Credit Errors
before and after Revising and Testing

                             Number of errors Number of errors
                               on tax returns   on tax returns Percentage decrease
                                 filed in 1999     filed in 2000     in errors from
Forms/Instructions            (before testing)   (after testing)       1999 to 2000
Earned Income Credit                 1,797,162         1,296,095                  28
Child Tax Credit                     1,430,394           934,289                  35
Source: IRS data.


IRS officials told us they attribute part of the decrease in EIC errors to a
new approach officials developed for structuring EIC forms and
instructions and part to the improvements in the draft documents that
resulted from testing the revised forms and instructions. Before the EIC
forms and instructions were revised, the instructions included a definition
and example describing a qualifying child that taxpayers had to interpret.9


7
Floyd J Fowler, 2002; “Survey Research Methods”; Third Edition; Sage Publications,
Applied Social Research Methods Series, volume 1 (2002). Also see R. Tourangeau, L.J.
Rips, and K. Rasinski, The Psychology of Survey Response (2000).
8
 As noted in table 1, IRS tested five forms during July 1997 through June 2002. IRS did not
determine the effect of testing on error rates after testing the EIC Schedule in 1998
according to IRS Research officials. IRS tested Schedule D in 2001. Any change in error
rates would be based on the 2002 filing season, and IRS has not yet analyzed 2002 figures.
IRS tested the Innocent Spouse form in 2002 and any change in error rates will not occur
until the 2003 filing season.
9
Most taxpayers eligible for the EIC have a qualifying child.




Page 9                                         GAO-03-486 Testing Forms and Instructions
                                Incorrectly claimed qualifying children have been a major source of EIC
                                errors. IRS revised the instructions so that taxpayers would answer a
                                series of “yes/no” questions to determine if they have a qualifying child
                                instead of relying on their interpretations of the definition or example of a
                                qualifying child. They then tested the old format and the new format. The
                                number of errors decreased substantially when taxpayers used the new
                                format. IRS then made some final changes to clarify the instructions based
                                on test results. In IRS officials’ opinions, the “yes/no” format made it
                                clearer for taxpayers to determine if they had a qualifying child.


Benefits of Testing Forms       The benefits of testing some changes to IRS’s forms and instructions can
and Instructions Can Far        considerably exceed IRS’s costs to do tests, especially because so many
Exceed IRS’s Costs              taxpayers can be affected by improvements in clarity that may result from
                                testing. IRS’s contract costs, including travel, for testing changes to the
                                EIC and Child Tax Credit forms and instructions were about $56,000 and
                                these costs may have been offset within IRS alone in the year that the
                                change was implemented. More significantly, if testing changes to forms
                                and instructions for these credits led to a 1-minute reduction, on average,
                                in the time taxpayers needed to understand and complete the forms during
                                the 2000 tax filing season, affected taxpayers would have saved 240,000
                                hours worth $1.2 million valued at minimum wage.

Taxpayers and IRS Can Realize   Testing has the potential to yield a wide range of benefits to taxpayers and
a Wide Range of Benefits and    IRS. Table 3 summarizes some of the potential benefits that could result if
Costs From Testing              testing helps clarify tax forms and instructions. If the form or instruction
                                that has been revised and tested remains unchanged, some potential
                                benefits could recur annually for the life of the form or instruction.




                                Page 10                                GAO-03-486 Testing Forms and Instructions
Table 3: Potential Benefits if Testing Helps Ensure the Clarity of Tax Forms and
Instructions

 IRS’s benefits                                   Taxpayers’ benefits
 • Fewer errors that need to be corrected         • Increased clarity of IRS’s forms and
   before processing returns                        instructions reduces
                                                       -time to complete tax forms
 •   Reduction in audits due to clarity-related        -expenses associated with obtaining
     taxpayer errors and redirection of audit              assistance in completing tax
     resources to other noncompliant                       forms
     taxpayers
                                                  •   Reduced burden from not having to
 •   Reduction in demand for taxpayer                 deal with IRS notices, such as reduced
     assistance at local IRS offices may allow        time to
     IRS to provide better service to other              -open and read IRS’s notices
     taxpayers                                           -decide what to do
                                                         -research tax records
 •   Reduction in demand for assistance via
     IRS’s toll-free telephone service related           -prepare response to IRS
     to clarity issues may enable IRS to                 -copy and mail response, if
     answer some calls that would otherwise                  necessary, to IRS
     have not been answered                              -call IRS for assistance

 •   Experience and knowledge gained
     through testing may lead to creation of
     clearer forms and instructions in the
     future
Source: GAO.


From the taxpayer’s perspective, benefits from testing could include
avoiding the burdens associated with (1) interacting with IRS if they make
mistakes due to unclear forms and instructions and (2) understanding and
complying with unclear forms and instructions. From IRS’s perspective,
benefits are generally in the form of opportunities to use its resources
better serving other taxpayers and enforcing the tax laws.10

Because IRS makes many changes to forms and instructions affecting
individual taxpayers every year, ranging from very simple to more
complex changes, the benefits can vary according to the type of changes
made. Some forms or instructions may change simply to update certain
dollar thresholds based on inflation and these changes may be unlikely to
be confusing or unclear to taxpayers. However, in other cases changes
may introduce new requirements or concepts to taxpayers, such as when



10
  Generally, because IRS has more demand for assistance from taxpayers than it can handle
and identifies more potentially noncompliant tax returns than it can address, benefits from
testing are unlikely to include reductions in IRS’s overall budget.




Page 11                                           GAO-03-486 Testing Forms and Instructions
                           new rules are established through legislation or regulation. Changes
                           intended to address situations like these may be more likely to be
                           confusing or unclear to taxpayers, which could result in a burden on
                           taxpayers to understand their obligations in preparing their tax forms and,
                           possibly, to errors that lead to subsequent interactions with IRS to correct
                           their returns.

                           To the extent that a form or instruction is unclear and the lack of clarity
                           leads to taxpayer errors, the method IRS uses to detect the errors can
                           affect the costs IRS incurs as a consequence. If a taxpayer’s error can be
                           detected by IRS and corrected under its “math error” procedures, which
                           rely extensively on automated processes, the cost to IRS to correct the
                           error is likely to be small.11 On the other hand, if unclear forms or
                           instructions lead to compliance errors that are detected and addressed
                           through audits conducted through the mail, in IRS offices, or in the
                           taxpayer’s location, the costs to IRS are likely to be higher in part because
                           these processes are more labor intensive.

                           The burden and costs taxpayers might avoid if testing helps clarify forms
                           and instructions, and thereby helps taxpayers avoid errors, can vary
                           substantially just as IRS’s costs can vary. In general, because taxpayers
                           need only respond if they disagree with an IRS notice stating that it has
                           corrected an error under its math error procedures, the taxpayer’s burden
                           and cost are likely to be lower than if IRS contacts the taxpayer as part of
                           an audit since audits require taxpayer responses and reviews of taxpayers’
                           books and records. 12

Testing Benefits Can Far   Illustrations we developed of the potential benefits and costs of testing
Exceed IRS’s Costs         forms and instructions show that at least in some cases benefits can be
                           substantially greater than the costs to IRS to do tests. The benefits of
                           testing to IRS alone can potentially exceed its testing costs in the first year
                           a change is implemented. But, primarily because a small change in the
                           time required of taxpayers to understand their tax obligations can total to


                           11
                            In the math error program, IRS uses computer comparisons and calculations to correct
                           clerical and mathematical errors on tax returns. These corrections involve addition and
                           subtraction errors; incorrect social security numbers, filing status and exemptions; and
                           missing schedules or forms.
                           12
                            Although in many cases taxpayers would not need to respond to a math error notice, in
                           some cases, such as when IRS has recalculated EIC or other benefits due to its detection of
                           an apparently invalid social security number, the taxpayer may be able to provide IRS
                           additional information to substantiate the original return.




                           Page 12                                       GAO-03-486 Testing Forms and Instructions
a large aggregate benefit, taking taxpayers’ benefits into account can yield
total benefits substantially above IRS’s costs.

IRS officials have not attempted to develop quantitative estimates of the
benefits to taxpayers and IRS that may result from testing forms and
instructions and the costs IRS incurs to achieve those benefits. IRS
officials did believe that because taxpayers made fewer errors when using
the revised EIC and the Child Tax Credit forms and instructions as shown
in table 2, IRS spent less time and money correcting errors related to
them.13 The officials said they could not quantify the cost savings because
IRS does not track error correction costs by type of error.

To provide some perspective on the potential magnitude of benefits and
costs that may be realized due to testing changes to forms and
instructions, we analyzed the changes IRS made to EIC and Child Tax
Credit forms and instructions. Our analyses are illustrations and not actual
assessments of benefits and costs that were associated with testing these
forms and instructions because complete data were not available on the
potential benefits and costs. Further, in constructing our illustrations we
sought to be conservative in estimating benefits, in part because we did
not have information on the full range of costs IRS incurred to test forms
and instructions. Our illustrations focus on (1) a narrow set of benefits to
IRS alone due to potential reductions in taxpayer errors, (2) those benefits
plus certain benefits to the taxpayers from reduced errors, and (3)
potential benefits to taxpayers in reduced time to do their taxes. See
appendix I for details on the methodology we used in developing our
illustrations.

Our first illustration quantifies a narrow set of benefits to IRS alone from
testing EIC and Child Tax Credit forms and instructions—that is, the
benefits IRS may have realized due to reduced numbers of errors that are
handled under its math error procedures. It is likely that to the extent
testing contributed to better taxpayer understanding of these two credits,


13
  Error correction costs relate to the error resolution process. When a taxpayer makes an
error, in some cases, IRS will correct the error. For example, if a taxpayer transposes
information from the Schedule D to an incorrect line on the form 1040, IRS can correct this
mistake for the taxpayer. IRS cannot correct certain errors made by taxpayers such as
claiming EIC for a child whose social security number on the tax return does not match
Social Security Administration records. In this case, because it cannot determine the
correct social security number for the child, IRS will disallow the claim and send an error
notice to the taxpayer that explains what IRS did and what the taxpayer can do if he/she
disagrees with IRS’s actions.




Page 13                                       GAO-03-486 Testing Forms and Instructions
IRS would have obtained other benefits. For instance, because improperly
claimed qualifying children is one of the leading causes of the EIC’s high
noncompliance rate,14 if clarified EIC forms and instructions lead fewer
taxpayers to improperly claim the EIC, IRS would likely be able to free
some of its EIC-related audit resources for other audits or to audit EIC
returns that it might otherwise have had insufficient resources to cover. In
fiscal year 2002, IRS used about 1,400 full-time equivalent (FTE) staff
years15 for correspondence audits of EIC issues.

Our analysis in table 4 shows the amount by which IRS’s potential cost
savings from not having to correct EIC and Child Tax Credit errors may
have exceeded its testing costs given differing assumptions about how
much testing alone may have contributed to reduced taxpayer errors. As
illustrated, IRS would have saved more in cost avoidance (thereby freeing
resources to work elsewhere) in the first year of the change alone than it
spent on the contracted testing of the forms and instructions if half of the
reduction in errors was due to testing.16 If only 10 or 25 percent of the
reduction was due to testing, then IRS would not have saved more than it
spent on the testing contract in the first year. However, some of the
benefits of a change in forms or instructions continue to be realized in
future years. Again, the illustration does not consider other benefits IRS
may have realized.




14
 Department of the Treasury, Internal Revenue Service: Compliance Estimates for Earned
Income Tax Credit Claimed on 1999 Returns (Washington, D.C.: Feb. 28, 2002).
15
 An FTE consists of one or more employees who collectively work 1 year. For example,
one full-time employee or two half-time employees equal one FTE.
16
 Because IRS officials attributed reduced errors both to their new approach to structuring
EIC guidance and to improvements to the new approach that resulted from testing, we
chose to illustrate reductions due to testing alone that accounted for half or less of the
decline in errors.




Page 14                                       GAO-03-486 Testing Forms and Instructions
Table 4: Illustration of Potential Savings to IRS Alone in 2000 from Testing EIC and Child Tax Credit Forms and Instructions

                                                   Assumed number                                                       Potential IRS cost
                       Differing assumed                    of errors        Estimated direct        Estimated cost to savings assuming
 Form and                  levels of errors        eliminated due to             labor cost to       correct eliminated   $56,000 cost to
                                                                                              a                                           b
 instruction                       avoided                    testing        correct an error                    errors     perform test
 EIC                                   50%                   250,534                    $0.36                   $90,000            $34,000
                                       25%                   125,267                     0.36                    45,000            (11,000)
                                       10%                     50,107                    0.36                    18,000            (38,000)
 Child tax credit                      50%                   248,053                     0.36                    89,000              33,000
                                       25%                   124,026                     0.36                    45,000            (11,000)
                                       10%                     49,611                    0.36                    18,000            (38,000)
Source: IRS and GAO.

                                              Note: GAO analysis of IRS data.
                                              a
                                               IRS officials provided (1) the number of FTEs, including labor-related overhead costs such as
                                              training and leave, for operating the Error Resolution System (ERS) and (2) the number of errors
                                              corrected by ERS during fiscal year 2002. According to IRS officials, ERS personnel are hired at GS-
                                              4 and the journeyman level is GS-6. Using the number of ERS FTEs, the number of errors corrected
                                              by ERS staff, and the GS-5, step 1 salary effective January 2002, as the salary for ERS staff
                                              members, we estimated that IRS incurred $0.36 in costs per error. (IRS officials were unable to
                                              provide the actual costs for ERS staff. We chose to use GS-5, step 1, because beginning ERS staff
                                              are GS-4s and more experienced ERS staff are GS-6s.) Total error correction costs may be higher
                                              because the $0.36 estimate does not include various other costs such as nonlabor-related overhead
                                              for equipment and supplies.
                                              b
                                               IRS contracted with a firm to conduct focus groups to test EIC and Child Tax Credit forms and
                                              instructions. The contract was for $54,000 and IRS spent an additional $2,000 on related travel
                                              costs. Because one contractor conducted the tests for both credits, IRS officials said it would be
                                              difficult to allocate the $56,000 between the tests for the two credits.


                                              To provide some perspective on how the potential benefits to taxpayers
                                              from testing EIC and Child Tax Credit forms could affect the overall
                                              benefits and costs of testing, we next looked at potential reduced burden
                                              from credit claimants receiving fewer notices due to reduced errors. First,
                                              we assumed that on average all taxpayers receiving an error notice from
                                              IRS take 2 or 5 minutes to deal with the notice. Based on those
                                              assumptions, we calculated the value to taxpayers of the time saved (using
                                              minimum wage levels) from not having to deal with IRS error notices.17 We
                                              used the same assumed reductions in errors due to testing that we made




                                              17
                                                We used minimum wage to value taxpayers’ time to be conservative in estimating
                                              potential benefits because EIC is targeted to low-income taxpayers. Minimum wage is a
                                              conservative estimate of taxpayers’ time even for EIC in that some taxpayers can qualify
                                              for EIC at incomes that are three times minimum wage. For purposes of the Paperwork
                                              Reduction Act, OMB values time for dealing with paperwork requirements like tax forms at
                                              $30 per hour.




                                              Page 15                                             GAO-03-486 Testing Forms and Instructions
                                           for table 4 and we netted taxpayers’ savings with the savings shown in
                                           table 4 for IRS alone.

                                           As table 5 shows, including testing-related benefits to taxpayers from
                                           decreased errors suggests that in the first year following testing of EIC and
                                           Child Tax Credit forms and instructions, the net benefit to taxpayers and
                                           IRS combined could have been positive except for our lowest assumption
                                           about the degree to which testing may have reduced taxpayer errors—our
                                           10 percent assumption.

Table 5: Illustration of Potential Savings to Taxpayers and IRS in 2000 from Error Reductions Alone from Testing EIC and
Child Tax Credit Forms and Instructions

                                                                                                          Potential net savings to
                                                                                                          taxpayers and IRS from
                                                                                                                 error reductions
                    Assumed number                Value to                Value to Potential IRS cost
                             of errors   taxpayers in time       taxpayers in time savings assuming
 Form and           eliminated due to    saved (2 minutes        saved (5 minutes     $56,000 cost to
 instruction                   testing         per notice)             per notice)      perform test      2 minutes     5 minutes
 EIC                          250,534             $43,000                $108,000             $34,000       $77,000      $142,000
                              125,267               22,000                  54,000            (11,000)        11,000        43,000
                                50,107               9,000                  22,000             (38,000      (29,000)      (16,000)
 Child tax credit             248,053               43,000                106,000               33,000        76,000      139,000
                              124,026               21,000                  53,000             (11,000        10,000        42,000
                                49,611               9,000                  21,000            (38,000)      (29,000)      (17,000)
Source: GAO.

                                           Note: GAO analysis.


                                           Finally, to illustrate the potential benefits if testing EIC and Child Tax
                                           Credit forms and instructions made them clearer and thereby reduced
                                           taxpayers’ time needed to understand and complete the credit forms, we
                                           calculated the value of time saved by taxpayers (using minimum wage
                                           levels) in understanding and completing EIC and Child Tax Credit forms
                                           assuming the time saved was 1 minute. Unlike for tables 5 and 6, all
                                           taxpayers who used the form or instructions to determine whether they
                                           qualified for either credit may have saved time if testing contributed to
                                           clearer EIC and Child Tax Credit forms and instructions. However,
                                           because we did not know how many taxpayers might have used the forms
                                           and instructions for this purpose, in calculating the value of time
                                           taxpayers may have saved we used only the number of taxpayers who
                                           claimed these credits and did not use paid preparers to prepare their tax
                                           returns.




                                           Page 16                                       GAO-03-486 Testing Forms and Instructions
Table 6 shows that if testing the credits’ forms and instructions helped
clarify them and that led taxpayers to take 1 minute less to understand and
complete the forms, credit claimants would have saved a total of about
240,000 hours worth $1.2 million at minimum wage levels.

Table 6: Illustration of Potential Benefit to Taxpayers in 2000 if Testing Reduced the
Time Needed to Understand and Complete EIC and Child Tax Credit Forms

                      Number of returns            Total hours saved          Potential value to
    Form and          claiming the credit     assuming 1 minute less          taxpayers of time
    Instruction               (millions)a                per taxpayer           saved (millions)
    EIC                               6.2                     100,000                       $0.5
    Child Tax
    Credit                             8.4                       140,000                       0.7
    Totals                            14.6                       240,000                       1.2
Source: GAO.

Note: GAO analysis.
a
 This is the number of tax returns prepared by taxpayers claiming the credits. Since taxpayers who
used a preparer might not need to read and understand the forms and instructions, we did not include
them in this table. Both the number of EIC and Child Tax Credit returns are based on the percentage
of returns prepared by taxpayers who claimed EIC since data on the percentage of returns prepared
by taxpayers who claimed the Child Tax Credit were not available.


To provide another perspective on the potential magnitude of benefits and
costs associated with testing changes to forms and instructions, we also
looked at IRS’s experience with the rate reduction credit. This one-time
credit was enacted in June 2001.18 When 2001 tax returns were processed
during 2002, over 8 million returns had errors related to the credit.19 IRS
did not test the instructions for computing the rate reduction credit that
was included on the Form 1040 for tax year 2001. According to IRS
officials, they did not test the instructions because the credit was a one-
time event, and in their judgment, they had insufficient time to test it. We
reported that some of the taxpayers’ errors were probably due to



18
  Taxpayers who were eligible to receive an advance tax refund in 2001 but who (1) did not
receive a check because IRS did not have their current addresses or (2) did not have
enough taxable income in 2000 to qualify for the maximum amount allowable, may have
been entitled to a rate reduction credit when filing their tax year 2001 returns. In addition,
taxpayers who were not eligible for an advance tax refund, such as those who did not have
taxable income in 2000, may have been entitled to a rate reduction credit provided they had
taxable income in 2001.
19
 This total only includes rate reduction credit errors corrected by IRS under its math error
authority as of September 27, 2002. Some electronically filed tax returns that IRS rejected
during screening for processing also had rate reduction credit errors.




Page 17                                           GAO-03-486 Testing Forms and Instructions
                                              taxpayers not understanding IRS’s instructions on how to compute the
                                              credit. We also reported that the demand for telephone assistance related
                                              to the credit was significant during the 2002 filing season, and that some of
                                              these calls, based on Taxpayer Advocate Service information, were made
                                              because taxpayers did not understand how to compute the credit.20

                                              Using the same approach to illustrate whether IRS alone may have realized
                                              benefits in excess of its testing costs as we did for EIC and Child Tax
                                              Credit changes, we developed the illustration shown in table 7.21 As shown,
                                              considering only IRS’s cost and assuming that all errors were corrected by
                                              IRS using its math error procedures and assuming IRS would have spent
                                              the same amount to test the rate reduction credit instructions as it did for
                                              EIC and Child Tax Credit tests, IRS may have been able to save between
                                              $233,000 and $666,000. Although this case is somewhat atypical since the
                                              rate reduction credit affected essentially all individual taxpayers and the
                                              number of errors related to the credit was unusually high, these figures
                                              illustrate that the potential for savings to IRS alone from testing
                                              instructions at times can substantially exceed its testing costs.

Table 7: Illustration of Potential Cost Savings to IRS in 2002 if the Rate Reduction Credit Instructions Had Been Tested and
Errors Were Reduced

                                                   Assumed number                                                    Potential IRS cost
 Number of returns     Differing assumed                   of errors      Estimated direct     Estimated cost to     savings assuming
 with rate reduction       levels of errors           eliminated by           labor cost to    correct eliminated       $56,000 cost to
 credit errors                     avoided                   testing      correct an error                 errors         perform test
 8,025,851                             25%                2,006,646                  $0.36               $722,000             $666,000
                                       15%                1,203,878                   0.36                433,000               377,000
                                       10%                  802,585                   0.36                289,000               233,000
Source: IRS and GAO.

                                              Note: GAO analysis of IRS data.


                                              However, just as with EIC and Child Tax Credits, taxpayers would have
                                              benefited if testing had been done, led to clearer instructions and
                                              consequently led to fewer taxpayer errors for the rate reduction credit.
                                              Using the same approach we used for EIC and Child Tax Credits, table 8



                                              20
                                               GAO-02-827.
                                              21
                                                Consistent with our effort to be conservative in estimating testing benefits, we assumed
                                              that testing would have reduced errors by no more than 25 percent because many
                                              taxpayers only needed to review a sentence or two of the instructions and did not need to
                                              do any calculations.




                                              Page 18                                         GAO-03-486 Testing Forms and Instructions
                                          shows the potential taxpayers’ savings from dealing with fewer rate
                                          reduction credit error notices and the net savings to taxpayers and IRS.

Table 8: Illustration of Potential Savings to Taxpayers and IRS in 2002 from Potential Error Reductions if Testing the Rate
Reduction Credit Had Reduced Taxpayer Errors

                                                                                                    Potential net savings to taxpayers
                                                                                                      and IRS from error reductions
                          Assumed            Value to              Value to        Potential IRS
 Number of               number of       taxpayers of          taxpayers of        cost savings
 returns with                 errors    time saved (2         time saved (5           assuming
 rate reduction       eliminated by       minutes per           minutes per      $56,000 cost to
 credit errors               testing           notice)               notice)        perform test           2 minutes             5 minutes
 8,025,851                2,006,646         $344, 000              $861,000            $666,000           $1,010,000            $1,527,000
                          1,203,878           207,000               517,000              377,000             584,000               894,000
                            802,585           138,000               344,000              233,000             371,000               577,000
Source: GAO.

                                          Note: GAO analysis.


                                          Table 9 shows that if testing had been done and it improved the clarity of
                                          the instructions enough to save taxpayers, on average, 30 seconds in
                                          understanding whether and how they needed to complete the credit line
                                          on their tax returns, the savings would have been larger than savings to
                                          IRS and taxpayers from avoided errors alone.22

                                          Table 9: Illustration of Potential Benefit to Taxpayers in 2002 if Testing Reduced the
                                          Time Needed to Understand and Complete the Rate Reduction Credit

                                                                                      Total hours saved               Potential value to
                                              Number of returns                assuming 30 seconds less         taxpayers of time saved
                                                                    a
                                              affected by the credit                        per taxpayer                       (millions)
                                              50,000,000                                         415,000                             $2.1
                                          Source: GAO.

                                          Note: GAO analysis.
                                          a
                                           This is the number of paper returns prepared by taxpayers. Since taxpayers who used a preparer
                                          might not need to read and understand the instructions, we did include them in this table. We only
                                          included the number of paper returns because information was not readily available on the number of
                                          returns filed electronically that were completed by taxpayers rather than a preparer.




                                          22
                                            We assumed an average taxpayer time savings of 30 seconds because many taxpayers
                                          only needed to review a sentence or two of the instructions and did not need to do any
                                          calculations.




                                          Page 19                                           GAO-03-486 Testing Forms and Instructions
                           Although IRS officials said that making greater use of testing to improve
IRS Has Not                clarity of forms and instructions could be beneficial, officials have not
Addressed                  addressed the two constraints—time and resources—that they state limit
                           their ability to do more testing of changes to forms and instructions. Time
Constraints to             constraints are not binding for some changes IRS considers to forms and
Increased Testing          instructions, although IRS cannot realistically test the unknown portion of
                           the changes that are due to laws passed shortly before, or even after, the
                           effective dates for the forms. Also, IRS’s procedures for developing and
                           revising forms (1) do not clearly specify which draft version of forms and
                           instructions should be tested with taxpayers or (2) when in the annual
                           forms development cycle testing should occur. In addition to tight time
                           frames, officials also say that limited resources, such as only one person
                           responsible for coordinating all testing efforts in the Forms and
                           Publications Division, preclude them from increasing tests of forms and
                           instructions. However, IRS has not documented which changes to forms
                           and instructions likely would benefit from testing or demonstrated the
                           benefits that are gained when testing is done. IRS’s planning and budgeting
                           process uses such information in determining the level of resources to be
                           allocated to various units.


Tight Time Frames Do Not   IRS officials told us that when new tax laws are enacted during the year
Always Exist and IRS’s     that require IRS to create or revise tax forms and instructions in time to
Procedures Do Not Always   distribute them to taxpayers by January 1, the start of the tax-filing season,
                           they lack time to test the forms and instructions before distributing them
Facilitate Testing         to taxpayers. However, not all changes to forms and instructions are time
                           constrained and IRS’s procedures lack a clear target for which version of
                           forms and instructions should be tested with taxpayers.

                           While sufficient data were not available to determine the portion of
                           changes IRS makes to forms and instructions that cannot be tested due to
                           time constraints, not all changes are time constrained. Due to the
                           variability in the time that may be required to test a form or instruction
                           and in the amount of time IRS needs to develop the initial form or
                           instruction to be tested, we cannot say definitively when IRS may or may
                           not have sufficient time to conduct tests. In some cases, IRS likely could
                           have sufficient time to do testing when it identifies a needed change to
                           forms or instructions itself since it largely controls the scheduling of this
                           work. Similarly, when the Congress passes a law that is not effective until
                           a future tax year, or that contains provisions that are not effective until a
                           future tax year, IRS may have sufficient time to conduct tests. For
                           example, the Economic Growth and Tax Reconciliation Act of 2001 was
                           passed on June 7, 2001, with some provisions effective for tax year 2001,


                           Page 20                                GAO-03-486 Testing Forms and Instructions
but others with later effective dates. The provisions modifying education
Individual Retirement Accounts were effective for taxable years beginning
after December 31, 2001. This gave IRS approximately 16 months to
develop and test any modifications to tax forms and instructions and make
final revisions before those forms and instructions needed to go to printing
for distribution by January 2003. When a law affects the current tax year,
i.e., changes how taxpayers will need to calculate their taxes in the next
tax-filing season, IRS is less likely to have sufficient time to test. Even in
such a case, however, the new law may be passed early enough to allow
testing.

IRS’s current procedures for developing and revising forms and
instructions do not clearly specify which draft version of forms and
instructions should be tested with taxpayers or when in the annual forms
development cycle testing should occur. Officials said that draft forms
may be tested with taxpayers either before or after they are posted to
IRS’s website for external comments by the public, tax practitioners,
software developers, and others. Tax Forms and Publications Division
officials said that they consider the particular circumstances surrounding
the development of each form and instruction when deciding which
version of a draft form or instruction they should test. However, because
IRS does not have a clear targeted time for testing, IRS’s ability to plan and
conduct tests maybe constrained. If IRS’s procedures defined a point in
the annual forms development cycle where a version of a draft form or
instruction would be available for testing, IRS would be able to establish
processes and deadlines designed to ensure that the opportunity for
testing is realized.

To the extent that a draft version of a form or instruction is available for
testing early in the process, it would give IRS a fuller range of options for
testing. For example, if IRS tested draft versions of forms and instructions
before or during the approximately 3-week period that the form is
available on its Web site, this would minimize any additional calendar time
that testing might otherwise add to IRS’s forms development process.

Figure 2 shows the points in IRS’s annual forms development process
where testing can occur. As illustrated, testing may be conducted early in
the process and late in the process.




Page 21                                GAO-03-486 Testing Forms and Instructions
Figure 2: Simplified Overview of Where Testing Can Occur in IRS’s Annual Tax Forms Development Process




                                       Note: GAO analysis of IRS data.


                                       Testing earlier drafts of forms and instructions would also enable officials
                                       to select from various testing alternatives depending on how early a draft
                                       is available for testing. We did not find a uniform amount of time needed
                                       to test a change to a form or instruction. At the low end of the spectrum,
                                       an official from NCHS said that it takes about 7 weeks to test that agency’s
                                       questionnaires using one-on-one interviews. IRS officials estimated that
                                       when IRS employees are used as focus group participants it requires about
                                       8 to 12 weeks to schedule and conduct the tests, analyze the data, and
                                       prepare a report summarizing the results. IRS officials estimated that
                                       when they contract with a private firm to conduct focus groups using
                                       private citizens, 24 to 32 weeks are required to obtain a contract, recruit
                                       participants, conduct the tests, analyze the results, and prepare a report.
                                       This time frame is based on using regular contracting processes involving
                                       developing a statement of work, soliciting bids, and selecting a contractor.
                                       Contract options exist that enable agencies to identify a firm or group of
                                       firms qualified to undertake work so that an expedited task order
                                       procedure can be used to select a firm for when needs arise. According to


                                       Page 22                                 GAO-03-486 Testing Forms and Instructions
                            IRS officials, they recently entered into a multiyear contract with two
                            vendors that will enable them to issue task orders when work is needed.

Data Are Not Collected      Although IRS’s Tax Forms and Publications Division officials believe
That Could Be Used by IRS   current resources are insufficient to support more testing of forms and
Management to Determine     instructions, they do not have some of the information needed to
                            determine whether to allocate additional resources. This information is
the Proper Allocation of    not available at least in part because division guidelines and policies do
Resources to Support        not require that it be gathered.
Testing
                            Officials said that because they have so few staff available to conduct tests
                            and have a limited budget to contract for testing, they could not increase
                            the number of tests they perform. According to the officials, currently only
                            1 of 103 persons in the division is trained in testing methods. In addition to
                            other duties, this person coordinates the tests for the division, such as the
                            test of EIC forms and instructions completed in 1999 by a private vendor
                            and the test of the innocent spouse application form completed by IRS in
                            2002. Officials also told us some staff who are primarily responsible for
                            creating and revising tax documents may occasionally assist in conducting
                            tests, such as the three persons involved in testing the innocent spouse
                            form. Officials also said the total budget for contract support for the
                            division was $150,000 in fiscal year 2002, $185,000 in fiscal year 2001, and
                            $130,000 in fiscal year 2000.

                            As part of its annual planning and budgeting process, IRS management
                            determines what resources will be needed to accomplish strategies and
                            implement programs. IRS’s planning and budget guidance requires that
                            each operating unit prepare a business plan that, among other things,
                            clearly defines priorities and resource requirements. Requests in the
                            business plan for resources must be substantiated with evidence that
                            allocating additional resources is justified.

                            However, the division does not systematically identify when testing would
                            be beneficial and does not routinely demonstrate the benefits to taxpayers
                            and IRS that have been gained from such testing. Officials do not identify
                            which of the many changes it makes to forms and instructions each year
                            would most likely benefit from testing. Thus, the officials cannot tell IRS
                            management how many opportunities to improve forms and instructions
                            may be lost due to current resource levels. Further, when tests are
                            performed, officials do not identify, quantitatively or qualitatively, the
                            benefits that taxpayers and IRS may have realized.




                            Page 23                                GAO-03-486 Testing Forms and Instructions
One reason that IRS does not have data on forgone testing opportunities is
that the division lacks formal, written guidelines and procedures for
determining when testing would be beneficial. Currently, testing is an
optional step in the process for developing forms. IRS’s Tax Forms and
Publications officials said that they decide which forms to test based on
informal guidelines and procedures and input from officials in IRS’s four
operating division program offices and the Taxpayer Advocate Service.
The informal guidelines and procedures call for officials to weigh, among
other things, whether a form or instruction (a) affects a large number of
taxpayers, (b) has a high error rate based on taxpayers’ prior use of the
form, (c) is perceived as complex, and (d) will be used for several filing
seasons. Also, according to IRS, the amount of time available to perform
tests is factored into testing decisions.

These informal guidelines do not require officials to consider in all cases
whether testing would be beneficial and to document the decisions made.
Accordingly, even if the informal guidelines are applied, and officials judge
that some forms or instructions could benefit from testing but cannot be
tested due to scarce resources, those decisions are not made
systematically and documented.

Further, although the factors the guidelines suggest taking into account
appear to have evolved from officials’ experience and therefore should be
useful, they do not consider some pertinent factors that could affect the
benefits likely to be realized from testing. For instance, the guidelines
suggest taking the number of affected taxpayers into account but not the
likely amount of burden they would face due to unclear forms or
instructions. They also do not clearly call for officials to consider the costs
to test forms and instructions and the benefits that may accrue throughout
IRS, such as in telephone service centers. In addition, these informal
guidelines and procedures automatically exclude testing forms and
instructions that will be used only one time. Also, according to IRS
officials, the time frame between the passage of new tax laws and when
the newly created or revised forms and instructions must be finalized may
preclude some forms and instructions from being tested. Even if one-time-
use forms meet other testing criteria, such as affecting a large number of
taxpayers who may perceive them as complex, IRS will not consider
testing them. As the rate reduction credit situation discussed earlier
illustrates, such automatic exclusions may not be appropriate in all
situations.

IRS officials do not have information on the results achieved when forms
and instructions are tested in part because the division does not have


Page 24                                 GAO-03-486 Testing Forms and Instructions
              policies that require such evaluations. When IRS obtained information on
              the reduction in error rates following testing of EIC and Child Tax Credit
              forms and instructions, the studies did not include collecting other
              information on the benefits that may have resulted for taxpayers and for
              IRS. For instance, the studies did not estimate the savings IRS may have
              realized in its telephone and walk-in service due to increased form clarity.

              Capturing fuller information on the results of testing would be consistent
              with IRS’s strategic planning and budgeting process, which emphasizes
              assessing the impact of current programs to efficiently allocate resources.
              Further, by evaluating results of testing decisions, IRS officials would be
              able to determine if their testing guidelines and procedures lead to good
              decisions about when testing is most likely to be beneficial. They may also
              be able to see if the methods they use to test—for example, focus groups
              formed by IRS employees or one-on-one interviews with individuals—yield
              the most effective test results.


              IRS continually faces the daunting task of developing and revising tax
Conclusions   forms and instructions to administer our ever-changing set of federal tax
              laws. Taxpayers rely on IRS for forms and instructions that are as clear
              and easy to understand as possible given the complexity of the tax laws
              and providing clear materials is a key goal of IRS’s Tax Forms and
              Publications Division.

              In attempting to meet this goal, IRS has tested an average of one set of
              forms and instructions each year over the last 5 years. In contrast, officials
              from three federal agencies that routinely collect information from the
              public say that testing documents for clarity before using them is their
              standard practice. They do so because they believe testing will ensure that
              their data collection documents are clear and that individuals will
              understand them and complete them accurately.

              Although it is difficult to gauge how much testing alone contributes to the
              clarity of tax forms and instructions, IRS officials believe testing has
              contributed to significant declines in taxpayer errors. Illustrations we
              developed based on IRS’s experience in testing forms and instructions
              suggest that IRS can completely recover its testing costs in the first year
              following testing in some circumstances and that when savings to
              taxpayers from more understandable forms and instructions are
              considered, total benefits even in the first year following tests can be
              several times IRS’s testing costs.



              Page 25                                GAO-03-486 Testing Forms and Instructions
                      Although they recognize that testing is beneficial, officials say time
                      constraints and limited resources preclude more testing. However, IRS’s
                      procedures do not clearly specify when draft versions of forms and
                      instructions should be available for testing. Having a clearly defined point
                      where testing would be performed would facilitate establishing
                      procedures and deadlines to better ensure that testing could be done even
                      within IRS’s annual forms update cycle. Further, IRS officials do not have
                      information that would help IRS management to determine whether to
                      allocate additional resources to support enhanced testing. Because IRS
                      lacks standard written procedures for testing, officials have not
                      documented cases where testing would likely be beneficial and have not
                      demonstrated the benefits that are gained from testing.


                      Because testing could potentially yield clearer and more understandable
Recommendations for   tax forms and instructions, thereby producing benefits both to taxpayers
Executive Action      and IRS, we recommend that the Acting Commissioner of Internal
                      Revenue take the following actions.

                      •   Develop written criteria for determining which changes to tax forms
                          and instructions should be tested with taxpayers before publication.

                      •   Develop official written guidance that incorporates those criteria and
                          ensure that the guidance requires staff that develop new or revised
                          forms and instructions to document which changes would merit testing
                          and why.

                      •   Clarify procedures by designating when in the annual forms
                          development process that a draft version of forms and instructions
                          should be available for testing with taxpayers.

                      •   Ensure that an appropriate range of evaluations are conducted of tests
                          that are performed to better establish the costs and benefits of
                          performing tests and to refine IRS’s approach to testing on the basis of
                          lessons learned.

                      •   Use information gained from documenting when changes to forms or
                          instructions likely would be beneficial and from evaluations of tests to
                          reassess an appropriate level of resources to perform testing.

                      The Acting Commissioner of Internal Revenue provided written comments
Agency Comments       on a draft of this report in an April 7, 2003, letter, which is reprinted in
and Our Evaluation    appendix III. The Acting Commissioner agreed with our recommendations.



                      Page 26                                GAO-03-486 Testing Forms and Instructions
We are encouraged that IRS plans to implement all but one of our
recommendations in time for the 2004 forms development cycle.
Understandably, the remaining recommendation to ensure that an
appropriate range of evaluations is conducted of tests would take more
time to put into practice. The Acting Commissioner also provided
additional comments and observations on our draft report.

The Acting Commissioner commented that the crux of our report is that
we do not believe IRS has performed adequate testing on new and revised
tax forms and instructions due to a lack of resources. He said that
resources for testing forms and instructions have been adequate for the
testing IRS wanted to perform. While not questioning whether resources
were adequate for the testing IRS performed, we concluded that IRS
officials do not have information needed to determine the level of
resources that should be allocated to testing forms and instructions.
Accordingly, we recommended that IRS systematically identify
opportunities to improve forms and instructions through testing and to
evaluate the costs and benefits when testing is done.

Although agreeing that testing is beneficial, the Acting Commissioner also
said that there are significant staff costs associated with testing that are
not included in our cost analysis. We recognize that our analysis excluded
staff costs and as stated in our draft report we sought to be conservative in
estimating the benefits of testing, in part because we did not have
information on the full range of costs IRS incurs when undertaking
projects to tests forms and instructions. During the course of our work, we
requested estimates of staff costs for testing but IRS was unable to provide
them. Nevertheless, at least in the cases we illustrated the potential
benefits of testing were so much greater than the costs that including staff
costs likely would not have substantially changed the results of our
illustrations.

The Acting Commissioner expressed concern about whether IRS could
have forms and instructions ready for the filing season if testing was done
late in the forms development cycle as shown in our figure 2 depicting
IRS’s process. We agree with the Acting Commissioner’s concern;
however, our figure shows the various points at which testing can occur in
IRS’s current processes based on interviews with IRS officials and the
documentation they provided us. As IRS implements our recommendation
to clarify when testing should be done, selecting a point as early as
possible would help maximize the number of changes that can be tested
during the annual forms update cycle.



Page 27                                GAO-03-486 Testing Forms and Instructions
The Acting Commissioner also said that he disagreed with our conclusion
that IRS’s experience with obtaining feedback on its products is limited or
recent. He said IRS uses various methods to obtain customer feedback. We
agree that IRS uses methods other than testing to obtain feedback on its
forms and instructions. However, our report describes the potential
benefits and costs of testing as a feedback method. In terms of testing, IRS
has only tested five forms and instructions during July 1997 through June
2002; in our view, this is a limited number of tests that were conducted
during the recent past.

The Acting Commissioner also disagreed that testing would result in
reduced demand for walk-in and toll-free assistance. He said that IRS lacks
data to support such a conclusion and, based on its experience, new forms
generate requests for assistance and error rates on them tend to be higher.
We recognize that there will always be a demand for taxpayer customer
assistance. However, we believe that reduced demand for assistance is a
potential benefit of testing. We note, for example, that IRS officials seek
input from telephone assistors when deciding which forms or instructions
need to be clarified, apparently believing that clarifying the forms and
instructions may help reduce calls to assistors. Finally, testing is one
means for ensuring that even for new forms and instructions requests for
assistance and errors made by taxpayers will be minimized.


We are sending copies of this report to the Chairman and Ranking
Minority Member of the House Committee on Ways and Means and its
Subcommittee on Oversight; the Secretary of the Treasury; the Acting
Commissioner of Internal Revenue; the Director of the Office of
Management and Budget; and other interested parties. We will make
copies available to others on request. In addition, the report will be
available at no charge on the GAO Web site at http://www.gao.gov.




Page 28                               GAO-03-486 Testing Forms and Instructions
This report was prepared under the direction of Charlie Daniel, Assistant
Director. If you have any questions regarding this report, please contact
him or me at (202) 512-9110. Key contributors to this report were David
Alexander, Christopher Currie, Ronald La Due Lake, Anne Laffoon,
Veronica Mayhand, Edward Nannenhorn, and Shellee Soliday.

Michael Brostek




Director, Tax Issues




Page 29                               GAO-03-486 Testing Forms and Instructions
              Appendix I: Objectives, Scope, and
Appendix I: Objectives, Scope, and
              Methodology



Methodology

              To determine how often IRS has used taxpayers to test the clarity of new
              or revised individual income tax forms and instructions, we interviewed
              officials in its W&I Division’s Tax Forms and Publications Division in
              Washington, D.C. We sought to obtain (a) an understanding of the process
              IRS used to develop and revise individual income tax forms and
              instructions and (b) gather information on the forms and instructions IRS
              tested using taxpayers for the 5-year period between July 1997 and June
              2002. We requested information for a 5-year span to help ensure that the
              information collected would reflect the amount of testing usually done by
              IRS. Our work did not include assessing IRS’s processes for developing
              and revising notices or publications or assessing the clarity of any specific
              tax forms or instructions.

              To obtain insights on the benefits of testing written documents, we
              interviewed officials from three federal agencies in the metropolitan
              Washington, D.C., area that perform extensive research and data
              collection using private citizens. We contacted NCHS, BLS, and the Census
              Bureau because they have broad experience in conducting tests of the
              clarity of written documents, such as forms, surveys, and instructions. We
              also contacted a private research firm that specializes in testing forms and
              surveys for a variety of public and private sector clients. We also reviewed
              our own guidance for developing and using questionnaires.23

              To determine the benefits to taxpayers and IRS of testing tax forms and
              instructions with taxpayers prior to their use by the public, we took
              several steps. First, we interviewed IRS officials including Tax Forms and
              Publications Division officials to obtain information on whether they
              perceived testing as being beneficial to taxpayers and IRS. We also
              interviewed IRS’s W&I Research Division officials in Indianapolis to obtain
              their views on whether testing tax forms and instructions benefits
              taxpayers and IRS. The research division officials, among other things,
              collect information on taxpayer errors that the Tax Forms and
              Publications Division uses when deciding which forms and instructions to
              test. These officials provided us with data on changes in error rates for
              EIC and Child Tax Credit forms and instructions before and after IRS
              revised and tested them.

              Second, we developed illustrations of the potential benefits and costs of
              IRS’s testing EIC and Child Tax Credit forms and instructions by analyzing


              23
               GAO/PEMD-10.1.7.




              Page 30                                GAO-03-486 Testing Forms and Instructions
Appendix I: Objectives, Scope, and
Methodology




the data on the changes in error rates obtained from IRS research officials.
To construct our illustrations, we used readily available data and made
certain assumptions. Data were not available on many of the potential
benefits of the changes and on the full range of costs IRS incurred to
conduct the tests. We developed similar illustrations of potential benefits
to taxpayers and IRS if the rate reduction credit instructions had been
tested. In all cases, the illustrations we developed are not actual
assessments of the costs and benefits that were associated with testing
forms and instructions, or that would have resulted if testing had
occurred. In developing the illustrations we sought to be conservative in
estimating benefits, in part because we did not have information on the
full range of costs IRS incurred to test forms and instructions. Because we
had to make various assumptions, our illustrations undoubtedly vary from
actual costs and benefits.

To determine the potential cost savings to IRS of testing these forms and
instructions, we first estimated IRS’s costs to correct a taxpayer error. Our
estimates of IRS’s cost to correct errors were limited to the labor cost
associated with correcting errors in IRS’s math error program. IRS
provided us the number of FTE staff years for operating the ERS that was
used to detect and correct math errors and the number of errors corrected
by ERS during fiscal year 2002. We used this information to calculate an
average labor cost to correct math errors that we then applied to
reductions in EIC and Child Tax Credit errors as well as to error
reductions that might have resulted from testing the rate reduction credit.
Total error correction costs may be higher because we did not include a
number of other costs associated with correcting errors. For example, we
excluded printing and postage costs for the notices sent to taxpayers. IRS
notices often cover more than one issue associated with a tax return and
data were not readily available to determine what portion of the postage
cost might be attributable to EIC or Child Tax Credit issues alone. In
addition, the estimate does not include costs such as equipment and rent.
We did not specifically test the accuracy of the cost information provided;
however, our audits of IRS’s annual financial statements have raised
concerns regarding IRS’s ability to identify all costs associated with a
given program or activity.24




24
 U.S. General Accounting Office, Major Management Challenges and Program Risks:
Department of the Treasury, GAO-03-109 (Washington, D.C.: January 2003).




Page 31                                   GAO-03-486 Testing Forms and Instructions
Appendix I: Objectives, Scope, and
Methodology




Because data were not available on the extent that testing reduced errors
related to EIC and Child Tax Credit forms and instructions, we assumed
different percentage reduction rates in errors due to testing these forms
and instructions. Our illustrations show potential savings based on
assumed percentage reduction in errors attributable to testing of 50, 25,
and 10 percent. Using our estimate of IRS’s costs to correct an error and
the assumed number of errors eliminated by testing, we arrived at the
costs that would have been incurred to correct those errors. The
difference between costs to correct eliminated errors and the costs to test
is the potential cost savings from testing. Cost savings due to reduced
errors likely would not mean reductions to IRS’s budget. Further, savings
likely would mean that IRS would provide services to other taxpayers or
would pursue other compliance or tax collection activities that it would
otherwise have been unable to do.

To determine IRS’s costs to test forms and instructions, we used IRS’s
actual contract costs for testing EIC and Child Tax Credits in 1999. IRS
spent a total of about $56,000 to test both EIC and Child Tax Credit forms
and instructions with focus groups. Although the cost of contracted
support for testing each form and instruction individually likely would
have been somewhat lower than this, we applied the total cost in each
case.25 Because IRS could not provide data, we did not include the costs
associated with IRS letting and managing the contract or the cost for Tax
Forms and Publications Division staff to work with the contractor in
conducting and managing the tests or the cost of any other IRS staff that
were involved in these tests.

To determine the potential benefits to taxpayers from testing EIC and
Child Tax Credit forms and instructions, we calculated estimated values to
taxpayers of the time saved if testing improved form and instruction
clarity, thereby reducing taxpayer errors and the burden of dealing with
IRS error notices. We also estimated taxpayers’ time saved if testing
reduced the time needed to understand and complete tax forms and
instructions. We developed similar illustrations for potential benefits to
taxpayers if the rate reduction credit had been tested. Our benefit
illustrations were based on a series of assumptions. For example, to
estimate the value to taxpayers of time saved from not having to deal with


25
  IRS contracted with a firm to conduct focus groups to test EIC and Child Tax Credit
forms and instructions. The contract was for $54,000 and IRS spent an additional $2,000 on
related travel costs. Because one contractor conducted both tests, IRS officials said it
would be difficult to allocate the $56,000 between the two credits.




Page 32                                       GAO-03-486 Testing Forms and Instructions
Appendix I: Objectives, Scope, and
Methodology




an IRS notice, we applied the minimum wage rate that was in effect after
IRS revised and tested the forms and instructions to estimates of time
taxpayers might save by not having to deal with an IRS notice.26 For these
illustrations we assumed, on average, that taxpayers who received an IRS
error notice might spend either 2 or 5 minutes to deal with it. Data were
unavailable on how much time taxpayers actually spend dealing with IRS’s
notices; however, according to Taxpayer Advocate Service information,
IRS’s notices are difficult for taxpayers to understand. Further, taxpayers
who decide to contest an IRS notice may take time to call or write letters
to IRS or to contact and work with a tax preparer. Data were not readily
available to determine what portion of taxpayers who received an EIC or
Child Tax Credit error notice contested IRS’s change to their tax returns.

For our illustrations of the estimated value of time saved by taxpayers if
testing reduced the time needed to understand and complete EIC and
Child Tax Credit forms and instructions, we assumed that taxpayers would
save on average 1 minute by using clearer forms and instructions. We
excluded from our illustrations those tax returns prepared by preparers
because the taxpayers might not have had to read and understand the
forms and instructions. Because data were not available on the number of
tax returns claiming the Child Tax Credit that were prepared by paid
preparers, we reduced the number of returns claiming the credit by the
same percentage of EIC returns that were prepared by preparers. The
percentage of EIC claimants using paid preparers exceeds the average for
all taxpayers. For the rate reduction credit, our illustration is based on a
30-second time savings and the number of taxpayers who filed on paper
and did not use preparers. We chose a 30-second potential savings for this
illustration because many taxpayers would have had to read only part of
the instructions to determine what to do. We aggregated the times for all
taxpayers and multiplied the total hours saved by the prevailing minimum
wage rate to arrive at estimated benefits to taxpayers.

To determine whether any factors limited IRS’s ability to use individual
taxpayers to test forms and instructions and, if so, how these factors can



26
  Our estimates of the value to taxpayers of time saved from using clearer forms and
instructions are conservative estimates. The minimum wage rate of $5.15 per hour that we
used in our illustrations is substantially less than OMB’s estimate of $30 per hour as the
average value of time for individuals and entities to provide information to the government.
Since EIC claimants are lower-income taxpayers, we used the minimum wage rate.
However, some taxpayers would qualify for EIC even at three times the income of a full
time minimum wage level employee.




Page 33                                       GAO-03-486 Testing Forms and Instructions
Appendix I: Objectives, Scope, and
Methodology




be addressed, we interviewed IRS’s Tax Forms and Publications Division’s
officials and analyzed supporting data they provided us. Regarding
officials’ view that they lacked sufficient time to do more testing, we
reviewed information on the amount of time IRS, NCHS, and the private
research firm we contacted took to perform various types of tests. We also
reviewed IRS’s process for developing new and revised forms and
instructions and determined how many weeks were available between the
dates that various laws were enacted or their provisions became effective
and IRS’s normal October 1st deadline for printing. Finally, regarding
officials’ view that they lacked sufficient resources to do more testing, we
obtained information on the resources available within IRS for testing.

We performed our work from May 2002 through March 2003 in accordance
with generally accepted government auditing standards.




Page 34                               GAO-03-486 Testing Forms and Instructions
              Appendix II: Use of Testing in NCHS, Census,
Appendix II: Use of Testing in NCHS, Census,
              and BLS



and BLS

              Like IRS, the three federal agencies we contacted create written
              documents to be completed by the public. The agencies create documents
              such as forms, surveys, and questionnaires that they use to collect
              information from the public to fulfill their missions. Unlike IRS, described
              below, these agencies routinely test their forms, surveys, and
              questionnaires prior to distribution to the public.

              •   The National Center for Health Statistics, the nation’s principal health
                  statistics agency, compiles statistical information to guide actions and
                  policies relevant to public health and health policy. According to
                  NCHS, obtaining accurate and usable health information is crucial to
                  successfully fulfilling its mission to provide reliable information to the
                  Centers for Disease Control and Prevention and the Department of
                  Health and Human Services. NCHS collects information through
                  various sources including questionnaires that it develops and
                  administers. Researchers at NCHS told us that in support of their
                  research they administer surveys and questionnaires each year in
                  addition to developing questionnaires used by the Centers for Disease
                  Control and Prevention. They test each questionnaire using one-on-one
                  interviews. When documents pertain to a particular rather than a
                  general population, the researchers recruit participants with
                  characteristics similar to those persons who might be completing the
                  forms or questionnaires. For example, researchers recruited asthmatics
                  to test a questionnaire related to asthma. NCHS tests forms in one-on-
                  one settings in which a participant may be asked to work through a
                  form while a moderator observes and then later interviews the
                  participant. This approach allows the researcher to identify specific
                  points at which the forms were confusing or problematic and learn why
                  the participant had difficulty. NCHS prefers to use one-on-one
                  interviews when conducting tests because this method closely
                  resembles the ways in which individuals will be completing the
                  documents since individuals will likely complete the documents by
                  themselves.

              •   The Census Bureau is the principal agency responsible for collecting
                  and providing data about the people and the economy of the United
                  States. An accurate census is important because census results are
                  used to reapportion seats in the House of Representatives, redraw
                  congressional districts and other political boundaries, and address
                  countless other public and private data needs. The Census Bureau
                  collects information through short-form and long-form questionnaires
                  that it develops, tests, and administers. In preparation for the 2000
                  Census, the Congress budgeted millions of dollars to develop and test
                  questionnaires during the 1990s. The Census Bureau’s policy requires



              Page 35                                        GAO-03-486 Testing Forms and Instructions
Appendix II: Use of Testing in NCHS, Census,
and BLS




    that demographic survey questionnaires be tested. It has used focus
    groups and one-on-one interviews to test its questionnaires and forms.
    For example, in fiscal year 1996, the Census Bureau decided to make
    fundamental changes to the traditional census design such as
    shortening census questionnaires. In that year, it budgeted funds to
    test, among other things, respondents’ understanding of race and
    ethnicity questions. The Census Bureau has also conducted detailed
    cost-benefit analyses of alternative designs; in 1992 it tested the
    simplified questionnaire in order to gauge whether the new form would
    increase response rates and reduce costly follow up with households
    that did not respond to the census.27

•   The Bureau of Labor Statistics, the principal fact-finding agency for the
    federal government in the broad field of labor economics and statistics,
    also depends on clear and understandable written documents to collect
    accurate information from the public. According to BLS’s policy,
    testing documents such as forms and surveys prior to use by the
    general public should be undertaken to help identify factors that may
    impede users’ ability to understand forms or surveys. Then these
    factors can be addressed in order to improve the clarity of written
    documents and increase the accuracy of responses. Testing should be
    done in the early stages of document development so that any
    problems with clarity can be identified early. BLS routinely tests its
    written documents using focus groups and one-on-one interviews, and
    uses the results of the tests to make improvements to the documents.




27
  U.S. General Accounting Office, Census Reform: Questionnaire Test Shows
Simplification Holds Promise, GAO/T-GGD-92-59 (Washington, D.C.: July 1, 1992) and
U.S. General Accounting Office, Decennial Census: 1990 Results Show Need for
Fundamental Reform, GAO/GGD-92-94 (Washington, D.C.: June 9, 1992) address
recommendations that the Census Bureau simplify its questionnaire in order to improve its
response rate and thereby reduce the number of follow-ups with nonrespondents and costs
associated with follow-up activities.




Page 36                                        GAO-03-486 Testing Forms and Instructions
          Appendix III: Comments from the Internal Revenue Service
Appendix III: Comments from the Internal
Revenue Service




                       Page 37                                       GAO-03-486 Testing Forms and Instructions
Appendix III: Comments from the Internal Revenue Service




             Page 38                                       GAO-03-486 Testing Forms and Instructions
Appendix III: Comments from the Internal Revenue Service




             Page 39                                       GAO-03-486 Testing Forms and Instructions
Appendix III: Comments from the Internal Revenue Service




             Page 40                                       GAO-03-486 Testing Forms and Instructions
           Appendix III: Comments from the Internal Revenue Service




(440129)
                        Page 41                                       GAO-03-486 Testing Forms and Instructions
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548