oversight

Veterans' Benefits: Improvements Needed in the Reporting and Use of Data on the Accuracy of Disability Claims Decisions

Published by the Government Accountability Office on 2003-09-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to Congressional Requesters




September 2003
                 VETERANS’
                 BENEFITS
                 Improvements Needed
                 in the Reporting and
                 Use of Data on the
                 Accuracy of Disability
                 Claims Decisions




GAO-03-1045 

                                               September 2003


                                               VETERANS’ BENEFITS

                                               Improvements Needed in the Reporting
Highlights of GAO-03-1045, a report to         and Use of Data on the Accuracy of
congressional requesters
                                               Disability Claims Decisions



The Veterans Benefits                          From fiscal years 2001 to 2002, VBA’s accuracy of decision-making in the
Administration (VBA) has a large               disability compensation and pension benefit programs declined from
inventory of claims for benefits               89 percent to 81 percent. The agency had reported a slight improvement in
under its compensation and                     accuracy between fiscal years 2001 and 2002—from 78 percent to 80 percent.
pension programs. The Secretary of             However, we found that these two annual figures were not comparable
the Department of Veterans Affairs
has pledged to substantially reduce
                                               because the agency had substantially changed the way it measured accuracy
this inventory in order to improve             for fiscal year 2002. Although VBA acknowledged a change in its accuracy
timeliness. In response, VBA                   measure in its annual report to the Congress, the agency did not revise its
emphasized producing more claims               2001 figure to allow for an appropriate comparison with 2002. VBA officials
decisions per year. GAO was asked              GAO spoke with suggested several factors that may have contributed to the
to ascertain how accuracy has                  decline in accuracy. We were not able to quantify the relative contribution of
changed since VBA increased its                these factors. These factors included VBA’s emphasis on increasing claims
emphasis on production and to                  decisions, the specific processing requirements of the Veterans Claims
report on the agency’s efforts to              Assistance Act (VCAA) of 2000, and the relative inexperience of VBA’s
ensure the accuracy of its                     claims processing staff.
decisions.
                                               To help ensure accountability for accuracy, VBA set accuracy standards for
                                               its regional offices. Although VBA has regional office-level accuracy data, it
GAO recommends that VBA                        has not made full use of this information to encourage better performance
                                               from regional offices with low accuracy scores. For example, in fiscal year
•	   report the accuracy of VBA                2002, VBA did not require offices with poor accuracy to prepare
     disability compensation and               improvement plans and gave performance awards to two offices that clearly
     pension claims decisions to the           failed to meet VBA’s accuracy goal.
     Congress and other
     stakeholders in a manner that             VBA Compensation and Pension Rating Products Decided and Accuracy Rates, Fiscal Years
     allows for valid comparisons              2001-2002
     of accuracy across fiscal years           Number completed                                                 Percentage
     and                                       (in thousands)
•	   better hold regional offices              1,000                                          89                      100
     accountable for the accuracy                800
                                                                         797                                81
                                                                                                                      80
     of their claims decisions, by
     increasing the use of its                   600        481                                                       60

     regional office accuracy data.              400                                                                  40

                                                 200                                                                  20
VBA concurred with GAO’s
                                                   0                                                                  0
recommendations and provided
                                                              Production                         Accuracy
                                                                                                    B
additional information about its
efforts to improve accuracy.                                      FY 2001                     FY 2002

                                               Source: GAO analysis of Veterans Benefits Administration data.

                                               Note: Accuracy rates in the above chart are estimates based on samples of rating products
                                               completed in fiscal years 2001 and 2002 and have margins of error of about plus or minus
                                               1 percentage point at the 95-percent level of confidence. We determined these accuracy rates by
                                               comparing substantially similar questions for the 2 fiscal years that focused on whether the decision
                                               to grant or deny benefits was correct. The fiscal year 2002 accuracy score includes a specific
www.gao.gov/cgi-bin/getrpt?GAO-03-1045.        question on proper VCAA pre-decision notification that VBA did not include in its fiscal year 2001
                                               quality reviews.
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Cynthia A.
Bascetta at (202) 512-7215 or
bascettac@gao.gov.
Contents 



Letter                                                                                                   1
               Results in Brief 
                                                                        2
               Background
                                                                               3
               Accuracy Declined in Fiscal Year 2002
                                                    6
               VBA Could Have Done More to Hold Regional Offices Accountable 

                 for Accuracy                                                                             8
               Conclusions                                                                               10
               Recommendations                                                                           11
               Agency Comments and Our Response                                                          11

Appendix I 	   Comments from the Veterans Benefits
               Administration                                                                            13




Figure
               Figure 1: Fiscal Year 2001 STAR Rating Checklist—Criteria for a
                        Correct VBA Decision                                                              5




               Abbreviations

               BVA               Board of Veterans’ Appeals 

               SIPA              Systematic Individual Performance Assessment 

               STAR              Systematic Technical Accuracy Review 

               VA                Department of Veterans Affairs 

               VBA               Veterans Benefits Administration 

               VCAA              Veterans Claims Assistance Act of 2000 

               VSR               veterans service representative 





               This is a work of the U.S. government and is not subject to copyright protection in the
               United States. It may be reproduced and distributed in its entirety without further
               permission from GAO. However, because this work may contain copyrighted images or
               other material, permission from the copyright holder may be necessary if you wish to
               reproduce this material separately.




               Page i                                                   GAO-03-1045 Veterans' Benefits
United States General Accounting Office
Washington, DC 20548



                                   September 30, 2003


                                   The Honorable Arlen Specter 

                                   Chairman 

                                   The Honorable Bob Graham 

                                   Ranking Minority Member 

                                   Committee on Veterans’ Affairs 

                                   United States Senate 


                                   The Honorable John D. Rockefeller IV 

                                   United States Senate 


                                   At the beginning of fiscal year 2002, over 400,000 claims for disability 

                                   compensation and pension benefits were waiting for Veterans Benefits 

                                   Administration (VBA) decisions. On average, veterans were waiting about 

                                   6 months and many were waiting more than a year. VBA has been making 

                                   a concerted effort to reduce the inventory of claims and complete 

                                   decisions in an average of 100 days by the end of fiscal year 2003. In order 

                                   to improve the timeliness of its disability claims decisions, VBA 

                                   established a goal of completing about 839,000 decisions in fiscal year 2002 

                                   and reducing its inventory of pending claims to about 316,000 by the end 

                                   of that year. During fiscal year 2002, VBA increased its claims decisions by

                                   two-thirds from about 481,000 to about 797,000 and reduced its inventory 

                                   of pending claims by about one-fifth from about 421,000 to about 346,000. 

                                   For fiscal year 2003, VBA’s goals are to produce about 806,000 decisions 

                                   and further reduce inventory to 250,000 claims. 


                                   In light of VBA’s emphasis on completing more claims decisions more 

                                   quickly, you asked us to assist the Committee in its oversight of VBA by 

                                   examining the agency’s efforts to ensure the accuracy of decision-making 

                                   in its disability compensation and pension benefits programs. In response, 

                                   we assessed (1) how accuracy may have changed since VBA increased its 

                                   emphasis on production and (2) how well VBA ensures the accuracy of its 

                                   decisions. 


                                   We reviewed VBA’s efforts to measure and report accuracy for fiscal years 

                                   2001 and 2002.1 To determine how accuracy had changed over that time, 



                                   1
                                    This report focuses on the accuracy of rating decisions. Rating decisions are primarily
                                   decisions on original claims for compensation and pension benefits and reopened claims.
                                   For example, veterans may file reopened claims if they believe that their service-connected
                                   conditions have worsened.



                                   Page 1                                                    GAO-03-1045 Veterans' Benefits
                   we independently calculated and compared VBA’s accuracy rates for fiscal
                   years 2001 and 2002. As part of our analysis of how well VBA ensures
                   national and regional office accuracy, we reviewed VBA’s accuracy
                   measurement system, known as the Systematic Technical Accuracy
                   Review (STAR) program, including how VBA incorporates accuracy into
                   its performance standards for both regional office directors and claims
                   processing staff. In addition, we examined VBA’s criteria for regional
                   office performance awards and the results of reviews by VBA survey
                   teams of regional offices. We discussed factors affecting accuracy with
                   staff at VBA regional offices in Phoenix, Arizona; St. Petersburg, Florida;
                   Boston, Massachusetts; Pittsburgh, Pennsylvania; and Houston, Texas.2 We
                   also discussed accuracy issues with representatives of the American
                   Legion, Disabled American Veterans, Paralyzed Veterans of America, and
                   Veterans of Foreign Wars to gain their perspectives on VBA efforts to
                   improve accuracy. We conducted our review from January through August
                   of 2003 in accordance with generally accepted government auditing
                   standards.


                   From fiscal years 2001 to 2002, VBA’s accuracy of decision making in the
Results in Brief   compensation and pension programs declined from 89 percent to 81
                   percent.3 We determined these accuracy rates by comparing substantially
                   similar questions for the 2 fiscal years that focused on whether the
                   decision to grant or deny benefits was correct. The agency had reported a
                   slight improvement in accuracy between fiscal years 2001 and 2002—from
                   78 percent to 80 percent. However, we found that these two annual figures
                   were not comparable because the agency had substantially changed the
                   way it measured accuracy for fiscal 2002. Although VBA acknowledged a
                   change in its accuracy measure in its annual report to the Congress, the
                   agency did not revise its 2001 figure to allow for an appropriate
                   comparison with 2002. VBA officials attributed the decline in accuracy
                   during this period to several factors. We were not able to quantify the
                   relative contribution of these factors. These factors included headquarters
                   emphasis on production, the specific processing requirements of the


                   2
                    Except for Pittsburgh, we selected the regional offices we visited based on a combination
                   of their fiscal year 2002 rating accuracy and the number of rating decisions they produced.
                   We visited the Pittsburgh Regional Office to observe how VBA conducts an inspection of a
                   regional office’s compensation and pension operations as well as to discuss factors
                   affecting accuracy.
                   3
                    The margin of error for rating accuracy VBA-wide for fiscal years 2001 and 2002 was about
                   1 percentage point at a 95-percent level of confidence.




                   Page 2                                                    GAO-03-1045 Veterans' Benefits
             Veterans Claims Assistance Act of 2000, and the relative inexperience of
             VBA’s claims processing staff.

             To help ensure accountability for accuracy, VBA set accuracy standards
             for its regional offices. However, VBA has not fully used data gathered on
             the accuracy of its regional office decisions to encourage better regional
             office performance. For example, while some regional offices were
             required to prepare plans to improve claims processing production, similar
             plans were not required for offices to improve accuracy. Also, two regional
             offices received overall performance awards despite not meeting VBA’s
             accuracy standard. Unless VBA strikes a balance between production
             goals and accuracy standards, it cannot ensure that as it improves decision
             timeliness, it will also improve decision accuracy.

             This report contains recommendations to the Secretary of Veterans Affairs
             to direct VBA to improve its reporting on and use of accuracy data. VBA
             concurred with our recommendations and provided additional information
             about its efforts to improve accuracy.


             Through its disability compensation program, VBA pays monthly benefits
Background   to veterans with service-connected disabilities (injuries or diseases
             incurred or aggravated while on active military duty) according to the
             severity of the disability. VBA’s pension benefit program pays monthly
             benefits to wartime veterans who have low incomes and are permanently
             and totally disabled for reasons not service-connected.4 In addition, VBA
             pays dependency and indemnity compensation to some deceased veterans’
             spouses, children, and parents and to survivors of service members who
             died on active duty. In fiscal year 2002, VBA paid over $22 billion in
             disability compensation to an average of about 2.4 million veterans and
             over 300,000 survivors. VBA also paid over $3 billion in pensions to an
             average of about 580,000 veterans and survivors.

             The amount of disability compensation largely depends on the degree to
             which a veteran is disabled. VBA determines the degree to which veterans
             are disabled in 10 percent increments on a scale of 0 to 100 percent. Basic
             monthly payments range from $104 for 10 percent disability to $2,193 for


             4
              Veterans who are 65 years or older do not have to be permanently and totally disabled to
             become eligible for pension benefits, as long as they meet the other requirements for
             income and military service. VBA also pays pensions to surviving spouses and unmarried
             children of deceased wartime veterans.




             Page 3                                                    GAO-03-1045 Veterans' Benefits
100 percent disability. About 65 percent of veterans receiving disability
compensation have disabilities rated at 30 percent and lower; about
8 percent have disabilities rated at 100 percent. The most common
impairments for veterans who began receiving compensation in fiscal year
2001 were tinnitus (ex., a ringing in the ear), auditory acuity impairment
rated at 0 percent (i.e., mild hearing difficulties), skeletal conditions,
arthritis due to trauma, post-traumatic stress disorder, and scars.
Eligibility and priority for other Department of Veterans Affairs (VA)
benefits and services such as health care and vocational rehabilitation are
affected by these VA disability ratings.

When a veteran submits a claim to any of VBA’s 57 regional offices, a
veterans service representative (VSR) is responsible for obtaining the
relevant evidence to evaluate the claim. Such evidence includes veterans’
military service records, medical examinations and treatment records
from VA medical facilities, and treatment records from private medical
service providers. Once a claim is developed (i.e., has all the necessary
evidence), a rating VSR, also called a rating specialist, evaluates the claim,
determines whether the claimant is eligible for benefits, and assigns a
rating based on degree of disability. Veterans with multiple disabilities
receive a single composite rating. For veterans claiming pension eligibility,
the regional office determines if the veteran served in a period of war, is
permanently and totally disabled for reasons not service-connected, and
meets the income thresholds for eligibility. A veteran who disagrees with
the regional office’s decision for either program can appeal sequentially to
VA’s Board of Veterans’ Appeals (BVA), the U.S. Court of Appeals for
Veterans Claims, and the U.S. Court of Appeals for the Federal Circuit.

In fiscal year 1999, VBA implemented a quality review system to improve
the measurement of the accuracy of its claims decisions. Under STAR,
VBA selects a random sample of completed claims decisions each month
from each of its 57 regional offices. STAR quality review staff review these
claims using a standard checklist. If a claim has any error, VBA counts the
entire claim as incorrect for accuracy rate computation purposes. STAR
then returns the case file and the results of the review to the regional
office that made the decision. If an error was found, the regional office is
required to either correct it or request reconsideration of the error by
STAR.




Page 4                                           GAO-03-1045 Veterans' Benefits
VBA made several significant changes to its STAR system in fiscal year
2002. First, in response to our March 1999 recommendation, VBA made the
STAR review staff organizationally independent of the regional offices to
ensure the independence of the reviews.5 Second, VBA more than doubled
the number of rating decisions it reviewed (from 3,209 in fiscal year 2001
to 6,646 in fiscal year 2002) in order to obtain more precise regional office
level accuracy scores. Third, VBA improved its key measure to focus more
on whether decisions to grant or deny benefits were correct. Previously,
VBA’s key measure also included decision documentation and notification
issues. Figure 1 shows the areas covered by the checklist used to measure
accuracy in fiscal year 2001. The revised key accuracy measure includes
only the first four areas of the checklist.

Figure 1: Fiscal Year 2001 STAR Rating Checklist—Criteria for a Correct VBA
Decision


    � Address all issues – The decision addressed all claimed disabilities and
          benefits, and any disabilities or benefits that could be inferred from the available
          evidence.

    � Proper development – VBA made an adequate attempt to obtain all evidence
          needed to decide the claim.

    � Grant or deny – The decision to grant or deny was correct; if granted, the
          percentage evaluation was correct.

    � Award actions – Where benefits were granted, the effective dates of benefits
          and payment amounts were correct.

    � Reasons and bases          – The decision discussed all applicable evidence and the
          basis of the decision was explained.

    � Notification – The claimant and power of attorney, if applicable, were correctly
          notified of the decision and were notified of appeal rights.

Source: Veterans Benefits Administration.




5
 For more information, see U.S. General Accounting Office, Veterans’ Benefits Claims:
Further Improvements Needed in Claims-Processing Accuracy, GAO/HEHS-99-35
(Washington, D.C.: Mar. 1, 1999) and Veterans’ Benefits: Quality Assurance for Disability
Claims Processing, GAO-01-930R (Washington, D.C.: Aug. 23, 2001).




Page 5                                                         GAO-03-1045 Veterans' Benefits
                       From fiscal years 2001 to 2002, VBA’s accuracy of decision making in the
Accuracy Declined in   compensation and pension programs declined from 89 percent to 81
Fiscal Year 2002       percent. We determined these accuracy rates by comparing substantially
                       similar questions for the 2 fiscal years that focused on whether the
                       decision to grant or deny benefits was correct.6 However, the agency
                       reported that it had improved its accuracy for that period. This
                       discrepancy occurred because VBA did not report comparable accuracy
                       data for the 2 fiscal years. VBA reported its fiscal year 2002 accuracy based
                       on a revised measure that excluded two areas of the checklist—reasons
                       and bases and notification. VBA then compared this to its fiscal year 2001
                       accuracy, which included these two areas of the checklist. VBA officials
                       attributed the decline in accuracy during this period to several factors. We
                       were not able to quantify the relative contribution of these factors. These
                       factors included headquarters emphasis on production, the specific
                       processing requirements of the Veterans Claims Assistance Act of 2000,
                       and the relative inexperience of VBA’s claims processing staff.

                       In fiscal year 2002, VBA made progress in increasing production of claims
                       decisions and reducing inventory. The number of claims decisions rose
                       substantially from about 481,000 in fiscal year 2001 to about 797,000 in
                       fiscal year 2002. During the same time period, the number of claims VBA
                       received increased from about 674,000 to 722,000. The higher level of
                       production permitted VBA to reduce its inventory of pending claims by
                       18 percent from about 421,000 to about 346,000.

                       Although accuracy actually declined, VBA reported in its performance
                       report to the Congress for fiscal year 2002 that the accuracy of its rating
                       decisions for compensation and pension benefits had improved slightly.
                       VBA reported that accuracy improved from 78 percent in fiscal 2001 to
                       80 percent in fiscal 2002.7 However, these rates were not comparable
                       because the new “benefit entitlement accuracy” rate for fiscal year 2002
                       was based on fewer areas of the checklist that focused on whether and to
                       what extent the claimant received a benefit. In its fiscal year 2002
                       performance report, VA made note of this change but did not recalculate


                       6
                        The fiscal year 2002 accuracy score includes a specific question on proper VCAA pre-
                       decision notification that VBA did not include in its fiscal year 2001 quality reviews.
                       7
                        Due to the time lag involved between receiving claims files and conducting the accuracy
                       reviews of disability compensation and pension decisions as well as deadlines for
                       incorporating data into its annual accountability report, reported accuracy rates can differ
                       from final rates for the reported year. VBA’s final rating accuracy rate for fiscal year 2002
                       was 81 percent.




                       Page 6                                                      GAO-03-1045 Veterans' Benefits
the fiscal 2001 rate to provide a correct comparison. When we recomputed
the agency’s accuracy rate for fiscal 2001 using its revised criteria, the
result was 89 percent, not 78 percent as reported. As a result, the correct
comparison showed a decline in accuracy from 89 percent to 81 percent.
The agency corroborated our finding after performing its own
computation at our request.

VBA officials we spoke with cited a number of factors that affected
accuracy over the fiscal years 2001 to 2002 period. A key factor cited was
VBA’s emphasis on production, such that accuracy became a lesser
priority. In 2001, the Secretary of Veterans Affairs set an ambitious goal to
reduce the agency’s average time to complete a rating decision from 173
days (in fiscal year 2000) to 100 days by the end of fiscal year 2003. Also,
the Secretary tasked VBA with reducing the inventory of rating claims to
250,000 by the end of September 2003. VBA managers responded by
emphasizing the completion of more rating cases each month. The result
has been a significant increase in production and a significant reduction in
the rating inventory since the end of fiscal year 2001.

In fiscal year 2002, VBA incorporated its new production targets into its
measures for regional office performance.8 Regional office directors
became accountable for specific targets for production, inventory
reduction, and timeliness improvement. The agency also established
regional office performance award criteria that gave more weight to
efficiency than accuracy. Three of the four criteria for cash awards were
based on production and timeliness and one on accuracy; bonuses could
be received without meeting the accuracy criteria.

VBA officials also attributed accuracy problems to the addition of more
specific processing requirements under the Veterans Claims Assistance
Act (VCAA) of 2000. The act broadened VBA’s responsibility to assist
veterans — such as notifying them of needed evidence and helping them
develop that evidence for their claims. VBA incorporated VCAA
requirements into its STAR rating accuracy checklist in fiscal year 2002. In
the first 4 months of fiscal year 2002, VCAA errors accounted for almost
half of all errors identified by STAR rating reviewers. In April 2002, VBA
required each regional office to re-train its claims processing staff on the


8
 VBA established specific monthly production targets for its regional offices in December
2001. For more information on these targets see U.S. General Accounting Office, Veterans’
Benefits: Despite Recent Improvements, Meeting Claims Processing Goals Will Be
Challenging, GAO-02-645T (Washington, D.C.: Apr. 26, 2002).




Page 7                                                   GAO-03-1045 Veterans' Benefits
                    new requirements.9 Over the last 8 months of fiscal year 2002, the
                    proportion of VCAA-related errors declined to about one-third of all errors.

                    VBA officials also cited the relative inexperience of VBA’s claims
                    processing staff. Regional office officials noted that they had many VSRs
                    and rating specialists with insufficient training and experience to be fully
                    proficient at developing and making decisions on claims. For example,
                    officials at two offices we visited noted that their accuracy was affected by
                    large numbers of inexperienced staff, due to a large increase in staff size
                    or a relatively high rate of turnover. VBA has found that attrition rates for
                    new claims processing staff hired over a 3-year period ranged from
                    0 percent to 49 percent at some regional offices.10


                    While VBA has established accuracy standards for its regional offices, it
VBA Could Have      has not made the best use of its accuracy data on regional offices. VBA
Done More to Hold   collected regional office accuracy data in fiscal year 2002 but has not fully
                    used the information to evaluate regional office performance, correct
Regional Offices    errors, and identify needed training that could reduce errors. Unless VBA
Accountable for     better uses these data to hold regional offices accountable for accuracy, it
                    cannot ensure that as decisions become timelier, they also become more
Accuracy            accurate. VBA is developing an agencywide system for evaluating
                    individual claims processing employees’ performance against VBA’s
                    individual accuracy standards.

                    In fiscal year 2002, VBA improved its STAR review system to provide
                    independent review of decisions and more precise accuracy data at the
                    regional office level. Regional accuracy scores are estimates based on
                    samples of cases and revealed high and low performers relative to the
                    agency’s fiscal year 2002 accuracy goal of 85 percent. Eleven regional
                    offices clearly failed to meet the goal while 3 clearly met or exceeded the




                    9
                     For more information on VCAA implementation, see U.S. General Accounting Office,
                    Veterans’ Benefits: VBA’s Efforts to Implement the Veterans Claims Assistance Act Need
                    Further Monitoring, GAO-02-412 (Washington, D.C.: July 1, 2002).
                    10
                     For more information, see U.S. General Accounting Office, Veterans Benefits
                    Administration: Better Collection and Analysis of Attrition Data Needed to Enhance
                    Workforce Planning, GAO-03-491 (Washington, D.C.: Apr. 28, 2003).




                    Page 8                                                 GAO-03-1045 Veterans' Benefits
goal.11 However, VBA managers did not fully use the regional office-
specific information generated by the STAR unit as a basis for improving
accuracy at the regional office level. For example, VBA did not require
regional offices that failed to meet the national accuracy goal of 85 percent
to prepare strategies for improvement. In contrast, in fiscal year 2002,
13 regional offices that did not meet established targets for production,
inventory, and timeliness were required to develop corrective action plans.
Also, as noted earlier, VBA did not make rating accuracy a prerequisite for
receiving a performance award. Three of the four criteria for cash awards
were based on production and timeliness and one on accuracy; a bonus
could be received without meeting the accuracy criteria. In fiscal year
2002, 2 offices received performance awards despite clearly not meeting
VBA’s accuracy goal. These offices had accuracy rates of 71 percent and
75 percent, respectively.

VBA’s regional offices did not always properly address STAR errors that
were returned to them, a violation of VBA policy that regional offices
either correct errors or formally request that the STAR unit reconsider its
error call. VBA survey teams, which review judgmental samples of STAR
cases with errors, found that the 18 regional offices they visited from
October 2001 through December 2002 had failed to respond properly to
over 40 percent of the errors reviewed. At 14 of the 18 offices, the survey
teams found problems significant enough to recommend improvements in
STAR error handling. For example, teams recommended that several
offices establish management controls to ensure that they properly
address all errors identified by the STAR unit. Effective April 1, 2003, VBA
established a new tracking system that required regional offices to report
to headquarters on actions taken to address all errors identified by the
STAR unit.

In addition to not correcting all errors, several of the regional offices
visited by the survey teams were not making adequate use of STAR errors
to identify trends that could be used to provide feedback and training to
claims processing staff. Officials at regional offices we visited commented
that this information is useful for identifying trends for feedback and
training purposes. For example, in fiscal year 2002, the results of STAR
reviews helped to alert management of the Boston Regional Office that


11
  It could not be determined if the remaining offices met the accuracy goal because of the
margins of error associated with their accuracy scores. In fiscal year 2002, the margins of
error for rating accuracy for each regional office ranged from 4 to 9 percentage points at
the 95-percent level of confidence.




Page 9                                                     GAO-03-1045 Veterans' Benefits
                many of its claims processors were having difficulties with the new VCAA
                legislation, permitting management to provide special training for the
                Boston staff. The agency survey teams recommended that 7 of the regional
                offices reviewed between October 2001 and December 2002 take steps to
                use STAR reviews for feedback and training.

                In addition to regional office-level accuracy standards, VBA has also
                established quantifiable accuracy standards for its experienced claims
                processing employees, as part of these employees’ annual performance
                evaluations.12 The accuracy standards for experienced rating specialists
                require that they correctly decide 85 percent of 60 randomly selected
                rating decisions per year (5 decisions per month).13 VBA is developing an
                agencywide review process, the Systematic Individual Performance
                Assessment (SIPA), to assess compliance with the individual accuracy
                standards. VBA developed a uniform checklist, similar to the STAR
                checklist for reviewing rating decisions, which offices could use to assess
                rating specialists’ accuracy. Unlike the STAR system, reviews are
                conducted before decisions are finalized.

                In April 2003, VBA surveyed regional offices to determine the status of
                their efforts to measure individual accuracy and assess the staff resources
                needed to implement SIPA VBA-wide. Individual reviews were being
                conducted by supervisors, such as team coaches, and nonsupervisory
                staff, such as Decision Review Officers, who are responsible for handling
                appeals. At the time of our review, VBA was in the process of deciding
                who would perform the reviews. In its efforts to develop SIPA as a
                consistent review process, VBA is considering issues such as ensuring that
                consistent feedback is provided to the employees and all regional offices
                are using the same review checklists.


                While VBA’s emphasis on production succeeded in increasing the number
Conclusions 	   of claims decisions made during fiscal year 2002, the accuracy of those
                decisions declined. Because VBA did not report comparable accuracy
                rates for fiscal years 2001 and 2002 in its performance and accountability
                report, Members of Congress, Department of Veterans Affairs
                management, and claimants did not know of this decline in accuracy. The


                12
                 In general, inexperienced claims processing employees have all of their work reviewed.
                13
                 These standards, according to VBA officials, are intended to be floors; individual regional
                offices can set higher standards.




                Page 10                                                    GAO-03-1045 Veterans' Benefits
                       lack of accurate performance data hampers stakeholders’ ability to
                       monitor VBA’s performance in serving the nation’s veterans.

                       In addition, VBA could better use the accuracy data it has to hold its
                       regional offices more accountable for accuracy. While VBA has made
                       progress in measuring regional office accuracy, it has not made full use of
                       the results to reward high performing offices or to improve poorly
                       performing offices. VBA has made significant progress in getting its
                       compensation and pension claims workload under control to improve
                       timeliness for veterans waiting for decisions on their claims. However,
                       VBA’s accuracy has declined. Unless VBA balances its emphasis on
                       producing more and faster decisions with an increased emphasis on
                       accuracy, it will be difficult to ensure that veterans are getting the benefits
                       to which they are entitled in a timely manner.


                       We recommend that the Secretary of Veterans Affairs direct the Under 

Recommendations        Secretary for Benefits to 


                   •   report the accuracy of VBA disability compensation and pension claims 

                       decisions to the Congress and other stakeholders in a manner that allows 

                       for valid comparisons of accuracy across fiscal years and 

                   •   better hold regional offices accountable for the accuracy of their claims 

                       decisions, by increasing the use of its regional office accuracy data, while 

                       at the same time maintaining an appropriate emphasis on production. 



                       In its written comments on a draft of this report (see app. I), VBA 

Agency Comments        concurred with our recommendations. However, VBA suggested that more 

and Our Response       effective use of STAR had been made to improve performance than our 

                       report concludes. We did note that VBA used STAR data to improve 

                       accuracy. Specifically, we noted in our report that VCAA-related errors 

                       declined to about one-third of all errors after VBA identified the errors 

                       through STAR and required retraining. We support VBA’s continued efforts 

                       to emphasize accuracy and timeliness as critical components of quality. 



                       We will send copies of this report to the Secretary of Veterans Affairs,
                       appropriate congressional committees, and other interested parties. We
                       will also make copies of this report available to others on request. The
                       report will also be available at no charge on GAO’s Web site at
                       http://www.gao.gov.




                       Page 11                                           GAO-03-1045 Veterans' Benefits
If you or your staffs have any questions regarding this report, please call
me at (202) 512-7215 or Irene Chu, Assistant Director, at (202) 512-7102. In
addition to those named, Susan Bernstein, Kristine Braaten, Kevin
Jackson, Joseph Natalicchio, Martin Scire, and Greg Whitney made key
contributions to this report.




Cynthia A. Bascetta
Director, Education, Workforce,
 and Income Security Issues




Page 12                                         GAO-03-1045 Veterans' Benefits
Appendix I: Comments from the Veterans
Benefits Administration




              Page 13          GAO-03-1045 Veterans' Benefits
Appendix I: Comments from the Veterans
Benefits Administration




Page 14                                  GAO-03-1045 Veterans' Benefits
           Appendix I: Comments from the Veterans
           Benefits Administration




(130228)
           Page 15                                  GAO-03-1045 Veterans' Benefits
                           The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission              Congress, exists to support Congress in meeting its constitutional responsibilities
                           and to help improve the performance and accountability of the federal
                           government for the American people. GAO examines the use of public funds;
                           evaluates federal programs and policies; and provides analyses,
                           recommendations, and other assistance to help Congress make informed
                           oversight, policy, and funding decisions. GAO’s commitment to good government
                           is reflected in its core values of accountability, integrity, and reliability.


                           The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of        through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and            text files of current reports and testimony and an expanding archive of older
                           products. The Web site features a search engine to help you locate documents
Testimony                  using key words and phrases. You can print these documents in their entirety,
                           including charts and other graphics.
                           Each day, GAO issues a list of newly released reports, testimony, and
                           correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                           daily. The list contains links to the full-text document files. To have GAO e-mail
                           this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail
                           alerts” under the “Order GAO Products” heading.


Order by Mail or Phone 	   The first copy of each printed report is free. Additional copies are $2 each. A
                           check or money order should be made out to the Superintendent of Documents.
                           GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                           single address are discounted 25 percent. Orders should be sent to:
                           U.S. General Accounting Office
                           441 G Street NW, Room LM
                           Washington, D.C. 20548
                           To order by Phone: 	 Voice:      (202) 512-6000
                                                TDD:        (202) 512-2537
                                                Fax:        (202) 512-6061


                           Contact:
To Report Fraud,
                           Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in        E-mail: fraudnet@gao.gov
Federal Programs           Automated answering system: (800) 424-5454 or (202) 512-7470


                           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs 	           U.S. General Accounting Office, 441 G Street NW, Room 7149
                           Washington, D.C. 20548