oversight

Veterans' Benefits Claims: Further Improvements Needed in Claims-Processing Accuracy

Published by the Government Accountability Office on 1999-03-01.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to the Ranking Minority Member,
                 Committee on Veterans’ Affairs, House
                 of Representatives


March 1999
                 VETERANS’
                 BENEFITS CLAIMS
                 Further Improvements
                 Needed in
                 Claims-Processing
                 Accuracy




GAO/HEHS-99-35
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Health, Education, and
      Human Services Division

      B-281315

      March 1, 1999

      The Honorable Lane Evans
      Ranking Minority Member
      Committee on Veterans’ Affairs
      House of Representatives

      Dear Mr. Evans:

      In fiscal year 1997, the Department of Veterans Affairs (VA) paid about $19
      billion in compensation and pension benefits to more than 3 million
      veterans and their dependents and survivors. The compensation program
      pays monthly benefits to veterans with service-connected disabilities
      (injuries or diseases incurred or aggravated while on active military duty).
      Veterans with service-connected disabilities are entitled to compensation
      benefits even if they are working and regardless of the amount they earn.
      By contrast, the pension program pays monthly benefits to wartime
      veterans who have low incomes and are permanently and totally disabled
      for reasons not connected to their service.

      VA has 58 regional offices (RO) that process veterans’ compensation and
      pension claims and decide whether to award benefits. The ROs develop
      evidence and adjudicate these claims under program guidance and policy
      provided by VA’s Veterans Benefits Administration (VBA). In recent years,
      the accuracy of RO claims processing has been the subject of concern and
      attention within VA and from the Congress and veterans’ service
      organizations. Although VBA had been reporting until recently that ROs
      process claims accurately more than 95 percent of the time, questions
      arose about RO accuracy because, when dissatisfied veterans appealed ROs’
      initial decisions, the Board of Veterans’ Appeals during fiscal years 1993-97
      reversed about 19 percent of the appealed decisions and remanded about
      47 percent back to ROs for further development and reconsideration. In
      fiscal year 1998, VBA pilot tested its new accuracy measurement system,
      known as the Systematic Technical Accuracy Review (STAR) system. Using
      the STAR system, VBA found an accuracy rate of only 64 percent for RO
      initial decisions, indicating that VBA needs to give more attention to
      ensuring that ROs make the correct decision the first time so that veterans
      need not make unnecessary appeals or be unnecessarily delayed in
      receiving benefits owed them.

      VBA   implemented the STAR system nationwide in October 1998. VBA sees
      STAR   as an improvement in its ability to measure accuracy and identify and




      Page 1                                   GAO/HEHS-99-35 Veterans’ Benefits Claims
                   B-281315




                   correct the causes of claims-processing errors. As agreed with your office,
                   this report addresses (1) the extent of improvements made by STAR in
                   measuring claims-processing accuracy, (2) additional efforts needed to
                   strengthen the system, and (3) challenges VBA faces in meeting goals for
                   improving claims-processing accuracy.

                   In conducting our review, we spoke with officials of and reviewed reports
                   and policy guidance by VBA, the Board of Veterans’ Appeals, and 15 ROs.
                   We also received available data from VBA, the Board, and the ROs.
                   Furthermore, we contacted and reviewed documents from several
                   veterans’ service organizations, the National Academy of Public
                   Administration, the Veterans’ Claims Adjudication Commission, and VA’s
                   Office of Inspector General. We also obtained information on the quality
                   assurance programs of several other organizations, including the Social
                   Security Administration (SSA), which administers the largest federal
                   disability benefits program. We conducted our review between
                   October 1997 and December 1998 in accordance with generally accepted
                   government auditing standards.


                   The new STAR system represents an important step forward by VBA in
Results in Brief   measuring the accuracy of compensation and pension claims processing.
                   Compared with the previous system, STAR focuses more on RO decisions
                   that are likely to contain processing errors, uses a stricter method for
                   computing accuracy rates, provides more data on the performance of
                   organizational levels within VBA, collects more data on processing errors,
                   and stores more accuracy review results in a centralized database for
                   review and analysis.

                   Even so, VBA can further strengthen STAR’s ability to identify error-prone
                   cases and claims-processing weaknesses so that it can take corrective
                   actions. VBA needs to better pinpoint error-prone cases and weaknesses in
                   the development of evidence by collecting more specific data on the types
                   of medical characteristics and deficiencies in medical evidence that are
                   most prevalent in incorrect decisions. VBA can also better address
                   vulnerabilities in the integrity of accuracy data. Currently, STAR reviewers
                   in ROs do not have sufficient separation of duties or adequate
                   independence to meet government standards for internal controls or
                   program performance audits. These shortcomings raise concern about the
                   integrity of STAR accuracy data, which are a key factor in the performance




                   Page 2                                  GAO/HEHS-99-35 Veterans’ Benefits Claims
             B-281315




             measurement system designed by VBA to meet the requirements of the
             Government Performance and Results Act of 1993.1

             While such system improvements are necessary, they alone are not
             sufficient for VBA to meet its goal for improving accuracy. Using the STAR
             pilot test’s 64-percent accuracy rate as a baseline, VBA’s goal is to achieve a
             93-percent accuracy rate by fiscal year 2004. As acknowledged by VBA,
             however, it faces management challenges that it must address successfully
             in order to meet its accuracy improvement goal. To do this, VBA recognizes
             that (1) its newly implemented performance measurement system must
             hold program managers accountable for performance and (2) the training
             program under development must effectively train the current RO
             workforce as well as the many new employees who will have to be hired in
             the coming decade to replace those who retire. It is too early to determine
             whether VBA’s efforts to meet these challenges will be successful.

             This report makes recommendations to (1) further strengthen VBA’s ability
             to identify error-prone cases by collecting more detailed data on the
             human body systems and specific impairments involved in disability
             claims as well as data on specific deficiencies in medical evidence and
             examinations, (2) implement a system for reviewing claims-processing
             accuracy that meets the government’s internal control standard on
             separation of duties and the program performance audit standard on
             organizational independence, and (3) keep the Congress informed on VBA’s
             progress in establishing stricter employee accountability and developing
             more effective training for claims adjudicators.


             VBA’s Compensation and Pension Service, located at VA headquarters,
Background   formulates the policy and guidance used by the RO staff who receive,
             develop, and evaluate veterans’ compensation and pension claims. The
             compensation program pays monthly benefits to veterans with
             service-connected disabilities (injuries or diseases incurred or aggravated
             while on active military duty). Veterans with service-connected disabilities
             are entitled to compensation benefits even if they are working and
             regardless of the amount they earn. By contrast, the pension program pays
             monthly benefits to wartime veterans who have low incomes and are
             permanently and totally disabled for reasons not connected to their
             service. In compensation cases, the payment amount varies according to

             1
              The Results Act requires agencies to clearly define their missions, set goals, measure performance,
             and make improvements. Agencies are required to submit annual reports on their success in achieving
             program performance goals for the previous fiscal year. The first performance reports for fiscal year
             1999 are due by March 31, 2000.



             Page 3                                                GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




degree of disability; in pension cases, the amount varies according to
financial need. When veterans are unable to manage their affairs, benefit
payments are made to guardians who serve as their fiduciary
representatives.

Adjudicating an original disability claim involves two basic
functions—“authorization” and “disability rating.” Authorization involves
obtaining records from the military services and information from the
veterans, such as medical records and information on income and
dependents. Disability rating involves establishing whether a veteran’s
impairment is service-connected and, if so, evaluating the veteran’s degree
of disability. VBA considers claims requiring a disability rating to be the
core workload of the compensation and pension program, and as a group,
cases requiring a disability rating are considered to be the most
error-prone in the program. In order to rate (or evaluate) a veteran’s
disability, ROs often determine that they need medical evidence in addition
to evidence obtained from the veteran’s physicians and other medical
providers. In such cases, they send veterans to the Veterans Health
Administration (VHA) for physical or mental examinations by VHA
physicians.2

From the medical evidence, ROs rate a veteran’s disability using VA’s
Schedule for Rating Disabilities, which lists physical and mental
conditions and assigns a disability rating to each condition. Under this
schedule, the degree of disability is expressed in 10-percent increments up
to 100-percent disability. A veteran can also receive a “zero-percent”
disability rating, which means the veteran’s condition is service-connected
but not severe enough to qualify for compensation payments on the basis
of the medical criteria specified in the rating schedule. If the veteran’s
condition later worsens, he or she may reapply, asking VA to increase the
rating from zero to 10 percent or more.

Evaluating the degree of disability for some conditions, such as mental
impairments, can require adjudicators to make subjective judgments that
are not always clear-cut. For veterans with multiple impairments, the RO
must rate each impairment separately and then combine them into a
composite rating. After a veteran is placed on the rolls, his or her
condition or circumstances may change in ways that can result in
adjustments to the RO’s initial decision. For example, a veteran may file a
claim for an increase in degree of disability if his or her medical condition

2
 VA is currently conducting a pilot test to study the effectiveness of using private medical providers to
perform these examinations.



Page 4                                                  GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




deteriorates. Or nonmedical issues may arise that require an adjustment to
the initial decision but do not require a disability rating in order to make
the new decision. Such cases could arise from changes in the status of the
veteran’s dependents or changes in the income of a veteran receiving
pension benefits.

After the RO notifies the veteran of its decision, the veteran, if dissatisfied,
may ask for a hearing before an RO hearing officer. The veteran also may
file a notice of disagreement with the RO and then file an appeal asking for
a review of the RO’s decision by the Board of Veterans’ Appeals, which
makes VA’s final decisions on appeals on behalf of the Secretary. The
Board may conduct a hearing if the veteran requests one. In deciding
appeals, the Board can grant benefits (reverse the RO decision), deny
benefits (affirm the RO decision), or remand (or return) the case to the RO
to develop further evidence and reconsider the claim. After further
development of a remanded claim, the RO either awards the claim or
returns it to the Board for a decision.

Before 1989, the Board’s decisions on appeals were final. In that year,
however, the Court of Veterans Appeals—established by the Veterans’
Judicial Review Act of 1988 (P.L. 100-687, Nov. 18, 1988)—began to hear
cases. As a result, the Board is no longer the final step in the claims
adjudication process. When a veteran disagrees with a decision of the
Board, the veteran may now appeal to the Court, which is independent of
VA. Additionally, either veterans or VA may appeal decisions of the Court of
Veterans Appeals to the Court of Appeals for the Federal Circuit.

Since veterans began appealing Board decisions to the Court of Veterans
Appeals, according to a court official, the Court has remanded more than
4,500 decisions back to the Board for further development and
reconsideration. According to the same official, this represents about
59 percent of the Board’s decisions that were appealed to the Court,
excluding dismissed cases. In turn, ROs have felt the repercussions of these
Court decisions as evidenced by significant increases in the Board’s
reversals and remands of appealed RO decisions. Before the advent of the
Court, the Board historically had annually awarded benefits in 12 to
14 percent of appealed RO decisions and had annually remanded another
12 to 24 percent back to ROs for further development.3 However, in the
years since the advent of the Court, the Board has annually awarded
benefits in about 14 to 20 percent of the cases it reviewed and remanded

3
 Veterans file relatively few appeals with the Board. In fiscal year 1997, for example, they filed appeals
in only 5.4 percent of all RO initial decisions.



Page 5                                                  GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




another 31 to 51 percent back to ROs for further development. Despite
these increases in awards and remands by the Board, VBA had continued to
report—until STAR was implemented—that ROs were accurately processing
compensation and pension claims more than 95 percent of the time. (See
app. I for more details on the reversal and remand rates of the Court and
the Board and on the accuracy rates reported by VBA.)

VBA  considers a disability claim to have been accurately processed if basic
eligibility has been determined correctly, the case file contains all required
medical and nonmedical documentary evidence, the RO’s decision on
service-connection and the rating given to each impairment are correct,
the payment amount is correct, and the RO properly notified the veteran of
the outcome of his or her claim.

Under the accuracy measurement system that was in operation from fiscal
year 1992 through fiscal year 1997, VBA headquarters annually reviewed
approximately 100 cases randomly selected from the cases completed by
each of 57 ROs.4 These cases were selected from the entire universe of
compensation and pension work products completed by the ROs. Using
this procedure, VBA produced a national accuracy rate with a reasonable
level of statistical precision.5 While each year’s sample was too small for
VBA to produce accuracy rates for each RO with a reasonable level of
statistical precision, VBA required each RO to self-review 300 to 900 cases
annually, depending on the size of the RO. These RO self-reviews were to
provide ROs with information needed to improve quality, not to compute
accuracy rates for measuring performance.

VBA’s new accuracy measurement system, STAR, is part of a customer
service and benefits delivery improvement effort that involves, among
other things, the restructuring of VBA’s organization and accountability
systems. Under the restructuring, VBA has grouped the ROs into nine
service delivery networks (SDN). An SDN does not have a centralized office
or staff. Instead, the ROs in each SDN are expected to closely collaborate
with one another, provide mutual support, share resources, operate
according to team-based principles, and share collective responsibility and


4
 Although 58 ROs receive and process claims, the RO in Cheyenne, Wyoming, reports administratively
though the Denver RO; therefore, cases completed by the Cheyenne RO were included in the universe
of cases from which the Denver RO sample was selected.
55
  Statistical precision refers to the amount of uncertainty in an estimate that results from sampling
variability at a given level of confidence. For example, if a sample that has a 95-percent confidence
level and a precision level of plus or minus 5 percentage points yields an estimated accuracy rate of
70 percent, this means that one can be 95-percent confident that the true accuracy rate is between
65 percent and 75 percent.



Page 6                                                 GAO/HEHS-99-35 Veterans’ Benefits Claims
                       B-281315




                       accountability for the SDN’s overall performance of all work assigned to it.
                       In meeting the requirements of the Results Act, VBA headquarters will
                       measure each SDN’s performance, and each SDN will assess the
                       performance of its ROs. This measurement will be made on the basis of five
                       performance factors: claims-processing accuracy (as determined by STAR),
                       timeliness of claims processing, unit cost, customer satisfaction, and
                       employee satisfaction and development.


                       The new STAR system represents an important step forward by VBA in
VBA Has Improved Its   measuring the accuracy of compensation and pension claims processing
Measurement of         and in providing data to identify error-prone cases and correct the causes
Claims-Processing      of errors, including those that result in reversals and remands by the
                       Board of Veterans’ Appeals. Compared with the previous accuracy
Accuracy               measurement system that VBA had been using since 1992, the STAR system
                       is a step forward because it focuses more on RO decisions that are likely to
                       contain claims-processing errors, uses a stricter method for computing
                       accuracy rates, provides more data on the performance of additional
                       organizational levels within VBA, collects more data on errors, and stores
                       the results of more accuracy reviews in a centralized database for further
                       review and analysis.

                       Whereas VBA had been reporting more than 95-percent accuracy under the
                       previous accuracy measurement system, VBA, in its pilot test of STAR,
                       reported that only 64 percent of veterans’ claims were processed
                       accurately. A primary reason for this difference is that the pilot test
                       focused only on the most complex and more error-prone RO work
                       products, those involving disability rating decisions. By contrast, the
                       previous system drew its sample of cases from the entire universe of RO
                       work products, including those not requiring an assessment of disability
                       and, therefore, less error-prone. The newly implemented STAR system
                       continues to focus on claims that involve disability ratings, but it also
                       includes a sample of cases that address issues typically not requiring
                       disability ratings and a separate sample of cases involving guardianship
                       issues for veterans unable to represent themselves. Separate accuracy
                       rates are computed for each of these two other samples.

                       Another reason that the STAR pilot test found an accuracy rate of
                       64 percent rather than 95 percent as reported under the previous system is
                       STAR’s stricter accuracy rate computation method. Under the previous
                       system, VBA categorized each error into one of three areas of the claims
                       adjudication process: (1) case control and development, (2) decision



                       Page 7                                  GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




elements, and (3) notification to the veteran. Thus, if a case had one error,
VBA would record this error under the appropriate area and show the two
other areas as error-free. After reviewing all cases, VBA computed separate
accuracy rates for each of the three claims adjudication areas and then
determined an overall accuracy rate by calculating the arithmetic mean (or
average) of the three accuracy rates. Under STAR, however, VBA does not
compute separate accuracy rates for the three areas of the claims
adjudication process. If a case has any errors in any area of the claims
adjudication process, the entire case is counted as incorrect for accuracy
rate computation purposes. This approach tends to result in a lower
accuracy rate than under the previous system. (See app. II for a
hypothetical example demonstrating how STAR’s computation method can
result in a lower accuracy rate.)

In addition to focusing more on error-prone RO decisions and using a
stricter accuracy rate computation method, STAR provides accuracy rates
with reasonable statistical precision not only for the nation as a whole but
also for each SDN.6 Under the previous system, VBA headquarters had
reviewed about 5,700 cases annually. Its sampling methodology allowed
VBA to produce an accuracy rate with reasonable statistical precision for
the nation as a whole. Under STAR, VBA headquarters will review about
7,400 cases annually. Its sampling methodology will enable VBA to provide
accuracy rates with reasonable statistical precision for the nation and each
SDN for the sample of cases requiring disability ratings and the sample of
cases typically not requiring such ratings (see app. II for SDN sample sizes
and statistical precision data). However, the sample of cases involving
guardianship issues will be too small to provide the same level of
statistical precision.

VBA originally considered designing STAR so that VBA headquarters also
could produce accuracy rates for each RO but dropped this option because
it would have required VBA headquarters to review an additional 50,000
cases annually. Instead, VBA opted to require each RO to review samples of
its own work products using STAR review procedures. As in the
headquarters review, these RO self-reviews will produce accuracy rates
with reasonable statistical precision for the sample of cases requiring
disability ratings and the sample of cases typically not requiring such
ratings. However, the sample of cases involving guardianship issues will be
too small to produce accuracy rates with the same level of statistical
precision. Nationwide, the ROs will review about 44,000 randomly selected
cases (see app. II for RO sample sizes and statistical precision data). VBA

6
 See app. II for more information on statistical precision.



Page 8                                                   GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




estimates that every 1,000 cases in these samples require about 1.0
full-time equivalent review staff per year.

STAR is also an improvement over the previous accuracy measurement
system because it provides more precise information on the inaccuracies it
identifies. Under the previous system, VBA’s database essentially captured
only whether a decision did or did not contain errors. By contrast, STAR
requires reviewers to answer a standardized series of questions about
whether the RO’s actions and decisions were correct or incorrect in various
steps of claims processing. The reviewers enter their answers to these
questions, along with brief narrative comments, in the STAR database. In
addition, because the need for further development of evidence is a
primary reason that the Board of Veterans’ Appeals remands many cases
to ROs, STAR asks reviewers to identify deficient evidence categories, such
as private medical evidence, VA medical center records, and service
records. Also, because the Board remands many cases to ROs to obtain
further medical examinations by VHA physicians, STAR asks reviewers to
indicate whether deficiencies in medical evidence supporting the decision
relate to VHA medical examinations. These data on deficiencies in evidence
are entered in the STAR database. The database also identifies cases
involving five special conditions that have medical implications: prisoner
of war, radiation exposure, Gulf War veteran, Agent Orange exposure, and
posttraumatic stress syndrome.

Additionally, STAR’s database captures the results of accuracy reviews
conducted by both VBA headquarters and the ROs, whereas under the
previous system, VBA’s database captured only the results of accuracy
reviews conducted by VBA headquarters. VBA planned to implement in
February 1999 a new centralized database on its internal network
(intranet) system that will permit both VBA headquarters and the ROs to
input the results of all STAR reviews into the database. Capturing RO data
will enrich the data available to analyze error trends, and both VBA
headquarters and the ROs will have access to the full complement of data
through the intranet.




Page 9                                  GAO/HEHS-99-35 Veterans’ Benefits Claims
                          B-281315




                          Although STAR represents a significant step forward in VBA’s ability to
VBA Can Strengthen        measure claims-processing accuracy and identify error-prone cases, VBA
Its Ability to Identify   can take further steps to improve this ability. These steps involve
Error-Prone Cases         collecting additional data for identifying and correcting error-prone cases
                          and addressing vulnerabilities in the integrity of accuracy reviews.
and Address
Vulnerabilities in the
Integrity of Accuracy
Reviews
VBA Can Improve the       Even with the improvements provided by STAR, VBA’s ability to identify
Ability to Identify       error-prone cases and target corrective actions is constrained by the
Error-Prone Cases         limited data that it captures on the medical characteristics of claimants
                          whose claims are processed incorrectly and on why medical evidence is
                          deficient. Data captured on claimants’ medical characteristics is currently
                          limited to identifying whether a veteran was a prisoner of war, served in
                          the Gulf War, or had posttraumatic stress syndrome, radiation exposure,
                          or Agent Orange exposure. More detailed medical characteristics data
                          could help pinpoint the specific types of claims in which errors occur.
                          Also, although STAR captures data on whether medical evidence and
                          medical examinations are adequate, it does not record statistical data
                          identifying why reviewers found the evidence or examinations supporting
                          RO final decisions to be deficient. Such data also could help pinpoint the
                          types of corrective actions that need to be taken to improve the accuracy
                          of RO decisions.

                          Limited studies by VBA demonstrate how capturing additional data in the
                          STAR database on medical issues could help VBA focus on corrective actions
                          that can reduce claims-processing errors and in turn reduce remands from
                          the Board of Veterans’ Appeals. In 1996, VBA and the Board of Veterans’
                          Appeals jointly conducted a limited study of remanded cases and reported
                          that inadequate medical examinations were the most frequent reason for
                          remands and that a majority of the remanded cases involved the need for
                          specialty examinations, such as orthopedic, psychiatric, neurologic,
                          audiologic, and ear-nose-throat examinations. Also, in 1996, the Milwaukee
                          RO reviewed claims that were awarded by the RO’s hearing officers after
                          the claims were initially denied. Of the cases in which the RO’s hearing
                          officers reversed the initial decision, the Milwaukee RO captured data on
                          the specific conditions, such as orthopedic impairments, that were
                          involved in significant numbers of cases, and using such data, the RO
                          identified specific corrective actions. According to Milwaukee RO officials,




                          Page 10                                 GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




this helped reduce the RO’s remand rate from the Board of Veterans’
Appeals. From fiscal year 1995 to fiscal year 1998, the Milwaukee RO
reduced its remand rate from about 40 percent to about 21 percent, one of
the lowest remand rates in the nation.7

SSA, which administers the largest federal disability program, has a quality
assurance system that captures detailed data on claimants’ medical
characteristics and on weaknesses in evidence. SSA has found that such
data are helpful in identifying error-prone cases and targeting corrective
actions. For each case reviewed, SSA’s system captures data on the specific
body systems involved, such as musculoskeletal, respiratory,
cardiovascular, and mental systems. Further, using codes from the
International Classification of Diseases, SSA’s system identifies each
claimant’s specific impairments. Additionally, when medical evidence is
judged not adequate, SSA’s system records the specific medical specialty
area in which evidence was lacking, such as orthopedic, psychiatric, and
neurologic areas, and it identifies the specific type of test, study, or other
medical evidence that was lacking.

Such data, according to an SSA quality assurance official, not only helps to
identify error-prone cases but can pinpoint specific evidentiary
weaknesses for cases involving specific body systems or impairments.
Also, this official stated that spending resources up front to capture such
data can reduce the need to conduct time-consuming special studies later
to understand why certain types of cases are being processed incorrectly.
According to the SSA quality assurance unit, the depth of the data collected
from quality assurance reviews also enables it to assess the
implementation of new or revised policies, perform analyses and make
recommendations for operational and systems corrective actions, and
provide broad levels of management information, such as information by
categories of impairments.

VBA agrees that the STAR system deployed at the beginning of fiscal year
1999 provides a sound start for beginning to address claims-processing
accuracy issues. VBA officials acknowledge, however, that they realized
when STAR was deployed that continuous improvement should be sought
to enhance its effectiveness. These VBA officials stated that VBA is open to
considering the collection of additional data in order to enhance STAR.




7
 Remand rates for the 58 ROs ranged from about 19 percent to 59 percent in fiscal year 1998.



Page 11                                               GAO/HEHS-99-35 Veterans’ Benefits Claims
                           B-281315




VBA Can Further Address    To ensure integrity in the operation of government programs, standards
Vulnerabilities in the     for internal controls call for separation of key duties, and standards for
Integrity of Performance   performance audits call for those who review and evaluate a program’s
                           performance to be organizationally independent of the program’s
Data                       managers.8 Under STAR, however, the RO staff who review the accuracy of
                           RO decisions are themselves responsible for making such decisions, and
                           they report to RO managers responsible for claims processing. Such an
                           arrangement does not meet the standard for separation of duties, nor does
                           it meet the independence standard. Both the RO reviewers and their
                           managers have an inherent self-interest in having as high an accuracy rate
                           as possible. This self-interest derives from the fact that accuracy is one of
                           five factors that determine RO performance scores, which VBA measures to
                           meet Results Act requirements. Thus, without adequate separation of
                           duties or adequate independence for RO reviewers, the integrity of both the
                           STAR review process and the resulting accuracy rates and performance
                           data reported under the Results Act are called into question.

                           The potential effect of impaired objectivity on performance data is
                           exemplified by findings reported by VA’s Inspector General in
                           September 1998.9 Because of concern about the accuracy of data used to
                           meet Results Act requirements, the Inspector General examined the
                           integrity of certain data used for Results Act reports. In this review, the
                           Inspector General found instances in which RO staff had manipulated data
                           on the timeliness of RO claims processing in order to make performance
                           appear to be better than it actually was. The Inspector General found that
                           weaknesses in internal controls had contributed to the lack of integrity in
                           the timeliness data reported under the Results Act. During our review,
                           some RO staff made comments on the integrity of accuracy reviews that
                           parallel the findings of the Inspector General. These RO staff told us that
                           ROs are biased against identifying their own errors. They also stated that
                           ROs in the past, after selecting samples of cases to review, had sometimes
                           “sanitized” or fixed problems in the case files before the cases underwent
                           quality review.

                           No data are available to indicate the extent to which RO reviewers might
                           attempt to overlook errors and sanitize case files to conceal errors in the
                           approximately 44,000 cases that ROs review annually under STAR. However,


                           8
                            See U.S. General Accounting Office, Standards for Internal Controls in the Federal Government
                           (Washington, D.C.: 1983), and Government Auditing Standards (Washington, D.C.: June 1994).
                           9
                            See Department of Veterans Affairs, Office of Inspector General, Audit of Data Integrity for Veterans
                           Claims Processing Performance Measures Used for Reports Required by the Government Performance
                           and Results Act, Report No. 8R5-B01-147 (Washington, D.C.: Sept. 22, 1998).



                           Page 12                                               GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




to the extent that such efforts may occur, the accuracy rates reported by
the ROs would be overstated. Furthermore, any attempts by ROs to conceal
errors and overstate their accuracy rates could also result in an
overstatement of the accuracy rates that VBA reports for SDNs and the
nation. This vulnerability in VBA’s data exists because the sample of 7,400
cases that VBA reviews annually is selected directly from the approximately
44,000 cases reviewed by the ROs. VBA reviews its sample of 7,400 cases
after the ROs complete their own reviews of these same cases. VBA believes
that it can detect most attempts to sanitize case files because such
attempts would likely require extensive backdating of corrected case file
documents, which VBA believes would be difficult to conceal. VBA
acknowledges, however, that it cannot ensure that it would detect every
such attempt in the cases it reviews. To the extent that VBA may not detect
all such attempts, the accuracy rates it reports for SDNs and the nation
would be overstated.

Ensuring the integrity of accuracy data will require that staff who review
claims-processing accuracy neither are responsible for claims processing
nor report to program managers responsible for claims processing. VBA
stated that resource restrictions prevent establishing independent
accuracy review units either in the ROs or at VA headquarters; however,
unless VBA provides adequate separation of duties and organizational
independence for accuracy reviewers, potential questions about the
integrity of accuracy-related performance data will likely persist. By
contrast, we found that SSA has quality assurance units at its headquarters
and in each of its 10 regional offices that are organizationally independent
of program management. The independent quality assurance unit in SSA
headquarters has overall responsibility for assessing disability
claims-processing accuracy. To do this, it oversees the operation of the
independent regional quality assurance units that review the accuracy of
statistically random samples of the disability decisions rendered by 54
state agencies that process disability claims for SSA.

VBA contends that it would be impractical to establish independent
accuracy review units in VBA’s 58 ROs, many of which are relatively small in
size. Establishing independent STAR units in ROs would be more practical if
only a relatively small number of large ROs processed all compensation
and pension claims. Under the present structure, however, a more
workable long-term solution could involve establishing an independent
headquarters unit responsible for conducting all reviews used to
determine the accuracy rates that go into the calculation of overall
performance scores for VBA headquarters, SDNs, and ROs.



Page 13                                  GAO/HEHS-99-35 Veterans’ Benefits Claims
                            B-281315




                            VBA has set a goal of achieving a claims-processing accuracy rate of
VBA Faces Challenges        93 percent by fiscal year 2004. This would be almost 30 percentage points
in Meeting Its Goal for     higher than the baseline rate of 64 percent established in the 1998 pilot test
Improving                   of STAR. VBA acknowledges, however, that the STAR system on its own
                            cannot ensure that VBA will meet its accuracy goal. Beyond any
Claims-Processing           improvements that VBA might make in the STAR system, VBA acknowledges
Accuracy                    that there are challenges it must address successfully in order to meet its
                            goal for improving accuracy. These challenges include effectively
                            establishing accountability for accuracy improvement and developing an
                            effective training program for the current and future workforce.


Establishing Stricter       In May 1998, VBA identified several root causes of quality problems in
Accountability              processing disability compensation and pension claims.10 One such cause
                            was a lack of employee accountability. VBA plans to focus on quality and
                            accountability with a quality assurance system that provides clear and fair
                            accountability at all organizational levels. To accomplish this goal, VBA is
                            implementing the “balanced scorecard” that scores the performance of VBA
                            headquarters, SDNs, and ROs on the basis of five performance factors:
                            claims-processing accuracy (as determined by STAR), timeliness of claims
                            processing, unit cost, customer satisfaction, and employee satisfaction and
                            development.

                            With the goal of achieving a 93-percent accuracy rate by fiscal year 2004,
                            VBA believes its balanced scorecard approach will, among other things,
                            drive organizational change, provide employees with feedback on
                            measures they can influence, and link the performance appraisal and
                            reward systems to organizational performance measures. VBA plans to use
                            the balanced scorecard to give RO managers incentives to work as teams in
                            their SDNs with a focus on meeting balanced scorecard performance
                            measures, including accuracy. The extent to which this strategy will
                            improve accountability and accuracy cannot yet be determined.


Developing More Effective   In our discussions with RO staff, many stated that VBA had not provided
Training                    adequate training for claims adjudicators. They stated, for example, that
                            there was confusion in the ROs on how to process cases because of
                            apparent conflicts between decisions of the Court of Veterans Appeals and
                            VA’s regulations and guidance. They also stated that too much of their
                            training was determined locally, resulting in inconsistent training among
                            the ROs. VBA acknowledged shortcomings in training and stated that it had

                            10
                              See VBA, Roadmap to Excellence—Planning the Journey (Washington, D.C.: May 1998).



                            Page 14                                            GAO/HEHS-99-35 Veterans’ Benefits Claims
                  B-281315




                  not fared well in preparing its workforce, with a resultant decline in
                  technical accuracy. VBA acknowledged the need for an effective,
                  centralized, and comprehensive training program that provides the
                  background necessary for its decisionmakers to render decisions
                  according to the statutes and regulations mandated for claims
                  adjudication.

                  Such training is important not only for current employees but also for the
                  many new employees whom VBA will have to hire to replace retiring
                  employees. According to VBA, it may lose up to 30 percent of its workforce
                  to retirement by fiscal year 2003. To develop a training program for RO
                  staff, VBA plans to identify the necessary employee skills and work
                  processes for every decisionmaking position, implement skill certification
                  or credentialing for these positions, and implement performance-based
                  training connected to measurable outcomes. VBA has already developed a
                  computer-based training module for processing appeals and is working on
                  modules for original disability claims, service-connected death indemnity
                  benefits, and pensions. VBA also plans to produce additional modules,
                  including one for training RO staff when they first assume disability rating
                  responsibilities. Whether these training efforts will enable VBA to meet its
                  accuracy goal cannot yet be determined.


                  Although VBA had been reporting until recently that ROs were processing
Conclusions and   claims accurately more than 95 percent of the time, the STAR pilot test in
Recommendations   fiscal year 1998 revealed that the accuracy rate for decisions involving
                  disability ratings was much lower, about 64 percent.11 This confirmed that
                  VBA needs to give more attention to ensuring that ROs make the correct
                  decision the first time. Making the correct decision the first time would
                  mean that veterans could avoid having to make unnecessary appeals and
                  would not be unnecessarily delayed in receiving benefits owed them.

                  Although the new STAR system represents genuine improvement in VBA’s
                  ability to measure accuracy and identify error-prone cases, VBA needs to
                  make further progress in collecting data for identifying difficult cases,
                  assessing adjudication difficulties, and developing corrective actions.
                  Despite its newly implemented STAR system, without further refinements in
                  the data collected on errors, significant inaccuracies are likely to persist
                  because VBA is constrained in its ability to pinpoint error-prone cases and
                  identify corrective actions. Moreover, the data produced from STAR

                  11
                   As mentioned, the lower accuracy rate under STAR is partially attributable to the fact that STAR
                  computes accuracy rates more strictly than the previous system, thereby tending to produce lower
                  accuracy rates (see app. II).



                  Page 15                                              GAO/HEHS-99-35 Veterans’ Benefits Claims
                         B-281315




                         reviews will be suspect because of weaknesses in internal controls and
                         lack of adherence to performance audit standards. We believe this can
                         potentially undermine progress made under STAR.

                         To further strengthen VBA’s ability to identify error-prone cases, ensure the
                         integrity of accuracy rate-related performance data reported under the
                         Results Act, and keep the Congress informed about VBA’s progress in
                         addressing challenges that must be met in order to improve accuracy, we
                         recommend that the Secretary of the Department of Veterans Affairs direct
                         the Under Secretary for Benefits to take the following actions.

                     •   For RO disability decisions found to be in error, revise STAR to collect more
                         detailed medical characteristics data, such as the human body systems, the
                         specific impairments, and the specific deficiencies in medical evidence
                         involved in these disability claims, so that VA can identify and focus
                         corrective actions on specific problems that RO adjudicators have in
                         correctly evaluating certain types of medical conditions or in correctly
                         determining whether medical evidence is adequate to make a decision.
                     •   Implement a claims-processing accuracy review function that meets the
                         government’s internal control standard on separation of duties and the
                         program performance audit standard on organizational independence.
                     •   In the annual Results Act reports, inform the Congress on VBA’s progress in
                         (1) establishing stricter employee accountability for the achievement of
                         performance goals and (2) developing more effective training for claims
                         adjudicators.


                         In commenting on our draft report, VA stated that it found the report to be
Agency Comments          a fair and balanced appraisal. VA concurred that its process for assessing
and Our Evaluation       claims accuracy is critical and stated that continued urgent action is
                         required for VA to meet its own and its stakeholders’ expectations. VA
                         stated that our recommendations were generally constructive but had
                         concern about our first two recommendations.

                         The first recommendation in our draft report was that VA “revise STAR to
                         include the collection of more detailed medical characteristics data on the
                         human body systems, and specific impairments involved in disability
                         claims as well as data on specific deficiencies in medical evidence and
                         examinations.” VA interpreted our recommendation to mean that STAR
                         should collect data on the quality of examinations conducted by VHA.
                         However, this was not the intent of our recommendation. The intent was
                         for STAR to collect additional data that would help VA better identify



                         Page 16                                 GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




(1) specific types of medical conditions that RO adjudicators have difficulty
evaluating correctly and (2) specific types of inadequacies in medical
evidence that are most prevalent in incorrect decisions. This would
provide a means for VA to develop corrective actions addressing the causes
of errors in the evaluation of medical conditions and of failure to collect
adequate medical evidence to make a supportable decision. We clarified
the recommendation and our discussion of this issue in our report.

The second recommendation in our draft report was that VA “implement a
claim processing accuracy review function that meets the government’s
internal control standard on separation of duties and the program
performance audit standard on organizational independence.” VA’s primary
concern about this recommendation was that current budget constraints
make it impractical to adopt approaches that would fully satisfy these
standards—for example, establishing a single, large centralized review unit
to assess all quality issues, including individual RO quality. However, while
current budget constraints may present problems in finding ways to fully
meet the standards immediately, we believe meeting these standards as
expeditiously as possible should be a continuing priority in VA’s future
planning process. Until the standards are met, the integrity of VA’s
claims-processing accuracy data will remain questionable. As VA stated in
its comments, “Effective reviews require an organizational commitment to
dedicate the necessary resources to the review process.”

With regard to the second recommendation, VA also stated that while the
STAR system is a compromise reflecting resource constraints, it has some
distinct advantages compared with quality reviews performed by a
consolidated, independent review unit. VA cited the value of having
reviews performed by local staff in each RO. Our recommendation would
not preclude local reviews, which we agree are important. Even if a single,
central unit were established for the purpose of assessing the degree to
which each RO processes claims accurately, it would still be critical for
local RO management to gather detailed local data on claims processing to
understand fully how to correct local processing problems. This function,
however, is different from local reviewers conducting accuracy reviews of
their own RO’s decisions, which our recommendation is intended to
eliminate.

VA also stated that it is concerned that a “permanent” independent review
staff would become stagnant. We disagree because the staff who perform
reviews would not have to be permanently assigned to the unit but could
instead be rotated to avert staff stagnation. VA furthermore expressed



Page 17                                  GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




concern about the cost and increased potential for losing active case files
that would result from mailing many more thousands of case files from the
58 ROs to a central review site. This concern, however, does not negate the
need to meet the standards for separation of duties and organizational
independence. Also, the concern could potentially be lessened by other
measures. For example, the Congressional Commission on
Servicemembers and Veterans Transition Assistance in its January 1999
report applauded VBA for consolidating the administration of its education
and loan programs into fewer than 10 ROs but pointed out that VBA has
made no effort to make a similar consolidation of the adjudication of
compensation claims.12 If VBA were ever to consolidate the adjudication of
claims into a few relatively large ROs, it would be more practical to locate
an independent STAR unit in each of these ROs to review the accuracy of
cases each one processed. Each RO STAR unit would then need to mail to a
central review unit only a relatively small random sample of the cases it
reviewed so that the central unit could ensure the reviews’
appropriateness and consistency.

VA’s   comments are printed in appendix III.




12
  The Commission was established by title VII of the Veterans’ Benefits Improvement Act of 1996 (P.L.
104-275) to examine a broad range of federal programs that provide transition assistance and benefits
to service members when they leave military service and to veterans.



Page 18                                               GAO/HEHS-99-35 Veterans’ Benefits Claims
B-281315




As agreed with your office, we plan no further distribution of this report
until 7 days from its date of issue, unless you publicly announce its
contents earlier. We will then send copies to the Chairman of the House
Committee on Veterans’ Affairs, the Secretary of the Department of
Veterans Affairs, other congressional committees, and others who are
interested. We will also make copies available to others upon request. If
you have any questions about this report, please call me at (202) 512-7101
or Irene P. Chu, Assistant Director, at (202) 512-7102. Other major
contributors to this report were Ira B. Spears, Mark Trapani, Connie D.
Wilson, Paul C. Wright, and Deborah L. Edwards.

Sincerely yours,




Stephen P. Backhus
Director, Veterans’ Affairs and
  Military Health Care Issues




Page 19                                 GAO/HEHS-99-35 Veterans’ Benefits Claims
Contents



Letter                                                                                             1


Appendix I                                                                                        22
Historical Data on
VBA’s
Claims-Processing
Accuracy Rates and
on Award and
Remand Rates of the
Board of Veterans’
Appeals
Appendix II                                                                                       24
                       Sample Sizes                                                               24
Sample Sizes and       Accuracy Rate Computation Methods                                          25
Accuracy Rate
Computation Methods
for Compensation and
Pension Cases
Appendix III                                                                                      27
Comments From the
Department of
Veterans Affairs
Tables                 Table I.1: RO National Accuracy Rates for Disability                       22
                         Compensation and Pension Claims, Fiscal Years 1993-97
                       Table I.2: Decisions by Board of Veterans’ Appeals Resulting in            22
                         Awards or Remands of Appealed Disability Compensation and
                         Pension Cases, Fiscal Years 1993-97
                       Table I.3: Disability Compensation Claim Decisions Appealed by             23
                         Veterans, Fiscal Year 1997
                       Table II.1: Annual Sample Sizes Under VBA’s STAR Accuracy                  25
                         Measurement System




                       Page 20                               GAO/HEHS-99-35 Veterans’ Benefits Claims
Contents




Table II.2: Hypothetical Computation of Accuracy Rates Under                 26
  the Pre-STAR Accuracy Measurement System




Abbreviations

ICD        International Classification of Diseases
RO         regional office
SDN        service delivery network
SSA        Social Security Administration
STAR       Systematic Technical Accuracy Review
VA         Department of Veterans Affairs
VBA        Veterans Benefits Administration
VHA        Veterans Health Administration


Page 21                                 GAO/HEHS-99-35 Veterans’ Benefits Claims
Appendix I

Historical Data on VBA’s Claims-Processing
Accuracy Rates and on Award and Remand
Rates of the Board of Veterans’ Appeals
                                         Before the Veterans Benefits Administration (VBA) implemented the
                                         Systematic Technical Accuracy Review (STAR) measurement system, it
                                         reported that regional offices (RO) accurately processed and adjudicated
                                         disability compensation and disability pension claims more than
                                         95 percent of the time during fiscal years 1993-97 (see table I.1).

Table I.1: RO National Accuracy Rates
for Disability Compensation and                                 Proper control
Pension Claims, Fiscal Years 1993-97                                      and                                   Proper
                                                                 development Correctness of             notification to       Overall RO
                                         Fiscal year                of claims  RO decisions                   veterans      accuracy rate
                                         1993                               97.5%               94.5%              93.4%               95.2%
                                         1994                               97.7                95.6               95.2                96.2
                                         1995                               97.2                95.4               95.9                96.2
                                         1996                               97.5                96.5               96.8                96.9
                                         1997                               98.0                96.6               96.9                97.2
                                         Source: VBA.



                                         The validity of such high accuracy rates, however, seemed inconsistent
                                         with the results of decisions made by the Board of Veterans’ Appeals when
                                         veterans appealed unfavorable RO decisions. The Board of Veterans’
                                         Appeals awarded benefits or remanded cases for further development
                                         more than 60 percent of the time when veterans appealed RO decisions
                                         during fiscal years 1993-97 (see table I.2).


Table I.2: Decisions by Board of Veterans’ Appeals Resulting in Awards or Remands of Appealed Disability Compensation
and Pension Cases, Fiscal Years 1993-97
                        Total disability         Total awards by Board                    Total remands by Board
                  decisions made by                                  Percent of total                                     Percent of total
Fiscal year                   Board                Number                 decisions                     Number                 decisions
1993                          22,924                    4,026                      17.6                   10,350                       45.1
1994                          19,343                    3,474                      18.0                    9,583                       49.5
1995                          24,834                    4,921                      19.8                   12,073                       48.6
1996                          29,818                    6,137                      20.6                   13,357                       44.8
1997                          37,936                    6,627                      17.5                   17,783                       46.9
                                         Source: Annual reports of the Chairman of the Board of Veterans’ Appeals, fiscal years 1993-97.



                                         Only a small proportion of RO decisions are appealed to the Board. For
                                         example, in fiscal year 1997, veterans filed notices of disagreement in
                                         about 14 percent of the disability compensation claims processed by ROs



                                         Page 22                                              GAO/HEHS-99-35 Veterans’ Benefits Claims
                                     Appendix I
                                     Historical Data on VBA’s Claims-Processing
                                     Accuracy Rates and on Award and Remand
                                     Rates of the Board of Veterans’ Appeals




                                     (see table I.3). The number of cases appealed, however, is less than the
                                     number of cases in which veterans file a notice of disagreement with VA. In
                                     some cases, after notices of disagreement are filed, ROs award the benefits
                                     sought, or some veterans decide not to continue with their appeals if the
                                     RO again denies benefits at this point.13 In fiscal year 1997, the Board
                                     received initial substantive appeals equivalent to about 5 percent of all
                                     disability compensation claims processed by ROs.

Table I.3: Disability Compensation
Claim Decisions Appealed by                                                                                  Number of       Percent of
Veterans, Fiscal Year 1997           Type of action                                                             cases           claims
                                     Disability compensation claims processed by ROs (original
                                     and reopened)                                                               486,425         100.0
                                     Notices of disagreement filed with ROs                                       66,566           13.7
                                     Initial substantive appeals filed requesting Board of Veterans’
                                     Appeals review                                                               26,033            5.4
                                     Source: VBA.




                                     13
                                       VBA does not maintain data on the number of these cases for which benefits are awarded.



                                     Page 23                                              GAO/HEHS-99-35 Veterans’ Benefits Claims
Appendix II

Sample Sizes and Accuracy Rate
Computation Methods for Compensation
and Pension Cases
               Under the pre-STAR accuracy measurement system, VBA annually reviewed
Sample Sizes   approximately 5,700 compensation and pension cases, or approximately
               100 cases randomly selected from the cases completed by each of 57 ROs.14
               These cases were selected from the entire universe of compensation and
               pension work products completed by the ROs. Using this procedure, VBA
               annually produced, with a reasonable level of statistical precision, a
               national accuracy rate for the entire body of compensation and pension
               work done by the ROs during the prior year. The sample of approximately
               100 cases selected for each RO was too small to produce accuracy rates for
               each RO with a reasonable level of statistical precision. However, VBA
               required each RO to self-review a sample of 300 to 900 cases annually,
               depending on the size of the RO. These RO self-reviews were intended to
               provide the RO with information needed to improve quality, not to compute
               accuracy rates for VBA to measure performance.

               Under STAR, VBA annually reviews 7,371 compensation and pension cases
               for the nine service delivery networks (SDN), and the 57 ROs self-review
               about 44,000 cases. These cases are made up of three separate samples:
               (1) rating-related work products; (2) authorization work products that
               require significant development, review, and administrative decision or
               award action but may not involve any rating-related action; and
               (3) principal guardianship files, referred to as fiduciary cases. (See table
               II.1 for SDN and RO sample sizes.) For rating-related work products and
               authorization work products that typically do not require rating-related
               action, the sampling methodology will allow VBA to produce accuracy rates
               with a reasonable level of statistical precision for the nation and each SDN.
               However, the sample of fiduciary cases is too small to provide accuracy
               rates with the same level of statistical precision. Similarly, for cases that
               are self-reviewed by ROs, the sampling methodology will allow each RO to
               produce accuracy rates with a reasonable level of statistical precision for
               rating-related work products and authorization work products typically
               not requiring ratings. Again, however, the sample of fiduciary cases is too
               small to provide accuracy rates with the same level of statistical precision.




               14
                 Although 58 ROs receive and process claims, the RO in Cheyenne, Wyoming, reports administratively
               through the Denver RO; therefore, cases completed by the Cheyenne RO are included in the universe
               of cases from which the Denver RO sample is selected.



               Page 24                                             GAO/HEHS-99-35 Veterans’ Benefits Claims
                                        Appendix II
                                        Sample Sizes and Accuracy Rate
                                        Computation Methods for Compensation
                                        and Pension Cases




Table II.1: Annual Sample Sizes Under VBA’s STAR Accuracy Measurement System
                            Samples reviewed by VBA headquarters for each
                                             of 9 SDNs                     Samples self-reviewed by each of 57 ROs
Types of cases reviewed     Sample size Statistical precisiona                        Sample sizeb Statistical precisiona
Total for eachc                     819 Not applicable                                      645-855 Not applicable
Cases requiring a rating            354 +/-5 percentage points                              300-352 +/-5 percentage points
Cases typically not                 325 +/-5 percentage points                              285-323 +/-5 percentage points
requiring a rating
Fiduciary cases                     140 +/-8 percentage points                                60-180 +/-7 to +/-13 percentage pointsd
              e
National total                     7,371 Not applicable                                       44,175 Not applicable
Cases requiring a rating           3,186 +/-2 percentage points                               19,388 Not applicable
Cases typically not                2,925 +/-2 percentage points                               17,947 Not applicable
requiring a rating
Fiduciary cases                    1,260 +/-3 percentage points                                6,840 Not applicable
                                        a
                                         Statistical precision refers to the amount of uncertainty in an estimate that results from sampling
                                        variability at a given level of confidence. These precision levels were calculated at the 95-percent
                                        confidence level. For example, an estimated accuracy rate of 70 percent at a precision level of
                                        plus or minus 5 percentage points means that one is 95-percent confident that the true accuracy
                                        rate is between 65 percent and 75 percent.
                                        b
                                         The range in sample sizes stems from the varying size of caseloads among ROs. The ROs with
                                        the smallest caseloads, for example, have the smallest sample size to review.
                                        c
                                        The totals in this row represent total sample size for each SDN and each RO.
                                        d
                                         Precision is dependent on sample size. Sampling errors range from plus or minus 7 percentage
                                        points for the sample of 180 cases to plus or minus 13 percentage points for the sample of 60.
                                        e
                                        The totals in this row represent national total sample size for 9 SDNs and 57 ROs.

                                        Source: VBA.




                                        For each case reviewed under the previous accuracy measurement system,
Accuracy Rate                           VBA categorized each error into one of three areas of the claims
Computation Methods                     adjudication process: (1) control and development of the claim,
                                        (2) decision elements, and (3) notification to the veteran. Thus, for
                                        example, if a case had only one error, VBA would record this error under
                                        the appropriate area of the claims adjudication process and would show
                                        the two other areas as error-free for that case. After all cases were
                                        reviewed, VBA would compute an accuracy rate for each of the three areas
                                        in the claims adjudication process. To arrive at an overall accuracy rate for
                                        the three areas combined, VBA computed their arithmetic mean (or
                                        average). For example, table II.2 shows a hypothetical outcome for
                                        accuracy reviews of 10 cases. Under the control and development area,




                                        Page 25                                                GAO/HEHS-99-35 Veterans’ Benefits Claims
                                          Appendix II
                                          Sample Sizes and Accuracy Rate
                                          Computation Methods for Compensation
                                          and Pension Cases




                                          one case has an error (a 90-percent accuracy rate); under the decision
                                          element area, two cases have errors (an 80-percent accuracy rate); and
                                          under the notification area, one case has an error (a 90-percent accuracy
                                          rate). For this sample of 10 cases as a whole, the overall accuracy rate is
                                          the average of these three accuracy rates, or 86.6 percent.

Table II.2: Hypothetical Computation of
Accuracy Rates Under the Pre-STAR                                                    Errors found in accuracy review
Accuracy Measurement System                                            Area 1: Control                          Area 3:
                                                                       and development       Area 2: Decision   Notification to
                                          Hypothetical case            of claim              elements           veteran
                                          1                            X                     X
                                          2
                                          3
                                          4
                                          5
                                          6                                                  X
                                          7
                                          8
                                          9
                                          10                                                                    X
                                          Total cases                  1                     2                  1
                                          with errors
                                          Accuracy rate for sample     9/10 = 90%            8/10 = 80%         9/10 = 90%
                                          of 10 casesa
                                          a
                                          Overall average accuracy rate: (90% + 80% + 90%) / 3 = 86.6%.



                                          For each case reviewed under STAR, however, VBA does not compute
                                          separate accuracy rates for the three areas of the claims adjudication
                                          process. If a case has any errors in any area of the claims adjudication
                                          process, the entire case is counted as incorrect for accuracy rate
                                          computation purposes. This approach tends to result in a lower accuracy
                                          rate than under the previous system. For example, in the hypothetical
                                          sample of 10 cases shown in table II.2, 3 cases would be counted as
                                          incorrect under STAR because they contain at least one processing error,
                                          and the resultant accuracy rate for the sample would be only 70 percent (7
                                          out of 10 cases with no errors = 70-percent accuracy rate), compared with
                                          the overall accuracy rate of 86.6 percent calculated under the previous
                                          system.




                                          Page 26                                          GAO/HEHS-99-35 Veterans’ Benefits Claims
Appendix III

Comments From the Department of
Veterans Affairs




               Page 27     GAO/HEHS-99-35 Veterans’ Benefits Claims
Appendix III
Comments From the Department of
Veterans Affairs




Page 28                           GAO/HEHS-99-35 Veterans’ Benefits Claims
Appendix III
Comments From the Department of
Veterans Affairs




Page 29                           GAO/HEHS-99-35 Veterans’ Benefits Claims
Appendix III
Comments From the Department of
Veterans Affairs




Page 30                           GAO/HEHS-99-35 Veterans’ Benefits Claims
           Appendix III
           Comments From the Department of
           Veterans Affairs




(105757)   Page 31                           GAO/HEHS-99-35 Veterans’ Benefits Claims
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested