oversight

VA Health Care: Reliability of Reported Outpatient Medical Appointment Wait Times and Scheduling Oversight Need Improvement

Published by the Government Accountability Office on 2012-12-21.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Requesters




                VA HEALTH CARE
December 2012




                Reliability of Reported
                Outpatient Medical
                Appointment Wait
                Times and Scheduling
                Oversight Need
                Improvement




GAO-13-130
                                            December 2012

                                            VA HEALTH CARE
                                            Reliability of Reported Outpatient Medical
                                            Appointment Wait Times and Scheduling Oversight
                                            Need Improvement
Highlights of GAO-13-130, a report to
congressional requesters




Why GAO Did This Study                      What GAO Found
VHA provided nearly 80 million              Outpatient medical appointment wait times reported by the Veterans Health
outpatient medical appointments to          Administration (VHA), within the Department of Veterans Affairs (VA), are
veterans in fiscal year 2011. While         unreliable. Wait times for outpatient medical appointments—referred to as
VHA has reported continued                  medical appointments—are calculated as the number of days elapsed from the
improvements in achieving access to         desired date, which is defined as the date on which the patient or health care
timely medical appointments, patient        provider wants the patient to be seen. The reliability of reported wait time
complaints and media reports about          performance measures is dependent on the consistency with which schedulers
long wait times persist.                    record the desired date in the scheduling system in accordance with VHA’s
GAO was asked to evaluate VHA’s             scheduling policy. However, VHA’s scheduling policy and training documents for
scheduling of timely medical                recording desired date are unclear and do not ensure consistent use of the
appointments. GAO examined (1) the          desired date. Some schedulers at Veterans Affairs medical centers (VAMC) that
extent to which VHA’s approach for          GAO visited did not record the desired date correctly. For example, three
measuring and monitoring medical            schedulers changed the desired date based on appointment availability; this
appointment wait times reflects how         would have resulted in a reported wait time that was shorter than the patient
long veterans are waiting for               actually experienced. VHA officials acknowledged limitations of measuring wait
appointments; (2) the extent to which       times based on desired date, and described additional information used to
VAMCs are implementing VHA’s                monitor veterans’ access to medical appointments, including patient satisfaction
policies and processes for appointment      survey results. Without reliable measurement of how long patients are waiting for
scheduling, and any problems                medical appointments, however, VHA is less equipped to identify areas that need
encountered in ensuring veterans’
                                            improvement and mitigate problems that contribute to wait times.
access to timely medical appointments;
and (3) VHA’s initiatives to improve        While visiting VAMCs, GAO also found inconsistent implementation of VHA’s
veterans’ access to medical                 scheduling policy that impedes VAMCs from scheduling timely medical
appointments. To conduct this work,         appointments. For example, four clinics across three VAMCs did not use the
GAO made site visits to 23 clinics at       electronic wait list to track new patients that needed medical appointments as
four VAMCs, the latter selected for         required by VHA scheduling policy, putting these clinics at risk for losing track of
variation in size, complexity, and          these patients. Furthermore, VAMCs’ oversight of compliance with VHA’s
location. GAO also reviewed VHA’s           scheduling policy, such as ensuring the completion of required scheduler
policies and data, and interviewed VHA
                                            training, was inconsistent across facilities. VAMCs also described other problems
officials.
                                            with scheduling timely medical appointments, including VHA’s outdated and
What GAO Recommends                         inefficient scheduling system, gaps in scheduler and provider staffing, and issues
                                            with telephone access. For example, officials at all VAMCs GAO visited reported
GAO recommends that VHA take                that high call volumes and a lack of staff dedicated to answering the telephones
actions to (1) improve the reliability of   impede scheduling of timely medical appointments. In January 2012, VHA
its medical appointment wait time           distributed telephone access best practices that, if implemented, could help
measures, (2) ensure VAMCs
                                            improve telephone access to clinical care.
consistently implement VHA’s
scheduling policy, (3) require VAMCs        VHA is implementing a number of initiatives to improve veterans’ access to
to allocate staffing resources based on     medical appointments such as expanded use of technology to interact with
scheduling needs, and (4) ensure that       patients and provide care, which includes the use of secure messaging between
VAMCs provide oversight of telephone        patients and their health care providers. VHA also is piloting a new initiative to
access and implement best practices         provide health care services through contracts with community providers that
to improve telephone access for             aims to reduce travel and wait times for veterans who are unable to receive
clinical care. VA concurred with GAO’s      certain types of care within VHA in a timely way.
recommendations.

View GAO-13-130. For more information,
contact Debra Draper at (202) 512-7114 or
draperd@gao.gov.

                                                                                     United States Government Accountability Office
Contents


Letter                                                                                            1
                       Background                                                                 5
                       Reported Medical Appointment Wait Times Are Unreliable; VHA
                         Uses Additional Methods to Monitor Patients’ Access to Medical
                         Appointments                                                           11
                       Inconsistent Implementation of VHA’s Scheduling Policy and Other
                         Problems Impede VAMCs’ Ability to Schedule Timely Medical
                         Appointments                                                           19
                       VHA Is Implementing a Number of Initiatives to Improve Access to
                         Medical Appointments                                                   29
                       Conclusions                                                              34
                       Recommendations for Executive Action                                     35
                       Agency Comments and Our Evaluation                                       35

Appendix I             Comments from the Department of Veterans Affairs                         41



Appendix II            GAO Contact and Staff Acknowledgments                                    45



Related GAO Products                                                                            46



Tables
                       Table 1: Selected Fiscal Year 2012 VHA Medical Appointment Wait
                                Time Performance Measures                                       10
                       Table 2: Number of Schedulers at Each VAMC Visited Who
                                Incorrectly Recorded the Medical Appointment Desired
                                Date, by Error Type                                             15
                       Table 3: Number of Clinics That Did Not Implement Selected
                                Elements of the VHA’s Scheduling Policy, by VAMC Visited        20




                       Page i                                GAO-13-130 VA Scheduling and Wait Times
Abbreviations

CBOC                       community-based outpatient clinic
FDPP                       Facility Director Performance Plan
NDPP                       Network Director Performance Plan
OIG                        Office of Inspector General
PACT                       Patient-Aligned Care Team
PAR                        Performance and Accountability Report
Project ARCH               Access Reached Closer to Home
VA                         Department of Veterans Affairs
VAMC                       VA Medical Center
VHA                        Veterans Health Administration
VISN                       Veterans Integrated Service Network
VistA                      Veterans Health Information Systems and
                            Technology Architecture



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                                         GAO-13-130 VA Scheduling and Wait Times
United States Government Accountability Office
Washington, DC 20548




                                   December 21, 2012

                                   Congressional Requesters

                                   The Veterans Health Administration (VHA), within the Department of
                                   Veterans Affairs (VA), operates one of the nation’s largest health care
                                   systems. Its medical facilities include 152 VA medical centers (VAMC), as
                                   well as more than 800 community-based outpatient clinics (CBOC) and
                                   ambulatory care centers that operate under supervision of the VAMCs. 1
                                   At these facilities, VHA provides outpatient care through primary and
                                   specialty care clinics. 2 In fiscal year 2011, there were more than 8 million
                                   veterans enrolled in VHA’s health system, and VHA provided nearly
                                   80 million outpatient medical appointments to veterans for primary and
                                   specialty care. 3

                                   Although access to timely medical appointments is critical to ensuring that
                                   veterans obtain needed medical care, long wait times and inadequate
                                   scheduling processes at VHA medical facilities have been long-standing
                                   problems. For example, in 2001, GAO reviewed the timeliness of medical
                                   appointments and found that two-thirds of the specialty care clinics visited
                                   had wait times longer than 30 days, although some clinics had made
                                   progress in reducing wait times, primarily by improving their scheduling
                                   processes and making better use of their staff. 4 Later, in 2007, the VA


                                   1
                                    Generally a CBOC or ambulatory care center is defined as a fixed health care site that is
                                   geographically distinct or separate from its parent VAMC. All CBOCs and ambulatory care
                                   centers generally provide primary care, and some may provide specialty care; services
                                   can vary by individual CBOC or ambulatory care center.
                                   2
                                    Outpatient clinics offer services to patients that do not require a hospital stay. Primary
                                   care addresses patients’ routine health needs and specialty care is focused on a specific
                                   specialty service such as orthopedics or dermatology. A “clinic” may be defined as an
                                   entity for dividing provider workload and scheduling different types of patient care
                                   appointments. A particular area of care, such as primary care or specialty care may have
                                   multiple clinics that vary in purpose and size. A VAMC can provide care in each area at
                                   the VAMC or its affiliated CBOCs and ambulatory care centers; for example, primary care
                                   could be provided through multiple primary care clinics at a VAMC’s different locations.
                                   Throughout the report we refer to a specific area of care at a specific location as a “clinic.”
                                   3
                                    Throughout the report we will use the term “medical appointments” to refer to outpatient
                                   medical appointments.
                                   4
                                    GAO, VA Health Care: More National Action Needed to Reduce Waiting Times, but
                                   Some Clinics Have Made Progress, GAO-01-953 (Washington, D.C.: Aug. 31, 2001).




                                   Page 1                                             GAO-13-130 VA Scheduling and Wait Times
Office of Inspector General (OIG) reported that VHA facilities did not
always follow VHA’s scheduling policies and processes and that the
accuracy of VHA’s reported wait times for medical appointments was
unreliable. 5 Most recently, in 2012, the VA OIG reported that VHA was
not providing all new veterans with timely access to full mental health
evaluations, and had overstated its success in providing veterans with
timely new and follow-up appointments for mental health treatment. 6

VHA has reported continued improvements in measuring and achieving
timely access to medical appointments. For example, in fiscal year 2011,
VHA had a goal of scheduling medical appointments within 14 days of the
patient’s or provider’s desired medical appointment date, 7 and in that
year, VA reported that it completed 95 percent of specialty care medical
appointments and 94 percent of primary care medical appointments
within this time frame. However, patient complaints and media reports
about long wait times have persisted, prompting renewed concerns about
excessive medical appointment wait times. You asked us to evaluate
VHA’s scheduling of timely medical appointments. We examined (1) the
extent to which VHA’s approach for measuring and monitoring medical
appointment wait times reflects how long veterans are waiting; (2) the
extent to which VAMCs are implementing VHA’s policies and processes
for medical appointment scheduling, and any problems encountered in
ensuring veterans’ access to timely medical appointments as identified by
VAMCs; and (3) VHA’s initiatives to improve veterans’ access to medical
appointments.

To address all three objectives, we interviewed VHA central office officials
responsible for medical appointment scheduling policy, medical
appointment wait time measurement, and initiatives to improve access to
timely medical appointments. We also conducted site visits to four
VAMCs selected for variation in size and complexity, geographic location,



5
 Department of Veterans Affairs, Office of Inspector General, Audit of the Veterans Health
Administration’s Outpatient Waiting Times, Report No. 07-00616-199 (Washington, D.C.:
Sept. 10, 2007).
6
 Department of Veterans Affairs, Office of Inspector General, Veterans Health
Administration: Review of Veterans’ Access to Mental Health Care, Report No. 12-00900-
168 (Washington, D.C.: Apr. 23, 2012).
7
 According to VHA’s scheduling policy, the desired appointment date, referred to as the
“desired date,” is the date on which the patient or provider wants the patient to be seen.




Page 2                                           GAO-13-130 VA Scheduling and Wait Times
or role as a pilot site for a VHA initiative to improve access to timely
medical appointments. 8 These four VAMCs were located in Dayton, Ohio;
Fort Harrison, Montana; Los Angeles, California; and Washington, D.C. At
each site, we visited the VAMC as well as the affiliated CBOCs that had
among the highest volume of medical appointments. 9 We also visited the
highest volume ambulatory care center at one VAMC that had such
facilities. 10 At each VAMC, CBOC, and ambulatory care center, we visited
an outpatient primary care clinic, and where available, one or more
outpatient specialty care clinics for a total of 23 clinics—9 primary care
and 14 specialty care clinics—the clinics were among those with the
highest medical appointment volume. 11 Results from our site visits cannot
be generalized to other VAMCs. We also interviewed the directors and
relevant staff of the four Veterans Integrated Service Networks (VISN), or
regional networks of care, for the sites we visited. 12

To examine the extent to which VHA’s approach for measuring and
monitoring medical appointment wait times reflects how long veterans are
waiting, we reviewed VHA’s outpatient medical appointment scheduling
policy and processes and training documents based on the policy, as well
as documents related to performance accountability that include wait time
measures. 13 At each of the four VAMCs we visited, we interviewed the
leadership team, scheduling managers, and managers from all of our


8
 VA assigns each VAMC a complexity score derived from multiple variables to measure
facility complexity arrayed along four categories, namely patient population served, clinical
services offered, education and research complexity, and administrative complexity.
9
 We visited the highest volume CBOC for three VAMCs and the second highest volume
for the fourth VAMC.
10
  CBOCs and ambulatory care centers provide outpatient primary care and may provide
specialty care services. Oversight for facility functions—including scheduling—occurs at
the VAMC level. CBOCs and ambulatory care centers generally are geographically distinct
from, but operate under the supervision of a parent VAMC, which maintains administrative
responsibility.
11
  During our site visits, we visited only outpatient clinics. Some CBOCs did not have
specialty care clinics. From this point forward, we use VAMC to refer collectively to the
VAMC and all of its affiliated CBOCs and ambulatory care centers.
12
 Each of VA’s 21 VISNs is responsible for managing and overseeing medical facilities
within a defined geographic area.
13
  VHA outpatient medical appointment scheduling policy is documented in VHA Directive
2010-027, VHA Outpatient Scheduling Processes and Procedures (June 9, 2010). We
refer to the directive as “VHA’s scheduling policy” from this point forward.




Page 3                                           GAO-13-130 VA Scheduling and Wait Times
selected clinics about oversight activities to ensure the accuracy of
scheduling data, and about what data and measures they use to manage
clinics and improve medical appointment timeliness. In addition, we
interviewed schedulers from the 9 primary care clinics and 10 of the
14 specialty care clinics we visited using a structured protocol to
determine the accuracy with which schedulers determined and recorded
medical appointment data for hypothetical patient medical appointments
into the scheduling system in accordance with VHA’s scheduling policy. 14
We also compared reported medical appointment wait times for the clinics
we visited to information gathered during our site visits. 15

To determine the extent to which VAMCs are implementing VHA’s
scheduling policy and processes, and to gather information on problems
encountered in scheduling timely medical appointments, we reviewed
VHA’s scheduling policy, interviewed VHA central office officials
responsible for the scheduling policy, and obtained information about
scheduling practices from officials at each of the four VAMCs we visited.
Specifically, for each of the VAMCs, we interviewed leadership,
scheduling managers, clinic managers, patient advocates, and case
managers. We obtained information about the VAMCs’ oversight to
ensure compliance with policy, and about problems staff at these facilities
say they experience in scheduling timely medical appointments. We
reviewed the implementation of selected elements of VHA’s scheduling
policy at both the VAMC and individual clinic level. These elements
included the use of VHA’s scheduling software to schedule medical
appointments, the use of the electronic wait list 16 for tracking patients new
to a clinic that are waiting for medical appointments, and the use of the
recall/reminder software to facilitate reminders for patients that need to
return to the clinic for follow-up medical appointments. We also obtained
from VHA and reviewed VAMCs’ fiscal year 2011 certifications of
compliance with VHA’s scheduling policy—a required annual self-


14
  We refer to clerical or administrative support staff with scheduling responsibilities as
“schedulers.” We did not complete a scheduler interview for 4 of the 14 specialty clinics
that we visited because either the clinic did not have a scheduler or it would have caused
delays in patient care to take a scheduler off duty for an interview.
15
  VHA Support Services Center maintains an internal VHA website that allows central
office-, VISN-, and VAMC-level staff to access reports on a variety of topics including
medical appointment data.
16
  The electronic wait list is a type of computer software application designed for recording,
tracking, and reporting veterans waiting for medical appointments.




Page 4                                           GAO-13-130 VA Scheduling and Wait Times
                          certification—as well as documentation of scheduler training completion
                          obtained from the four VAMCs visited. Finally, we reviewed patient
                          complaints about telephone responsiveness collected by each VAMC’s
                          Office of the Patient Advocate.

                          To examine VHA’s initiatives to improve veterans’ access to timely
                          medical appointments, we interviewed VHA central office officials to
                          obtain information about selected initiatives and reviewed relevant VHA
                          documents outlining these initiatives. We also interviewed officials at the
                          VAMCs we visited about the implementation of these initiatives and
                          officials at the Billings Clinic, a non-VA health care facility involved in the
                          implementation of one of the initiatives. 17

                          We conducted this performance audit from February 2012 to December
                          2012 in accordance with generally accepted government auditing
                          standards. Those standards require that we plan and perform the audit to
                          obtain sufficient, appropriate evidence to provide a reasonable basis for
                          our findings and conclusions based on our audit objectives. We believe
                          that the evidence obtained provides a reasonable basis for our findings
                          and conclusions based on our audit objectives.


                          VHA’s health care system is geographically divided into 21 VISNs, each
Background                of which is headed by a VISN director. Each VISN is comprised of a
                          network of VAMCs, and the VISN office serves as the basic budgetary
                          and decision-making unit for providing health care services to veterans
                          within that geographical area. Each VAMC and its affiliated CBOCs and
                          ambulatory care centers are headed by a VAMC director who manages
                          administrative functions, and a chief of staff who manages clinical
                          functions for these facilities. VHA’s Central Office establishes system-
                          wide scheduling policy.


VHA Medical Appointment   VHA’s scheduling policy establishes processes and procedures for
Scheduling Policy         scheduling medical appointments, and for ensuring the competency of
                          staff directly or indirectly involved in the scheduling process. This policy is
                          designed to help VAMCs meet VHA’s commitment to scheduling medical



                          17
                            VHA uses non-VA care to reduce wait times and backlogs and to provide veterans
                          access to specialists not available through VHA.




                          Page 5                                       GAO-13-130 VA Scheduling and Wait Times
appointments with no undue waits or delays. Specifically, VHA’s
scheduling policy includes, but is not limited to, the following
requirements:

•    Requires VAMCs to use VHA’s Veterans Health Information Systems
     and Technology Architecture (VistA) medical appointment scheduling
     system to schedule medical appointments. 18

•    Requires VAMCs to keep appointment schedules open and available
     for patients to make medical appointments at least 3 to 4 months into
     the future.

•    Requires schedulers to record in the VistA scheduling system the date
     on which the patient or provider wants the patient to be seen as the
     desired date. To determine the desired date, schedulers should be in
     communication with the patient when scheduling the medical
     appointment.

•    Requires schedulers to record the desired date correctly and
     describes how to determine and record the desired date for new
     patients—patients who haven’t been seen by a health care provider in
     a clinic within the past 2 years, including those scheduled in response
     to a consult request—as well as specifying how to determine the
     desired date for established patients’ follow-up medical
     appointments—patients who have been seen within the past two
     years. 19

•    Requires VAMCs to track new patients waiting for medical
     appointments using the electronic wait list within VistA and to remind
     established patients of follow-up medical appointments using the
     recall/reminder software within VistA, which enables clinics to create a
     list of established patients who need follow-up medical appointments
     more than 3 or 4 months in the future.




18
  From this point forward, the VistA medical appointment scheduling system will be
referred to as the VistA scheduling system.
19
  Consults—generally requests for specialty care appointments—are most often
communicated electronically through an application in the electronic medical record within
VistA.




Page 6                                          GAO-13-130 VA Scheduling and Wait Times
Additionally, VHA has a separate directive that establishes policy on the
provision of telephone service related to clinical care, including facilitating
telephone access for medical appointment management. 20

Officials at the VHA central office, VISN, and VAMC all have oversight
responsibilities for the implementation of VHA’s scheduling policy. In the
VHA central office, the Director of Systems Redesign, through the Office
of the Deputy Undersecretary for Health for Operations and Management,
is responsible for the oversight and implementation of medical
appointment scheduling requirements. This oversight includes
measurement and monitoring of ongoing performance. Each VISN
director, or designee, is responsible for oversight of enrollment and
medical appointment scheduling for eligible veterans. 21 Each VAMC
director, or designee, is responsible for ensuring that clinics’ scheduling of
medical appointments complies with VHA’s scheduling policy, including
clinics in affiliated CBOCs and ambulatory care centers. In addition, the
VAMC director is responsible for ensuring that any staff who can
schedule medical appointments in the VistA scheduling system has
completed VHA scheduler training. 22

Starting in fiscal year 2007, VHA required every VAMC to annually self-
certify compliance with VHA’s scheduling policy. 23 This certification is
signed by the VAMC director and also encompasses scheduling
compliance in affiliated CBOCs and ambulatory care centers. For fiscal
year 2011, the certification required VAMCs to self-certify compliance,
partial compliance, or noncompliance with more than 30 individual



20
 VHA Directive 2007-033, Telephone Service for Clinical Care (Oct. 11, 2007).
21
  To obtain VHA healthcare services, veterans generally must enroll with VHA and
register at a specific VAMC.
22
  Specifically, VAMCs are required to maintain a list of all staff who can schedule medical
appointments in the VistA scheduling system and VAMC directors are required to ensure
successful completion of required training by all staff on the list. Schedulers are not to be
allowed to schedule medical appointments in the VistA scheduling system without proof of
their successful completion of this training.
23
  For fiscal year 2011, the most recent certification available at the time of our review,
VHA Systems Redesign collected VAMC directors’ certification through a web-based
template. As part of the certification, VAMC directors certify that they have completed,
using VISN-approved processes and procedures, a yearly standardized audit of
schedulers on the timeliness and appropriateness of scheduling actions and the accuracy
of desired dates.




Page 7                                            GAO-13-130 VA Scheduling and Wait Times
                        aspects of VHA’s scheduling policy as well as overall compliance, partial
                        compliance, or noncompliance with VHA’s scheduling policy as a whole.
                        According to officials, VHA’s central office does not penalize
                        noncompliance with the certification and expects oversight to be
                        managed locally. VHA’s central office uses this certification of compliance
                        as a tool for VAMCs to identify and improve performance on important
                        aspects of the policy.


VistA Information       VistA is the single integrated health information system used throughout
Technology System       VHA in all of its health care settings. There are many different VistA
                        applications for clinical, administrative, and financial functions, including
                        VHA’s electronic medical record, known as the Computerized Patient
                        Record System, and the scheduling system. As we reported in May 2010,
                        the VistA scheduling system is more than 25 years old and inefficient in
                        facilitating care coordination between different sites. 24 In 2000, VHA
                        began an initiative to modernize the scheduling system, but VA
                        terminated the project in 2009. We also reported that VA’s efforts to
                        successfully replace the scheduling system were hindered by
                        weaknesses in its project management processes and lack of effective
                        oversight.


Wait Time Measurement   In 1995, VHA established a goal of scheduling primary and specialty care
and Performance         medical appointments within 30 days to ensure veterans’ timely access to
                        care. 25 In fiscal year 2011, VHA shortened the wait time goal to 14 days
                        for both primary and specialty care medical appointments based on
                        improved performance reported in previous years. Specifically, VA’s
                        reported wait times for fiscal year 2010 showed that nearly all primary
                        care and specialty care medical appointments were scheduled within
                        30 days of desired date. In fiscal year 2012, VHA added a goal of
                        completing primary care medical appointments within 7 days of the
                        desired date.



                        24
                          GAO, Information Technology: Management Improvements Are Essential to VA’s
                        Second Effort to Replace Its Outpatient Scheduling System, GAO-10-579 (Washington,
                        D.C.: May 27, 2010).
                        25
                          VHA also has a goal of scheduling compensation and pension examinations within
                        30 days. Compensation and pension examinations may be provided to veterans to
                        establish a claim for disability compensation; appointment wait times for these
                        appointments are outside the scope of this report.




                        Page 8                                       GAO-13-130 VA Scheduling and Wait Times
To facilitate accountability for achieving its wait time goals, VHA includes
wait time measures—referred to as performance measures—in its VISN
and VAMC directors’ performance contracts known as Network Director
Performance Plans (NDPP) and Facility Director Performance Plans
(FDPP), respectively. 26 Wait time performance measures also are
included in VA’s budget submissions and performance reports to
Congress and stakeholders; the performance reports are published
annually in VA’s Performance and Accountability Report (PAR). 27
However, the medical appointment wait time performance measures
included in the NDPPs and FDPPs differ from the measures that are
reported in the PAR. (See table 1.) For example, in fiscal year 2012,
VHA’s wait time goal of 7 days for primary care medical appointments
was reflected in the NDPP and FDPP performance measures, but the
fiscal year 2012 PAR reported primary care wait time performance using
a 14-day standard. 28 The performance measures have also changed over
time. 29




26
  Directors’ performance contracts include measures against which directors are rated at
the end of the fiscal year, which determine their performance pay. The contracts include
system-wide performance measures, as well as individualized performance measures that
are selected based on specific problems or needs of the respective VISN or VAMC.
27
  VA prepares a congressional budget justification that provides details supporting the
policy and funding decisions in the President’s budget request submitted to Congress prior
to the beginning of each fiscal year. The budget justification articulates what VA plans to
achieve with the resources requested; it includes performance measures by program
area. VA also publishes an annual PAR, which contains performance targets and results
achieved against those targets in the previous year.
28
  VHA officials told us the department is working to better coordinate consistency of the
performance measures.
29
  For example, the fiscal year 2011 NDPP and FDPP included a measure “percent of
patients waiting for a primary care appointment longer than 14 days from the desired date”
instead of the primary care measures included in fiscal year 2012.The fiscal year 2010
NDPP and FDPP included the measure “percent of patients waiting for a primary care
appointment longer than 30 days from the desired date.” In addition, the fiscal year 2011
PAR included three rather than four wait time measures that did not break out the new
and established patients for primary and specialty care; the fiscal year 2012 PAR included
separate measures for new and established patients. The fiscal year 2010 PAR also
included three measures, one of which measured the “percent of new patient
appointments completed within 30 days of the appointment create date.”




Page 9                                           GAO-13-130 VA Scheduling and Wait Times
                                       At the time of our review, all of VHA’s medical appointment wait time
                                       performance measures reflected the number of days elapsed from the
                                       patient’s or provider’s desired date, which is recorded in the VistA
                                       scheduling system by VAMCs’ schedulers. According to VHA central
                                       office officials, VHA measures wait times based on desired date in order
                                       to capture the patient’s experience waiting and to reflect the patient’s or
                                       provider’s wishes; which is not reflected by other available wait time
                                       measures.

Table 1: Selected Fiscal Year 2012 VHA Medical Appointment Wait Time Performance Measures

Performance plan or report             Wait time performance measure
Network Director’s Performance Plans   Percent of patients waiting for a specialty care appointment longer than 14 days from the
                                                   b
and Facility Director’s Performance    desired date
                       a
Plans (NDPP, FDPP)                     Percent of primary care appointments completed within 7 days of the desired date
                                                                                                                                   c


                                       Same day access with primary care provider (percent of requested same day primary care
                                                                             c
                                       appointments completed within one day)
VA Budget Submission and Performance   Percent of new patient primary care appointments completed within 14 days of the desired
                                d          e
and Accountability Report (PAR)        date
                                       Percent of established patient primary care appointments completed within 14 days of the
                                       desired date
                                       Percent of new patient specialty care appointments completed within 14 days of the
                                                   e
                                       desired date
                                       Percent of established patient specialty care appointments completed within 14 days of
                                       the desired date
                                       Source: VA.

                                       Notes: New and established patient appointments, and primary care and specialty care appointments
                                       refer to outpatient medical appointments.
                                       a
                                        The NDPP and FDPP are the network (VISN) and facility (VAMC) directors’ contracts that include
                                       performance measures against which directors are rated at the end of the fiscal year, and are
                                       monitored throughout the year.
                                       b
                                        For fiscal year 2012, the specialty care medical appointment wait time measure represents the
                                       percent of patients waiting for at least one appointment longer than 14 days from the desired date at
                                       a given point in time. The measure was collected twice a month and VISN and VAMC directors were
                                       rated against the average of these scores at the end of the year. The plans also included additional
                                       performance measures specifically for mental health appointment timeliness that are outside the
                                       scope of this report.
                                       c
                                        For fiscal year 2012, the plans included five individual measures related to primary care and a sixth
                                       measure that was a composite of the five individual measures. Two of the five individual primary care
                                       measures pertained to primary care medical appointment wait times; these two wait time measures
                                       represent the percent of appointments completed in the specified time frame. The three other
                                       individual measures were percentage of total encounters that occur by telephone (telephone
                                       utilization), percentage of primary care appointments with patient’s assigned primary care provider
                                       (continuity of care), and percentage of patients discharged from hospital who were contacted by their
                                       primary care provider within 2 days (post-hospital discharge contact). At the end of the year, VISN
                                       and VAMC directors were scored for each of the five individual primary care measures and rated
                                       against the single composite measure. They were rated as meeting the composite measure if they
                                       met targets for at least three of the five individual primary care measures.




                                       Page 10                                               GAO-13-130 VA Scheduling and Wait Times
                              d
                               VA prepares a congressional budget justification that provides details supporting the policy and
                              funding decisions in the President’s budget request that is submitted to Congress prior to the
                              beginning of each fiscal year. VA also publishes an annual PAR, which contains performance targets
                              and results achieved against those targets in the previous year. For fiscal year 2012, the four PAR
                              wait time measures represent the percent of each type of appointment completed within 14 days of
                              the desired date. The cumulative year-to-date scores were reported on the PAR.
                              e
                               Generally, for wait time measurement purposes, VA defines new patients as those who have not
                              been seen in a particular clinic at that facility within the last 2 years. This includes appointments for
                              newly enrolled patients as well as those scheduled in response to a consult request.



                              Medical appointment wait times used for measuring and assessing
Reported Medical              performance toward VHA’s wait time goals are unreliable due to problems
Appointment Wait              with recording the appointment desired date in the VistA scheduling
                              system. Acknowledging limitations of the wait time measures, VHA uses
Times Are Unreliable;         additional information to monitor patients’ access to medical
VHA Uses Additional           appointments.
Methods to Monitor
Patients’ Access to
Medical
Appointments
Reported Medical              VHA measures its medical appointment wait times as the number of days
Appointment Wait Times        that have elapsed from the patient’s or provider’s desired date.
Derived from Desired Date     Consequently, the reliability of reported wait time performance is
                              dependent on the consistency with which schedulers record the desired
Are Unreliable                date in the VistA scheduling system. However, aspects of VHA’s
                              scheduling policy and training documents regarding how to determine and
                              record the desired date are unclear and do not ensure replicable and
                              reliable use of the desired date. In addition, we found that some
                              schedulers at select VAMCs did not correctly implement other aspects of
                              VHA’s scheduling policy for recording the desired date.

Unclear Desired Date Policy   Aspects of VHA’s scheduling policy and related training documents on
                              how to determine and record the desired date are unclear and do not
                              ensure replicable and reliable recording of the desired date by the large
                              number of staff across VHA who can schedule medical appointments in
                              the VistA scheduling system. 30 Specifically, VHA’s scheduling policy and



                              30
                                According to a VHA official, there were more than 50,000 staff across VHA who could
                              schedule appointments at the time of our review.




                              Page 11                                                  GAO-13-130 VA Scheduling and Wait Times
related scheduler training documents do not provide consistent guidance
about when or whether the desired date should be based on the patient’s
or provider’s preference. While the policy defines desired date as “the
date on which the patient or provider wants the patient to be seen,” it also
instructs that the “the desired date needs to be defined by the patient” for
new patient medical appointments, medical appointments scheduled in
response to consult requests, and established patient follow-up medical
appointments. When there is a conflict between the provider and patient
desired date, the scheduler is instructed to contact the provider for a
decision on the return time frame, but the policy and training documents
do not clearly describe under what circumstances the provider’s date
should be used as the desired date. Further, providers may designate a
desired appointment time frame for a follow-up medical appointment
rather than a specific date; in such cases, the policy is unclear as to
which date within the provider’s designated time frame the scheduler
should enter as the desired date. The scheduling policy and training do
not provide sufficient guidance to ensure consistent use of desired date in
these various scheduling scenarios.

VHA central office officials responsible for developing VHA’s scheduling
policy and related training documents told us that the desired date is
intentionally broad to account for all of the scheduling scenarios that may
exist. However, leadership officials from the four VAMCs we visited and
their corresponding VISNs reported problems with the unclear guidance
on the desired date definition, and difficulties achieving consistent and
correct use of the desired date by their schedulers. In addition, given the
ambiguity in the scheduling policy and related training documents, there
are different interpretations of the desired date between officials at
different levels. For example, a VISN director stated that if a provider
gives a desired time frame, the scheduler is to use the earliest date in that
range as the desired date; whereas a provider in a specialty care clinic at
the VAMC we visited within that VISN stated that the clinic uses the latest
date in the range to meet the 14-day specialty care medical appointment
scheduling goal.

Additionally, when presented with various scheduling scenarios,
schedulers at the VAMCs we visited determined and recorded the desired
date differently. For example, when posed with the question “What date
do you enter into the scheduling system as the desired date for an
established patient follow-up medical appointment?”, 12 schedulers said
they would enter the patient’s desired date, 4 said the provider’s date,
and the remaining 3 said they used the next available medical
appointment date. When posed with the question “If the patient’s stated


Page 12                                  GAO-13-130 VA Scheduling and Wait Times
                              desired date conflicts with the provider’s designated desired date or time
                              frame, what date do you enter as the desired date?”, 1 scheduler said
                              that the patient’s desired date would be entered, while another said the
                              desired date has to come from the provider. The variation in schedulers’
                              interpretation of the desired date suggests confusion about its correct use
                              in different scheduling scenarios.

Errors in Recording Desired   Although unclear about when to use the patient’s or provider’s desired
Date                          date, VHA’s scheduling policy clearly instructs that, in all circumstances,
                              the desired date should be defined without regard to schedule capacity,
                              and should not be altered once established to reflect a medical
                              appointment date the patient accepts because of lack of medical
                              appointment availability on the desired date. However, we found that at
                              least one scheduler from each of the VAMCs we visited did not correctly
                              implement these aspects of the policy when recording the desired date in
                              the VistA scheduling system for specific hypothetical scheduling
                              situations. 31 As summarized in table 2, we identified the following three
                              types of errors, each of which would have resulted in desired dates that
                              did not accurately reflect the patients’ or providers’ desired date, as well
                              as potentially result in the reporting of more favorable wait times for those
                              medical appointments. 32

                              •    Determined appointment availability prior to establishing desired date:
                                   Although VHA’s scheduling policy requires schedulers to establish the
                                   desired date for a medical appointment without regard to the schedule
                                   capacity, four schedulers from three VAMCs determined the clinic’s
                                   next available medical appointment dates before establishing a
                                   desired date. Therefore, reported wait times for these appointments
                                   may not have accurately reflected how long patients actually waited.

                              •    Altered original desired date based on appointment availability: Three
                                   schedulers from two VAMCs established a desired date that was
                                   recorded in the VistA scheduling system independent of schedule
                                   capacity, but later altered the desired date because of appointment


                              31
                                Because the policy and training documents are unclear about when the desired date is
                              defined by the patient or defined by the provider, we only identified errors related to
                              aspects of the policy and training regarding how to determine and record the desired date
                              that hold true despite the ambiguity.
                              32
                                For reporting on the PAR, VHA measures medical appointment wait times as the
                              number of days between the desired date and appointment date.




                              Page 13                                         GAO-13-130 VA Scheduling and Wait Times
    availability. Specifically, two of the three schedulers altered the
    originally established desired date to match the agreed-upon
    appointment date, which would have incorrectly resulted in no wait
    time reported for the appointment. The third scheduler altered the
    established desired date when there was no appointment availability
    within 2 weeks of that date; which would have resulted in an
    incorrectly reported wait time that was shorter than the patient actually
    waited from his or her original desired date.

•   Recorded a new desired date when rescheduling appointment:
    Additionally, eight schedulers from three VAMCs incorrectly recorded
    a new desired date when rescheduling an appointment cancelled by
    the clinic rather than keeping the original desired date as required by
    VHA’s scheduling policy. Changing the desired date in this way would
    incorrectly decrease the reported wait times for the rescheduled
    appointments; veterans actually would wait longer than the reported
    wait times indicated.




Page 14                                  GAO-13-130 VA Scheduling and Wait Times
Table 2: Number of Schedulers at Each VAMC Visited Who Incorrectly Recorded the Medical Appointment Desired Date, by
Error Type

                                                             Number of schedulers who demonstrated error
                                                           (number of schedulers interviewed at each VAMC)
                                                                                                                                     Total
                                                                                                                           schedulers who
Desired date recording error                               VAMC A (3)           VAMC B (7)      VAMC C (4)   VAMC D (5) demonstrated error
Determined medical appointment availability prior                          1                2           1             0                        4
                            a
to establishing desired date
Altered original desired date based on medical                             0                2           1             0                        3
                         a
appointment availability
Recorded a new desired date when rescheduling a                            0                3           3             2                        8
                   b
medical appointment
Total number of errors                                                     1                7           5             2                       15
Total number of schedulers who demonstrated
at least one error                                                         1                6           3             2                       12
                                            Source: GAO analysis of scheduler interviews.

                                            Notes: This table presents the results of our interviews with 19 schedulers from the four VAMCs
                                            visited. We used structured questions to test how schedulers would determine and enter the desired
                                            date for specific medical appointment types using hypothetical patients. We identified three types of
                                            desired date scheduling errors which, for actual patient medical appointments, would have resulted in
                                            a desired date that would not accurately reflect the patient or provider’s desired date, as well as
                                            potentially result in reporting of more favorable wait times for those medical appointments. A
                                            scheduler could have demonstrated more than one error.
                                            a
                                             According to VHA’s scheduling policy, when scheduling new patient medical appointments—
                                            including medical appointments in response to a consult request—and established patient follow-up
                                            medical appointments, the medical appointment desired date needs to be defined by the patient
                                            without regard to schedule capacity. Once the patient’s desired date has been established, it must not
                                            be altered to reflect a medical appointment date patients agree to accept due to lack of medical
                                            appointment availability on the original desired date.
                                            b
                                             According to VHA’s scheduling policy, when a medical appointment is cancelled and rescheduled by
                                            the clinic, the scheduler must record as the desired date for the new medical appointment, the
                                            desired date for the original medical appointment. Three of the 19 schedulers did not respond to
                                            questions about rescheduling medical appointments.


                                            During our site visits, staff at some clinics told us they change medical
                                            appointment desired dates to show clinic wait times within VHA’s
                                            performance goals. A scheduler at one primary care clinic specifically
                                            stated that she changes the recorded desired date to the patient’s
                                            agreed-upon appointment date in order to show shorter wait times for the
                                            clinic. A provider at a specialty care clinic at another VAMC said providers
                                            in that clinic change the desired dates of their follow-up appointments if a
                                            patient cannot be scheduled within the 14-day performance goal.

                                            In addition, the reported wait times, derived from desired date, for one of
                                            the specialty care clinics we visited were inconsistent with the VAMC’s
                                            account of appointment scheduling backlogs and scheduling challenges,



                                            Page 15                                                   GAO-13-130 VA Scheduling and Wait Times
                            indicating reported wait time inaccuracies. At the time of our site visit,
                            officials from this clinic indicated that long waits for new patient
                            appointments had existed prior to our visit and told us that the next
                            available appointment for a new patient was in 6 to 8 weeks. However,
                            reported wait time data for the month we visited showed that the clinic
                            completed all new patient appointments on the desired date, resulting in
                            an unlikely high percentage of appointments with zero-day wait times that
                            was inconsistent with information gathered during our site visit, raising
                            questions about whether the desired date was recorded in accordance
                            with VHA’s scheduling policy. Furthermore, according to reported wait
                            times for the VAMC, this clinic completed nearly all new patient
                            appointments within 14 days of the desired date for the 2 months prior to
                            our visit; and, similarly, in the 2 months after our visit, reported wait times
                            for this clinic show completion of all new patient appointments within the
                            14-day time frame. 33


VHA Officials Cited         VHA central office officials told us that they recognized the potential
Importance of Desired       reliability issues of using the desired date for measuring wait times, but
Date in Capturing Patient   stated that use of the desired date is the best approach for capturing
                            patient experience and preference. Officials told us that there is no single
Preference and              industry standard for measuring how long patients wait for appointments
Supplement Wait Time        and commonly used measures—such as capacity measures—do not
Measures with Other         account for patient preference or reflect how long the patient actually
Information to Monitor      waited for an appointment. 34 In addition, officials told us that the VistA
                            scheduling system was not designed to capture data for management
Patient Access to           purposes, which has limited VHA’s options for developing wait time
Appointments                measures. Over the years, VHA has tried using many different
                            approaches to measuring wait times, such as capacity measures and
                            using the date the appointment was created rather than the desired date




                            33
                              This is based on a measure similar to the performance measure “percent of new patient
                            specialty care medical appointments completed within 14 days of the desired date,”
                            reported on the PAR.
                            34
                               Clinic capacity is the supply of available future appointments. Capacity measures
                            typically count the number of days between the day the measure is taken and the day the
                            first or third next available appointment occurs.




                            Page 16                                        GAO-13-130 VA Scheduling and Wait Times
to determine wait times. 35 Although these measures were not officially
used for performance accountability or reported on the PAR or NDPP and
FDPP in fiscal year 2012, data on these measures are available to VISNs
and VAMCs for performance monitoring. Officials told us that improving
how wait times are measured is an ongoing effort, and they have
conducted research to identify wait time measures that most closely
correlate with patient satisfaction and positive outcomes. At the time of
our review, VHA had not implemented changes to wait time performance
measures based on the results of this research.

In addition to measuring medical appointment wait times, VHA central
office officials reported that VHA also uses other information to monitor
patients’ access to medical appointments and to assist VISNs and
VAMCs in managing clinics.

Patient Satisfaction Measures: VHA central office and VISN officials with
whom we spoke identified patient satisfaction as another important
indicator of patient access to medical appointments and VA has
incorporated measures of self-reported patient satisfaction in its
performance assessments. 36 Specifically, the annual PAR includes a
measure of overall patient satisfaction with VHA inpatient and outpatient
healthcare in addition to the wait time measures derived from desired
date. Separate measures related to patient satisfaction with obtaining
outpatient care were also among the measures available for VISN and
VAMC directors to include in their fiscal years 2011 and 2012
performance plans (NDPP and FDPP). 37 VHA also makes the satisfaction



35
  For example, in the 2010 PAR, VA reported a wait time performance measure for new
patient medical appointments based on days from the date the appointment was created.
The appointment create date is automatically generated in the VistA scheduling system
and therefore not prone to scheduler error. VHA officials told us that wait time measures
based on create date do not reflect patient preference and therefore can incorrectly
characterize wait times, particularly for established patient follow-up appointments which
may be scheduled months before they are completed.
36
  Since 2002, VHA has measured veterans’ perceived access through a monthly Survey
of Health Experiences of Patients —a survey of satisfaction with inpatient and outpatient
care similar to the Department of Health and Human Services Consumer Assessment of
Healthcare Providers Survey.
37
  Specifically, those measures were: Getting Needed Care—combines responses from
questions regarding how much of a problem, if any, patients had with various aspects of
getting needed care; and Getting Care Quickly—combines responses from questions
regarding how often patients received various types of care in a timely manner.




Page 17                                         GAO-13-130 VA Scheduling and Wait Times
measures available to VISNs and VAMCs for continuous performance
monitoring as well as available to the public. One of the four VAMCs we
visited included the satisfaction measures in their performance plan for
fiscal year 2012, and officials cited monitoring these measures on a
regular basis. Officials from one VISN also specifically cited comparing its
VAMCs’ patient satisfaction scores to reported wait times to identify
inconsistencies. However, the director of another VAMC said he does not
rely on the satisfaction measures to monitor access because the data are
dated by the time the VAMC sees the results, and instead, he relies on
the scheduling data derived from wait time measures.

Clinic Management Information: In addition to wait time measures, VHA
has other information available for VISNs and VAMCs to manage clinics
and monitor and improve clinic access, such as no-show rates and
consult lists. Several clinic officials reported monitoring no-show rates—
the rate at which patients do not appear for their scheduled
appointment—in order to reduce unused appointments, for example, by
identifying and providing additional appointment reminders to patients
with frequent no-shows. Officials from multiple specialty clinics said they
monitor lists of consults—requests for specialty care appointments—to
ensure they are acted upon in a timely manner. 38 Although the time
between when the provider requests a consult and when the specialty
clinic reviews the consult can affect the total time a patient waits for a
specialty appointment, this time is not reflected in current wait time
performance measures.




38
  Consults are most often communicated electronically through an application in the
electronic medical record within VistA. The electronic medical record is separate from the
scheduling system.




Page 18                                         GAO-13-130 VA Scheduling and Wait Times
                             The four VAMCs we reviewed did not consistently implement certain
Inconsistent                 elements of VHA’s scheduling policy, including oversight requirements,
Implementation of            which may result in increased wait time or delays in scheduling medical
                             appointments. VAMCs also described other problems with scheduling
VHA’s Scheduling             timely medical appointments, including outdated technology, gaps in
Policy and Other             staffing of schedulers and providers, and telephone access problems.
Problems Impede
VAMCs’ Ability to
Schedule Timely
Medical
Appointments
Inconsistent                 The four VAMCs we visited did not consistently implement VHA’s
Implementation of VHA’s      scheduling policy, which is intended to facilitate the creation of medical
Scheduling Policy Hinders    appointments that meet patients’ needs with no undue waits or delays.
                             This policy includes the use of the VistA scheduling system to schedule
VAMCs’ Ability to Schedule   medical appointments, and the use of the electronic wait list to track new
Timely Medical               patients waiting for medical appointments. (See table 3 for information on
Appointments                 the number of clinics we visited that did not implement selected elements
                             of the VHA’s scheduling policy.) Inconsistent implementation of VHA’s
                             scheduling policy can result in increased wait time or delays in obtaining
                             medical appointments.




                             Page 19                                 GAO-13-130 VA Scheduling and Wait Times
Table 3: Number of Clinics That Did Not Implement Selected Elements of the VHA’s Scheduling Policy, by VAMC Visited

                                                                                  Number of clinics at each VAMC that did not implement the
                                                                                             element of VHA’s scheduling policy
                                                                                                (clinics visited at each VAMC)
Element of VHA’s scheduling policy
(number of clinic responses)                                                              VAMC A (4)   VAMC B (9)      VAMC C (5)         VAMC D (5)
Using VistA in scheduling of medical appointments (23)                                            0              1                   0                  0
Scheduling medical appointments while in direct communication
with the patient (20)                                                                             0              1                   5                  0
Using the electronic wait list for patients who have not been seen
                                                     a
before in the clinic and are waiting to be scheduled (21)                                         0              1                   2                  1
Using recall/reminder software for medical appointments needed
more than 3 to 4 months into future (22)                                                          0              0                   5                  0
Keeping medical appointment schedules open at least 3 to
4 months into future (19)                                                                         1              1                   1                  1
Total number instances in which elements of VHA’s
scheduling policy were not implemented                                                            1              4                 13                   2
Total number of clinics that did not implement at least one
element of VHA’s scheduling policy                                                                1              2                   5                  2
                                             Source: GAO analysis of interviews at 23 clinics.

                                             Notes: Except for the element “using VistA in scheduling of medical appointments,” we did not report
                                             responses for all 23 clinics we visited because interview responses were incomplete for some
                                             elements of the VHA’s scheduling policy as depicted in the table.
                                             a
                                              Officials from 12 clinics told us that they do not use the electronic wait list because their clinic did not
                                             have new patients waiting for appointments or their clinic schedules patients for appointments with
                                             long wait times.


Use of VistA Scheduling System               One of the clinics we visited did not use the VistA scheduling system to
                                             determine available medical appointment dates and times, and to
                                             schedule medical appointments, as required by VHA’s scheduling policy.
                                             Officials noted that this clinic lacked a full-time staff person dedicated to
                                             scheduling, and therefore, the providers called their patients to schedule
                                             their own medical appointments. Clinic staff reported that providers
                                             recorded medical appointments on sheets of paper and gave those
                                             sheets to a scheduler, who maintained a paper calendar of all medical
                                             appointments; this scheduler later recorded the appointment into the
                                             VistA scheduling system. Failing to use VistA to schedule medical
                                             appointments could create additional backlogs or scheduling errors
                                             because the schedule in VistA may not accurately reflect providers’
                                             availability. According to one provider in this clinic, for example, “staff
                                             from other departments look in VistA [scheduling system] and it looks like
                                             the clinic is not booked, so they’ll send their patients as walk-in
                                             appointments. However, the clinic is really fully booked and patients are
                                             waiting.”



                                             Page 20                                                     GAO-13-130 VA Scheduling and Wait Times
Communication with Patients       Officials from six clinics across two different VAMCs reported that staff
                                  scheduled new patient or established patient follow-up medical
                                  appointments without speaking to patients, and then notified patients of
                                  the scheduled medical appointment by letter, if the appointment was at
                                  least a few weeks away. This method of scheduling—referred to as
                                  “blind” scheduling by one official —is not in accordance with VHA’s
                                  scheduling policy and could result in missed medical appointments for
                                  patients who do not receive the letters, or are not available at the
                                  scheduled time because patients are not involved in the scheduling
                                  process. One scheduler noted that he sent medical appointment letters
                                  because he didn’t have time to call all patients to schedule appointments
                                  as he performs scheduling duties for 27 different clinics. Furthermore,
                                  outdated or incorrect patient contact information is an impediment to
                                  scheduling appointments via letters; an official in one of the six clinics told
                                  us that the databases containing patient contact information used to send
                                  such letters often do not have veterans’ correct or up-to-date contact
                                  information.

Use of the Electronic Wait List   Officials in four clinics across three VAMCs that had backlogs of patients
                                  waiting for medical appointments stated that they do not use the
                                  electronic wait list, the official VHA wait list used to track patients with
                                  whom a clinic does not have an established relationship. 39 Clinics that do
                                  not use the electronic wait list may be at risk of losing track of new
                                  patients waiting for medical appointments. For example, at one specialty
                                  clinic with a backlog of consult requests, medical appointments for new
                                  patients were backed up almost 3 months; VAMC officials reported
                                  tracking patients waiting for medical appointments by printing paper
                                  copies of the consult requests from the electronic medical record. A
                                  provider at this clinic expressed concern that the clinic manager “has a
                                  tall stack of unscreened consult referrals just sitting on her desk, and no
                                  one is addressing them.”




                                  39
                                    According to VHA’s scheduling policy, the electronic wait list is used to keep track of
                                  patients with whom the provider does not yet have an established relationship and who
                                  cannot be scheduled for appointments in target time frames. No other wait list formats
                                  (such as paper or electronic spreadsheets) are to be used for tracking requests for
                                  medical appointments.




                                  Page 21                                          GAO-13-130 VA Scheduling and Wait Times
Use of Recall/reminder          Officials from one VAMC stated that it did not have the required
Software                        recall/reminder software to facilitate reminders for patients who need to
                                return to the clinic for follow-up medical appointments more than 3 to
                                4 months into the future; therefore, none of its clinics, including the five
                                clinics that we visited, were able to use it as intended. 40 Instead clinics at
                                this VAMC use a work-around in the scheduling system to remind clerks
                                to print and send letters reminding patients to call and schedule their
                                follow-up medical appointments. However, this work-around is not
                                automated and relies on schedulers to remember to generate a list of
                                patients who need follow-up medical appointments, and print and send
                                those letters. The VAMC is in the process of implementing recall/reminder
                                software, according to officials.

Medical Appointment Schedule    One clinic in each of the four VAMCs visited did not keep their medical
Availability                    appointment schedules open 3 to 4 months into the future as required by
                                VHA’s scheduling policy. 41 Instead, these four clinics allowed medical
                                appointments to be booked only 1 to 2 months into the future. Limiting the
                                future medical appointment schedule may limit patients’ ability to
                                schedule a follow-up medical appointment before leaving the clinic, as
                                recommended by the policy, and also may result in additional work for
                                clinic staff to send recall/reminder letters to patients for medical
                                appointments less than 3 to 4 months away.

Oversight of VHA’s Scheduling   The VAMCs we visited inconsistently implemented certain oversight
Policy                          requirements in VHA’s scheduling policy—specifically, completion of
                                training and certification of compliance. VAMC officials stressed the
                                importance of scheduler training for ensuring correct implementation of
                                VHA’s scheduling policy; however, certain VAMCs did not ensure




                                40
                                  Patients are entered into the recall/reminder software for the date they are to return to
                                the clinic—which should be identified by the provider—and VistA automatically generates
                                correspondence to the patient (post card or letter) a week or 2 prior to that date to remind
                                the patient to call the clinic and schedule a medical appointment.
                                41
                                  VHA’s scheduling policy states that for clinics to most efficiently operate, “schedules
                                must be open and available for the patient to make [medical] appointments at least three
                                to four months into the future. Permissions may be given to schedulers to make
                                appointments beyond these limits when doing so is appropriate and consistent with patient
                                or provider requests. Blocking the scheduling of future [medical] appointments by limiting
                                the maximum days into the future an appointment can be scheduled is inappropriate and
                                is disallowed.”




                                Page 22                                          GAO-13-130 VA Scheduling and Wait Times
completion of the training by all staff who were required to complete it. 42
Although all VAMCs we visited provided a list of staff who can schedule
appointments, three VAMCs did not provide documentation that all staff
on the list had successfully completed the required training. For example,
officials from one VAMC stated that it maintained a list of staff who can
schedule appointments, and a separate list of staff who had completed
the training, but only in response to GAO’s request for documentation did
the VAMC identify staff with scheduling access who needed to complete
the training. Further, three of the 19 schedulers we interviewed said they
completed training other than the required VHA scheduler training.
Completion of required VHA scheduler training and maintaining up-to-
date documentation of schedulers’ completion of the training is
particularly important for ensuring consistent implementation of VHA’s
scheduling policy, given the high rates of scheduler turnover described by
officials.

All four of the VAMCs we visited completed the required self-certification
of compliance with the VHA’s scheduling policy for fiscal year 2011, three
of which certified overall compliance, and one certified overall
noncompliance. 43 However, leadership officials from two VAMCs,
including the only one of the four that certified overall noncompliance,
were initially uncertain who completed the certification or the steps taken
to complete it, indicating that VAMCs are not always using the self-
certification process to identify and improve problems with compliance
with VHA’s scheduling policy.




42
  VAMCs are required to maintain a list of all staff who can schedule medical
appointments in the VistA scheduling system, and are required to ensure successful
completion of required VHA scheduler training by all staff on that list. Schedulers are not
to be allowed to schedule medical appointments in the VistA scheduling system without
proof of their successful completion of this training.
43
  Of the 144 VAMCs that completed the certification for fiscal year 2011, 109 certified
overall compliance, 27 certified partial compliance, and 8 certified noncompliance.




Page 23                                          GAO-13-130 VA Scheduling and Wait Times
VAMCs Identified Other         VAMCs identified several problems that can impede the timely scheduling
Problems with Scheduling       of medical appointments, which also may impact their compliance with
Medical Appointments,          VHA’s scheduling policy.
Including Issues with
Outdated Technology,
Staffing Gaps, and
Telephone Access
Problems with Outdated VistA   VHA central office officials and officials from all of the VAMCs we visited
Scheduling System              said the VistA scheduling system is outdated and inefficient, which
                               hinders the timely scheduling of medical appointments. In particular,
                               officials said the scheduling system requires schedulers to use
                               commands requiring many keystrokes and does not allow them to view
                               multiple screens at once. Schedulers must open and close multiple
                               screens to check a provider’s or clinic’s full availability when scheduling a
                               medical appointment, which is time-consuming and can lead to errors.
                               For example, providers have separate schedules within VistA to
                               accommodate the various types of services they provide. 44 Because the
                               scheduling system cannot display multiple schedules on the same
                               screen, schedulers have to enter and exit multiple screens to check a
                               provider’s full daily schedule when scheduling a medical appointment. If
                               schedulers do not open all of the necessary screens, they may
                               unknowingly create scheduling errors such as booking two medical
                               appointments at the same time in different sections of a provider’s
                               schedule. Further, staff at one VAMC told us the problem of not being
                               able to easily view a provider’s full schedule can result in the failure to
                               ensure that appointments are cancelled when a provider requests it. This
                               error could cause patients to come to the VAMC unnecessarily or a failure
                               to reschedule cancelled appointments in a timely way, both of which
                               might lead to increased wait times for those patients.

                               Officials from all the VAMCs we visited also noted that the VistA
                               scheduling system is not easily adapted to meet clinic needs. For
                               example, staff cannot create a provider schedule in the scheduling
                               system that is longer than 8 hours. If a provider wants to extend his or her
                               schedule on certain days, staff must create additional clinic schedules in



                               44
                                 For example, a physical therapist may have a separate schedule for amputee clinic,
                               general physical therapy, or other types of services.




                               Page 24                                        GAO-13-130 VA Scheduling and Wait Times
the scheduling system for that provider, which can result in more delays
and possible errors because schedulers have to check additional screens
for medical appointment availability. Furthermore, officials told us that the
scheduling system does not automatically interface with VHA’s electronic
medical record, 45 which makes the scheduling process more time-
consuming as schedulers alternate between the two software applications
to ensure medical appointments are made in accordance with providers’
guidance.

VAMC officials described steps they take to ensure schedulers use VistA
in accordance with the scheduling directive, including ongoing scheduler
training and supervisory reviews of scheduler performance. However, as
noted above, a lack of clarity in the desired date training documents and a
lack of documentation of scheduler training at certain facilities may limit
the effectiveness of these interventions. One VAMC provides schedulers
with dual monitors to enable them to open multiple screens at once.
Another VAMC told us they considered this solution in their primary care
clinic, but found that limited physical space in the clinic did not
accommodate additional monitors.

In response to ongoing problems with the VistA scheduling system, VHA
undertook an initiative to replace it in 2000, but VA abandoned the
replacement due to weaknesses in project management and a lack of
effective oversight. VA released a new request for information in
December 2011 to gather information about vendors and possible
software packages that could replace the current scheduling system. In
September 2012, VHA told us that vendors’ responses to the request for
information indicated that VHA will be able to choose among several
viable software packages. According to officials, VA’s next step is to
compare different vendors’ software packages through the summer of
2013, and subsequently issue a request for vendor proposals. 46




45
  The electronic medical record is a component of the VistA system that includes patient
health information and enables providers to record notes, such as when the provider
would like to see the patient for a follow-up appointment, and place orders for procedures,
x-rays, and laboratory tests, among other things.
46
  In October 2012, VA announced a contest seeking proposals for a new medical
appointment scheduling system from commercial software developers. The contest is
intended to reduce risks in the future procurement and implementation of a new
scheduling system.




Page 25                                          GAO-13-130 VA Scheduling and Wait Times
Problems with Scheduler          VHA central office officials and officials from all of the VAMCs we visited
Turnover and Provider Staffing   stated that shortages or turnover of schedulers also creates problems for
Gaps                             the timely scheduling of medical appointments. Officials said that
                                 schedulers perform many important roles, including greeting patients,
                                 checking patients in and out of clinics, answering telephone calls,
                                 scheduling medical appointments for primary care, as well as specialty
                                 care consults, and performing other administrative support functions on
                                 behalf of the clinical staff. Officials explained, however, that high stress
                                 and a demanding workload as well as the entry-level pay grade of the
                                 scheduler position leads to high turnover. Further, officials told us that
                                 high-performing schedulers often are quickly promoted to other positions
                                 within VA. According to VHA officials, most scheduler positions are
                                 classified as a low grade within the government general schedule pay
                                 scale with little room for upward movement within the grade. Officials at
                                 two of the VAMCs we visited told us they are working to raise the pay
                                 level for schedulers; for example, one VAMC has begun to assess
                                 scheduler position descriptions to determine whether they can be
                                 reclassified to allow for more flexibility in determining scheduler salaries
                                 based on the variation in their assigned duties.

                                 Given the important role of schedulers in the scheduling process, officials
                                 said that even temporary staffing gaps or shortages can cause medical
                                 appointment delays or wait times. Staff with whom we spoke in several
                                 clinics said that when scheduler staffing is lacking, including when a
                                 scheduler is on short-term leave, it is difficult to cover all the scheduler’s
                                 duties, and that such gaps can cause delays for patients. Further, we
                                 were told that scheduler staffing gaps resulted in inefficient use of clinical
                                 staff time. For example, at one specialty clinic that lacked its own
                                 scheduler, providers routinely scheduled their own medical appointments,
                                 which took away from time seeing patients, and also resulted in incorrect
                                 scheduling practices. Given the training needs associated with using the
                                 VistA scheduling system, following VHA’s scheduling policy, and ensuring
                                 the correct use of desired date, high rates of scheduler turnover could
                                 contribute to inconsistent use of desired date in the scheduling process or
                                 other appointment scheduling problems.

                                 Officials at two VAMCs noted that scheduler staffing gaps are
                                 compounded by recent changes in their roles and responsibilities as VHA
                                 implements a new team-based model of primary care, which calls for one
                                 scheduler to be assigned to each primary care team. Officials told us that
                                 these changes generally increase the administrative demands placed on
                                 schedulers, as they are asked to respond to team duties while continuing
                                 to answer phones, greet patients, and register new patients, among other


                                 Page 26                                  GAO-13-130 VA Scheduling and Wait Times
responsibilities. Officials from two VAMCs told us they had requested
approval to hire additional staff to meet these added administrative
needs.

Scheduler staffing gaps may also create problems managing patient flow
through clinics, which can impede scheduling of follow-up appointments,
according to officials at two of the VAMCs we visited. Staff at these
VAMCs told us that they sometimes do not have sufficient schedulers
available to staff check-out desks, and staff at one VAMC added that as a
result patients might “fall through the cracks,” leaving follow-up medical
appointments unscheduled unless the patient remembers to call in to
schedule the appointment. In addition, when patients do not check out,
schedulers are responsible for tracking patients needing follow-up
medical appointments. This situation may be exacerbated in clinics that
do not use the required recall/reminder software to facilitate the
scheduling of follow-up medical appointments more than 3 to 4 months in
the future, adding further to the backlog of patients in need of follow-up
medical appointments.

Officials from all of the VAMCs we visited told us that provider shortages
also contribute to scheduling backlogs in certain locations and specialties.
Recruitment and retention of providers was a particular challenge for
VAMCs in rural areas, areas with high costs of living, and for certain
provider specialties. All of the VAMCs we visited described gaps in
provider staffing in certain specialty care clinics. Officials at all VAMCs
also stated that a lack of salary competitiveness or the length of time to
hire new providers into the VA system also contributed to gaps in provider
staffing and scheduling backlogs.

Gaps in provider staffing also can result from providers being on
extended or unexpected leave, including vacation time, sick leave, or
military deployments. These absences may result in longer wait times for
patients. For example, officials at one VAMC told us that even a brief
absence of one provider on leave can cause significant wait times, and
that it is difficult to catch up and eliminate the backlog.

Staff from some clinics described steps they take to reduce backlogs
caused by gaps in provider staffing, including overbooking provider
schedules and scheduling temporary Saturday hours. Officials at one
VAMC told us that they employ a “floater” primary care physician to
provide coverage for providers on leave, but an official at another clinic
told us that they were unable to hire additional providers to meet the
demand for medical appointments.


Page 27                                  GAO-13-130 VA Scheduling and Wait Times
Problems with Telephone   Officials at all of the VAMCs we visited told us that high call volumes and
Access                    a lack of staff dedicated to answering the telephones impede the timely
                          scheduling of medical appointments. 47 Despite VHA’s telephone policy
                          requiring the provision of continuous telephone service for clinical care
                          and medical appointment management, VAMC officials noted that
                          schedulers are frequently overwhelmed by high call volumes and are
                          unable to respond to calls in a timely way. In addition, officials at one
                          VAMC told us that outdated telephone technology, and the lack of a
                          dedicated VAMC-wide call center, limited their ability to improve their
                          telephone responsiveness. VHA has reported that telephone access to
                          VHA health services has historically been a frustrating experience for
                          veterans, including dropped calls, multiple transfers, and long waits to
                          reach a staff person able to resolve their inquiries. 48 Further, patients at
                          all of the VAMCs we visited registered complaints about the difficulty of
                          reaching outpatient clinic staff by telephone and unreturned telephone
                          calls. According to information on patient complaints provided by the four
                          VAMCs we visited, patient complaints about unreturned telephone calls
                          ranked among the top two categories of complaints in fiscal year 2012 at
                          all four VAMCs. 49 Further, staff at two of the VAMCs reported that their
                          telephone calls to outpatient clinics within their own VAMC went
                          unanswered, and one added that their inability to reach staff in their own
                          clinics also was an obstacle to timely medical appointment scheduling.




                          47
                            VHA’s policy on telephone service for clinical care, VHA Directive 2007-033, establishes
                          VHA’s policy of providing telephone access for appointment management and continuous
                          access to health care advice. The telephone directive also establishes recommended
                          benchmarks for telephone service at VA facilities. VAMCs differ in how they manage the
                          telephones; for example, some VAMCs establish VAMC-wide call centers to answer and
                          direct incoming calls. VAMC telephone systems generally serve the VAMC including its
                          affiliated CBOCs and ambulatory care centers.
                          48
                           Veterans Health Administration, Telephone Systems Improvement Guide, Second
                          Edition (December 2011).
                          49
                            Each VAMC has a patient advocate who accepts and addresses patient complaints. The
                          patient advocate records complaints in the patient advocate tracking system and tracks
                          complaints in various categories, including “phone calls not returned, letters not
                          answered.” Two of the four VAMCs provided information on patient complaints for fiscal
                          year 2012 from October 1, 2011, through May 31, 2012. One VAMC provided information
                          for fiscal year 2012 through June 25, 2012, and the other provided information for fiscal
                          year 2012 through August 31, 2012.




                          Page 28                                         GAO-13-130 VA Scheduling and Wait Times
                         In January 2012, VHA distributed suggested best practices for improving
                         telephone design, service, and access in its Telephone Systems
                         Improvement Guide. 50 This guide outlines steps VHA found to be effective
                         means of improving telephone service and maintaining health care
                         access, including regularly monitoring the purpose and volume of
                         telephone calls; establishing dedicated staff to answering calls, especially
                         at times of peak call volume; and training staff responsible for answering
                         telephones in call centers. To address telephone issues, officials at one
                         VAMC we visited told us they were developing a proposal to establish a
                         call center with a new telephone system, to be staffed by schedulers
                         dedicated to answering the telephones. Officials at a different VAMC
                         stated that a scheduling supervisor periodically checks schedulers’
                         telephones to ensure that voice mail messages are listened to and that
                         calls are returned.


                         VHA is implementing several initiatives to improve veterans’ access to
VHA Is Implementing      medical appointments. Specifically, these initiatives focus on more
a Number of              patient-centered care; using technology to provide care, through means
                         such as telehealth; and using care outside of VHA to reduce travel and
Initiatives to Improve   wait times for veterans who are unable to receive certain types of
Access to Medical        outpatient care in a timely way through local VHA facilities. VHA officials
                         told us they are monitoring the implementation of these initiatives;
Appointments             however, in some cases, more information is needed to determine their
                         impact on timely access to care over time.


Patient-Centered Care    VHA’s patient-centered medical home model for primary care, Patient
Initiatives              Aligned Care Teams (PACT), is intended, in part, to improve access to
                         medical appointments and care coordination through the use of
                         interdisciplinary care teams and technology to communicate with patients.
                         Implementation of PACT began in 2010, and is an ongoing effort,
                         according to VHA officials. PACT differs from how primary care was
                         previously delivered by assigning each patient to an interdisciplinary
                         team. The PACT team is intended to be comprised of a primary care
                         provider, registered nurse care manager, a clinical support staff member
                         such as a licensed practical nurse, and a scheduler. 51 These teams offer


                         50
                          Veterans Health Administration, Telephone Systems Improvement Guide.
                         51
                           VHA officials noted that even in primary care clinics in which PACT implementation has
                         begun, some PACT teams are not yet fully staffed in accordance with the model.




                         Page 29                                        GAO-13-130 VA Scheduling and Wait Times
patients a centralized way to get questions answered by nurses or other
clinical support staff and aim to reduce the need for face-to-face medical
appointments, thereby enabling more efficient use of providers’ time. For
example, at one of the VAMCs we visited, patients are given a direct
telephone number to contact their PACT team and leave a voice mail
message to be returned by the team’s registered nurse. Encouraging
PACT teams’ use of telephone communication and telephone
appointments is intended to enable patients to more quickly obtain
answers to some of their administrative and medical questions, such as
requests for prescription refills, without having to schedule a face-to-face
medical appointment. VHA officials told us that they expect PACT teams’
use of telephone communication and telephone appointments will open
up face-to-face medical appointment slots for patients who need them
and might enable clinics to reduce backlogs and improve access to same-
day primary care medical appointments.

Officials at two VAMCs we visited told us that the transition to the PACT
model has created some initial scheduling and staffing difficulties. For
example, officials at these VAMCs noted that it is difficult for scheduling
staff to respond to their PACT team duties in addition to meeting other
responsibilities such as answering phones, checking in patients,
registering new patients, and scheduling for more than one clinic. This is
compounded by the fact that not all PACT teams have been assigned
their own scheduler, as prescribed by the PACT model, so an individual
scheduler is sometimes serving multiple PACT teams. Officials at these
two VAMCs explained that they would need to hire more schedulers to
meet the goal of assigning one to each PACT team.

To measure the progress of PACT implementation and its impact on
access to quality care, VHA is collecting data and tracking a series of
measures in a monthly internal data report. Five of the PACT measures
are (1) primary care medical appointments completed within 7 days of the
desired date; 52 (2) same day access with primary care provider, or the
percentage of appointments completed within 1 day; (3) telephone
utilization, or the percentage of total encounters that occur by telephone;
(4) continuity of care, or the percentage of primary care appointments
with the patient’s assigned primary care provider; and (5) post-hospital


52
  In contrast to the PACT 7-day wait time measure reported in the NDPP and FDPP, VA
reported primary care wait time performance as the completion of appointments within
14 days of desired date in its fiscal year 2012 PAR.




Page 30                                       GAO-13-130 VA Scheduling and Wait Times
                    discharge contact, or percentage of patients discharged from the hospital
                    who were contacted by their primary care provider within 2 days. 53 As
                    described earlier, accurate measurement of medical appointment wait
                    times—including the first two PACT measures—is dependent upon the
                    correct recording of the desired date in the VistA scheduling system. In
                    fiscal year 2012, PACT measures were also included in the NDPP and
                    FDPP. 54


Initiatives Using   Part of VHA’s goal of achieving improved access to medical appointments
Technology          is the increased use of technology such as telehealth and secure
                    messaging. 55 Use of these tools is intended to improve communication
                    between patients and providers and open up providers’ schedules for
                    needed face-to-face medical appointments, thereby improving access to
                    face-to-face appointments.

                    VHA telehealth includes:

                    •     home telehealth for chronic disease management such as diabetes;

                    •     real-time clinic-based video telehealth, in which patients at a local
                          CBOC may connect with a VHA provider at a different location to
                          receive services that are unavailable at the CBOC, such as mental
                          health or speech pathology;

                    •     and store-and-forward telehealth, in which digital images such as
                          x-rays or images of skin problems, are taken, stored, and sent to an
                          expert for review and consultation.




                    53
                        There are additional measures in the monthly internal data report.
                    54
                      VISN and VAMC directors were scored for each of the five PACT measures listed and
                    rated against a sixth composite PACT measure. They were rated as meeting the
                    composite PACT measure if they met targets for three of the five individual PACT
                    measures.
                    55
                      Telehealth is the delivery of health care services using telecommunications technology.
                    Using technology such as videoconferencing, telehealth changes the location where
                    health care services are delivered. Secure messaging is VHA’s web-based message
                    service that allows patients to communicate nonemergency health-related information with
                    their health care team.




                    Page 31                                           GAO-13-130 VA Scheduling and Wait Times
                         VHA officials told us that the use of telehealth can reduce both travel and
                         wait times for medical appointments and help meet the needs of patients
                         with chronic conditions. All VAMCs we visited told us they were using
                         telehealth to improve access to care.

                         Another initiative that uses technology to reduce unnecessary face-to-
                         face medical appointments is VHA’s My HealtheVet, a web-based
                         program that enables veterans to create and maintain a web-based
                         personal health record with secure access to health information; services
                         such as prescription refill requests; and secure messaging. Secure
                         messaging allows veterans to communicate electronically with their health
                         care team. According to VHA, of the more than 8 million veterans enrolled
                         in VHA, 1.4 million are registered in My HealtheVet as of August 2012,
                         and more than 437,000 have created secure messaging accounts. A
                         recent VA study reports that secure messaging may improve access,
                         patient perceptions about access, and provides for better
                         communication. 56


Non-VA Care Initiative   VHA uses non-VA care to reduce wait times and backlogs and to provide
                         veterans’ access to specialists not available through VHA. 57 Under a
                         statutory requirement to help veterans receive care closer to home, VHA
                         is piloting a new model of non-VA care known as Project ARCH (Access




                         56
                          Kim Nazi, Department of Veterans Affairs Experiences with System-wide Transformation
                         Activities that Foster Continuous Learning and Improvement, Institute of Medicine
                         Consensus Study on the Learning Healthcare System in America (May 2012).
                         57
                           Non-VA care is medical care paid for by VA but provided to veterans outside of VA.
                         Non-VA care may be offered on a temporary basis to a veteran when medical services are
                         not available due to a lack of available VA specialists, long wait times, or when VA care is
                         only available at extraordinary distances from a veteran’s home. VAMCs do not track wait
                         times for patients using non-VA care.




                         Page 32                                          GAO-13-130 VA Scheduling and Wait Times
Received Closer to Home). 58 Project ARCH is a five-site, 3-year pilot
program administered by the VHA Office of Rural Health to provide health
care services through contracts with local community providers. 59
According to VHA officials, Project ARCH might help alleviate wait times
for specialty care services with high demand, or for which there is a
shortage of local providers.

At the Montana Project ARCH pilot site, which we visited as part of our
site visit to the Montana VAMC, staff from the VAMC and the Billings
Clinic, a non-VA provider delivering services to veterans through Project
ARCH, identified both benefits and obstacles for patients enrolled in
Project ARCH. For example, though VAMC and Billings Clinic staff noted
that Project ARCH reduced both travel and wait times for Montana
veterans in need of orthopedic care, Billings Clinic staff also noted that
difficulties in coordinating care for veterans moving between VHA and
non-VA providers at times resulted in delays in providing care to those
and other veterans. Additionally, problems with processing authorizations
for certain services were among the concerns raised in an April 2012
evaluation of the Montana Project ARCH program. 60

Project ARCH contractors must submit monthly reports, including
information on medical appointment scheduling timeliness, wait times,
and other topics. For example, the contractor for the Project ARCH


58
  See Pub. L. No. 110-387, § 403, 122 Stat. 4110, 4124 (2008). Veterans are eligible to
participate in the program if they reside in a location where a pilot site is located and if
they are enrolled in VA health care when the program starts, and meet any of the
following criteria: live more than (1) 60 miles driving distance from the nearest VA health
care facility providing primary care services, if the veteran is seeking such services;
(2) 120 miles driving distance from the nearest VA health care facility providing acute
hospital care, if the veteran is seeking such care; or (3) 240 miles driving distance from
the nearest VA health care facility providing tertiary care, if the veteran is seeking such
care. Nonenrolled veterans who are eligible to enroll in VA health care because they
served in a combat theater after November 11, 1998, are also eligible to participate in the
program. Health care delivery contracts for services covered under Project ARCH were
awarded to Humana Veterans, a health services support contractor, in four pilot sites, and
to a health care provider in the fifth pilot site.
59
 Project ARCH services are currently being piloted at five sites, including Northern
Maine; Farmville, Virginia; Pratt, Kansas; Flagstaff, Arizona; and Billings, Montana.
60
  VA is required to evaluate the program and prepare an annual report to Congress for
each of the 3 years of the pilot. VHA engaged a contractor to conduct site visits and
provide VHA with quarterly progress reports on Project ARCH implementation. The first
progress report for Montana VAMC’s Project ARCH program was produced in April 2012.




Page 33                                          GAO-13-130 VA Scheduling and Wait Times
              Program in Montana is required to report on the extent to which it is
              meeting VHA’s 14-day wait time goal for medical appointments—
              according to VHA officials, the contractor must meet a 90 percent target.
              These wait times may not accurately reflect how long patients are waiting
              for a medical appointment, however, because the wait time is counted
              from the time the contractor receives the authorization from VA, rather
              than from the time the patient or provider requests a medical
              appointment.


              VHA officials have expressed an ongoing commitment to providing
Conclusions   veterans with timely access to medical appointments and have reported
              continued improvements in achieving this goal. However, unreliable wait
              time measurement has resulted in a discrepancy between the positive
              wait time performance VA has reported and veterans’ actual experiences.
              Ambiguity in what constitutes the medical appointment desired date—the
              date VHA uses as the basis for measuring wait time—as well as
              manipulation of the desired date to meet goals have contributed to these
              inaccuracies. With more than 50,000 schedulers making approximately
              80 million medical appointments in fiscal year 2011, establishing a clear
              definition of the desired date or finding and reporting another acceptable
              measure of wait time is key to understanding how long veterans are
              actually waiting for medical appointments. Without reliable measurement
              of how long patients are waiting for medical appointments, VHA is less
              equipped to identify and address factors that contribute to wait times, or
              gauge the success of its initiatives to improve access to timely medical
              appointments, including efforts to improve primary care medical
              appointments.

              More consistent adherence to VHA’s scheduling policy and oversight of
              the scheduling process, as well as the allocation of staffing resources in
              accordance with clinics’ demands for scheduling of medical
              appointments, would potentially reduce medical appointment wait times.
              Furthermore, persistent problems with telephone access must be
              resolved to assure veterans’ ability to schedule timely medical
              appointments. Ultimately, VHA’s ability to ensure and accurately monitor
              access to timely medical appointments is critical to ensuring quality health
              care to veterans, who may have medical conditions that worsen if access
              is delayed.




              Page 34                                 GAO-13-130 VA Scheduling and Wait Times
                      To ensure reliable measurement of veterans’ wait times for medical
Recommendations for   appointments, we recommend that the Secretary of VA direct the Under
Executive Action      Secretary for Health to take actions to improve the reliability of wait time
                      measures either by clarifying the scheduling policy to better define the
                      desired date, or by identifying clearer wait time measures that are not
                      subject to interpretation and prone to scheduler error.

                      To better facilitate timely medical appointment scheduling and improve
                      the efficiency and oversight of the scheduling process, we recommend
                      that the Secretary of VA direct the Under Secretary for Health to take
                      actions to ensure that VAMCs consistently and accurately implement
                      VHA’s scheduling policy, including use of the electronic wait list, as well
                      as ensuring that all staff with access to the VistA scheduling system
                      complete the required training.

                      To improve timely medical appointment scheduling, we recommend that
                      the Secretary of VA direct the Under Secretary for Health to develop a
                      policy that requires VAMCs to routinely assess clinics’ scheduling needs
                      and resources to ensure that the allocation of staffing resources is
                      responsive to the demand for scheduling medical appointments.

                      To improve timely medical appointments and to address patient and staff
                      complaints about telephone access, we recommend that the Secretary of
                      VA direct the Under Secretary for Health to ensure that all VAMCs
                      provide oversight of telephone access and implement best practices
                      outlined in its telephone systems improvement guide.


                      In reviewing a draft of this report, VA generally agreed with our
Agency Comments       conclusions and concurred with our recommendations. (VA’s comments
and Our Evaluation    are reprinted in app. I.) In summary, VA stated that VHA officials have
                      closely followed our review and proactively taken steps in response to our
                      findings. Specifically, VHA is revising and improving directives, policies,
                      training, clinic management tools, and oversight related to scheduling
                      practices. VA further stated that VHA is committed to routinely assessing
                      clinics’ scheduling needs and resources and developing practices and
                      guidelines to ensure adequate staffing resources for scheduling medical
                      appointments.




                      Page 35                                  GAO-13-130 VA Scheduling and Wait Times
VA described its plans to address each recommendation as follows:

•   In response to our recommendation that VA take actions to improve
    the reliability of wait time measures, VA concurred and stated that
    VHA will revise its scheduling policy to implement more reliable wait
    time measures and new processes to better define desired date with a
    targeted completion date of November 1, 2013.

•   In response to our recommendation that VA take actions to ensure
    that VAMCs consistently and accurately implement VHA’s scheduling
    policy and ensure that all staff complete required training, VA
    concurred and stated that the revised scheduling policy will include
    improvements and standardization of the use of the electronic wait
    list. Additionally, VHA will require VISNs to update each VAMC’s
    scheduler master list and verify that all schedulers on the list have
    completed required training, and will require schedulers to complete a
    standardized training update on the revised scheduling policy. The
    targeted completion date for these activities is November 1, 2013.

•   In response to our recommendation that VA develop a policy that
    requires VAMCs to routinely assess clinics’ scheduling needs and
    resources, VA concurred and stated that VHA will ask VAMCs to
    routinely assess clinics’ availability and ensure staff is distributed to
    meet access standards in clinics. However, VA has not specified
    requirements for VAMCs to complete these assessments nor has the
    agency provided a timeline for this process. Because schedulers are
    key to ensuring timely appointment scheduling, we believe that VA
    should establish a targeted completion date for requiring these
    assessments in policy or guidance.

•   In response to our recommendation that VA ensure that all VAMCs
    provide oversight of telephone access and implement best practices
    outlined in its telephone improvement guide, VA concurred and stated
    that VHA will require each VISN director to assess current phone
    service and develop strategic improvement telephone service plans to
    improve service. Additionally, VHA will identify a process to monitor
    performance on a quarterly basis for at least 1 year after the
    assessment. The targeted completion date for the telephone service
    assessments and plans is March 30, 2013.




Page 36                                  GAO-13-130 VA Scheduling and Wait Times
As arranged with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 28 days after its
issuance date. At that time, we will send copies of this report to
appropriate congressional committees, the Secretary of Veterans Affairs
and other interested parties. In addition, the report is available at no
charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact
me at (202) 512-7114 or draperd@gao.gov. Contact points for our Offices
of Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made key contributions to this report
are listed in appendix II.




Debra A. Draper
Director, Health Care




Page 37                                   GAO-13-130 VA Scheduling and Wait Times
List of Requesters

The Honorable Jeff Miller
Chairman
Committee on Veterans’ Affairs
House of Representatives

The Honorable Karen Bass
House of Representatives

The Honorable Shelley Berkley
House of Representatives

The Honorable Howard L. Berman
House of Representatives

The Honorable Brian P. Bilbray
House of Representatives

The Honorable Mary Bono Mack
House of Representatives

The Honorable Ken S. Calvert
House of Representatives

The Honorable John Campbell
House of Representatives

The Honorable Lois Capps
House of Representatives

The Honorable Judy Chu
House of Representatives

The Honorable Jim Costa
House of Representatives

The Honorable Susan Davis
House of Representatives

The Honorable David Dreier
House of Representatives



Page 38                          GAO-13-130 VA Scheduling and Wait Times
The Honorable Elton Gallegly
House of Representatives

The Honorable Joe Heck
House of Representatives

The Honorable Duncan D. Hunter
House of Representatives

The Honorable Darrell Issa
House of Representatives

The Honorable Jerry Lewis
House of Representatives

The Honorable Kevin McCarthy
House of Representatives

The Honorable Howard P. McKeon
House of Representatives

The Honorable Gary Miller
House of Representatives

The Honorable Grace Napolitano
House of Representatives

The Honorable Laura Richardson
House of Representatives

The Honorable Dana Rohrabacher
House of Representatives

The Honorable Lucille Roybal-Allard
House of Representatives

The Honorable Ed Royce
House of Representatives

The Honorable Loretta Sanchez
House of Representatives




Page 39                               GAO-13-130 VA Scheduling and Wait Times
The Honorable Adam Schiff
House of Representatives

The Honorable Brad Sherman
House of Representatives

The Honorable Henry A. Waxman
House of Representatives




Page 40                         GAO-13-130 VA Scheduling and Wait Times
Appendix I: Comments from the Department
             Appendix I: Comments from the Department of
             Veterans Affairs



of Veterans Affairs




             Page 41                                       GAO-13-130 VA Scheduling and Wait Times
Appendix I: Comments from the Department of
Veterans Affairs




Page 42                                       GAO-13-130 VA Scheduling and Wait Times
Appendix I: Comments from the Department of
Veterans Affairs




Page 43                                       GAO-13-130 VA Scheduling and Wait Times
Appendix I: Comments from the Department of
Veterans Affairs




Page 44                                       GAO-13-130 VA Scheduling and Wait Times
Appendix II: GAO Contact and Staff
                  Appendix II: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Debra A. Draper, (202) 512-7114 or draperd@gao.gov
GAO Contact
                  In addition to the contact named above, Bonnie Anderson, Assistant
Staff             Director; Rebecca Abela; Jennie Apter; Rich Lipinski; Sara Rudow; and
Acknowledgments   Ann Tynan made key contributions to this report.




                  Page 45                               GAO-13-130 VA Scheduling and Wait Times
Related GAO Products
             Related GAO Products




             VA Mental Health: Number of Veterans Receiving Care, Barriers Faces,
             and Efforts to Increase Access. GAO-12-12. Washington, D.C.:
             October 14, 2011.

             Information Technology: Department of Veterans Affairs Faces Ongoing
             Management Challenges. GAO-11-663T. Washington, D.C.:
             May 11, 2011.

             Information Technology: Management Improvements Are Essential to
             VA’s Second Effort to Replace Its Outpatient Scheduling System.
             GAO-10-579. Washington, D.C.: May 27, 2010.

             VA Health Care: Access for Chattanooga-Area Veterans Needs
             Improvement. GAO-04-162. Washington, D.C.: January 30, 2004.

             VA Health Care: More National Action Needed to Reduce Waiting Times,
             but Some Clinics Have Made Progress. GAO-01-953. Washington, D.C.:
             August 31, 2001.

             Veterans’ Health Care: VA Needs Better Data on Extent and Causes of
             Waiting Times. GAO/HEHS-00-90. Washington, D.C.: May 31, 2000.




(291013)
             Page 46                              GAO-13-130 VA Scheduling and Wait Times
                      The Government Accountability Office, the audit, evaluation, and
GAO’s Mission         investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
                      Website: http://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in   E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.