oversight

Defense Health Care: Appointment Timeliness Goals Not Met; Measurement Tools Need Improvement

Published by the Government Accountability Office on 1999-09-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting Office

GAO               Report to the Chairman and Ranking
                  Minority Member, Subcommittee on
                  Military Personnel, Committee on Armed
                  Services, House of Representatives

September 1999
                  DEFENSE HEALTH
                  CARE
                  Appointment
                  Timeliness Goals Not
                  Met; Measurement
                  Tools Need
                  Improvement




GAO/HEHS-99-168
                   United States
GAO                General Accounting Office
                   Washington, D.C. 20548

                   Health, Education, and
                   Human Services Division

                   B-279619

                   September 30, 1999

                   The Honorable Steve Buyer
                   Chairman
                   The Honorable Neil Abercrombie
                   Ranking Minority Member
                   Subcommittee on Military Personnel
                   Committee on Armed Services
                   House of Representatives

                   Over the past decade, the Department of Defense (DOD) has faced the same
                   challenges in delivering health care to its beneficiaries as the nation’s
                   health care system has for the general population, including increasing
                   costs and uneven access to care. In 1993, after years of demonstration
                   programs designed to explore options to manage the delivery of health
                   care more effectively, DOD restructured its health care system into
                   TRICARE, its managed care program. Today, about 8.2 million active-duty
                   personnel, their dependents, and retirees are eligible to receive care in this
                   $15.6 billion-per-year health-care system. Care for eligible beneficiaries is
                   provided mostly in military treatment facilities (MTFs), supplemented by
                   networks of contracted civilian providers. To help ensure timely access to
                   care, TRICARE established appointment timeliness standards and goals
                   similar to those of private health plans for the beneficiaries who choose to
                   enroll in TRICARE’s health maintenance organization option, called Prime.

                   While TRICARE was designed in part to improve beneficiaries’ access to
                   health care, beneficiaries have complained about the difficulties they
                   encounter obtaining care, including the length of time needed to get an
                   appointment. As you requested, this report provides information on DOD’s
                   performance in scheduling appointments, and possible reasons why Prime
                   enrollees might not obtain appointments within the appointment
                   timeliness goals. We also provide information on improvements needed to
                   DOD’s measurement tools. We conducted our work between April 1998 and
                   June 1999 in accordance with generally accepted government auditing
                   standards. See appendix I for our scope and methodology.


                   After correcting definitional discrepancies in DOD data, we found DOD has
Results in Brief   not achieved its goal of scheduling 98 percent of acute and routine
                   appointments within the timeliness standards it established. About
                   70 percent of appointments for a routine visit at MTFs were scheduled
                   within the standard, while between 80 and 97 percent of appointments for




                   Page 1                              GAO/HEHS-99-168 Access to Defense Health Care
B-279619




acute care, preventive care, or specialists were scheduled within the
relevant standards. DOD’s analysis of appointment timeliness is consistent
with our findings, and the Department has reported that the MTFs’
performance has fallen short of its expectations.

There are several reasons why active duty members and other enrollees
may not obtain appointments within the standards. For example, Prime
beneficiaries sometimes request an appointment date later than the one
offered that was within the standard, although DOD does not have the data
needed to identify the actual number of these requests. Another factor is
the extent to which MTFs provide care to nonenrolled beneficiaries. We
found that about 16 percent of the appointment slots were given to
nonenrolled beneficiaries. DOD permits nonenrollees, including retirees
over age 65, to make appointments and obtain care in MTFs because it
believes treating these beneficiaries is necessary to support medical
readiness and training requirements. DOD has made no analysis, however,
of the extent to which this policy adversely affects the ability of the
enrolled population to obtain care and treatment or the effect of any
resulting shortfall on readiness and training. Another factor affecting
appointment availability is that military beneficiaries traditionally utilize
health care at a higher rate than do private-sector beneficiaries. Research
by the Congressional Budget Office (CBO) has shown that instituting a
copayment for care provided in MTFs could reduce demand for care and
improve appointment timeliness by freeing up appointments for
active-duty members and other Prime enrollees.

As currently configured, DOD’s data tools—its Customer Satisfaction
Survey and Composite Health Care System (CHCS) appointment scheduling
system—are inadequate for measuring appointment timeliness against the
access standards. Survey weaknesses include reliance on the beneficiaries’
ability to correctly recall details of the appointments, a low response rate,
and no analysis of the beneficiaries who do not respond—all of which
affect the accuracy of the information on how well appointment standards
were met. CHCS also has weaknesses. In particular, the appointment names
used in the MTF’s appointment scheduling system do not directly relate to
the access standards. Although DOD has some efforts under way to
improve its Survey, the efforts will not overcome its inherent weaknesses,
such as its reliance on beneficiary recall. DOD also has several efforts under
way to improve the data contained in the CHCS appointment scheduling
system, including standardizing the appointment names across the military
health-care system and associating them with the timeliness standards.
Once implemented, CHCS promises to become a good source of the



Page 2                              GAO/HEHS-99-168 Access to Defense Health Care
             B-279619




             appointment timeliness information DOD needs to effectively manage and
             monitor access to care. This report makes recommendations to the
             Secretary of Defense to improve appointment timeliness measurement and
             access to care for active-duty members and other Prime enrollees.


             DOD’s  primary medical mission is to maintain the health of 1.6 million
Background   active-duty service personnel and to provide health care to them during
             military operations. DOD additionally offers health care to 6.6 million
             nonactive-duty beneficiaries, including dependents of active-duty
             personnel, military retirees, and dependents of retirees. Under TRICARE,
             most care is provided in MTFs worldwide and is supplemented by civilian
             providers. TRICARE is a triple-option health benefit program designed to
             give beneficiaries a choice among a health maintenance organization
             (Prime), a preferred provider organization (Extra), and a fee-for-service
             benefit (Standard).1 TRICARE Prime is the only option for which
             beneficiaries must enroll; active-duty members are automatically enrolled
             in Prime. Active-duty family members and retirees and their dependents
             under age 65 are also eligible to enroll in Prime. Retirees and their
             dependents and survivors over age 65 are not eligible to enroll in Prime,
             but can still obtain care in MTFs if space or resources are available.
             Beneficiaries can also obtain care from civilian providers. Beneficiaries
             who obtain care within the MTFs, including Prime enrollees, pay nothing
             for their outpatient visits. However, beneficiaries obtaining care from
             civilian providers are subject to out-of-pocket costs ranging from
             25 percent of the allowable charge for a TRICARE Standard office visit to
             a copayment of $6 or $12 for a Prime enrollee visiting a provider outside
             the MTF but in the TRICARE network.2

             Under section 712 of the National Defense Authorization Act for Fiscal
             Year 1996 (P.L. 104-106), DOD was required to establish priorities for
             accessing care within MTFs.3 Under DOD’s implementing policy, active-duty
             personnel have highest priority, followed by active-duty family members
             enrolled in Prime; retirees, their family members, and survivors enrolled in
             Prime; nonenrolled active-duty family members; and nonenrolled retirees,
             their family members, and survivors. In addition, DOD policy specifies that
             MTF commanders have the discretion to grant exceptions to the access


             1
             DOD previously provided health care under the Civilian Health and Medical Program of the
             Uniformed Services, a fee-for-service program.
             2
              Dependents of lower-rank active-duty members pay $6 for an outpatient visit, while dependents of
             higher-rank active-duty members, and retirees and their dependents and survivors, pay $12.
             3
              10 U.S.C. section 1097(c).



             Page 3                                         GAO/HEHS-99-168 Access to Defense Health Care
    B-279619




    priority rules for various reasons, such as giving groups or individuals
    higher priority to meet requirements of graduate medical education
    programs.

    To better ensure timely access to health care, DOD established appointment
    timeliness standards for Prime enrollees similar to the standards used in
    private-sector managed care programs. DOD’s standards, which apply to
    MTFs and the civilian network, established the following maximum wait
    times between the day a Prime enrollee requests an appointment with his
    or her primary-care physician and the actual date of the visit:

•   1 day for acute illness care, defined as visits requiring physician
    intervention and urgent in nature;
•   1 week for routine visits, defined as requiring physician intervention but
    nonurgent in nature;
•   4 weeks for well visits, defined as health maintenance and prevention, and
    nonurgent in nature; and
•   4 weeks for specialty care referrals from a primary-care physician to a
    specialist.

    In June 1998, DOD established a goal that at least 98 percent of acute and
    routine primary-care appointments for Prime enrollees should be
    scheduled within the time allowed by the standards. In March 1999, DOD
    lowered its 98-percent goal to what DOD considers a more achievable goal
    of 90 percent because most of the MTFs failed to meet the 98-percent goal.
    According to a DOD official, lowering the target to 90 percent provides
    more opportunity for MTFs to achieve DOD’s established goal.

    Section 713 of the Strom Thurmond National Defense Authorization Act
    for Fiscal Year 1999 (P.L. 105-261) established requirements for DOD to
    collect data on the timeliness of appointments in order to measure
    performance in meeting the primary-care access standards established
    under TRICARE.4 This requirement is consistent with the Government
    Performance and Results Act of 1993, which requires agencies to define
    their missions clearly, set goals, measure performance, and report on their
    accomplishments. DOD uses information from a Customer Satisfaction
    Survey to meet this legislative requirement. The Survey asks a sample of
    patients a number of questions about a specific visit with a particular
    medical provider in an MTF, including the severity of the need for the visit
    (such as whether the visit was urgent or routine), the number of days
    between requesting the appointment and the actual appointment date, and

    4
     10 U.S.C 1073.



    Page 4                              GAO/HEHS-99-168 Access to Defense Health Care
                       B-279619




                       their satisfaction with the care they received. DOD aggregates the results of
                       selected questions as a measure of how well or poorly MTFs as a group
                       performed in meeting the access standards. Information on appointment
                       timeliness is also contained in the appointment-scheduling module of the
                       CHCS system. CHCS is considered the primary health-care information
                       system of the military health-care system, and is used by all MTFs to
                       capture patient demographic information, schedule appointments, and to
                       order prescriptions and ancillary services. It also contains information on
                       the timeliness of scheduled appointments for virtually all clinics in the
                       MTFs.



                       Available data do not permit us or other analysts to precisely measure the
DOD Experiencing       extent to which DOD is meeting its access standards. However, after
Difficulty Achieving   correcting the DOD data for definitional discrepancies, we were able to
Appointment            develop an assessment of appointment timeliness. Our analysis shows that
                       appointments obtained by Prime enrollees, including active-duty members,
Timeliness Goals       were not always scheduled within the timeliness standards. Furthermore,
                       about the same percentage of appointments for nonenrolled beneficiaries
                       were scheduled within the standards as were those for active-duty and
                       Prime enrollees. These findings are consistent with DOD’s own analysis,
                       which concluded that its performance in appointment timeliness has not
                       met its expectations and goals.

                       According to DOD officials, there are several reasons why appointments for
                       active-duty and other Prime beneficiaries are not scheduled within the
                       standard. These include beneficiaries turning down an appointment that
                       was offered to them within the standard, and nonenrolled beneficiaries
                       being scheduled for appointments that otherwise would have been
                       available for active-duty and Prime beneficiaries. Several options exist to
                       increase the percentage of appointments scheduled within the standards
                       and improve access for active-duty and other enrollees. These options
                       include DOD conducting an assessment of the extent to which medical
                       readiness and training needs can be met without treating nonenrolled
                       beneficiaries, and stricter enforcement of the access to care priorities
                       based on this assessment. Also, requiring a copayment for care provided in
                       the MTFs could reduce the traditionally higher usage of military health-care
                       (as compared to utilization in private health plans) and help DOD achieve
                       its appointment timeliness goals.




                       Page 5                              GAO/HEHS-99-168 Access to Defense Health Care
                                              B-279619




DOD Has Not Achieved                          Our analysis of Customer Satisfaction Survey and CHCS appointment data
Timeliness Goal for                           indicates that DOD fell short of its original goal that 98 percent of acute and
Active-Duty and Prime                         routine primary-care appointments for Prime enrollees, including
                                              active-duty members, be scheduled within the period of time set in the
Enrollees                                     standards. For example, both data sources show that only about
                                              70 percent of routine appointments for Prime enrollees were scheduled
                                              within the required 1 week of the request for the appointment. While the
                                              98-percent goal was in place for the time period we analyzed, the
                                              performance for scheduling acute and routine appointments was even
                                              below DOD’s lowered goal of 90 percent. Table 1 summarizes DOD’s
                                              appointment standards, goals, and the percentage of appointments within
                                              the standards for active-duty members and other Prime enrollees.


Table 1: DOD Appointment Scheduling Standards, Goals, and Appointments Scheduled Within Standards for Active-Duty
and Other Prime Enrollees
                                                                                         Prime enrollee appointments
                                     Goal for             Active-duty appointments        scheduled within standard
                     Appointment     appointments to    scheduled within standard (%)     (excluding active-duty) (%)
                       scheduling        be scheduled                  Customer                                  Customer
                       standard for      within standard             Satisfaction                              Satisfaction
Appointment type       Prime enrollees   (%)                        Survey dataa        CHCS datab            Survey dataa         CHCS datab
Primary care acute     1 day             98                                     84                 91                     80                     92
Primary care routine   1 week            98                                     81                 81                     71                     65
Primary care well      4 weeks           No goal                                96                 91                     97                     81
Specialty referral     4 weeks           No goal                                94                 96                     94                     91
                                              a
                                              Data for 117 MTFs with clinics that had more than 200 visits per month for the 5-month period of
                                              May 1, 1998, to September 30, 1998. Sampling errors are no greater than +/-3 percentage points.
                                              b
                                              Data for appointments scheduled between October 1, 1997, and September 30, 1998, at five
                                              MTFs and between January 1, 1998, and December 31, 1998, at one MTF.

                                              Source: GAO analysis of DOD data.




Timeliness for Prime                          Among the beneficiaries who obtained appointments, the percentage of
Enrollees Similar to That                     appointments scheduled for active-duty and other Prime enrollees (those
for Nonenrolled                               with the highest priority) within the standards was similar to the
                                              percentage of appointments within the standards for nonenrolled
Beneficiaries                                 beneficiaries (who have the lowest priority). For example, the Customer
                                              Satisfaction Survey indicates that the percentage of acute and well
                                              primary-care appointments scheduled for active-duty members within the
                                              standards (84 and 96 percent, respectively) was similar to the percentage
                                              for nonenrolled appointments (81 percent and 95 percent, respectively).



                                              Page 6                                        GAO/HEHS-99-168 Access to Defense Health Care
                                         B-279619




                                         The CHCS data show that the appointment timeliness for other enrollees
                                         and the nonenrolled for all appointment types was also similar. Table 2
                                         summarizes the appointment timeliness for active-duty members, other
                                         Prime enrollees, and nonenrolled beneficiaries.


Table 2: Comparison of Appointments Scheduled Within Standards for Active-Duty, Other Prime Enrollees, and
Nonenrolled Beneficiaries
                                                       Prime enrollee appointments
                       Active-duty appointments         scheduled within standard           Nonenrolled appointments
                     scheduled within standard (%)      (excluding active-duty) (%)       scheduled within standard (%)
                          Customer                               Customer                                   Customer
                        Satisfaction                           Satisfaction                               Satisfaction
Appointment type       Survey dataa     CHCS datab            Survey dataa         CHCS datab            Survey dataa         CHCS datab
Primary care acute               84                 91                    80                  92                     81                     88
Primary care
routine                          81                 81                    71                  65                     69                     69
Primary care well                96                 91                    97                  81                     95                     78
Specialty referral               94                 96                    94                  91                     90                     93
                                         a
                                         Data for 117 MTFs with clinics that had more than 200 visits per month for the 5-month period of
                                         May 1, 1998, to September 30, 1998. Sampling errors are no greater than +/-3 percentage points.
                                         b
                                         Data for appointments scheduled between October 1, 1997, and September 30, 1998, at five
                                         MTFs and between January 1, 1998, and December 31, 1998, at one MTF.

                                         Source: GAO analysis of DOD data.



                                         While the data show similarities in the timeliness of appointments for
                                         enrolled and nonenrolled beneficiaries, it is important to note that the
                                         majority of the appointments—84 percent—were for enrolled
                                         beneficiaries. Also, there are no data showing the number of nonenrolled
                                         beneficiaries who were unable to obtain an appointment.


DOD Reports Performance                  In October 1998, DOD reported that it had a serious problem providing
in Meeting Access                        timely access to care, based on its analysis of Customer Satisfaction
Standards Has Not Met Its                Survey data for the May to July 1998 period. According to DOD, less than
                                         15 percent of the 115 MTFs included in its analysis were able to schedule
Expectations                             acute appointments within the standard, and DOD characterized the
                                         performance of many of the MTFs as “dismal.” Over the next 5 months, DOD
                                         said that although it had noticed some improvements, the achievement of
                                         the access standards continued to fall below its goal. In March 1999, the
                                         Executive Director of the TRICARE Management Activity stated that
                                         access must improve and tasked the Surgeons General and regional



                                         Page 7                                        GAO/HEHS-99-168 Access to Defense Health Care
                       B-279619




                       TRICARE management offices to work with MTFs to identify access
                       problems and make needed improvements.


Reasons Why            According to DOD, appointments may not be scheduled within the
Appointments Are Not   standards either because the beneficiary requests a later appointment for
Scheduled Within the   personal convenience, or because there are no appointment slots
                       available. DOD does not have data to identify the actual number of personal
Standards              convenience requests, but is planning some revisions to the CHCS
                       appointment system to capture information on whether the beneficiary
                       accepts the first offered appointment or requests a later one.

                       Appointment availability at the MTFs is also affected by the extent to which
                       care is provided to nonenrolled beneficiaries. Our review of CHCS
                       appointment data at six MTFs shows that about 16 percent of the
                       appointments were for beneficiaries who were not enrolled in TRICARE
                       Prime.5 According to DOD, providing medical care to other beneficiaries,
                       including those over age 65, provides medical proficiency training that
                       supports military medical readiness and training requirements. DOD has
                       made no analysis of the extent to which providing care to these
                       beneficiaries adversely affects the ability of the enrolled population to
                       obtain care or the effect of any resulting shortfall on readiness and
                       training.

                       Another factor that affects the availability of appointments for active-duty
                       and other Prime enrollees is the extent to which care in the MTFs is
                       overutilized by beneficiaries. Studies have shown that the per-capita
                       utilization of DOD health care services by military beneficiaries has
                       historically been much higher than in civilian health plans, due in part to
                       the lack of a cost-sharing requirement in MTFs. As we have previously
                       reported, research has shown that the lack of a cost-sharing requirement
                       leads to a higher utilization of health care.6 CBO has reported that sharing
                       costs with beneficiaries reduces health-care utilization. In its April 1999
                       report, CBO concluded that requiring a copayment from beneficiaries who
                       use MTFs would help curb excessive use.7 Furthermore, according to CBO,


                       5
                        Although we obtained appointment data from eight MTFs, we were only able to use data from six due
                       to limitations and discrepancies in the data that could not be corrected.
                       6
                        Defense Health Care: Challenges Facing DOD in Implementing Nationwide Managed Care
                       (GAO/T-HEHS-94-145, Apr. 19, 1994), and Addressing the Deficit: Budgetary Implications of Selected
                       GAO Work for Fiscal Year 1998 (GAO/OCG-97-2, Mar. 14, 1997).
                       7
                         Maintaining Budgetary Discipline: Spending and Revenue Options, CBO (Washington, D.C.,
                       Apr. 1999).



                       Page 8                                        GAO/HEHS-99-168 Access to Defense Health Care
                         B-279619




                         concerns that increasing cost-sharing requirements could discourage
                         beneficiaries from seeking necessary care are not well founded, especially
                         for the military health-care beneficiaries. CBO reports that cost-sharing
                         requirements do not prevent beneficiaries at ages and income levels
                         typical of military beneficiaries from seeking needed care.


Options to Improve       Several options could improve appointment timeliness for active-duty and
Appointment Timeliness   other Prime enrollees. One is to more rigorously implement the access
and Access               priorities. Some of the MTFs we visited had procedures to give appointment
                         priority to active-duty and other enrollees, such as specifying certain times
                         of day for active-duty and Prime enrollees to request appointments, after
                         which appointments were available for all beneficiaries, whether enrolled
                         or not. However, once beneficiaries are booked into appointments, the
                         appointment priority no longer exists. One option is to “bump” a
                         nonenrollee who has an appointment when an enrolled beneficiary needs
                         an appointment and none is available within the required time frame.
                         Second, if each MTF identified what percentage of the care provided to
                         nonenrollees was necessary to achieve their medical readiness and
                         training requirements, the rest of the care could be reallocated to
                         active-duty and other enrollees to improve their access. Third, establishing
                         a beneficiary copayment for care in the MTF could reduce the demand for
                         care in the MTF and free up more appointments for active-duty members
                         and other Prime enrollees. A standard practice used by commercial
                         managed care plans to bring about more appropriate utilization is
                         requiring enrolled beneficiaries to pay a copayment for care. Commercial
                         plan copayments for outpatient physician visits range from about $5 to
                         $15, with most beneficiaries paying $10 per visit. DOD’s civilian copayment
                         requirement of $6 or $12 per visit is consistent with commercial plans.

                         While these options are intended to improve appointment timeliness and
                         availability for enrolled beneficiaries, it is possible that the options may
                         cause some nonenrolled beneficiaries to seek care elsewhere and
                         experience higher out-of-pocket costs. The options may also affect the
                         timeliness of the care they receive. However, we did not evaluate the
                         extent to which this might occur.




                         Page 9                              GAO/HEHS-99-168 Access to Defense Health Care
                              B-279619




                              While DOD has been measuring appointment timeliness, the tools it uses
Weaknesses in Data            have several weaknesses that limit their usefulness in providing
Tools Prevent                 management information on the extent to which beneficiaries are
Accurate Assessment           obtaining appointments within the prescribed standards. For example,
                              problems associated with the design and administration of DOD’s Customer
of Appointment                Satisfaction Survey, such as the accuracy of beneficiary-reported data and
Timeliness                    the small number of visits included in the sample, prevent using the Survey
                              to measure MTFs’ performance against the appointment timeliness
                              standards. CHCS appointment system data can provide information on
                              appointment timeliness at each MTF, but cannot be used to compare the
                              data against the standards or across the military health care system unless
                              certain modifications are made. Although DOD has efforts under way to
                              improve the Survey, its reliance on beneficiaries’ recall of their
                              appointment experience is an inherent weakness that fundamentally limits
                              the Survey’s usefulness in this area. However, the efforts under way to
                              improve CHCS should address the weaknesses and make CHCS a good
                              source of data to measure and monitor MTF performance in scheduling
                              appointments within the standards.


Weaknesses Associated         While DOD’s Customer Satisfaction Survey provides information on how
With Survey Data Affect       beneficiaries perceive their health care experiences, it has weaknesses
Usefulness                    when used for measuring the performance of MTFs in meeting access
                              standards. For example, the quality of data is entirely dependent on the
                              beneficiary’s ability to remember the number of days it took to get a
                              specific appointment. However, because beneficiaries can receive the
                              survey up to 45 days after the appointment, they may have difficulty
                              accurately recalling their experience, thus calling into question the validity
                              of the Survey results.

                              Another weakness is that DOD relies on the respondents to correctly
                              classify their appointment types in their survey responses. DOD uses the
                              respondents’ classifications to determine which access standard should be
                              related to the appointments. The Survey asks each respondent to classify
                              the purpose of his or her visit as one of the following:

                          •   Care for illness or injury which the patient felt required him or her to see a
                              doctor right away;
                          •   Routine care for a nonurgent condition;
                          •   Well-patient visit for preventive care (checkup); or
                          •   Specialty care, referral visit.




                              Page 10                             GAO/HEHS-99-168 Access to Defense Health Care
B-279619




However, our discussions with beneficiaries revealed that they were
uncertain about the correct category for their visits. Beneficiaries did not
understand the difference between a routine visit for a nonurgent
condition and a well-patient visit for preventive care, and were unsure
about how to categorize a follow-up visit with either a primary-care
physician or specialist. Also, the design of the question intends that
primary-care visits would be identified by responses to the first three
categories and specialty care by selecting the fourth choice. Beneficiaries
said they would select the first option if they felt that they needed care
from a specialist right away. Our analysis of Survey responses confirmed
the potential for this error. We found that about two-thirds of the
responses from beneficiaries who received care in a specialty clinic
marked one of the first three categories, and thus were misclassified as
primary care.

Even if beneficiaries were able to interpret the questions and report their
experiences accurately, the sample size of the Survey is too small to
provide precise estimates of clinic performance. Each month, DOD
randomly selects 35 visits from each clinic that receives at least 200 visits
per month. Given the survey response rate of 40 percent, this sample size
yields about 14 responses per month for each clinic sampled. Even if data
were aggregated and analyzed every 6 months, a sample size of only
around 85 for the period could be expected, which would provide
information only on very large changes in performance at the clinic level.
For example, an increase in the appointments scheduled within the
standards from 70 percent to 80 percent would not represent a statistically
significant change based on a sample size of 85.

The response rate of the Survey also calls into question the validity of the
Survey results. While the Survey results provide information on those who
responded, DOD knows little about the experiences of the 60 percent or
more of surveyed beneficiaries who did not respond.8 Without conducting
a nonrespondent analysis, DOD cannot determine the extent to which their
health-care experiences were similar to or different from experiences of
patients who did complete the survey.9 Because the group of


8
 A DOD official involved with the Survey told us that the response rate has historically been 40 percent
or less.
9
 One way to assess the extent to which nonrespondents differ from respondents is to conduct a
nonresponse analysis. A nonresponse analysis is a technique used to determine the difference between
those who responded and those who did not respond to a survey, and the extent to which the
respondents represent the overall population. A nonresponse analysis for a mail survey is usually
conducted by administering the survey over the telephone.



Page 11                                         GAO/HEHS-99-168 Access to Defense Health Care
                            B-279619




                            nonrespondents is so large, their experiences, if different from the
                            experiences of the respondents, could dramatically change the survey
                            results.

                            In regard to measuring civilian provider appointment timeliness, DOD is
                            developing a survey modeled after the Customer Satisfaction Survey.
                            However, the limitations of the MTF Customer Satisfaction Survey would
                            also apply to the civilian survey. Thus, while the civilian survey might
                            provide some general indications about beneficiaries’ experiences with
                            civilian providers, it would not capture precise data needed to assess how
                            well the access standards are being met in the civilian network.


CHCS Appointment Data       Appointment data taken directly from the CHCS appointment scheduling
Need Modification to Be a   system used by all MTFs potentially could be the best data DOD has
Viable Measurement Tool     available to measure the performance of MTFs in meeting the access
                            standards. While CHCS data are not vulnerable to the limitations inherent in
                            the Customer Satisfaction Survey, the CHCS has other shortcomings that
                            limit its current usefulness as a tool to measure appointment timeliness.

                            A critical weakness of the CHCS data for appointment-measuring purposes
                            is that the appointment names used in the MTF’s appointment scheduling
                            system do not directly relate the types of visits to the standards. We found
                            that four of the eight MTFs in our study used appointment names within
                            their scheduling systems that could not be linked to only one appointment
                            timeliness standard. For example, at one MTF the appointment name
                            “PRIME” was used to book acute, routine, and well primary-care
                            appointments, which are each subject to different access standards. At
                            another MTF, the appointment name “PACU” was used to book acute and
                            routine appointments, while the name “ROUP” was used to book routine,
                            follow-up, and well appointments. In these cases, more than one
                            timeliness standard would be applicable and the MTF would not know
                            which standard to use to measure its performance in making timely
                            appointments. Unless MTFs link their appointment names to a single
                            standard, they will be unable to determine the extent to which their
                            appointments are in compliance with the appointment timeliness
                            standards.

                            In addition, the lack of standard appointment names among the MTFs
                            prevents DOD from consolidating individual facility CHCS data into regional
                            or systemwide data. Under DOD’s current procedures, each MTF has the
                            flexibility to design a unique appointment system. This practice hampers



                            Page 12                            GAO/HEHS-99-168 Access to Defense Health Care
                          B-279619




                          DOD’s ability to collect and monitor appointment data across the military
                          health-care system. For example, DOD would have to know exactly which
                          names were used in every MTF’s appointment system and which of the
                          access standards applied to the appointment name. In our review of
                          appointment names in use at eight MTFs, we found 14 different names for
                          appointments associated with the timeliness standard for acute
                          appointments, 18 different names for appointments associated with the
                          timeliness standard for routine appointments, and 35 different names for
                          appointments associated with the timeliness standard for well visits. Even
                          though some of these MTFs could associate the appointment names they
                          used with the applicable timeliness standard, the lack of consistent and
                          standard appointment names across a system of more than 450 MTFs with
                          potentially thousands of appointment names would make any effort to
                          collect, monitor, and regularly report on systemwide appointment data a
                          complex and complicated undertaking.


DOD Attempting to         DOD  officials told us they recognize the weaknesses of the Survey and CHCS
Resolve Data Weaknesses   data and have some efforts under way or planned to address some of the
                          weaknesses. With regard to the Survey, they acknowledged that
                          beneficiaries are confused when trying to categorize their appointments,
                          especially because the Survey does not define the categories. According to
                          DOD officials, providing some general definitions with examples of
                          appointments could help beneficiaries responding to the Survey. However,
                          they believe it is not possible to provide sufficient examples to completely
                          eliminate the confusion and ensure correct categorization, and are not
                          planning any revisions to that Survey question. The officials also agreed
                          that memory recall about the number of days it took to get an appointment
                          was a concern. DOD officials said that sending the Survey closer to the
                          appointment date might improve memory recall, but the administrative
                          tasks associated with selecting the sample and mailing the Survey could
                          not be hastened. Related to the lack of a nonrespondent analysis, DOD
                          recently decided to conduct the first analysis of nonrespondents in fiscal
                          year 2000.

                          With regard to the CHCS data, DOD has efforts under way or planned that
                          should address the critical weaknesses that affect the usefulness of the
                          data in measuring appointment timeliness. DOD officials told us that a
                          policy is being developed requiring MTFs to correlate their primary-care
                          appointments to the three timeliness standards and to standardize the
                          appointment names across the military health-care system. Other
                          enhancements are also planned that will improve the accuracy of CHCS



                          Page 13                            GAO/HEHS-99-168 Access to Defense Health Care
              B-279619




              appointment timeliness data. Officials estimate that CHCS data would be
              reliable for monitoring and measuring access in MTFs by March 2000 after
              these changes and improvements are tested and implemented throughout
              the military health-care system. If successful, DOD could rely on CHCS and
              cease using the Survey as a means of measuring compliance with the
              timeliness standards at the MTF, regional, and systemwide level.


              Active-duty and other Prime enrollees have not been able to obtain
Conclusions   appointments within the prescribed timeliness standards to the extent that
              DOD expected when it first established goals for TRICARE. Moreover, the
              performance in meeting standards is about the same for active-duty
              members, who have the highest priority, and nonenrolled beneficiaries,
              who have the lowest priority. In some cases, appointments are scheduled
              outside the standards due to the beneficiary’s request for a later
              appointment to meet personal needs. However, appointments within the
              standards for enrolled beneficiaries may not be available because
              nonenrolled beneficiaries have filled available appointment slots ahead of
              them. Providing care to nonenrollees, especially those who are eligible to
              enroll in Prime, counters the program’s intention that eligible beneficiaries
              enroll, and reinforces some beneficiaries’ view that they can still obtain
              care in the MTFs without enrolling.

              There are several options DOD could test to improve the availability of
              appointments for active-duty and other enrolled beneficiaries. These
              include more vigorously enforcing systemwide access priorities, to the
              extent of giving appointments booked for nonenrollees to enrolled
              beneficiaries in need of an appointment within the standard. Also,
              eliminating care to nonenrolled beneficiaries that exceeds medical training
              requirements could result in more available appointments. Lastly,
              instituting a copayment in the MTFs could lead to more appropriate
              utilization of care in MTFs, thereby opening up additional appointment slots
              for enrollees. While copayments could help improve appointment
              timeliness, potential benefits actually go well beyond this. Copayments
              would also serve to equalize the cost-sharing for all beneficiaries,
              regardless of whether they receive care from military or civilian providers,
              by eliminating the inherent inequity of providing more generous health
              benefits to those who live near an MTF. It would also allow physicians to
              refer beneficiaries to the most appropriate provider—whether military or
              civilian—without regard to the financial implications of the referral for the
              beneficiary.




              Page 14                            GAO/HEHS-99-168 Access to Defense Health Care
                  B-279619




                  The two data tools that provide information on appointment timeliness in
                  MTFs have significant weaknesses that affect the accuracy and sufficiency
                  of the data. DOD is undertaking efforts to address the weaknesses of the
                  CHCS appointment system by requiring MTFs to associate appointment
                  names with the access standards and by establishing standard
                  appointment names across the system. Regarding the Customer
                  Satisfaction Survey, we agree that it provides DOD with meaningful
                  information on how beneficiaries feel about their health-care experiences
                  and can be used for this purpose. Furthermore, the planned analysis of
                  nonrespondents will further improve the data. However, two remaining
                  weaknesses in the Survey—beneficiary categorization of appointments
                  and reliance on memory recall—are sufficiently significant to continue to
                  call into question the validity of the results as a measure of either civilian
                  providers’ or MTFs’ performance against the standards. While we recognize
                  the challenges DOD faces in obtaining comprehensive information on
                  civilian providers’ performance, accurate and appropriate data to measure
                  how well MTFs are meeting the standards can and should be obtained from
                  MTF data sources, not from beneficiaries. Therefore, the CHCS system
                  should be the primary data source for determining MTF compliance with
                  the access standard. In our view, it is imperative that DOD implement
                  changes to the CHCS system as soon as possible so that it can meet its
                  responsibilities to beneficiaries and more effectively manage access to the
                  MTFs.



                  We recommend that the Secretary of Defense direct the Assistant
Recommendations   Secretary of Defense (Health Affairs) to measure and monitor
                  beneficiaries’ access to health care in a more comprehensive and accurate
                  manner by directing that CHCS be used in lieu of the Customer Satisfaction
                  Survey to measure compliance with the appointment timeliness standards
                  in the MTFs and that the necessary modifications be made to CHCS so that
                  appointment names are linked to the appropriate access standard and
                  standardized across the military health-care system. The Secretary should
                  direct that the results be reported at all levels—individual facility, service-
                  and system-wide, and by the various beneficiary categories.

                  The Secretary should also direct a test of a policy that appointments
                  scheduled for nonenrolled beneficiaries are subject to cancellation if an
                  active-duty member or other Prime enrollee requests care and no other
                  appointment is available within the access standard. This test could be
                  implemented in those MTFs having the greatest difficulty scheduling




                  Page 15                              GAO/HEHS-99-168 Access to Defense Health Care
                  B-279619




                  active-duty members and other Prime enrollees within the access
                  standards.

                  The Secretary should also test the option of instituting copayments within
                  the MTFs comparable to those in the civilian networks to help bring about
                  more appropriate utilization of military care and thus free up appointment
                  space.


                  We provided a draft of this report to DOD for review and comment, but DOD
Agency Comments   has not provided comments.


                  As agreed with your offices, we are sending copies of this report to the
                  Honorable William C. Cohen, Secretary of Defense, and will make copies
                  available to others upon request. Please contact me on (202) 512-7111 or
                  Michael T. Blair, Jr., Assistant Director, on (404) 679-1944 if you or your
                  staff have any questions. Other major contributors to this report are listed
                  in appendix II.




                  Stephen P. Backhus, Director
                  Veterans’ Affairs and Military Health
                    Care Issues




                  Page 16                            GAO/HEHS-99-168 Access to Defense Health Care
Page 17   GAO/HEHS-99-168 Access to Defense Health Care
Contents



Letter                                                                                        1


Appendix I                                                                                   20

Scope and
Methodology
Appendix II                                                                                  24

GAO Contacts and
Staff
Acknowledgments
Tables             Table 1: DOD Appointment Scheduling Standards, Goals, and                  6
                     Appointments Scheduled Within Standards for Active-Duty and
                     Other Prime Enrollees
                   Table 2: Comparison of Appointments Scheduled Within                       7
                     Standards for Active-Duty, Other Prime Enrollees, and
                     Nonenrolled Beneficiaries
                   Table I.1: Primary and Specialty Care Clinics at Eight MTFs               22
                     Providing Appointment Data




                   Abbreviations

                   CBO       Congressional Budget Office
                   CHCS      Composite Health Care System
                   DOD       Department of Defense
                   MTF       military treatment facility


                   Page 18                         GAO/HEHS-99-168 Access to Defense Health Care
Page 19   GAO/HEHS-99-168 Access to Defense Health Care
Appendix I

Scope and Methodology


                        To obtain information on the Department of Defense’s (DOD) access
Scope                   policies and measures, we met with officials in the Office of the Assistant
                        Secretary of Defense (Health Affairs) and the TRICARE Management
                        Activity, who are responsible for managing the military health-care
                        program. We also spoke with staff of the three Services’ Surgeons General
                        and three of the TRICARE managed care support contractors. We
                        discussed local access policies and appointment procedures with officials
                        at 15 military treatment facilities (MTF) and visited 12 of the facilities. We
                        also reviewed DOD standards for primary- and specialty-care appointment
                        timeliness and DOD’s policy on priority for access to care in MTFs.

                        We researched the access standards used by commercial managed care
                        plans and how they measure their performance against these standards.
                        We reviewed accreditation standards related to access to care from two
                        health-care industry accreditation bodies—the Joint Commission on
                        Accreditation of Healthcare Organizations and the National Committee for
                        Quality Assurance. We also gathered information on appointment
                        timeliness and access standards used by individual private-sector health
                        care plans, as well as cost-sharing requirements.


                        For the purpose of this study, we defined access as appointment
Methodology             timeliness—measuring the number of elapsed days between the date that
                        the beneficiary requests an appointment and the scheduled appointment
                        date. We selected this measure because (1) it was the access measure for
                        which DOD had established criteria or standards against which its
                        performance could be measured, and (2) appointment data were available
                        throughout the military health care system from the Customer Satisfaction
                        Survey and the Composite Health Care System (CHCS). However, because
                        of limitations in DOD’s data from both sources, we could not use the data as
                        they existed in DOD’s systems, and designed a methodology and analysis
                        approach (discussed below) to minimize the effect of the limitations.


Customer Satisfaction   We made several adjustments to the Survey data to minimize the
Survey                  weaknesses and correct discrepancies. Because we had concerns about
                        whether beneficiaries had correctly classified their visits, we did not use
                        their classification from the Survey. Instead, we used the sample selection
                        data that provided information on the clinics the beneficiaries visited.
                        From these data, we identified primary-care visits, which we defined as a
                        visit to one of the four clinics DOD now considers to be a primary-care
                        clinic throughout the military health care system—family practice, primary



                        Page 20                             GAO/HEHS-99-168 Access to Defense Health Care
                        Appendix I
                        Scope and Methodology




                        care, flight medicine, or pediatrics. In consultation and agreement with
                        DOD officials responsible for administering the Survey, we considered all
                        appointments not associated with one of the four primary clinics to be a
                        specialty appointment. We did not include information on visits from
                        certain clinics for which the access standards were not applicable, such as
                        mental health clinics and emergency departments. Our analysis covered
                        survey responses for appointments in the 5-month period from May 1,
                        1998, to September 30, 1998. We selected the beginning date of May 1,
                        1998, so that our analysis contained only responses since the survey
                        instrument was revised in May 1998. September 30, 1998, was the latest
                        date for which data were available at the time of our request. To determine
                        appointment timeliness and compliance with the primary-care standards,
                        we relied on the beneficiaries’ response to the question asking them how
                        many days it took to obtain the appointment and compared it to the
                        relevant access standard determined by the clinic of their visit, as
                        discussed above, and the beneficiaries’ categorization as to the urgency of
                        the visit (for those determined to be primary-care visits). In analyzing the
                        data by beneficiary category and enrollment status, we found and
                        corrected discrepancies. We considered all active-duty respondents as
                        enrolled in Prime, and retirees over age 65 as not enrolled in Prime,
                        regardless of how they responded to the survey question about their
                        enrollment status.


CHCS Appointment Data   As a result of our discussions with MTF officials, we confirmed that the
                        CHCS appointment system could provide the information we needed to
                        assess the appointment timeliness at the MTF level. This information
                        included beneficiary category and enrollment status, the date the
                        beneficiary requested the appointment, the date of the scheduled
                        appointment, and appointment type or name. Our analysis of CHCS data
                        also confirmed that the key limitation with these data was determining for
                        each appointment name in the system which appointment timeliness
                        standard was relevant.

                        We asked eight MTFs to provide the appointment names used in their
                        scheduling system that were subject to each of the access standards for
                        their primary care.10 We also asked them to provide the same information
                        on appointment names for selected specialty clinics. After obtaining the
                        information on the appointment name used, we asked each of the eight
                        MTFs to provide us with 12 months of appointment data for the identified


                        10
                         We selected the MTFs to represent the different Services, TRICARE contractors, areas of the
                        country, and size of medical facility.



                        Page 21                                       GAO/HEHS-99-168 Access to Defense Health Care
                                            Appendix I
                                            Scope and Methodology




                                            primary-care and specialty clinics. Table I.1 shows the eight MTFs and the
                                            clinics for which we obtained data.


Table I.1: Primary and Specialty Care Clinics at Eight MTFs Providing Appointment Data
MTF                                Primary care clinics              Specialty care clinics
Fort Benning/Martin Army           Internal Medicine, Family Practice Gynecology, Internal Medicine, Optometry, Podiatry, Allergy,
Community Hospital                 (2 clinics), Aviation Medicine     Physical Therapy, Orthopedics, Ophthalmology, Dermatology,
                                                                      Nutrition, Obstetrics, Otorhinolaryngology, Urology
Davis-Monthan AFB/355th Medical    Family Practice (2 clinics)         Gynecology, Internal Medicine, Optometry, Physical Therapy,
Group                                                                  General Surgery, Orthopedics, Dermatology
Fort Hood/Darnall Army Community Family Care (3 clinics), Pediatrics   Gynecology, Internal Medicine, Optometry, Podiatry, Allergy,
Hospital                         (3 clinics)                           Physical Therapy, General Surgery, Orthopedics,
                                                                       Ophthalmology, Dermatology, Nutrition, Neurology,
                                                                       Obstetrics, Women’s Health, Urology
Naval Hospital Oak Harbor          Family Practice (3 clinics),        Gynecology, Internal Medicine, Optometry, Physical Therapy,
                                   Primary Care (3 clinics),           General Surgery, Obstetrics
                                   Pediatrics (3 clinics), Aviation
                                   Medicine (3 clinics)
Mountain Home AFB/366th Medical    Primary Care (3 clinics)            Gynecology, Internal Medicine, Optometry, Physical Therapy,
Group                                                                  General Surgery, Obstetrics
Wilford Hall Medical Center        Family Medicine, Internal           Gynecology, Obstetrics, Optometry, Podiatry, Allergy,
                                   Medicine, Women’s Health,           Physical Therapy, Cardiology, General Surgery,
                                   General Pediatrics                  Ophthalmology, Dermatology, Nutrition, Neurology,
                                                                       Otorhinolaryngology, Orthopedics, Urology
Fort Rucker/Lyster Army            Aviation Medicine, Ambulatory       Gynecology, Internal Medicine, Optometry, Physical Therapy,
Community Hospital                 Care, Family Practice               General Surgery, Orthopedics, Ophthalmology, Dermatology
Naval Hospital Jacksonville        Primary Care, Family Practice (2    Allergy, Dermatology, General Surgery, Gynecology, Internal
                                   clinics), Pediatrics                Medicine, Neurology, Nutrition, Ophthalmology, Orthopedics,
                                                                       Otorhinolaryngology, Physical Therapy, Urology

                                            From the clinic appointment data, we calculated the number of days
                                            between the date the beneficiary requested the appointment and the date
                                            of the scheduled appointment. We analyzed these data by different
                                            variables, including beneficiary category and whether the beneficiary was
                                            enrolled in TRICARE Prime. For our analysis, we assumed that the
                                            patient’s visit to the provider was in fact for the type of visit indicated by
                                            the appointment name and timeliness standard. We could not correct the
                                            data to reflect any instances in which patients were scheduled into
                                            appointment types that were different from the type they requested.
                                            Another discrepancy we found was that in some cases active-duty
                                            personnel were recorded as not enrolled in Prime, when they are actually
                                            considered automatically enrolled. We corrected for this by recoding all
                                            active-duty as enrolled in Prime, regardless of the enrollment status field
                                            in the CHCS data. Similarly, we recoded all retirees over age 65 as not




                                            Page 22                                  GAO/HEHS-99-168 Access to Defense Health Care
Appendix I
Scope and Methodology




enrolled despite the enrollment status shown in the data. We ultimately
had to exclude data from two MTFs—Fort Hood and Wilford Hall Medical
Center—because some of their appointment types were associated with
more than one standard, which precluded comparing the data against the
access standards.




Page 23                          GAO/HEHS-99-168 Access to Defense Health Care
Appendix II

GAO Contacts and Staff Acknowledgments


                  Michael T. Blair, Jr., (404) 679-1944
GAO Contacts      Nancy T. Toolan, (404) 679-1983


                  In addition to those mentioned above, Sylvia D. Jones, Linda S. Lootens,
Staff             Deborah L. Edwards, and Beverly Brooks-Hall made key contributions to
Acknowledgments   this report.




(101613)          Page 24                             GAO/HEHS-99-168 Access to Defense Health Care
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested