oversight

Nursing Home Quality: CMS Should Improve Efforts to Monitor Implementation of the Quality Indicator Survey [Reissued on March 9, 2012]

Published by the Government Accountability Office on 2012-02-01.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Requesters




February 2012
                NURSING HOME
                QUALITY
                CMS Should Improve
                Efforts to Monitor
                Implementation of the
                Quality Indicator
                Survey



                   This report was revised on March 9, 2012, to include omitted
                   page 2 of the agency comment letter from the Department of
                   Health and Human Services in appendix I, page 23.




GAO-12-214
                                            February 2012

                                            NURSING HOME QUALITY
                                            CMS Should Improve Efforts to Monitor
                                            Implementation of the Quality Indicator Survey
Highlights of GAO-12-214, a report to
congressional requesters




Why GAO Did This Study                      What GAO Found
The Centers for Medicare & Medicaid         CMS has commissioned three studies to evaluate the QIS-based survey process
Services (CMS) is responsible for           but does not routinely monitor the extent to which the objectives established for
ensuring that nursing homes receiving       the QIS are being met. The first two studies, completed in 2006 and 2007, were
federal funding meet federal quality        commissioned to determine whether the new survey process could be used in
standards. CMS contracts with state         real-world settings and whether the QIS was helping the agency meet several of
survey agencies to conduct periodic         the objectives established for the process. The studies’ findings suggested that
inspections of nursing homes using a        surveyors could use the survey process and recommended changes intended to
CMS-developed survey process.               help improve the QIS. CMS officials reported taking steps to address the studies’
Weaknesses in the survey process led
                                            findings and recommendations. In 2009, CMS commissioned a third study that
CMS to develop and begin
                                            was completed in 2011 and identified aspects of the QIS process that could
implementing the Quality Indicator
Survey (QIS), a new electronic survey.
                                            affect the consistency with which surveyors identify quality problems. For
CMS developed the QIS to achieve            example, the study found that during resident interviews, surveyors did not
several objectives, including improving     consistently probe for further information when provided with incomplete
the efficiency, accuracy, and               responses to interview questions. However, CMS does not have the means to
consistency of the survey process.          routinely monitor the extent to which the QIS is helping improve the survey
CMS expects the QIS to be                   process as intended. Such routine, ongoing monitoring would be consistent with
implemented nationally by 2018. We          federal internal control standards and could include the use of performance goals
examined the extent to which CMS            and measures. CMS does have access to some data, such as the amount of
(1) evaluates whether progress is           time surveyors have spent inspecting facilities, that could be used to help
being made in meeting the objectives        develop performance goals and measures.
of the QIS and (2) monitors and
facilitates states’ implementation of the   CMS has taken some steps to monitor and facilitate states’ implementation of the
QIS. To do this, GAO reviewed               QIS-based routine survey, but CMS’s efforts are not systematic. As part of the
relevant CMS documents, including           agency’s efforts to monitor states’ implementation, CMS primarily uses quarterly
CMS-commissioned studies of the             teleconferences with state survey agency officials to obtain information on the
QIS, policies, and guidance. GAO also       extent to which each state has completed training all its surveyors to use the
interviewed officials from CMS’s            QIS. However, states may not always participate in the teleconferences, and
Central and all 10 Regional Offices as      those that do may not provide complete information on their progress. As a
well as officials from a judgmental         result, the information CMS obtains through its monitoring of states’ progress
sample of nine state survey agencies.       may be incomplete. For example, CMS was unable to provide GAO with
                                            information on training completion dates of all state surveyors in states that
What GAO Recommends                         completed training. Although CMS reported it plans to develop a more systematic
GAO recommends that CMS develop a           process for monitoring states’ implementation progress, it has not established a
means to routinely monitor the extent       time line for doing this. To help facilitate states’ implementation of the QIS, CMS
to which progress is being made in          provides states with guidance, gives presentations, and offers states
meeting the objectives of the QIS and       opportunities to share their implementation experiences through quarterly
systematic methods for monitoring and       teleconferences. However, CMS does not have a systematic method for
facilitating states’ efforts to implement   obtaining, compiling and sharing information on state experiences, especially
the QIS. The Department of Health and       information on approaches states have taken to help facilitate implementation of
Human Services fully concurred with         the QIS. Systematically sharing such information—for example, through CMS’s
all of the recommendations.                 annual conference in which all state survey agencies participate—could help the
                                            agency facilitate implementation in states that have not begun QIS
                                            implementation.


View GAO-12-214. For more information,
contact Linda Kohn at (202) 512-7114 or
kohnl@gao.gov.

                                                                                    United States Government Accountability Office
Contents


Letter                                                                                                  1
             Background                                                                                4
             CMS Commissioned Studies to Evaluate the QIS, but Does Not
               Routinely Monitor Progress toward Meeting the Objectives of
               the QIS                                                                                  6
             CMS Has Taken Steps to Monitor and Facilitate States’
               Implementation of the QIS, but CMS’s Efforts Are Not
               Systematic                                                                              12
             Conclusions                                                                               17
             Recommendations for Executive Action                                                      18
             Agency Comments and Our Evaluation                                                        18

Appendix I   Comments from the Department of Health and Human Services                                 21




             Abbreviations

             CMS               Centers for Medicare & Medicaid Services
             HHS               Department of Health and Human Services
             QIS               Quality Indicator Survey
             RTI               Research Triangle Institute International
             SSA               Social Security Act

             This is a work of the U.S. government and is not subject to copyright protection in the
             United States. The published product may be reproduced and distributed in its entirety
             without further permission from GAO. However, because this work may contain
             copyrighted images or other material, permission from the copyright holder may be
             necessary if you wish to reproduce this material separately.




             Page i                                      GAO-12-214 Nursing Home Quality Assessment
United States Government Accountability Office
Washington, DC 20548




                                   February 1, 2012

                                   The Honorable Herb Kohl
                                   Chairman
                                   Special Committee on Aging
                                   United States Senate

                                   The Honorable Charles E. Grassley
                                   Ranking Member
                                   Committee on the Judiciary
                                   United States Senate

                                   Nursing homes in the United States provide skilled nursing, rehabilitation,
                                   and custodial care to approximately 1.5 million elderly and disabled
                                   individuals. Federal and state governments share responsibility for
                                   ensuring that the approximately 15,700 nursing homes in the nation
                                   receiving payment from Medicare and/or Medicaid programs provide
                                   quality care in a safe environment for their highly vulnerable residents.
                                   Specifically, the Centers for Medicare & Medicaid Services (CMS), an
                                   agency within the Department of Health and Human Services (HHS), is
                                   responsible for ensuring that nursing homes receiving payment from
                                   Medicare and Medicaid programs meet federal quality standards
                                   established pursuant to the Omnibus Budget Reconciliation Act of 1987. 1
                                   To meet this responsibility, CMS, among other activities, contracts with
                                   state survey agencies to conduct on-site nursing home inspections, which
                                   consist of routine surveys and complaint investigations, using a process
                                   developed by CMS. 2

                                   In 1998, we found weaknesses in the traditional process used to conduct
                                   routine surveys, including poor measurement of serious care problems in




                                   1
                                    Pub. L. No. 100-203, Title IV, Subtitle C, §§ 4201 et seq., 101 Stat. 1330, 1330-160, et
                                   seq. (amending Titles XVIII and XIX of the Social Security Act). According to CMS,
                                   combined Medicare and Medicaid payments for nursing home care in 2010 were about
                                   $91 billion.
                                   2
                                    Under contract with CMS, state survey agencies conduct inspections of health care
                                   facilities—such as nursing homes, home health agencies, intermediate care facilities for
                                   the mentally retarded, and hospitals—that participate in Medicare and Medicaid.




                                   Page 1                                       GAO-12-214 Nursing Home Quality Assessment
nursing homes. 3 For example, we reported that the resident sample used
in the traditional process was inadequate and limited a surveyors’ ability
to identify care problems and their prevalence within a facility. To address
the problems we reported, and in response to criticism from others, CMS
revised the routine survey process by developing the Quality Indicator
Survey (QIS). According to CMS, the QIS was developed to help the
agency meet several objectives, including improving surveyors’
documentation of quality concerns; improving the efficiency of the survey
process by focusing resources on facilities, and on areas within facilities,
with the greatest quality concerns; and improving the accuracy and
consistency with which surveyors identify deficiencies.

CMS began implementation of the QIS-based routine survey process in
2005, by launching demonstrations of the QIS in five states. To
implement the QIS, CMS relies on the state survey agencies, as these
agencies are largely responsible for the administration of nursing home
inspections. 4 To help assist state survey agencies in implementing the
QIS, CMS provides policies, guidance and some training for state
surveyors. CMS officials told us that as of September 2011, 26 states had
trained or started training surveyors to use the QIS and 27 states had not
yet started the training. CMS expects to complete implementation of the
QIS by 2018 in all 50 states, the District of Columbia, Puerto Rico, and
the U.S. Virgin Islands. 5

You have raised questions about CMS’s efforts to implement the QIS. We
previously provided you with information on the status of the agency’s
efforts to implement the QIS nationally. 6 This report focuses on certain
aspects of CMS’s implementation of the QIS and will examine the extent
to which CMS (1) evaluates whether progress is being made in meeting



3
 See GAO, California Nursing Homes: Care Problems Persist Despite Federal and State
Oversight, GAO/HEHS-98-202 (Washington, D.C.: July 27, 1998).
4
 State survey agencies administer and have discretion over many survey activities and
policies, including hiring and retaining a surveyor workforce, training surveyors, and
conducting supervisory reviews of surveys. For example, state survey agencies are
largely responsible for training all surveyors in their states to use the QIS; in 2011, there
were roughly 4,700 surveyors nationwide.
5
 In this report, we use the term “states” to refer to this group of states and territories.
6
 See GAO, Nursing Home Quality: Implementation of the Quality Indicator Survey,
GAO-11-403R (Washington, D.C.: Apr. 6, 2011).




Page 2                                         GAO-12-214 Nursing Home Quality Assessment
the objectives of the QIS and (2) monitors and facilitates states’
implementation of the QIS.

To examine the steps CMS has taken to evaluate the QIS, we reviewed
relevant CMS documents, including the agency’s plan for evaluation and
improvement of the QIS and independent studies of the QIS
commissioned by the agency. We also interviewed CMS officials from
CMS’s Central Office. We asked these officials for information on CMS’s
efforts to evaluate the QIS, including any efforts to routinely monitor the
QIS, a practice consistent with federal internal control standards. 7
Additionally, we interviewed the CMS contractors—Abt Associates Inc.
and Research Triangle Institute International (RTI)—that conducted the
independent studies of the QIS.

To examine the extent to which CMS monitors and facilitates states’
implementation of the QIS, we reviewed CMS policies on the QIS as well
as guidance the agency provided to state survey agencies that described
their respective roles in implementing the QIS. We reviewed CMS’s plan
for implementing the QIS nationally. We also interviewed officials from
CMS’s Central Office and all 10 Regional Offices to learn more about the
agency’s efforts to monitor and facilitate states’ implementation of the
QIS. 8 We also analyzed this information to determine whether CMS’s
monitoring and facilitation efforts are consistent with federal internal
control standards. 9 Additionally, we conducted structured interviews with
state survey agency officials from a judgmental sample of nine states
(Arizona, Colorado, Connecticut, Florida, Maryland, Minnesota, New
York, Ohio, and West Virginia), to obtain information about these states’
experience with implementing the QIS, including the extent to which CMS
monitored their progress with implementation and helped to facilitate
implementation in their state. We selected our sample from the 21 states
that had begun implementation as of February 2011. These states were
selected based on criteria such as variation in the states’ QIS
implementation status, the percentage of nursing home inspections
conducted using the QIS-based process within a state, and state size.


7
 See GAO, Standards for Internal Control in the Federal Government,
GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999).
8
 CMS’s Regional Offices are located in Atlanta, Boston, Chicago, Dallas, Denver, Kansas
City, New York, Philadelphia, Seattle, and San Francisco.
9
See GAO/AIMD-00-21.3.1.




Page 3                                     GAO-12-214 Nursing Home Quality Assessment
                           The findings from our structured interviews are limited to the nine states
                           in our sample and are not representative; therefore, we will not be able to
                           generalize these findings to all states.

                           We conducted this performance audit from June 2011 through December
                           2011 in accordance with generally accepted government auditing
                           standards. Those standards require that we plan and perform the audit to
                           obtain sufficient, appropriate evidence to provide a reasonable basis for
                           our findings and conclusions based on our audit objectives. We believe
                           that the evidence obtained provides a reasonable basis for our findings
                           and conclusions based on our audit objectives.


                           Oversight of nursing homes is a shared federal-state responsibility. Based
Background                 on statutory requirements, CMS defines standards that nursing homes
                           must meet to participate in the Medicare and Medicaid programs and
                           contracts with state survey agencies to assess whether homes meet
                           these standards. A range of statutorily defined sanctions is available to
                           CMS and the states to help ensure that homes maintain compliance with
                           federal quality requirements. CMS also is responsible for monitoring the
                           adequacy of state survey activities.


Nursing Home Inspections   Every nursing home that receives payment from Medicare and/or
                           Medicaid must undergo a nursing home inspection. These inspections
                           consist of routine surveys that are conducted on average every
                           12 months and complaint investigations that are conducted in response to
                           allegations of quality problems. 10 During a routine survey, teams of state
                           surveyors—generally consisting of registered nurses, social workers,
                           dieticians, or other specialists—evaluate a nursing home’s compliance
                           with federal nursing home quality standards. There are approximately 200
                           different standards that focus on the delivery of care, resident outcomes
                           and facility conditions. The standards are grouped into 15 categories,
                           such as Quality of Life, Resident Assessment, Quality of Care, and
                           Administration. For example, there are 23 standards within the Quality of
                           Care category ranging from the prevention of pressure sore development
                           to keeping the resident environment as free of accident hazards as is


                           10
                             Every nursing home is required to receive an inspection not less than once every 15
                           months and the statewide average interval for these surveys must not exceed 12 months.
                           42 C.F.R. § 488.308 (2010).




                           Page 4                                    GAO-12-214 Nursing Home Quality Assessment
              possible. During nursing home inspections, state surveyors may cite
              deficiencies—areas in which facilities fail to meet federal quality
              standards—and nursing homes must prepare a correction plan to address
              most deficiencies. 11 State surveyors may also conduct what are known as
              revisit surveys to ensure that the homes corrected or took steps to correct
              identified deficiencies.

              To ensure the reliability of state survey agencies’ inspections of nursing
              homes, federal survey teams composed of CMS staff from its 10 Regional
              offices conduct what are known as federal monitoring surveys, which
              include comparative and observational surveys. In a comparative survey,
              a federal survey team conducts an independent inspection of a home
              recently surveyed by a state survey agency in order to compare and
              contrast its findings with those of the state survey team. In an
              observational survey, federal surveyors accompany a state survey team
              to a nursing home to evaluate the team’s on-site survey performance and
              ability to document deficiencies. The federal monitoring surveys are to be
              conducted annually in at least 5 percent of state-surveyed Medicare and
              Medicaid nursing homes in each state. 12 CMS has developed the QIS-
              based tool for routine surveys and plans to develop QIS-based tools for
              all other types of nursing home inspections: revisit surveys, complaint
              investigations and federal monitoring surveys. 13


QIS Process   The QIS process is similar to the traditional survey method. 14 For
              example, both the QIS and the traditional survey process involve
              analyzing information available about the nursing home facilities and their


              11
                For serious deficiencies, CMS generally imposes sanctions, which may result in the
              termination of a nursing home’s participation in the Medicare and Medicaid programs.
              12
                Under the Social Security Act (SSA), HHS is required to conduct a certain number of
              federal validation surveys, also referred to as federal monitoring surveys, each year. In no
              case is HHS to survey less than a total of five nursing homes in a state per year. SSA
              §1819(g)(3)(B)(Medicare) (codified at 42 U.S.C. §1395i-3(g)(3)(B)); SSA §
              1919(g)(3)(B)(Medicaid) (codified at 42 U.S.C. § 1396r(g)(3)(B)).
              13
                Revisit surveys ensure that nursing homes remediate certain deficiencies. A complaint
              investigation generally focuses on a specific allegation regarding a resident’s care or
              safety. Federal monitoring surveys are done to ensure the reliability of state survey
              agencies’ routine surveys.
              14
                See GAO-11-403R for more details on similarities and differences between the QIS and
              the traditional survey processes.




              Page 5                                       GAO-12-214 Nursing Home Quality Assessment
                             residents before entering the nursing home in order to identify areas of
                             care that have been historically problematic and residents who may be at
                             higher risk for poor care; interviewing residents, family members, and
                             staff; observing staff and nursing home practices; reviewing medical
                             records; and citing deficiencies when necessary. However, the QIS differs
                             from the traditional survey in two key areas:

                             •    The QIS provides an electronic means of conducting nursing home
                                  inspections. The collection, recording and analysis of information and
                                  documentation of findings are done electronically using a tablet
                                  computer; the traditional method is paper-based.

                             •    The QIS software draws an expanded random sample of up to
                                  70 residents for surveyors to interview, observe, and review their
                                  medical records during the inspection. In contrast, the traditional
                                  survey method allows surveyors to review various reports, data sets,
                                  and guidance and use their judgment to select a sample of residents,
                                  consisting of about 20 percent of the current resident census.

                             CMS has commissioned three studies to evaluate the QIS, an electronic
CMS Commissioned             tool that CMS developed to assess the quality of nursing home care, and
Studies to Evaluate          has taken some steps to improve the QIS in response to the findings of
                             the studies. However, the agency does not routinely monitor the extent to
the QIS, but Does Not        which progress is being made in meeting the objectives established for
Routinely Monitor            the QIS. 15
Progress toward
Meeting the
Objectives of the QIS

CMS Has Commissioned         The first CMS-commissioned study was completed by Abt Associates Inc.
Studies to Evaluate          in 2006 and tested the QIS under real-world conditions. This study
Various Aspects of the QIS   analyzed survey data available from three states in which CMS had
                             begun to conduct demonstrations of the QIS: Connecticut, Kansas, and


                             15
                               According to CMS, the objectives of the QIS include improving surveyors’
                             documentation of deficiencies; improving the efficiency of the routine survey process by
                             focusing survey resources on facilities and areas within facilities where the greatest quality
                             concerns exist; and improving the accuracy and consistency with which surveyors identify
                             deficiencies.




                             Page 6                                        GAO-12-214 Nursing Home Quality Assessment
Ohio. The study was intended to examine surveyors’ experiences using
the QIS to conduct actual nursing home inspections, and identify ways to
improve the QIS. In order to meet this general purpose, the study
assessed several areas, including whether surveyors were able to use an
electronic tool to conduct routine surveys in a real-world setting, whether
surveyors could complete the routine survey using the QIS in the same
amount of time that surveyors needed to complete routine surveys using
the traditional methodology, and whether any changes were needed to
improve the QIS tool. Among other results, the study found that state
surveyors were able to use the QIS-based process to conduct routine
surveys and generally perceived the process to be an improvement over
the traditional survey method. However, the study also found that
surveyors generally needed more time to complete routine surveys using
the QIS tool rather than the traditional survey method. 16 The study cited
several potential reasons for this time difference, ranging from problems
with the QIS software and hardware to forms within the QIS survey
process that could be made less time-consuming. These findings led the
researchers to recommend ways to streamline and otherwise improve the
QIS tool in order to reduce the amount of time surveyors needed to
complete the QIS-based routine survey. CMS revised the QIS survey tool
and process based on many of these recommendations. These revisions
included providing more guidance on certain aspects of the QIS process
during QIS training; eliminating certain duplicative requirements for
collecting information; and altering various survey questions, for example,
by adjusting the wording of survey questions to make them more
understandable to nursing home residents and reducing the time required
for surveyors to explain them.

In 2007, Abt Associates Inc. completed the second CMS-commissioned
study that evaluated the use of the QIS-based routine survey process in
the field. This study included survey data from all five demonstration
states (California, Connecticut, Kansas, Louisiana, and Ohio) and was
intended to evaluate, among other things, whether the QIS achieved
certain of the agency’s objectives for the process. 17 These objectives



16
  According to the report, the researchers were unable to draw firm conclusions regarding
this comparison due to data limitations such as the limited number of QIS surveys
conducted at the time.
17
  Although California participated in the five-state study, CMS allowed the state to
postpone further implementation of QIS due to budgetary concerns within the state.




Page 7                                      GAO-12-214 Nursing Home Quality Assessment
included providing a routine survey tool that improves surveyor
accuracy—that is, helps surveyors better identify deficiencies; improves
surveyor efficiency—that is, helps surveyors focus their time on facilities
with the greatest number of quality concerns; and improves surveyors’
documentation of deficiencies. In terms of whether the QIS led to
improved accuracy or improved documentation of deficiencies, the report
found that the QIS-based routine survey and the traditional survey
approach generally led to comparable results. The researchers, therefore,
concluded that the QIS did not significantly enhance or diminish
surveyors’ ability to accurately identify or document deficiencies.
Similarly, the study suggested that in general, surveyors using the QIS
tool completed routine surveys as efficiently, but not more efficiently, as
surveyors using the traditional survey methodology. 18 The study made
several recommendations to CMS for improving the QIS process,
including that CMS further evaluate how well the QIS protocols use
preliminary information gathered during the survey to identify problem
areas in the facility for more in-depth investigation. Further, the study
recommended that CMS clarify the agency’s investigative protocols,
which CMS publishes and makes available within the QIS software to
help surveyors assess nursing home compliance with federal quality
standards. 19 Following completion of this study, CMS decided to
implement the QIS nationally and CMS officials told us they worked to
improve aspects of the QIS process by taking steps to address the
study’s findings and recommendations. 20

In 2009, CMS commissioned a third study to evaluate the QIS-based
routine survey process and make recommendations for improvement.
This study, conducted by RTI and completed in 2011, was intended to


18
  The researchers noted that the study’s findings were limited by factors such as small
sample size and an inability to control for certain external factors, such as surveyor skills,
that may have affected results of the study.
19
  These protocols, called critical element pathways, assist surveyors in completing a
systematic review of problematic areas in facilities that are selected for in-depth
investigation during the survey. For example, the critical element pathway for the federal
standard regarding the prevention and treatment of pressure ulcers in nursing facilities
provides a checklist of items for the surveyor to assess as well as questions, such as
whether the facility comprehensively assessed the resident’s condition, for the surveyor to
answer during this portion of the survey.
20
  According to CMS officials, the agency considered the results of the second study,
among other factors, when making the decision to continue implementation of the QIS on
a national scale.




Page 8                                        GAO-12-214 Nursing Home Quality Assessment
                            identify aspects of the QIS process that could affect how consistently
                            surveyors identify quality problems and to suggest ways to improve these
                            aspects of the process. This study included survey data from 19 states
                            that had begun to implement the QIS. 21 The study found that various
                            aspects of the QIS process, including the way resident interviews were
                            conducted and the surveyor initiation process (in which surveyors identify
                            additional problem areas, not identified by the QIS software, for further
                            investigation during surveys), might affect the consistency with which
                            surveyors identify quality problems during nursing home inspections. 22
                            For example, the study found that surveyors did not consistently probe for
                            further information when residents gave incomplete answers to interview
                            questions, and that some surveyors were likely to initiate problem areas
                            during surveys on their own, while others relied mainly on the QIS
                            software to identify areas for investigation. The study also identified
                            potential methods for addressing these issues and improving the
                            consistency of the QIS process, including providing additional guidance to
                            surveyors on interviewing techniques and developing additional
                            standardized training for new surveyors to ensure they are familiar with
                            federal regulations. CMS officials told us that they intend to consider the
                            recommendations made in this study in their efforts to continue improving
                            the QIS.


CMS Has Not Established     Although CMS has commissioned past studies to evaluate the QIS, the
the Means to Routinely      agency does not monitor the QIS on an ongoing basis. Such ongoing
Monitor Progress toward     monitoring of the QIS could allow CMS to collect data to routinely
                            evaluate the extent to which progress is being made in meeting all of the
Meeting the Objectives of   objectives established for the QIS to improve the survey process as
the QIS, Such as            intended. Monitoring, a key component of federal internal control
Developing Performance      standards, helps provide management with reasonable assurance that an
Goals and Measures          agency or its programs and initiatives are achieving effectiveness and
                            efficiency of operations and compliance with applicable laws and



                            21
                              The study included both qualitative and quantitative analyses. Some analyses in the
                            study used data from fewer than 19 states.
                            22
                              According to CMS, the QIS software uses preliminary survey data to select problematic
                            areas in facilities for further investigation during surveys. However, surveyors have the
                            option to initiate the selection of areas for investigation outside of those selected by the
                            QIS software. Surveyors may rely on their professional experience and judgment to make
                            these selections.




                            Page 9                                       GAO-12-214 Nursing Home Quality Assessment
regulations, among other factors. 23 One way agencies can monitor
program operations on an ongoing basis to ensure that the programs are
meeting their objectives is through the use of performance goals and
measures. Performance measurement involves identifying performance
goals and measures, establishing performance baselines by tracking
performance over time, identifying targets for improving performance, and
measuring progress against those targets. 24 In the case of the QIS-based
routine survey process, information collected through performance goals
and measures could help CMS routinely monitor the extent to which the
objectives of the QIS are being achieved. This information could also
inform future efforts by CMS to modify and improve the QIS process as
needed. CMS officials acknowledged the need for certain performance
measures—specifically with regard to examining the effect of the QIS on
surveyor consistency–but noted that they did not have performance goals
or measures in place as of December 2011. This lack of performance
goals and measures can affect CMS’s ability to effectively monitor the
QIS.

CMS currently collects some data that would allow it to begin monitoring
aspects of the performance of the QIS. For example, the agency currently
collects information from routine surveys—such as data that describes
the amount of time surveyors have spent inspecting facilities, as well as
the deficiencies that surveyors cited during each inspection—that could
be used to help measure the efficiency of the QIS process. 25 CMS could
use these data to conduct analyses of surveyor efficiency, similar to the
analysis conducted by Abt Associates Inc., and derive efficiency scores




23
  The five components of internal control are Control Environment, Risk Assessment,
Control Activities, Information and Communications, and Monitoring. See also GAO,
Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1
(Washington, D.C.: November 1999).
24
  See Pub. L. No. 103-62, § 4, 107 Stat. 285, 287 (generally codified, as amended, to
31 U.S.C. §1115-1117).
25
  More specifically, CMS data that could be used to measure efficiency include the
number, scope, and severity of deficiencies cited during QIS-based surveys, as well as
the resources, in terms of the amount of time or number of staff, required to complete
these surveys.




Page 10                                    GAO-12-214 Nursing Home Quality Assessment
that could be used for monitoring. 26 These efficiency scores could also be
used to measure changes in surveyor efficiency over time, with the goal
of attaining an agency-established target efficiency score.

Additionally, CMS could use data from desk audit reports to help measure
progress towards the objective of improving surveyor consistency.
According to CMS officials, desk audit reports are standardized reports
that are generated and sent at established intervals to Regional Offices
and state survey agencies. 27 These reports contain data related to
surveys, including data intended to describe surveyor performance during
each QIS-based routine survey conducted. Specifically, desk audit
reports contain data on approximately 30 different aspects of QIS
surveys, such as the problem areas that were identified for further
investigation during a survey and the number of residents for whom a
surveyor did not document complete information during a survey. The
reports also indicate how these data compare to rates at the state,
regional, or national level. 28 CMS officials told us that these reports are
intended to serve as a tool to help officials from Regional Offices and
state survey agencies identify and respond to inconsistencies in QIS
surveyor performance. Since data in the reports can illustrate variation in
how surveyors use the QIS, they could also be used as a starting point for
CMS to create performance goals and measures for consistency and help
the agency routinely evaluate whether progress is being made toward the
objective of improved consistency.




26
  Abt Associates Inc. indicated that a lack of QIS data limited their ability to draw firm
conclusions about the effect of the QIS on surveyor efficiency. This study was conducted
when CMS first began to implement the QIS and included data from five states. As of
September 2011, CMS reported that 26 states have begun QIS implementation.
Therefore, more QIS data should be currently available for analysis.
27
  The reports are generated by a CMS contractor and sent to Regional Offices on a
quarterly basis and to state survey agencies generally after every 10 QIS-based routine
surveys conducted in the state.
28
  CMS officials noted that a version of the report for the CMS Central Office is under
development; however, the agency has not yet established a time line for completing this
work.




Page 11                                     GAO-12-214 Nursing Home Quality Assessment
                            CMS has taken some steps to monitor states’ progress implementing the
CMS Has Taken Steps         QIS, but the agency’s monitoring of states’ efforts is not systematic. As a
to Monitor and              result, CMS has incomplete information on states’ implementation
                            progress. CMS has also taken steps to facilitate states’ implementation of
Facilitate States’          the QIS, such as holding quarterly teleconferences for all states
Implementation of the       implementing the QIS, but the agency lacks a systematic method for
QIS, but CMS’s Efforts      obtaining, compiling, and sharing information on states’ experiences
                            implementing the QIS.
Are Not Systematic


CMS Has Taken Some          CMS has taken some steps to monitor states’ progress implementing the
Steps to Monitor States’    QIS-based routine survey, but these efforts are not systematic. As part of
Progress Implementing the   its national implementation plan, CMS established three milestones to
                            guide states’ implementation of the QIS-based routine survey. 29 These
QIS but Does Not Have a
                            milestones include (1) start dates by which each state is to begin training
Systematic Method of        state surveyors to use the QIS, (2) completion of training for all surveyors
Monitoring                  within three years of the training start date, and (3) use of the QIS for all
                            routine surveys by state surveyors after all surveyors in the state have
                            completed training. CMS policy requires that all routine surveys in a state
                            should be conducted using the QIS after all surveyors in a state have
                            been trained. CMS officials told us that the agency uses various methods
                            to periodically monitor states’ progress on the three milestones.

                            •    Monitoring of state training start dates—CMS officials said that
                                 they track start dates for the initial training of state surveyors to use
                                 the QIS-based routine survey; however, the information that CMS
                                 obtains only pertains to a small group of surveyors in each state.
                                 Specifically, CMS arranges and pays for a contractor to train a small
                                 group of surveyors in states to (1) conduct routine surveys using the
                                 QIS-based process and (2) become registered QIS trainers who will




                            29
                              CMS has established a phased approach for implementing the QIS nationally. The
                            agency has grouped states into six cohorts. States within a cohort are to meet the
                            milestones for beginning and completing surveyor training and using the QIS for all routine
                            surveys within approximately the same general timeframe. According to CMS officials, the
                            agency has rolled out the QIS in three of the six cohorts as of September 2011.




                            Page 12                                     GAO-12-214 Nursing Home Quality Assessment
     be responsible for training the rest of surveyors in the state. 30 CMS
     officials told us the training contractors are responsible for providing
     CMS with information on the start dates of states’ training of this small
     group of surveyors.

•    Monitoring of state surveyor training completion dates—CMS
     officials told us that they periodically monitor the extent to which each
     state completes training of all its surveyors to use the QIS-based
     process, but CMS’s primary method of obtaining this information is not
     systematic and may not allow CMS to collect complete information.
     CMS officials were unable to provide us with information on the
     training completion dates of all state surveyors in those states that the
     agency considered to have completed training. To monitor training
     completion, CMS officials told us that they review information on the
     number of state surveyors trained to use the QIS and compare the
     time between states’ training start dates and when states inform CMS
     that they have trained all of their surveyors. CMS officials told us they
     obtain this training completion information primarily through the
     agency’s quarterly teleconference with state survey agency officials
     implementing the QIS, a forum where states provide information on
     their implementation progress, including status of state surveyor
     training completion. However, we found that the information CMS
     obtains through this method may be incomplete because (1) states
     may not participate in these teleconferences and (2) some states that
     participate may not provide complete information on their
     implementation progress. For example, we observed a quarterly
     teleconference in July 2011, during which at least 7 out of the
     24 states that participated in the call did not provide the agency with
     an update on state surveyor training completion during the
     teleconference.

•    Monitoring the use of the QIS—CMS officials said that they review
     routine survey data to track the number of routine surveys performed
     using the QIS-based process; however, CMS may not have complete
     information to help determine the extent to which states are using the
     QIS. Specifically, agency officials said they periodically review
     whether state surveyors are conducting routine surveys using the


30
  To begin training of state surveyors, states select an initial core group of eight
surveyors—consisting of two survey teams of four surveyors each—to be trained by the
CMS contractor to use the QIS. It is up to state survey agencies to train all remaining
surveyors within CMS’s 3 year implementation time frame.




Page 13                                     GAO-12-214 Nursing Home Quality Assessment
     QIS-based process exclusively after states complete training of all
     state surveyors to use the QIS. 31 To do this, CMS officials pull
     information on the number of surveys completed using the QIS tool
     from their nursing home survey database. However, because CMS
     may not have complete information on the training completion dates
     for all state surveyors, agency officials may not be able to accurately
     determine the date after which states should have begun conducting
     routine surveys exclusively using the QIS-based process. 32
CMS acknowledges that the agency does not have a systematic process
in place that would allow the agency to obtain comprehensive information
on states’ progress implementing the QIS. The ability to perform effective,
ongoing monitoring in the course of normal operations is an integral
element of managing a program, according to federal internal control
standards. 33 CMS officials told us that while they continue to periodically
monitor states’ progress on the QIS milestones using various means, the
agency plans to develop a more systematic process for monitoring states’
progress toward completion of state surveyor training and use of the QIS-
based process for routine surveys. According to CMS, this method will
provide a way for the agency to centrally collect information directly from
the states, including information on states’ completion of surveyor
training. The agency plans to use this information to help monitor states’
overall progress toward implementing the QIS nationally. However,
officials said the agency has not yet established a time line for the
development of this method.




31
  Under CMS policy, once all of a state’s surveyors are trained in QIS that state is
expected to conduct all routine surveys using QIS. CMS officials acknowledged that not all
states reporting to have completed training of all surveyors may be using the QIS
exclusively.
32
  State survey agencies may continue to use the traditional methodology to conduct
nursing home inspections until all surveyors in that state have been trained to use the
QIS. Additionally, those states that have not yet begun QIS implementation continue to
use the traditional methodology.
33
 See GAO/AIMD-00-21.3.1.




Page 14                                     GAO-12-214 Nursing Home Quality Assessment
CMS Has Taken Some            CMS has taken steps to facilitate states’ implementation of the QIS. The
Steps to Facilitate States’   agency has issued periodic guidance to state survey agencies and
Implementation of the QIS,    conducted presentations on the QIS process at various venues, including
                              CMS’s annual conference with state survey agency officials. In addition,
but Efforts to Obtain and     as stipulated by its guidance, CMS provides opportunities for state survey
Share Information on          agencies to discuss implementation issues and concerns with the agency
States’ Experiences Are       and other states. CMS achieves this in part through its quarterly
Not Systematic                teleconferences for all states currently implementing the QIS. During
                              these teleconferences, state survey agency officials have opportunities to
                              share their experiences implementing the QIS, including how they
                              address any challenges they encounter. CMS also provides updates and
                              clarification of agency policies that are to guide state survey agencies as
                              they implement the QIS during the teleconferences. For example, during
                              the July 2011 CMS quarterly teleconference with state survey agencies,
                              agency officials updated state survey agency officials on potential
                              changes that the agency was considering regarding its policy on the
                              number of surveyors assigned to conduct routine surveys using the QIS
                              process. 34 Further, the CMS Central Office and CMS Regional Office staff
                              provides information and assistance on an as-needed basis to individual
                              states. Through these efforts, CMS officials stated they learn of common
                              challenges states face during implementation, such as problems with
                              using QIS software. The agency has provided some assistance to states
                              to resolve these challenges, including providing guidance and
                              implementing QIS software upgrades.

                              Despite the agency’s efforts, CMS does not have a systematic method for
                              obtaining, compiling, and sharing information with state survey agencies
                              on states’ experiences with implementing the QIS. According to federal
                              internal control standards, an agency should establish internal controls
                              that help ensure adequate communication and information sharing with
                              key external stakeholders that may significantly affect the agency’s ability
                              to meet its goals. 35 CMS obtains and shares information with states
                              primarily through the quarterly teleconferences or direct assistance
                              provided upon request to state survey agencies—the agency’s key



                              34
                                According to CMS, agency policy previously required a minimum of four state surveyors
                              per team to conduct routine surveys using the QIS process, regardless of the size of the
                              nursing home facility. As of August 2011, CMS revised this policy to allow states to
                              determine the number of surveyors per team for facilities with fewer than 40 beds.
                              35
                               See GAO/AIMD-00-21.3.1.




                              Page 15                                    GAO-12-214 Nursing Home Quality Assessment
external stakeholders. While states share some information on their
implementation experiences during CMS’s quarterly teleconferences,
such disclosures are voluntary. Additionally, participation in the
teleconferences is limited to only states that are implementing the QIS.
Further, states may not always participate in these teleconferences, so
information communicated during these teleconferences may not reach
all states. While CMS officials summarize the teleconference proceedings
in the form of meeting minutes and share these minutes with meeting
participants, the agency does not share this information with state survey
agencies that have not yet begun implementation of the QIS.

We learned that some states have independently developed various
approaches to help their state facilitate implementation of the QIS. While
some states have shared these approaches with CMS, the agency is not
systematically analyzing, compiling, and sharing these efforts with all
other states.

•    Officials in some states helped facilitate implementation by taking the
     initiative to complete state surveyor QIS training in less than 3 years.
     For example, some state survey agency officials told us they worked
     to efficiently train their surveyors to use the QIS-based process so
     that their survey workload would not be affected. 36

•    One state survey agency facilitated state surveyor training regionally
     by training surveyors in areas of the state with high concentrations of
     large nursing home facilities first. Officials said that they took this
     approach because it allowed them to efficiently balance their survey
     workload.

•    All nine state survey agencies we spoke to reported technology-
     related issues that state surveyors encountered when using the QIS
     software and tablet computers during nursing home inspections. 37
     State survey agency officials from two states reported inviting state
     survey agency information technology staff to participate in QIS



36
  Some state survey agency officials told us that QIS training can affect state survey
workloads. Because surveyors’ completion of QIS training required them to be offline for
at least 4 weeks, fewer state surveyors were then available to conduct surveys for this
period during the implementation process.
37
  These technological issues included software malfunctions that caused surveyors’ tablet
computers to crash or erase survey data collected during nursing home inspections.




Page 16                                     GAO-12-214 Nursing Home Quality Assessment
                  surveyor training in order to enhance the staff’s ability to resolve the
                  technological issues surveyors expected to encounter when using the
                  QIS software and tablet computers.

              •   Some state survey agency officials we spoke to reported that some
                  surveyors in their state experienced difficulties with using QIS
                  software and tablet computers due to a learning curve or a lack of
                  computer skills needed to operate these tools. For example, officials
                  from four states reported developing additional training materials
                  related to basic computer skills or general information on the QIS in
                  order to improve state surveyors’ proficiency with using these QIS
                  tools.

              Providing a systematic method—such as using CMS’s annual conference
              in which all state survey agencies participate—to share information on
              state approaches to facilitate implementation and resolve implementation
              challenges could help the agency facilitate implementation in states that
              have begun implementation or will implement the QIS in the future.


              CMS developed the QIS in order to improve nursing home inspections,
Conclusions   which are used to help ensure the quality of nursing home care delivered
              each year to a vulnerable population of more than one million people.
              After conducting demonstrations and evaluating the results of studies,
              CMS determined that the QIS should be implemented nationwide by
              2018, using a phased implementation. To date, CMS has made progress
              implementing the QIS in 26 states. As this implementation continues over
              the next 6 years, it is important that CMS know the extent to which the
              implementation of the QIS is improving the survey process and whether
              states are making progress toward and meeting CMS’s implementation
              milestones.

              Although CMS commissioned several studies of the QIS and has shown a
              willingness to adapt the QIS process based on some study findings, the
              agency has not established the means—such as performance goals and
              measures—that would allow for routine, ongoing monitoring of the extent
              to which the QIS is helping CMS improve the survey process as intended.
              Such performance goals and measures could be developed in part using
              data CMS currently collects. Routine monitoring that includes
              performance goals and measures for the QIS could also help CMS
              continue to identify areas for improvement as CMS officials expect to
              continue to make changes intended to improve the QIS throughout its
              phased implementation. While studies commissioned at key points can be



              Page 17                              GAO-12-214 Nursing Home Quality Assessment
                      valuable, the information CMS would obtain on the QIS through routine
                      monitoring is needed for this ongoing effort.

                      CMS also does not have reasonable assurance that the phased
                      implementation across the states is making sufficient progress in order to
                      meet the 2018 goal. Although CMS uses various means to monitor and
                      facilitate states’ efforts to implement the QIS, these efforts are not
                      systematic. As a result, CMS lacks complete information on the extent to
                      which states have met their implementation milestones, including the
                      extent to which states are using the QIS tool once all surveyors in the
                      state have completed training. CMS recognizes that it needs a more
                      systematic methodology for monitoring states’ progress, but has not yet
                      implemented it. To help facilitate implementation of the QIS, CMS
                      provides the opportunity for all states implementing the QIS to discuss
                      their implementation challenges; however, CMS could better leverage the
                      experience of the states by systematically obtaining, compiling, and
                      sharing strategies states have used to address their challenges
                      implementing the QIS, for example sharing such information through
                      CMS’s annual conference in which all state survey agencies participate.
                      Such information could help the remaining states that have not yet begun
                      implementation to learn from the experience of others.


                      As CMS continues to implement the QIS, we recommend the
Recommendations for   Administrator of CMS take the following three actions:
Executive Action
                      •   develop a means—such as performance goals and measures—to
                          routinely monitor the extent to which CMS is making progress in
                          meeting the objectives established for the QIS;

                      •   develop and implement a systematic methodology to track state
                          survey agencies’ progress with implementation activities; and

                      •   develop and implement a systematic method for obtaining, compiling,
                          and sharing information from state survey agencies about their
                          implementation experiences.

                      On behalf of CMS, HHS reviewed a draft of this report and provided
Agency Comments       written comments, which appear in appendix I. In its written comments,
and Our Evaluation    HHS fully concurred with all three of our recommendations and said that it
                      would work to implement them expeditiously. Specifically, HHS said that
                      CMS intends to:



                      Page 18                             GAO-12-214 Nursing Home Quality Assessment
•   Establish new or modify existing measures or processes to more
    effectively monitor CMS’s progress towards meeting QIS goals and
    objectives. HHS informed us that CMS has begun to review available
    data related to the QIS—for example, data on survey workload,
    survey deficiencies, and the number of surveys performed—to more
    effectively monitor the QIS and guide improvement efforts.

•   Formalize the data collection method used to track states’ progress
    with QIS implementation activities. According to HHS, the agency
    plans to institute an automated process to obtain updated information
    on states’ progress with training surveyors which, when combined
    with existing data, could provide a more accurate assessment of
    implementation activities in any state.

•   Enhance existing and add new methods of sharing information on the
    QIS with states. HHS stated that CMS plans to expand existing
    information sharing opportunities—for example, quarterly calls and
    presentations at annual meetings—to be available to all states, not
    just those in the process of implementing the QIS. It will also give
    consideration to using web-based capability to facilitate sharing
    information on QIS implementation.

In addition to the comments related to our recommendations, HHS noted
that CMS has reassessed the timetable for QIS expansion to additional
states due to budgetary concerns and the QIS will not be introduced to
additional states in fiscal year 2012 as planned. According to HHS, CMS
will instead focus efforts on more effectively implementing the QIS in
those states where training of state surveyors has begun or has been
completed, addressing barriers to QIS implementation, and completing
the remaining components of the QIS. This reassessment of the QIS
timeline raises concerns that nationwide implementation of the QIS will
extend beyond 2018, as currently planned. The longer it takes the agency
to implement the QIS nationwide, the longer it will take to fully realize the
intended benefits of the QIS—improving the nursing home inspection
process and helping to ensure the quality of nursing home care delivered
each year to more than one million people.


As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from the
report date. At that time, we will send copies of this report to the
Secretary of HHS, the Administrator of CMS, and relevant congressional




Page 19                              GAO-12-214 Nursing Home Quality Assessment
committees. In addition, the report will be available at no charge on the
GAO website at http://www.gao.gov.

If you or your staffs have any questions regarding this report, please
contact me at (202) 512-7114 or kohnl@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report were E. Anne
Laffoon, Assistant Director; George Bogart; Krister Friday; Seta
Hovagimian; Samantha Poppe; and Priyanka Sethi.




Linda T. Kohn
Director, Health Care




Page 20                              GAO-12-214 Nursing Home Quality Assessment
Appendix I: Comments from the Department
             Appendix I: Comments from the Department of
             Health and Human Services



of Health and Human Services




             Page 21                                   GAO-12-214 Nursing Home Quality Assessment
Appendix I: Comments from the Department of
Health and Human Services




Page 22                                   GAO-12-214 Nursing Home Quality Assessment
Appendix I: Comments from the Department of
Health and Human Services




Page 23                                   GAO-12-214 Nursing Home Quality Assessment
Appendix I: Comments from the Department of
Health and Human Services




Page 24                                   GAO-12-214 Nursing Home Quality Assessment
           Appendix I: Comments from the Department of
           Health and Human Services




(290932)
           Page 25                                   GAO-12-214 Nursing Home Quality Assessment
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.