United States Government Accountability Office GAO Report to Congressional Requesters February 2012 NURSING HOME QUALITY CMS Should Improve Efforts to Monitor Implementation of the Quality Indicator Survey This report was revised on March 9, 2012, to include omitted page 2 of the agency comment letter from the Department of Health and Human Services in appendix I, page 23. GAO-12-214 February 2012 NURSING HOME QUALITY CMS Should Improve Efforts to Monitor Implementation of the Quality Indicator Survey Highlights of GAO-12-214, a report to congressional requesters Why GAO Did This Study What GAO Found The Centers for Medicare & Medicaid CMS has commissioned three studies to evaluate the QIS-based survey process Services (CMS) is responsible for but does not routinely monitor the extent to which the objectives established for ensuring that nursing homes receiving the QIS are being met. The first two studies, completed in 2006 and 2007, were federal funding meet federal quality commissioned to determine whether the new survey process could be used in standards. CMS contracts with state real-world settings and whether the QIS was helping the agency meet several of survey agencies to conduct periodic the objectives established for the process. The studies’ findings suggested that inspections of nursing homes using a surveyors could use the survey process and recommended changes intended to CMS-developed survey process. help improve the QIS. CMS officials reported taking steps to address the studies’ Weaknesses in the survey process led findings and recommendations. In 2009, CMS commissioned a third study that CMS to develop and begin was completed in 2011 and identified aspects of the QIS process that could implementing the Quality Indicator Survey (QIS), a new electronic survey. affect the consistency with which surveyors identify quality problems. For CMS developed the QIS to achieve example, the study found that during resident interviews, surveyors did not several objectives, including improving consistently probe for further information when provided with incomplete the efficiency, accuracy, and responses to interview questions. However, CMS does not have the means to consistency of the survey process. routinely monitor the extent to which the QIS is helping improve the survey CMS expects the QIS to be process as intended. Such routine, ongoing monitoring would be consistent with implemented nationally by 2018. We federal internal control standards and could include the use of performance goals examined the extent to which CMS and measures. CMS does have access to some data, such as the amount of (1) evaluates whether progress is time surveyors have spent inspecting facilities, that could be used to help being made in meeting the objectives develop performance goals and measures. of the QIS and (2) monitors and facilitates states’ implementation of the CMS has taken some steps to monitor and facilitate states’ implementation of the QIS. To do this, GAO reviewed QIS-based routine survey, but CMS’s efforts are not systematic. As part of the relevant CMS documents, including agency’s efforts to monitor states’ implementation, CMS primarily uses quarterly CMS-commissioned studies of the teleconferences with state survey agency officials to obtain information on the QIS, policies, and guidance. GAO also extent to which each state has completed training all its surveyors to use the interviewed officials from CMS’s QIS. However, states may not always participate in the teleconferences, and Central and all 10 Regional Offices as those that do may not provide complete information on their progress. As a well as officials from a judgmental result, the information CMS obtains through its monitoring of states’ progress sample of nine state survey agencies. may be incomplete. For example, CMS was unable to provide GAO with information on training completion dates of all state surveyors in states that What GAO Recommends completed training. Although CMS reported it plans to develop a more systematic GAO recommends that CMS develop a process for monitoring states’ implementation progress, it has not established a means to routinely monitor the extent time line for doing this. To help facilitate states’ implementation of the QIS, CMS to which progress is being made in provides states with guidance, gives presentations, and offers states meeting the objectives of the QIS and opportunities to share their implementation experiences through quarterly systematic methods for monitoring and teleconferences. However, CMS does not have a systematic method for facilitating states’ efforts to implement obtaining, compiling and sharing information on state experiences, especially the QIS. The Department of Health and information on approaches states have taken to help facilitate implementation of Human Services fully concurred with the QIS. Systematically sharing such information—for example, through CMS’s all of the recommendations. annual conference in which all state survey agencies participate—could help the agency facilitate implementation in states that have not begun QIS implementation. View GAO-12-214. For more information, contact Linda Kohn at (202) 512-7114 or firstname.lastname@example.org. United States Government Accountability Office Contents Letter 1 Background 4 CMS Commissioned Studies to Evaluate the QIS, but Does Not Routinely Monitor Progress toward Meeting the Objectives of the QIS 6 CMS Has Taken Steps to Monitor and Facilitate States’ Implementation of the QIS, but CMS’s Efforts Are Not Systematic 12 Conclusions 17 Recommendations for Executive Action 18 Agency Comments and Our Evaluation 18 Appendix I Comments from the Department of Health and Human Services 21 Abbreviations CMS Centers for Medicare & Medicaid Services HHS Department of Health and Human Services QIS Quality Indicator Survey RTI Research Triangle Institute International SSA Social Security Act This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page i GAO-12-214 Nursing Home Quality Assessment United States Government Accountability Office Washington, DC 20548 February 1, 2012 The Honorable Herb Kohl Chairman Special Committee on Aging United States Senate The Honorable Charles E. Grassley Ranking Member Committee on the Judiciary United States Senate Nursing homes in the United States provide skilled nursing, rehabilitation, and custodial care to approximately 1.5 million elderly and disabled individuals. Federal and state governments share responsibility for ensuring that the approximately 15,700 nursing homes in the nation receiving payment from Medicare and/or Medicaid programs provide quality care in a safe environment for their highly vulnerable residents. Specifically, the Centers for Medicare & Medicaid Services (CMS), an agency within the Department of Health and Human Services (HHS), is responsible for ensuring that nursing homes receiving payment from Medicare and Medicaid programs meet federal quality standards established pursuant to the Omnibus Budget Reconciliation Act of 1987. 1 To meet this responsibility, CMS, among other activities, contracts with state survey agencies to conduct on-site nursing home inspections, which consist of routine surveys and complaint investigations, using a process developed by CMS. 2 In 1998, we found weaknesses in the traditional process used to conduct routine surveys, including poor measurement of serious care problems in 1 Pub. L. No. 100-203, Title IV, Subtitle C, §§ 4201 et seq., 101 Stat. 1330, 1330-160, et seq. (amending Titles XVIII and XIX of the Social Security Act). According to CMS, combined Medicare and Medicaid payments for nursing home care in 2010 were about $91 billion. 2 Under contract with CMS, state survey agencies conduct inspections of health care facilities—such as nursing homes, home health agencies, intermediate care facilities for the mentally retarded, and hospitals—that participate in Medicare and Medicaid. Page 1 GAO-12-214 Nursing Home Quality Assessment nursing homes. 3 For example, we reported that the resident sample used in the traditional process was inadequate and limited a surveyors’ ability to identify care problems and their prevalence within a facility. To address the problems we reported, and in response to criticism from others, CMS revised the routine survey process by developing the Quality Indicator Survey (QIS). According to CMS, the QIS was developed to help the agency meet several objectives, including improving surveyors’ documentation of quality concerns; improving the efficiency of the survey process by focusing resources on facilities, and on areas within facilities, with the greatest quality concerns; and improving the accuracy and consistency with which surveyors identify deficiencies. CMS began implementation of the QIS-based routine survey process in 2005, by launching demonstrations of the QIS in five states. To implement the QIS, CMS relies on the state survey agencies, as these agencies are largely responsible for the administration of nursing home inspections. 4 To help assist state survey agencies in implementing the QIS, CMS provides policies, guidance and some training for state surveyors. CMS officials told us that as of September 2011, 26 states had trained or started training surveyors to use the QIS and 27 states had not yet started the training. CMS expects to complete implementation of the QIS by 2018 in all 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands. 5 You have raised questions about CMS’s efforts to implement the QIS. We previously provided you with information on the status of the agency’s efforts to implement the QIS nationally. 6 This report focuses on certain aspects of CMS’s implementation of the QIS and will examine the extent to which CMS (1) evaluates whether progress is being made in meeting 3 See GAO, California Nursing Homes: Care Problems Persist Despite Federal and State Oversight, GAO/HEHS-98-202 (Washington, D.C.: July 27, 1998). 4 State survey agencies administer and have discretion over many survey activities and policies, including hiring and retaining a surveyor workforce, training surveyors, and conducting supervisory reviews of surveys. For example, state survey agencies are largely responsible for training all surveyors in their states to use the QIS; in 2011, there were roughly 4,700 surveyors nationwide. 5 In this report, we use the term “states” to refer to this group of states and territories. 6 See GAO, Nursing Home Quality: Implementation of the Quality Indicator Survey, GAO-11-403R (Washington, D.C.: Apr. 6, 2011). Page 2 GAO-12-214 Nursing Home Quality Assessment the objectives of the QIS and (2) monitors and facilitates states’ implementation of the QIS. To examine the steps CMS has taken to evaluate the QIS, we reviewed relevant CMS documents, including the agency’s plan for evaluation and improvement of the QIS and independent studies of the QIS commissioned by the agency. We also interviewed CMS officials from CMS’s Central Office. We asked these officials for information on CMS’s efforts to evaluate the QIS, including any efforts to routinely monitor the QIS, a practice consistent with federal internal control standards. 7 Additionally, we interviewed the CMS contractors—Abt Associates Inc. and Research Triangle Institute International (RTI)—that conducted the independent studies of the QIS. To examine the extent to which CMS monitors and facilitates states’ implementation of the QIS, we reviewed CMS policies on the QIS as well as guidance the agency provided to state survey agencies that described their respective roles in implementing the QIS. We reviewed CMS’s plan for implementing the QIS nationally. We also interviewed officials from CMS’s Central Office and all 10 Regional Offices to learn more about the agency’s efforts to monitor and facilitate states’ implementation of the QIS. 8 We also analyzed this information to determine whether CMS’s monitoring and facilitation efforts are consistent with federal internal control standards. 9 Additionally, we conducted structured interviews with state survey agency officials from a judgmental sample of nine states (Arizona, Colorado, Connecticut, Florida, Maryland, Minnesota, New York, Ohio, and West Virginia), to obtain information about these states’ experience with implementing the QIS, including the extent to which CMS monitored their progress with implementation and helped to facilitate implementation in their state. We selected our sample from the 21 states that had begun implementation as of February 2011. These states were selected based on criteria such as variation in the states’ QIS implementation status, the percentage of nursing home inspections conducted using the QIS-based process within a state, and state size. 7 See GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999). 8 CMS’s Regional Offices are located in Atlanta, Boston, Chicago, Dallas, Denver, Kansas City, New York, Philadelphia, Seattle, and San Francisco. 9 See GAO/AIMD-00-21.3.1. Page 3 GAO-12-214 Nursing Home Quality Assessment The findings from our structured interviews are limited to the nine states in our sample and are not representative; therefore, we will not be able to generalize these findings to all states. We conducted this performance audit from June 2011 through December 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Oversight of nursing homes is a shared federal-state responsibility. Based Background on statutory requirements, CMS defines standards that nursing homes must meet to participate in the Medicare and Medicaid programs and contracts with state survey agencies to assess whether homes meet these standards. A range of statutorily defined sanctions is available to CMS and the states to help ensure that homes maintain compliance with federal quality requirements. CMS also is responsible for monitoring the adequacy of state survey activities. Nursing Home Inspections Every nursing home that receives payment from Medicare and/or Medicaid must undergo a nursing home inspection. These inspections consist of routine surveys that are conducted on average every 12 months and complaint investigations that are conducted in response to allegations of quality problems. 10 During a routine survey, teams of state surveyors—generally consisting of registered nurses, social workers, dieticians, or other specialists—evaluate a nursing home’s compliance with federal nursing home quality standards. There are approximately 200 different standards that focus on the delivery of care, resident outcomes and facility conditions. The standards are grouped into 15 categories, such as Quality of Life, Resident Assessment, Quality of Care, and Administration. For example, there are 23 standards within the Quality of Care category ranging from the prevention of pressure sore development to keeping the resident environment as free of accident hazards as is 10 Every nursing home is required to receive an inspection not less than once every 15 months and the statewide average interval for these surveys must not exceed 12 months. 42 C.F.R. § 488.308 (2010). Page 4 GAO-12-214 Nursing Home Quality Assessment possible. During nursing home inspections, state surveyors may cite deficiencies—areas in which facilities fail to meet federal quality standards—and nursing homes must prepare a correction plan to address most deficiencies. 11 State surveyors may also conduct what are known as revisit surveys to ensure that the homes corrected or took steps to correct identified deficiencies. To ensure the reliability of state survey agencies’ inspections of nursing homes, federal survey teams composed of CMS staff from its 10 Regional offices conduct what are known as federal monitoring surveys, which include comparative and observational surveys. In a comparative survey, a federal survey team conducts an independent inspection of a home recently surveyed by a state survey agency in order to compare and contrast its findings with those of the state survey team. In an observational survey, federal surveyors accompany a state survey team to a nursing home to evaluate the team’s on-site survey performance and ability to document deficiencies. The federal monitoring surveys are to be conducted annually in at least 5 percent of state-surveyed Medicare and Medicaid nursing homes in each state. 12 CMS has developed the QIS- based tool for routine surveys and plans to develop QIS-based tools for all other types of nursing home inspections: revisit surveys, complaint investigations and federal monitoring surveys. 13 QIS Process The QIS process is similar to the traditional survey method. 14 For example, both the QIS and the traditional survey process involve analyzing information available about the nursing home facilities and their 11 For serious deficiencies, CMS generally imposes sanctions, which may result in the termination of a nursing home’s participation in the Medicare and Medicaid programs. 12 Under the Social Security Act (SSA), HHS is required to conduct a certain number of federal validation surveys, also referred to as federal monitoring surveys, each year. In no case is HHS to survey less than a total of five nursing homes in a state per year. SSA §1819(g)(3)(B)(Medicare) (codified at 42 U.S.C. §1395i-3(g)(3)(B)); SSA § 1919(g)(3)(B)(Medicaid) (codified at 42 U.S.C. § 1396r(g)(3)(B)). 13 Revisit surveys ensure that nursing homes remediate certain deficiencies. A complaint investigation generally focuses on a specific allegation regarding a resident’s care or safety. Federal monitoring surveys are done to ensure the reliability of state survey agencies’ routine surveys. 14 See GAO-11-403R for more details on similarities and differences between the QIS and the traditional survey processes. Page 5 GAO-12-214 Nursing Home Quality Assessment residents before entering the nursing home in order to identify areas of care that have been historically problematic and residents who may be at higher risk for poor care; interviewing residents, family members, and staff; observing staff and nursing home practices; reviewing medical records; and citing deficiencies when necessary. However, the QIS differs from the traditional survey in two key areas: • The QIS provides an electronic means of conducting nursing home inspections. The collection, recording and analysis of information and documentation of findings are done electronically using a tablet computer; the traditional method is paper-based. • The QIS software draws an expanded random sample of up to 70 residents for surveyors to interview, observe, and review their medical records during the inspection. In contrast, the traditional survey method allows surveyors to review various reports, data sets, and guidance and use their judgment to select a sample of residents, consisting of about 20 percent of the current resident census. CMS has commissioned three studies to evaluate the QIS, an electronic CMS Commissioned tool that CMS developed to assess the quality of nursing home care, and Studies to Evaluate has taken some steps to improve the QIS in response to the findings of the studies. However, the agency does not routinely monitor the extent to the QIS, but Does Not which progress is being made in meeting the objectives established for Routinely Monitor the QIS. 15 Progress toward Meeting the Objectives of the QIS CMS Has Commissioned The first CMS-commissioned study was completed by Abt Associates Inc. Studies to Evaluate in 2006 and tested the QIS under real-world conditions. This study Various Aspects of the QIS analyzed survey data available from three states in which CMS had begun to conduct demonstrations of the QIS: Connecticut, Kansas, and 15 According to CMS, the objectives of the QIS include improving surveyors’ documentation of deficiencies; improving the efficiency of the routine survey process by focusing survey resources on facilities and areas within facilities where the greatest quality concerns exist; and improving the accuracy and consistency with which surveyors identify deficiencies. Page 6 GAO-12-214 Nursing Home Quality Assessment Ohio. The study was intended to examine surveyors’ experiences using the QIS to conduct actual nursing home inspections, and identify ways to improve the QIS. In order to meet this general purpose, the study assessed several areas, including whether surveyors were able to use an electronic tool to conduct routine surveys in a real-world setting, whether surveyors could complete the routine survey using the QIS in the same amount of time that surveyors needed to complete routine surveys using the traditional methodology, and whether any changes were needed to improve the QIS tool. Among other results, the study found that state surveyors were able to use the QIS-based process to conduct routine surveys and generally perceived the process to be an improvement over the traditional survey method. However, the study also found that surveyors generally needed more time to complete routine surveys using the QIS tool rather than the traditional survey method. 16 The study cited several potential reasons for this time difference, ranging from problems with the QIS software and hardware to forms within the QIS survey process that could be made less time-consuming. These findings led the researchers to recommend ways to streamline and otherwise improve the QIS tool in order to reduce the amount of time surveyors needed to complete the QIS-based routine survey. CMS revised the QIS survey tool and process based on many of these recommendations. These revisions included providing more guidance on certain aspects of the QIS process during QIS training; eliminating certain duplicative requirements for collecting information; and altering various survey questions, for example, by adjusting the wording of survey questions to make them more understandable to nursing home residents and reducing the time required for surveyors to explain them. In 2007, Abt Associates Inc. completed the second CMS-commissioned study that evaluated the use of the QIS-based routine survey process in the field. This study included survey data from all five demonstration states (California, Connecticut, Kansas, Louisiana, and Ohio) and was intended to evaluate, among other things, whether the QIS achieved certain of the agency’s objectives for the process. 17 These objectives 16 According to the report, the researchers were unable to draw firm conclusions regarding this comparison due to data limitations such as the limited number of QIS surveys conducted at the time. 17 Although California participated in the five-state study, CMS allowed the state to postpone further implementation of QIS due to budgetary concerns within the state. Page 7 GAO-12-214 Nursing Home Quality Assessment included providing a routine survey tool that improves surveyor accuracy—that is, helps surveyors better identify deficiencies; improves surveyor efficiency—that is, helps surveyors focus their time on facilities with the greatest number of quality concerns; and improves surveyors’ documentation of deficiencies. In terms of whether the QIS led to improved accuracy or improved documentation of deficiencies, the report found that the QIS-based routine survey and the traditional survey approach generally led to comparable results. The researchers, therefore, concluded that the QIS did not significantly enhance or diminish surveyors’ ability to accurately identify or document deficiencies. Similarly, the study suggested that in general, surveyors using the QIS tool completed routine surveys as efficiently, but not more efficiently, as surveyors using the traditional survey methodology. 18 The study made several recommendations to CMS for improving the QIS process, including that CMS further evaluate how well the QIS protocols use preliminary information gathered during the survey to identify problem areas in the facility for more in-depth investigation. Further, the study recommended that CMS clarify the agency’s investigative protocols, which CMS publishes and makes available within the QIS software to help surveyors assess nursing home compliance with federal quality standards. 19 Following completion of this study, CMS decided to implement the QIS nationally and CMS officials told us they worked to improve aspects of the QIS process by taking steps to address the study’s findings and recommendations. 20 In 2009, CMS commissioned a third study to evaluate the QIS-based routine survey process and make recommendations for improvement. This study, conducted by RTI and completed in 2011, was intended to 18 The researchers noted that the study’s findings were limited by factors such as small sample size and an inability to control for certain external factors, such as surveyor skills, that may have affected results of the study. 19 These protocols, called critical element pathways, assist surveyors in completing a systematic review of problematic areas in facilities that are selected for in-depth investigation during the survey. For example, the critical element pathway for the federal standard regarding the prevention and treatment of pressure ulcers in nursing facilities provides a checklist of items for the surveyor to assess as well as questions, such as whether the facility comprehensively assessed the resident’s condition, for the surveyor to answer during this portion of the survey. 20 According to CMS officials, the agency considered the results of the second study, among other factors, when making the decision to continue implementation of the QIS on a national scale. Page 8 GAO-12-214 Nursing Home Quality Assessment identify aspects of the QIS process that could affect how consistently surveyors identify quality problems and to suggest ways to improve these aspects of the process. This study included survey data from 19 states that had begun to implement the QIS. 21 The study found that various aspects of the QIS process, including the way resident interviews were conducted and the surveyor initiation process (in which surveyors identify additional problem areas, not identified by the QIS software, for further investigation during surveys), might affect the consistency with which surveyors identify quality problems during nursing home inspections. 22 For example, the study found that surveyors did not consistently probe for further information when residents gave incomplete answers to interview questions, and that some surveyors were likely to initiate problem areas during surveys on their own, while others relied mainly on the QIS software to identify areas for investigation. The study also identified potential methods for addressing these issues and improving the consistency of the QIS process, including providing additional guidance to surveyors on interviewing techniques and developing additional standardized training for new surveyors to ensure they are familiar with federal regulations. CMS officials told us that they intend to consider the recommendations made in this study in their efforts to continue improving the QIS. CMS Has Not Established Although CMS has commissioned past studies to evaluate the QIS, the the Means to Routinely agency does not monitor the QIS on an ongoing basis. Such ongoing Monitor Progress toward monitoring of the QIS could allow CMS to collect data to routinely evaluate the extent to which progress is being made in meeting all of the Meeting the Objectives of objectives established for the QIS to improve the survey process as the QIS, Such as intended. Monitoring, a key component of federal internal control Developing Performance standards, helps provide management with reasonable assurance that an Goals and Measures agency or its programs and initiatives are achieving effectiveness and efficiency of operations and compliance with applicable laws and 21 The study included both qualitative and quantitative analyses. Some analyses in the study used data from fewer than 19 states. 22 According to CMS, the QIS software uses preliminary survey data to select problematic areas in facilities for further investigation during surveys. However, surveyors have the option to initiate the selection of areas for investigation outside of those selected by the QIS software. Surveyors may rely on their professional experience and judgment to make these selections. Page 9 GAO-12-214 Nursing Home Quality Assessment regulations, among other factors. 23 One way agencies can monitor program operations on an ongoing basis to ensure that the programs are meeting their objectives is through the use of performance goals and measures. Performance measurement involves identifying performance goals and measures, establishing performance baselines by tracking performance over time, identifying targets for improving performance, and measuring progress against those targets. 24 In the case of the QIS-based routine survey process, information collected through performance goals and measures could help CMS routinely monitor the extent to which the objectives of the QIS are being achieved. This information could also inform future efforts by CMS to modify and improve the QIS process as needed. CMS officials acknowledged the need for certain performance measures—specifically with regard to examining the effect of the QIS on surveyor consistency–but noted that they did not have performance goals or measures in place as of December 2011. This lack of performance goals and measures can affect CMS’s ability to effectively monitor the QIS. CMS currently collects some data that would allow it to begin monitoring aspects of the performance of the QIS. For example, the agency currently collects information from routine surveys—such as data that describes the amount of time surveyors have spent inspecting facilities, as well as the deficiencies that surveyors cited during each inspection—that could be used to help measure the efficiency of the QIS process. 25 CMS could use these data to conduct analyses of surveyor efficiency, similar to the analysis conducted by Abt Associates Inc., and derive efficiency scores 23 The five components of internal control are Control Environment, Risk Assessment, Control Activities, Information and Communications, and Monitoring. See also GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999). 24 See Pub. L. No. 103-62, § 4, 107 Stat. 285, 287 (generally codified, as amended, to 31 U.S.C. §1115-1117). 25 More specifically, CMS data that could be used to measure efficiency include the number, scope, and severity of deficiencies cited during QIS-based surveys, as well as the resources, in terms of the amount of time or number of staff, required to complete these surveys. Page 10 GAO-12-214 Nursing Home Quality Assessment that could be used for monitoring. 26 These efficiency scores could also be used to measure changes in surveyor efficiency over time, with the goal of attaining an agency-established target efficiency score. Additionally, CMS could use data from desk audit reports to help measure progress towards the objective of improving surveyor consistency. According to CMS officials, desk audit reports are standardized reports that are generated and sent at established intervals to Regional Offices and state survey agencies. 27 These reports contain data related to surveys, including data intended to describe surveyor performance during each QIS-based routine survey conducted. Specifically, desk audit reports contain data on approximately 30 different aspects of QIS surveys, such as the problem areas that were identified for further investigation during a survey and the number of residents for whom a surveyor did not document complete information during a survey. The reports also indicate how these data compare to rates at the state, regional, or national level. 28 CMS officials told us that these reports are intended to serve as a tool to help officials from Regional Offices and state survey agencies identify and respond to inconsistencies in QIS surveyor performance. Since data in the reports can illustrate variation in how surveyors use the QIS, they could also be used as a starting point for CMS to create performance goals and measures for consistency and help the agency routinely evaluate whether progress is being made toward the objective of improved consistency. 26 Abt Associates Inc. indicated that a lack of QIS data limited their ability to draw firm conclusions about the effect of the QIS on surveyor efficiency. This study was conducted when CMS first began to implement the QIS and included data from five states. As of September 2011, CMS reported that 26 states have begun QIS implementation. Therefore, more QIS data should be currently available for analysis. 27 The reports are generated by a CMS contractor and sent to Regional Offices on a quarterly basis and to state survey agencies generally after every 10 QIS-based routine surveys conducted in the state. 28 CMS officials noted that a version of the report for the CMS Central Office is under development; however, the agency has not yet established a time line for completing this work. Page 11 GAO-12-214 Nursing Home Quality Assessment CMS has taken some steps to monitor states’ progress implementing the CMS Has Taken Steps QIS, but the agency’s monitoring of states’ efforts is not systematic. As a to Monitor and result, CMS has incomplete information on states’ implementation progress. CMS has also taken steps to facilitate states’ implementation of Facilitate States’ the QIS, such as holding quarterly teleconferences for all states Implementation of the implementing the QIS, but the agency lacks a systematic method for QIS, but CMS’s Efforts obtaining, compiling, and sharing information on states’ experiences implementing the QIS. Are Not Systematic CMS Has Taken Some CMS has taken some steps to monitor states’ progress implementing the Steps to Monitor States’ QIS-based routine survey, but these efforts are not systematic. As part of Progress Implementing the its national implementation plan, CMS established three milestones to guide states’ implementation of the QIS-based routine survey. 29 These QIS but Does Not Have a milestones include (1) start dates by which each state is to begin training Systematic Method of state surveyors to use the QIS, (2) completion of training for all surveyors Monitoring within three years of the training start date, and (3) use of the QIS for all routine surveys by state surveyors after all surveyors in the state have completed training. CMS policy requires that all routine surveys in a state should be conducted using the QIS after all surveyors in a state have been trained. CMS officials told us that the agency uses various methods to periodically monitor states’ progress on the three milestones. • Monitoring of state training start dates—CMS officials said that they track start dates for the initial training of state surveyors to use the QIS-based routine survey; however, the information that CMS obtains only pertains to a small group of surveyors in each state. Specifically, CMS arranges and pays for a contractor to train a small group of surveyors in states to (1) conduct routine surveys using the QIS-based process and (2) become registered QIS trainers who will 29 CMS has established a phased approach for implementing the QIS nationally. The agency has grouped states into six cohorts. States within a cohort are to meet the milestones for beginning and completing surveyor training and using the QIS for all routine surveys within approximately the same general timeframe. According to CMS officials, the agency has rolled out the QIS in three of the six cohorts as of September 2011. Page 12 GAO-12-214 Nursing Home Quality Assessment be responsible for training the rest of surveyors in the state. 30 CMS officials told us the training contractors are responsible for providing CMS with information on the start dates of states’ training of this small group of surveyors. • Monitoring of state surveyor training completion dates—CMS officials told us that they periodically monitor the extent to which each state completes training of all its surveyors to use the QIS-based process, but CMS’s primary method of obtaining this information is not systematic and may not allow CMS to collect complete information. CMS officials were unable to provide us with information on the training completion dates of all state surveyors in those states that the agency considered to have completed training. To monitor training completion, CMS officials told us that they review information on the number of state surveyors trained to use the QIS and compare the time between states’ training start dates and when states inform CMS that they have trained all of their surveyors. CMS officials told us they obtain this training completion information primarily through the agency’s quarterly teleconference with state survey agency officials implementing the QIS, a forum where states provide information on their implementation progress, including status of state surveyor training completion. However, we found that the information CMS obtains through this method may be incomplete because (1) states may not participate in these teleconferences and (2) some states that participate may not provide complete information on their implementation progress. For example, we observed a quarterly teleconference in July 2011, during which at least 7 out of the 24 states that participated in the call did not provide the agency with an update on state surveyor training completion during the teleconference. • Monitoring the use of the QIS—CMS officials said that they review routine survey data to track the number of routine surveys performed using the QIS-based process; however, CMS may not have complete information to help determine the extent to which states are using the QIS. Specifically, agency officials said they periodically review whether state surveyors are conducting routine surveys using the 30 To begin training of state surveyors, states select an initial core group of eight surveyors—consisting of two survey teams of four surveyors each—to be trained by the CMS contractor to use the QIS. It is up to state survey agencies to train all remaining surveyors within CMS’s 3 year implementation time frame. Page 13 GAO-12-214 Nursing Home Quality Assessment QIS-based process exclusively after states complete training of all state surveyors to use the QIS. 31 To do this, CMS officials pull information on the number of surveys completed using the QIS tool from their nursing home survey database. However, because CMS may not have complete information on the training completion dates for all state surveyors, agency officials may not be able to accurately determine the date after which states should have begun conducting routine surveys exclusively using the QIS-based process. 32 CMS acknowledges that the agency does not have a systematic process in place that would allow the agency to obtain comprehensive information on states’ progress implementing the QIS. The ability to perform effective, ongoing monitoring in the course of normal operations is an integral element of managing a program, according to federal internal control standards. 33 CMS officials told us that while they continue to periodically monitor states’ progress on the QIS milestones using various means, the agency plans to develop a more systematic process for monitoring states’ progress toward completion of state surveyor training and use of the QIS- based process for routine surveys. According to CMS, this method will provide a way for the agency to centrally collect information directly from the states, including information on states’ completion of surveyor training. The agency plans to use this information to help monitor states’ overall progress toward implementing the QIS nationally. However, officials said the agency has not yet established a time line for the development of this method. 31 Under CMS policy, once all of a state’s surveyors are trained in QIS that state is expected to conduct all routine surveys using QIS. CMS officials acknowledged that not all states reporting to have completed training of all surveyors may be using the QIS exclusively. 32 State survey agencies may continue to use the traditional methodology to conduct nursing home inspections until all surveyors in that state have been trained to use the QIS. Additionally, those states that have not yet begun QIS implementation continue to use the traditional methodology. 33 See GAO/AIMD-00-21.3.1. Page 14 GAO-12-214 Nursing Home Quality Assessment CMS Has Taken Some CMS has taken steps to facilitate states’ implementation of the QIS. The Steps to Facilitate States’ agency has issued periodic guidance to state survey agencies and Implementation of the QIS, conducted presentations on the QIS process at various venues, including CMS’s annual conference with state survey agency officials. In addition, but Efforts to Obtain and as stipulated by its guidance, CMS provides opportunities for state survey Share Information on agencies to discuss implementation issues and concerns with the agency States’ Experiences Are and other states. CMS achieves this in part through its quarterly Not Systematic teleconferences for all states currently implementing the QIS. During these teleconferences, state survey agency officials have opportunities to share their experiences implementing the QIS, including how they address any challenges they encounter. CMS also provides updates and clarification of agency policies that are to guide state survey agencies as they implement the QIS during the teleconferences. For example, during the July 2011 CMS quarterly teleconference with state survey agencies, agency officials updated state survey agency officials on potential changes that the agency was considering regarding its policy on the number of surveyors assigned to conduct routine surveys using the QIS process. 34 Further, the CMS Central Office and CMS Regional Office staff provides information and assistance on an as-needed basis to individual states. Through these efforts, CMS officials stated they learn of common challenges states face during implementation, such as problems with using QIS software. The agency has provided some assistance to states to resolve these challenges, including providing guidance and implementing QIS software upgrades. Despite the agency’s efforts, CMS does not have a systematic method for obtaining, compiling, and sharing information with state survey agencies on states’ experiences with implementing the QIS. According to federal internal control standards, an agency should establish internal controls that help ensure adequate communication and information sharing with key external stakeholders that may significantly affect the agency’s ability to meet its goals. 35 CMS obtains and shares information with states primarily through the quarterly teleconferences or direct assistance provided upon request to state survey agencies—the agency’s key 34 According to CMS, agency policy previously required a minimum of four state surveyors per team to conduct routine surveys using the QIS process, regardless of the size of the nursing home facility. As of August 2011, CMS revised this policy to allow states to determine the number of surveyors per team for facilities with fewer than 40 beds. 35 See GAO/AIMD-00-21.3.1. Page 15 GAO-12-214 Nursing Home Quality Assessment external stakeholders. While states share some information on their implementation experiences during CMS’s quarterly teleconferences, such disclosures are voluntary. Additionally, participation in the teleconferences is limited to only states that are implementing the QIS. Further, states may not always participate in these teleconferences, so information communicated during these teleconferences may not reach all states. While CMS officials summarize the teleconference proceedings in the form of meeting minutes and share these minutes with meeting participants, the agency does not share this information with state survey agencies that have not yet begun implementation of the QIS. We learned that some states have independently developed various approaches to help their state facilitate implementation of the QIS. While some states have shared these approaches with CMS, the agency is not systematically analyzing, compiling, and sharing these efforts with all other states. • Officials in some states helped facilitate implementation by taking the initiative to complete state surveyor QIS training in less than 3 years. For example, some state survey agency officials told us they worked to efficiently train their surveyors to use the QIS-based process so that their survey workload would not be affected. 36 • One state survey agency facilitated state surveyor training regionally by training surveyors in areas of the state with high concentrations of large nursing home facilities first. Officials said that they took this approach because it allowed them to efficiently balance their survey workload. • All nine state survey agencies we spoke to reported technology- related issues that state surveyors encountered when using the QIS software and tablet computers during nursing home inspections. 37 State survey agency officials from two states reported inviting state survey agency information technology staff to participate in QIS 36 Some state survey agency officials told us that QIS training can affect state survey workloads. Because surveyors’ completion of QIS training required them to be offline for at least 4 weeks, fewer state surveyors were then available to conduct surveys for this period during the implementation process. 37 These technological issues included software malfunctions that caused surveyors’ tablet computers to crash or erase survey data collected during nursing home inspections. Page 16 GAO-12-214 Nursing Home Quality Assessment surveyor training in order to enhance the staff’s ability to resolve the technological issues surveyors expected to encounter when using the QIS software and tablet computers. • Some state survey agency officials we spoke to reported that some surveyors in their state experienced difficulties with using QIS software and tablet computers due to a learning curve or a lack of computer skills needed to operate these tools. For example, officials from four states reported developing additional training materials related to basic computer skills or general information on the QIS in order to improve state surveyors’ proficiency with using these QIS tools. Providing a systematic method—such as using CMS’s annual conference in which all state survey agencies participate—to share information on state approaches to facilitate implementation and resolve implementation challenges could help the agency facilitate implementation in states that have begun implementation or will implement the QIS in the future. CMS developed the QIS in order to improve nursing home inspections, Conclusions which are used to help ensure the quality of nursing home care delivered each year to a vulnerable population of more than one million people. After conducting demonstrations and evaluating the results of studies, CMS determined that the QIS should be implemented nationwide by 2018, using a phased implementation. To date, CMS has made progress implementing the QIS in 26 states. As this implementation continues over the next 6 years, it is important that CMS know the extent to which the implementation of the QIS is improving the survey process and whether states are making progress toward and meeting CMS’s implementation milestones. Although CMS commissioned several studies of the QIS and has shown a willingness to adapt the QIS process based on some study findings, the agency has not established the means—such as performance goals and measures—that would allow for routine, ongoing monitoring of the extent to which the QIS is helping CMS improve the survey process as intended. Such performance goals and measures could be developed in part using data CMS currently collects. Routine monitoring that includes performance goals and measures for the QIS could also help CMS continue to identify areas for improvement as CMS officials expect to continue to make changes intended to improve the QIS throughout its phased implementation. While studies commissioned at key points can be Page 17 GAO-12-214 Nursing Home Quality Assessment valuable, the information CMS would obtain on the QIS through routine monitoring is needed for this ongoing effort. CMS also does not have reasonable assurance that the phased implementation across the states is making sufficient progress in order to meet the 2018 goal. Although CMS uses various means to monitor and facilitate states’ efforts to implement the QIS, these efforts are not systematic. As a result, CMS lacks complete information on the extent to which states have met their implementation milestones, including the extent to which states are using the QIS tool once all surveyors in the state have completed training. CMS recognizes that it needs a more systematic methodology for monitoring states’ progress, but has not yet implemented it. To help facilitate implementation of the QIS, CMS provides the opportunity for all states implementing the QIS to discuss their implementation challenges; however, CMS could better leverage the experience of the states by systematically obtaining, compiling, and sharing strategies states have used to address their challenges implementing the QIS, for example sharing such information through CMS’s annual conference in which all state survey agencies participate. Such information could help the remaining states that have not yet begun implementation to learn from the experience of others. As CMS continues to implement the QIS, we recommend the Recommendations for Administrator of CMS take the following three actions: Executive Action • develop a means—such as performance goals and measures—to routinely monitor the extent to which CMS is making progress in meeting the objectives established for the QIS; • develop and implement a systematic methodology to track state survey agencies’ progress with implementation activities; and • develop and implement a systematic method for obtaining, compiling, and sharing information from state survey agencies about their implementation experiences. On behalf of CMS, HHS reviewed a draft of this report and provided Agency Comments written comments, which appear in appendix I. In its written comments, and Our Evaluation HHS fully concurred with all three of our recommendations and said that it would work to implement them expeditiously. Specifically, HHS said that CMS intends to: Page 18 GAO-12-214 Nursing Home Quality Assessment • Establish new or modify existing measures or processes to more effectively monitor CMS’s progress towards meeting QIS goals and objectives. HHS informed us that CMS has begun to review available data related to the QIS—for example, data on survey workload, survey deficiencies, and the number of surveys performed—to more effectively monitor the QIS and guide improvement efforts. • Formalize the data collection method used to track states’ progress with QIS implementation activities. According to HHS, the agency plans to institute an automated process to obtain updated information on states’ progress with training surveyors which, when combined with existing data, could provide a more accurate assessment of implementation activities in any state. • Enhance existing and add new methods of sharing information on the QIS with states. HHS stated that CMS plans to expand existing information sharing opportunities—for example, quarterly calls and presentations at annual meetings—to be available to all states, not just those in the process of implementing the QIS. It will also give consideration to using web-based capability to facilitate sharing information on QIS implementation. In addition to the comments related to our recommendations, HHS noted that CMS has reassessed the timetable for QIS expansion to additional states due to budgetary concerns and the QIS will not be introduced to additional states in fiscal year 2012 as planned. According to HHS, CMS will instead focus efforts on more effectively implementing the QIS in those states where training of state surveyors has begun or has been completed, addressing barriers to QIS implementation, and completing the remaining components of the QIS. This reassessment of the QIS timeline raises concerns that nationwide implementation of the QIS will extend beyond 2018, as currently planned. The longer it takes the agency to implement the QIS nationwide, the longer it will take to fully realize the intended benefits of the QIS—improving the nursing home inspection process and helping to ensure the quality of nursing home care delivered each year to more than one million people. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of HHS, the Administrator of CMS, and relevant congressional Page 19 GAO-12-214 Nursing Home Quality Assessment committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions regarding this report, please contact me at (202) 512-7114 or email@example.com. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were E. Anne Laffoon, Assistant Director; George Bogart; Krister Friday; Seta Hovagimian; Samantha Poppe; and Priyanka Sethi. Linda T. Kohn Director, Health Care Page 20 GAO-12-214 Nursing Home Quality Assessment Appendix I: Comments from the Department Appendix I: Comments from the Department of Health and Human Services of Health and Human Services Page 21 GAO-12-214 Nursing Home Quality Assessment Appendix I: Comments from the Department of Health and Human Services Page 22 GAO-12-214 Nursing Home Quality Assessment Appendix I: Comments from the Department of Health and Human Services Page 23 GAO-12-214 Nursing Home Quality Assessment Appendix I: Comments from the Department of Health and Human Services Page 24 GAO-12-214 Nursing Home Quality Assessment Appendix I: Comments from the Department of Health and Human Services (290932) Page 25 GAO-12-214 Nursing Home Quality Assessment GAO’s Mission The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no Obtaining Copies of cost is through GAO’s website (www.gao.gov). Each weekday afternoon, GAO Reports and GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, Testimony go to www.gao.gov and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov. Contact: To Report Fraud, Waste, and Abuse in Website: www.gao.gov/fraudnet/fraudnet.htm E-mail: firstname.lastname@example.org Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, email@example.com, (202) 512- Congressional 4400, U.S. Government Accountability Office, 441 G Street NW, Room Relations 7125, Washington, DC 20548 Chuck Young, Managing Director, firstname.lastname@example.org, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Please Print on Recycled Paper.
Nursing Home Quality: CMS Should Improve Efforts to Monitor Implementation of the Quality Indicator Survey [Reissued on March 9, 2012]
Published by the Government Accountability Office on 2012-02-01.
Below is a raw (and likely hideous) rendition of the original report. (PDF)