oversight

Health Care Quality Measurement: HHS Should Address Contractor Performance and Plan for Needed Measures

Published by the Government Accountability Office on 2012-01-13.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States Government Accountability Office

GAO            Report to Congressional Committees




January 2012
               HEALTH CARE
               QUALITY
               MEASUREMENT
               HHS Should Address
               Contractor
               Performance and Plan
               for Needed Measures




GAO-12-136
                                             January 2012

                                             HEALTH CARE QUALITY MEASUREMENT
                                             HHS Should Address Contractor Performance and
                                             Plan for Needed Measures
Highlights of GAO-12-136, a report to
congressional committees




Why GAO Did This Study                       What GAO Found
The Medicare Improvements for                NQF has made progress on projects under its contract activities, as of August
Patients and Providers Act of 2008           2011. Specifically, NQF has completed or made progress on 60 of 63 projects.
(MIPPA) directed the Department of           For example, NQF has completed projects to endorse measures related to
Health and Human Services (HHS) to           various topics, including nursing homes. However, for more than half of the
enter into a 4-year contract with an         projects, NQF did not meet or did not expect to meet the initial time frames
entity to perform various activities         approved by HHS. For example, NQF completed one project to retool
related to health care quality               measures—that is, convert previously endorsed quality measures to an
measurement. In January 2009, HHS            electronic format. While the retooling project was expected to be completed by
awarded a contract to the National
                                             September 2010, its completion was delayed by 3 months. NQF and HHS
Quality Forum (NQF), a nonprofit
                                             officials identified various reasons that contributed to this delay, including an
organization that endorses health care
quality measures—that is, recognizes
                                             expansion of the project’s scope and complexity. As a result of the delay, HHS
certain ones as national standards. In       did not have all the retooled measures it expected to include in its Electronic
2010, the Patient Protection and             Health Records (EHR) Incentive Program. The delay of this project was also a
Affordable Care Act (PPACA)                  contributing factor to NQF exceeding its estimated cost for its entire contract
established additional duties for NQF.       activity related to EHR by about $560,000 in the second contract year—
This is the second of two reports            January 14, 2010, through January 13, 2011. While HHS monitored NQF’s
MIPPA required GAO to submit on              progress through monthly progress reports and approved changes to time frames
NQF’s contract with HHS. In this             and costs, HHS did not use all of the tools for monitoring that are required under
report—which covers NQF’s                    the contract. Specifically, HHS did not conduct an annual performance evaluation
performance under the contract from          to assess timeliness and cost issues that could have helped to inform NQF’s
January 14, 2010, through August 31,         future scope of work. Until August 2011, HHS did not enforce the provision for
2011—GAO examines (1) the status of          NQF to submit a financial graph to compare monthly costs for each contract
projects under NQF’s required contract       activity with cost estimates, which is information not included in monthly progress
activities and (2) the extent to which       reports. These tools could have provided additional, more detailed information to
HHS used or planned to use the               help identify instances in which NQF might have been at risk of not meeting time
measures it has received from NQF            frames or exceeding cost estimates, which could have provided HHS an
under the contract to meet its quality       opportunity to make any appropriate changes to NQF’s activities.
measurement needs, as of August
2011. GAO interviewed NQF and HHS
officials, reviewed relevant laws, and       HHS had used or planned to use about half of the measures—164 of 344—that it
reviewed HHS and NQF documents.              received from NQF under the contract, as of August 2011. For example, HHS
                                             used 44 measures that NQF retooled under the contract in its EHR Incentive
What GAO Recommends                          Program. HHS officials stated that the 44 measures used in the program
GAO recommends HHS: (1) use all              contained errors, which required corrections. HHS officials also have not yet
monitoring tools required under the          tested the retooled measures to assess the feasibility of implementing them in
contract to help address NQF’s               the electronic format; therefore, HHS runs the risk that some of these measures
performance, (2) complete testing of         may not work as intended when implemented. HHS officials told GAO they
retooled measures, and                       expect to evaluate if and how they could use all of the remaining measures HHS
(3) comprehensively plan for its quality     received under the contract. However, HHS has not determined how PPACA
measurement needs. HHS neither               requirements for quality measurement may have changed its needs for endorsed
agreed nor disagreed with these              quality measures. As a result, HHS has not established a comprehensive plan
recommendations. NQF concurred with          that identifies its measurement needs and time frames for obtaining endorsed
many of the findings in the report and       measures and that accounts for relevant PPACA requirements. Without such a
provided additional context.                 plan, HHS may be limited in its efforts to prioritize which specific measures it
                                             needs to develop and to have endorsed by NQF during the remainder of the NQF
View GAO-12-136. For more information,       contract. As a result, HHS may be unable to ensure that the agency receives the
contact Linda T. Kohn at (202) 512-7114 or   quality measures needed to meet PPACA requirements, including time frames
kohnl@gao.gov.                               for implementing quality measurement programs.

                                                                                     United States Government Accountability Office
Contents


Letter                                                                                    1
               Background                                                                 7
               NQF Made Progress on Projects under Each of Its Contract
                 Activities but Has Not Met Expected Time Frames and Has
                 Exceeded Cost Estimates                                                13
               NQF Reported Costs and Fixed Fees of over $20 Million from
                 January 2010 through August 2011                                       27
               HHS Used or Planned to Use Some NQF Measures Received under
                 the Contract but Has Not Comprehensively Planned for Other
                 Measurement Needs to Implement PPACA                                   30
               Conclusions                                                              37
               Recommendations for Executive Action                                     39
               Agency and Other External Comments and Our Evaluation                    40

Appendix I     Health Care Quality Measurement Framework                                45

               HHS’s Role in Planning and Identifying Gap Areas                         47

Appendix II    How GAO Categorized NQF’s 16 Tasks under the Contract into
               Nine Contract Activities                                                 49



Appendix III   Status of NQF’s Nine Contract Activities and the Projects under
               Each Activity GAO Identified                                             51



Appendix IV    Summary of Projects to Review the National Quality Forum’s
               Endorsement Process                                                      62



Appendix V     Comments from the Department of Health and Human Services                64



Appendix VI    Comments from the National Quality Forum                                 69




               Page i                                    GAO-12-136 NQF Contract Performance
Appendix VII   GAO Contact and Staff Acknowledgments                                    74



Tables
               Table 1: Description of the Nine NQF Contract Activities
                        Categorized by GAO                                                9
               Table 2: Status of Measures Used or Planned for Use under the
                        NQF Contract, as of August 31, 2011                             33
               Table 3: GAO’s Categorization of the NQF Contract’s 16 Tasks into
                        Nine Activities                                                 50
               Table 4: Status of Projects under NQF’s Contract Activity to
                        Provide Recommendations on a National Strategy and
                        Priorities for Quality Measurement, as of August 31, 2011       52
               Table 5: Status of Projects under NQF’s Contract Activity to
                        Endorse Health Care Quality Measures, as of August 31,
                        2011                                                            53
               Table 6: Status of Projects under NQF’s Contract Activity to
                        Maintain Previously Endorsed Health Care Quality
                        Measures, as of August 31, 2011                                 54
               Table 7: Status of Projects under NQF’s Contract Activity to
                        Promote the Development and Use of Electronic Health
                        Records, as of August 31, 2011                                  56
               Table 8: Status of Projects under NQF’s Contract Activity to Submit
                        an Annual Report to Congress and the Secretary of HHS,
                        as of August 31, 2011                                           57
               Table 9: Status of Projects under NQF’s Contract Activity to
                        Provide Multistakeholder Input into HHS’s National
                        Strategy for Quality Improvement in Healthcare, as of
                        August 31, 2011                                                 58
               Table 10: Status of Projects under NQF’s Contract Activity to
                        Provide Multistakeholder Input on Selection of Quality
                        Measures, as of August 31, 2011                                 59
               Table 11: Status of Projects under NQF’s Other Health Care Quality
                        Measurement Activity Not Identified in MIPPA or PPACA,
                        as of August 31, 2011                                           60
               Table 12: Status of Projects under NQF’s Administrative Contract
                        Activity                                                        61




               Page ii                                   GAO-12-136 NQF Contract Performance
Figures
          Figure 1: Estimated Time Frames and Completion Dates for All
                   Five Projects under NQF’s Endorsement Contract
                   Activity, as of August 31, 2011                                 20
          Figure 2: Estimated Time Frames for Eight Projects Related to
                   NQF’s Contract Activity on the Promotion and Use of
                   EHRs, as of August 31, 2011                                     22
          Figure 3: NQF’s Reported Contract Activity Costs and Fixed Fees
                   Totaling Approximately $12.8 Million during the Second
                   Contract Year (Jan. 14, 2010—Jan. 13, 2011)                     28
          Figure 4: NQF’s Reported Contract Activity Costs and Fixed Fees
                   Totaling Approximately $9.6 Million during the Period of
                   the Third Contract Year Covered in Our Review (Jan. 14,
                   2011—Aug. 31, 2011)                                             29
          Figure 5: Health Care Quality Measurement Framework                      46




          Page iii                                  GAO-12-136 NQF Contract Performance
Abbreviations

ASPE                                Assistant Secretary for Planning and
                                      Evaluation
CHIP                                Children’s Health Insurance Program
CHIPRA                              Children’s Health Insurance Program
                                      Reauthorization Act of 2009
CDP                                 consensus development process
CMS                                 Centers for Medicare & Medicaid Services
EHR                                 Electronic Health Record
ESRD                                end-stage renal disease
FAR                                 Federal Acquisition Regulation
HAC                                 Healthcare Acquired Condition
HHS                                 Department of Health and Human Services
IT                                  Information Technology
IEP                                 Internal Evaluation Plan
LTCH                                Long-Term Care Hospitals
MAP                                 Measure Applications Partnership
MAT                                 Measure Authoring Tool
Mathematica                         Mathematica Policy Research, Inc.
MIPPA                               Medicare Improvements for Patients and
                                      Providers Act of 2008
NPP                                 National Priorities Partnership
NQF                                 National Quality Forum
National Quality Strategy           National Strategy for Quality Improvement
                                      in Healthcare
NTTAA                               National Technology Transfer and
                                      Advancement Act of 1995
OMB                                 Office of Management and Budget
PPACA                               Patient Protection and Affordable Care Act
PPS                                 Prospective Payment System
QDM                                 Quality Data Model
SRE                                 Serious Reportable Event



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page iv                                             GAO-12-136 NQF Contract Performance
United States Government Accountability Office
Washington, DC 20548




                                   January 13, 2012

                                   Congressional Committees

                                   The nation’s health care spending was about $2.5 trillion in 2009. Health
                                   care spending has been increasing in recent years by an average of
                                   nearly 7 percent per year since 2000, and is projected to reach over
                                   $4.5 trillion by 2020. 1 However, higher levels of spending, such as
                                   spending more to increase the number or technical complexity of
                                   treatments, do not necessarily lead to a corresponding increase in the
                                   quality of care. 2 To better align quality and health care spending, the
                                   Department of Health and Human Services (HHS) has developed various
                                   programs and initiatives to measure health care quality and to provide
                                   financial incentives to health care providers to improve quality and to
                                   reduce the use of unnecessary and costly services. These programs and
                                   initiatives include, for example, pay-for-performance programs that rely on
                                   information collected on various health care quality measures. 3 Health
                                   care quality measures are used to evaluate how health care is delivered,
                                   and information obtained from such measures can promote accountability
                                   among health care providers. The Patient Protection and Affordable Care
                                   Act (PPACA), enacted in March 2010, directed HHS to expand its current
                                   programs and initiatives, such as pay-for-performance programs, and to
                                   implement new ones that will use health care quality measures. PPACA
                                   establishes specific time frames for HHS to implement these programs
                                   and initiatives that rely on various health care quality measures over the
                                   next several years. For example, PPACA requires HHS to establish a




                                   1
                                    Office of the Actuary, Centers for Medicare & Medicaid Services (CMS), National Health
                                   Expenditures Tables, table 1, downloaded 09/2/11 from
                                   https://www.cms.gov/NationalHealthExpendData/downloads/tables.pdf. In 2009, the
                                   national spending on health care accounted for 17.6 percent of gross domestic product,
                                   which totaled $14.1 trillion. The federal government share of health care spending for that
                                   same year was $678 billion, or 27 percent.
                                   2
                                    GAO, Value in Health Care: Key Information for Policymakers to Assess Efforts to
                                   Improve Quality While Reducing Costs, GAO-11-445 (Washington, D.C.: July 26, 2011).
                                   3
                                    The Physician Quality Reporting System, for example, is an HHS quality reporting
                                   program that provides an incentive payment to eligible professionals, such as physicians,
                                   who satisfactorily report data on various quality measures for services that are covered by
                                   Medicare.




                                   Page 1                                              GAO-12-136 NQF Contract Performance
hospital value-based purchasing program. 4 HHS estimates that in fiscal
year 2013 its Hospital Value-Based Purchasing Program 5 will provide
$850 million in Medicare incentive payments to hospitals that meet certain
performance measures. 6 Additionally, HHS is required under PPACA to
establish value-based payments for physicians, develop a plan for value-
based purchasing programs for skilled nursing home facilities and home
health agencies, and create pilot programs that test the implementation of
a value-based purchasing program for other providers—psychiatric, long-
term care, inpatient rehabilitation, and cancer hospitals as well as hospice
programs.

For many of HHS’s health care quality measurement programs and
initiatives, HHS is required to give consideration to the quality measures
endorsed by a consensus-based entity, such as the National Quality
Forum (NQF). NQF is the entity in the United States with the lead
responsibility for endorsing health care quality measures—that is,
determining which measures should be recognized as national standards.
The Medicare Improvements for Patients and Providers Act of 2008
(MIPPA) directed the agency to enter into a 4-year contract with an entity
to endorse health care quality measures and conduct other activities, and
HHS contracted with NQF in January 2009. 7 NQF is a nonprofit
organization established in 1999 in order to foster agreement on national
standards for measurement and public reporting of health care
performance data. MIPPA provided $10 million per year for each of fiscal
years 2009 through 2012—$40 million in total—from the Medicare Trust
Funds for this 4-year contract, which covers the period from January 14,
2009, through January 13, 2013. PPACA provided an additional


4
 Value-based purchasing programs are pay-for-performance programs that require
providers to collect and report information on health care quality measures and adjust
payment levels based on providers’ performance against the measures.
5
 For payment purposes, CMS divides hospital providers into various categories, including
acute care inpatient hospitals, long-term care hospitals, and psychiatric hospitals, among
others. The Hospital Value-Based Purchasing Program applies to acute care inpatient
hospitals.
6
 See 76 Fed. Reg. 2454 (Jan. 13, 2011). PPACA specified that the total amount of value-
based payments must be equal to the total amount of reductions to payments for all
hospitals, so the program has a net budget-neutral impact. Also see Pub. L. No. 111-148,
§ 3001, 124 Stat. 119, 353-62 (2010).
7
 Pub. L. No. 110-275, § 183, 122 Stat. 2494, 2583-86. The contract may be renewed at
the end of the 4-year period after a subsequent bidding process.




Page 2                                              GAO-12-136 NQF Contract Performance
$100 million—$20 million per year for each of fiscal years 2010 through
2014—for additional health care quality measurement work to be
conducted by NQF and HHS through fiscal year 2014. 8 Because PPACA
does not specify the proportion of this amount for NQF activities, HHS
officials told us that they plan to obligate approximately $10 million per
year for the NQF contract, beginning in 2011, and that they may extend
the contract through fiscal year 2014.

MIPPA also directed us to submit two reports on the performance of and
costs incurred by NQF under its 4-year contract with HHS. 9 On July 14,
2010, we issued the first report, which contained information about NQF’s
performance and costs during the first contract year, which began
January 14, 2009, and ended January 13, 2010. 10 This second report
covers NQF’s performance and costs for the second contract year—
January 14, 2010, through January 13, 2011—and part of the third
contract year—January 14, 2011, through August 31, 2011. Specifically,
this report examines (1) the status of projects under the activities NQF is
required to complete under its contract with HHS, (2) the costs and fixed
fees NQF reported under the contract from January 2010 through August
2011, and (3) the extent to which HHS used or planned to use the
measures it receives from NQF under the contract to meet its quality
measurement needs, as of August 31, 2011. In addition, appendix I of this
report provides information about a framework for the various stages of a
health care quality measure, including measure endorsement, as
described by HHS and NQF officials and others.

To describe the status of projects under NQF’s required contract activities
from January 2010 through August 2011, we reviewed relevant provisions
in MIPPA and PPACA and documents including the HHS contract with
NQF, NQF’s 2009, 2010, and 2011 final annual work plans that cover the



8
Pub. L. No. 111-148, § 3014(c), 124 Stat. 119, 387 (2010).
9
 MIPPA states that GAO’s reports shall be submitted no later than 18 months and
36 months, respectively, after the effective date of the contract.
10
  GAO, Health Care Quality Measurement: The National Quality Forum Has Begun a
4-Year Contract with HHS, GAO-10-737 (Washington, D.C.: July 14, 2010). In this report,
we found that (1) NQF had begun work for each of the five duties required by MIPPA
related to quality measures, (2) NQF reported costs and fixed fees totaling approximately
$6.5 million for the first contract year, and (3) NQF and HHS rely on reviews of NQF
invoices in order to help ensure that NQF’s reported costs are proper.




Page 3                                             GAO-12-136 NQF Contract Performance
contract activities in response to MIPPA, 11 NQF’s 2011 technical proposal
for the contract activities in response to PPACA, 12 and NQF’s monthly
progress and monthly internal evaluation plan reports that NQF is
required to submit to HHS. 13 For purposes of our work, we categorized
the 16 tasks that NQF is required to perform under the contract from
January 2010 through August 2011 into 9 contract activities. (See app. II
for more detailed information on how we categorized these contract
activities.) In addition, we identified the various projects NQF is required
to perform under the 9 contract activities we identified. Specifically, for the
purposes of our work, we identified 63 projects NQF is required to
perform under the 9 contract activities, as shown in appendix III. We then
compared the information we obtained from NQF on its progress on the
63 projects under the 9 activities to the requirements for NQF’s
performance under the contract with HHS and the expected time frames
and cost estimates initially established by HHS and NQF in the final
annual MIPPA work plans and other NQF documents for completing the
work. We also interviewed NQF officials responsible for implementing the
contract and HHS officials responsible for managing the contract and
overseeing NQF’s performance. We also compared evidence from HHS
and NQF documents with evidence from discussions with HHS and NQF
officials to corroborate any issues or challenges with NQF’s performance.
Based on our comparison and interviews with officials, we identified any
changes to the work NQF is required to perform and changes to the time
frames or cost estimates established in the final work plans developed
annually to respond to MIPPA and a budget developed as part of NQF’s
technical proposal to respond to PPACA.

To assess the costs and fixed fees NQF reported under the contract from
January 2010 through August 2011, we reviewed NQF documents that



11
  Under the contract, NQF is required to work with HHS to develop an annual work plan
for each contract year that outlines the contract activities in response to MIPPA. The work
plan also delineates scheduled time frames for the work NQF is expected to perform that
contract year and cost estimates for the activities. NQF officials told us HHS obligates
funds on a yearly basis following HHS’s approval of the work plan.
12
  The technical proposal is submitted to HHS and incorporated into the scope of work.
13
  The contract requires that monthly progress reports include NQF’s accomplishments for
the month under each of the contract activities, any problems encountered or anticipated
and the impact on project time frames, and a financial graph that compares NQF’s
monthly reported costs for each of the contract activities with initial cost estimates, among
other things.




Page 4                                               GAO-12-136 NQF Contract Performance
include financial information related to the contract, such as NQF’s final
work plans developed in 2010 and 2011 to respond to MIPPA and a
budget developed as part of NQF’s technical proposal to respond to
PPACA that describe NQF’s estimated costs for each of the nine activities
we identified under the contract. We obtained and summarized
information from NQF monthly invoices submitted to HHS from January
14, 2010, through August 31, 2011, for direct 14 and indirect 15 costs
incurred by NQF under the contract and fixed fees. 16 We compared
NQF’s reported costs with its cost estimates for the activities in NQF’s
2010 final annual work plan and determined whether or to what extent
NQF’s reported costs exceeded its cost estimates. We did not assess the
accuracy of NQF’s reported costs. 17 We also reviewed the HHS contract
with NQF and any contract modification since January 14, 2010. We also
interviewed NQF and HHS officials responsible for reviewing and
approving the costs and fixed fees submitted under the contract. Based
on our review of documents and interviews with NQF and HHS officials,
we determined that the reported costs and fixed-fee data were sufficiently
reliable for the purposes of this report.

To examine the extent to which HHS used or planned to use measures it
receives from NQF under the contract to meet its quality measurement
needs, as of August 2011, we reviewed statutory requirements and
documents related to HHS health care quality measurement efforts. We
relied on HHS to identify programs and initiatives that use or will use
health care quality measures, such as programs or initiatives established


14
  Direct costs are costs incurred specifically for the contract, such as labor costs and
payments to subcontractors and consultants.
15
  Indirect costs are costs that are not directly attributable to a specific project or function.
These costs cover additional items such as employee benefits, overhead, and
administrative costs.
16
  NQF’s contract requires that, in accordance with 48 C.F.R. § 52.216-8, the fixed fee be
paid monthly until fee payments reach 85 percent of the total amount of the fixed fee
authorized, and that after they reach 85 percent HHS may withhold a reserve up to
15 percent or $100,000, whichever is less, to protect the government’s interest.
17
  NQF and HHS rely on reviews of NQF invoices in order to help ensure that NQF’s
reported costs are proper. NQF officials told us that they review the invoices prior to
submitting them to HHS and carry out other activities, such as using an electronic system
to track labor hours, in order to help ensure that the costs they report in the invoices are
proper. HHS also conducts reviews of NQF invoices, which are governed by HHS policies
and procedures and by requirements applicable to federal contracts generally. For more
information, see GAO-10-737.




Page 5                                                  GAO-12-136 NQF Contract Performance
by MIPPA, PPACA, and the Health Information Technology for Economic
and Clinical Health Act. 18 As a result, the health care quality programs
and initiatives mentioned in our report may not represent a
comprehensive list of all those that use or plan to use measures received
from NQF under the contract. We interviewed officials from various
divisions within HHS that may use measures that were nationally
recognized—or NQF-endorsed—in their various programs and initiatives
related to health care quality measurement in order to corroborate the
information we obtained from documentation. We obtained information
about how HHS selects measures for its programs and initiatives from our
interviews with HHS officials and reviews of HHS documents. We
determined how HHS has used and planned to use measures that were
received under the contract with NQF by identifying all the measures that
HHS received under the contract and matching each measure to the
quality measures being used or considered for use in HHS programs and
initiatives. 19 For the purpose of this report, we defined a measure as used
if the measure was identified by HHS and included in an HHS program or
initiative as of August 31, 2011, and indicated as such by a final rule or
other HHS documentation. We defined a measure as planned for use if
the measure was identified by HHS for potential use in an HHS program
or initiative as of August 31, 2011, and indicated as such by a proposed
rule or other HHS documentation. In addition, we reviewed HHS
documents and interviewed HHS officials to determine whether or to what
extent HHS has plans for using work conducted under the NQF contract
to meet PPACA requirements related to quality measures for HHS
programs and initiatives. For example, we reviewed HHS plans or
strategies such as the 2011 National Strategy for Quality Improvement in




18
  Pub. L. No. 111-5, div. A, title XIII, div. B, title IV, 123 Stat. 115, 226, 467 (2009).
19
  HHS officials explained that any measure developer can submit a measure to be
considered for NQF endorsement. Each measure developer has its own purpose or plans
for implementing a measure, which may or may not involve HHS quality and public
reporting programs. Therefore, all measures submitted may not be applicable to a
particular HHS health care quality program or initiative. HHS officials told us that they will
review all the measures received under the contract to determine if they are applicable to
their health care quality programs or initiatives.




Page 6                                                  GAO-12-136 NQF Contract Performance
             Healthcare (National Quality Strategy), 20 that describe the agency’s plans
             to help meet its quality measurement needs.

             We conducted this performance audit from April 2011 through January
             2012 in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objective. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objectives.


             NQF is a nonprofit organization established in 1999 in order to foster
Background   agreement, or consensus, on national standards for measuring and public
             reporting of health care performance data. 21 Its membership includes
             more than 400 organizations that represent multiple sectors of the health
             care system, including providers, consumers, and researchers. NQF’s
             mission focuses on three core areas: (1) building consensus on national
             priorities and goals for performance improvement and working in
             partnership to achieve them, (2) endorsing national consensus standards
             for measuring and publicly reporting on performance, and (3) promoting
             the attainment of national goals through education and outreach
             programs.




             20
               The 2011 National Strategy for Quality Improvement in Healthcare, also referred to as
             the National Quality Strategy, focused on six priorities: (1) making care safer; (2) ensuring
             person- and family-centered care; (3) promoting effective communication and coordination
             of care; (4) promoting the most effective prevention and treatment of the leading causes of
             mortality, starting with cardiovascular disease; (5) working with communities to promote
             wide use of best practices to enable healthy living; and (6) making quality care more
             affordable. The National Quality Strategy was due on January 1, 2011, as required by
             PPACA, and was issued on March 21, 2011. HHS is expected to provide annual updates
             to the report.
             21
               NQF is recognized as a consensus body under the National Technology Transfer and
             Advancement Act of 1995 (NTTAA) and the Office of Management and Budget (OMB)
             Circular A-119, which provides NQF the authority needed to facilitate its public/private
             partnership. OMB Circular A-119 defines “consensus” as a “general agreement, not
             necessarily unanimity” so that attempts may be made to resolve objections and members
             may have an opportunity to amend their votes after reviewing comments. In addition, the
             circular states that the voluntary consensus process must adhere to the following five
             elements: openness, balance, due process, consensus, and an appeal mechanism. See
             Pub. L. No. 104-113, 110 Stat. 775 (1996).




             Page 7                                               GAO-12-136 NQF Contract Performance
                          Prior to its contract with HHS, NQF established a consensus development
                          process (CDP) to evaluate available health care quality measures to
                          determine which ones are qualified to be endorsed—that is, recognized—
                          as national standards. Under this process, organizations that develop
                          quality measures submit them to NQF for consideration, in response to
                          specific solicitations by NQF. 22 NQF forms a committee of experts from its
                          member organizations as well as other organizations and agencies to
                          conduct an objective and transparent review of these quality measures
                          against four standardized criteria established by NQF, such as whether
                          the measures are scientifically acceptable. 23 After this committee
                          evaluates the measures against these criteria, NQF’s process allows for a
                          period during which its member organizations and the public may
                          comment on the committee’s recommendation for each measure. The
                          process also provides for a period for its member organizations to vote on
                          whether the measures should be endorsed by NQF as a national
                          standard. Ultimately, NQF’s board of directors makes a final decision on
                          whether NQF should formally endorse the measures.

                          As of October 2011, NQF has endorsed over 600 health care quality
                          measures in 27 areas, such as cancer and diabetes. HHS uses NQF-
                          endorsed measures in its programs and initiatives to promote quality
                          measurement, and NQF continues to endorse quality measures separate
                          from its contract with HHS.


NQF’s Contract with HHS   NQF’s work under the contract includes endorsement of quality measures
                          and other activities that are expected to support HHS’s quality
                          measurement efforts, such as through value-based purchasing programs.
                          Specifically, NQF’s work under the contract consists of various projects
                          under the nine contract activities related to health care quality
                          measurement. The work plans developed annually to respond to MIPPA
                          and NQF’s technical proposal to respond to PPACA delineate the projects



                          22
                            NQF does not develop quality measures itself; rather, NQF endorses quality measures
                          developed by other organizations, such as the Joint Commission, the National Committee
                          for Quality Assurance, and the American Medical Association.
                          23
                            A review for scientific acceptability includes an assessment of data from measure testing
                          conducted by measure developers to help ensure that the measure produces consistent
                          and credible results about the quality of care when implemented. The remaining three
                          criteria NQF established for its CDP process are (1) importance to measure and report,
                          (2) usability, and (3) feasibility.




                          Page 8                                              GAO-12-136 NQF Contract Performance
                                            NQF is required to conduct under the nine contract activities, as well as
                                            expected time frames and cost estimates for the projects for each year.
                                            Table 1 provides more detailed information on the nine contract activities.
                                            Some of these activities are required by either MIPPA or PPACA, while
                                            others are quality measurement activities established by HHS or
                                            administrative activities.

Table 1: Description of the Nine NQF Contract Activities Categorized by GAO

NQF contract activity categorized by GAO        Description of activity
Contract activities required by MIPPA or PPACA
Recommendations on a National Strategy          NQF will synthesize evidence and convene key stakeholders to make
and Priorities for Quality Measurement          recommendations on a national strategy and priorities for health care performance
(as required by MIPPA)                          measurement.
Endorsement of Health Care Quality Measures     NQF will conduct endorsement of quality measures using its consensus
(as required by MIPPA)                          development process to determine which ones should be endorsed as national
                                                standards. Specifically, it will provide for the endorsement of quality measures
                                                through projects that focus on priority areas identified by HHS, such as patient
                                                outcomes and patient safety. In addition to endorsing measures, this activity
                                                includes an evaluation of NQF’s endorsement process to identify ways to improve
                                                its efficiency and effectiveness.
Maintenance of Endorsed Quality Measures        NQF will use its process to ensure that endorsed measures are maintained—that is,
(as required by MIPPA)                          updated or retired. This process is similar to NQF’s endorsement process, in that it
                                                involves a review of measures against established criteria, a period for public
                                                                                                            a
                                                comment, and a final decision by NQF’s board of directors. Under this activity, NQF
                                                will conduct maintenance projects using its 3-year review cycle that compares
                                                previously endorsed measures against new measures submitted for endorsement.
                                                In addition to the 3-year review cycles, NQF can also maintain measures through:
                                                (1) annual updates, which is a process where NQF annually reviews and receives
                                                information on a measure to determine whether the measure has undergone any
                                                changes; (2) ad hoc reviews where NQF reviews an endorsed measure at any time
                                                based on justifiable evidence to substantiate the review; and (3) time-limited
                                                endorsement reviews, which is a process where NQF will review time-limited
                                                endorsed measures—measures that have been endorsed for a limited period of
                                                time to allow for testing by measure developers.
Promotion of the Development and Use of         NQF will undertake projects to promote the development and use of electronic
Electronic Health Records (as required by       health records for use in quality measurement, including to “retool” measures—that
MIPPA)                                          is, convert previously endorsed quality measures to an electronic format that is
                                                compatible with electronic health records. In addition, NQF will undertake projects
                                                related to developing tools to capture data for performance measurement, creating
                                                a framework that defines a standardized set of data that should be captured in
                                                patients’ electronic health records, and providing data to providers to support clinical
                                                decisions. These efforts had not been performed by NQF prior to the contract.




                                            Page 9                                              GAO-12-136 NQF Contract Performance
NQF contract activity categorized by GAO             Description of activity
Annual Report to Congress and the Secretary          NQF is required to submit an annual report to Congress and the Secretary of HHS
of Health and Human Services (as required            that provides a summary of NQF’s progress on activities under the contract during
by MIPPA and amended by PPACA)                       the year.
                                                     PPACA established additional requirements that NQF must perform for this activity
                                                     that include providing additional information in its annual report to Congress and the
                                                     Secretary of HHS, beginning with its report due March 1, 2012. Specifically, NQF
                                                     will include additional information in the report on gaps in endorsed and
                                                     nonendorsed health care quality measures and a summary of activities conducted
                                                     by multistakeholder groups during the third contract year (January 14, 2011 –
                                                     January 13, 2012).
Multistakeholder Input into HHS’s National      In 2010, NQF convened a multistakeholder group to provide input on national
Strategy for Quality Improvement in Health Care priorities for improvement in population health and in the delivery of health care
(as required by PPACA)                          services for consideration under HHS’s National Strategy for Quality Improvement in
                                                                                              b
                                                Healthcare, or the National Quality Strategy.
                                                In 2011, NQF will continue to engage a multistakeholder group to provide input on a
                                                plan for measuring and improving health and health care focusing on the six
                                                                                                              c
                                                priorities identified in HHS’s 2011 National Quality Strategy. In addition, NQF will
                                                                                                            d
                                                conduct endorsement projects for cross-cutting measures that can support national
                                                implementation of these priorities.
Multistakeholder Input on the Selection of           NQF will convene a multistakeholder group to provide input on the selection of
Quality Measures for use in Payment Programs         measures for use in various public programs, including measures for use in quality
and Value-Based Purchasing Programs Under            reporting programs for physicians, postacute care programs under Medicare,
                                                                                                                     e
PPACA, Other Private/Public Payers, and Other        Prospective Payment System (PPS) exempt cancer hospitals, and hospice care. In
Programs (as required by PPACA)                      addition, the group will provide input on measures that address quality issues
                                                     identified for vulnerable populations, such as dual-eligible beneficiaries.
Other health care quality measurement activities not identified in MIPPA or PPACA
Other health care quality measurement activity       NQF will work on additional health care quality measurement projects established
                                                     by HHS to fill immediate areas of need, including two white papers related to
                                                     resource use and efficiency; efficiency and resource use endorsement projects; and
                                                                                                f
                                                     projects related to measure harmonization.
Administrative activities
Administrative activity                              NQF will perform a number of administrative activities that focus on project planning
                                                     and contract management: (1) conduct an opening meeting between HHS and
                                                     NQF, (2) develop an annual work plan, (3) develop a monthly internal evaluation
                                                     plan for quality assurance, (4) develop a monthly progress report, (5) conduct
                                                                                                                g
                                                     weekly conference calls, and (6) develop a public website.
                                            Source: GAO analysis of MIPPA, PPACA, and the NQF contract as well as the 2010 and 2011 NQF final annual MIPPA work plans and
                                            2011 PPACA technical proposal.
                                            a
                                             Prior to its contract with HHS, NQF established a process for maintenance of measures. In May
                                            2010, NQF updated this process by requiring all previously endorsed measures to go through a
                                            comprehensive review every 3 years that will include a comparison against newly submitted
                                            measures submitted for endorsement. For example, NQF will review all previously endorsed and
                                            newly submitted measures related to cardiovascular health simultaneously.
                                            b
                                                NQF convened the National Priorities Partnership to complete this work.
                                            c
                                             The six priorities identified in the National Quality Strategy are: (1) making care safer; (2) ensuring
                                            person- and family-centered care; (3) promoting effective communication and coordination of care;
                                            (4) promoting the most effective prevention and treatment of the leading causes of mortality, starting
                                            with cardiovascular disease; (5) working with communities to promote wide use of best practices to
                                            enable healthy living; and (6) making quality care more affordable.
                                            d
                                                Cross-cutting measures are those that affect all or most patients, such as patient safety.




                                            Page 10                                                                GAO-12-136 NQF Contract Performance
e
 PPS-exempt cancer hospitals are those exempt from the Prospective Payment System (PPS), which
is a system of Medicare reimbursement in which providers receive a predetermined, fixed payment
based on the service provided and its classification, such as the patient’s diagnosis at the time of
hospital admission.
f
 Harmonization is a process where related measures are standardized in a manner by which they can
be used in multiple settings when relevant. NQF defines “related measures” as those that either
(1) address the same concepts for measure focus (target process, condition, event, outcome) but a
different target population; or (2) address different concepts for measure focus (target process,
condition, event, outcome) but the same target population.
g
Development of a public website was part of the 2010 final annual MIPPA work plan but this work
was canceled in 2011.


To help determine the activities and the projects under the nine contract
activities that NQF is expected to perform during each contract year, HHS
has established an interagency workgroup that comprises officials from
multiple divisions within HHS, including the Agency for Healthcare
Research and Quality, the Centers for Medicare & Medicaid Services
(CMS), and the Office of the National Coordinator for Health Information
Technology. The workgroup is responsible for prioritizing and selecting
the activities and projects under each activity that NQF is expected to
perform during each contract year. 24 HHS officials told us that the
representatives from these various HHS agencies provide input on the
work NQF is expected to perform, including determining quality measures
requested from NQF for their respective programs. The activities and
projects selected by the interagency workgroup become part of NQF’s
scope of work under the contract. 25 Some of the projects under the
contract activities that NQF is expected to perform during the year will be
ongoing from the previous contract year while new work will be
incorporated into the work plan as necessary.

For the NQF contract, HHS selected a cost-plus-fixed-fee contract, under
which HHS reimburses NQF for actual costs incurred under the contract
in addition to a fixed fee that is unrelated to costs. Cost-plus-fixed-fee
contracts are used for efforts such as research, design, or study efforts
where costs and technical uncertainties exist and it is desirable to retain


24
  The office of the Assistant Secretary for Planning and Evaluation (ASPE)—a component
of HHS—is responsible for facilitating the workgroup discussions, which occur at least
biweekly.
25
  The contract requires NQF to develop a work plan for each contract year that describes
the MIPPA projects under the activities that NQF is required to perform. The MIPPA
workplan includes expected time frames and cost estimates for projects under the contract
activities. The work NQF is required to perform in response to PPACA is outlined in a
technical proposal submitted to HHS and incorporated into the scope of work.




Page 11                                                  GAO-12-136 NQF Contract Performance
as much flexibility as possible in order to accommodate change. 26
However, this type of contract provides only a minimum incentive to the
contractor to control costs. As we reported in 2009, these contracts are
suitable when the cost of work to be done is difficult to estimate and the
level of effort required is unknown. 27 This cost-plus-fixed-fee contract is
NQF’s first cost-reimbursement contract. For cost-reimbursement
contracts, the Federal Acquisition Regulation (FAR) 28 requires
appropriate government surveillance 29 during performance to provide
reasonable assurance that efficient methods and effective cost controls
are used. Under the FAR, contracts are to contain a provision for agency
approval of a contractor’s subcontracts. 30 HHS’s contract with NQF
contains this provision and also requires the approval of consultants
engaged under the contract. The review and approval of NQF’s use of
subcontractors and consultants require appropriate support
documentation provided by NQF to HHS, including a description of the
services, the proposed price, and a negotiation memo that reflects the
principal elements of the price negotiations between NQF and the
subcontractor or consultant. Under its contract with HHS, NQF has
utilized 31 subcontractors and 16 consultants since January 14, 2010, to
provide support to NQF on many of the contract activities and associated
projects.

Two HHS components are principally responsible for administering the
NQF contract: the office of the Assistant Secretary for Planning and
Evaluation (ASPE) and CMS—an agency within HHS. 31 Specifically, the



26
 See 48 C.F.R. §§ 16.301-2, 16.306 (2010).
27
 For more information on cost-plus-fixed-fee contracts, see GAO, Contract Management:
Extent of Federal Spending under Cost-Reimbursement Contracts Unclear and Key
Controls Not Always Used, GAO-09-921 (Washington, D.C.: Sept. 30, 2009).
28
  48 C.F.R. ch.1. The FAR establishes uniform policies for acquisition of supplies and
services by executive agencies. Agency acquisition regulations may implement or
supplement the FAR.
29
  Surveillance includes oversight of a contractor’s work to provide assurance that the
contractor is providing timely and quality goods or services and to help mitigate any
contract performance problems.
30
 48 C.F.R. §§ 44.204(a)(1), 52.244-2 (2010).
31
  Within CMS, the Office of Acquisition and Grants Management is responsible for
administering the NQF contract.




Page 12                                             GAO-12-136 NQF Contract Performance
                         project officer for the NQF contract is a representative of ASPE. 32 This
                         individual is responsible for program management and works with the
                         contracting officer to oversee the contract. The contracting officer for the
                         NQF contract, responsible for administering the contract, is a
                         representative of CMS. 33 The contract outlines various activities that the
                         contracting officer is required to perform, such as conducting an annual
                         evaluation of the contractor’s performance.


                         From January 14, 2010, through August 31, 2011, NQF made progress
NQF Made Progress        on projects under its contract activities. However, our review of NQF
on Projects under        documents found that NQF had not met or did not expect to meet time
                         frames on more than half of the projects, and it exceeded its cost
Each of Its Contract     estimates for projects under three of the contract activities. HHS did not
Activities but Has Not   use all tools for monitoring that are required under the contract.
Met Expected Time
Frames and Has
Exceeded Cost
Estimates




                         32
                           The project officer serves as the technical representative of the contracting officer, and
                         provides technical direction to NQF for all tasks described in the NQF contract. In addition,
                         the project officer monitors NQF’s performance and reviews invoices for payment. Another
                         HHS official within ASPE, the project manager, also provides support to the project officer
                         for the NQF contract.
                         33
                           The contracting officer enters into, administers, and terminates government contracts.
                         The contracting officer negotiates and prepares contract documents, modifies terms or
                         conditions of the contract, and approves payment of invoices, among other tasks.




                         Page 13                                              GAO-12-136 NQF Contract Performance
NQF Made Progress on         From January 14, 2010, through August 31, 2011, NQF has made
Projects under Each of the   progress on 60 of the 63 projects under the activities required under its
Contract Activities as of    contract with HHS. Specifically, NQF had completed 26 projects and was
                             continuing to work on the remaining 34 projects. 34 (App. III provides the
August 31, 2011
                             status of all contract activities and the projects under each activity NQF
                             was expected to perform during our reporting period.) Examples of
                             projects under the contract activities include both completed and
                             continuing projects:

                             •    Endorsement of Measures Activity. NQF endorsed 101 measures
                                  since the beginning of the second contract year by conducting work
                                  on endorsement projects on different topic areas. 35 Specifically, NQF
                                  completed two projects to endorse 38 outcome measures related to
                                  20 high-priority conditions identified by CMS that account for the
                                  majority of Medicare’s costs, 36 and mental health and child health
                                  conditions; and 21 performance measures for chronic and postacute
                                  care nursing facilities. NQF also worked on two projects related to
                                  child health quality and patient safety. As of August 2011, NQF
                                  endorsed 41 child health quality measures and 1 patient safety
                                  measure under these projects. NQF expected to complete the child
                                  health quality project in September 2011 and the patient safety project




                             34
                               The contract activities and related projects that NQF is expected to perform during each
                             year are determined by HHS and NQF, as they develop NQF’s scope of work. As a result,
                             the number of projects varies by year, and some project time frames may extend beyond
                             1 contract year. For our reporting period, we determined that NQF is conducting work on
                             63 projects. As of August 31, 2011, of the 63 projects we identified, 26 have been
                             completed, 34 are in progress, 2 have been canceled, and 1 has not yet started. See
                             app. III for more details on each of these projects.
                             35
                               HHS told us that they consider an NQF measure to be endorsed when the NQF Board
                             ratifies the measures. However, NQF is required to submit a final report to HHS following
                             ratification to complete an endorsement project. For purposes of our report, we considered
                             an endorsement project to be completed when the board ratifies the measures that are the
                             subject of the project.
                             36
                               The 20 conditions are acute myocardial infarction, Alzheimer’s disease and related
                             disorders, atrial fibrillation, breast cancer, cataract, chronic kidney disease, chronic
                             obstructive pulmonary disorder, colorectal cancer, congestive heart failure, diabetes,
                             endometrial cancer, glaucoma, hip/pelvic fracture, ischemic heart disease, lung cancer,
                             major depression, osteoporosis, prostate cancer, rheumatoid arthritis and osteoarthritis,
                             and stroke/transient ischemic attack.




                             Page 14                                             GAO-12-136 NQF Contract Performance
     in December 2011. 37 In addition, NQF completed a contractually
     required review of its endorsement process, subcontracting with
     Mathematica Policy Research, Inc. (Mathematica). The review
     focused on the timeliness and effectiveness of the endorsement
     process; 38 identified inefficiencies, including those that may contribute
     to delays; and recommended, among other steps, that NQF create a
     schedule for its endorsement process for measure developers and
     develop feasible time lines that include clear goals for each
     endorsement project. HHS officials stated that Mathematica’s
     recommendations were valuable because much of the work under the
     NQF contract needs to be completed in an accelerated timeline to
     help fill critical measurement gaps associated with HHS’s health care
     quality programs and initiatives. For more information about this
     review, see appendix IV.

•    Maintenance of Endorsed Quality Measures Activity. NQF
     maintained—that is, updated or retired—124 measures under the
     contract since the beginning of the second contract year. 39 These
     included 41 measures reviewed under NQF’s 3-year review cycle
     related to diabetes, mental health, and musculoskeletal conditions. 40
     In addition, 83 measures were maintained under NQF’s other
     maintenance review processes. 41 NQF was also continuing to work on
     maintenance projects it initiated in 2010 for measures related to
     cardiovascular and surgery measures. As of August 2011, the two


37
  NQF completed the child health quality project by endorsing three additional child health
quality measures in September 2011, as scheduled. As of November 2011, NQF was
continuing to conduct work on the patient safety project, such as through the endorsement
of an additional patient safety measure in September 2011. NQF expected to complete
additional review of measures under this project by December 2011.
38
  Mathematica reviewed 23 projects in its analysis, including 7 endorsement projects
conducted under the HHS contract.
39
  Under NQF’s maintenance process, NQF can retire a measure, or remove its
endorsement status for various reasons, such as if the board determines a measure is
obsolete.
40
  Although these measures were reviewed under the contract, they were not reviewed
under NQF’s updated Endorsement Maintenance Process because the reviews began
prior to its implementation.
41
  During our reporting period, NQF maintained 80 measures using its time-limited
endorsement review process in areas such as infectious diseases and neurology. NQF
also maintained 3 measures under its ad hoc review process related to various conditions,
including patient safety.




Page 15                                             GAO-12-136 NQF Contract Performance
     projects were expected to be completed by December 2011 and
     January 2012, respectively. 42

•    Promotion of the Development and Use of Electronic Health
     Records Activity. NQF has made progress on three projects related
     to retooling—that is, converting previously endorsed quality measures
     to an electronic format that is compatible with electronic health
     records (EHR). First, NQF completed initial retooling of 113
     measures. This work is intended to allow data from EHRs to be used
     for quality measurement, which is a part of HHS’s long-term goal to
     use health information technology to exchange information and
     improve quality of care. Second, as of August 2011, NQF convened
     an expert review panel to review the retooled measures to ensure that
     each retooled measure is properly formatted, the logic is correctly
     stated, and the intent of the measures is maintained in the electronic
     format that will use data obtained from EHRs, instead of from claims
     as originally formatted. Third, as of August 2011, NQF was expected
     to complete another project to provide an updated list of the 113
     retooled measures to HHS by December 2011, which would
     incorporate any revisions identified by the expert review panel and
     others involved in the retooling process. After these updated
     measures are completed, HHS officials told us that they will contract
     with other entities to conduct testing of some of the 113 retooled
     measures to assess the feasibility of implementing the measures in
     the electronic format. Although NQF’s endorsement process requires
     that measure developers submit data on validity and reliability testing
     of measures they submit for endorsement, this testing does not
     include feasibility testing for implementing the measures in an
     electronic format for performance measurement. As of December
     2011, HHS officials did not provide an expected date of completion for
     this feasibility testing but told us that they have awarded two contracts
     that include this in their scope of work. 43 In addition to the retooling


42
  As of November 2011, NQF was still completing the cardiovascular and surgery projects
and expected to complete the work by December 2011 and January 2012, respectively, as
scheduled.
43
  HHS officials told us that they awarded contracts to Mathematica Policy Research, Inc.,
and Abt Associates to conduct feasibility testing on 69 of 113 retooled measures. These
contracts were awarded on June 8, 2011, and July 8, 2011, respectively. For the
remaining 44 retooled measures, HHS officials told us that HHS issued a solicitation for a
contract to conduct feasibility testing of the measures; however, they did not receive any
bids. As of December 2011, they said that they will not be issuing a solicitation for a new
contract to test the 44 retooled measures.




Page 16                                             GAO-12-136 NQF Contract Performance
     projects, NQF is developing a software tool—the Measure Authoring
     Tool—to allow measure developers to create standardized electronic
     measures that help capture information in EHRs so that less retooling
     would be needed in the future. As of August 2011, NQF was
     completing final testing of the beta, or initial, version of this tool. NQF
     expected to complete testing and publish an updated version for
     public use by January 2012. 44

•    Multistakeholder Input into HHS’s National Quality Strategy
     Activity. NQF convened the National Priorities Partnership (NPP), a
     multistakeholder group expected to provide annual input on national
     priorities, among other things, to be considered in the National Quality
     Strategy. 45 As of August 2011, the NPP was completing a report on
     this input, which was then published in September 2011. The report
     noted the need for a national comprehensive strategy that identifies
     core sets of standardized measures to meet each of the national
     priorities HHS identified in the 2011 National Quality Strategy, among
     other things. The NPP noted in the report that a common data
     platform, core measure set, and public reporting mechanism are key
     components of the infrastructure for performance measurement. It
     also highlighted that a strategic plan, road map, and timeline for
     establishing an infrastructure should be accelerated to allow for rapid
     implementation over the next 5 years. Additionally, the NPP reported
     that it was critical that all federal programs drive toward the
     establishment of a common platform for measurement and reporting.

•    Multistakeholder Input on the Selection of Quality Measures
     Activity. NQF has convened the Measure Applications Partnership
     (MAP). The MAP is a multistakeholder group that is expected to
     conduct work in two areas. First, the MAP is expected to provide input
     to the Secretary of HHS on the selection of quality measures for use
     in payment programs and value-based purchasing programs required
     by PPACA, among others. The MAP will review a list of measures
     published by the Secretary of HHS on December 1 of each year, and


44
  As of October 2011, NQF completed testing of the beta version that was made available
for public use in September 2011 and expected to publish an updated version in January
2012, as scheduled.
45
   The National Priorities Partnership (NPP) represents over 48 major national
organizations from the public and private sectors, including consumers, purchasers,
clinicians, and communities. Under PPACA, HHS is required to solicit annual input from
such a group on the National Quality Strategy.




Page 17                                            GAO-12-136 NQF Contract Performance
                            develop a report that contains a framework to help guide measure
                            selection. The MAP will provide its annual input beginning February 1,
                            2012, for measures used in the following 11 programs: hospice,
                            hospital inpatient, hospital outpatient, physician offices, cancer
                            hospitals, end-stage renal disease (ESRD) facilities, inpatient
                            rehabilitation facilities, long-term care hospitals, hospital value-based
                            purchasing, psychiatric hospitals, and home health care. Second, the
                            MAP is expected to publish reports that provide input on the selection
                            of measures for use in various quality reporting programs, including
                            those for physicians. As of August 2011, the MAP had held meetings
                            and initiated its work for reports due October 1, 2011. 46

                       •    Other Health Care Quality Measurement Activity. NQF completed
                            a project to endorse six imaging efficiency measures. NQF was also
                            continuing to work on a project to identify existing quality measures
                            and gap areas related to measurement of regionalized emergency
                            care services. 47


NQF Did Not Meet       Our review of NQF documents found that NQF had not met or did not
Expected Time Frames   expect to meet time frames on more than half of the projects under the
and Exceeded Cost      contract activities that were completed or ongoing, as of August 2011.
                       Specifically, our review of documents found that NQF had not met
Estimates              expected time frames on 18 of the 26 projects it completed under the nine
                       contract activities. Further, NQF did not expect to meet time frames on 14
                       of the 34 projects on which it was continuing to work. The delays of these
                       projects under the contract activities varied in time from about 1 to
                       12 months. HHS officials told us they approved all changes to the time
                       frames, which were established by HHS and NQF in NQF’s 2010 and
                       2011 final annual MIPPA work plans and the PPACA technical proposal.
                       Appendix III provides the status for all projects related to each of the nine
                       contract activities, including information on their expected and actual time
                       frames for completion during our reporting period. Examples of projects
                       under the contract activities for which NQF did not meet or did not expect
                       to meet expected time frames include the following:



                       46
                         On October 4, 2011, NQF reported that it submitted these reports to HHS on October 1,
                       2011, as scheduled.
                       47
                         Regionalized emergency medical care refers to directing patients to emergency facilities
                       with optimal capabilities for a given type of illness or injury to help coordinate emergency
                       care across a region.




                       Page 18                                              GAO-12-136 NQF Contract Performance
•   Endorsement of Measures Activity. NQF did not meet or did not
    expect to meet time frames for all five endorsement projects under the
    endorsement contract activity. 48 (See app. III for details on the five
    projects.) For example, NQF was expected to complete an
    endorsement project for nursing home quality measures in July 2010;
    however, the measures were not endorsed until February 2011. (See
    fig. 1 for estimated time frames and actual completion dates for all
    projects related to the endorsement contract activity.)




48
  As of August 2011, three of these five projects had been completed. However, NQF was
still completing the patient safety project, which was delayed approximately 10 months to
allow for harmonization of measures, and the child health quality project.




Page 19                                            GAO-12-136 NQF Contract Performance
Figure 1: Estimated Time Frames and Completion Dates for All Five Projects under NQF’s Endorsement Contract Activity, as
of August 31, 2011




                                        Note: For purposes of our report, we consider an endorsement project completed when the board
                                        ratifies its measures.
                                        a
                                         NQF had three separate reviews of measures during Phases I and II, resulting in three separate
                                        board endorsements. The circles reflect those three reviews.
                                        b
                                         NQF endorsed one out of two patient safety measures under phase II of the patient safety project.
                                        The remaining measure was still under consideration by the board and is scheduled to be reviewed
                                        for endorsement on September 15, 2011.
                                        c
                                         Mathematica’s report on its evaluation of NQF’s CDP process, or the endorsement process, and
                                        NQF’s response to Mathematica’s recommendations in the report were published in January 2011.
                                        NQF solicited comments on its proposed revisions and additions to its endorsement process, as a
                                        result of the Mathematica evaluation, through April 2011.
                                        d
                                         NQF endorsed 41 out of 44 child health quality measures on August 15, 2011. As of August 2011,
                                        the remaining 3 measures were still under consideration by the board and were scheduled to be
                                        reviewed for endorsement on September 15, 2011.




                                        Page 20                                                  GAO-12-136 NQF Contract Performance
     NQF officials stated that several factors contributed to NQF exceeding
     the expected time frames for the five endorsement projects, including
     the high volume of measures submitted for review, the amount of time
     it took to harmonize measures 49 between measure developers, and a
     need for additional technical expertise on review panels.

•    Promotion of the Development and Use of Electronic Health
     Records Activity. NQF did not meet or did not expect to meet
     expected time frames for five out of eight projects related to the EHR
     contract activity. 50 For example, NQF was expected to complete its
     initial retooling of 113 endorsed quality measures into electronic
     formats by September 2010, but this effort was not completed until
     December 2010. (See fig. 2 for estimated time frames and actual
     completion dates for all projects related to the EHR contract activity.)
     In addition, NQF was expected to complete the project to convene an
     expert panel to review the 113 retooled measures by January 2011.
     However, the panel did not complete its review of the 113 measures
     until June 2011.




49
  Harmonization is a process where related measures are standardized in a manner by
which they can be used in multiple settings when relevant. NQF defines “related
measures” as those that either (1) address the same concepts for measure focus (target
process, condition, event, outcome) but a different target population; or (2) address
different concepts for measure focus (target process, condition, event, outcome) but the
same target population. NQF officials stated in some instances NQF needed to facilitate
harmonization discussions between measure developers to help ensure that the most
appropriate measure was being used, and that the measure portfolio did not include
duplicative measures, and to avoid confusion within the health care community.
50
  Two of the five projects that have not met expected time frames are still in progress and
are expected to be completed by the end of the third contract year. The remaining three
have been completed.




Page 21                                             GAO-12-136 NQF Contract Performance
Figure 2: Estimated Time Frames for Eight Projects Related to NQF’s Contract Activity on the Promotion and Use of EHRs, as
of August 31, 2011




                                        a
                                         In January 2011, NQF completed a version of the tool that was released for beta testing (reflected by
                                        the circle). NQF is now updating the version, which it expects to release in January 2012.
                                        b
                                         NQF officials told us that the final report was completed in April 2010; however, additional work was
                                        required to complete the project through the beginning of July 2010.
                                        c
                                         NQF is expected to publish several versions of the QDM throughout the contract term to allow for
                                        ongoing review and updates. The circles indicate the versions that have been published or are
                                        expected to be published, as of August 2011.
                                        d
                                         NQF was unable to complete all 113 retooled measures by September 2010, as initially scheduled.
                                        NQF submitted versions of 44 measures in July 2010, as requested by HHS. The final versions of all
                                        113 measures were not completed until December 2010.




                                        Page 22                                                    GAO-12-136 NQF Contract Performance
e
 NQF officials told us the eMeasure Review Panel was not able to complete its review of the retooled
measures by its initial due date of January 13, 2011, because all 113 measures were not retooled
until December 2010. As a result, the review was not completed until June 2011. As of August 2011,
the review panel was expected to submit its final report in October 2011. NQF officials told us in
October 2011 this report was completed, as scheduled.


     According to HHS and NQF officials, several factors contributed to
     NQF exceeding expected time frames for the retooling project under
     the EHR contract activity. HHS officials stated that the first set of 44
     retooled measures submitted had errors that required correction. For
     example, HHS officials stated that they found errors in the electronic
     coding of these 44 retooled measures requiring NQF and its
     subcontractors who retooled the measures to make corrections. In
     addition, HHS and NQF officials stated that after starting the retooling
     project, they quickly learned that the estimated time frames for the
     retooling project, as well as other projects related to the EHR activity,
     were overly ambitious, given the scope and complexity of the work.
     For example, HHS officials noted that retooling of quality measures
     into electronic format had never been attempted before and the
     technical complexity and labor required to complete the project were
     greater than anticipated. NQF officials also told us that HHS’s
     requests to modify the scope of work for this project often required
     changing the time frame for completing the retooled measures. These
     factors resulted in an extension of the project that delayed the final
     delivery of the 113 retooled measures as well as contributed to the
     need for additional staff at NQF.

•    Other Health Care Quality Measurement Activity. NQF was
     expected to complete two projects under the other health care quality
     measurement activity related to efficiency and resource use—one
     white paper on resource use 51 and another on geographic-level
     efficiency 52 by July 2010. These white papers were intended to
     provide information for an endorsement project on resource-use
     measures that began in January 2011. However, as of August 2011,
     the resource-use paper was still under review by HHS, and NQF



51
  Resource-use measures compare resources used (i.e., cost of care) with either (1) a
specific population, (2) a specific service, or (3) a series of related services over time, to
help identify the resources necessary for delivering high-quality care and to promote
greater efficiency in delivery of health care services.
52
  Geographic-level efficiency measures would be used for quality measure reporting at
the geographic and population level.




Page 23                                                  GAO-12-136 NQF Contract Performance
      officials stated they expected to receive comments in September
      2011. 53 The geographic-level efficiency paper was canceled in June
      2010 at the request of HHS. NQF initially intended to subcontract the
      work on these two projects, but officials told us that they were unable
      to identify a subcontractor at the level of funding approved for this
      project. As a result, HHS approved NQF’s proposal to complete this
      work internally. HHS officials stated that the drafts NQF submitted on
      both topics were poor in quality and did not meet its needs, resulting
      in HHS requesting additional revisions for the resource-use white
      paper that delayed its completion, and requesting the cancellation of
      the geographic-level efficiency white paper.

•     Administrative Activity. NQF did not meet the expected time frames
      for completing one of the required projects under the administrative
      activity—finalizing its annual work plan. Specifically, the NQF contract
      requires NQF to develop an annual work plan and to receive final
      approval from HHS within the first 4 weeks of each contract year;
      however, NQF did not meet this requirement in 2010 or 2011. For
      example, the final 2011 MIPPA annual work plan was not developed
      by NQF and approved by HHS until April 1, 2011. According to NQF
      and HHS officials, the 2011 MIPPA work plan was not developed and
      approved on time due to extended discussions on the scope and cost
      estimates of NQF’s EHR activities. HHS officials told us that the
      primary reason for the extended discussions was that they expected
      the costs to reflect all the work needed to complete the Measure
      Authoring Tool (MAT) by the end of the second contract year.
      However, they said that NQF only submitted a beta version of the tool
      by the end of the second contract year, which was not the version
      expected by HHS. NQF officials told us that the version was never
      intended to be final but rather a beta version, consistent with their
      understanding of HHS’s expectations. As a result, HHS and NQF
      officials needed to evaluate the scope of work and cost estimates for
      this and other projects. Further, NQF officials told us the delay in
      completing the 2011 MIPPA annual work plan resulted in the
      interruption of NQF’s ongoing work related to the MAT under the EHR
      contract activity. The delay also delayed its receipt of funding for
      some new or ongoing work under the contract. In some instances,
      NQF chose to start new or continue ongoing work with its own
      funding. For example, NQF officials stated that NQF began work



53
    HHS officials told us that they accepted the final version of this paper in October 2011.




Page 24                                                GAO-12-136 NQF Contract Performance
     related to the MAP using its own funds until HHS authorized the work.
     In addition, the delay in completing the 2011 MIPPA work plan
     resulted in the need to set the start date for fall 2011 rather than
     earlier in the contract year for some of the projects under the
     maintenance activity.

NQF also exceeded its cost estimates for projects under three of the
contract activities. HHS officials told us they approved the changes to the
cost estimates and in some cases modified NQF’s scope of work to help
ensure that NQF’s costs did not exceed the amount HHS had obligated
for the contract activities. NQF officials stated that in certain cases, not
meeting expected time frames contributed to NQF exceeding these cost
estimates. For example, the delays in projects related to the EHR contract
activity, including expanding the scope of the retooling project,
contributed to NQF exceeding its cost estimate of about $3.8 million for
the entire EHR contract activity by about $560,000 in the second contract
year. 54 In another example, the delays in finalizing the 2010 and 2011
MIPPA work plans contributed in part to NQF exceeding its cost estimate
for developing and finalizing these plans, which is a project under the
administrative contract activity. Specifically, while NQF estimated that
completion of the annual work plan would cost approximately $77,000,
NQF reported an actual cost of $176,590. In addition, NQF also exceeded
its cost estimate for the endorsement contract activity during the second
contract year for various reasons, including a need for additional technical
experts for review panels. Specifically, NQF exceeded estimated costs of
about $3.1 million for the entire endorsement activity by about $146,000
in the second contract year. 55 While HHS officials told us they approved
all changes to the cost estimates, in certain cases they reduced the scope
of NQF’s work in 2011 to ensure that total available funding for the
contract year was not exceeded and that sufficient funding was available
for ongoing projects. For example, HHS officials told us that they had
hoped to start several new endorsement projects beginning in 2011;


54
  NQF officials told us that it was originally required to retool 44 measures under the
contract; however, HHS expanded the scope of work in 2010 to 113 measures. This
required additional labor and other resources that increased the costs associated with the
retooling project.
55
  Our previous work shows that without the ability to generate reliable cost estimates,
programs are at risk of experiencing cost overruns, missed deadlines, and performance
shortfalls. GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.:
March 2009).




Page 25                                             GAO-12-136 NQF Contract Performance
                            however, these projects were not included in the 2011 final annual MIPPA
                            work plan so that funding would be available for NQF to complete its
                            ongoing projects, including work that was delayed under the EHR
                            contract activity. In addition, HHS requested that NQF discontinue its
                            work on one project related to the development of a public website for
                            2011, which is associated with the administrative contract activity.


HHS Did Not Use All         HHS officials told us that to help monitor NQF’s performance on the
Monitoring Tools Required   projects under the contract activities, they rely on NQF to report any
under the NQF Contract      issues, including those related to time frames or cost estimates, in the
                            monthly progress reports that NQF is required to submit to HHS or during
                            phone calls held at least monthly. While HHS monitored NQF’s progress
                            and approved changes to the time frames and cost estimates for the
                            projects under the contract activities, HHS did not use available tools for
                            monitoring that are required under NQF’s contract. These tools could
                            have helped to provide an opportunity for HHS to make any appropriate
                            changes to NQF’s projects. For example, HHS did not conduct an annual
                            performance evaluation required by the contract that would assess
                            timeliness and cost control issues, among other things, for the previous
                            contract year. 56 The results of such an evaluation could help HHS officials
                            to consider potential timeliness and cost issues when determining NQF’s
                            scope of work for the next year. Further, while monthly progress reports
                            and invoices include information on NQF’s costs, these documents do not
                            compare reported costs to initial cost estimates. HHS officials told us that,
                            prior to August 2011, they had not enforced a contractual requirement for
                            NQF to submit—nor had it received from NQF—a financial graph in its
                            monthly progress reports that provides information comparing NQF’s
                            monthly incurred costs for each of the contract activities with initial cost
                            estimates. Instead, HHS officials informally requested that NQF provide
                            them with the financial status of the contract activities in midyear 2010,
                            which helped them to plan for NQF’s work under the contract for 2011.
                            Having a financial graph in the monthly progress report could have helped
                            HHS officials to identify instances where any contract activity was
                            approaching or exceeding NQF’s initial cost estimates prior to HHS’s
                            midyear review. This, in turn, could have provided HHS and NQF an
                            opportunity to adjust estimates of future costs for these or related


                            56
                              The requirement in the contract to conduct annual performance evaluations is consistent
                            with a FAR provision that provides that agencies may specify that interim performance
                            evaluations be performed for multiyear contracts. See 48 C.F.R. § 42.1502(a) (2010).




                            Page 26                                            GAO-12-136 NQF Contract Performance
                        activities earlier in the contract year. HHS officials had asked NQF to
                        begin to include such a financial graph in its monthly progress reports
                        beginning in August 2011.


                        From January 14, 2010, through August 31, 2011, NQF reported a total of
NQF Reported Costs      approximately $22.4 million in costs and fixed fees on monthly invoices
and Fixed Fees of       submitted to HHS for projects under activities conducted in response to
                        MIPPA and PPACA. Specifically, NQF reported about $12.8 million in
over $20 Million from   total costs and fixed fees for the contract activities it performed during the
January 2010 through    second contract year—January 14, 2010, through January 13, 2011. 57
August 2011             From January 14, 2011, through August 31, 2011, part of the third
                        contract year, NQF reported an additional $9.6 million in total costs and
                        fixed fees. 58

                        During the second contract year, the majority of NQF’s reported costs
                        were related to the promotion of the development and use of EHRs
                        (36 percent, or $4.6 million) and endorsement of health care quality
                        measures (26 percent, or $3.3 million). Figure 3 illustrates the costs and
                        fixed fees NQF reported for eight of the nine contract activities we
                        reviewed that occurred during the second contract year. The ninth
                        contract activity relates to multistakeholder input on the selection of
                        quality and efficiency measures, as directed by PPACA. This contract
                        activity did not begin until after January 14, 2011, which is the start of the
                        third contract year.




                        57
                          According to HHS, total funding obligated was approximately $13.6 million for the
                        second contract year, which covered January 14, 2010, through January 13, 2011.
                        58
                          According to HHS, total funding obligated was approximately $25 million for the third
                        contract year, which covers January 14, 2011, through January 13, 2012.




                        Page 27                                             GAO-12-136 NQF Contract Performance
Figure 3: NQF’s Reported Contract Activity Costs and Fixed Fees Totaling
Approximately $12.8 Million during the Second Contract Year (Jan. 14, 2010—
Jan. 13, 2011)




Note: The figure describes eight of the nine contract activities we reviewed.


For the part of the third contract year covered in our review—January 14,
2011, through August 31, 2011—almost one-half of NQF’s reported costs
were for the activity to promote the development and use of EHRs and for
the activity to provide multistakeholder input on the selection of quality
and efficiency measures. Each of these activities accounted for
22 percent, or about $2.1 million of NQF’s reported costs. Other costs
reported by NQF include those for the activity related to providing
multistakeholder input into HHS’s annual National Quality Strategy
($1.55 million, or 16 percent) and those for the activity related to the
maintenance of endorsed quality measures activity (13 percent, or



Page 28                                                    GAO-12-136 NQF Contract Performance
$1.29 million). Figure 4 illustrates the costs and fixed fees NQF reported
for the part of the third contract year covered in our review for each of the
nine contract activities we reviewed.

Figure 4: NQF’s Reported Contract Activity Costs and Fixed Fees Totaling
Approximately $9.6 Million during the Period of the Third Contract Year Covered in
Our Review (Jan. 14, 2011—Aug. 31, 2011)




Page 29                                        GAO-12-136 NQF Contract Performance
                      According to HHS, as of August 2011, about $55.2 million remains
                      available for the NQF contract. 59 About $15.1 million in MIPPA funding
                      remains available for work to be conducted through January 13, 2013. In
                      addition, HHS plans to obligate approximately $40.1 million of its PPACA
                      funding through 2014 for NQF’s activities related to health care quality
                      measurement in response to PPACA. 60


                      For its various programs or initiatives, HHS has used or planned to use
HHS Used or Planned   about one-half of the quality measures that NQF has endorsed,
to Use Some NQF       maintained, or retooled under the contract, as of August 31, 2011, and
                      HHS officials expect to evaluate if and how the remaining measures will
Measures Received     be used. However, HHS has not comprehensively determined how it will
under the Contract    use NQF’s work under the contract to implement PPACA requirements
but Has Not           related to quality measures.

Comprehensively
Planned for Other
Measurement Needs
to Implement PPACA




                      59
                        As of August 2011, NQF had reported costs and fixed fees totaling $28.9 million since
                      the contract was awarded in January 2009. Specifically in our July 2010 report, we
                      highlighted that NQF reported costs and fixed fees totaling approximately $6.5 million for
                      the first contract year—January 14, 2009, to January 13, 2010. In addition, NQF had
                      reported about $22.4 million in costs and fixed fees from January 14, 2010, through
                      August 31, 2011—our reporting period that covers the second and part of the third
                      contract years. See GAO-10-737.
                      60
                        From January 14, 2011, through August 31, 2011, NQF reported about $4 million in
                      costs and fixed fees for PPACA-required activities. HHS officials told us that they plan to
                      obligate $10 million of the PPACA funding for health care quality measurement each fiscal
                      year from 2011 through 2014. Therefore, at least $40 million in PPACA funding is
                      expected to be available to NQF under the contract. HHS officials said that because
                      PPACA does not specify the proportion of the $100 million for NQF’s and HHS’s health
                      care quality measurement work for NQF, they determined that $10 million per year was a
                      reasonable amount.




                      Page 30                                             GAO-12-136 NQF Contract Performance
HHS Has Used or Planned    According to HHS officials, HHS has used or planned to use 61 about one-
to Use 164 of the 344      half (164) of the 344 health care quality measures it has received from
Measures That NQF Has      NQF through various endorsement, maintenance, and retooling projects
                           under the contract, as of August 31, 2011. 62 For example, of the 164
Endorsed, Maintained, or
                           measures used or planned for use, 44 were used in CMS’s Medicare and
Retooled under the         Medicaid EHR Incentive Program after being retooled—that is, converted
Contract                   to an electronic format that is compatible with EHRs—under the NQF
                           contract. 63 Although these 44 retooled measures were used in the EHR
                           Incentive Program, HHS officials stated that NQF and HHS detected
                           coding and other errors in the versions of the 44 retooled measures that
                           were published in the program’s final rule in July 2010 that required NQF
                           to make corrections to them after publication of the final rule. NQF did not
                           submit the revised versions of the 44 retooled measures published in the
                           final rule to HHS until December 2010. HHS officials stated that because
                           the final rule had already been published prior to receiving the final



                           61
                             For the purpose of this report, we defined a measure as used if the measure was
                           identified by HHS and included in an HHS program or initiative as of August 31, 2011, and
                           indicated as such by a final rule or other HHS documentation. We defined a measure as
                           planned for use if the measure was identified by HHS for potential use in an HHS program
                           or initiative as of August 31, 2011, and indicated as such by a proposed rule or other HHS
                           documentation.
                           62
                             Measures that are endorsed, maintained, and retooled under the NQF contract are
                           counted separately, even if they are the same measure. Therefore, the 344 measures
                           HHS has received from NQF do not represent 344 distinct measures. For example, a
                           measure that had been maintained under the contract and later retooled would be counted
                           as two separate measures. Therefore, the measure would appear twice in our count of the
                           total number of measures received from NQF—once as a maintained measure and then
                           separately as a measure that was retooled. Similarly, the measure would also be counted
                           as two separate measures for purposes of determining the number used or planned for
                           use. If both versions of the measure were used or planned for use in HHS programs or
                           initiatives, then it would also appear twice in our count of measures used or planned for
                           use.
                           63
                             See 75 Fed. Reg. 44,314 (July 28, 2010). The Medicare and Medicaid EHR Incentive
                           programs will provide incentive payments to eligible Medicare and Medicaid professionals
                           and hospitals that adopt and use health information technology, and reduce payments for
                           those Medicare professionals or hospitals that do not. The Medicare and Medicaid EHR
                           programs are divided into three stages. Stage I, which is underway, focuses on
                           establishing the functionalities in certified EHR technology that will allow for continuous
                           quality improvement and ease of information exchange. Stages II and III will potentially
                           become more stringent as the capabilities of health information technology infrastructure
                           increase. In 2011, part of the requirement for Stage I of the Medicare EHR Program is that
                           eligible professionals attest that they have met the program’s reporting requirements,
                           which includes that they report on a certain number of measures that were retooled by
                           NQF.




                           Page 31                                             GAO-12-136 NQF Contract Performance
formatted measures, CMS listed general guidance on its website to
address the errors. HHS officials told us that these 44 measures are
being used but have not yet been tested to assess the feasibility of
implementing them in the electronic format. 64 Until the testing is complete,
HHS runs the risk that some of these measures may not work as intended
when implemented in electronic format for performance measurement. As
a result, the agency does not have reasonable assurance that the
retooled versions of the measures will correctly capture information from
EHRs. In addition to the 44 retooled measures used in the EHR Incentive
Program, HHS also has used or planned to use 120 measures that it
received from endorsement and maintenance projects under the NQF
contract for various HHS programs and initiatives. (See table 2 for details
on specific programs in which HHS has used or planned to use health
care quality measures received from NQF under the contract.)




64
  HHS officials said that the 44 measures are not included as part of the scope of work
under the two contracts under which feasibility testing of the retooled measures will be
performed.




Page 32                                             GAO-12-136 NQF Contract Performance
Table 2: Status of Measures Used or Planned for Use under the NQF Contract, as of August 31, 2011

                                  Total measures        Measures used      Description of HHS programs and initiatives
                               received from NQF         or planned for    for which NQF measures are being used or
Projects                       under the contract           use by HHS     planned for use
Promote the development and use of electronic health records
Measure retooling                             113                    44    Forty-four measures have been used in Stage I of
                                                                           the Electronic Health Records (EHR) Incentive
                                                                                     a
                                                                           Program.
Total retooled measures                       113                    44
Endorse health care quality measures
Patient outcomes                                  38                  6   Six measures are planned for use in various
                                                                          programs including the Hospital Inpatient Quality
                                                                                             b
                                                                          Reporting Program.
Patient safety                                    1                   0   No measures used or planned for use.
Nursing homes                                     21                 17    Seventeen measures are planned for use in
                                                                           various programs including Nursing Home
                                                                                     c
                                                                           Compare.
Child health quality                              41                  8    Eight measures have been used in CMS Medicaid
                                                                           Programs as part of CMS’s Children’s Health
                                                                           Insurance Program Reauthorization Act of 2009
                                                                           (CHIPRA) Initial Core Set of Children’s Health Care
                                                                                             d
                                                                           Quality Measures.
Total endorsed measures                       101                    31
                                                            e
Maintain previously endorsed health care quality measures
                                  f
3-Year maintenance review cycle
Diabetes                                          13                 10    Twenty-three measures have been used or are
                                                                           planned for use in various programs and initiatives
                                                                                                                             g
                                                                           including the Physician Quality Reporting System
                                                                           and the Hospital Outpatient Quality Reporting
                                                                                     h
                                                                           Program.
Mental health                                     11                  3
Musculoskeletal conditions                        17                 10
Other maintenance review processes
Ad hoc review                                     3                   2   Two measures have been used or are planned for
                                                                          use in various programs, including the EHR
                                                                          Incentive Program and the Hospital Inpatient
                                                                          Quality Reporting Program.
Time-limited review                               80                 63    Sixty-three measures have been used or are
                                                                           planned for use in various programs and initiatives,
                                                                           including the EHR Incentive Program and the
                                                                           Physician Quality Reporting System.
Total maintained measures                     124                    88




                                        Page 33                                          GAO-12-136 NQF Contract Performance
                                  Total measures                  Measures used                Description of HHS programs and initiatives
                               received from NQF                   or planned for              for which NQF measures are being used or
Projects                       under the contract                     use by HHS               planned for use
Other health care quality measurement activity
Imaging efficiency measures                          6                                 1       One measure is planned for use in the Hospital
                                                                                               Outpatient Quality Reporting Program.
Total measures from other                            6                                 1
quality measurement activity
Total measures                                   344                                164
                                        Source: GAO analysis of final rules, proposed rules, and other HHS documentation.

                                        Note: Measures that are endorsed, maintained, and retooled under the NQF contract are counted
                                        separately, even if they are the same measure. Therefore, the 344 measures HHS has received from
                                        NQF do not represent 344 distinct measures. For example, a measure that had been maintained
                                        under the contract and later retooled would be counted as two separate measures. Therefore, the
                                        measure would appear twice in our count of the total number of measures received from NQF—once
                                        as a maintained measure and then separately as a measure that was retooled. Similarly, the measure
                                        would also be counted as two separate measures for purposes of determining the number used or
                                        planned for use. If both versions of the measure were used or planned for use in HHS programs or
                                        initiatives, then it would also appear twice in our count of measures used or planned for use.
                                        a
                                         The EHR Incentive Program provides incentive payments to eligible Medicare and Medicaid
                                        providers that adopt and use health information technology (IT) and reduces payments for those
                                        Medicare professionals or hospitals that do not.
                                        b
                                         The Hospital Inpatient Quality Reporting program provides hospitals a financial incentive to report
                                        the quality of their services as well as provides CMS data to help consumers make more informed
                                        decisions about their health care.
                                        c
                                         Nursing Home Compare is a tool of the Nursing Home Quality Initiative that allows consumers,
                                        providers, states, and researchers to compare information on health quality measures among various
                                        nursing homes.
                                        d
                                         CMS’s CHIPRA Initial Core Set of Children’s Health Care Quality Measures identifies an initial core
                                        set of child health quality measures for voluntary use by state Medicaid and Children’s Health
                                        Insurance Program (CHIP) programs to evaluate the quality of care and health outcomes in children
                                        enrolled in Medicaid and CHIP programs.
                                        e
                                          NQF can also maintain measures through its (1) annual updates, (2) ad hoc requests, and (3) time-
                                        limited endorsement review processes. NQF officials told us that as of August 2011, 83 measures
                                        have been maintained using these processes, including 80 under time-limited review and 3 under ad
                                        hoc request.
                                        f
                                        NQF officials told us the review of the diabetes, mental health, and musculoskeletal measures were
                                        maintained under its previous 3-year maintenance review process because the project began in
                                        September 2009, which is prior to the updated process that took effect in August 2010.
                                        g
                                         The Physician Quality Reporting System provides an incentive payment to eligible professionals who
                                        select among 240 measures to report.
                                        h
                                         CMS’s Hospital Outpatient Quality Reporting is a pay-for-performance program for outpatient
                                        hospital services.


                                        HHS officials told us that they expect to evaluate if and how they could
                                        use all of the remaining 180 of the 344 quality measures that were
                                        endorsed, maintained, or retooled under the NQF contract that are not
                                        currently in use or planned for use in HHS programs or initiatives.
                                        According to HHS officials, any measure developer can submit a measure
                                        to be considered for NQF endorsement. Therefore, all the measures



                                        Page 34                                                                    GAO-12-136 NQF Contract Performance
received under the contract may not be applicable to a particular HHS
health care quality program or initiative. HHS officials told us that they will
review the remaining 180 measures to determine if they are applicable to
their health care quality programs or initiatives. The officials expect that
many of these measures will be used in HHS programs or initiatives
required by PPACA. For example, HHS officials told us that they will
consider implementation of most of the retooled measures in future
stages of the EHR Incentive Program. In addition, PPACA directed HHS
to establish a hospital value-based purchasing program, as well as to
make plans or begin pilot programs for value-based purchasing in other
settings of care. The hospital value-based purchasing program will use
various quality measures and depend on the information collected on
them to determine payments to providers. PPACA also required the
development of no less than 10 provider-level outcome measures for
hospitals and physicians by March 2012. Further, PPACA directed HHS
to identify quality measures that could be used to evaluate hospice
programs and publish these measures by October 1, 2012. HHS officials
told us that they are in the process of determining whether or to what
extent the remaining 180 measures HHS has received under the NQF
contract can be used to address the new measurement needs and
priorities established by PPACA. HHS officials told us that they prefer to
use NQF-endorsed measures to meet HHS’s measurement needs
because these quality measures are nationally recognized standards and
in some cases HHS is required to use them. 65




65
  See, e.g., Pub. L. No. 111-148, § 3004(a), 124 Stat. 119, 368 (2010) (codified as
amended at 42 U.S.C. § 1395ww(m)(5)) (directing HHS to publish measures for long-term
care hospitals and requiring the department to consider measures endorsed by NQF). For
specific areas for which NQF has not endorsed feasible and practical measures, HHS may
specify measures that have not been endorsed by NQF as long as due consideration is
given to measures that have been endorsed or adopted by a consensus organization.




Page 35                                          GAO-12-136 NQF Contract Performance
HHS Does Not Have a        Although HHS has taken steps to determine how it can use the measures
Comprehensive Plan for     received under the contract with NQF, the agency does not have a
Determining How It Will    comprehensive plan for determining how it will use the remainder of the
                           work conducted under NQF’s contract to implement PPACA
Use NQF’s Work under the   requirements, including plans for additional quality measures that need to
Contract to Implement      be endorsed during the remaining contract years. HHS officials told us
PPACA Requirements         that HHS determines on an annual basis which activities—including work
Related to Quality         on quality measures—NQF is to perform under the contract through the
Measurement                interagency workgroup. The workgroup is comprised of representatives
                           from various HHS agencies and allows them to provide input on their
                           needs, including quality measures that need endorsement from NQF, for
                           their respective programs. However, HHS officials told us that each HHS
                           program assesses its quality measurement needs separately and
                           provides varying levels of detail about its needs. Therefore, it is unclear
                           the extent to which all programs consistently incorporate PPACA’s quality
                           measurement requirements and deadlines into these assessments. The
                           NPP’s September 2011 report noted the importance of greater alignment
                           of national quality measurement efforts, including the establishment of a
                           comprehensive measurement strategy that identifies core measure sets,
                           among other things. In addition, the report noted that all federal programs
                           should work toward the establishment of a common platform for
                           measurement and reporting. Without a comprehensive plan that
                           delineates HHS’s quality measurement needs, and given that each
                           program assesses its quality measurement needs separately, the
                           interagency workgroup may not be able to systematically ensure that all
                           of HHS’s quality measurement needs that implement PPACA
                           requirements align with the selection and prioritization of activities for
                           NQF to complete under the contract.

                           While HHS has begun various efforts to assess its quality measurement
                           needs, the lack of a plan that comprehensively determines the impact of
                           PPACA on its needs could affect the agency’s progress on its quality
                           measurement efforts as well as how it selects and prioritizes NQF’s
                           contract activities. Officials told us that prior to PPACA’s enactment, CMS
                           maintained a 5-year plan that listed its measurement needs based on
                           agency priorities and the priorities established by the NPP for some of its




                           Page 36                                    GAO-12-136 NQF Contract Performance
              programs. 66 However, officials told us that they have not updated this list
              to reflect the requirements related to quality measurement and time
              frames established by PPACA. In March 2011, HHS published the
              National Quality Strategy as required by PPACA, which included six
              priority areas of focus. The report was required by PPACA to include
              agency-specific plans, goals, benchmarks, and standardized quality
              metrics for each priority area, but did not do so. HHS officials stated that
              this document describes HHS’s initial plan for these elements and that
              they may be included in future versions of the strategy. In June 2011,
              HHS officials told us that they plan to convene a Quality Measurement
              Task Force within CMS with a goal to comprehensively align, coordinate,
              and approve the development, maintenance, and implementation of
              health care quality measures for use in various CMS programs. As of
              August 2011, the task force was in an early stage of development, and
              therefore it is too early to determine whether it will accomplish its goal.
              Although these various HHS efforts are key steps toward helping the
              agency meet its quality measurement needs, they are not guided by a
              comprehensive plan that synthesizes key priority areas identified in
              various sources, such as those reported by the NPP or in the National
              Quality Strategy, for which measures may be needed. Without such a
              plan, HHS may be limited in its efforts to prioritize which specific
              measures it needs to develop and have endorsed by NQF for its health
              care quality programs and initiatives established by PPACA. As a result,
              HHS may be unable to ensure that the agency receives the quality
              measures needed to meet PPACA requirements and specified time
              frames related to quality measurement.


              Health care quality measures are increasingly important to HHS as it uses
Conclusions   and will continue to use them in its existing and forthcoming programs
              and initiatives to evaluate health care delivery. For example, HHS’s value-
              based purchasing programs are pay-for-performance programs that will


              66
                The NPP identified six priorities in a November 2008 report: (1) patient and family
              engagement, (2) population health, (3) patient safety, (4) care coordination, (5) palliative
              and end-of-life care, and (6) elimination of overuse. In its October 2010 report to HHS, the
              NPP identified two additional priority areas: equitable access to ensure that all patients
              have access to affordable, timely, and high-quality care; and infrastructure supports, such
              as health information technology (IT), to address underlying system changes necessary to
              attain the goals of the other priority areas. See National Priorities Partnership, Input To
              The Secretary Of Health And Human Services On Priorities For The 2011 National Quality
              Strategy (Washington, D.C.: 2010).




              Page 37                                             GAO-12-136 NQF Contract Performance
require providers to collect and report information on health care quality
measures and adjust payment levels based on providers’ performance
against the measures. PPACA has increased HHS’s quality measurement
needs, and the time frames specified in the law have also increased the
urgency of obtaining endorsed quality measures—which are nationally
recognized standards and in some cases are required by statute—to
meet these needs. Given that NQF is the entity in the United States with
lead responsibility for endorsing health care quality measures, NQF’s
endorsement activities under the contract are of key importance to help
meet HHS’s quality measurement needs.

However, NQF’s endorsement process takes time. For more than half of
the projects, including all five projects in the endorsement activity, NQF
did not meet or did not expect to meet the initial time frames approved by
HHS. In addition, projects under three of the contract activities have
exceeded initial cost estimates, which resulted in HHS’s modification of
NQF’s scope of work in some instances to help ensure that NQF’s costs
did not exceed the funding allocated for the contract activities. While HHS
received information in monthly progress reports to help monitor NQF’s
performance under the contract, the agency did not use all of the
monitoring tools required under the contract to help address issues
related to time lines and cost estimates. These monitoring tools included
an annual performance evaluation that could help HHS officials consider
potential issues related to NQF’s time frames and cost estimates when
planning work for the next year and a financial graph to be included in
NQF’s monthly progress reports. The graph would have compared
reported costs to initial cost estimates, which is something that monthly
progress reports do not do. Although HHS officials reported that they
recently began in August 2011 to enforce the contractual requirement for
NQF to submit the graph, they have not implemented the required annual
performance evaluation. By not taking advantage of these tools, HHS
runs the risk of not having detailed and timely information that could help
identify instances in which NQF might be at risk of not meeting time
frames or exceeding estimated costs. Identifying such instances could
provide an opportunity for HHS to make any appropriate changes to
NQF’s scope of work, including setting priorities to ensure that HHS
receives the quality measures it needs in a timely manner.

With the time remaining under the contract, HHS has an opportunity to
ensure that the work performed under NQF’s contract better meets the
agency’s needs for its programs and initiatives. However, HHS has not
developed a plan that comprehensively identifies its quality measurement
needs for its programs and initiatives in light of PPACA’s requirements or


Page 38                                    GAO-12-136 NQF Contract Performance
                      determines how it will use the work conducted during the remaining years
                      of the NQF contract to help it meet these needs. In addition, critical tasks
                      may need to be completed outside of the NQF contract. For example,
                      HHS requested that NQF retool 113 measures under the contract and
                      used 44 of the 113 measures that included errors in its EHR Incentive
                      Program. As of November 2011, feasibility testing related to
                      implementation of the retooled measures had not been completed, and
                      HHS expected to perform this work outside of the NQF contract. Until the
                      testing is completed, HHS runs the risk that some of the retooled
                      measures may not work as intended when implemented in electronic
                      format for performance measurement, which is a concern because use of
                      these measures is an important component of HHS’s long-term goal for
                      providers to use health information technology (IT) to exchange
                      information and improve the quality of care.

                      Without a comprehensive plan, HHS lacks assurance that its selection of
                      the work to be performed by NQF—and the approximately $55.2 million
                      that the agency expects to spend for remaining work under the NQF
                      contract—will be prioritized in the most effective way possible. Given that
                      PPACA includes time frames for the implementation of quality
                      measurement programs, NQF’s pace in completing some of the work
                      under the contract—particularly the endorsement activity—raises
                      concerns. If the endorsement projects continue to require extended
                      completion times, HHS runs the risk of not having all the endorsed
                      measures it needs for implementing its programs and initiatives. Should
                      this occur, HHS may need to select nonendorsed measures for its
                      programs and initiatives that have not undergone an objective and
                      transparent review by NQF.


                      To help ensure that HHS receives the quality measures it needs to
Recommendations for   effectively implement its quality measurement programs and initiatives
Executive Action      within required time frames, we recommend that the Secretary of HHS
                      take the following three actions:

                      •   use monitoring tools required under the NQF contract to obtain
                          detailed and timely information on NQF’s performance and use that
                          information to inform any appropriate changes to time frames,
                          projects, and cost estimates for the remaining contract years;

                      •   ensure that testing of the electronic versions of the measures retooled
                          by NQF that are being used or are planned for use in the Medicare
                          and Medicaid EHR Incentive programs is completed in a timely


                      Page 39                                    GAO-12-136 NQF Contract Performance
                         manner to help identify potential errors and address issues of
                         implementation; and

                     •   develop a comprehensive plan that identifies the quality measurement
                         needs of HHS programs and initiatives, including PPACA
                         requirements, and provides a strategy for using the work NQF
                         performs under the contract to help meet these needs.

                     We provided a draft of this report to HHS and NQF for review and
Agency and Other     comment. HHS neither agreed nor disagreed with our recommendations
External Comments    and provided general comments. NQF concurred with many of the
                     findings in the report and provided clarification and additional context on
and Our Evaluation   the findings and recommendations. HHS and NQF’s letters conveying
                     their comments are reproduced in appendixes V and VI, respectively. In
                     addition to the overall comments discussed below, we received technical
                     comments from HHS and NQF, which we incorporated into our report as
                     appropriate.

HHS Comments         HHS’s comments included separate general comments from CMS and
                     ASPE that provided context on aspects of our findings and
                     recommendations. CMS’s comments stated that the draft report suggests
                     that CMS must use all of the measures endorsed by NQF, and noted that
                     not all NQF-endorsed measures are suitable for HHS quality reporting
                     and public reporting programs. Although our draft report did not state that
                     CMS must use all of the measures endorsed by NQF, we modified it to
                     note specifically, among other things, that all measures received under
                     the contract may not be applicable to a particular HHS health care quality
                     program or initiative. CMS also stated that the report suggests that CMS
                     has not developed measurement plans for various provisions of PPACA
                     related to quality reporting, public reporting, and value-based purchasing
                     programs. CMS provided additional context for current planning efforts to
                     address these requirements, including its Quality Measurement Task
                     Force. The draft report acknowledged this and other CMS planning efforts
                     to address the health care quality requirements contained in PPACA and
                     noted that, as of August 2011, this initiative was just beginning. Further,
                     while various efforts are underway and CMS’s comments state that it has
                     documented how quality measures will be used to address all relevant
                     provisions of PPACA, CMS has not provided documentation of
                     comprehensive plans to address PPACA requirements that include
                     alignment across programs, detailed time frames to meet PPACA
                     deadlines, or how it will use the NQF contract to help ensure that it
                     receives the endorsed measures it needs to meet these requirements.



                     Page 40                                    GAO-12-136 NQF Contract Performance
ASPE’s comments noted, with respect to our first recommendation, that
HHS used all except two of the monitoring tools called for in the contract.
As noted in the draft report, HHS began receiving the monthly financial
graph—one of the two monitoring tools—from NQF in August 2011. Also,
ASPE noted its plans to update its performance evaluation system with
NQF performance information for the first 2 contract years—the period
January 14, 2009, through January 13, 2011—and to complete a final
performance evaluation at the end of the contract in January 2013, which
is the end of the fourth contract year. It did not indicate any plans to
conduct the annual performance evaluation for the third contract year—
January 14, 2011, through January 13, 2012—which would be consistent
with the contract’s requirements. With respect to our second
recommendation, ASPE provided technical comments and also told us
that CMS issued a contract solicitation to test the retooled measures, but
CMS did not receive any bids. Instead, ASPE noted in its comments that
two of CMS’s current contractors will conduct feasibility testing on 69 of
the 113 retooled measures that are planned for use in HHS’s EHR
Incentive programs. CMS does not plan to issue a solicitation for a new
contract to test the feasibility of the remaining 44 retooled measures,
which are currently being used in HHS’s EHR Incentive Program. We
noted these comments in the report. Regarding our third
recommendation, ASPE stated that the measures that are not currently in
“use” are being evaluated by HHS and that any conclusions that they will
not be used are not accurate. Our draft report provided information on
which measures were used or planned for use as of August 2011, and
indicated that the remaining measures may be used in the future.
Specifically, the report noted that HHS officials expect that many of these
measures will be used in HHS programs or initiatives, and that HHS
officials told us that they will review all the measures received under the
contract to determine if they are applicable to their health care quality
programs or initiatives. ASPE’s comments also noted that our draft report
did not include information on all NQF-endorsed measures used by the
various agencies within HHS. As noted in the draft report, we relied on
HHS to identify programs and initiatives across HHS that use or plan to
use these health care quality measures and recognize that those included
in our report may not represent a comprehensive list of all health care
quality programs and initiatives. As we recommended in our report,
having a comprehensive plan could help HHS identify programs or
initiatives that use or plan to use health care quality measures, including
those endorsed by NQF.




Page 41                                    GAO-12-136 NQF Contract Performance
NQF Comments   NQF’s comments state that it is providing its services to HHS under a
               cost reimbursement contract, which is used in circumstances where
               aspects of performance, such as time frames, cost estimates, and scope
               of work, cannot be reasonably estimated, and therefore, should not be
               expected. As noted in the draft report, the contract type used for this work
               is used for efforts such as research, design, or study efforts where costs
               and technical uncertainties exist and it is desirable to retain as much
               flexibility as possible in order to accommodate change. However, the draft
               report also noted that this type of contract provides only a minimum
               incentive to the contractor to control costs. Given the risk associated with
               this type of contract, the fact that NQF has not met expected time frames
               on about half of its projects as of August 2011, and that NQF exceeded
               its initial cost estimates for some of its projects under its contract
               activities, it is especially important that HHS obtain detailed and timely
               information on NQF’s performance and use that information to inform any
               appropriate changes to time frames, projects, and cost estimates for the
               remaining contract years, as noted in our recommendations. NQF’s
               comments also state that time frames and costs for the work performed
               under the contract were initial estimates based on an early understanding
               of the work, that HHS and NQF understood that there would likely be
               changes to them as a result of the complexity and novelty of the work,
               and that they have worked collaboratively throughout the contract period
               to address these and other factors. As noted in the draft report, the final
               work plans, the technical proposal, and other documents that we
               reviewed included initial time frames for all projects and costs for the work
               performed during the contract year that were approved by HHS in
               collaboration with NQF. The draft report also notes several examples of
               reasons why the time frames and costs were modified over time.
               Contributing factors include the high volume of measures submitted,
               changes to the scope of work, and the novelty and complexity of the
               work.


               We are sending copies of this report to the Secretary of Health and
               Human Services and other interested parties. In addition, the report will
               be available at no charge on GAO’s website at http://www.gao.gov.




               Page 42                                     GAO-12-136 NQF Contract Performance
If you or your staff have any questions about this report, please contact
me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Office
of Congressional Relations and Office of Public Affairs can be found on
the last page of this report. Other major contributors to this report are
listed in appendix VII.




Linda T. Kohn
Director, Health Care




Page 43                                   GAO-12-136 NQF Contract Performance
List of Committees

The Honorable Max Baucus
Chairman
The Honorable Orrin G. Hatch
Ranking Member
Committee on Finance
United States Senate

The Honorable Tom Harkin
Chairman
The Honorable Michael B. Enzi
Ranking Member
Committee on Health, Education, Labor, and Pensions
United States Senate

The Honorable Fred Upton
Chairman
The Honorable Henry A. Waxman
Ranking Member
Committee on Energy and Commerce
House of Representatives

The Honorable David Camp
Chairman
The Honorable Sander M. Levin
Ranking Member
Committee on Ways and Means
House of Representatives




Page 44                                GAO-12-136 NQF Contract Performance
Appendix I: Health Care Quality
              Appendix I: Health Care Quality Measurement
              Framework



Measurement Framework

              Figure 5 illustrates a health care quality measurement framework of the
              various stages that a quality measure will go through, as described by the
              Department of Health and Human Services (HHS) and National Quality
              Forum (NQF) officials and others. These stages include measure
              development, endorsement, selection, and use, among others. This
              framework also shows examples of which entities, including HHS and
              NQF, are involved in each of the stages.




              Page 45                                       GAO-12-136 NQF Contract Performance
Appendix I: Health Care Quality Measurement
Framework




Figure 5: Health Care Quality Measurement Framework




a
 NQF officials told us that the development of data platforms allows measure developers to create
quality measures directly in an electronic format as well as retool—that is, convert measures from a
paper-based format to an electronic format. For example, NQF’s work to create a measure authoring
tool provides measure developers with a tool to create standardized electronic measures that help
capture information in electronic health records. NQF officials told us development of data platforms,
such as through the use of the measure authoring tool, is increasingly important as NQF will begin to
require that measure developers include electronic specifications in quality measures beginning in
May 2012.




Page 46                                                    GAO-12-136 NQF Contract Performance
                        Appendix I: Health Care Quality Measurement
                        Framework




                        As an example of actions taken during the second stage of this quality
HHS’s Role in           measurement framework, HHS officials described two different processes
Planning and            used for planning and identifying gap areas.
Identifying Gap Areas   The Centers for Medicare & Medicaid Services (CMS) Office of Clinical
                        Standards and Quality 1 has developed a standardized approach to
                        identify quality measures that it uses in its health quality initiatives and
                        programs using CMS’s Measures Management System. 2 The Measures
                        Management System requires the convening of a technical expert panel
                        in the initial planning stage. 3 Once convened, the technical expert panel is
                        expected to work with measure developers who will gather information
                        that will help the panel determine whether measures need to be
                        developed for a program or initiative. During this stage, measure
                        developers may conduct environmental scans or literature reviews, to
                        determine the existence of measures that could be used for a program or
                        initiative. If a measure does not exist, then the developer will work with
                        CMS to develop the needed measures for the program or initiative,
                        including measure testing. Upon development of the measures, the
                        technical expert panel will evaluate them based on (1) importance to
                        making significant gains in health care quality and improving health
                        outcomes, (2) scientific acceptability of the measure properties including
                        tests of reliability and validity, (3) usability, and (4) feasibility. Measures
                        recommended by the panel are generally submitted for NQF
                        endorsement.

                        In contrast, CMS’s Center for Medicaid, Children’s Health Insurance
                        Program (CHIP), and Survey & Certification Office—the CMS center
                        which implements CHIP—uses a measure identification process that
                        relies on existing measures rather than development of new measures,
                        according to officials. This office worked with a technical advisory group,


                        1
                            CMS is an agency within HHS.
                        2
                         The Measures Management System is used by divisions within CMS and provides
                        guidance to CMS-contracted measure developers on the development and maintenance
                        of measures. It was developed to help CMS manage an ever-increasing demand for
                        quality measures to use in its various public reporting and quality programs as well as in
                        value-based purchasing initiatives.
                        3
                         The technical expert panel consists of a group of recognized experts in relevant fields
                        including clinicians, statisticians, quality improvement experts, methodologists, and
                        pertinent measure developers who are recruited to provide input on the measure under
                        development by the contract.




                        Page 47                                             GAO-12-136 NQF Contract Performance
Appendix I: Health Care Quality Measurement
Framework




the Subcommittee on Children’s Healthcare Quality Measures for
Medicaid and CHIP, to recommend an initial core set of measures for the
CHIP. With assistance from CMS, the subcommittee evaluated measures
based on importance, validity, and feasibility. CMS officials told us that
they considered existing NQF-endorsed and non-NQF-endorsed
measures based on the measurement needs of the program, and relied
on measure testing conducted by the measure developers. Officials
stated that they have also relied on the subcommittee to evaluate
candidate measures for Medicaid child health programs. Officials said
that they are not required to submit measures that will be used for
Medicaid programs for NQF endorsement.




Page 48                                       GAO-12-136 NQF Contract Performance
Appendix II: How GAO Categorized NQF’s 16
              Appendix II: How GAO Categorized NQF’s 16
              Tasks under the Contract into Nine Contract
              Activities


Tasks under the Contract into Nine Contract
Activities
              From January 14, 2010, through August 31, 2011, the National Quality
              Forum’s (NQF) contract with the Department of Health and Human
              Services (HHS) included 16 tasks that NQF is required to perform. For
              purposes of our work, we categorized these tasks into nine contract
              activities. Specifically, in certain cases, we grouped activities that covered
              related areas of work into a single contract activity. For example, we
              consolidated the six administrative activities NQF is required to perform
              into a single contract activity. (See table 3 that shows how we
              consolidated these contract activities.)

              NQF was required to perform specific projects under the nine contract
              activities we identified. For example, under the endorsement contract
              activity, NQF was required to complete an endorsement project related to
              patient outcome measures. For purposes of our work, we identified and
              reviewed 63 projects NQF is required to perform under the nine contract
              activities, as shown in appendix III.




              Page 49                                       GAO-12-136 NQF Contract Performance
                                              Appendix II: How GAO Categorized NQF’s 16
                                              Tasks under the Contract into Nine Contract
                                              Activities




Table 3: GAO’s Categorization of the NQF Contract’s 16 Tasks into Nine Activities
                           a
NQF contract’s 16 tasks                                                     Nine contract activities categorized by GAO
                                                                                                                b
1.   Opening meeting between HHS and NQF                                    1.    Administrative activity
2.   Development of the annual work plan
3.   Quality assurance (development of an internal evaluation
     plan and monthly call to discuss the plan)
4.   Weekly conference calls
5.   Monthly progress reports
                                                     c
6.   Develop public website for project documents
7.   Formulation of national strategy and priorities for health care 2.           Recommendations on a National Strategy and Priorities for
     quality measures, as specified by the Medicare                               Quality Measurement (MIPPA)
     Improvements for Patients and Providers Act of 2008
     (MIPPA)
8.   Implementation of a consensus process for endorsement of               3.    Endorsement of Health Care Quality Measures (MIPPA)
     health care quality measures (MIPPA)
9.   Implementation of a consensus process for maintenance of               4.    Maintenance of Endorsed Quality Measures (MIPPA)
     previously endorsed health care quality measures (MIPPA)
10. Promotion of electronic health records (MIPPA)                          5.    Promotion of the Development and Use of Electronic Health
                                                                                  Records (MIPPA)
11. Submission of annual report to Congress and the Secretary 6.                  Annual Report to Congress and the Secretary of HHS (MIPPA
    of Health and Human Services (MIPPA)                                          and PPACA)
12. Additional Requirements for Annual Report—Measure Gaps
    and Inadequacies and Input of the Multistakeholder Groups,
    as specified by the Patient Protection and Affordable Care
    Act (PPACA)
13. Focused Measure Development, Harmonization, and                         7.    Other Health Care Quality Measurement Activity Not Identified
    Endorsement Efforts to Fill Critical Gaps in Performance                      in MIPPA or PPACA
    Measurement (not specified by MIPPA or PPACA)
14. Provision of Input on Priorities for the National Strategy for          8.    Multistakeholder Input into HHS’s National Strategy for Quality
    Quality Improvement in Healthcare (PPACA)                                     Improvement in Health Care (as required by PPACA)
15. Input to the Annual National Quality Strategy (PPACA)
16. Selection of Quality Measures for Use in Payment Programs 9.                  Multistakeholder Input on the Selection of Quality Measures for
    and Value-Based Purchasing Programs under PPACA,                              use in Payment Programs and Value-Based Purchasing
    Other Private/Public Payers, and Other Programs (PPACA)                       Programs Under PPACA, Other Private/Public Payers, and
                                                                                  Other Programs (as required by PPACA)
                                              Source: GAO analysis of MIPPA, PPACA, the NQF contract, the 2010 and 2011 final MIPPA work plans, and the 2011 PPACA
                                              technical proposal.
                                              a
                                              The numbering of the 16 tasks, or contract activities, does not reflect the numbering used under the
                                              NQF contract with HHS.
                                              b
                                               A number of activities NQF performs under the HHS contract are administrative and are not directly
                                              related to NQF’s work on health care quality measurement. Rather, these activities focus on project
                                              planning and contract management efforts.
                                              c
                                               Development of a public website was part of the 2010 final annual work plan but this work was
                                              canceled in 2011.




                                              Page 50                                                                GAO-12-136 NQF Contract Performance
Appendix III: Status of NQF’s Nine Contract
              Appendix III: Status of NQF’s Nine Contract
              Activities and the Projects under Each Activity
              GAO Identified


Activities and the Projects under Each
Activity GAO Identified
              The tables below provide a status update on the projects that the National
              Quality Forum (NQF) is required to complete under the nine contract
              activities we identified (see app. II). The contract activities and the
              projects under the activities NQF is expected to perform are determined
              on an annual basis by the Department of Health and Human Services
              (HHS) and NQF. As a result, the number of projects under the contract
              activities varies by contract year. For our reporting period—January 2010
              through August 2011—we determined that NQF was required to conduct
              work on 63 projects under the contract activities we reviewed. To
              determine initial time frames for each project, we calculated the
              approximate time between expected start and end dates established in
              NQF’s 2009, 2010, and 2011 final annual Medicare Improvements for
              Patients and Providers Act of 2008 (MIPPA) work plans, the 2011 Patient
              Protection and Affordable Care Act (PPACA) technical proposal, and
              other NQF documents. Actual time frames were determined by
              calculating the approximate time between the actual start date and the
              actual date of completion. For projects that were not yet complete as of
              August 2011, we included an expected time frame based on the
              approximate difference between the actual start date and the expected
              date of completion. NQF and HHS officials stated that any changes to the
              initial time frames were approved by HHS.




              Page 51                                           GAO-12-136 NQF Contract Performance
                                             Appendix III: Status of NQF’s Nine Contract
                                             Activities and the Projects under Each Activity
                                             GAO Identified




Table 4: Status of Projects under NQF’s Contract Activity to Provide Recommendations on a National Strategy and Priorities
for Quality Measurement, as of August 31, 2011

Projects under contract activity to                                                                                                     Expected         Actual time
provide recommendations on a                                                                                                          time frame               frame
national strategy and priorities for                                                                                                  (estimated          (estimated
quality measurement (MIPPA)          Description of project                                                            Status            months)            months)
Formulate national strategies and      Report that prioritizes 20 high-impact Medicare                                                           14                   15
priorities                             conditions and associated measure gaps.
Evaluation of the uses of NQF-         Report that investigates the use of NQF-endorsed                                                            9           exp. 18
endorsed measures                      measures.
Measurement development and            Report that sets a national measure development and                                                       10                   10
endorsement agenda                     endorsement agenda.
                                                                                                                                                                            a
Measurement framework for multiple Report that presents an analysis of measures being                                                            14           exp. 13
chronic conditions                 used to assess quality of care for people with multiple
                                   chronic conditions.
Advice to the Office of the National   Report that analyzes measures targeted for use in the                                                       2                       2
Coordinator on meaningful use of       Medicare Electronic Health Record Incentive Program,
measures                               specifically examining how health information technology
                                       (IT) tools can improve the efficiency, quality, and safety
                                       of health care delivery.
                                             Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress
                                             and Internal Evaluation Plan (IEP) reports from January 2010 to August 2011.

                                             Note: Status is defined by the following symbols.
                                              = Completed          = In progress        exp. = expected time frame
                                             a
                                              The initial time line for this work was June 2010 through August 2011, which is approximately
                                             14 months. However, due to delays in approving the subcontractor NQF engaged to perform the
                                             work, the project was not started on time. As a result, the modified time line for the project is April
                                             2011 through May 2012, which represents a delay of approximately 9 months.




                                             Page 52                                                                  GAO-12-136 NQF Contract Performance
                                         Appendix III: Status of NQF’s Nine Contract
                                         Activities and the Projects under Each Activity
                                         GAO Identified




Table 5: Status of Projects under NQF’s Contract Activity to Endorse Health Care Quality Measures, as of August 31, 2011

                                                                                                                                   Expected            Actual time
Projects under NQF’s contract                                                                                                    time frame                  frame
activity to endorse health care                                                                                                  (estimated             (estimated
                           a
quality measures (MIPPA)           Description of project                                                          Status           months)               months)
                                                                                                                                                                        b
Patient outcomes                   Endorsement of health care outcome measures related                                                       14             13-19
                                   to 20 high-impact Medicare conditions identified by the
                                   Centers for Medicare & Medicaid Services (CMS), and
                                   mental health, and child health conditions.
                                                                                                                                                                        c
Patient safety                     Endorsement of patient safety measures and other                                                          17       23, exp. 27
                                   projects related to patient safety, including a framework
                                   for public reporting of patient safety information,
                                   endorsement of serious reportable events (SRE), and
                                   collaboration with state-based reporting agencies.
Nursing homes                      Endorsement of performance measures for chronic and                                                       10                   17
                                   postacute care nursing facilities.
                                                                                                                                                   d
Evaluation of Consensus            A comprehensive analysis of NQF’s endorsement                                                              9                   18
Development Process                process by Mathematica Policy Research, Inc., to
                                   identify areas of improvement related to timeliness and
                                   effectiveness.
                                                                                                                                                                        e
Child health quality               Endorsement of measures that could be used in public                                                      11           exp. 16
                                   reporting for certain conditions or cross-cutting areas
                                   applicable to the Medicaid population.
                                         Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress
                                         and IEP reports from January 2010 to August 2011.

                                         Note: Status is defined by the following symbols.
                                          = Completed          = In progress        exp. = expected time frame
                                         a
                                          HHS officials told us that they consider an NQF measure endorsed when the board ratifies the
                                         measures. However, NQF is required to submit a final report to HHS following ratification to complete
                                         the project. For purposes of our report, we consider the project completed when the board endorses
                                         the measures.
                                         b
                                          The patient outcomes project was conducted in three phases. Phases I and II were initially expected
                                         to be completed in 13 months; however, the project was completed in 19 months. NQF officials told
                                         us a high volume of measures contributed to the delays. Phase II was initially expected to take
                                         15 months but it took 17 months.
                                         c
                                          As of August 2011, NQF endorsed one patient safety measure under phase II of the endorsement
                                         project and expects to complete this phase in September 2011 following the review of an additional
                                         patient safety measure. Phase I is expected to be completed by December 2011.
                                         d
                                          In the 2009 final work plan, this project was expected to be completed between October 2009 and
                                         July 2010, or approximately 9 months. NQF officials stated the delays in completing this work were a
                                         result of additional time needed for data collection efforts, among other things.
                                         e
                                          In August 2011, NQF announced endorsement of 41 out of 44 child health quality measures under
                                         the child health quality endorsement project. The remaining 3 measures are still under consideration
                                         and are scheduled to be reviewed for endorsement in September 2011.




                                         Page 53                                                                  GAO-12-136 NQF Contract Performance
                                           Appendix III: Status of NQF’s Nine Contract
                                           Activities and the Projects under Each Activity
                                           GAO Identified




Table 6: Status of Projects under NQF’s Contract Activity to Maintain Previously Endorsed Health Care Quality Measures, as
of August 31, 2011

                                                                                                                            Expected                  Actual time
Projects under NQF’s contract                                                                                               time frame                frame
activity to maintain health care                                                                                            (estimated                (estimated
                           a
quality measures (MIPPA)           Description of project                                                            Status months)                   months)
                                                                                                                                            c
Updated 3-year endorsement       Six maintenance projects using NQF’s process that                                              Various              All ongoing
maintenance cycle review process reviews previously endorsed measures across 22 topic
projects (6 projects)            areas every 3 years using three measure maintenance
                                        b
                                 cycles. The projects relate to the following conditions:
                                 •    Cardiovascular
                                 •    Surgical
                                 •    Cancer
                                 •    Pulmonary/critical care
                                 •    Perinatal
                                 •    Renal care
3-Year maintenance cycle review    Three maintenance projects conducted under the old NQF                                       No specific          20
                                                                                                                                            d
prior to updated process           maintenance review process related to the following                                           time frame
(3 projects)                       conditions:
                                   •   Diabetes
                                   •   Mental health
                                   •   Musculoskeletal conditions
Time-limited endorsement review    Maintenance review of measures that have been granted a                                      No specific          No specific
                                                                                                                                            e
                                   time-limited endorsement—measures that require                                                time frame           time frame
                                   additional testing to be conducted by measure developers
                                   within 12 months of endorsement.
                                                                                                                                            f
Annual measure maintenance         Review of measures submitted by measure developers to                                        Various              Ongoing
update                             determine whether the measure has undergone any
                                   change.
                                                                                                                                                            g
Ad hoc review                      Review of measures that were initiated during the contract                                   n/a                  n/a
                                   year by request and based on justifiable evidence to
                                   substantiate the review.
                                           Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress
                                           and IEP reports from January 2010 to August 2011.

                                           Note: Status is defined by the following symbols.
                                            = Completed          = In progress        n/a = no time frame was established
                                           a
                                            In addition to the 3-Year Maintenance Review cycles, NQF can also maintain measures through
                                           (1) annual updates, (2) ad hoc requests, and (3) time-limited endorsement reviews. As of August
                                           2011, NQF has maintained 3 measures under ad hoc requests and 80 measures under time-limited
                                           endorsement reviews. NQF officials told us they also began annual updates for 59 measures during
                                           our reporting period.




                                           Page 54                                                                  GAO-12-136 NQF Contract Performance
Appendix III: Status of NQF’s Nine Contract
Activities and the Projects under Each Activity
GAO Identified




b
 NQF has developed three cycles of review—cycles A, B, and C. Cycle A focuses on measures
related to cardiovascular; surgery; prevention; endocrine; GU/GYN; mental health; and
musculoskeletal disorders. Cycle B focuses on measures related to cancer; pulmonary/critical care;
palliative and end-of-life care; Perinatal; renal; safety; disparities; and care coordination. Cycle C will
focus on health care infrastructure; head/eye/ear/nose/throat conditions; infectious disease;
neurology; patient experience and engagement; functional status; and gastrointestinal. NQF is
expected to complete Cycle A by the end of 2010, Cycle B by the end of 2011, and Cycle C by the
end of 2012. As of August 2011, the cardiovascular and surgery measures from Cycle A are still
under review.
c
 As of August 2011, NQF was conducting six maintenance projects under its updated 3-year review
cycle. The initial and estimated time frames for these projects vary by project. For example, review for
phases I and II of the cardiovascular measures is expected to take 15 months and review for phase I
of the surgical measures is expected to take 13 months and phase 2 is expected to take 16 months.
NQF also initiated the review of renal measures in May 2011 and was initially expected to take
7 months; however, the time line has been extended to 8 months. As of November 2011, NQF has
issued a call for measures for Perinatal measures and expects to issue a call for cancer and
pulmonary/critical care measures in November 2011 and January 2012, respectively.
d
 NQF officials stated that no estimated time frame for completion was established for these three
projects. Although officials stated they anticipated the length of the review to be similar to other NQF
projects, they encountered delays that extended the time frame, such as a high volume of measures
under review.
e
 NQF officials provided us with a time line for completing time-limited reviews that included an initial
start date of January 2011 but did not provide specific expected dates of completion.
f
    NQF officials told us that the annual reviews did not begin until 2011 and are conducted in cycles.
g
 NQF officials told us that there is no time line established for ad hoc reviews because requests for
ad hoc reviews will be considered by NQF on a case-by-case basis and can be made at any time.




Page 55                                                       GAO-12-136 NQF Contract Performance
                                           Appendix III: Status of NQF’s Nine Contract
                                           Activities and the Projects under Each Activity
                                           GAO Identified




Table 7: Status of Projects under NQF’s Contract Activity to Promote the Development and Use of Electronic Health Records,
as of August 31, 2011

Project under NQF’s
contract activity to                                                                                                                  Expected         Actual time
promote the development                                                                                                             time frame               frame
and use of electronic health                                                                                                        (estimated          (estimated
records (MIPPA)                      Description of project                                                          Status            months)            months)
Measure authoring tool               Development of a software tool that measure developers                                                    15            exp. 28
                                     will use to create standardized eMeasures.
Clinical decision support            Report that provides a framework for Clinical Decision                                                    14                   17
                                     Support rules and key elements for health IT systems to
                                     help ensure compatibility and functionality with the
                                     Quality Data Model.
eMeasure creation and testing        Establish a uniform standard for electronic measures.                                                     13                   13
                                                                                                                                                                          a
Quality data model                   Maintenance and revision of a health IT model that                                                        28           exp. 28
                                     allows data from electronic health records to be collected
                                     and used in quality measurement.
Health information technology (IT)   Expert panel convened to examine the information                                                          12                   14
utilization expert panel             needed to measure effective health IT use.
Initial measure retooling            “Retooling”—that is, converting 113 NQF-endorsed                                                          19                   22
                                     health care quality measures from paper-based format to
                                     an eMeasure format.
Update of retooled measures          Publish a list of updated versions of the initial 113                                                     10            exp. 10
                                     measures under the contract that describes any changes
                                     made to the measures, including those identified during
                                     public comment and by a review panel.
                                                                                                                                                                          b
eMeasure syntactical review panel    Expert panel that conducts a thorough review of the                                                         9     15, exp. 19
                                     retooled measures to help ensure that a measure’s
                                     intent remains intact for continued NQF endorsement.
                                           Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress
                                           and IEP reports from January 2010 to August 2011.

                                           Note: Status is defined by the following symbols.
                                            = Completed          = In progress        exp. = expected time frame
                                           a
                                            NQF officials told us they anticipate a number of versions will be published annually under the
                                           contract to allow for ongoing review and updates. NQF expects to publish an updated version for
                                           public use in January 2012.
                                           b
                                            In June 2011, the review panel completed its review of the 113 retooled measures and expects to
                                           publish a report that synthesizes comments on and updates to the measures by October 2011. NQF
                                           officials told us that a request by HHS to expand the scope of this work extended the time line by
                                           3 months.




                                           Page 56                                                                  GAO-12-136 NQF Contract Performance
                                         Appendix III: Status of NQF’s Nine Contract
                                         Activities and the Projects under Each Activity
                                         GAO Identified




Table 8: Status of Projects under NQF’s Contract Activity to Submit an Annual Report to Congress and the Secretary of HHS,
as of August 31, 2011

Project under NQF’s
contract activity to submit                                                                                                           Expected      Actual
an annual report to Congress                                                                                                        time frame time frame
and the Secretary of HHS                                                                                                            (estimated (estimated
(MIPPA and PPACA)                Description of project                                                               Status           months)    months)
Report due March 1, 2010         Summary of NQF’s progress on activities under the contract                                                     12                12
                                 during the first contract year (2009).
Report due March 1, 2011         Summary of NQF’s progress on activities under the contract                                                     13                12
                                 during the second contract year (2010).
Report due March 1, 2012         Summary of NQF’s progress on activities under the contract,                                                    13         exp. 12
                                 including a report on gaps in endorsed and nonendorsed
                                 health care quality measures and a summary of activities
                                 conducted by multistakeholder groups during the third
                                 contract year (2011).
                                         Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress
                                         and IEP reports from January 2010 to August 2011.

                                         Note: Status is defined by the following symbols.
                                          = Completed          = In progress         exp. = expected time frame




                                         Page 57                                                                  GAO-12-136 NQF Contract Performance
                                               Appendix III: Status of NQF’s Nine Contract
                                               Activities and the Projects under Each Activity
                                               GAO Identified




Table 9: Status of Projects under NQF’s Contract Activity to Provide Multistakeholder Input into HHS’s National Strategy for
Quality Improvement in Healthcare, as of August 31, 2011

Project under NQF’s
contract activity to provide                                                                                                              Expected         Actual time
multistakeholder input to                                                                                                               time frame               frame
HHS’s National Quality                                                                                                                  (estimated          (estimated
Strategy (PPACA)                  Description of project                                                                 Status            months)            months)
Recommendations on HHS’s          Multistakeholder input on national priorities related to                                                           1                    1
National Strategy for Quality     improvement in population health and in delivery of health care
Improvement in Healthcare         services that HHS will consider when developing the National
(2010)                            Quality Strategy issued in 2011.
Provision of input on priorities Multistakeholder input on a plan for measuring and improving                                                        5             exp. 6
for HHS’s annual National        health and health care with a focus on the six priorities
Quality Strategy                 identified in HHS’s National Quality Strategy.
Provision of input on             Quarterly webinars, facilitated by the National Priorities                                                         7             exp. 5
                         a
Partnership for Patients          Partnership (NPP), for purposes of gathering input issues
                                  related to HHS’s Partnership for Patients initiative.
Care coordination                 A two-phase project that will include an endorsement project                                         Phase 1: 6              Phase 1:
                                  related to prioritized gap areas related to care coordination,                                                                 exp. 10
                                  such as transitions across settings and providers; and (2) two
                                  reports that will address issues related to care coordination                                        Phase 2: 13              Phase 2:
                                  measures, such as readiness of health information technology                                                                   exp. 15
                                  systems.
Palliative and end-of-life care   Endorsement project for individual and composite measures or                                                       9            exp.. 9
                                  cross-cutting measures related to palliative and end-of life care.
Patient safety                    Initiative that includes (a) an endorsement project for patient                                      (a) 9 or 10,            (a) exp.9
                                  safety complications measures; (b) work related to SRE                                                       to be
                                                                                                                                                   b
                                  codification; and (c) development of an integrated strategy for                                      determined
                                  high-profile areas in patient safety.
                                                                                                                                                 (b) 7         (b) exp. 8
                                                                                                                                                 (c) 6         (c) exp. 6
Population health                 Endorsement project for (a) preventive services and population                                                (a) 9         (a) exp. 9
                                  health measures and (b) a report that provides information
                                  related to key topic areas related to population health, including
                                  applicable NQF evaluation criteria.                                                                            (b) 5         (b) exp. 6

Patient reported outcomes         Two reports—one that addresses issues related to use of                                                            9             exp. 9
                                  patient reported outcomes measures, including methodological
                                  issues, and another that discusses information needed to
                                  prepare measures for the endorsement process.
                                               Source: GAO analysis of NQF’s contract with HHS, the 2011 PPACA technical proposal, and NQF’s monthly progress and IEP reports
                                               from January 2010 to August 2011.

                                               Note: Status is defined by the following symbols.
                                                = Completed          = In progress         exp. = expected time frame
                                               a
                                                In April 2011, HHS launched the Partnership for Patients, which is a national partnership with the
                                               goal to save 60,000 lives by improving patient safety through reduction in preventable injuries,
                                               complications, and hospital readmissions over the next 3 years.




                                               Page 58                                                                  GAO-12-136 NQF Contract Performance
                                          Appendix III: Status of NQF’s Nine Contract
                                          Activities and the Projects under Each Activity
                                          GAO Identified




                                          b
                                           NQF officials told us that they expect the time lines for the endorsement project to be extended
                                          because the project will be conducted in phases to accommodate additional measures. However, as
                                          of August 2011, NQF officials told us they are still finalizing the scope of work for this project,
                                          including determining if there will be any impact on the overall time line.



Table 10: Status of Projects under NQF’s Contract Activity to Provide Multistakeholder Input on Selection of Quality
Measures, as of August 31, 2011

Project under NQF’s
contract activity to provide                                                                                                         Expected         Actual time
multistakeholder input on                                                                                                          time frame               frame
selection of quality measures                                                                                                      (estimated          (estimated
(PPACA)                             Description of project                                                          Status            months)            months)
Multistakeholder prerulemaking      Multistakeholder input provided annually by the Measure                                                   11            exp. 11
input                               Applications Partnership (MAP) to HHS on the selection
                                    of measures to be implemented through the federal
                                                                             a
                                    rulemaking process for various programs.
Measures for use in improvement     Multistakeholder input provided to HHS by the MAP to                                                   (a) 7         (a) exp. 7
of quality reporting programs       help develop a coordination strategy for performance
                                    measurement and identify health care quality measures                                                 (b) 11        (b) exp. 11
                                    for use in four quality reporting programs: (a) physicians,
                                    (b) postacute care programs under Medicare, (c) PPS-                                                   (c) 15        (c) exp. 15
                                    exempt cancer hospitals, and (d) hospice care.
                                                                                                                                          (d) 15        (d) exp. 15
Measures for dual eligible          Multistakeholder input provided to HHS by the MAP on a                                                    15            exp. 15
beneficiaries                       measurement strategy and potential new health care
                                    quality measures that address quality issues identified
                                    for dual eligible beneficiaries, such as multiple chronic
                                    conditions.
Measurement strategy for            Multistakeholder input provided to HHS by the MAP on a                                                    18            exp. 17
readmissions and healthcare-        coordination strategy for readmission and HAC
acquired conditions (HAC)           measurement across public and private payers.
                                          Source: GAO analysis of NQF’s contract with HHS, the 2011 PPACA technical proposal, and NQF’s monthly progress and IEP reports
                                          from January 2010 to August 2011.

                                          Note: Status is defined by the following symbols.
                                           = In progress        exp. = expected time frame
                                          a
                                           The MAP will provide input on measures beginning February 1, 2012, based on a list of measures
                                          published on December 1 of each year by the Secretary of HHS. The 11 programs included in this
                                          prerulemaking input are: hospice, hospital inpatient, hospital outpatient, physicians’ offices, cancer
                                          hospitals, end-stage renal disease (ESRD) facilities, inpatient rehabilitation facilities, long-term care
                                          hospitals, CMS hospital value-based purchasing, psychiatric hospitals, and home health care.




                                          Page 59                                                                  GAO-12-136 NQF Contract Performance
                                           Appendix III: Status of NQF’s Nine Contract
                                           Activities and the Projects under Each Activity
                                           GAO Identified




Table 11: Status of Projects under NQF’s Other Health Care Quality Measurement Activity Not Identified in MIPPA or PPACA,
as of August 31, 2011

Project under NQF’s                                                                                                            Expected time Actual time
other health care                                                                                                                      frame      frame
quality measurement                                                                                                               (estimated (estimated
contract activity                Description of project                                                          Status             months)     months)
                                                                                                                                                                          a
Imaging efficiency measures      Endorsement project for imaging efficiency measures                                                              8           15-18
                                 because Medicare spends approximately $14 billion
                                 annually on outpatient imaging studies.
                                                                                                                                                    b
Resource-use measures            Endorsement project for measures to help assess the                                                       12-14         exp.11-14
                                 cost of health care services provided.
Resource-use white paper         White paper on resource-use measures that will inform a                                                          9          exp. 22
                                 2011 endorsement project on this topic.
Geographic-level efficiency      White paper on geographic-level efficiency measures that                            --                   canceled (June 2010)
measurement white paper          are used for quality measure reporting at the geographic
                                 and population level.
                                                                                                                                                                          c
Harmonization                    Report that provides operational guidance related to                                                             9                  9
                                 harmonization to be used in future NQF consensus
                                 development projects.
                    d
ICD-10 conversion                Report that examines implications of the transition from                                                       11                  15
                                 an HHS code set that is used to classify health care on
                                 NQF’s measure maintenance process and provides
                                 recommendations to assist measure developers and
                                 NQF during the transition.
                        e
Measure development              Maintenance of subcontracts with eight measure                                                                 n/a
                                 developers to allow for development of critical measures
                                 as needed.
Regionalized emergency medical Identification of existing quality measures and gap areas                                                        14           exp. 18
care services                  related to measurement of regionalized emergency care
                               services to inform creation of a framework to guide
                               measure development in this area.
                                           Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress
                                           and IEP reports from January 2010 to August 2011.

                                           Note: Status is defined by the following symbols.
                                            = Completed          = In progress         = Not started        - - = Canceled       exp. = expected time frame
                                           n/a = no time frame was established
                                           a
                                            The imaging efficiency measures endorsement project was initially expected to endorse measures
                                           by July 2010; however, due to measure harmonization efforts, NQF divided the endorsement review
                                           in two phases. The first six measures were endorsed in February 2011. In May 2011, NQF issued a
                                           second report announcing endorsement of another measure.
                                           b
                                            The resource use endorsement project will be conducted in two cycles—cycle 1 is expected to be
                                           completed in 11 months and cycle 2 is expected to be completed in 14 months.
                                           c
                                            The initial time line for completing this project was February 2010 through November 2010; however,
                                           the actual time line for completion was from March 2010 through December 2010, which delayed
                                           delivery of the report by 1 month.




                                           Page 60                                                                  GAO-12-136 NQF Contract Performance
                                           Appendix III: Status of NQF’s Nine Contract
                                           Activities and the Projects under Each Activity
                                           GAO Identified




                                           d
                                            HHS uses the International Classification of Diseases, Ninth Revision, Clinical Modification
                                           (ICD-9-CM) code sets to classify health care. In 2013, HHS will transition to the Tenth Revision, or
                                           ICD-10-CM/PCS. The ICD-10 update activity aims to help HHS prepare for this transition by
                                           examining the impact of the change on measure development and submitting measures for NQF
                                           endorsement.
                                           e
                                            HHS officials stated that HHS has never implemented this project and likely will not do so through
                                           the remainder of the contract period.



Table 12: Status of Projects under NQF’s Administrative Contract Activity

                                                                                                                             Expected time             Actual time
Project under NQF’s                                                                                                          frame                     frame
administrative                                                                                                               (estimated                (estimated
contract activity                Description of project                                                        Status        months)                   months)
Opening meeting with HHS         Meeting between NQF and HHS officials at the                                               This is a single
                                 beginning of the contract year to review activities to be                                   event held
                                 performed during for the year.                                                              annually.
2010 annual work plan            Development of a draft and final work plan that outlines                                   3                         4
                                 specific activities to be performed during the 2010
                                 contract year.
2011 annual work plan            Development of a draft and final work plan that outlines                                   5                         6
                                 specific activities to be performed during the 2011
                                 contract year related to NQF’s activities funded by
                                 MIPPA.
Quality assurance and monthly    Monthly report compiled by NQF that includes process                                       All completed in reporting
Internal Evaluation Plan (IEP)   and outcome measures to help monitor NQF’s                                                  period, ongoing annually
reports                          progress under the contract.
Weekly conference call with      Weekly conference call with HHS and NQF to go over                                         Not all held in reporting
                                                                                                                                     a
HHS                              projects under each activity.                                                               period, ongoing annually
Monthly progress report          Monthly report compiled by NQF that includes a                                             All completed in reporting
                                                                                                                                                      b
                                 summary for each activity and the status of project                                         period, ongoing annually
                                 planning, implementation, management, quality
                                 assurance, and evaluation that has occurred during the
                                 previous month’s contract period.
Public website                   Overhaul and maintenance of NQF’s website to allow                               --         canceled (February 2011)
                                 measure developers, members, and the public easier
                                 access to relevant documents.
                                           Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress
                                           and IEP reports from January 2010 to August 2011.

                                           Note: Status is defined by the following symbols.
                                            = Completed         - - = Canceled
                                           a
                                            NQF officials told us that this meeting did not occur each week, as scheduled, because in some
                                           instances those involved determined it was unnecessary.
                                           b
                                            From January 2010 to June 2010, NQF was required to submit its monthly progress reports by the
                                           1st of every month; however, it did not submit these reports until the 15th of the month. In June 2010,
                                           HHS modified the due date to the 15th, beginning in July 2010. Since that time, all reports have been
                                           submitted on time.




                                           Page 61                                                                  GAO-12-136 NQF Contract Performance
Appendix IV: Summary of Projects to Review
              Appendix IV: Summary of Projects to Review
              the National Quality Forum’s Endorsement
              Process


the National Quality Forum’s Endorsement
Process
              As part of a project under its contract with the Department of Health and
              Human Services (HHS), NQF was required to review its endorsement
              process. To complete this project, the National Quality Forum (NQF)
              subcontracted with Mathematica Policy Research, Inc. (Mathematica), to
              conduct a review of NQF’s endorsement process, as requested by HHS.
              HHS officials stated that, given the importance of the endorsement
              process as part of the health care quality measurement framework, they
              requested that an objective and thorough review of NQF’s endorsement
              process that focused on timeliness, efficiency, and effectiveness should
              be conducted. For example, they stated that they were interested in
              whether there were any efficiencies that could be implemented to shorten
              the process while maintaining an objective review of the health care
              quality measures that were evaluated under the process. Mathematica
              initiated its review of NQF’s endorsement process in October 2009 and
              completed the work in December 2010.

              In December 2010, Mathematica submitted a final report to NQF and
              recommended eight areas where improvements could be made and
              inefficiencies could be addressed in the endorsement process. In the final
              report, Mathematica noted that the current process is lengthy and the
              timeliness of the endorsement projects varies substantially. The report
              further noted that the length of the endorsement process affects the
              availability of endorsed measures for end users, such as HHS. To help
              reduce the time required to complete projects, Mathematica
              recommended that NQF create a schedule for its endorsement process
              for measure developers and develop feasible time lines that include clear
              goals for each endorsement project.

              As of May 2011, NQF officials stated that NQF has taken steps or plans
              to take steps in its future projects to address the eight areas for
              improvement Mathematica identified. For example, as of May 2011, NQF
              has solicited measures earlier based on a tentative annual project
              schedule to reduce the time lines of its endorsement process and
              reduced the period for voting by NQF member organizations from 30 to
              15 days. NQF officials stated that they believe their efforts to implement
              the recommendations will shorten the time lines for the endorsement
              projects by 3 to 4 months without compromising the integrity of the
              endorsement process and measures to be evaluated under the process.
              HHS officials stated Mathematica’s recommendations were valuable
              because much of the work under the NQF contract needs to be
              completed in an accelerated time line to help fill critical measurement
              gaps associated with HHS’s health care quality programs and initiatives.
              They noted that it is too soon to tell the effects of these changes on the


              Page 62                                      GAO-12-136 NQF Contract Performance
Appendix IV: Summary of Projects to Review
the National Quality Forum’s Endorsement
Process




endorsement process, but they plan to monitor implementation of the
changes in NQF’s 2011 endorsement projects under the contract. In
addition, as of September 2011, HHS approved a new project under the
contract to identify how the endorsement process can best align with
HHS’s time frame for needed measures. As part of this project, NQF is
expected to work with a consulting group to identify key performance
metrics and define milestones and time lines to help streamline its
endorsement process.




Page 63                                      GAO-12-136 NQF Contract Performance
Appendix V: Comments from the Department
             Appendix V: Comments from the Department
             of Health and Human Services



of Health and Human Services




             Page 64                                    GAO-12-136 NQF Contract Performance
Appendix V: Comments from the Department
of Health and Human Services




Page 65                                    GAO-12-136 NQF Contract Performance
Appendix V: Comments from the Department
of Health and Human Services




Page 66                                    GAO-12-136 NQF Contract Performance
Appendix V: Comments from the Department
of Health and Human Services




Page 67                                    GAO-12-136 NQF Contract Performance
Appendix V: Comments from the Department
of Health and Human Services




Page 68                                    GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National
             Appendix VI: Comments from the National
             Quality Forum



Quality Forum




             Page 69                                   GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National
Quality Forum




Page 70                                   GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National
Quality Forum




Page 71                                   GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National
Quality Forum




Page 72                                   GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National
Quality Forum




Page 73                                   GAO-12-136 NQF Contract Performance
Appendix VII: GAO Contact and Staff
                  Appendix VII: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Linda T. Kohn, (202) 512-7114 or kohnl@gao.gov
GAO Contact
                  In addition to the contact named above, Will Simerl, Assistant Director;
Staff             La Sherri Bush; Krister Friday; Amy Leone; Carla Lewis; John Lopez;
Acknowledgments   Elizabeth Martinez; Lisa Motley; Teresa Tucker; Carla Willis; and William
                  T. Woods made key contributions to this report.




(290911)
                  Page 74                                   GAO-12-136 NQF Contract Performance
                      The Government Accountability Office, the audit, evaluation, and
GAO’s Mission         investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.