oversight

Comparative Effectiveness: Agency for Healthcare Research and Quality's Process for Awarding Recovery Act Funds and Disseminating Results

Published by the Government Accountability Office on 2012-02-29.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Requesters




February 2012
                COMPARATIVE
                EFFECTIVENESS
                Agency for Healthcare
                Research and
                Quality’s Process for
                Awarding Recovery
                Act Funds and
                Disseminating Results




GAO-12-332
                                           February 2012

                                           COMPARATIVE EFFECTVENESS
                                           Agency for Healthcare Research and Quality’s
                                           Process for Awarding Recovery Act Funds and
                                           Disseminating Results
Highlights of GAO-12-332, a report to
congressional requesters




Why GAO Did This Study                     What GAO Found
The American Recovery and                  AHRQ used its standard, competitive review processes and criteria to select the
Reinvestment Act of 2009 (Recovery         recipients of CER grants and contracts using Recovery Act funds. Specifically, to
Act) provided $1.1 billion to the          select the recipients of Recovery Act CER grants, AHRQ used its standard
Department of Health and Human             review process that includes peer review of grant applications, the development
Services (HHS) for comparative             of funding recommendations by a team of senior officials within AHRQ, and final
effectiveness research (CER), which is     funding determination by the agency’s director. As part of this process, AHRQ
research that compares different           used its standard criteria to evaluate grant applications, as well as additional
interventions and strategies to prevent,   requirements that were specific to each funding opportunity. To select
diagnose, treat, and monitor health
                                           contractors who would receive Recovery Act funds, AHRQ used its standard
conditions. Of this amount, HHS’s
                                           contracting processes and criteria that are governed by the Federal Acquisition
Agency for Healthcare Research and
Quality (AHRQ) received $474 million
                                           Regulation, which establishes uniform policies for acquisition of supplies and
to support and disseminate the results     services by executive agencies, and the Public Health Service Act. These
of CER. GAO was asked to describe          processes included an evaluation of all contract proposals using standard criteria
issues including the (1) process and       adapted to the specific needs of each project. Between February 2009 and
criteria AHRQ used to award Recovery       September 2010, AHRQ awarded $311 million of its $474 million in Recovery Act
Act funds for CER, including steps to      CER funds through 110 grants. AHRQ also awarded $161 million of its Recovery
coordinate CER awards with other           Act CER funding through 34 contracts. The contracts and grants AHRQ awarded
HHS entities in order to avoid             supported both AHRQ’s agency-specific and HHS’s departmentwide CER priority
unnecessary duplication of effort; and     areas. In an effort to avoid unnecessary duplication of CER awards, AHRQ
(2) plans AHRQ has for disseminating       participated in HHS working groups, developed a CER spending plan, and
the results of CER it funded under the     queried HHS databases to check for duplicative awards.
Recovery Act.
To address these objectives, GAO
reviewed relevant documentation,
including AHRQ’s policies and
procedures for selecting the recipients
of grants; internal documents that
describe the award of Recovery Act
grants and contracts; and Recovery
Act contractors’ work plans. GAO also
analyzed AHRQ data on the number
and type of grants and contracts
awarded Recovery Act CER funds.
GAO interviewed AHRQ officials on the
selection of Recovery Act CER
grantees and contractors, including
coordination with other HHS agencies
that received Recovery Act CER funds,
and the plans the agency has to
disseminate the results of CER funded      According to AHRQ officials, the agency plans to disseminate the results of
by the Recovery Act. AHRQ provided         Recovery Act-funded CER using a range of existing mechanisms. These
technical comments, which GAO
                                           mechanisms include written products, training programs, social media tools, and
incorporated as appropriate.
                                           AHRQ’s website. AHRQ is also developing additional strategies to disseminate
                                           CER results. AHRQ awarded four contracts using Recovery Act funds totaling
                                           approximately $42.3 million to promote innovative approaches for disseminating
View GAO-12-332. For more information,     CER results. A variety of efforts are conducted under these contracts, including
contact Linda Kohn (202) 512-7114 or
kohnl@gao.gov.                             efforts to educate clinicians and develop regional dissemination offices.

                                                                                   United States Government Accountability Office
Contents


Letter                                                                                      1
               Scope and Methodology                                                        3
               Background                                                                   5
               AHRQ Used Standard Competitive Processes and Criteria and
                 Coordinated within HHS to Make Recovery Act Awards                       13
               AHRQ Plans to Use Its Existing Mechanisms and Develop
                 Additional Strategies to Disseminate CER Results                         21
               AHRQ Has Begun to Monitor PCORI and Identify Resources That
                 Could Enable It to Fulfill Its PPACA Responsibilities Related to
                 PCORI                                                                    25
               Agency Comments                                                            26

Appendix I     The Agency for Healthcare Research and Quality’s (AHRQ)
               Mission, Research, Priorities, and Budget                                  28



Appendix II    AHRQ CER Grants Awarded Using Recovery Act CER Funds,
               by Priority Area                                                           32



Appendix III   AHRQ CER Contracts Awarded Using Recovery Act CER Funds,
               by Priority Area                                                           34



Appendix IV    AHRQ Mechanisms That Support Dissemination of CER                          36



Appendix V     Example of Dissemination of Comparative Effectiveness Research
               by the Agency for Healthcare Research and Quality                          37



Appendix VI    GAO Contact and Staff Acknowledgments                                      38




               Page i                       GAO-12-332 AHRQ Comparative Effectiveness Research
Tables
          Table 1: AHRQ’s Funding for CER Activities, Fiscal Years 2007
                   through 2011                                                       6
          Table 2: HHS Departmentwide Priority Areas for CER Funded by
                   the Recovery Act                                                  6
          Table 3: AHRQ Priority Areas for CER Funded by the Recovery Act            7
          Table 4: AHRQ Budget, by Fiscal Year and Research Area                    31
          Table 5: Grant Applications Received and Reviewed, and Grants
                   Awarded, by AHRQ’s CER Priority Areas                            32
          Table 6: Grant Applications Received and Reviewed, and Grants
                   Awarded, by HHS Departmentwide CER Priority Areas                33
          Table 7: Contract Proposals Received and Reviewed, and Contracts
                   Awarded, by AHRQ’s CER Priority Areas                            34
          Table 8: Contract Proposals Received and Reviewed, and Contracts
                   Awarded, by HHS Departmentwide CER Priority Areas                35


Figures
          Figure 1: AHRQ’s Standard Competitive Process for Selecting
                   Grant Recipients                                                 10
          Figure 2: Distribution of AHRQ’s Recovery Act Funds for CER
                   Grants, by Priority Area                                         15
          Figure 3: Distribution of AHRQ’s Recovery Act Funds for CER
                   Contracts, by Priority Area                                      18




          Page ii                     GAO-12-332 AHRQ Comparative Effectiveness Research
Abbreviations

AHRQ                       Agency for Healthcare Research and Quality
ASPE                       Assistant Secretary for Planning and Evaluation
CER                        comparative effectiveness research
CER-CIT                    Comparative Effectiveness Research-Coordination
                             and Implementation Team
COTR                       contracting officer’s technical representative
FAR                        Federal Acquisition Regulation
FOA                        funding opportunity announcement
GSA                        General Services Administration
HHS                        Department of Health and Human Services
MMA                        Medicare Prescription Drug, Improvement, and
                             Modernization Act of 2003
NIH                        National Institutes of Health
OMB                        Office of Management and Budget
PHSA                       Public Health Service Act
PPACA                      Patient Protection and Affordable Care Act
PCORI                      Patient-Centered Outcomes Research Institute
PCORTF                     Patient-Centered Outcomes Research Trust Fund
Recovery Act               American Recovery and Reinvestment Act of 2009
VHA                        Veterans Health Administration



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page iii                            GAO-12-332 AHRQ Comparative Effectiveness Research
United States Government Accountability Office
Washington, DC 20548




                                   February 29, 2012

                                   The Honorable Fred Upton
                                   Chairman
                                   Committee on Energy and Commerce
                                   House of Representatives

                                   The Honorable Joe Barton
                                   House of Representatives

                                   Since the enactment of the Medicare Prescription Drug, Improvement,
                                   and Modernization Act of 2003 (MMA), the Agency for Healthcare
                                   Research and Quality (AHRQ), an agency within the Department of
                                   Health and Human Services (HHS), has been one of several federal
                                   agencies responsible for supporting and disseminating the results of
                                   comparative effectiveness research (CER). 1, 2 CER is research that
                                   compares different interventions and strategies to prevent, diagnose,
                                   treat, and monitor health conditions. AHRQ has supported CER activities
                                   and disseminated results by awarding grants and contracts to research
                                   centers and academic organizations to carry out this work. These
                                   grantees and contractors review and synthesize scientific evidence
                                   through research reviews; generate new scientific evidence and analytical
                                   tools in original research reports; and compile research findings that are
                                   translated into formats for a variety of audiences. The results of CER can
                                   be used by both patients and clinicians to make health care decisions
                                   about which treatment or intervention may be most effective or beneficial
                                   for a given patient. 3

                                   Under the American Recovery and Reinvestment Act of 2009 (Recovery
                                   Act), AHRQ received significant funding to support and disseminate the
                                   results of CER. Specifically, while AHRQ’s fiscal years 2009 and 2010


                                   1
                                    This mission was assigned to AHRQ by the MMA. Pub. L. No. 108-173, § 1013, 117
                                   Stat. 2066, 2438-41 (2003) (codified at 42 U.S.C. § 299b-7). Other federal agencies that
                                   conduct CER include the National Institutes of Health (NIH) and the Veterans Health
                                   Administration (VHA), a component of the Department of Veterans Affairs.
                                   2
                                    AHRQ also refers to CER as patient-centered outcomes research.
                                   3
                                    U.S. Department of Health and Human Services, Federal Coordinating Council for
                                   Comparative Effectiveness Research: Report to the President and the Congress
                                   (Washington, D.C.: June 30, 2009).




                                   Page 1                              GAO-12-332 AHRQ Comparative Effectiveness Research
budgets for CER activities were $50 million and $21 million, respectively,
the Recovery Act provided AHRQ with an additional $474 million for CER
for this period—$300 million that was appropriated to AHRQ and
$174 million that was appropriated to the HHS Office of the Secretary and
allocated to AHRQ. 4 The Recovery Act, which was enacted on
February 17, 2009, required that AHRQ obligate these funds by
September 30, 2010. 5 AHRQ’s $474 million in funding was part of
$1.1 billion HHS received under the Recovery Act to support CER
through its various agencies, including AHRQ and the National Institutes
of Health (NIH).

The Patient Protection and Affordable Care Act (PPACA) gave AHRQ
additional responsibilities related to CER—in particular, responsibilities
related to the Patient-Centered Outcomes Research Institute (PCORI).
PCORI is a nonprofit corporation established by PPACA to, among other
things, improve the quality and relevance of CER, and disseminate the
results of this research. 6 AHRQ is to help PCORI meet its mission in
several ways, which include broadly disseminating the results of the
research that PCORI conducts or funds; 7 developing a publicly available


4
 The act appropriated $400 million to the HHS Office of the Secretary to distribute to HHS
agencies to carry out CER; the Office of the Secretary allocated $174 million of these
funds to AHRQ. HHS agencies that received a portion of these funds awarded contracts
and grants through internal agency processes to carry out specific CER projects in four
priority areas. Other HHS agencies that received a portion of these funds include the
Centers for Medicare & Medicaid Services, the Food and Drug Administration, NIH, and
the Centers for Disease Control and Prevention.
5
 An obligation is a definite commitment that creates a legal liability of the government that
will give rise to a payment immediately or in the future. An agency incurs an obligation
when it awards a grant or contract.
6
  PPACA authorized the establishment of PCORI, a nonprofit corporation, to assist
patients, clinicians, purchasers, and policymakers in making informed health decisions by
advancing the quality and relevance of evidence concerning the manner in which
diseases, disorders, and other health conditions can effectively and appropriately be
prevented, diagnosed, treated, monitored, and managed through research and evidence
synthesis that considers variations in subpopulations and dissemination of research
findings in several areas. PCORI’s duties include identifying research priorities,
establishing a research project agenda, and carrying out that agenda using a variety of
methodological approaches. PPACA required the Comptroller General of the United
States to appoint 19 members to PCORI’s Board of Governors, which he did in September
2010. See Pub. L. No. 111-148, §§ 6301(a), 10602, 124 Stat. 119, 727-38, 1005 (2010)
(to be codified at 42 U.S.C. § 1320e).
7
 Pub. L. No. 111-148, § 6301(b), 124 Stat. 119, 738-40 (2010) (to be codified at 42 U.S.C.
§ 299b-37).




Page 2                               GAO-12-332 AHRQ Comparative Effectiveness Research
              database for this research; and promoting the incorporation of CER
              findings into health information technology systems that support clinical
              decision making. 8

              In June 2011, we provided information to congressional committees on
              HHS’s use of CER funds that were available under the Recovery Act and
              PPACA. 9 You have also asked that we report on AHRQ’s procedures for
              awarding Recovery Act CER funds and AHRQ’s plans to address the
              various new responsibilities established for it by PPACA as they relate to
              PCORI. In this report, we describe (1) the process and criteria AHRQ
              used to award Recovery Act funds for CER, including steps to coordinate
              these awards with other HHS entities in order to avoid unnecessary
              duplication of effort; (2) the plans AHRQ has for disseminating the results
              of CER it funded under the Recovery Act; and (3) the steps AHRQ has
              taken to perform its roles and responsibilities related to PCORI under
              PPACA.


              To describe the process and criteria AHRQ used to award its $474 million
Scope and     in Recovery Act CER funds, we reviewed relevant statutes as well as
Methodology   documentation on the process and criteria AHRQ uses to (1) determine
              the scientific and technical merit of grant applications and contract
              proposals, and (2) select grant recipients and contractors. We reviewed
              spending plans, which outline AHRQ’s and HHS’s plans for spending
              Recovery Act funds. We also reviewed summary statements that describe
              AHRQ’s process for selecting grantees and contractors to be awarded
              Recovery Act funds. In addition, we reviewed AHRQ’s Management
              Operations Manual, the agency’s written guidance that provides policies
              and procedures for selecting grant recipients. 10 We interviewed AHRQ
              and other HHS officials to learn about the processes and criteria they
              used to select the grantees and contractors that received awards funded



              8
               See Pub. L. No. 111-148, §§ 6301(a), (b), 10602, 124 Stat. 119, 727-47, 1005 (2010)
              (to be codified at 42 U.S.C. §§ 299b-37, 1320e).
              9
               GAO, HHS Research Awards: Use of Recovery Act and Patient Protection and
              Affordable Care Act Funds for Comparative Effectiveness Research, GAO-11-712R
              (Washington, D.C.: June 14, 2011).
              10
                The sections of AHRQ’s Management Operations Manual that provide policies and
              procedures for selecting the recipients of grants are primarily composed of HHS’s
              Awarding Agency Grants Administration Manual, which is HHS’s grant application manual.




              Page 3                             GAO-12-332 AHRQ Comparative Effectiveness Research
with the $474 million in Recovery Act CER funds and to coordinate these
awards with other HHS agencies that also received Recovery Act CER
funds. Because NIH also received Recovery Act CER funds, we
conducted interviews with NIH officials to confirm the methods and
processes used by AHRQ to coordinate funding opportunity
announcements (FOA), contract solicitations, and awards with NIH in an
effort to prevent the unnecessary duplication of effort in awarding
Recovery Act CER funds. 11 Finally, we obtained data from AHRQ on the
number and type of awards made between February 2009 and
September 2010 using the $474 million in Recovery Act CER funds. We
relied on Recovery Act award data provided by AHRQ and did not audit
the reported data. To determine whether AHRQ and the Office of the
Secretary’s Recovery Act CER award data were sufficiently reliable for
our analyses, we conducted a reliability assessment of the data we used
by reviewing existing information about the data, conducting quality
control checks, and interviewing agency officials knowledgeable about the
data. We determined that the data were sufficiently reliable for the
purposes of this report.

To describe AHRQ’s plans to disseminate results from CER funded with
Recovery Act funds, we reviewed the Recovery Act and other relevant
statutes to determine AHRQ’s responsibilities for disseminating CER. We
reviewed agency documents, including AHRQ contractors’ work plans
describing specific goals and activities; AHRQ’s general publications,
including reports and guides posted on AHRQ’s website and electronic
newsletters; and samples of AHRQ’s CER, including original research
reports, treatment guides, and slide presentations used for educating
clinicians. We also interviewed AHRQ officials and a contractor to
understand how the agency plans to disseminate the results of CER
funded with Recovery Act funds and to obtain information on plans AHRQ
has for assessing the effectiveness of its dissemination efforts.

To describe the steps AHRQ has taken to fulfill its roles and
responsibilities related to PCORI under PPACA, we reviewed provisions
in PPACA to identify these roles and responsibilities, which include to
broadly disseminate CER findings to various audiences; develop a


11
  FOAs and contract solicitations announce an agency’s intent to fund research or other
work through a grant or contract, respectively. FOAs and solicitations include a description
of the research or work to be performed and criteria against which applicants will be
considered.




Page 4                              GAO-12-332 AHRQ Comparative Effectiveness Research
                           publicly available database to collect evidence and research; and
                           promote the timely incorporation of CER findings into health information
                           technology systems that support clinical decision making. 12 While AHRQ
                           is required to conduct a number of activities under PPACA, we focused
                           our review on those activities that are related to PCORI. We also
                           reviewed AHRQ’s PPACA spending plan for fiscal years 2011 and 2012,
                           which describes the agency’s plans for using funds made available by
                           PPACA, as well as PCORI presentation materials and meeting reports.
                           We also conducted interviews with AHRQ officials to determine the steps
                           the agency has taken to meet its responsibilities related to PCORI under
                           PPACA.

                           We conducted this performance audit from February 2011 to February
                           2012 in accordance with generally accepted government auditing
                           standards. Those standards require that we plan and perform the audit to
                           obtain sufficient, appropriate evidence to provide a reasonable basis for
                           our findings and conclusions based on our audit objectives. We believe
                           that the evidence obtained provides a reasonable basis for our findings
                           and conclusions based on our audit objectives.


                           AHRQ supports CER by awarding grants and contracts to entities in order
Background                 to conduct CER and perform related activities, such as the dissemination
                           of CER results. 13 As 1 of 12 agencies within HHS, AHRQ’s overarching
                           mission is to improve the quality, safety, efficiency, and effectiveness of
                           health care for all Americans. (For more information on AHRQ’s mission,
                           research, priorities, and budget, see app. I.)


CER and the Recovery Act   The Recovery Act provided a significant amount of funding for AHRQ to
                           conduct CER activities. (See table 1.)




                           12
                             See Pub. L. No. 111-148, §§ 6301(a), (b), 10602, 124 Stat. 119, 727-47, 1005 (2010)
                           (to be codified at 42 U.S.C. §§ 299b-37, 1320e).
                           13
                             The term dissemination refers to developing and distributing messages that are derived
                           from CER for target audiences such as clinicians, consumers, or policymakers in order to
                           inform health care delivery or practice.




                           Page 5                             GAO-12-332 AHRQ Comparative Effectiveness Research
                                       Table 1: AHRQ’s Funding for CER Activities, Fiscal Years 2007 through 2011

                                                                        Funding from annual                        Funding from other
                                                                                              a
                                           Fiscal year            appropriations (in millions)              appropriations (in millions)
                                           2007                                               $15.0
                                           2008                                                 30.0
                                                                                                                                            b
                                           2009                                                 50.0                                $474.0
                                           2010                                                 21.0
                                                                                                                                             c
                                           2011                                                 21.0                                   $8.0
                                       Source: AHRQ.
                                       a
                                        Language in committee reports accompanying annual appropriations acts directs AHRQ to use
                                       specified amounts for CER activities from a lump sum appropriation.
                                       b
                                        The Recovery Act was enacted on February 17, 2009, and AHRQ received a total of $474 million for
                                       CER under the Recovery Act — $300 million that was appropriated to AHRQ and $174 million that
                                       was appropriated to the HHS Office of the Secretary and allocated to AHRQ. The Recovery Act
                                       required that these funds be obligated by September 30, 2010. An obligation is a definite commitment
                                       that creates a legal liability of the government that will give rise to a payment immediately or in the
                                       future.
                                       c
                                        These funds were made available to AHRQ in fiscal year 2011 from the Patient-Centered Outcomes
                                       Research Trust Fund.


                                       HHS developed departmentwide priorities for CER. (See table 2.) Of the
                                       total amount of $474 million in Recovery Act funds available to AHRQ for
                                       CER, $174 million of these funds were allocated to AHRQ by the Office of
                                       the Secretary and were used to support the HHS departmentwide
                                       priorities for CER.

Table 2: HHS Departmentwide Priority Areas for CER Funded by the Recovery Act

Priority area                          Description
Data Infrastructure                    Enhance existing data infrastructure and develop new databases, networks, and registries
                                       to support CER. Essential investments include longitudinal databases to link claims data
                                       for individual patients over a long period of time and patient registries that will
                                       prospectively collect clinical data on patients with specific diseases or on specific tests or
                                       procedures.
Dissemination, Translation, and        Ensure innovative strategies to invest in the dissemination and implementation of CER
Implementation                         with the ultimate goal being improved health outcomes. Funded projects include efforts to
                                       advance the dissemination of CER to patients and providers; and dissemination and
                                       implementation efforts at the delivery system and community levels.
Research                               Provide information on the relative strengths and weaknesses of various medical
                                       interventions.
Inventory and Evaluation               Catalogue CER activities and infrastructure in order to track investments in CER going
                                       forward.
                                       Source: AHRQ.




                                       Page 6                                   GAO-12-332 AHRQ Comparative Effectiveness Research
                                       AHRQ developed seven agency-specific CER priority areas to guide its
                                       spending of Recovery Act funds. (See table 3.) Of the total amount of
                                       $474 million in Recovery Act funds available to AHRQ for CER,
                                       $300 million of these funds were appropriated to the agency and,
                                       therefore, supported the agency’s seven CER priority areas.

Table 3: AHRQ Priority Areas for CER Funded by the Recovery Act

Priority area                         Description
Horizon Scanning                      Identify new and emerging issues for CER investments and establish an approach to
                                      investigate and prioritize areas for investigation relevant to the 14 priority conditions that
                                                                                  a
                                      guide the Effective Health Care Program.
Evidence Synthesis                    Increase the number of CER reviews conducted by AHRQ’s Evidence-based Practice
                                      Centers.
Evidence Gap Identification           Identify gaps in evidence research and prioritize needs for future reviews.
Translation and Dissemination         Expand the translation of findings on CER for different audiences, such as consumers,
                                      clinicians, and policymakers, and disseminate those findings.
Evidence Generation                   Measure the effectiveness of treatments for priority conditions with a concentration in
                                      under-represented populations, including children, the elderly, and racial and ethnic
                                      minorities.
Training and Career Development       Enhance the research and methodological capacity for conducting and improving CER and
                                      the development of data sources and research infrastructure.
Community Forum                       Formally engage stakeholders in CER efforts and develop a process for formal advice and
                                      guidance.
                                       Source: AHRQ.
                                       a
                                        These priority conditions are arthritis and nontraumatic joint disorders; cancer; cardiovascular
                                       disease, including stroke and hypertension; dementia, including Alzheimer’s disease; depression and
                                       other mental health disorders; developmental delays, attention-deficit hyperactivity disorder, and
                                       autism; diabetes mellitus; functional limitations and disability; infectious diseases, including HIV/AIDS;
                                       obesity; peptic ulcer disease and dyspepsia; pregnancy, including preterm birth; pulmonary
                                       disease/asthma; and substance abuse.


CER and the Patient                    The enactment of PPACA in 2010 gave AHRQ new roles and
Protection and Affordable              responsibilities related to disseminating CER and building capacity for
Care Act                               research, and appropriated funds for carrying out these activities. Several
                                       of these responsibilities relate to work conducted by PCORI. Established
                                       in November 2010, PCORI was authorized to help coordinate CER at a
                                       national level by developing national priorities for CER and conducting
                                       and funding CER activities. PPACA directs AHRQ to broadly disseminate
                                       CER findings to physicians, other health care providers, patients, payers,
                                       and policymakers; develop a publicly available database to collect
                                       evidence and research; promote the timely incorporation of CER findings




                                       Page 7                                    GAO-12-332 AHRQ Comparative Effectiveness Research
                             into health information technology systems that support clinical decision
                             making; and establish a process for receiving feedback about the value of
                             information disseminated by AHRQ. 14

                             To fund this work, PPACA established the Patient-Centered Outcomes
                             Research Trust Fund (PCORTF). The act specified that percentages of
                             this trust fund be provided to the Secretary of HHS and AHRQ in each of
                             fiscal years 2011 through 2019. 15 Specifically, AHRQ received $8 million
                             from this trust fund in fiscal year 2011 and will receive $24 million in fiscal
                             year 2012, representing 16 percent of the total amount appropriated to
                             this trust fund in each of these fiscal years. In subsequent fiscal years,
                             AHRQ will continue to receive 16 percent of the total amount appropriated
                             to the trust fund, which will be based on the net revenues from fees on
                             health insurance and self-insured plans, amounts transferred from the
                             Medicare trust funds, and appropriations to PCORTF from the General
                             Fund of the Treasury.


AHRQ’s Standard              A portion of AHRQ’s work, including its work related to CER, is conducted
Competitive Process for      through grants awarded to research centers and academic organizations
Selecting Grant Recipients   to fund research ideas developed by a grant applicant. Grant applications
                             are submitted in response to publicly available FOAs, which announce
                             AHRQ’s intention to award research grants. AHRQ has established a
                             standard competitive process that is governed by federal law to select
                             grant recipients. 16 According to AHRQ officials, this multistep process
                             includes: (1) an initial review of received applications; (2) preliminary
                             scoring of applications; (3) review and final scoring of applications at a
                             peer review panel meeting; (4) the development of preliminary funding
                             recommendations; (5) review by a senior leadership team within AHRQ;
                             and (6) a final determination of funding by the agency director.



                             14
                               See Pub. L. No. 111-148, §§ 6301(a), (b), 10602, 124 Stat. 119, 727-47, 1005 (2010)
                             (to be codified at 42 U.S.C. §§ 299b-37, 1320e).
                             15
                               The trustee of PCORTF is to provide for the transfer from PCORTF of 20 percent of the
                             amounts appropriated or credited to PCORTF for each of fiscal years 2011 through 2019
                             to the Secretary of HHS. Of the amounts transferred, the Secretary of HHS is to distribute
                             80 percent to AHRQ and 20 percent to the Secretary of HHS. See 26 U.S.C. § 9511.
                             16
                               AHRQ’s process for selecting grant recipients is governed by the Public Health Service
                             Act (PHSA) and implementing regulations, which require the use of peer review for grant
                             applications to ensure fair, competent, and objective assessment of their scientific and
                             technical merit. See 42 U.S.C.§ 299c-1; 42 C.F.R. pt. 67, subpt. A.




                             Page 8                              GAO-12-332 AHRQ Comparative Effectiveness Research
To evaluate the grant applications it receives in response to FOAs,
AHRQ’s peer reviewers, most of whom are authorities in their respective
fields and not government employees, use five standard core criteria to
score and rank the applications. These five standard core criteria are
(1) the significance in addressing an important problem; (2) the
investigators’ ability to carry out the research; (3) the originality or
innovation of the project; (4) the development of an adequate research
approach or framework; and (5) the scientific environment in which the
applicant plans to conduct the research. Each FOA also contains criteria
that are specific to the announcement. While these other specific criteria
are not individually scored, they are used to evaluate the applications
during the peer review panel meetings. (See fig. 1 for an overview of
AHRQ’s standard process for selecting grant recipients.)




Page 9                        GAO-12-332 AHRQ Comparative Effectiveness Research
Figure 1: AHRQ’s Standard Competitive Process for Selecting Grant Recipients




                                        a
                                        Each application is reviewed by three peer reviewers who assign preliminary scores.




                                        Page 10                                GAO-12-332 AHRQ Comparative Effectiveness Research
                            b
                             AHRQ establishes a minimum score, or triage line, that the grant applications must receive in order
                            to proceed to the next level of peer review. The triage line is established so that the number of
                            applications that fall above the triage line can be reasonably reviewed by the peer review panel at the
                            next stage; the poorest-scoring 50 percent to 60 percent of applications are eliminated; or the peer
                            review panel can review about three times the number of applications AHRQ anticipates funding
                            under a given FOA.
                            c
                             The summary statement includes peer reviewers’ written critiques, budget recommendations,
                            administrative notes, and the final overall impact score for the application.
                            d
                             “Rank order” refers to the relative position of an application among a listing of applications that have
                            undergone peer review. The listing of applications is ranked in the order of the overall impact score
                            calculated during the peer review process, from most to least meritorious.
                            e
                                The program official is the AHRQ official responsible for the program area of the FOA.
                            f
                             “Out-of-order funding” occurs when applications are funded out of rank order (i.e., not in accordance
                            with the rank order of most-meritorious to least-meritorious based on the overall impact scores
                            calculated during the peer review process). A program official may recommend and the agency may
                            ultimately decide to make out-of-order funding decisions for a number of reasons including the need
                            to address important agency research priorities, avoid duplication, or meet specific requirements in
                            the original FOA that the more meritorious applications cannot meet. The Director of AHRQ makes
                            the final determination to fund grant applications, including any out-of-order funding decisions. For
                            any out-of-order funding decision, a justification memo is prepared to provide written justification of
                            why out-of-order funding is being recommended.
                            g
                             The Director of AHRQ may not approve an application for funding unless the application has been
                            recommended for approval by a peer review group. 42 U.S.C. § 299c-1(b); 42 C.F.R. § 67.16(b) .


AHRQ’s Standard             AHRQ uses a separate competitive process to award contracts, which
Competitive Processes for   fund specific activities defined by AHRQ. This process for selecting
Awarding Contracts          contract proposals for award is governed by the Federal Acquisition
                            Regulation (FAR) 17 and the Public Health Service Act (PHSA) and
                            implementing regulations. 18 The advertising of available contracting
                            opportunities, which occurs through different types of solicitations, varies
                            depending on the type of contracting mechanism used. AHRQ generally
                            uses three types of contracting mechanisms.

                            •       Stand-alone contracts. This contracting vehicle involves the
                                    issuance of new, stand-alone contracts. Proposals are submitted in
                                    response to publicly-available solicitations referred to as requests for
                                    proposals. A request for proposals details the specific tasks or ideas
                                    that an agency needs a contractor to fulfill, such as delivery of a
                                    certain service or research of a clearly defined topic.



                            17
                              48 C.F.R. ch. 1. The FAR establishes uniform policies for acquisition of supplies and
                            services by executive agencies. Agency acquisition regulations may implement or
                            supplement the FAR.
                            18
                              The PHSA and implementing regulations require the use of peer review of contract
                            proposals. See 42 U.S.C. § 299c-1; 42 C.F.R. pt. 67, subpt. B.




                            Page 11                                    GAO-12-332 AHRQ Comparative Effectiveness Research
•    Task orders. This contracting vehicle involves issuing task orders
     under an existing, master contract, thereby giving a contractor a new
     task to perform. Proposals are submitted in response to solicitations
     called requests for task orders, which are issued to contractors
     already awarded contracts by the agency.

•    General Services Administration (GSA)-schedule task orders.
     This contracting vehicle involves the use of contracts that have been
     awarded by GSA for governmentwide use. GSA-schedule task orders
     are issued under existing master contracts awarded by GSA. These
     task orders are solicited through a request for quote solicitation that is
     competed among these contractors.

Upon receiving proposals in response to a solicitation, AHRQ typically
evaluates contract proposals for award using standard contracting
procedures and criteria, which are governed by the FAR and the PHSA
and implementing regulations. A technical review panel of agency officials
and external experts evaluates each proposal submitted in response to a
solicitation against standard criteria that are tailored to the specific needs
of each solicitation. These criteria include (1) demonstrated knowledge
and understanding of the contract requirements; (2) the proposed
approach to address tasks and subtasks listed in the proposal; (3) the
qualifications and experience of key management personnel, such as the
project director and project manager; (4) the potential contractor’s ability
to meet the project’s milestones; (5) the facility, equipment, and space
available to support the project goals and objectives; and (6) the past
performance of the potential contractor using information from references
or other government customers. Based on this evaluation of each
proposal’s scientific and technical merit and cost, the review panel, along
with the contracting officer’s technical representative (COTR), 19 identifies
the entity they believe should be awarded the contract and forwards the
recommendation to the contracting officer, the federal official who has




19
  A COTR is a government official who is designated by the agency’s contracting officer to
assist in the technical monitoring or administration of a contract.




Page 12                            GAO-12-332 AHRQ Comparative Effectiveness Research
                           authority to enter into a contract. 20 The contracting officer reviews the
                           recommendation and makes a final award decision. 21


                           AHRQ used its standard, competitive review process and criteria to select
AHRQ Used Standard         grant recipients and award 110 Recovery Act-funded CER grants, totaling
Competitive                approximately $311 million. In addition, AHRQ primarily used its standard
                           contracting processes and criteria to select contract proposals and enter
Processes and              into 34 contracts for CER using Recovery Act funding, totaling
Criteria and               approximately $161 million. AHRQ also took several steps to coordinate
Coordinated within         with other HHS agencies when soliciting and awarding CER grants and
                           contracts.
HHS to Make
Recovery Act Awards

AHRQ Used Its Standard     Between February 2009 and September 2010, AHRQ used its standard,
Competitive Review         competitive grant review process to select grant recipients and ultimately
Process and Criteria to    award 110 CER grants using approximately $311 million in Recovery Act
                           CER funds. Specifically, AHRQ used its standard process for selecting
Select Grant Recipients    grantees, which includes an initial review and preliminary scoring of
and Award 110 CER Grants   applications; review and final scoring of applications at peer review panel
                           meetings; development of preliminary funding recommendations; review
                           of funding recommendations by a senior leadership team within AHRQ;
                           and a final determination of funding by the agency director.

                           As part of the standard process AHRQ used to select the recipients of
                           Recovery Act-funded CER grants, peer reviewers used the agency’s
                           standard core criteria to score and subsequently rank the applications for
                           this funding. In addition to its core criteria, AHRQ also used other criteria
                           in its process of selecting recipients of Recovery Act CER grants. These
                           other criteria were specific to each FOA and often varied depending on
                           the CER study requested under that announcement. These other criteria
                           may be used to assess, for example, a grant application in terms of the


                           20
                             Contracting officers are responsible for ensuring performance of all necessary actions
                           for effective contracting. Contracting officers enter into, administer, and terminate
                           contracts.
                           21
                             The contracting officer may not award a contract unless the proposal has been
                           recommended for approval by the review panel. See 42 U.S.C. § 299c-1(b); 42 C.F.R.
                           § 67.103.




                           Page 13                             GAO-12-332 AHRQ Comparative Effectiveness Research
adequacy of the protection afforded human subjects; the inclusion of
certain priority populations in the study; 22 the extent to which privacy and
security issues have been addressed; the partnerships that the applicant
has with the proposed population; and the degree of responsiveness in
addressing the purpose and objective of the FOA. AHRQ officials told us
that the agency’s peer reviewers, program officials, and members of the
senior leadership team used both standard core criteria and other criteria
outlined in the FOAs to determine which grant applications should be
recommended to the Director of AHRQ for funding.

AHRQ issued 14 FOAs for Recovery Act CER grant opportunities. Using
its grant review process and criteria, AHRQ received 536 grant
applications and awarded 110 CER grants between February 2009 and
September 2010, totaling approximately $311 million. 23, 24 The roughly
$311 million in grants AHRQ awarded supported HHS’s departmentwide
and AHRQ’s agency-specific CER priorities. (See fig. 2 and app. II for
more information on the award of AHRQ’s CER grants with Recovery Act
funds.)




22
  AHRQ is directed to conduct and support research with respect to health care for priority
populations. These priority populations include low-income groups, women, the elderly,
minorities, individuals with disabilities, and recipients of rural health care. See 42 U.S.C.
§ 299(c)(1) . For more information on AHRQ’s priority populations, see
http://www.ahrq.gov/populations/.
23
  The 14 FOAs issued and 110 CER grants awarded addressed projects in three of the
four HHS-departmentwide CER priority areas and three of AHRQ’s seven agency-specific
CER priority areas. Some priority areas were supported by more than one FOA. AHRQ
agency-specific and HHS departmentwide priority areas not supported with grants were
supported through contracts.
24
  AHRQ made these awards by September 30, 2010, the end of the period in which the
Recovery Act funds were available for obligation. See GAO-11-712R. AHRQ awarded an
additional $161 million through contracts and spent approximately $2 million for
administrative purposes.




Page 14                              GAO-12-332 AHRQ Comparative Effectiveness Research
Figure 2: Distribution of AHRQ’s Recovery Act Funds for CER Grants, by Priority Area




                                        a
                                            AHRQ supported the remaining four CER priority areas through contracts.
                                        b
                                            AHRQ supported the one remaining HHS departmentwide CER priority area through contracts.
                                        c
                                         Percentages do not add up to 100 percent due to rounding.


                                        For 55 of the 110 grants, the Director of AHRQ exercised her discretion to
                                        make out-of-order funding decisions. 25 An out-of-order funding decision
                                        occurs when grant applications are funded out of rank order; that is, they
                                        are not funded in accordance with the rank order of most-meritorious to
                                        least-meritorious overall impact scores calculated during the peer review
                                        process. Out-of-order funding decisions can be made for a number of
                                        reasons, including the need to address important agency research
                                        priorities, avoid duplication, or meet specific requirements in the original
                                        FOA that the more meritorious applicants cannot meet.
                                        Recommendations for out-of-order funding decisions are made by the




                                        25
                                          The Director of AHRQ may not approve an application for funding unless the application
                                        has been recommended for approval by a peer review group. 42 U.S.C. § 299c-1(b);
                                        42 C.F.R. § 67.16(b).




                                        Page 15                                   GAO-12-332 AHRQ Comparative Effectiveness Research
                           program official or senior leadership team, but the final decision to award
                           grants is made by the Director of AHRQ. 26


AHRQ Primarily Used Its    Between February 2009 and September 2010, AHRQ primarily used its
Standard Competitive       standard competitive contracting process and criteria to select contract
Contracting Process and    proposals and enter into 34 contracts using approximately $161 million in
                           Recovery Act CER funds. According to AHRQ officials, a review panel,
Criteria to Award 34 CER   composed of external experts and AHRQ staff, evaluated all Recovery
Contracts                  Act CER contract proposals using the standard criteria that are tailored to
                           the specific needs of each contract solicitation. These criteria include
                           evaluating each proposal’s technical approach, management plan, and
                           key personnel. AHRQ officials reported that a contracting officer used the
                           results of the panel’s evaluation to make a final selection of contractors
                           that presented the best value to meet the needs of work specified in each
                           Recovery Act CER solicitation.

                           To meet the September 30, 2010, deadline established by the Recovery
                           Act, AHRQ made one change to its standard contracting process.
                           Specifically, for 1 of the 13 contract solicitations AHRQ conducted an
                           initial review of the contract proposals it received in order to determine
                           whether these contract proposals were duplicative or responsive to the
                           solicitation’s requirements. 27 The agency received 23 task order contract
                           proposals in response to this solicitation. Agency officials explained that
                           because they received a large number of proposals in response to this
                           solicitation, they decided to conduct the initial review to identify which of
                           the received proposals would continue through the agency’s standard,
                           competitive contract review process. Four of the 23 task order contract
                           proposals were found to be nonresponsive or duplicative of another
                           previously funded study and, therefore, were not considered for further
                           review.




                           26
                             When we reviewed AHRQ documentation of out-of-order funding decisions made during
                           the selection of Recovery Act CER grant recipients, we found that although AHRQ officials
                           generally followed the agency’s internal policies and procedures to document and review
                           these decisions, the documentation lacked the level of detail required by AHRQ’s policies
                           at the time these grants were awarded in 2009 and 2010. The agency revised its policy on
                           the level of detail required to document out-of-order funding decisions on October 3, 2011,
                           to reflect current practices.
                           27
                            This solicitation was for the Evidence Generation priority area.




                           Page 16                             GAO-12-332 AHRQ Comparative Effectiveness Research
AHRQ awarded task orders under existing AHRQ master contracts, task
orders under existing GSA master contracts, and stand-alone contracts
using Recovery Act CER funds. The agency primarily awarded task
orders under existing master contracts when awarding contracts with
Recovery Act CER funds. Specifically, of the 34 contracts that AHRQ
awarded using Recovery Act CER funds, 30 were task orders under
either existing AHRQ contracts or existing GSA-schedule contracts, and
the remaining 4 were stand-alone contracts. AHRQ awarded multiple task
orders within the Evidence Synthesis and Gap Identification priority areas
under existing contracts that AHRQ entered into prior to the passage of
the Recovery Act. Officials stated that issuing task orders under existing
master contracts, which included a GSA-schedule task order, facilitated
the quick and efficient award of Recovery Act funds in instances where
the agency or GSA had existing master contracts with entities capable of
conducting work the agency wanted to support with Recovery Act CER
funds. 28 AHRQ officials stated that this approach was faster and more
cost-effective than entering into new, stand-alone contracts. AHRQ
officials also said that they used stand-alone contracts only in instances
where there were existing contracts with entities that could perform the
planned work.

AHRQ issued 13 CER contract solicitations between February 2009 and
September 2010. Using its standard contracting process and criteria, the
agency received 80 contract proposals and entered into 34 contracts
totaling almost $161 million. 29, 30 These contracts supported HHS’s
departmentwide and AHRQ’s agency-specific CER priorities. (See fig. 3
and app. III for more information on the award of AHRQ’s CER contracts
with Recovery Act funds.)


28
  GAO previously reported that federal agencies primarily awarded Recovery Act funds
under existing contracts, such as through task orders. See GAO, Recovery Act:
Contracting Approaches and Oversight Used by Selected Federal Agencies and States,
GAO-10-809 (Washington, D.C.: July 15, 2010).
29
  The 13 contract solicitations issued and 34 CER contracts awarded addressed projects
in AHRQ and HHS-departmentwide CER priority areas. Some priority areas were
supported by more than one contract solicitation. AHRQ agency-specific and HHS
departmentwide priority areas not supported with contracts were supported through
grants.
30
  AHRQ made these awards by September 30, 2010, the end of the period in which the
Recovery Act funds were available for obligation. See GAO-11-712R. AHRQ awarded an
additional $311 million through grants and spent approximately $2 million for
administrative purposes.




Page 17                           GAO-12-332 AHRQ Comparative Effectiveness Research
Figure 3: Distribution of AHRQ’s Recovery Act Funds for CER Contracts, by Priority Area




                                        a
                                         The remaining two of the seven AHRQ priority areas were supported by grants. In addition, contracts
                                        awarded for the Evidence Synthesis and Evidence Gap Identification priority areas were combined.
                                        b
                                         Percentages do not add up to 100 percent due to rounding.


AHRQ Also Reported                      AHRQ officials reported that they used five mechanisms in order to
Taking Steps to Coordinate              coordinate with other HHS agencies to avoid unnecessary duplication
Recovery Act CER Awards                 when creating FOAs for grants and solicitations for contracts and when
                                        awarding Recovery Act CER funds. Specifically, AHRQ participated in a
with Other HHS Agencies                 federal interagency coordination council and an HHS working group;
to Avoid Unnecessary                    contributed to an HHS spending plan to coordinate the department’s
Duplication                             solicitations; participated with NIH in another working group to coordinate
                                        both solicitations and CER awards; and queried HHS databases when
                                        awarding Recovery Act funds to identify potentially duplicative projects.

                                        •    Federal Coordinating Council for CER—AHRQ participated on the
                                             Federal Coordinating Council for CER (“the Council”), a body created
                                             by the Recovery Act to foster coordination for CER across the federal
                                             government in an effort to reduce duplication and encourage the
                                             coordinated and complementary use of resources. In addition to
                                             AHRQ, officials from the Veterans Health Administration (VHA), the



                                        Page 18                                GAO-12-332 AHRQ Comparative Effectiveness Research
     Department of Defense, and NIH also served on the Council.
     According to AHRQ officials, the Council provided a mechanism for
     coordinating, among other things, the establishment of CER priorities
     and some Recovery Act CER grant announcements and contract
     solicitations. The Council was terminated by PPACA in March 2010.

•    CER Coordination and Implementation Team (CER-CIT)—In
     addition to the Council, AHRQ officials participated in the CER-CIT, a
     departmentwide effort to coordinate investments in CER supported
     with Recovery Act funds. Organized by HHS, the CER-CIT served as
     a centralized forum for HHS officials to assess FOAs for grants and
     solicitations for contracts. AHRQ officials stated that the CER-CIT’s
     process helped ensure that the FOAs and solicitations ultimately
     posted by AHRQ for grants and contracts were not duplicative of
     FOAs and solicitations posted by other entities within HHS. For
     example, during the CER-CIT’s review of two proposed AHRQ CER
     FOAs, reviewers identified aspects of the proposed announcements
     that were potentially duplicative of other proposed or existing projects.

•    HHS Intra-Agency Spending Plan—AHRQ contributed to an intra-
     agency spending plan developed by HHS that describes how all HHS
     agencies anticipated using the funding they received under the
     Recovery Act for CER. AHRQ contributed to this intra-agency
     spending plan by developing an agency-specific spending plan that
     described AHRQ’s research priorities and how the agency anticipated
     using its $300 million in Recovery Act CER funds to support these
     priorities. 31 HHS incorporated AHRQ’s spending plan into the
     department’s intra-agency spending plan. According to AHRQ
     officials, officials from the HHS Office of the Secretary, who were
     responsible for coordinating this effort, reviewed the spending plans it
     received from AHRQ and other agencies, and this helped ensure that
     AHRQ’s Recovery Act CER solicitations were not unnecessarily
     duplicative of other CER efforts within HHS.

•    AHRQ-NIH Working Group—In addition to the departmentwide
     working group that was primarily focused on coordination of FOAs
     and contract solicitations, AHRQ and NIH formed a working group to
     coordinate the award of Recovery Act funds to avoid unnecessarily


31
  Agencies receiving Recovery Act funds were required by the Office of Management and
Budget (OMB) to develop a spending plan that documented, among other things, how the
agency planned to use its Recovery Act funds.




Page 19                           GAO-12-332 AHRQ Comparative Effectiveness Research
     funding duplicative projects. 32 AHRQ officials stated that before
     making awards, they checked with NIH through this working group to
     ensure that the two agencies were not funding duplicative projects.
     For example, during one meeting members noted where issues of
     duplication needed to be further discussed to ensure that studies were
     complementary and not duplicative. In addition to reviewing awards,
     AHRQ officials reported that this working group met regularly during
     the award of the Recovery Act CER funds to share spending plans,
     share solicitations, and provide updates on each agency’s respective
     CER activities.

•    Querying of HHS Databases—AHRQ officials stated that in order to
     avoid funding unnecessarily duplicative work with Recovery Act funds,
     they queried HHS databases prior to awarding any Recovery Act CER
     funds to ensure that other similar projects were not funded elsewhere
     within HHS. 33 AHRQ officials stated that if, in the process of querying
     these databases, a duplicative award was identified, AHRQ would
     contact the appropriate HHS project officer listed in the database to
     discuss the award in more detail. AHRQ and NIH officials confirmed
     that this process resulted in the identification of grant proposals for
     training awards that were potentially duplicative of projects AHRQ had
     funded. Once identified, it was decided that NIH would not fund these
     potentially duplicative awards.




32
  According to AHRQ officials, the AHRQ-NIH Working Group is a subcommittee of the
NIH Comparative Effectiveness Research Coordinating Committee. AHRQ and NIH are
the HHS agencies most active in CER.
33
 According to AHRQ officials, the databases they queried were the Information for
Management and Planning Analysis and Coordination II and NIH’s Research Portfolio
Online Reporting Tools.




Page 20                           GAO-12-332 AHRQ Comparative Effectiveness Research
                              AHRQ plans to use a range of existing mechanisms, such as written
AHRQ Plans to Use             products, training, social media tools, and its website, to disseminate
Its Existing                  results of CER funded through the Recovery Act. According to AHRQ
                              officials, the agency will determine which specific mechanisms will be
Mechanisms and                used to disseminate CER results by considering the unique
Develop Additional            characteristics of the research. In addition, AHRQ awarded four contracts
Strategies to                 using Recovery Act CER funds to develop and implement innovative
                              approaches for disseminating CER results, including Recovery Act-
Disseminate CER               funded CER.
Results
AHRQ Plans to Use a           AHRQ officials stated that the agency plans to use a range of existing
Range of Existing             mechanisms to disseminate Recovery Act-funded CER results as such
Mechanisms, Such as           results become available. As of December 6, 2011, 30 Recovery Act CER
                              projects were completed or in draft and some dissemination activities had
Written Products, Training,   begun. 34 The mechanisms AHRQ plans to use to disseminate Recovery
Social Media Tools, and Its   Act-funded CER include written products, training programs, social media
Website, to Disseminate       tools, learning networks, and AHRQ’s website. The different types of
Recovery Act CER Results      written products that AHRQ develops for CER results and other research
                              include comprehensive research reviews that summarize existing
                              research on a CER topic; original research reports that introduce new
                              CER results; and plain language publications that summarize the findings
                              of research on the benefits and harms of different treatment options and
                              which are tailored to clinicians, consumers, or policymakers. AHRQ’s
                              training programs include web-based conferences that feature
                              presentations by experts accompanied by instructional slides for
                              clinicians. In addition, the agency employs social media tools to
                              disseminate notices of CER results including electronic newsletters, audio
                              podcasts, and Twitter. AHRQ is also drawing on an existing agency
                              learning network for Medicaid medical directors that was created in 2005.
                              This group convenes periodically to discuss ways to advance the health
                              of Medicaid beneficiaries, including how evidence-based research
                              findings can be used to improve quality of care. 35 AHRQ’s website
                              provides access to CER results through search tools and links to its



                              34
                                While Recovery Act funds were required to be obligated by September 30, 2010, project
                              end dates may occur after that date. Some projects may take several years to complete
                              and be ready for dissemination by AHRQ.
                              35
                                In addition to this learning network, AHRQ established a learning network that focuses
                              on tools and products related to patient safety.




                              Page 21                             GAO-12-332 AHRQ Comparative Effectiveness Research
                           written and social media formats. (See app. IV for a more detailed
                           description of these existing mechanisms.)

                           AHRQ officials explained that the agency determines which specific
                           mechanisms will be used to disseminate particular CER results by
                           considering the unique characteristics of the research such as the type of
                           research conducted, its potential impact, the strength of the evidence,
                           and the audiences that can best make use of the information. AHRQ then
                           develops a marketing plan that identifies key messages, target
                           audiences, and the mechanisms to be used to disseminate CER to those
                           audiences. For example, AHRQ’s marketing plan for a CER project that
                           examines certain treatments for type 2 diabetes targeted consumers as
                           well as primary care clinicians and certain specialist clinicians and other
                           health professionals. While this CER project was not funded with
                           Recovery Act funds, AHRQ officials confirmed that the process they use
                           to customize the dissemination for this project is the same process the
                           agency will follow for disseminating Recovery Act-funded CER results. To
                           disseminate the CER results of this project to consumers, AHRQ
                           developed a consumer guide and a series of audio podcasts. To reach
                           clinicians, AHRQ developed a clinician’s guide and a webcast program
                           with educational slides. These products were targeted to be distributed
                           through multiple channels including AHRQ’s website, as well as its
                           newsletters and list-servs. Notices about these products were also sent
                           directly to a range of general media news services; consumer health and
                           advocacy publications; and a wide range of key national organizations
                           that included those representing primary care, specialty clinicians, and
                           payers. (See app. V for a more detailed description of this dissemination
                           effort.)


AHRQ Is Developing         In addition to its existing mechanisms for disseminating the results of
Additional Strategies to   CER, AHRQ is in the process of developing additional dissemination
Disseminate CER Results    strategies. Specifically, in September 2010, AHRQ awarded four
                           Recovery Act-funded contracts to develop and implement innovative
                           approaches for disseminating CER results, including Recovery Act-
                           funded CER. The specific purpose of each of the four contracts is
                           described below.

                           •   Academic Detailing. Academic detailing involves face-to-face
                               educational sessions by trained clinicians, including physicians,
                               nurses, pharmacists, and others, who visit health professionals in their
                               practice settings. The goal of these sessions is to share evidence-
                               based information and facilitate use of that information to improve


                           Page 22                       GAO-12-332 AHRQ Comparative Effectiveness Research
     patient care. AHRQ awarded a contract in the amount of $11,680,060
     for the purpose of implementing academic detailing for CER from
     2011 through 2013. The plan under this contract calls for academic
     detailing to 1,300 primary care providers and 200 health care system
     practice sites. Each provider or site will receive one face-to-face visit
     every 6 months plus follow-up e-mail communications and supporting
     materials. The academic detailing will focus on six CER topics over
     the 3-year period. 36 Between February 2011 and October 2011, work
     completed under this contract resulted in over 1,562 visits to providers
     and practice sites. These visits involved the discussion of AHRQ’s
     CER results related to the treatment of type 2 diabetes.

•    Continuing Education Modules. AHRQ awarded a contract in the
     amount of $3,981,168 for the purpose of developing and
     disseminating 45 accredited online continuing education programs for
     health care professionals, including physicians, physician assistants,
     pharmacists, nurses, nurse practitioners, medical assistants, and
     other health professionals. These programs translate CER results into
     a variety of formats, for example, videos featuring case studies and
     journal supplements. As of November 30, 2011, 13 approved
     continuing education programs were completed, including programs
     on CER results related to hip fractures, hypertension, prostate cancer,
     breast cancer, heart disease, and diabetes.

•    Regional Dissemination and Partnership Offices. AHRQ awarded
     a contract in the amount of $8,613,876 to create five regional offices
     for the purpose of establishing partnerships to facilitate dissemination
     and use of CER results by regional health care organizations,
     businesses, unions, and consumer groups. Collaborative efforts are
     expected to result in local and regional meetings, web conferences,
     training programs, and distribution of CER results to partner
     organizations’ memberships.



36
  The first two topics selected for academic detailing under this project address CER
related to type 2 diabetes: “Comparative Effectiveness, Safety and Indications of Pre-
mixed Insulin Analogues for Adults with Type 2 Diabetes” and “Comparative Effectiveness
and Safety of Oral Diabetes Medications for Adults with Type 2 Diabetes.” The third topic
selected for academic detailing visits is in the area of heart and blood vessel conditions
and will be based on the reports titled, “Comparative Effectiveness of Angiotensin-
Converting Enzyme Inhibitors, Angiotensin II Receptor Antagonists,” and “Direct Renin
Inhibitors for Treating Essential Hypertension.” As of December 2011 additional topics
were expected to cover other heart and blood vessel conditions; muscle, bone, and joint
conditions; and/or mental health conditions.




Page 23                             GAO-12-332 AHRQ Comparative Effectiveness Research
•   Publicity Center. AHRQ awarded a contract in the amount of
    $17,999,988 to develop and implement a national strategic
    communications plan for AHRQ’s CER results. The communications
    plan calls for the development of national partnerships with consumer,
    clinician, policymaker, and business audiences; marketing efforts,
    including the use of social media, focused on disseminating results of
    research; and creation of new website portals with established sites
    reaching patients and clinicians. For example, under the contract,
    partnerships have been established with such organizations as the
    National Rural Health Association, the National Alliance for
    Caregiving, the American College of Cardiology, and the American
    Medical Student Association.

Along with its efforts to develop additional strategies for disseminating
CER results, AHRQ is taking steps to evaluate the effectiveness of these
strategies. Specifically, using its Recovery Act funds AHRQ awarded a
contract in the amount of $2,371,179 for the purpose of evaluating some
of AHRQ’s dissemination strategies by collecting data about
dissemination. Under the contract, information will be collected about
changes over time in the level of awareness, understanding, use, and
perceived benefits of CER . This information will be gathered from
clinicians, patients, consumers, health system decision makers,
purchasers, and policymakers. In addition, this evaluation includes plans
to collect process and outcomes data for each of the additional
dissemination strategies being developed under Recovery Act-funded
contracts including academic detailing, continuing education, regional
dissemination, and the national publicity center.

AHRQ officials noted that evaluating the impact of dissemination of its
CER results is important but also challenging. They noted that clinician
practice behavior often changes slowly and is affected by many variables,
thereby making it difficult to directly attribute changes to information
AHRQ has disseminated. In addition, once CER results are disseminated
to target audiences through, for example, AHRQ’s website or one of its
educational programs, it is often not feasible to track secondary
dissemination from those audiences to others.




Page 24                      GAO-12-332 AHRQ Comparative Effectiveness Research
                       While PCORI is in the early stages of development, AHRQ has begun to
AHRQ Has Begun to      monitor PCORI’s needs to determine what resources might be needed by
Monitor PCORI and      AHRQ to fulfill its PPACA responsibilities related to PCORI and identify
                       existing resources that the agency can use to fulfill these
Identify Resources     responsibilities. 37 These responsibilities include broadly disseminating the
That Could Enable It   research findings published by PCORI; developing a publicly available
to Fulfill Its PPACA   database to collect government-funded evidence and research from
                       public, private, not-for-profit, and academic sources; promoting the timely
Responsibilities       incorporation of PCORI-generated CER findings into health information
Related to PCORI       technology systems that support clinical decision making; and
                       establishing a process for receiving feedback about the value of
                       information disseminated by AHRQ. 38

                       AHRQ officials report that they are monitoring PCORI’s needs to
                       determine what resources might be needed by AHRQ to fulfill its PPACA
                       responsibilities related to PCORI. The director of AHRQ serves on
                       PCORI’s Board of Governors and another high-level AHRQ official serves
                       on PCORI’s methodology committee, which allows AHRQ to obtain
                       information on the resources the institute might need and when these
                       resources might be needed. In addition, according to AHRQ officials, the
                       agency has shared information with PCORI members about AHRQ’s
                       existing resources at various PCORI meetings.

                       AHRQ officials reported that they are also in the process of identifying
                       existing resources, including existing capabilities and ongoing projects,
                       that the agency can leverage to fulfill its responsibilities related to PCORI.
                       For example, AHRQ officials are exploring whether contracts the agency
                       awarded to evaluate AHRQ’s CER dissemination efforts could be
                       leveraged to meet the agency’s responsibilities to obtain, on behalf of
                       PCORI, feedback from health care professionals on the CER information
                       disseminated by AHRQ. In addition, AHRQ is currently assessing whether
                       a research database being developed by HHS’s Office of the Assistant
                       Secretary for Planning and Evaluation could be used to, among other
                       things, store and make publicly available CER funded and generated by
                       PCORI.



                       37
                        The Board of Governors for PCORI incorporated the institute in November 2011, and
                       PCORI is in the process of hiring staff to run this nonprofit organization.
                       38
                         See Pub. L. No. 111-148, §§ 6301(a), (b), 10602, 124 Stat. 119, 727-47, 1005 (2010) (to
                       be codified at 42 U.S.C. §§ 299b-37, 1320e).




                       Page 25                            GAO-12-332 AHRQ Comparative Effectiveness Research
                  In addition, AHRQ has developed spending plans for fiscal years 2011
                  and 2012 that describe how AHRQ will use the funds it receives from
                  PCORTF to fulfill the agency’s responsibilities related to PCORI. 39 These
                  plans describe proposed FOAs and contract solicitations that would
                  expand opportunities for AHRQ to disseminate CER information through
                  a variety of channels to different target audiences, for example, public
                  service announcements targeting consumers and symposia and
                  publications targeting researchers and health care professionals. AHRQ
                  officials stated that the fiscal year 2011 plan has been approved by OMB,
                  and the fiscal year 2012 plan was under review by HHS as of December
                  2011. AHRQ officials stated that they have issued one FOA for a project
                  described in the fiscal year 2011 spending plan but, as of December
                  2011, have not made any awards.


                  We provided a draft of this report to AHRQ for review and comment.
Agency Comments   AHRQ provided technical comments, which we incorporated where
                  appropriate.


                  As agreed with your offices, unless you publicly announce the contents of
                  this report earlier, we plan no further distribution until 30 days from the
                  report date. At that time, we will send copies to the Secretary of HHS,
                  interested congressional committees, and others. In addition, the report
                  will be available at no charge on the GAO website at http://www.gao.gov.




                  39
                   The funds provided to AHRQ through PCORTF are available until expended. See 26
                  U.S.C. § 9511(d)(2)(B).




                  Page 26                         GAO-12-332 AHRQ Comparative Effectiveness Research
If you or your staffs have questions about this report, please contact me
at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report. GAO staff who made key contributions to this report are
listed in appendix VI.




Linda T. Kohn
Director, Health Care




Page 27                      GAO-12-332 AHRQ Comparative Effectiveness Research
Appendix I: The Agency for Healthcare
              Appendix I: The Agency for Healthcare
              Research and Quality’s (AHRQ) Mission,
              Research, Priorities, and Budget


Research and Quality’s (AHRQ) Mission,
Research, Priorities, and Budget
              AHRQ’s mission is to improve the quality, safety, efficiency, and
              effectiveness of health care for all Americans. The purpose of AHRQ’s
              research is to help people make more informed decisions and improve
              the quality of health care services. AHRQ, formerly known as the Agency
              for Health Care Policy and Research, is 1 of 12 agencies within the U.S.
              Department of Health and Human Services (HHS). While the National
              Institutes of Health focuses on biomedical research to prevent, diagnose,
              and treat disease and the Centers for Disease Control and Prevention
              focuses on population health and the role of community-based
              interventions to improve health, AHRQ’s research focus is on long-term
              and systemwide improvement of health care quality and effectiveness.

              AHRQ conducts work in five broad focus areas. These areas include
              comparing effectiveness of treatments; quality improvement and patient
              safety; health information technology; prevention and care management;
              and health care value.

              •   Comparing the effectiveness of treatments. AHRQ’s comparative
                  effectiveness research (CER) 1 provides patients and physicians with
                  information on which medical treatments work best for a given
                  condition. This includes comparisons of drugs, medical devices, tests,
                  surgeries, or ways to deliver health care in an effort to help patients
                  and their families understand which treatments work best and how
                  their risks compare. Initiatives under this focus area include:
                  •     The John M. Eisenberg Center for Clinical Decisions and
                        Communications Sciences

                  •     Evidence-based Practice Centers

                  •     The Centers for Education and Research on Therapeutics

                  •     The Developing Evidence to Inform Decisions about Effectiveness
                        Network
                  Quality improvement and patient safety. AHRQ funds and
                  disseminates research that identifies root causes of threats to patient
                  safety, provides information on the scope and impact of medical
                  errors, and examines effective ways to make system-level changes to
                  help prevent errors. Initiatives under this focus area include:


              1
              AHRQ also refers to CER as patient-centered outcomes research.




              Page 28                            GAO-12-332 AHRQ Comparative Effectiveness Research
Appendix I: The Agency for Healthcare
Research and Quality’s (AHRQ) Mission,
Research, Priorities, and Budget




    •     Preventing healthcare-associated infections

    •     Medical liability reform

    •     Patient Safety Organizations

    •     TeamSTEPPS ®

    •     Patient safety culture assessment tools
•   Health information technology (health IT). AHRQ provides support
    to give access to and encourage the adoption of health IT. The
    agency has focused its health IT activities on the following three
    goals:
    •     Improve health care decision making

    •     Support patient-centered care

    •     Improve the quality and safety of medication management
•   Prevention and care management. AHRQ translates evidence-
    based knowledge into recommendations for clinical preventative
    services. AHRQ initiatives under this focus area include:
    •     The U.S. Preventative Services Task Force

    •     The Patient-Centered Medical Home

    •     The Practice-Based Research Network
•   Health care value. AHRQ aims to find greater value in health care by
    producing the measures, data, tools, evidence, and strategies that
    health care organizations, systems, insurers, purchasers, and
    policymakers need to improve the value and affordability of health
    care. Initiatives under this focus area include:
    •     The Medical Expenditure Panel Survey

    •     The Healthcare Cost and Utilization Project

    •     Quality Indicators

    •     The annual National Healthcare Quality Report and National
          Healthcare Disparities Report

    •     State snapshots



Page 29                            GAO-12-332 AHRQ Comparative Effectiveness Research
Appendix I: The Agency for Healthcare
Research and Quality’s (AHRQ) Mission,
Research, Priorities, and Budget




    •     The Consumer Assessment of Healthcare Providers and Systems

    •     The National Guideline Clearinghouse

    •     The National Quality Measures Clearinghouse
In addition to the above focus areas, AHRQ also conducts crosscutting
activities related to quality, effectiveness, and efficiency. Activities include
data collection and measurement; dissemination and translation; and
program evaluation. In addition, support is provided for the investigator-
initiated and targeted research grants and contracts that focus on health
services research in the areas of quality, effectiveness, and efficiency.
These activities provide the core infrastructure used by the other focus
areas.

AHRQ staff and budget. AHRQ currently employs approximately 300
staff. The agency’s fiscal year 2010 budget was $402.6 million, of which
$270.7 million went to research on health costs, quality, and outcomes.
The President’s fiscal year 2012 budget request for AHRQ was
$366.4 million, a decrease of approximately $31 million from fiscal year
2010. 2 (See table 4 for the funding amounts under AHRQ’s focus areas.)




2
 AHRQ received $369 million for fiscal year 2012 under the Consolidated Appropriations
Act, 2012, approximately $2.6 million more than requested. AHRQ’s plans for the
additional $2.6 million were not yet available at the time we completed our review. In fiscal
year 2012, AHRQ will also receive a transfer from the Patient-Centered Outcomes
Research Trust Fund (PCORTF) in the amount of $24 million, bringing the total funds
available for fiscal year 2012 to $393 million.




Page 30                              GAO-12-332 AHRQ Comparative Effectiveness Research
                                              Appendix I: The Agency for Healthcare
                                              Research and Quality’s (AHRQ) Mission,
                                              Research, Priorities, and Budget




Table 4: AHRQ Budget, by Fiscal Year and Research Area

                                                           Fiscal year 2010                       Fiscal year 2012 Budget
                                                                            a
                                                           Budget (actual)                           (President’s Budget)     Percent change over
                                                        (dollars in millions)                         (dollars in millions)   fiscal year 2010 (+/-)
Research on Health Costs, Quality, and Outcomes (by focus area)
Patient-centered health research/effective
                                                                                     b
health care                                                                  $21.0                                   $21.6                     +2.9
Patient safety                                                                90.59                                  64.62                    -28.7
Health information technology (health IT)                                     27.65                                  27.57                      -0.3
Prevention and care management                                                  15.9                                  23.3                    +46.5
Health care value                                                               3.73                                  3.73                        0
Crosscutting activities related to quality,
effectiveness, and efficiency research                                      111.79                                   91.78                    -17.9
Subtotal - Research on Health Costs, Quality,
and Outcomes                                                                  270.7                                  232.6                   -14.07
Other budget line items
                                      c
Medical Expenditure Panel Surveys                                               58.8                                  59.3                    +0.85
                  d
Program support                                                                 67.6                                  74.5                   +10.21
Prevention and Public Health Fund                                                 5.5                                    0                     -100
Transfer from the Patient-Centered Outcomes
                                e
Research Trust Fund (PCORTF)                                                         0                                24.0                      N/A
AHRQ total program level                                                    $402.6                                  $390.4                    -3.03
                                              Source: AHRQ fiscal year 2012 Congressional Budget Justification.
                                              a
                                               In fiscal year 2011, AHRQ received $372.1 million under the Department of Defense and Full-Year
                                              Continuing Appropriations Act, 2011, and $8 million transferred from PCORTF. In fiscal year 2011,
                                              AHRQ allocated funds to programs and activities as they had done under the fiscal year 2010
                                              appropriation.
                                              b
                                              In addition to this amount, AHRQ received $300 million under the American Recovery and
                                              Reinvestment Act of 2009 for CER, which was used within this focus area.
                                              c
                                               The Medical Expenditure Panel Surveys is the only national source for annual data on how
                                              Americans use and pay for medical care. The survey collects detailed information from families on
                                              access, use, expense, insurance coverage, and quality. Data are disseminated to the public.
                                              d
                                               The program support budget category supports the strategic direction and overall management of
                                              the agency. Program support activities for AHRQ include salaries, travel, rent, supplies,
                                              transportation, printing, and other reproduction costs, supplies, equipment, and furniture.
                                              e
                                               AHRQ receives funding from PCORTF for fiscal year 2011 through fiscal year 2019 to carry out
                                              activities outlined by the Patient Protection and Affordable Care Act.




                                              Page 31                                             GAO-12-332 AHRQ Comparative Effectiveness Research
Appendix II: AHRQ CER Grants Awarded
                                          Appendix II: AHRQ CER Grants Awarded Using
                                          Recovery Act CER Funds, by Priority Area



Using Recovery Act CER Funds, by Priority
Area
Table 5: Grant Applications Received and Reviewed, and Grants Awarded, by AHRQ’s CER Priority Areas

                                                    Number of grant                  Number of grant     Number of
Source of Recovery                                     applications            applications submitted       grants
                                                                  a
Act funds          AHRQ CER priority area                received                      for peer review    awarded         Amount awarded
Recovery Act Funds   Horizon Scanning                                     0                         0               0                          0
Appropriated to      Evidence Synthesis                                   0                         0               0                          0
AHRQ
                     Evidence Gap
                     Identification                                       0                         0               0                          0
                                          b
                     Evidence Generation                               183                         94              19       $148,827,978.00
                     Translation and
                     Dissemination                                      91                         49              28          35,670,901.00
                     Training and
                     Development                                        30                         26               8          15,384,771.00
                     Community Forum                                      0                         0               0                          0
                     Total                                             304                        169              55       $199,883,650.00
                                          Source: GAO analysis of AHRQ data.
                                          a
                                           The number of grant applications received includes all applications received electronically. Some
                                          applications may have been received but not reviewed due to incomplete submission of the
                                          application.
                                          b
                                           AHRQ and the HHS Office of the Secretary jointly funded one grant. For purposes of our report, we
                                          counted this grant under the number of grants awarded for the HHS Office of the Secretary’s
                                          Dissemination and Translation priority area (see table 7). However, the amount of Recovery Act CER
                                          funds the HHS Office of the Secretary and AHRQ awarded to this grant are reflected under the
                                          amount awarded for the HHS Office of the Secretary’s Dissemination and Translation priority area
                                          and AHRQ’s Evidence Generation priority area.




                                          Page 32                                     GAO-12-332 AHRQ Comparative Effectiveness Research
                                           Appendix II: AHRQ CER Grants Awarded Using
                                           Recovery Act CER Funds, by Priority Area




Table 6: Grant Applications Received and Reviewed, and Grants Awarded, by HHS Departmentwide CER Priority Areas


Source of                                         Number of grant                     Number of grant     Number of
Recovery Act         HHS departmentwide              applications               applications submitted       grants
                                                                a
funds                CER priority area                 received                         for peer review    awarded         Amount awarded
Recovery Act         Data Infrastructure                             109                            65            39          $77,366,386.00
funds appropriated   Dissemination and
to the HHS Office               b
                                                                       21                           16             6           16,173,967.00
                     Translation
of the Secretary
and allocated to     Research                                        102                            60            10           17,841,781.00
AHRQ                 Inventory and Evaluation                           0                            0             0                            0
                     Total                                           232                           141            55         $111,382,134.00
                                           Source: GAO analysis of AHRQ data.
                                           a
                                            The number of grant applications received includes all applications received electronically. Some
                                           applications may have been received but not reviewed due to incomplete submission of the
                                           application.
                                           b
                                            AHRQ and the HHS Office of the Secretary jointly funded one grant. For purposes of our report, we
                                           counted this grant under the number of grants awarded for the HHS Office of the Secretary’s
                                           Dissemination and Translation priority area. However, the amount of Recovery Act CER funds the
                                           HHS Office of the Secretary and AHRQ awarded to this grant are reflected under the amount
                                           awarded for the HHS Office of the Secretary’s Dissemination and Translation priority area and
                                           AHRQ’s Evidence Generation priority area.




                                           Page 33                                       GAO-12-332 AHRQ Comparative Effectiveness Research
Appendix III: AHRQ CER Contracts Awarded
                                       Appendix III: AHRQ CER Contracts Awarded
                                       Using Recovery Act CER Funds, by Priority
                                       Area


Using Recovery Act CER Funds, by Priority
Area
Table 7: Contract Proposals Received and Reviewed, and Contracts Awarded, by AHRQ’s CER Priority Areas

                                                                                Number of contracts awarded
                                       Number of            Number of
Source of                                contract             contract                GSA
Recovery Act      AHRQ CER             proposals            proposals            schedule       Task          Stand               Amount
                                                a
funds             priority area         received             reviewed           task order    orders          alone              awarded
Recovery Act      Horizon Scanning                  4                       4           0           0              1       $9,499,986.00
funds             Evidence
appropriated to
                  Synthesis                        14                   14              0          10              0       49,904,490.00
AHRQ
                  Evidence Gap
                                b
                  Identification                    0                       0           0           0              0                     0
                  Evidence
                  Generation                       23                   19              0           7              1       28,874,761.00
                  Translation and
                  Dissemination                     0                       0           0           0              0                     0
                  Training and
                  Development                       0                       0           0           0              0                     0
                  Community Forum                   9                       9           0           0              1        9,999,742.00
                  Total                            50                   46              0          17              3     $98,278,979.00
                                       Source: GAO analysis of AHRQ data.
                                       a
                                        The number of contract proposals received includes all applications received electronically. Some
                                       proposals may have been received but not reviewed for a variety of reasons, including proposals that
                                       were incomplete.
                                       b
                                        AHRQ combined Evidence Synthesis and Gap Identification awards under a single solicitation when
                                       announcing the availability of these funds and within awards because, according to agency officials,
                                       having a single solicitation for these two priority areas reduced the amount of work related to these
                                       awards, thereby expediting the award process. As a result, AHRQ funded both of these priority areas,
                                       but advertised projects and made awards for these priority areas under a single solicitation.




                                       Page 34                                     GAO-12-332 AHRQ Comparative Effectiveness Research
                                         Appendix III: AHRQ CER Contracts Awarded
                                         Using Recovery Act CER Funds, by Priority
                                         Area




Table 8: Contract Proposals Received and Reviewed, and Contracts Awarded, by HHS Departmentwide CER Priority Areas

                                                                                   Number of contracts awarded
                                             Number of            Number of
Source of          HHS                         contract             contract             GSA
Recovery Act       departmentwide            proposals            proposals         schedule      Task         Stand               Amount
                                                      a
funds              CER priority areas         received             reviewed        task order   orders         alone              awarded
Recovery Act       Data Infrastructure                  11                    11           0           4             1     $13,683,264.00
funds              Dissemination and
appropriated to
                   Translation                          15                    15           4           2             0      46,856,206.00
the HHS Office
of the Secretary   Research                               2                   2            0           1             0        1,060,353.00
and allocated to   Inventory and
AHRQ               Evaluation                             2                   2            0           2             0        1,008,456.00
                   Total                                30                    30           4           9             1     $62,608,279.00
                                         Source: GAO analysis of AHRQ data.
                                         a
                                          The number of contract proposals received includes all applications received electronically. Some
                                         proposals may have been received but not reviewed for a variety of reasons, including proposals that
                                         were incomplete.




                                         Page 35                                    GAO-12-332 AHRQ Comparative Effectiveness Research
Appendix IV: AHRQ Mechanisms That
                                     Appendix IV: AHRQ Mechanisms That Support
                                     Dissemination of CER



Support Dissemination of CER


Types of mechanisms supporting
dissemination                        Examples of AHRQ’s dissemination mechanisms
Written products                     •    Research reviews and original research reports: These written products draw on
                                          completed scientific studies to make comparisons of different health care
                                          interventions or summarize original clinical research to explore practical questions
                                          about the effectiveness of treatments.
                                     •    Summary treatment guides for clinicians, consumers, policymakers: Short, plain-
                                          language guides summarize the findings of research reviews on the benefits and
                                          harms of different treatment options.
                                     •    Education modules and presentation slides: These resources are for clinicians
                                          pursuing continuing education credits and for faculty who are instructing clinicians.
Training programs                    •    Webcasts: Researchers and clinicians participate in online programs to discuss
                                          research findings.
                                     •    Conference series: Scientific meetings on state-of-the art concepts in
                                          communication, health literacy, and medical decision making.
Social media tools                   •    Audio podcasts: The Healthcare 411 audio podcast series shares news and
                                          information with consumers that they can use in health care decision making,
                                          through 60-second audio news programs and longer format interviews.
                                     •    Online videos: AHRQHealthTV provides videos for consumers about a range of
                                          health topics on AHRQ’s YouTube channel.
                                     •    Twitter updates: Short messages are broadcast that can be accessed by computer
                                          or mobile phone.
                                     •    RSS Feeds: Subscribers receive news and alerts about AHRQ programs through
                                          their RSS reader.
                                     •    E-mail updates: Subscribers receive e-mail updates on topics they are interested in.
Learning networks and case studies   •    The Medicaid Medical Directors Learning Network: This is one example of a network
                                          formed by AHRQ to create an ongoing collaborative relationship to disseminate
                                          AHRQ products, tools, and research to help members make policy and practice
                                          decisions related to clinical treatment.
                                     •    Impact Case Studies: AHRQ tracks and summarizes how AHRQ-funded research,
                                          tools, and products are actually implemented by state governments, medical
                                          practices, clinics, and hospitals. Makes summary information available to other
                                          potential users.
Website and other resources          •    AHRQ’s website ahrq.gov provides access to its written products, training programs,
                                          and social media tools, as well as useful search functions and other resources.
                                     •    Health Care Innovations Exchange: The Exchange offers health professionals and
                                          researchers searchable tools to access information about evidence-based
                                          innovations suitable for a range of health care settings and populations, as well as
                                          opportunities to network with other professionals who have implemented these
                                          innovations.
                                     Source: AHRQ.




                                     Page 36                              GAO-12-332 AHRQ Comparative Effectiveness Research
Appendix V: Example of Dissemination of
                           Appendix V: Example of Dissemination of
                           Comparative Effectiveness Research by the
                           Agency for Healthcare Research and Quality


Comparative Effectiveness Research by the
Agency for Healthcare Research and Quality

                           Comparative Effectiveness and Safety of Premixed Insulin Analogues in Type 2
CER topic                  Diabetes: A Systematic Review
Background                 Type 2 diabetes is an increasingly common chronic disease that occurs in people who
                           have too much glucose in their blood. Blood glucose levels are high either because their
                           cells are resistant to insulin (a hormone that helps convert glucose into energy) or
                           because their pancreas does not produce enough insulin. Insulin analogues are used
                           approximately as commonly as human insulin by diabetics who require insulin to regulate
                           blood glucose levels. Created by genetically modifying human proteins, insulin analogues
                           were developed as an alternative to human insulin to provide tighter control of blood
                           sugar levels.
Key findings               This study summarizes the effectiveness of insulin analogues against traditional human
                           insulin for type 2 diabetics. Researchers compared the effectiveness of three kinds of
                           synthetic insulin against their human insulin counterparts, against each other, and against
                           other antidiabetic medications. The report found that insulin analogues are more effective
                           than human insulin for treating certain diabetes-related symptoms such as high blood
                           sugar after meals. However, it also found that human insulin appears to be more effective
                           than insulin analogues in treating other aspects of diabetes, including lowering blood
                           sugar levels when patients go 8 hours or more without eating, typically overnight.
Products and formats       •    Press release                       •   Consumer guide (for adults)
                           •    Comparative effectiveness           •   Clinician’s guide
                                report                              •   Webcast and slides (for clinicians)
                           •    Executive summary                   •   Audio podcast (for consumers)
Key audiences              Over 90 target organizations, publications, and electronic venues are identified in the
                           marketing plan in the following categories: clinicians, insurers, payers, pharmacy and
                           drugs associations, federal direct and funded medical care programs, consumer-oriented
                           disease organizations, and government. Provider categories targeted include retail and
                           health system pharmacists; family physicians and general internists; pharmacologists;
                           nurse practitioners; physician assistants; and endocrinologists.
Media outreach             General media news services including radio, television news, and major daily
                           newspapers; consumer and advocacy publications; African-American media; and
                           translation to Spanish-only and Hispanic media.
Other electronic targets   Medscape, WebMD, ModernMedicine.com, ePocrates
AHRQ publications          Electronic Newsletter, Effective Health Care Listserv, Research Activities newsletter,
                           Website Spotlight
                           Source: AHRQ.




                           Page 37                              GAO-12-332 AHRQ Comparative Effectiveness Research
Appendix VI: GAO Contact and Staff
                  Appendix VI: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Linda T. Kohn, (202) 512-7114 or kohnl@gao.gov
GAO Contact
                  In addition to the contact named above, E. Anne Laffoon, Assistant
Staff             Director; Shaunessye Curry; Mary Giffin; Andrea E. Richardson;
Acknowledgments   Lisa Motley; Krister Friday; and Jessica C. Smith made key contributions
                  to this report.




(290922)
                  Page 38                              GAO-12-332 AHRQ Comparative Effectiveness Research
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.