National Institutes of Health: Problems in Implementing Policy on Women in Study Populations

Published by the Government Accountability Office on 1990-06-18.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Atiounting   OfBlce

on Delivery        Problems in Implementing Policy
Expected at        on Women in     Study   Populations
1O:OO a.m. EDT
June 18, 1990

                   Statement of
                   Mark V. Nadel, Associate   Director
                   National  and Public Health Issues
                   Human Resources Division
                   Before the
                   Subcommittee on Health and
                     the Environment
                   Committee on Energy and Commerce
                   House of Representatives

                                                              GAO Form 160 (12/m

The National       Institutes     of Health (NIH) has made little     progress
in implementing         its policy   to encourage the inclusion     of women in
research study populations.            Although the policy    first  was
announced in October 1986, guidance for implementation               was not
published    until      July 1989, and the policy    was not applied
consistently       before the 1990 grant review      cycles.

  -- The policy    on women has not been well communicated or
     understood    within    NIH or in the research community.      For
     example, the grant application         booklet has not been revised
     to instruct     applicants    about the policy   on women. As a
     result,    NIH still    is receiving   proposals  that are not
     responsive    to the policy.
  -- We found inconsistencies        in how the policy     has been applied
     in a key stage of the grant review process.              The Division     of
     Research Grants, which handles most grant applications,
     instructs     reviewers   not to consider     the inclusion     of women as
     a factor    of scientific     merit in the initial      evaluation    of
     grant applications.         In contrast,   the National     Heart, Lung,
     and Blood Institute       and the Alcohol,      Drug Abuse, and Mental
     Health Administration,        another public Health Service agency,
     instruct    their   reviewers   to consider     study population
     composition      as part of scientific     merit in the initial       review.
  -- NIH's policy       on women applies  only to extramural research.
     The smaller      intramural  research program has no policy.
  -- Although the original      policy announcement encouraged
     researchers   to analyze study results     by gender, NIH officials
     have taken little    action to implement this element of the
  --    Because implementation         of the policy    began so late,  we could
        not determine       its effect   on the demographic composition      of
        study populations.         Furthermore,    there is no readily
        accessible      source of data on the demographics        of NIH study
        populations,      either   from the NIH Director’s     Office or from
        the institutes.

Mr. Chairman           and Members of the Subcommittee:

I am pleased           to be here today               to discuss          our review      of the
progress      the National               Institutes        of Health        (NIH) has made in
implementing           its     policy      to encourage           the inclusion         of women in
study     populations.             You asked us to report                   on the steps          NIH has
taken     to carry           out this      policy        and what effect            the policy         has had
on the study           populations           of NIH-funded           research.         We reviewed                 four
institutes       in depth          and obtained            more limited          information           from nine
other     institutes           and one center.I

In brief,       we found          that      NIH has not adequately                  implemented         its
policy.       Although          NIH announced             its    policy     over     3 years      ago,        it
has just      begun to apply                it    systematically           during     the grant         review
process.        NIH's         various       institutes          have not consistently              applied
the policy,        and NIH has no way to measure the policy's                                    impact            on
the research           it     funds.        Furthermore,          the policy         applies      to
extramural       research          only,         and not to NIH's           own intramural             research

lThe four institutes        were the National       Cancer Institute;      National
Heart, Lung, and Blood Institute:           National    Institute     of Allergy    and
Infectious     Diseases;    and National    Institute     on Aging.     We also
obtained     some information     from the Alcohol,       Drug Abuse, and Mental
Health Administration         (ADAMHA), another agency of the Public Health
Service,     on its implementation       of policies    concerning     study

NIH, which       is part         of the Public             Health          Service       (PHS),      is the
principal       federal      agency supporting                    biomedical            research.          It     has a
1990 budget       of $7.6 billion.                  The 1985 Report                   of the Public              Health
Service      Task Force          on Women's Health.Issues                          recommended increased
research       on health         problems         affecting             women.         In response,             NIH
promulgated       a policy             to ensure         that     women are included                  in study
populations       unless         it     would be scientifically                        inappropriate             to do
so.      NIH has funded               some projects             that     studied        only      men, even
though      the diseases              being     researched             affect      both men and women.
According       to NIH, the underrepresentation                                  of women in such studies
"has resulted           in significant              gaps in knowledge."                        In studies         of
some diseases           and treatments,              excluding             women raises            serious
questions       about whether                 the research             results        can be applied             to

An example       of the problem                 is a National              Heart,       Lung,      and Blood
Institute       study     of 22,000             male physicians                  begun in 1981.            It     found
that     men who took        an aspirin             every        other          day reduced        their
incidence       of heart         attacks.           Institute             officials        told     us women
were not included            in this            study,      because to do so would have
increased       the cost.              However,      we now have the dilemma                        of not
knowing       whether     this         preventive          strategy             would help        women, harm
them,       or have no effect.

Following        publication              of the 1985 Public            Health       Service      Task Force
report,        the NIH Director              established        the NIH Advisory               Committee       on
Women's Health              Issues        to monitor     implementation              of the Task Force's
recommendations                 in NIH.      The committee's            work led to a policy                that
was first          announced         in October        1986 and restated              in a January          1987'
announcement.2                  The 1987 announcement

          -- urged        grant      applicants        to consider          the inclusion          of women
          in the study             populations      of all      clinical           research     efforts;

          --   stated       that     if    women were not to be included,                      applicants
          should       provide       a clear      rationale       for      their     exclusion:       and

          --   said      that      researchers      should      note and evaluate               gender

The 1987 policy                 announcement       urged      rather       than     required      attention
to these        issues.

To understand             NIH's       implementation          of the policy,            it    may be useful
to digress            briefly       and describe        the organization              of NIH and the

2The announcement first  appeared in the WIH Guide for Grants and
Contracts  of October 24, 1986.  The second announcement appeared
in the January 23, 1987 issue of the NIH Guide, and was co-
sponsored by ADAM?IA.
process       it     uses to award research                        grants.        NIH consists             of 13
research           institutes              and several         other     components.3

Applications             for         NIH research           grants      are received            by the Division
of Research            Grants,             and go through            a dual      review        process.           (See
chart,       NIH Grant               Review Process.)                The first         level      of review         takes
place      either        in the Division                   or in an institute.                  A group       of
outside       experts               evaluates          the scientific          and technical            merit       of
each proposal.                      If    the scientific           review      group      recommends approval
of a propasal,                 it        assigns       the application           a numerical           priority
score.        This       score            is the most important                factor      in NIH's         ultimate
decision           to fund a proposed                    study.        For each application,                  the
group's       executive                  secretary--       who is an RIH staff                 member assigned                to
each review            panel--prepares                   a summary statement               with     reviewers'
comments and recommendations.

An application                 approved            by the scientific             review        group      receives        a
second level             of review              by,the      advisory         council      of the appropriate
institute.             After             evaluating        the proposal's          scientific             merit     and
program       relevance,                  the council        makes a funding              recommendation.
The institute              director             makes the final              decision      on whether             to fund
proposals.             About one-third                   of the recommended proposals                       are

3Each institute      conducts laboratory   and clinical    research
through an intramural       program and supports other research
organizations     through an extramural    program of grants and
contracts.      In fiscal   year 1988, extramural    awards represented                                                  84
percent of the NIH budget.

The Office             of the NIH Director            has depended more on persuasion                    of
NIH staff             and outside      scientists       than       on central         direction      to take
action.          At the time         we began our work in January                      1990, NIH had
made little             progress      in carrying          out its     1987 policy         on women.
Although             some steps     have been taken            since       January,      several     problems
have characterized                 implementation:

          --    It     has been very       slow:

          -- The policy            has not been well            communicated           or understood
          within        NIH and in the scientific                  research      community,        and has
          been applied            inconsistently           among NIH components:

          --    Encouragement          of gender       analysis,           a key part       of the
          policy,        has not been implemented;                   and

          --    It     is impossible       to determine            the impact         of the policy.

I will         discuss      each of these           problems       in turn.


Most of the responsibility                    for     policy      implementation                  was left            to
the     individual       institutes,          which     have responded                with        varying
degrees      of effort        and speed.            After       publication           of the policy                   in
1986 and 1987,           some institutes              began to inform                their        staff         and
researchers           about   the policy            and some incorporated                    it     in their
grant      review      process.        Others        waited      for      further        guidance.
Because of the differences                     in implementation                  among the           institutes
and the lack           of records,          we cannot          describe          precisely          the timing
of each institute's                actions.          But of the four                institutes             we
reviewed        in depth,      two began to apply                  the policy            before           NIH
provided        additional         instructions          and two began afterwards.

It    took    NIH almost          3 years      to issue         detailed          implementation
guidelines           to its   staff.          A comprehensive                  memorandum applying                    to
all     extramural        research      did not appear                 until      July       1989.4         That
memorandum strengthened                  implemention            of the policy                to include
minorities           in studies,       as well        as providing               guidelines           for       the
policy       on women.5           The 1989 memorandum sets                       out the          following

4An earlier           memorandum in November 1987 provided    limited
instructions,           but it applied only to contracts,    a small
proportion           of the funds NIH awards to researchers.
'The September 25, 1987 NIH Guide for Grants                                      and Contracts
announced a policy encouraging  the inclusion                                     of minorities                 in
study populations.
        -- NIH solicitations                      for     research      applications           should      urge
        the     inclusion              of women and minorities                 in study        populations
        and require              applicants             to provide      a rationale           -if they     are

        --     Executive              secretaries         of scientiiic          review       groups      are to
        ensure         that      reviewers          address        the application's
        responsiveness                  to the policy          and indicate           in their         summary
        statements             reviewers'           recommendations            on this        issue.

The Division             of Research             Grants      is responsible           for     the first       level
of review         for     most proposals                 received      by NIH.        In the Division,
scientific         reviewers              did not begin            to apply      the policy        until      the
February         1990 grant             review      cycle.         Three of the four             institutes            we
reviewed         in depth             began to apply          the policy         by fall       1989, but          in
the National             Institute           of Allergy        and Infectious               Diseases,
reviewers         will        first       implement        the policy         this    month.        Because of
these        delays,      many scientific                 review     groups      are just        beginning         to
send to institute                     councils      summary statements               that     highlight
concerns         about        the exclusion              of women from studies.

We found problems               in the extent                  to which       the policy          is understood
and applied         by grant           applicants,              NIH staff,           and scientific              experts
who review         proposals           for     NIH funding.

The application               booklet        used by most NIH grant                      applicants--PHS
Form 398 --contains               no reference                 to the policy           to include          women in
study     populations.                This     form'is           a primary        source     of instructions
to investigators               initiating              their        own proposals.           A revised
version     of the        form and its                 instructions           will     not appear          until
Apri1.1991,            over    4 years         after       the policy          was first          articulated.

As a result,            NIH is still             receiving            many proposals             that     are not
responsive         to the policy.                 We reviewed             about       50 recent          grant
applications,            most proposing                 studies        on conditions             that     affect
both men and women.                     About twenty                percent     of the proposals
provided        no information                on the sex of the study                      population.             Over
one-third        indicated            that     both       sexes would be included                       but did      not
say in what proportions.                         Some proposals               for     all-male          studies
provided        no rationale             for     that      design.

We found        that     some NIH staff                 were unaware            of their
responsibilities                for     implementing                the policy.          In addition,              some
reviewers        demonstrated                limited           understanding          of the policy.                For

example,        a recent          proposal          to conduct          an all-male                study        related        to
coronary        artery         disease       was approved             by the scientific                        review
group with            the comment that                  the exclusion              of females            was
appropriate            because         the disease           studied         disproportionately                      affects
men.       While       this      observation             may be true,              it        may be inadequate               as a
rationale          for       excluding.women,               because         coronary            artery         disease       is
also      a serious           health      problem         in women.           The institute                    council       also
approved        this         proposal      for      funding.

During       a key stage           of the review               process,            the policy             on women is
applied        inconsistently.                   The Division              of Research             Grants         and some
institutes            instruct          members of scientific                      review         groups         not to
consider        the       inclusion        of women and minorities                             in the study
population            as a factor          of scientific               and technical                   merit      that
would       affect        the priority             score.          Instead,             if    the review          group
raises       a problem           with     the composition                  of the study                population,           it
should       be addressed               in an administrative                      note        in the summary
statement.               These administrative                  notes        are used to highlight
matters        that       do not pertain                directly       to scientific                   merit,       such as
care of experimental                     animals.           The institute                    council      may then take
that      issue       into     account           in reaching         its      recommendation.

Officials            of the Division               of Research             Grants            and these          institutes
told      us that         in practice             there     may be exceptions                     to this          review
policy.           Reviewers         can include             the study             population             as a criterion
for    the priority              score      if     it     is clear         that         the proposed             population

would make it                impossible             to answer the scientific                         question          posed by
the investigator.                     In addition,              the study            population          will      affect
the priority                score     if      an application              is responding               to an institute
solicitation                that     specifies         inclusion           of women as a review                        factor.

In contrast                to this         review     policy,         National             Heart,     Lung,      and Blood
Institute        officials                 told     us that      their          reviewers           consider       adequate
inclusion         of women and minorities                            an element             of scientific              merit
and factor            it     into     the priority              score.          Likewise,           ADAMHA instructs
its     reviewers            to evaluate             plans      for      inclusion           of women as part                  of
their       overall          evaluation             of the technical                 merit      of applications.


NIH's       intramural              research         program          has no policy             on the inclusion
of women in study                    populations.               In an August                1989 report,           the
Advisory         Committee            on Women's Health                   Issues           recommended that               NIH
take     steps        to encourage                inclusion           of women in intramural                     as well
as extramural                 studies.            The Director            of NIH has not                formally
transmitted                that     report        to intramural                officials        or instructed               them
to develop            a policy.               In response             to our review,                the Human
Research         Review Panel                 of the NIH Clinical                    Center         placed      this      issue
on the agenda of its                        June meeting.


             ACTION w

Although        the     1987 policy               announcement               also     encouraged          researchers
to analyze            study      results          by gender,              NIH officials            have taken
little       action         to implement               this     element         of the policy.               The 1989
memorandum setting                     out guidelines                for     policy       implementation           calls
for      attention          to issues           of research               design      and sample size,             but
does not specify                 the need for                 gender        analysis.            NIH officials
showed us solicitations                         that      cited        the importance              of including
women in study                populations.                We noted,           however,           that     few suggested
studies       be designed               to assess             different         results          for    men and women.
NIH officials                differ       among themselves                   in their          views      on the types
of studies            for     which       gender        analysis            is appropriate.

                      TO DETERMINE

You asked us to report                         on the extent                to which           the NIH policy       has
resulted        in inclusion                  of women in clinical                    study       populations.
Because policy                 implementation                 began so late,              it     is too soon to
determine            what,      if     any,      effect        it    is having          on the demographic
composition            of study              populations.              Additionally,              given     the   lack     of
data      on previous                study     populations,                analysis       of the policy's
impact       is virtually                impossible.

Steps could             be taken,         however,              to maintain            data     that      would be
useful         for     future       monitoring             of the inclusion                   of women in studies.
At present,              no central           NIH office              collects         the types          of
demographic              data     on study           populations            that       you requested.              Several
years      ago, NIH revived                   its        Inventory        of Clinical            Trials        and the
current         data      collection           form does              ask for       information            about    the
gender         composition           planned             for    study     populations.                 However,     the
gender         question          is not categorized                     specifically            enough to provide
complete             information.             As another              means of monitoring                  inclusion        of
women in study                 populations,               some institutes               plan     to begin
collecting             demographic            data        on studies          they      fund.

To ensure             effective        implementation                   of its      policy       to encourage           the
inclusion             of women in study                   populations,             the Director            of NIH
should         take      the      following          steps:

          --    Inform          NIH staff,           grant       reviewers,            and the community               of
          researchers              NIH supports                of the reasons             for    the policy         and
          how it         should       be carried               out;

          --    Direct          NIH institutes                 to maintain           readily       accessible          data
          to allow          assessment              of    the extent          to which          women are included
          in studies;

       --   Ensure    that      the planned            revision       of the grant      application
       booklet       (PHS    Form     398) adds a section               explaining      the policy
       and instructing              applicants         to respond       to the requirement            to
       include      women in study           populations,             or to justify      their
       exclusion:       and

       --Instruct       members of review                   groups    always    to determine
       whether      the gender          of the study           population       is an issue      of
       scientific       merit        affecting         the priority         score,    and to
       document      their      decisions         in the summary statements.

This   concludes      my statement,              Mr.    Chairman.         I would     be happy to
answer any questions                you may have.

GAO NIH Grant Review Process