National Institutes of Health: Problems in Implementing Policy on Women in Study Populations

Published by the Government Accountability Office on 1990-07-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting   Office

on Delivery          Problems in Implementing    Policy
Expected at          on Women in Study Populations
1O:OO a.m. EDT
July 24, 1990

                     Statement of
                     Mark V. Nadel, Associate Director
                     National  and Public Health Issues
                     Human Resources Division
                     Before the
                     Subcommittee on Housing and
                       Consumer Interest
                     Select Committee on Aging
                     House of Representatives

                     c34-6          / \4\4!a
                                                              GAO   Form160(12/87)

The National        Institutes      of Health   (NIH) has made little      progress
in implementing          its policy    to encourage the inclusion        of women in
research     study populations.          Although     the policy   first  was
announced in October 1986, guidance                for implementation     was not
published     until      July 1989, and the policy        was not applied
consistently        before the 1990 grant review cycles.
  --    The policy    on women has not been well communicated          or
        understood    within    NIH or in the research      community.    For
        example, the grant application          booklet   has not been revised
        to instruct     applicants     about the policy    on women. As a
        result,    NIH still     is receiving   proposals   that are not
        responsive    to the policy.
   --   We found inconsistencies           in how the policy       has been applied
        in a key stage of the grant review process.                   The Division   of
        Research Grants,        which handles most grant applications,
        instructs     reviewers      not to consider      the inclusion      of women
        as a factor      of scientific       merit in the initial        evaluation   of .
        grant applications.           In contrast,     the National      Heart, Lung,
        and Blood Institute          and the Alcohol,       Drug Abuse, and Mental
        Health Administration,          another    Public    Health Service
        agency, instruct        their   reviewers    to consider      study
        population      composition     as part of scientific         merit in the
        initial    review.
   --   NIH's policy      on women applies   only to extramural   research.
        The smaller     intramural  research    program has no policy.
   -- Although    the original   policy    announcement encouraged
      researchers     to analyze    study results   by gender, NIH
      officials     have taken little     action  to implement  this           element
      of the policy.
   -- Because implementation         of the policy    began so late,    we could
      not determine       its effect    on the demographic   composition    of
      study populations.         Furthermore,    there is no readily
      accessible      source of data on the demographics        of NIH study
      populations,      either   from the NIH Director's     Office   or from
      the institutes.
Madam Chairman               and Members of the                  Subcommittee:

I am pleased        to be here              today     to discuss             our review           of the
progress      the National               Institutes          of Health          (NIH)     has made in
implementing        its        policy       to encourage            the      inclusion        of women in
study     populations           and what effect                  the policy       has had on the study
populations        of NIH-funded                 research.           You asked us to provide
information        based on testimony                     we presented           before       the
Suhcomittee        on Health             and the Environment                  of the House Committee
on Energy       and Commerce,               and to provide              additional           information            on
the National        Institute             on Aging         and the percentage                 of women in
senior      positions          at NIH.           We reviewed          four      institutes          in depth
and obtained        more limited                 information          from      nine      other     institutes
arrci on<: center.l

in brief,       we found          that      NIH has not adequately                       implemented         its
policy.        Although          NIH announced             its      policy      over      3 years      ago,        it
has just       begun to apply               it    systematically              during       the grant         review
process.        NIH's         various        institutes            have not consistently                   applied
the policy,        and NIH has no way to measure                               the policy's           impact            on
the research            it    funds.         Furthermore,            the policy           applies      to

IThe four institutes        were the National        Cancer Institute;       National
Heart, Lung, and Blood Institute:           National      Institute    of Allergy     and
Infectious     Diseases:    and National    Institute       on Aging.     We also
obtained     some information     from the Alcohol,         Drug Abuse, and Mental
Health Administration         (ADAMHA), another agency of the Public Health
Service,     on its implementation       of policies     concerning      study
populations.                                       .

extramural            research           only,      and not          to NIH's      own intramural
research        projects.


NIH,     which        is part          of the       Public       Health        Service      (PHS),     is the
principal            federal          agency       supporting          biomedical          research.          It         has
a 1990 budget              of     $7.6      billion.            The total        percentage          of women in
senior        policy       and research                positions         at NIH is         31 percent;             for
the SES only,              that        figure        is   14 percent.

The 1985 Report                  of the          Public       Health     Service         Task Force      on
h'omen's       Health           Issues      recommended              increased      research         on health
problems            affecting          women.           In response,           NIH promulgated           a policy
to ~r~sur-e that                women are          included          in study      populations         unless             it
would        be scientifically                    inappropriate           to do so.           NIH has funded
some projects              that        studied         only     men, even though              the diseases
being        researched           affect          both    men and women.                 According     to NIH,
the underrepresentation                           of women in such studies                    "has resulted                    in
significant             gaps in knowledge."                        In studies        of some diseases                    and
treatments,             excluding           women raises               serious     questions         about
whether         the research               results        can be applied             to women.

An example             of the problem                  is a National           Heart,      Lung,and          Blood
 Institute           study       of    22,000          male physicians             begun in 1981.              It
 found       that      men who took               an aspirin           every     other     day reduced              their

incidence         of heart             attacks.          Institute             officials        told       us women
were not         included         in this          study,       because           to do so would             have

increased         the     cost.           However,        we now have               the      dilemma       of not
knowing        whether       this         preventive           strategy           would       help     women, harm
them.        or have       no effect.

Another        example        is the          National         Institute            on Aging's          Baltimore
Longitudinal              Study        of Aging.             This     community-based                study      of the
effects         of aging          was begun in 1958 with                         an all-male           study
population.               Women        were added to the                study          in 1978,        but because
the study          included            only     men    for     the     first        twenty       years,        less
information             on the aging              process           in women is available                    for

Following          publication                of the     1985 Public               Health       Service        Task
Force        report,       the NIH Director                   established              the NIH Advisory
Committee          on Women's Health                   Issues         to monitor             implementation             of
the Task Force's                  recommendations                   in NIH.           The committee's              work
led     to     a policy         that      was first           announced            in October          1986 and
restated          in a January                1987 announcement.2                      The 1987 announcement

          --    urged      grant         applicants           to consider              the     inclusion           of women
             in the     study       populations              of all        clinical          research        efforts:

2The announcement   first appeared in the NIH Guide fc >r Grants and
Contracts  of October 24, 1986.    The second announcer nent appeared
in the January 23, 1987 issue of the NIH Guide, and was co:
sponsored by ADAMHA.
         --     stated         that       if     women were not              to be included,                    applicants
         should         provide          a clear           rationale         for        their       exclusion;           and

         --     said     that         researchers             should       note         and evaluate                gender

The 1987 policy                  announcement               urged        rather         than      required           attention
to these          issues.

To understand               NIH's         implementation                 of the policy,                 it     may be
useful         to digress              briefly           and describe             the organization                   of NIH
and the process                  it     uses to award research                          grants.              NIH consists
of 13 research                  institutes           and several             other            components.3

Applications              for         NIH research            grants        are received                by the Division
of Research             Grants,           and go through                 a dual         review        process.           The
first         level      of review               takes     place        either         in the Division                or in an
institute.               A group          of outside            experts           evaluates           the      scientific
and technical               merit         of each proposal.                       If        the   scientific           review
group         recommends              approval           of a proposal,                it     assigns         the
application              a numerical               priority         score.             This       score        is the most
important             factor          in NIH's           ultimate        decision             to fund         a proposed

3Each institute      conducts    laboratory and clinical     research
through an intramural        program and supports    other research
organizations     through an extramural     program of grants and
contracts.      In fiscal    year 1988, extramural     awards represented                                                        84
percent of the NIH budget.
study.          For each application,                  the group's         executive           secretary--
who is an NIH staff                   member assigned           to each review                panel--
prepares         a summary statement               with       reviewers'          comments and

An application              approved        by the      scientific         review        group     receives              a
second         level      of review        by the advisory             council       of the appropriate
institute.              After     evaluating          the proposal's             scientific         merit          and
program         relevance,         the council          makes a funding              recommendation.
The institute              director        makes the       final       decision        on whether           to
fund     proposals.             About      one-third       of the recommended                   proposals           are

NIH !'4ADE LITTLE               PROGRESS

The Office              of the NIH Director             has depended             more on persuasion                  of
NIH staff              and outside       scientists         than      on central         direction            to
take     action.           At the       time    we began our work                in January          1990,         NIH
had made little                 progress       in carrying           out   its     1987 policy           on
women.           Although        some steps        have been taken               since        January,
several          problems        have characterized                implementation:

          --     It     has been very          slow;

         --     The policy             has not been well                communicated              or understood
         within          NIH and in the              scientific              research        community,             and has
         been applied                 inconsistently            among NIH components;

          --     Encouragement                  of gender      analysis,            a key part             of the
          policy,            has not       been implemented:                  and

          --     It     is     impossible          to determine              the    impact       of the policy.

I will         discuss            each of these           problems           in turn.


Most of the responsibility                            for     policy         implementation                was left            to
the      individual               institutes,        which        have responded               with        varying
degrees          of effort            and speed.             After      publication            of the policy                   in
1986 and 1987,                    some institutes             began to inform                 their        staff         and
researchers                  about     the policy           and some incorporated                     it    in their
grant          review         process.           Others      waited      for       further       guidance.
Because          of the differences                    in implementation                   among the          institutes
and the          lack         of records,          we cannot           describe           precisely         the      timing
of each institute's                       actions.           But of the            four      institutes             we
reviewed              in depth,          two began to apply                  the    policy       before            NIH
Provided              additional          instructions            and two began afterwards.                               The
Kational              Institute          on Aging         began to implement                  the policy             in
It    took      NIH almost          3 years         to issue        detailed         implementation
guidelines          to its       staff.        A comprehensive                    memorandum applying                     to
all      extramural        research          did     not appear           until      July       1989.*           That
memorandum strengthened                      implemention             of the policy               to include
minorities          in studies,            as well          as providing            guidelines             for      the
policy         on women.s           The 1989 memorandum sets                        out     the        following

          --    NIH solicitations                  for     research       applications                 should       urge
          the     inclusion         of women and minorities                        in study            populations
          and require           applicants               to provide       a rationale             if      they      are

          --     Executive       secretaries               of scientific            review        groups            are to
          ensure      that      reviewers            address        the application's
          responsiveness             to the policy                 and indicate           in their           summary
           statements         reviewers'             recommendations               on this        issue.

The Division            of Research            Grants         is responsible              for     the       first
 level         of review      for    most proposals                 received         by NIH.             In the
 Division,         scientific             reviewers          did    not    begin      to apply            the policy

 OAn earlier           memorandum in November 1987 provided    limited
 instructions,           but it applied only to contracts,    a small
 proportion           of the funds NIH awards to researchers.
 sThe September 25, 1987 NIH Guide for Grants                                         and Contracts
 anrlounced a policy encouraging the inclusion                                        of minorities                  in
 stuciy populations.
until      the February                1990 grant          review         cycle.           Three    of the four
institutes           we reviewed             in depth            including          The National            Institute
on Aging,           began to apply                the policy             by fall          1989,    but     in the
National           Institute           of Allergy          and Infectious                  Diseases,        reviewers
will      first       implement          the policy              this      month.          Because       of these
delays,           many scientific                review      groups         are     just     beginning          to send
to institute             councils           summary statements                     that     highlight         concerns
about       the exclusion               of women from                 studies.


We found           problems        in the extent                 to which          the policy            is understood
and applied             by grant          applicants,              NIH staff,             and scientific
experts           who review           proposals           for     NIH funding.

The application                  booklet         used by most NIH grant                       applicants--PHS
 Form 398 --contains                   no reference               to the policy              to include         women in
 study       populations.               This       form     is a primary              source       of instructions
 to investigators                 initiating           their            own proposals.             A revised
 version          of the        form     and its          instructions             will      not   appear       until
 April       1991,      over      4 years          after         the policy          was first           articulated.

 As a result,             NIH     is     still      receiving             many proposals             that      are not
 responsive            to the       policy.           We reviewed                about      50 recent        grant
 applications,                 most proposing              studies         on conditions             that      affect

both      men and women.                   About         twenty        percent         of the proposals
provided           no information                 on the         sex of the            study       population.                 Over
one-third           indicated             that        both      sexes would            be included               but did         not
say in what proportions.                               Some proposals                 for   all-male             studies
provided           no rationale                 for     that     design.

We found           that      some NIH staff                    were unaware            of their
responsibilities                   for         implementing              the policy.              In addition,             some
reviewers           demonstrated                 limited         understanding               of the policy.                    For
example,           a recent         proposal             to conduct           an all-male                study       related
to coronary               artery         disease         was approved               by the        scientific            review
group       with      the     comment that                    the exclusion            of females              was
appropriate               because         the disease              studied          disproportionately
affects        men.          While        this        observation            may be true,                it    may be
inadequate            as a rationale                    for     excluding           women, because                  coronary
artery        disease         is also            a serious           health         problem            in women.           The
instjtute           council             also     approved          this      proposal            for     funding,

During        a key stage                of the         review       process,           the policy             on women is
applied        inconsistently.                        The Division                of Research            Grants         and some
institutes,               including             the National              Institute          on Aging,              instruct
members of scientific                           review         groups      not      to consider               the    inclusion
of women and minorities                               in the      study      population                as a factor          of
scientific            and technical                    merit      that      would       affect          the priority
score.         Instead,            if      the review            group       raises         a problem            with      the
composition               of the         study         population,           it     should        be addressed                 in an

administrative               note      in the       summary statement.                       These
administrative               notes      are used to highlight                           matters         that        do not
pertain        directly         to scientific              merit,             such as care              of
experimental              animals,          The institute                council           may then            take    that
issue       into     account          in reaching           its        recommendation.

Officials           of the Division                of Research                Grants       and these            institutes
told      us that          in practice           there     may be exceptions                       to this           review
policy.            Reviewers          can include           the        study      population             as a
criterion           for     the priority            score         if    it     is clear          that        the proposed
population           would      make it          impossible             to answer           the scientific
question           posed by the            investigator.                     In addition,           the        study
population           will      affect       the priority                score       if     an application                   is
responding           to an institute                solicitation                 that      specifies            inclusion
of women as a review                     factor.

In contrast               to this       review      policy,            National           Heart,        Lung,         and
Blood       Institute          officials           told     us that             their      reviewers            consider
adequate           inclusion          of women and minorities                            an element            of
scientific           merit      and factor            it    into        the priority               score.           Likewise,
ADAMHA instructs                its      reviewers          to evaluate                  plans     for       inclusion           of
women as part               of their        overall         evaluation              of the technical                    merit
of applications.


NIH's     intramural           research      program            has no policy              on the          inclusion
of women in study               populations.              In an August                  1989 report,               the
Advisory          Committee         on Women's Health                  Issues       recommended                 that       NIH
take     steps      to encourage           inclusion            of women in intramural                           as well
as extramural            studies.          The Director                of NIH has not               formally
transmitted          that      report      to intramural                officials           or instructed                   them

to develop          a policy.           In response             to our review,               the     Human
Research          Review      Panel     of the NIH Clinical                      Center      placed             this       issue
on the      agenda of its             June meeting.

The National            Institute         on Aging         (NIA)        provides          a good example                    of
the     problems       that     can arise          from        the     lack      of emphasis              on including
women in NIH's              intramural         research              program.           The Baltimore
Longitudinal           Study        of Aging       is part            of NIA's          intramural              program.
Its     failure      to recruit           women as study                    subjects      during          its      first
twenty      years      has resulted            in some research                   results          that         can be
appliced          to men only.            Research        supported              by other          components               of
NIA's      intramural          program      also       has generated                   more information                    on men
than     on women.


Although            the     1987 policy              announcement                also     encouraged           researchers
to analyze             study        results          by gender,               NIH officials            have taken
little        action            to implement               this      element        of the policy.                The 1989
memorandum setting                        out     guidelines             for     policy       implementation              calls
for       attention             to issues           of research                design     and sample            size,     but
does not            specify          the     need for             gender        analysis.           NIH officials
showed us solicitations                             that      cited        the     importance              of including
women in study                   populations.                 We noted,            however,          that     few suggested
studies            be designed              to assess             different         results          for     men and women.
NIH officials                   differ       among themselves                     in their         views      on the      types
of       studies          for    which          gender      analysis             is appropriate.


---            OF POLICY

You asked us to report                             on the extent                 to which          the NIH policy           has
resulted            in inclusion                  of women in clinical                      study     populations.
Because            policy         implementation                  began so late,              it     is too       soon to
determine             what,        if      any,     effect          it   is      having      on the demographic
composition                 of study            populations.               Additionally,              given       the     lack    of
data        on previous                 study      populations,                analysis       of the policy's
impact         is virtually                 impossible.

Steps       could         be taken,         however,           to maintain             data      that      would     be
useful          for     future       monitoring          of the          inclusion          of women in studies.
At present,               no central           NIH office             collects         the types           of
demographic               data     on study          populations              that     you requested.                Several
years       ago,        NIH revived            its     Inventory             of Clinical          Trials         and the
current          data      collection           form         does ask for             information           about        the
gender          composition            planned         for     study         populations.               However,         the
gender          question           is not      categorized              specifically             enough to provide
complete              information.             As another             means of monitoring                    inclusion         of
women in study                   populations,           some         institutes         plan       to begin
collecting              demographic            data     on studies             they     fund.


To ensure              effective         implementation                 of    its     policy       to encourage            the
inclusion              of women in study                populations,                 the Director               of NIH
should          take      the      following          steps:

           --     Inform         NIH staff,           grant      reviewers,            and the           community        of
           researchers              NIH supports              of the reasons               for     the     policy        and
           how it         should       be carried             out;

           --     Direct         NIH institutes               to maintain             readily           accessible        data
           to allow          assessment              of the      extent           to which       women are included
           in studies;

        --     Ensure      that      the planned       revision       of the grant        application
        booklet          (PHS Form 398) adds a section                  explaining        the policy
        and instructing               applicants       to respond        to the requirement              to
        include        women in study            populations,         or to justify         their
        exclusion;           and

        --Instruct           members of review             groups     always     to determine
        whether          the gender       of the      study     population       is an issue        of
        scientific           merit     affecting       the priority          score,     and to
        document          their      decisions      in the      summary statements.

Following         our original           testimony,        the Acting        Director     of NIH said
he would        give      serious      consideration          to these       recommendations,            and
by law,        federal       agencies      have 60 days to notify                Congress      on actions
taken        in response          to GAO recommendations.

This    concludes          my statement,           Madam Chairman.             I would    be happy to
answer       any questions            you may have.