United States General Accounting Office Testimony GAO For Release NATIONAL INSTITUTES OF HEALTH: on Delivery Problems in Implementing Policy Expected at on Women in Study Populations 1O:OO a.m. EDT Tuesday July 24, 1990 Statement of Mark V. Nadel, Associate Director National and Public Health Issues Human Resources Division Before the Subcommittee on Housing and Consumer Interest Select Committee on Aging House of Representatives GAO/T-HRD-90-50 c34-6 / \4\4!a GAO Form160(12/87) SUMMARYOF GAO TESTIMONY BY MARK V. NADEL ON PROBLEMS IN IMPLEMENTING THE NATIONAL INSTITUTES OF HEALTH POLICY ON WOMENIN STUDY POPULATIONS The National Institutes of Health (NIH) has made little progress in implementing its policy to encourage the inclusion of women in research study populations. Although the policy first was announced in October 1986, guidance for implementation was not published until July 1989, and the policy was not applied consistently before the 1990 grant review cycles. -- The policy on women has not been well communicated or understood within NIH or in the research community. For example, the grant application booklet has not been revised to instruct applicants about the policy on women. As a result, NIH still is receiving proposals that are not responsive to the policy. -- We found inconsistencies in how the policy has been applied in a key stage of the grant review process. The Division of Research Grants, which handles most grant applications, instructs reviewers not to consider the inclusion of women as a factor of scientific merit in the initial evaluation of . grant applications. In contrast, the National Heart, Lung, and Blood Institute and the Alcohol, Drug Abuse, and Mental Health Administration, another Public Health Service agency, instruct their reviewers to consider study population composition as part of scientific merit in the initial review. -- NIH's policy on women applies only to extramural research. The smaller intramural research program has no policy. -- Although the original policy announcement encouraged researchers to analyze study results by gender, NIH officials have taken little action to implement this element of the policy. -- Because implementation of the policy began so late, we could not determine its effect on the demographic composition of study populations. Furthermore, there is no readily accessible source of data on the demographics of NIH study populations, either from the NIH Director's Office or from the institutes. Madam Chairman and Members of the Subcommittee: I am pleased to be here today to discuss our review of the progress the National Institutes of Health (NIH) has made in implementing its policy to encourage the inclusion of women in study populations and what effect the policy has had on the study populations of NIH-funded research. You asked us to provide information based on testimony we presented before the Suhcomittee on Health and the Environment of the House Committee on Energy and Commerce, and to provide additional information on the National Institute on Aging and the percentage of women in senior positions at NIH. We reviewed four institutes in depth and obtained more limited information from nine other institutes arrci on<: center.l in brief, we found that NIH has not adequately implemented its policy. Although NIH announced its policy over 3 years ago, it has just begun to apply it systematically during the grant review process. NIH's various institutes have not consistently applied the policy, and NIH has no way to measure the policy's impact on the research it funds. Furthermore, the policy applies to IThe four institutes were the National Cancer Institute; National Heart, Lung, and Blood Institute: National Institute of Allergy and Infectious Diseases: and National Institute on Aging. We also obtained some information from the Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA), another agency of the Public Health Service, on its implementation of policies concerning study populations. . 1 extramural research only, and not to NIH's own intramural research projects. BACKGROUND NIH, which is part of the Public Health Service (PHS), is the principal federal agency supporting biomedical research. It has a 1990 budget of $7.6 billion. The total percentage of women in senior policy and research positions at NIH is 31 percent; for the SES only, that figure is 14 percent. The 1985 Report of the Public Health Service Task Force on h'omen's Health Issues recommended increased research on health problems affecting women. In response, NIH promulgated a policy to ~r~sur-e that women are included in study populations unless it would be scientifically inappropriate to do so. NIH has funded some projects that studied only men, even though the diseases being researched affect both men and women. According to NIH, the underrepresentation of women in such studies "has resulted in significant gaps in knowledge." In studies of some diseases and treatments, excluding women raises serious questions about whether the research results can be applied to women. An example of the problem is a National Heart, Lung,and Blood Institute study of 22,000 male physicians begun in 1981. It found that men who took an aspirin every other day reduced their 2 incidence of heart attacks. Institute officials told us women were not included in this study, because to do so would have increased the cost. However, we now have the dilemma of not knowing whether this preventive strategy would help women, harm them. or have no effect. Another example is the National Institute on Aging's Baltimore Longitudinal Study of Aging. This community-based study of the effects of aging was begun in 1958 with an all-male study population. Women were added to the study in 1978, but because the study included only men for the first twenty years, less information on the aging process in women is available for analysis. Following publication of the 1985 Public Health Service Task Force report, the NIH Director established the NIH Advisory Committee on Women's Health Issues to monitor implementation of the Task Force's recommendations in NIH. The committee's work led to a policy that was first announced in October 1986 and restated in a January 1987 announcement.2 The 1987 announcement -- urged grant applicants to consider the inclusion of women in the study populations of all clinical research efforts: 2The announcement first appeared in the NIH Guide fc >r Grants and Contracts of October 24, 1986. The second announcer nent appeared in the January 23, 1987 issue of the NIH Guide, and was co: sponsored by ADAMHA. 3 -- stated that if women were not to be included, applicants should provide a clear rationale for their exclusion; and -- said that researchers should note and evaluate gender differences. The 1987 policy announcement urged rather than required attention to these issues. To understand NIH's implementation of the policy, it may be useful to digress briefly and describe the organization of NIH and the process it uses to award research grants. NIH consists of 13 research institutes and several other components.3 Applications for NIH research grants are received by the Division of Research Grants, and go through a dual review process. The first level of review takes place either in the Division or in an institute. A group of outside experts evaluates the scientific and technical merit of each proposal. If the scientific review group recommends approval of a proposal, it assigns the application a numerical priority score. This score is the most important factor in NIH's ultimate decision to fund a proposed 3Each institute conducts laboratory and clinical research through an intramural program and supports other research organizations through an extramural program of grants and contracts. In fiscal year 1988, extramural awards represented 84 percent of the NIH budget. 4 study. For each application, the group's executive secretary-- who is an NIH staff member assigned to each review panel-- prepares a summary statement with reviewers' comments and recommendations. An application approved by the scientific review group receives a second level of review by the advisory council of the appropriate institute. After evaluating the proposal's scientific merit and program relevance, the council makes a funding recommendation. The institute director makes the final decision on whether to fund proposals. About one-third of the recommended proposals are funded. NIH !'4ADE LITTLE PROGRESS Ik IMPLEMENTING POLICY The Office of the NIH Director has depended more on persuasion of NIH staff and outside scientists than on central direction to take action. At the time we began our work in January 1990, NIH had made little progress in carrying out its 1987 policy on women. Although some steps have been taken since January, several problems have characterized implementation: -- It has been very slow; 5 -- The policy has not been well communicated or understood within NIH and in the scientific research community, and has been applied inconsistently among NIH components; -- Encouragement of gender analysis, a key part of the policy, has not been implemented: and -- It is impossible to determine the impact of the policy. I will discuss each of these problems in turn. IMPLE?lENTATION VERY SLOW Most of the responsibility for policy implementation was left to the individual institutes, which have responded with varying degrees of effort and speed. After publication of the policy in 1986 and 1987, some institutes began to inform their staff and researchers about the policy and some incorporated it in their grant review process. Others waited for further guidance. Because of the differences in implementation among the institutes and the lack of records, we cannot describe precisely the timing of each institute's actions. But of the four institutes we reviewed in depth, two began to apply the policy before NIH Provided additional instructions and two began afterwards. The Kational Institute on Aging began to implement the policy in 1987. . 6 It took NIH almost 3 years to issue detailed implementation guidelines to its staff. A comprehensive memorandum applying to all extramural research did not appear until July 1989.* That memorandum strengthened implemention of the policy to include minorities in studies, as well as providing guidelines for the policy on women.s The 1989 memorandum sets out the following procedures: -- NIH solicitations for research applications should urge the inclusion of women and minorities in study populations and require applicants to provide a rationale if they are excluded. -- Executive secretaries of scientific review groups are to ensure that reviewers address the application's responsiveness to the policy and indicate in their summary statements reviewers' recommendations on this issue. The Division of Research Grants is responsible for the first level of review for most proposals received by NIH. In the Division, scientific reviewers did not begin to apply the policy OAn earlier memorandum in November 1987 provided limited instructions, but it applied only to contracts, a small proportion of the funds NIH awards to researchers. sThe September 25, 1987 NIH Guide for Grants and Contracts anrlounced a policy encouraging the inclusion of minorities in stuciy populations. 7 until the February 1990 grant review cycle. Three of the four institutes we reviewed in depth including The National Institute on Aging, began to apply the policy by fall 1989, but in the National Institute of Allergy and Infectious Diseases, reviewers will first implement the policy this month. Because of these delays, many scientific review groups are just beginning to send to institute councils summary statements that highlight concerns about the exclusion of women from studies. POLICY POORLY COMMUNICATED, INCONSISTENTLY APPLIED We found problems in the extent to which the policy is understood and applied by grant applicants, NIH staff, and scientific experts who review proposals for NIH funding. The application booklet used by most NIH grant applicants--PHS Form 398 --contains no reference to the policy to include women in study populations. This form is a primary source of instructions to investigators initiating their own proposals. A revised version of the form and its instructions will not appear until April 1991, over 4 years after the policy was first articulated. As a result, NIH is still receiving many proposals that are not responsive to the policy. We reviewed about 50 recent grant applications, most proposing studies on conditions that affect 8 both men and women. About twenty percent of the proposals provided no information on the sex of the study population. Over one-third indicated that both sexes would be included but did not say in what proportions. Some proposals for all-male studies provided no rationale for that design. We found that some NIH staff were unaware of their responsibilities for implementing the policy. In addition, some reviewers demonstrated limited understanding of the policy. For example, a recent proposal to conduct an all-male study related to coronary artery disease was approved by the scientific review group with the comment that the exclusion of females was appropriate because the disease studied disproportionately affects men. While this observation may be true, it may be inadequate as a rationale for excluding women, because coronary artery disease is also a serious health problem in women. The instjtute council also approved this proposal for funding, During a key stage of the review process, the policy on women is applied inconsistently. The Division of Research Grants and some institutes, including the National Institute on Aging, instruct members of scientific review groups not to consider the inclusion of women and minorities in the study population as a factor of scientific and technical merit that would affect the priority score. Instead, if the review group raises a problem with the composition of the study population, it should be addressed in an 9 administrative note in the summary statement. These administrative notes are used to highlight matters that do not pertain directly to scientific merit, such as care of experimental animals, The institute council may then take that issue into account in reaching its recommendation. Officials of the Division of Research Grants and these institutes told us that in practice there may be exceptions to this review policy. Reviewers can include the study population as a criterion for the priority score if it is clear that the proposed population would make it impossible to answer the scientific question posed by the investigator. In addition, the study population will affect the priority score if an application is responding to an institute solicitation that specifies inclusion of women as a review factor. In contrast to this review policy, National Heart, Lung, and Blood Institute officials told us that their reviewers consider adequate inclusion of women and minorities an element of scientific merit and factor it into the priority score. Likewise, ADAMHA instructs its reviewers to evaluate plans for inclusion of women as part of their overall evaluation of the technical merit of applications. 10 NO POLICY ON WOMENIN INTRAMURAL STUDIES NIH's intramural research program has no policy on the inclusion of women in study populations. In an August 1989 report, the Advisory Committee on Women's Health Issues recommended that NIH take steps to encourage inclusion of women in intramural as well as extramural studies. The Director of NIH has not formally transmitted that report to intramural officials or instructed them to develop a policy. In response to our review, the Human Research Review Panel of the NIH Clinical Center placed this issue on the agenda of its June meeting. The National Institute on Aging (NIA) provides a good example of the problems that can arise from the lack of emphasis on including women in NIH's intramural research program. The Baltimore Longitudinal Study of Aging is part of NIA's intramural program. Its failure to recruit women as study subjects during its first twenty years has resulted in some research results that can be appliced to men only. Research supported by other components of NIA's intramural program also has generated more information on men than on women. . 11 LITTLE ACTION TAKEN TO ENCOURAGEGENDER ANALYSIS Although the 1987 policy announcement also encouraged researchers to analyze study results by gender, NIH officials have taken little action to implement this element of the policy. The 1989 memorandum setting out guidelines for policy implementation calls for attention to issues of research design and sample size, but does not specify the need for gender analysis. NIH officials showed us solicitations that cited the importance of including women in study populations. We noted, however, that few suggested studies be designed to assess different results for men and women. NIH officials differ among themselves in their views on the types of studies for which gender analysis is appropriate. I!!POSSIBLE TO DETERMINE IWACT --- OF POLICY You asked us to report on the extent to which the NIH policy has resulted in inclusion of women in clinical study populations. Because policy implementation began so late, it is too soon to determine what, if any, effect it is having on the demographic composition of study populations. Additionally, given the lack of data on previous study populations, analysis of the policy's impact is virtually impossible. 12 Steps could be taken, however, to maintain data that would be useful for future monitoring of the inclusion of women in studies. At present, no central NIH office collects the types of demographic data on study populations that you requested. Several years ago, NIH revived its Inventory of Clinical Trials and the current data collection form does ask for information about the gender composition planned for study populations. However, the gender question is not categorized specifically enough to provide complete information. As another means of monitoring inclusion of women in study populations, some institutes plan to begin collecting demographic data on studies they fund. RECOYMESDATIONS -2 To ensure effective implementation of its policy to encourage the inclusion of women in study populations, the Director of NIH should take the following steps: -- Inform NIH staff, grant reviewers, and the community of researchers NIH supports of the reasons for the policy and how it should be carried out; -- Direct NIH institutes to maintain readily accessible data to allow assessment of the extent to which women are included in studies; 13 -- Ensure that the planned revision of the grant application booklet (PHS Form 398) adds a section explaining the policy and instructing applicants to respond to the requirement to include women in study populations, or to justify their exclusion; and --Instruct members of review groups always to determine whether the gender of the study population is an issue of scientific merit affecting the priority score, and to document their decisions in the summary statements. Following our original testimony, the Acting Director of NIH said he would give serious consideration to these recommendations, and by law, federal agencies have 60 days to notify Congress on actions taken in response to GAO recommendations. This concludes my statement, Madam Chairman. I would be happy to answer any questions you may have. 14
National Institutes of Health: Problems in Implementing Policy on Women in Study Populations
Published by the Government Accountability Office on 1990-07-24.
Below is a raw (and likely hideous) rendition of the original report. (PDF)