United States General Accounting Office GAO Report to the Ranking Minority Member Committee on Veterans Affairs, House of Representatives December 1999 AGENT ORANGE Actions Needed to Improve Communications of Air Force Ranch Hand Study Data and Results GAO/NSIAD-00-31 Letter 3 Appendixes Appendix I: Objectives, Scope, and Methodology 26 Appendix II: Statistical Power of the Ranch Hand Study 28 Appendix III: Duties of the Ranch Hand Advisory Committee 31 Appendix IV: Roles of Air Force, National Academy of Sciences, and Department of Veterans Affairs in Evaluating Health Effects of Herbicides 32 Appendix V: Comments From the Department of Defense 34 Appendix VI: Comments From the Food and Drug Administration 37 Tables Table 1: Minimum Detectable Effect in 1982 29 Table 2: Minimum Detectable Effect in 1992 29 Table 3: Minimum Detectable Effect in 2002 30 Table 4: Summary of Roles and Criteria in Evaluating the Health Effects of Herbicides 33 Figures Figure 1: Ranch Hand Study Milestones and Published Articles and Reports 10 Figure 2: Communication Flow From the Advisory Committee to Air Force Scientists Conducting the Ranch Hand Study 18 Page 1 GAO/NSIAD-00-31 Agent Orange Page 2 GAO/NSIAD-00-31 Agent Orange United States General Accounting Office National Security and Washington, D.C. 20548 International Affairs Division B-282636 Leter December 17, 1999 The Honorable Lane Evans Ranking Minority Member Committee on Veterans Affairs House of Representatives Dear Mr. Evans: From 1962 to early 1971, the United States sprayed herbicides, including Agent Orange, over Vietnam. In the late 1970s, concerns began to emerge over the long-term health problems of Vietnam veterans. Although these veterans could have been exposed to many potential health hazards, including pesticides, infectious diseases, and treatments for tropical diseases, attention focused on herbicides. Several herbicides, including Agent Orange, contain dioxin,1 which is known to cause a variety of adverse health effects in animals. While there is scientific evidence of some associations between exposure to herbicides (or the dioxin they contain) and adverse human health conditions, the effect of this exposure on human health remains controversial. There has been long-standing congressional interest in and concern about the effects of exposure to herbicides such as Agent Orange. Congress has held at least 26 hearings on the subject since 1978, some of them involving scrutiny of scientific studies. One key effort to examine the long-term health effects of servicemembers’ exposure to herbicides in Vietnam is an ongoing Air Force study known as the Ranch Hand study. This study was designed to investigate whether exposure to herbicides in Vietnam led or would lead to adverse health. To this end, the study follows the health (morbidity) and mortality rates of the Ranch Hands—the almost 1,300 Air Force personnel who sprayed herbicides from the air in Vietnam—in contrast to those of a comparison group. This comparison group consists of Air Force military personnel who were not involved in the spraying and who were matched to the Ranch Hands in terms of age, race, and military occupation. The 25-year study, which began in 1982 and is scheduled to end in fiscal year 2006, is projected to cost over $140 million. 1 The chemical 2, 3, 7, 8-tetrachlorodibenzo-p-dioxin. Page 3 GAO/NSIAD-00-31 Agent Orange B-282636 The study’s protocol, which outlines the study’s purpose, methods, and procedures, requires the Air Force to disclose periodically the study’s findings in official reports and in a journal of stature. The protocol also mandated a monitoring group, which currently consists of an Advisory Committee of nine scientists (three of whom were nominated by veterans’ organizations), to oversee the conduct of the study and provide an independent review of the study’s findings. Findings from the Ranch Hand study, along with those of other studies (e.g., on U.S. and foreign Vietnam veterans, chemical factory workers, or populations exposed environmentally to dioxin), are used by the Department of Veterans Affairs to determine whether veterans are eligible to receive disability compensation for conditions believed to be connected to military service. In 1984 and again in 1991, Congress revised the process used by the Department of Veterans Affairs to determine whether Vietnam veterans are eligible to receive compensation for health effects from exposures to herbicides.2 Since 1991, the Department has been required to contract with the National Academy of Sciences to perform reviews of scientific literature, including the Ranch Hand study, on the associations between diseases and herbicide exposure. Largely on the basis of these reviews, the Department of Veterans Affairs determines which diseases show a positive association between herbicide exposure and the disease. Vietnam veterans with such diseases are then eligible for disability compensation. Since its inception, the Ranch Hand study has been controversial. For instance, news articles and statements in the Congressional Record have alleged that government officials delayed and withheld information on the study’s findings, improperly influenced the study’s design and implementation, and failed to provide adequate veterans’ representation on the Advisory Committee. In addition, veterans’ organizations such as the American Legion and the Vietnam Veterans of America have criticized the Department of Defense’s conduct with regards to studies of Agent Orange. Recent congressional concerns have centered not on the scientific design or implementation of the study but on the proper dissemination and reporting of study results and on proper independent oversight. On the basis of your concerns, we examined the conduct and findings of the study and assessed the study’s impact on determinations of veterans’ 2 P.L. 98-542 (1984) and P.L. 102-4 (1991). Page 4 GAO/NSIAD-00-31 Agent Orange B-282636 compensation. As agreed with your office, we did not evaluate the study’s scientific design. Specifically, we (1) assessed whether the study’s findings and data have been properly and promptly reported and disseminated; (2) examined the statistical limitations of the study and whether they have been adequately reported and communicated; (3) examined the measures established to monitor the study’s conduct and to prevent improper influence, particularly those involving the Advisory Committee; and (4) assessed the impact of the study on determinations of diseases for which Vietnam veterans are eligible to receive compensation benefits. To examine the conduct and findings of the Ranch Hand study, we obtained and reviewed the study protocol, various study memoranda and correspondence, published study reports and peer-reviewed journal articles, executive summaries and Air Force press releases, and other available documents related to the study. In addition, we interviewed Ranch Hand study investigators, Advisory Committee members, Air Force program officials, Department of Veterans Affairs officials, veterans’ representatives, and scientists involved in research on the health effects associated with exposure to herbicides and dioxin. We also visited Brooks Air Force Base to interview members of the Ranch Hand study team and to review data collection and reporting procedures. Our scope and methodology are described in more detail in appendix I. Results in Brief To date, the Air Force has conducted all scheduled phases of the Ranch Hand study, including up to five full physical examinations of each participating Ranch Hand and comparison, and has periodically reported the results in official reports. However, there have been some delays over the years in the publication of morbidity and mortality findings in scientific journals and in the update of a key report on reproductive outcomes. The pace of publication has increased in the past few years. No journal articles on mortality or morbidity outcomes were published until 1990, even though the first mortality and morbidity reports were published in 1983 and 1984, respectively, and the Advisory Committee repeatedly recommended that such articles be published. Several reasons contributed to the publishing delays, including the need to collect and analyze data for five other morbidity and mortality reports published between 1985 and 1990. The Air Force reported preliminary reproductive outcomes in 1984 but did not publish a more detailed update until 1992 because it decided to verify the data extensively and perform additional data analyses without releasing any interim findings. Although the number of published reports has increased, public access to study data is limited by the relatively small Page 5 GAO/NSIAD-00-31 Agent Orange B-282636 amount of data currently available and its storage format (magnetic tape), which is difficult to use. Scientists and veterans’ groups critical of the study want access to all study data to verify the Air Force’s findings and to perform additional analyses. To date, the public can only access data the Air Force analyzed in 1987. The Air Force intends to make all other data available by the end of year 2000. Like many epidemiological studies,3 the Ranch Hand study has a number of inherent limitations, but the Air Force has not clearly or effectively communicated these limitations to the public. Two limitations are the difficulty in detecting increases in risks of rare diseases (because of the small size of the Ranch Hand population) and the fact that the study’s findings cannot be generalized to all Vietnam veterans (because Ranch Hands and ground troops were exposed to different levels of herbicides in different ways). In its first two morbidity reports in 1984 and 1987, however, the Air Force described the study’s lack of findings (i.e., few diseases with increased incidence among Ranch Hands) as “reassuring,” possibly leading to the misinterpretation that the study showed herbicides to be safe. Although the Air Force no longer uses such language, it still reports findings to the public through press releases and executive summaries that do not make the study’s limitations clear. During the study’s first several years, some measures intended to ensure that the study is conducted independently and without bias were not carried out as planned. To ensure independent review and prevent any appearance of Air Force management bias, the study protocol (published in 1982) mandated: (1) that an independent group of scientists (currently the Advisory Committee) be established to monitor the study’s conduct and (2) that Air Force scientists, subject to review by the monitoring group, have primary responsibility over the scientific aspects of the study. Shortcomings in the Advisory Committee’s makeup and in the processes through which the Committee and others communicated scientific advice were not fully corrected until 1989. Until that year, the Committee did not include any veterans’ representatives, as required by the study protocol. In addition, according to documents we reviewed dating from 1984 and 1985, Air Force management and the White House at the time tried to direct certain aspects of the Air Force scientists’ research. These attempts deviated from the protocol’s requirement that Air Force scientists retain 3 Scientific studies of the incidence, distribution, or control of diseases in human populations. Page 6 GAO/NSIAD-00-31 Agent Orange B-282636 primary responsibility over the study’s scientific conduct. Furthermore, the White House’s actions bypassed review by the Advisory Committee. Although these early problems were resolved through executive and congressional actions, some problems remain with the extent of the Committee’s outreach to veterans. Although the Committee’s public meetings are announced in the Federal Register, the Committee has not directly informed veterans’ organizations of these meetings. In addition, the Committee’s informal process for soliciting nominations has resulted in not all interested veterans’ organizations being notified of opportunities to nominate Committee members. The Ranch Hand study has had significant impact on one decision regarding compensation to Vietnam veterans. Partly on the basis of the study’s results, Congress passed legislation providing compensation for veterans’ children born with spina bifida.4 The study has not contributed to decisions by the Department of Veterans Affairs to compensate Vietnam veterans for any other diseases. Because the Ranch Hand study’s small sample size provides limited potential for addressing possible links between herbicide exposure and many forms of cancer, it has not contributed to any National Academy of Sciences or Department of Veterans Affairs recommendations regarding cancer. The finding of a possible association between herbicide exposure and diabetes mellitus (diabetes), first reported by the Ranch Hand study in 1991, has led to further review of the issue by the Academy, as well as by Air Force and other scientists. However, the Department of Veterans Affairs has not yet determined that Vietnam veterans with the disease are eligible for compensation. We are recommending several additional actions intended to improve communication of Air Force Ranch Hand study data and results. Background Before the Ranch Hand study began, official government and nongovernment reviewers of the study protocol expressed concern that the public would perceive the study as not credible because the fact that the Air Force would conduct the study might give the appearance of conflict of interest. An interagency working group that reviewed the study concluded, however, that the appearance of conflict of interest could be properly and 4 Spina bifida is a birth defect in which the vertebral column (backbone) fails to close and that may allow herniation of the spinal cord. Page 7 GAO/NSIAD-00-31 Agent Orange B-282636 adequately addressed through independent review and monitoring. Thus, an Advisory Committee was created to regularly review and assess the conduct of the study, review all changes to the study protocol, and provide written comments and recommendations on the study’s findings. The Committee, chartered by the Department of Health and Human Services, was initially responsible to the Agent Orange Working Group, an interagency group that monitored all government activities related to herbicides such as Agent Orange.5 The Agent Orange Working Group was headed by the Department of Health and Human Services and included representatives from many government agencies and offices, including the Department of Defense, the Department of Veterans Affairs, and the White House Office of Science and Technology Policy. In 1989, the Department of Health and Human Services transferred administration of the Committee to the Food and Drug Administration. The duties of the Advisory Committee are described in more detail in appendix III. The process of translating scientific evidence into policy decisions on compensation benefits for veterans has also been controversial. Scientific studies typically strive to produce evidence of a “cause-and-effect” relationship, and use a relatively high standard of proof (e.g., 95-percent confidence). In 1989, a federal court ruled that the Department of Veterans Affairs’ use of similar standards in making compensation decisions was too restrictive.6 According to the court, Congress had intended the Department of Veterans Affairs to require only a “statistical association” with a disease. Furthermore, the court concluded that veterans were to be given the benefit of the doubt in compensation decisions. Subsequently, in 1991, Congress established the current process for these decisions, including biennial reviews of the scientific literature by the National Academy of Sciences.7 The Academy has conducted three such reviews for Agent Orange, the most recent published in 1999. Additional information on the roles of the Ranch Hand study, the National Academy of Sciences, and the Department of Veterans Affairs is in appendix IV. 5 The Agent Orange Working Group was replaced by the Agent Orange Task Force in 1990. The final meeting of the Agent Orange Task Force occurred in 1994. 6 Nehmer v. U.S. Veterans’ Administration, 712 F. Supp. 1404 (N.D.Cal.1989). 7 P.L. 102-4 (1991). Page 8 GAO/NSIAD-00-31 Agent Orange B-282636 Publication of Study Publication of study findings in peer-reviewed journals was slow to begin, but the number of articles has increased considerably in recent years (see Results and Release of fig. 1). An update of a report on reproductive outcomes was delayed to Study Data Were Slow allow the completion of additional analyses and data collection. The Air Force has also been slow to release study data to the public. Only analyzed in the Early Years data from one physical examination (not all five examinations conducted thus far or all collected data) is currently available. In addition, the data is available only on magnetic tape rather than a more easily accessible format such as compact disc. Page 9 GAO/NSIAD-00-31 Agent Orange B-282636 Figure 1: Ranch Hand Study Milestones and Published Articles and Reports Note: “Journal articles” are articles (not including letters, comments, and errata) included in the National Library of Medicine’s MEDLINE database as of October 1999. “Reports” are official government reports published by the Air Force. *: Data currently publicly available. Source: Air Force, MEDLINE. Page 10 GAO/NSIAD-00-31 Agent Orange B-282636 Publication of Study In spite of the requirement in the study protocol as well as guidance from Findings Slow to Begin the Advisory Committee that study findings be disclosed periodically in scientific journals, no journal publications appeared until 1989, nearly 7 years after the study had begun. Publication of study results on morbidity or mortality did not begin in scientific journals until 1990. The study protocol initially mandated that the Air Force issue official reports on morbidity after each physical examination and on mortality at least every 5 years and that the results of the study be published in a journal of stature. The Air Force did not publish journal articles, however, until after it had obtained the third round of physical examinations in 1987 and after several morbidity and mortality reports had been released. Several reasons contributed to the delay, including the need to collect and analyze data for five other morbidity and mortality reports published between 1985 and 1990. The Advisory Committee repeatedly urged publication of study findings in journals. In 1986, it recommended that results be published in the “open literature,” and in 1987 it repeated the recommendation, stating that “first and foremost, there is an urgent need to publish in widely read peer-reviewed medical journals.” The pace of article publication has accelerated since 1990. Eleven articles on morbidity and mortality had been published by mid-1999, and Air Force scientists told us that over a dozen articles are currently in submission or preparation for publication in peer-reviewed journals. Update of Report on An update of a preliminary reproductive outcome report was delayed Reproductive Outcomes 8 years because the Air Force performed additional data collection and analyses without releasing any interim information. The Air Force first Was Delayed reported preliminary reproductive outcomes in a baseline morbidity study in 1984. It subsequently began verifying all medical records to check for birth defects instead of relying on reports of interviews with Ranch Hands, comparisons, and their spouses, which can be inaccurate. The Air Force anticipated completion of the effort within 1 year. Although a draft interim update, using partially verified data, was prepared later in 1984, the Advisory Committee suggested it not be released before additional work was done. The Air Force later expanded its verification of medical records through age 18 (rather than up to age 1) to identify birth defects that may not have manifested themselves earlier. This expanded verification was not completed until late 1988. From 1987 through 1990, the Air Force (in collaboration with the Centers for Disease Control and Prevention) measured dioxin levels in the blood of all study participants as part of its morbidity study and decided to analyze birth defects also using these measurements before publishing a report. The reproductive outcome Page 11 GAO/NSIAD-00-31 Agent Orange B-282636 update with all the additional analyses and data was finally published in 1992. Although the verification and additional analyses ultimately strengthened the scientific basis of the reproductive outcome portion of the study, the length of the delay, in conjunction with the lack of any interim report, led to concerns by some veterans and scientists that the study had been purposely delayed to suppress information. One reason underlying the concern was the apparently high standard of verification. Through our review of other birth defect studies and interviews with scientists, we found that the degree of birth defects verification the Air Force accomplished was highly unusual and virtually unprecedented for a study of its size. We examined two other studies of birth defects in children of Vietnam veterans. One of them verified only the health status of babies at birth, the other verified only those with reported birth defects through age 1, not those reported as healthy. Because the Ranch Hand study produced evidence that led Congress to authorize compensation for Vietnam veterans whose children have spina bifida, one veterans’ organization told us that earlier reporting, even just of preliminary findings, might have led to earlier compensation. In addition, one scientist told us that such preliminary information could have been useful for family planning or prenatal care decisions. Public Access to Study Data Two major factors currently limit public access to study data: (1) only one Is Currently Limited set of analyzed data—not all sets of analyzed data or all collected data—is available and (2) data is only available on magnetic tape, not on a more common format such as compact disc or the Internet. Two scientists and one veterans’ group told us they were interested in obtaining study data. They said that access to study data would enable them to verify the accuracy of the Air Force analyses and to perform additional analyses on the data. They cited the limited amount of study data currently available and its format among the barriers to their use of study data for these purposes. Currently, only data files from Air Force analyses of 1987 physical examination data are available, while data files from 1982, 1985, 1992, and 1997 are not. The Air Force database consists of two types of files: (1) raw data files, which include all the data collected at physical examinations, and (2) analysis data files, which were used to perform analyses for study reports. The content of raw and analysis data files overlap, but each contains data not included in the other. The data currently available to the Page 12 GAO/NSIAD-00-31 Agent Orange B-282636 public are analysis files from 1987. Raw data files from 1987 and 1992 have been processed by the Air Force for release and are in the process of becoming publicly available. Air Force officials report that they expect all raw and analysis data files, including reproductive outcome data, to be available by the end of year 2000. One scientist and one veterans’ group representative told us they believe the Air Force should release all the study’s data to permit additional analysis of the data. According to these individuals, release of the data is particularly important because of past allegations that government officials may have improperly influenced the study’s conduct. Currently, Ranch Hand study scientists send data on computer diskettes to the Defense Technical Information Center, which transfers the data to magnetic tape and sends it to the National Technical Information Service of the Department of Commerce. The Service said it provides the data to the public in the same format in which it is received, and charges about $450 to cover costs. In comparison, two other major government health databases we examined are both available on compact disc. These are the National Cancer Institute’s “Surveillance, Epidemiology, and End Results” database of cancer incidence throughout the United States, available free of charge from the Department of Health and Human Services, and the National Health and Nutrition Examination Survey, a Centers for Disease Control and Prevention database of the health and nutritional status of the U.S. population, that can be purchased for $20 from the Government Printing Office. Several veterans’ representatives said that they do not have the necessary computer equipment to use magnetic tapes, and one scientist told us that other formats such as compact discs would be easier to use. Air Force officials also told us they are planning to release the data on the Internet but noted that the National Technical Information Service would remain their primary means of release. Study Limitations Have All scientific studies have inherent limitations. Two major limitations of the Ranch Hand study are the difficulty in detecting increases in risks of rare Not Been Clearly diseases (because of the small size of the Ranch Hand population) and the Communicated to the fact that the study’s findings cannot be generalized to all Vietnam veterans (because Ranch Hands and ground troops were exposed to different levels Public of herbicides in different ways). In spite of the study’s sensitive and controversial nature, early study reports contained language that may have been misinterpreted to mean that the study showed herbicides were safe. Although the Air Force no longer uses such language, recent press releases Page 13 GAO/NSIAD-00-31 Agent Orange B-282636 and executive summaries still do not clearly communicate the study’s limitations to the public. Two Major Study The small number of participants in the Ranch Hand study—currently Limitations fewer than 1,200 Ranch Hands and less than 1,800 comparisons—makes it very difficult to detect a statistically significant increase in the risk of rare diseases. For instance, the lifetime risk of non-Hodgkins lymphoma (a cancer often linked to herbicide exposure) in Caucasian males in the United States is about 2 percent. Even if the Ranch Hand study were to follow participants through their entire lives (which it does not), because of the requirement for statistical significance, it could not confidently report any adverse effect unless the incidence of the disease in Ranch Hands were at least 3.4 percent. Appendix II contains a more detailed explanation of the study’s statistical limitations as they relate to the size of its study population and the detectable increased risk of cancer. Another limitation is that it is difficult to generalize the results of the study to all Vietnam veterans because Ranch Hands were exposed to herbicides in different ways than most ground troops in Vietnam. For instance, by comparing dioxin blood levels with Ranch Hands’ reported exposures to herbicides, Air Force scientists found evidence that Ranch Hands were exposed to herbicides primarily through their skin. Many ground-troop veterans, however, reported exposure to herbicides through other mediums such as contaminated food and water and contaminated clothing worn for extended periods of time. They believe that Ranch Hands were only minimally exposed through these mediums because they consumed uncontaminated food and water, wore clean clothes, and could wash regularly. These differences have not been extensively studied, so little is known about their potential effects. However, blood measurements of dioxin levels suggest, though not conclusively, that Ranch Hands’ exposure levels were significantly higher than both those of the study’s comparison group and of many ground troops who have been tested. 8 Because the comparison group was not involved in spraying herbicides, its low dioxin levels were consistent with 8 Since 1991, the study’s morbidity reports have concentrated on analyzing the association between dioxin levels and health rather than the health contrast between Ranch Hands and the comparison group. Almost half of the Ranch Hands have dioxin levels similar to those of the comparison group, further reducing the study size when applying the analysis of dioxin levels versus health. Page 14 GAO/NSIAD-00-31 Agent Orange B-282636 lack of exposure to herbicides. Congress had proposed a program to test veterans’ blood for dioxin, under the condition that such a program would make a contribution to scientific knowledge, but the National Academy of Sciences stated that such testing would not be useful unless it were part of a specific scientific study. Therefore, most Vietnam veterans’ blood dioxin levels are untested. Furthermore, according to several scientists we interviewed and scientific documents we reviewed, low levels of dioxin in the blood today do not guarantee that an individual had low amounts of exposure to herbicides in Vietnam. Reasons include (1) different herbicides contain different amounts of dioxin; (2) the general population is also exposed to detectable levels of dioxin masking exposures during the Vietnam War; (3) dioxin is slowly excreted from the body, so subjects may have had higher dioxin levels in the past; and (4) the rate at which dioxin is excreted differs, depending on the amount of body fat, so individuals with similar dioxin levels today may have had different exposures in the past. Early Reports Could Have The importance of communicating the study’s limitations to the public has Led to Misinterpretation of been stressed and highlighted by a number of scientists and officials over the years. As far back as 1980, the predecessor of the Agent Orange Findings to Mean Working Group9 recommended that the study proceed only on condition Herbicides Were Safe that the public “clearly understand that the stated health goal in the Air Force [Ranch Hand study] Protocol may not be fully realized” and that this “does not imply flaws in protocol design; it is to emphasize the inherent limitation of the study size.” In addition, in 1984 the White House Office of Science and Technology Policy urged the Air Force to respond to criticisms and perceived misunderstandings of the study by discussing more explicitly “the meaning of the number of cases found (or not found) … for each disease included in the study,” adding that “the discussion should be in terms of the confidence interval of these numbers and their comparison to general population data on these diseases.” In its early reports, and particularly in executive summaries and press releases, the Air Force used language that may have led the results of the Ranch Hand study to be interpreted inaccurately. As far back as 1980, a review panel of government scientists emphasized that “it needs to be clearly understood that failure to identify increased risk in a variety of health parameters is to be interpreted as inconclusive and not necessarily a 9 The Scientific Panel of the Interagency Work Group on Phenoxy Herbicides and Contaminants. Page 15 GAO/NSIAD-00-31 Agent Orange B-282636 true lack of effect” (emphasis in original). However, in describing the results of the first two morbidity reports, published in 1984 and 1987, the Air Force described the overall study conclusions as “reassuring” because they detected few statistically significant increased risks. The use of such language, as well as a statement by the Deputy Air Force Surgeon General at a press conference in 1984 saying that the results showed “nothing that would keep us from using it [herbicides such as Agent Orange] again,” may have led to the misinterpretation that the Ranch Hand study proved that herbicides were safe. Since 1990, the Air Force has not used this type of language. Current Air Force scientists acknowledge that the word “reassuring” in particular may have led to improper interpretations. Study Limitations Not Fully We reviewed all Air Force reports, executive summaries, and press releases and Clearly Communicated on the Ranch Hand study and found that the Air Force still reports study results to the public without consistently making clear that any lack of to the Public in Executive conclusive evidence may be due to the study’s limitations—not necessarily Summaries and Press a lack of long-term health effects from herbicides. While study limitations Releases have been addressed in the full text of the reports, they have not been consistently explained in executive summaries and press releases, which are more accessible to the general public. For example, the executive summary and press release for the baseline morbidity study, published in 1984, only stated that statistical limits exist, and gave no further description. The executive summary of the third morbidity report, published in 1990, stated that the study had a good chance of detecting a statistically significant effect only if a high number of study subjects (i.e., 5 percent or more in the comparison group, and 10 percent or more in the Ranch Hands) contracted a particular disease. In addition, only in the executive summary and press release of its 1991 morbidity report did the Air Force mention study limitations due to the differences in exposure routes and levels between Ranch Hands and ground troops. However, in this case, the Air Force only stressed the limits with which any positive evidence of ill health could be applied to other Vietnam veterans and did not mention the limits on what conclusions could be drawn from a lack of evidence in the study. All other press releases and executive summaries have not mentioned study limitations at all. Several scientists we spoke with also voiced concern about the Air Force’s public reporting of study results. One scientist with expertise in dioxin research told us that Air Force press releases and executive summaries continue to be misleading because they are not adequately balanced by an account of the limitations of the study. A scientist on the Advisory Page 16 GAO/NSIAD-00-31 Agent Orange B-282636 Committee also commented at a Committee meeting in August 1999 that the draft of the upcoming Air Force morbidity report stressed the study’s lack of findings too strongly and did not adequately discuss the limits to inferring either positive or negative effects. A former Air Force scientist, however, told us that the Air Force’s public reporting of study findings should be viewed in the context of media and unsubstantiated reports of major incidences of disease and death caused by herbicides. Many Early Oversight Two measures specified by the study protocol to ensure independent review and prevent any appearance of Air Force management bias were not Problems Have Been followed prior to 1989. These were the requirements that the Advisory Resolved, but Some Committee include scientists nominated by veterans’ organizations and that Air Force scientists, subject to review by the Advisory Committee, Veterans Groups Want retain primary responsibility over scientific aspects of the study. Executive More Outreach agencies (in 1985 and 1989) and Congress (in 1988 and 1990) took actions to correct these problems. However, problems remain with how well the Advisory Committee keeps veterans’ organizations informed of its activities. Early Problems With the Prior to 1989, the Advisory Committee did not fulfill a key requirement of Advisory Committee’s the study protocol: that it include scientists nominated by veterans’ organizations. The requirement stated that about one-third of Committee Composition and the members should be scientists nominated by veterans’ organizations. Communication of According to our review, the Committee did not include any such scientists Scientific Advice from 1982 through 1989. Documents we reviewed indicate that one reason was that the Agent Orange Working Group, which was responsible for administering the Committee, appeared to view the protocol’s description of the Advisory Committee as only giving a suggestion. The second problem that existed before 1989 was that outside entities tried to direct how Air Force scientists conducted certain aspects of the study, even though the study protocol specified that Air Force scientists “are responsible for … all data analysis, and for all data interpretation of analyses subject to review by the outside monitoring group.” This problem stemmed from the fact that before 1989, the Committee did not communicate its formal recommendations directly to Air Force scientists conducting the study. Rather, the Committee gave its formal recommendations first to the Agent Orange Working Group for approval, creating opportunities for others to influence the study. Figure 2 shows the Page 17 GAO/NSIAD-00-31 Agent Orange B-282636 flow of communications from the Committee to study scientists before and after 1989. Figure 2: Communication Flow From the Advisory Committee to Air Force Scientists Conducting the Ranch Hand Study Note: Solid lines indicate formal communication (e.g., agency memoranda). Dotted lines indicate informal communication (e.g., verbal comments or meeting minutes). Source: Air Force. Documents we reviewed from the period before 1989 indicate that the Air Force and the White House Office of Science and Technology Policy Page 18 GAO/NSIAD-00-31 Agent Orange B-282636 (a member of the Agent Orange Working Group) tried to direct certain aspects of the Air Force scientists’ research. For instance, in a memorandum sent in October 1984, the commander of the School of Aerospace Medicine (where the study is conducted), Brooks Air Force Base, ordered the Air Force scientists to perform certain analyses for mortality reports and to present the results in a specific format. The memorandum noted that the direction came from the Air Force Deputy Surgeon General and that it was “in accordance with the comments of the Advisory Committee.” However, according to the study protocol, Air Force scientists, not Air Force management, have ultimate responsibility for all data analysis and interpretations. One of the scientists involved at the time in the study expressed concern that the actions ordered by the commander would weaken the study or de-emphasize the significance of the results. In addition, in a letter also sent in 1984, the White House Office of Science and Technology Policy wrote that it had “directed that the Air Force undertake … actions,” including the extensive verification of reported birth defects. This verification contributed to the 8-year delay in updating study findings on reproductive outcomes. Again, this direction infringed on the Air Force scientists’ decision-making responsibilities in the scientific conduct of the study. Furthermore, the White House later provided guidance directly to the Air Force, requesting that the Advisory Committee ensure that this guidance be implemented. However, the study protocol only authorized the Advisory Committee to provide written comments and recommendations on the conduct of the Ranch Hand study to the White House Office of Science and Technology Policy, not the reverse. The protocol also required that all scientific aspects of the study undergo review by the Advisory Committee. Both Air Force management and the Advisory Committee later expressed concern about the appropriateness of the White House’s communication to the Air Force scientists and sought clarification from the Agent Orange Working Group. Many Early Problems In 1985, the Agent Orange Working Group decided that all Solved by 1990 recommendations intended for the Air Force scientists conducting the study should be communicated first to the Committee. This decision was made primarily in response to concerns about conflicting advice received by the Air Force scientists from bodies such as the White House Office of Science and Technology Policy. We found no evidence that any recommendations made since then have bypassed the Committee. In 1988, Congress mandated that veterans’ organizations be allowed to nominate Page 19 GAO/NSIAD-00-31 Agent Orange B-282636 three of the Committee’s nine members,10 and we verified the Committee’s compliance with this requirement. In 1989, the Department of Health and Human Services transferred administration of the Committee from the Agent Orange Working Group to the Food and Drug Administration, thus eliminating the Agent Orange Working Group as an intermediary in the chain of communication. Congress passed additional legislation in 1990 stating that “no officer or employee of the Federal Government may interfere in or impair direct communication between the Advisory Committee and … [the Air Force] scientists.”11 Remaining Problems in Although procedural problems in the early years of the study have been Informing Veterans of resolved, veterans’ groups and several scientists we spoke with continue to question the credibility of the study. In addition, representatives of Committee Activities veterans’ groups told us that the Committee’s communication with veterans’ organizations has been intermittent and inconsistent. Two veterans’ groups told us that they had not been notified repeatedly by either the Air Force or the Advisory Committee of public Committee meetings. Although they acknowledged that public Committee meetings are announced in the Federal Register, they stated that similar committees, such as those run by the Department of Veterans Affairs or the National Academy of Sciences, are more proactive in keeping veterans informed of their activities. One group, the Vietnam Veterans of America, which represents veterans with the most interest in herbicides such as Agent Orange, told us that lack of notification persisted even after it had requested directly to the Committee that it be informed of meetings in advance. In addition, the process for soliciting nominations for members of the Advisory Committee has been relatively unstructured and has led to incomplete dissemination of Committee membership openings, as well as inaccuracies in the identification of veterans’ nominees. The Food and Drug Administration is responsible for soliciting nominations from veterans’ organizations. However, openings have not been widely publicized. Rather, the Food and Drug Administration reported using an informal process, which involves asking previous Committee consultants to become nominees and telephone calls to some veterans’ organizations. 10 P.L. 100-687 section 1205 (1988). 11 P.L. 101-510 section 1468 (1990). Page 20 GAO/NSIAD-00-31 Agent Orange B-282636 The Vietnam Veterans of America told us that as a consequence, it has never been asked to nominate Committee members. Our review of Committee documents showed that veterans’ organizations that have nominated Committee members include the American Legion, the Veterans of Foreign Wars, and the Ranch Hand Association. In addition, our review revealed that one current Committee member was incorrectly identified as having been nominated by a veterans’ organization and that another Committee member who was nominated by a veterans’ organization was not identified as such. Study Has Had Limited The Ranch Hand study had significant impact on the decision to provide compensation for veterans’ children born with spina bifida. The study has Impact on not contributed to decisions by the Department of Veterans Affairs to Determination of compensate veterans for any other diseases. The finding of a possible association between herbicide exposure and diabetes mellitus, first Diseases for Which reported by the Ranch Hand study in 1991, has led to greater discussion by Veterans Are Eligible to the National Academy of Sciences and further study by Air Force and other Receive Compensation scientists, but Vietnam veterans with the disease are not yet eligible for compensation. Study Contributed to the The most significant impact of the Ranch Hand study to date has been on a Decision to Compensate for decision to allow compensation to Vietnam veterans’ children born with spina bifida. According to a scientist at the National Academy of Sciences, Spina Bifida because of the high quality of the Ranch Hand birth defect study, the Academy decided in 1996 to upgrade its evaluation of evidence for an association between herbicide exposure and spina bifida from “insufficient/inadequate” to “limited/suggestive.” Subsequently, the Department of Veterans Affairs requested, and Congress approved, legislation allowing the Department of Veterans Affairs to provide compensation to Vietnam veterans’ children with the disease. Page 21 GAO/NSIAD-00-31 Agent Orange B-282636 Study Has Not Contributed The Ranch Hand study has had limited impact on either National Academy to Decisions to Allow of Sciences reports or Department of Veterans Affairs decisions concerning associations between herbicide exposure and other diseases.12 Although Compensation for Other findings from other studies have enabled veterans with any of several Diseases cancers to be eligible for compensation, the Ranch Hand study has had almost no impact on these decisions because of the small size of the Ranch Hand population and the relative rarity of many cancers. In addition, scientific studies such as the Ranch Hand study are involved only in determining for which diseases veterans may be eligible for compensation, not in any other part of the compensation process. There are other requirements, such as obtaining medical diagnosis of a disease and determination of disability, that Vietnam veterans must satisfy in order to receive compensation. Moreover, reports and articles by the Ranch Hand study comprise only a small fraction of the information the National Academy of Sciences reviews and the Department of Veterans Affairs then considers when weighing scientific evidence. A finding of a possible association between herbicide exposure and diabetes mellitus was first suggested in a Ranch Hand study report in 1991. This and subsequent findings by the study have led to increased research on the subject and more reviews by the National Academy of Sciences. Although in its February 1999 report the Academy described the evidence of an association between herbicide exposure and diabetes mellitus as “equivocal,” the Academy still concluded, as it did in its earlier reports, that there was “inadequate/insufficient” evidence for such an association. However, a Task Force within the Department of Veterans Affairs recommended in April 1999 to allow Vietnam veterans with diabetes mellitus to become eligible for compensation. The Task Force wrote that because the weights of positive and negative evidence were approximately equal, the requirement to give veterans the benefit of the doubt (as stated by law) necessitated providing compensation for diabetes mellitus. The Department of Veterans Affairs has asked the Academy to conduct an additional evaluation of the scientific literature on the subject. The evaluation is scheduled to be completed by the end of 1999. 12 Currently, the Department of Veterans Affairs allows compensation for nine diseases (including six cancers) in veterans and one birth defect in children of veterans. Page 22 GAO/NSIAD-00-31 Agent Orange B-282636 Conclusions The Air Force has conducted many aspects of the Ranch Hand study in a rigorous manner. However, some past and present problems in the conduct of the study have led some veterans to view the study with suspicion and to express concerns about the study’s conduct. In particular, several veterans’ groups and scientists are critical of the study’s methods and results and want greater access to study data. Until all study data is publicly available, people critical of the study cannot fully verify the Air Force’s analyses. Making all study data publicly available would increase the credibility of Ranch Hand study results. In addition, without full access to data, outside scientists cannot perform alternative or additional analyses, which would facilitate open scientific debate on the merits of the Air Force’s methodologies and analyses. The Air Force anticipates making all its study data available to the public by the end of year 2000, but those with interest in the data say that unless the data is available in a more easily accessible format (such as compact disc or the Internet), they would have difficulty in using it. In addition, while communication of study limitations to the public has improved over the years, additional improvements are possible. Lack of sufficient knowledge of the study’s limitations can lead to misleading or incorrect interpretations of the study’s findings. Language used in early reports that may have led to such interpretations is no longer being used, but because of the study’s sensitive and controversial nature, it is important that publicly accessible documents such as executive summaries and press releases provide accurate information on the study’s limitations. Early problems in the Advisory Committee’s composition and communications have been remedied, but problems remain with the Committee’s outreach to veterans. The Committee’s reliance on indirect means, such as the Federal Register, to notify veterans’ organizations of its activities perpetuates the impression that the Committee does not seek veterans’ input. Moreover, better notification of Committee openings would help ensure that veterans’ groups perceive the Committee as fulfilling its role as an independent and unbiased oversight body. Recommendations To facilitate public access to study data and more effective communication of study limitations, we recommend that the Secretary of Defense direct the Air Force scientists in charge of the Ranch Hand study, in consultation with the Advisory Committee, to Page 23 GAO/NSIAD-00-31 Agent Orange B-282636 • establish and publicize a timetable for the release of all study data and provisions to release the data in a format (such as compact disc or the Internet) that is easily accessible to the general public and • include more information on the study’s limitations in press releases and executive summaries, which should address the limited applicability of study results (especially negative results) to other Vietnam veterans and the limited ability of the study to detect small to moderate increases in risks of rare diseases. In addition, to facilitate dissemination of information on the Advisory Committee’s activities to veterans, we recommend that the Secretary of Health and Human Services direct the Committee’s Executive Secretary at the Food and Drug Administration to provide direct and timely notification to veterans’ organizations of scheduled Advisory Committee meetings and of opportunities for veterans to nominate Committee members. Agency Comments In written comments on a draft of this report, the Department of Defense concurred with our recommendations and indicated that it is taking steps to address them. The Department also commented on our finding that the publication of study findings in scientific journals was slow to begin, stating that the time required to write and publish journal articles is typically 3-5 years and that work on the papers published in 1990 actually started in 1986. While we recognize that the publication process can be lengthy, it should be noted that the journal articles published in 1990 only included analysis of data collected in 1987 and reported in 1990 in an official Air Force report. No journal articles were published earlier based on data collected in 1982 and 1985 and respectively released in Air Force reports in 1984 and 1987. Other comments from the Department were technical in nature, and we incorporated them where appropriate. Comments from the Department of Defense are reproduced in their entirety in appendix V. The Food and Drug Administration also provided written comments on a draft of this report. It stated that in general it found the report well done and accurate. With regard to our recommendation that the Secretary of Health and Human Services direct the Advisory Committee’s Executive Secretary to facilitate dissemination of information to veterans, it indicated that they are working with the Office of Veterans Affairs and Military Liaison of the Department of Health and Human Services to ensure that veterans’ organizations are notified in a timely manner of Advisory Committee meetings and vacancies. The Food and Drug Administration Page 24 GAO/NSIAD-00-31 Agent Orange B-282636 also provided technical comments that we incorporated where appropriate. Comments from the Food and Drug Administration are reproduced in their entirety in appendix VI. As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies of this report to other appropriate congressional committees. We will also send copies to the Honorable William S. Cohen, Secretary of Defense; the Honorable F. Whitten Peters, Secretary of the Air Force; and the Honorable Donna E. Shalala, Secretary of Health and Human Services. In addition, we will make copies available to others upon request. If you have any questions about this report, please call me at (202) 512-3652 or John Oppenheim at (202) 512-3111. Weihsueh Chiu was a key contributor to this report. Sincerely yours, Kwai-Cheung Chan Director Special Studies and Evaluations Page 25 GAO/NSIAD-00-31 Agent Orange Appendix I Objectives, Scope, and Methodology AA ppp ep ned nx idx eIis Our objectives were to (1) assess whether the study’s interim findings and data have been properly and promptly reported and disseminated, (2) examine the statistical limitations of the study and whether they have been adequately reported and communicated, (3) examine the measures established to monitor the study’s conduct and to prevent improper influence, and (4) assess the impact of the Ranch Hand study on National Academy of Sciences reports and Department of Veterans Affairs determinations regarding diseases for which Vietnam veterans are eligible to receive compensation benefits. To assess whether the study’s interim findings and data have been properly and promptly reported and disseminated, we reviewed the study protocol and memoranda and correspondence by the Air Force and the Advisory Committee to determine the schedule for reporting study findings. In addition, we reviewed these documents to examine the reasons for any delays. We also obtained and reviewed published study reports and peer-reviewed articles to determine the actual date of these publications. We conducted site visits to Brooks Air Force Base to interview scientists in charge of the Ranch Hand study and to review data collection and reporting procedures. Finally, we reviewed the current state of public access to study data and queried several scientists and veterans’ organizations about their ability to access the data. To examine the statistical limitations of the study and the reporting and communication of these limitations, we reviewed the study protocol, the study reports, and the memoranda and correspondence obtained from the Air Force and the Advisory Committee. In addition, we reviewed literature on epidemiology and communication of health risks. We reviewed study reports, executive summaries, and Air Force press releases to assess how study limitations have been communicated. We also queried scientists about aspects of communicating study limitations. Using the National Cancer Institute’s Surveillance, Epidemiological, and End Results database, we determined the expected incidence of several cancers in Ranch Hands based on the rate in the U.S. population and performed a calculation of the statistical power of the study (see app. II). To examine measures established to monitor the study’s conduct and to prevent improper influence, we reviewed the study protocol, memoranda, and correspondence obtained from the Air Force and the Advisory Committee and documentation provided by the Food and Drug Administration on veterans’ representation on the Committee. We documented the roles and activities of key organizations involved in Page 26 GAO/NSIAD-00-31 Agent Orange Appendix I Objectives, Scope, and Methodology making scientific recommendations to Air Force scientists and compared them with requirements outlined in the study protocol. We also reviewed requirements on the selection and composition of the Ranch Hand Advisory Committee and reporting requirements as stated by law. To assess the impact of the Ranch Hand study on National Academy of Sciences reports and Department of Veterans Affairs determinations of diseases for which veterans are eligible to receive compensation benefits, we reviewed laws outlining the process to determine for which diseases Vietnam veterans are eligible for compensation. We then obtained and reviewed reports by the Academy and the Department of Veterans Affairs. In addition, we interviewed officials from these organizations to determine how research results lead to compensation decisions. We also reviewed those portions of the Federal Register that announce and codify regulations governing compensation of Vietnam veterans. In conducting our study, we contacted current and former members, affiliates, or representatives from the following organizations: Advisory Committee, Executive Secretary at National Center for Toxicological Research, Food and Drug Administration; Air Force Research Laboratory, Brooks Air Force Base, San Antonio, Texas; Air Force Surgeon General, Bolling Air Force Base, Washington, D.C.; American Legion, Washington, D.C.; Department of Veterans Affairs, Washington, D.C.; National Academy of Sciences, Washington, D.C.; Veterans of Foreign Wars, Washington, D.C.; and Vietnam Veterans of America. In addition, we spoke with several scientists involved in research on the health effects associated with exposure to herbicides and dioxin. We performed our work from January through October 1999 in accordance with generally accepted government auditing standards. Page 27 GAO/NSIAD-00-31 Agent Orange Appendix II Statistical Power of the Ranch Hand Study Appendx iI This appendix describes in more detail the limited statistical power of the study caused by the small size of the Ranch Hand population. There were a total of about 1,200 Ranch Hands in 1982, and there are currently fewer than 1,150 still alive. An estimate of the ideal statistical power of the Ranch Hand study from 1982 to the last physical examination, to be performed in 2002, is summarized for several cancers in tables 1-3. The last column of each table shows the expected number of cases among Ranch Hands necessary for the study to be likely to detect an increased risk of cancer with 95-percent significance. Statistical power increases with time because cancers, like most health problems, are more common among older people. This calculation is “ideal” because it assumes that all Ranch Hands and comparisons participate and that none of them die. It also does not account for a possible “healthy-soldier effect,” in which military populations are healthier than nonmilitary ones because entrance into military service requires passing a medical examination that screens out applicants with chronic illnesses and because remaining in the service requires maintaining good enough health to perform relatively rigorous duties. Because these numbers represent what the study could detect under the most ideal conditions, the actual power of the study is probably less than indicated here. As shown in these tables, the study cannot detect a doubling of cancer incidence (relative risk of at least 2) until at least 10 years into the study for many types of site-specific cancers. For all sites combined, the study can detect a doubling in risk. Page 28 GAO/NSIAD-00-31 Agent Orange Appendix II Statistical Power of the Ranch Hand Study Table 1: Minimum Detectable Effect in 1982 Cases expected Number of among Ranch cases Hands Cumulative expected Minimum corresponding to expected among relative riska minimum relative percentage comparison likely to be risk likely to be Cancer site with cancer group detected detected Liver 0.012 0.14 42 6.0 Prostate 0.088 1.0 7.5 7.9 Non-Hodgkins Lymphoma 0.10 1.2 6.7 8.3 Melanoma 0.17 2.0 4.9 9.7 All anatomical sites 2.2 26 1.7 43 Table 2: Minimum Detectable Effect in 1992 Cases expected Number of among Ranch cases Hands Cumulative expected Minimum corresponding to expected among relative riska minimum relative percentage comparison likely to be risk likely to be Cancer site with cancer group detected detected Liver 0.053 0.63 11 7.1 Prostate 1.1 13 2.0 27 Non-Hodgkins Lymphoma 0.36 4.3 3.1 14 Melanoma 0.49 5.9 2.7 16 All anatomical sites 7.8 94 1.3 122 Page 29 GAO/NSIAD-00-31 Agent Orange Appendix II Statistical Power of the Ranch Hand Study Table 3: Minimum Detectable Effect in 2002 Cases expected Number of among Ranch cases Hands Cumulative expected Minimum corresponding to expected among relative riska minimum relative percentage comparison likely to be risk likely to be Cancer site with cancer group detected detected Liver 0.18 2.1 4.6 9.9 Prostate 5.5 65 1.4 90 Non-Hodgkins Lymphoma 0.86 10 2.2 22 Melanoma 1.2 15 1.9 28 All anatomical sites 22 258 1.2 299 a The minimum relative risk is the smallest relative risk that must exist in order for the likelihood that the study can detect the effect with 95-percent significance to be greater than 50 percent. The last column is the corresponding minimum number of cases expected among Ranch Hands in order for the study to detect an adverse effect. Numbers are rounded to two digits unless otherwise shown. Note: The Ranch Hands used in this analysis were the 1,197 who were identified and alive as of December 31, 1982. An identical number of comparisons were assumed. They were stratified by birth in 5-year bins, with blacks and non-blacks considered separately. The age-specific cancer incidence rates for 1973-96 (from the Surveillance, Epidemiological, and End Results database) were used. Years before 1973 used the 1973 rates and years after 1996 used the 1996 rates. The cumulative incidence was calculated assuming all Ranch Hands left Vietnam in 1967. Sources: GAO analysis of National Cancer Institute and Air Force data. Page 30 GAO/NSIAD-00-31 Agent Orange Appendix III Duties of the Ranch Hand Advisory Committee Appendx Ii This appendix discusses the duties of the Advisory Committee in overseeing the Ranch Hand study. In general, the Committee conducts scientific reviews of the study and provides the Air Force with comments and advice. However, this advice, which may be in the form of either formal or informal recommendations, is nonbinding. The Committee’s primary purpose, as described in the study protocol, is to review and assess the study’s conduct. The Committee performs this role through both site visits and by reviewing study reports before their release. Site visits include visits to Brooks Air Force Base, where the Air Force scientists are located, and to locations where physical examinations are performed. The Committee also reviews all official Air Force reports on the study prior to release, including executive summaries. The Committee does not review peer-reviewed journal articles but has consistently encouraged the Air Force to publish its results in these journals. The Committee convenes meetings when the Air Force has drafted reports to be reviewed and has not met on a regular basis. According to the study protocol, the Committee should also review suggested changes in the data collected and methods of analysis used by the study. To this end, the Committee reviews the content of physical examinations, along with the statements of work for those examinations. In addition, the Air Force has contracted out the overall management, logistics, statistical analyses, and writing of Air Force morbidity reports subsequent to the first report in 1984. The Committee reviews plans for statistical analyses. Page 31 GAO/NSIAD-00-31 Agent Orange Appendix IV Roles of Air Force, National Academy of Sciences, and Department of Veterans Affairs in Evaluating Health Effects of Herbicides Appendx IV i In evaluating the health effects of herbicides, the Air Force, the National Academy of Sciences, and the Department of Veterans Affairs have different goals, use different criteria, and produce different types of products. These differences are illustrated in table 4. The goal of the Ranch Hand study is to determine whether health effects are attributable to herbicide exposure, while the goals of the Academy and the Department of Veterans Affairs emphasize establishing only an association. Although statistical methods are the primary tools the Air Force uses in the study, the “strength of association” of a result is only one of the many criteria Air Force scientists use in evaluating evidence of a causal relationship between adverse health and exposure to herbicides. The National Academy of Sciences and the Department of Veterans Affairs, on the other hand, both look primarily at evidence of a “statistical association” between diseases in humans and exposure to herbicides. The Academy considers evidence of a causal relationship separately. The Ranch Hand study and the reviews by the National Academy of Sciences apply a relatively high standard in evaluating evidence. The Department of Veterans Affairs, on the other hand, is charged with using a specified standard of evidence intended to give veterans the “benefit of the doubt.” Specifically, a “positive association” for the Department of Veterans Affairs is defined by law as when “the credible evidence for the association is equal to or outweighs the credible evidence against the association.” Although both the Air Force and the National Academy of Sciences report findings of lesser significance, their emphasis is on establishing relationships in which they are reasonably confident. Air Force scientists noted, for instance, that though not all their criteria need to be satisfied for a finding to be reported, the more indicators that point to a causal relationship, the stronger the finding. The Academy defines “sufficient” evidence as findings in which “a positive association has been observed … in studies in which chance, bias, and confounding could be ruled out with reasonable confidence.” A second Academy category, one of “limited/suggestive” evidence, includes situations in which “evidence is suggestive of an association … but is limited because chance, bias, and confounding could not be ruled out with confidence.” Page 32 GAO/NSIAD-00-31 Agent Orange Appendix IV Roles of Air Force, National Academy of Sciences, and Department of Veterans Affairs in Evaluating Health Effects of Herbicides Table 4: Summary of Roles and Criteria in Evaluating the Health Effects of Herbicides Goal(s) Evidence evaluated Standards/criteria used Product(s) Uses of products Air Force Existence of health Data collected by Air Plausibility. Ranch Hand National Academy Ranch effects in Vietnam Force on Ranch Hands Internal and external study official of Sciences Hand study veterans attributable to and comparison group. consistency. reports. reviews. exposure to herbicides. Strength of association. Peer-reviewed Department of Statistical power. journal articles. Veterans Affairs Biological mechanism. compensation Validity or bias. determinations. Statistical significance. Scientific base of Statistical assumptions. knowledge. a National Existence of statistical Scientific literature on Extent to which scientific data Biennial reports: Department of Academy of association between effects of herbicides in permit determinations. Veterans and Veterans Affairs a Sciences disease in Vietnam humans. Strength of evidence. Agent Orange. compensation a veterans and Appropriateness of methods. determinations. herbicides. Recommendations for future studies. a Existence of an Scientific literature on Extent to which scientific data increased risk among Vietnam veterans and permit determinations. Vietnam veterans. other human populations. a Existence of a plausible Scientific literature on Extent to which scientific data biological mechanism effects of herbicides in permit determinations. or other causal humans and animals. Validity of animal models. relationship. Biochemical origin of dose response. a Department Existence of a positive National Academy of Whether evidence for an Department Task Compensation of Veterans association between Sciences review of association equals or outweighs Force Reports. determinations. Affairs disease in humans and scientific literature. evidence against an Regulations of Recommendations herbicide exposure. Other available association, considering diseases with for future studies. scientific and medical (a) academy reviews, presumption of information and (b) statistical significance, service analyses. (c) capability of replication, and connection. (d) capability to withstand peer review. a Criterion required by law. Sources: Air Force, Public Law, National Academy of Sciences, Department of Veterans Affairs, and Federal Register. Page 33 GAO/NSIAD-00-31 Agent Orange Appendix V Comments From the Department of Defense Appendx V i Page 34 GAO/NSIAD-00-31 Agent Orange Appendix V Comments From the Department of Defense See p. 24. See p. 24. See p. 24. Page 35 GAO/NSIAD-00-31 Agent Orange Appendix V Comments From the Department of Defense Page 36 GAO/NSIAD-00-31 Agent Orange Appendix VI Comments From the Food and Drug Administration Appendx V iI Page 37 GAO/NSIAD-00-31 Agent Orange Appendix VI Comments From the Food and Drug Administration p See p. 24. (713035) Leter Page 38 GAO/NSIAD-00-31 Agent Orange Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary, VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. GI00 Official Business Penalty for Private Use $300 Address Correction Requested
Agent Orange: Actions Needed to Improve Communications of Air Force Ranch Hand Study Data and Results
Published by the Government Accountability Office on 1999-12-17.
Below is a raw (and likely hideous) rendition of the original report. (PDF)