oversight

Agent Orange: Actions Needed to Improve Communications of Air Force Ranch Hand Study Data and Results

Published by the Government Accountability Office on 1999-12-17.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting Office

GAO               Report to the Ranking Minority Member
                  Committee on Veterans Affairs, House of
                  Representatives


December 1999
                  AGENT ORANGE

                  Actions Needed to
                  Improve
                  Communications of
                  Air Force Ranch Hand
                  Study Data and
                  Results




GAO/NSIAD-00-31
Letter                                                                                3


Appendixes   Appendix I:    Objectives, Scope, and Methodology                       26
             Appendix II:   Statistical Power of the Ranch Hand Study                28
             Appendix III: Duties of the Ranch Hand Advisory Committee               31
             Appendix IV: Roles of Air Force, National Academy of Sciences,
               and Department of Veterans Affairs in Evaluating Health
               Effects of Herbicides                                                 32
             Appendix V:    Comments From the Department of Defense                  34
             Appendix VI: Comments From the Food and Drug
               Administration                                                        37


Tables       Table 1: Minimum Detectable Effect in 1982                              29
             Table 2: Minimum Detectable Effect in 1992                              29
             Table 3: Minimum Detectable Effect in 2002                              30
             Table 4: Summary of Roles and Criteria in Evaluating
               the Health Effects of Herbicides                                      33


Figures      Figure 1: Ranch Hand Study Milestones and Published
               Articles and Reports                                                  10
             Figure 2: Communication Flow From the Advisory
               Committee to Air Force Scientists Conducting the
               Ranch Hand Study                                                      18




             Page 1                                         GAO/NSIAD-00-31 Agent Orange
Page 2   GAO/NSIAD-00-31 Agent Orange
United States General Accounting Office                                                            National Security and
Washington, D.C. 20548                                                                      International Affairs Division



                                    B-282636                                                                                 Leter




                                    December 17, 1999

                                    The Honorable Lane Evans
                                    Ranking Minority Member
                                    Committee on Veterans Affairs
                                    House of Representatives

                                    Dear Mr. Evans:

                                    From 1962 to early 1971, the United States sprayed herbicides, including
                                    Agent Orange, over Vietnam. In the late 1970s, concerns began to emerge
                                    over the long-term health problems of Vietnam veterans. Although these
                                    veterans could have been exposed to many potential health hazards,
                                    including pesticides, infectious diseases, and treatments for tropical
                                    diseases, attention focused on herbicides. Several herbicides, including
                                    Agent Orange, contain dioxin,1 which is known to cause a variety of
                                    adverse health effects in animals. While there is scientific evidence of some
                                    associations between exposure to herbicides (or the dioxin they contain)
                                    and adverse human health conditions, the effect of this exposure on human
                                    health remains controversial.

                                    There has been long-standing congressional interest in and concern about
                                    the effects of exposure to herbicides such as Agent Orange. Congress has
                                    held at least 26 hearings on the subject since 1978, some of them involving
                                    scrutiny of scientific studies. One key effort to examine the long-term
                                    health effects of servicemembers’ exposure to herbicides in Vietnam is an
                                    ongoing Air Force study known as the Ranch Hand study. This study was
                                    designed to investigate whether exposure to herbicides in Vietnam led or
                                    would lead to adverse health. To this end, the study follows the health
                                    (morbidity) and mortality rates of the Ranch Hands—the almost 1,300 Air
                                    Force personnel who sprayed herbicides from the air in Vietnam—in
                                    contrast to those of a comparison group. This comparison group consists
                                    of Air Force military personnel who were not involved in the spraying and
                                    who were matched to the Ranch Hands in terms of age, race, and military
                                    occupation. The 25-year study, which began in 1982 and is scheduled to end
                                    in fiscal year 2006, is projected to cost over $140 million.



                                    1
                                     The chemical 2, 3, 7, 8-tetrachlorodibenzo-p-dioxin.




                                    Page 3                                                      GAO/NSIAD-00-31 Agent Orange
B-282636




The study’s protocol, which outlines the study’s purpose, methods, and
procedures, requires the Air Force to disclose periodically the study’s
findings in official reports and in a journal of stature. The protocol also
mandated a monitoring group, which currently consists of an Advisory
Committee of nine scientists (three of whom were nominated by veterans’
organizations), to oversee the conduct of the study and provide an
independent review of the study’s findings.

Findings from the Ranch Hand study, along with those of other studies
(e.g., on U.S. and foreign Vietnam veterans, chemical factory workers, or
populations exposed environmentally to dioxin), are used by the
Department of Veterans Affairs to determine whether veterans are eligible
to receive disability compensation for conditions believed to be connected
to military service. In 1984 and again in 1991, Congress revised the process
used by the Department of Veterans Affairs to determine whether Vietnam
veterans are eligible to receive compensation for health effects from
exposures to herbicides.2 Since 1991, the Department has been required to
contract with the National Academy of Sciences to perform reviews of
scientific literature, including the Ranch Hand study, on the associations
between diseases and herbicide exposure. Largely on the basis of these
reviews, the Department of Veterans Affairs determines which diseases
show a positive association between herbicide exposure and the disease.
Vietnam veterans with such diseases are then eligible for disability
compensation.

Since its inception, the Ranch Hand study has been controversial. For
instance, news articles and statements in the Congressional Record have
alleged that government officials delayed and withheld information on the
study’s findings, improperly influenced the study’s design and
implementation, and failed to provide adequate veterans’ representation on
the Advisory Committee. In addition, veterans’ organizations such as the
American Legion and the Vietnam Veterans of America have criticized the
Department of Defense’s conduct with regards to studies of Agent Orange.

Recent congressional concerns have centered not on the scientific design
or implementation of the study but on the proper dissemination and
reporting of study results and on proper independent oversight. On the
basis of your concerns, we examined the conduct and findings of the study
and assessed the study’s impact on determinations of veterans’


2
P.L. 98-542 (1984) and P.L. 102-4 (1991).




Page 4                                           GAO/NSIAD-00-31 Agent Orange
                   B-282636




                   compensation. As agreed with your office, we did not evaluate the study’s
                   scientific design. Specifically, we (1) assessed whether the study’s findings
                   and data have been properly and promptly reported and disseminated;
                   (2) examined the statistical limitations of the study and whether they have
                   been adequately reported and communicated; (3) examined the measures
                   established to monitor the study’s conduct and to prevent improper
                   influence, particularly those involving the Advisory Committee; and
                   (4) assessed the impact of the study on determinations of diseases for
                   which Vietnam veterans are eligible to receive compensation benefits.

                   To examine the conduct and findings of the Ranch Hand study, we obtained
                   and reviewed the study protocol, various study memoranda and
                   correspondence, published study reports and peer-reviewed journal
                   articles, executive summaries and Air Force press releases, and other
                   available documents related to the study. In addition, we interviewed
                   Ranch Hand study investigators, Advisory Committee members, Air Force
                   program officials, Department of Veterans Affairs officials, veterans’
                   representatives, and scientists involved in research on the health effects
                   associated with exposure to herbicides and dioxin. We also visited Brooks
                   Air Force Base to interview members of the Ranch Hand study team and to
                   review data collection and reporting procedures. Our scope and
                   methodology are described in more detail in appendix I.



Results in Brief   To date, the Air Force has conducted all scheduled phases of the Ranch
                   Hand study, including up to five full physical examinations of each
                   participating Ranch Hand and comparison, and has periodically reported
                   the results in official reports. However, there have been some delays over
                   the years in the publication of morbidity and mortality findings in scientific
                   journals and in the update of a key report on reproductive outcomes. The
                   pace of publication has increased in the past few years. No journal articles
                   on mortality or morbidity outcomes were published until 1990, even though
                   the first mortality and morbidity reports were published in 1983 and 1984,
                   respectively, and the Advisory Committee repeatedly recommended that
                   such articles be published. Several reasons contributed to the publishing
                   delays, including the need to collect and analyze data for five other
                   morbidity and mortality reports published between 1985 and 1990. The Air
                   Force reported preliminary reproductive outcomes in 1984 but did not
                   publish a more detailed update until 1992 because it decided to verify the
                   data extensively and perform additional data analyses without releasing
                   any interim findings. Although the number of published reports has
                   increased, public access to study data is limited by the relatively small



                   Page 5                                            GAO/NSIAD-00-31 Agent Orange
B-282636




amount of data currently available and its storage format (magnetic tape),
which is difficult to use. Scientists and veterans’ groups critical of the study
want access to all study data to verify the Air Force’s findings and to
perform additional analyses. To date, the public can only access data the
Air Force analyzed in 1987. The Air Force intends to make all other data
available by the end of year 2000.

Like many epidemiological studies,3 the Ranch Hand study has a number of
inherent limitations, but the Air Force has not clearly or effectively
communicated these limitations to the public. Two limitations are the
difficulty in detecting increases in risks of rare diseases (because of the
small size of the Ranch Hand population) and the fact that the study’s
findings cannot be generalized to all Vietnam veterans (because Ranch
Hands and ground troops were exposed to different levels of herbicides in
different ways). In its first two morbidity reports in 1984 and 1987,
however, the Air Force described the study’s lack of findings (i.e., few
diseases with increased incidence among Ranch Hands) as “reassuring,”
possibly leading to the misinterpretation that the study showed herbicides
to be safe. Although the Air Force no longer uses such language, it still
reports findings to the public through press releases and executive
summaries that do not make the study’s limitations clear.

During the study’s first several years, some measures intended to ensure
that the study is conducted independently and without bias were not
carried out as planned. To ensure independent review and prevent any
appearance of Air Force management bias, the study protocol (published in
1982) mandated: (1) that an independent group of scientists (currently the
Advisory Committee) be established to monitor the study’s conduct and
(2) that Air Force scientists, subject to review by the monitoring group,
have primary responsibility over the scientific aspects of the study.
Shortcomings in the Advisory Committee’s makeup and in the processes
through which the Committee and others communicated scientific advice
were not fully corrected until 1989. Until that year, the Committee did not
include any veterans’ representatives, as required by the study protocol. In
addition, according to documents we reviewed dating from 1984 and 1985,
Air Force management and the White House at the time tried to direct
certain aspects of the Air Force scientists’ research. These attempts
deviated from the protocol’s requirement that Air Force scientists retain


3
 Scientific studies of the incidence, distribution, or control of diseases in human
populations.




Page 6                                                        GAO/NSIAD-00-31 Agent Orange
             B-282636




             primary responsibility over the study’s scientific conduct. Furthermore, the
             White House’s actions bypassed review by the Advisory Committee.
             Although these early problems were resolved through executive and
             congressional actions, some problems remain with the extent of the
             Committee’s outreach to veterans. Although the Committee’s public
             meetings are announced in the Federal Register, the Committee has not
             directly informed veterans’ organizations of these meetings. In addition, the
             Committee’s informal process for soliciting nominations has resulted in not
             all interested veterans’ organizations being notified of opportunities to
             nominate Committee members.

             The Ranch Hand study has had significant impact on one decision
             regarding compensation to Vietnam veterans. Partly on the basis of the
             study’s results, Congress passed legislation providing compensation for
             veterans’ children born with spina bifida.4 The study has not contributed to
             decisions by the Department of Veterans Affairs to compensate Vietnam
             veterans for any other diseases. Because the Ranch Hand study’s small
             sample size provides limited potential for addressing possible links
             between herbicide exposure and many forms of cancer, it has not
             contributed to any National Academy of Sciences or Department of
             Veterans Affairs recommendations regarding cancer. The finding of a
             possible association between herbicide exposure and diabetes mellitus
             (diabetes), first reported by the Ranch Hand study in 1991, has led to
             further review of the issue by the Academy, as well as by Air Force and
             other scientists. However, the Department of Veterans Affairs has not yet
             determined that Vietnam veterans with the disease are eligible for
             compensation.

             We are recommending several additional actions intended to improve
             communication of Air Force Ranch Hand study data and results.



Background   Before the Ranch Hand study began, official government and
             nongovernment reviewers of the study protocol expressed concern that the
             public would perceive the study as not credible because the fact that the
             Air Force would conduct the study might give the appearance of conflict of
             interest. An interagency working group that reviewed the study concluded,
             however, that the appearance of conflict of interest could be properly and

             4
              Spina bifida is a birth defect in which the vertebral column (backbone) fails to close and
             that may allow herniation of the spinal cord.




             Page 7                                                      GAO/NSIAD-00-31 Agent Orange
B-282636




adequately addressed through independent review and monitoring. Thus,
an Advisory Committee was created to regularly review and assess the
conduct of the study, review all changes to the study protocol, and provide
written comments and recommendations on the study’s findings. The
Committee, chartered by the Department of Health and Human Services,
was initially responsible to the Agent Orange Working Group, an
interagency group that monitored all government activities related to
herbicides such as Agent Orange.5 The Agent Orange Working Group was
headed by the Department of Health and Human Services and included
representatives from many government agencies and offices, including the
Department of Defense, the Department of Veterans Affairs, and the White
House Office of Science and Technology Policy. In 1989, the Department of
Health and Human Services transferred administration of the Committee to
the Food and Drug Administration. The duties of the Advisory Committee
are described in more detail in appendix III.

The process of translating scientific evidence into policy decisions on
compensation benefits for veterans has also been controversial. Scientific
studies typically strive to produce evidence of a “cause-and-effect”
relationship, and use a relatively high standard of proof (e.g., 95-percent
confidence). In 1989, a federal court ruled that the Department of Veterans
Affairs’ use of similar standards in making compensation decisions was too
restrictive.6 According to the court, Congress had intended the Department
of Veterans Affairs to require only a “statistical association” with a disease.
Furthermore, the court concluded that veterans were to be given the
benefit of the doubt in compensation decisions. Subsequently, in 1991,
Congress established the current process for these decisions, including
biennial reviews of the scientific literature by the National Academy of
Sciences.7 The Academy has conducted three such reviews for Agent
Orange, the most recent published in 1999. Additional information on the
roles of the Ranch Hand study, the National Academy of Sciences, and the
Department of Veterans Affairs is in appendix IV.




5
The Agent Orange Working Group was replaced by the Agent Orange Task Force in 1990.
The final meeting of the Agent Orange Task Force occurred in 1994.
6
Nehmer v. U.S. Veterans’ Administration, 712 F. Supp. 1404 (N.D.Cal.1989).
7
P.L. 102-4 (1991).




Page 8                                                    GAO/NSIAD-00-31 Agent Orange
                         B-282636




Publication of Study     Publication of study findings in peer-reviewed journals was slow to begin,
                         but the number of articles has increased considerably in recent years (see
Results and Release of   fig. 1). An update of a report on reproductive outcomes was delayed to
Study Data Were Slow     allow the completion of additional analyses and data collection. The Air
                         Force has also been slow to release study data to the public. Only analyzed
in the Early Years       data from one physical examination (not all five examinations conducted
                         thus far or all collected data) is currently available. In addition, the data is
                         available only on magnetic tape rather than a more easily accessible format
                         such as compact disc.




                         Page 9                                              GAO/NSIAD-00-31 Agent Orange
                                         B-282636




Figure 1: Ranch Hand Study Milestones and Published Articles and Reports




                                         Note: “Journal articles” are articles (not including letters, comments, and errata) included in the
                                         National Library of Medicine’s MEDLINE database as of October 1999. “Reports” are official
                                         government reports published by the Air Force.
                                         *: Data currently publicly available.
                                         Source: Air Force, MEDLINE.




                                         Page 10                                                              GAO/NSIAD-00-31 Agent Orange
                         B-282636




Publication of Study     In spite of the requirement in the study protocol as well as guidance from
Findings Slow to Begin   the Advisory Committee that study findings be disclosed periodically in
                         scientific journals, no journal publications appeared until 1989, nearly
                         7 years after the study had begun. Publication of study results on morbidity
                         or mortality did not begin in scientific journals until 1990. The study
                         protocol initially mandated that the Air Force issue official reports on
                         morbidity after each physical examination and on mortality at least every
                         5 years and that the results of the study be published in a journal of stature.
                         The Air Force did not publish journal articles, however, until after it had
                         obtained the third round of physical examinations in 1987 and after several
                         morbidity and mortality reports had been released. Several reasons
                         contributed to the delay, including the need to collect and analyze data for
                         five other morbidity and mortality reports published between 1985 and
                         1990. The Advisory Committee repeatedly urged publication of study
                         findings in journals. In 1986, it recommended that results be published in
                         the “open literature,” and in 1987 it repeated the recommendation, stating
                         that “first and foremost, there is an urgent need to publish in widely read
                         peer-reviewed medical journals.” The pace of article publication has
                         accelerated since 1990. Eleven articles on morbidity and mortality had
                         been published by mid-1999, and Air Force scientists told us that over a
                         dozen articles are currently in submission or preparation for publication in
                         peer-reviewed journals.


Update of Report on      An update of a preliminary reproductive outcome report was delayed
Reproductive Outcomes    8 years because the Air Force performed additional data collection and
                         analyses without releasing any interim information. The Air Force first
Was Delayed
                         reported preliminary reproductive outcomes in a baseline morbidity study
                         in 1984. It subsequently began verifying all medical records to check for
                         birth defects instead of relying on reports of interviews with Ranch Hands,
                         comparisons, and their spouses, which can be inaccurate. The Air Force
                         anticipated completion of the effort within 1 year. Although a draft interim
                         update, using partially verified data, was prepared later in 1984, the
                         Advisory Committee suggested it not be released before additional work
                         was done. The Air Force later expanded its verification of medical records
                         through age 18 (rather than up to age 1) to identify birth defects that may
                         not have manifested themselves earlier. This expanded verification was not
                         completed until late 1988. From 1987 through 1990, the Air Force (in
                         collaboration with the Centers for Disease Control and Prevention)
                         measured dioxin levels in the blood of all study participants as part of its
                         morbidity study and decided to analyze birth defects also using these
                         measurements before publishing a report. The reproductive outcome


                         Page 11                                            GAO/NSIAD-00-31 Agent Orange
                              B-282636




                              update with all the additional analyses and data was finally published in
                              1992.

                              Although the verification and additional analyses ultimately strengthened
                              the scientific basis of the reproductive outcome portion of the study, the
                              length of the delay, in conjunction with the lack of any interim report, led to
                              concerns by some veterans and scientists that the study had been
                              purposely delayed to suppress information. One reason underlying the
                              concern was the apparently high standard of verification. Through our
                              review of other birth defect studies and interviews with scientists, we
                              found that the degree of birth defects verification the Air Force
                              accomplished was highly unusual and virtually unprecedented for a study
                              of its size. We examined two other studies of birth defects in children of
                              Vietnam veterans. One of them verified only the health status of babies at
                              birth, the other verified only those with reported birth defects through age
                              1, not those reported as healthy. Because the Ranch Hand study produced
                              evidence that led Congress to authorize compensation for Vietnam
                              veterans whose children have spina bifida, one veterans’ organization told
                              us that earlier reporting, even just of preliminary findings, might have led to
                              earlier compensation. In addition, one scientist told us that such
                              preliminary information could have been useful for family planning or
                              prenatal care decisions.


Public Access to Study Data   Two major factors currently limit public access to study data: (1) only one
Is Currently Limited          set of analyzed data—not all sets of analyzed data or all collected data—is
                              available and (2) data is only available on magnetic tape, not on a more
                              common format such as compact disc or the Internet. Two scientists and
                              one veterans’ group told us they were interested in obtaining study data.
                              They said that access to study data would enable them to verify the
                              accuracy of the Air Force analyses and to perform additional analyses on
                              the data. They cited the limited amount of study data currently available
                              and its format among the barriers to their use of study data for these
                              purposes.

                              Currently, only data files from Air Force analyses of 1987 physical
                              examination data are available, while data files from 1982, 1985, 1992, and
                              1997 are not. The Air Force database consists of two types of files: (1) raw
                              data files, which include all the data collected at physical examinations,
                              and (2) analysis data files, which were used to perform analyses for study
                              reports. The content of raw and analysis data files overlap, but each
                              contains data not included in the other. The data currently available to the



                              Page 12                                            GAO/NSIAD-00-31 Agent Orange
                         B-282636




                         public are analysis files from 1987. Raw data files from 1987 and 1992 have
                         been processed by the Air Force for release and are in the process of
                         becoming publicly available. Air Force officials report that they expect all
                         raw and analysis data files, including reproductive outcome data, to be
                         available by the end of year 2000. One scientist and one veterans’ group
                         representative told us they believe the Air Force should release all the
                         study’s data to permit additional analysis of the data. According to these
                         individuals, release of the data is particularly important because of past
                         allegations that government officials may have improperly influenced the
                         study’s conduct.

                         Currently, Ranch Hand study scientists send data on computer diskettes to
                         the Defense Technical Information Center, which transfers the data to
                         magnetic tape and sends it to the National Technical Information Service of
                         the Department of Commerce. The Service said it provides the data to the
                         public in the same format in which it is received, and charges about $450 to
                         cover costs. In comparison, two other major government health databases
                         we examined are both available on compact disc. These are the National
                         Cancer Institute’s “Surveillance, Epidemiology, and End Results” database
                         of cancer incidence throughout the United States, available free of charge
                         from the Department of Health and Human Services, and the National
                         Health and Nutrition Examination Survey, a Centers for Disease Control
                         and Prevention database of the health and nutritional status of the U.S.
                         population, that can be purchased for $20 from the Government Printing
                         Office. Several veterans’ representatives said that they do not have the
                         necessary computer equipment to use magnetic tapes, and one scientist
                         told us that other formats such as compact discs would be easier to use. Air
                         Force officials also told us they are planning to release the data on the
                         Internet but noted that the National Technical Information Service would
                         remain their primary means of release.



Study Limitations Have   All scientific studies have inherent limitations. Two major limitations of the
                         Ranch Hand study are the difficulty in detecting increases in risks of rare
Not Been Clearly         diseases (because of the small size of the Ranch Hand population) and the
Communicated to the      fact that the study’s findings cannot be generalized to all Vietnam veterans
                         (because Ranch Hands and ground troops were exposed to different levels
Public                   of herbicides in different ways). In spite of the study’s sensitive and
                         controversial nature, early study reports contained language that may have
                         been misinterpreted to mean that the study showed herbicides were safe.
                         Although the Air Force no longer uses such language, recent press releases




                         Page 13                                           GAO/NSIAD-00-31 Agent Orange
                  B-282636




                  and executive summaries still do not clearly communicate the study’s
                  limitations to the public.


Two Major Study   The small number of participants in the Ranch Hand study—currently
Limitations       fewer than 1,200 Ranch Hands and less than 1,800 comparisons—makes it
                  very difficult to detect a statistically significant increase in the risk of rare
                  diseases. For instance, the lifetime risk of non-Hodgkins lymphoma
                  (a cancer often linked to herbicide exposure) in Caucasian males in the
                  United States is about 2 percent. Even if the Ranch Hand study were to
                  follow participants through their entire lives (which it does not), because
                  of the requirement for statistical significance, it could not confidently
                  report any adverse effect unless the incidence of the disease in Ranch
                  Hands were at least 3.4 percent. Appendix II contains a more detailed
                  explanation of the study’s statistical limitations as they relate to the size of
                  its study population and the detectable increased risk of cancer.

                  Another limitation is that it is difficult to generalize the results of the study
                  to all Vietnam veterans because Ranch Hands were exposed to herbicides
                  in different ways than most ground troops in Vietnam. For instance, by
                  comparing dioxin blood levels with Ranch Hands’ reported exposures to
                  herbicides, Air Force scientists found evidence that Ranch Hands were
                  exposed to herbicides primarily through their skin. Many ground-troop
                  veterans, however, reported exposure to herbicides through other
                  mediums such as contaminated food and water and contaminated clothing
                  worn for extended periods of time. They believe that Ranch Hands were
                  only minimally exposed through these mediums because they consumed
                  uncontaminated food and water, wore clean clothes, and could wash
                  regularly. These differences have not been extensively studied, so little is
                  known about their potential effects.

                  However, blood measurements of dioxin levels suggest, though not
                  conclusively, that Ranch Hands’ exposure levels were significantly higher
                  than both those of the study’s comparison group and of many ground
                  troops who have been tested. 8 Because the comparison group was not
                  involved in spraying herbicides, its low dioxin levels were consistent with

                  8
                   Since 1991, the study’s morbidity reports have concentrated on analyzing the association
                  between dioxin levels and health rather than the health contrast between Ranch Hands and
                  the comparison group. Almost half of the Ranch Hands have dioxin levels similar to those of
                  the comparison group, further reducing the study size when applying the analysis of dioxin
                  levels versus health.




                  Page 14                                                    GAO/NSIAD-00-31 Agent Orange
                              B-282636




                              lack of exposure to herbicides. Congress had proposed a program to test
                              veterans’ blood for dioxin, under the condition that such a program would
                              make a contribution to scientific knowledge, but the National Academy of
                              Sciences stated that such testing would not be useful unless it were part of
                              a specific scientific study. Therefore, most Vietnam veterans’ blood dioxin
                              levels are untested. Furthermore, according to several scientists we
                              interviewed and scientific documents we reviewed, low levels of dioxin in
                              the blood today do not guarantee that an individual had low amounts of
                              exposure to herbicides in Vietnam. Reasons include (1) different herbicides
                              contain different amounts of dioxin; (2) the general population is also
                              exposed to detectable levels of dioxin masking exposures during the
                              Vietnam War; (3) dioxin is slowly excreted from the body, so subjects may
                              have had higher dioxin levels in the past; and (4) the rate at which dioxin is
                              excreted differs, depending on the amount of body fat, so individuals with
                              similar dioxin levels today may have had different exposures in the past.


Early Reports Could Have      The importance of communicating the study’s limitations to the public has
Led to Misinterpretation of   been stressed and highlighted by a number of scientists and officials over
                              the years. As far back as 1980, the predecessor of the Agent Orange
Findings to Mean
                              Working Group9 recommended that the study proceed only on condition
Herbicides Were Safe          that the public “clearly understand that the stated health goal in the Air
                              Force [Ranch Hand study] Protocol may not be fully realized” and that this
                              “does not imply flaws in protocol design; it is to emphasize the inherent
                              limitation of the study size.” In addition, in 1984 the White House Office of
                              Science and Technology Policy urged the Air Force to respond to criticisms
                              and perceived misunderstandings of the study by discussing more
                              explicitly “the meaning of the number of cases found (or not found) … for
                              each disease included in the study,” adding that “the discussion should be
                              in terms of the confidence interval of these numbers and their comparison
                              to general population data on these diseases.”

                              In its early reports, and particularly in executive summaries and press
                              releases, the Air Force used language that may have led the results of the
                              Ranch Hand study to be interpreted inaccurately. As far back as 1980, a
                              review panel of government scientists emphasized that “it needs to be
                              clearly understood that failure to identify increased risk in a variety of
                              health parameters is to be interpreted as inconclusive and not necessarily a


                              9
                               The Scientific Panel of the Interagency Work Group on Phenoxy Herbicides and
                              Contaminants.




                              Page 15                                                  GAO/NSIAD-00-31 Agent Orange
                              B-282636




                              true lack of effect” (emphasis in original). However, in describing the
                              results of the first two morbidity reports, published in 1984 and 1987, the
                              Air Force described the overall study conclusions as “reassuring” because
                              they detected few statistically significant increased risks. The use of such
                              language, as well as a statement by the Deputy Air Force Surgeon General
                              at a press conference in 1984 saying that the results showed “nothing that
                              would keep us from using it [herbicides such as Agent Orange] again,” may
                              have led to the misinterpretation that the Ranch Hand study proved that
                              herbicides were safe. Since 1990, the Air Force has not used this type of
                              language. Current Air Force scientists acknowledge that the word
                              “reassuring” in particular may have led to improper interpretations.


Study Limitations Not Fully   We reviewed all Air Force reports, executive summaries, and press releases
and Clearly Communicated      on the Ranch Hand study and found that the Air Force still reports study
                              results to the public without consistently making clear that any lack of
to the Public in Executive
                              conclusive evidence may be due to the study’s limitations—not necessarily
Summaries and Press           a lack of long-term health effects from herbicides. While study limitations
Releases                      have been addressed in the full text of the reports, they have not been
                              consistently explained in executive summaries and press releases, which
                              are more accessible to the general public. For example, the executive
                              summary and press release for the baseline morbidity study, published in
                              1984, only stated that statistical limits exist, and gave no further
                              description. The executive summary of the third morbidity report,
                              published in 1990, stated that the study had a good chance of detecting a
                              statistically significant effect only if a high number of study subjects
                              (i.e., 5 percent or more in the comparison group, and 10 percent or more in
                              the Ranch Hands) contracted a particular disease. In addition, only in the
                              executive summary and press release of its 1991 morbidity report did the
                              Air Force mention study limitations due to the differences in exposure
                              routes and levels between Ranch Hands and ground troops. However, in
                              this case, the Air Force only stressed the limits with which any positive
                              evidence of ill health could be applied to other Vietnam veterans and did
                              not mention the limits on what conclusions could be drawn from a lack of
                              evidence in the study. All other press releases and executive summaries
                              have not mentioned study limitations at all.

                              Several scientists we spoke with also voiced concern about the Air Force’s
                              public reporting of study results. One scientist with expertise in dioxin
                              research told us that Air Force press releases and executive summaries
                              continue to be misleading because they are not adequately balanced by an
                              account of the limitations of the study. A scientist on the Advisory



                              Page 16                                          GAO/NSIAD-00-31 Agent Orange
                          B-282636




                          Committee also commented at a Committee meeting in August 1999 that
                          the draft of the upcoming Air Force morbidity report stressed the study’s
                          lack of findings too strongly and did not adequately discuss the limits to
                          inferring either positive or negative effects. A former Air Force scientist,
                          however, told us that the Air Force’s public reporting of study findings
                          should be viewed in the context of media and unsubstantiated reports of
                          major incidences of disease and death caused by herbicides.



Many Early Oversight      Two measures specified by the study protocol to ensure independent
                          review and prevent any appearance of Air Force management bias were not
Problems Have Been        followed prior to 1989. These were the requirements that the Advisory
Resolved, but Some        Committee include scientists nominated by veterans’ organizations and
                          that Air Force scientists, subject to review by the Advisory Committee,
Veterans Groups Want      retain primary responsibility over scientific aspects of the study. Executive
More Outreach             agencies (in 1985 and 1989) and Congress (in 1988 and 1990) took actions
                          to correct these problems. However, problems remain with how well the
                          Advisory Committee keeps veterans’ organizations informed of its
                          activities.


Early Problems With the   Prior to 1989, the Advisory Committee did not fulfill a key requirement of
Advisory Committee’s      the study protocol: that it include scientists nominated by veterans’
                          organizations. The requirement stated that about one-third of Committee
Composition and the
                          members should be scientists nominated by veterans’ organizations.
Communication of          According to our review, the Committee did not include any such scientists
Scientific Advice         from 1982 through 1989. Documents we reviewed indicate that one reason
                          was that the Agent Orange Working Group, which was responsible for
                          administering the Committee, appeared to view the protocol’s description
                          of the Advisory Committee as only giving a suggestion.

                          The second problem that existed before 1989 was that outside entities tried
                          to direct how Air Force scientists conducted certain aspects of the study,
                          even though the study protocol specified that Air Force scientists “are
                          responsible for … all data analysis, and for all data interpretation of
                          analyses subject to review by the outside monitoring group.” This problem
                          stemmed from the fact that before 1989, the Committee did not
                          communicate its formal recommendations directly to Air Force scientists
                          conducting the study. Rather, the Committee gave its formal
                          recommendations first to the Agent Orange Working Group for approval,
                          creating opportunities for others to influence the study. Figure 2 shows the




                          Page 17                                           GAO/NSIAD-00-31 Agent Orange
                                        B-282636




                                        flow of communications from the Committee to study scientists before and
                                        after 1989.



Figure 2: Communication Flow From the Advisory Committee to Air Force Scientists Conducting the Ranch Hand Study




                                        Note: Solid lines indicate formal communication (e.g., agency memoranda). Dotted lines indicate
                                        informal communication (e.g., verbal comments or meeting minutes).
                                        Source: Air Force.


                                        Documents we reviewed from the period before 1989 indicate that the Air
                                        Force and the White House Office of Science and Technology Policy



                                        Page 18                                                          GAO/NSIAD-00-31 Agent Orange
                      B-282636




                      (a member of the Agent Orange Working Group) tried to direct certain
                      aspects of the Air Force scientists’ research. For instance, in a
                      memorandum sent in October 1984, the commander of the School of
                      Aerospace Medicine (where the study is conducted), Brooks Air Force
                      Base, ordered the Air Force scientists to perform certain analyses for
                      mortality reports and to present the results in a specific format. The
                      memorandum noted that the direction came from the Air Force Deputy
                      Surgeon General and that it was “in accordance with the comments of the
                      Advisory Committee.” However, according to the study protocol, Air Force
                      scientists, not Air Force management, have ultimate responsibility for all
                      data analysis and interpretations. One of the scientists involved at the time
                      in the study expressed concern that the actions ordered by the commander
                      would weaken the study or de-emphasize the significance of the results. In
                      addition, in a letter also sent in 1984, the White House Office of Science and
                      Technology Policy wrote that it had “directed that the Air Force undertake
                      … actions,” including the extensive verification of reported birth defects.
                      This verification contributed to the 8-year delay in updating study findings
                      on reproductive outcomes. Again, this direction infringed on the Air Force
                      scientists’ decision-making responsibilities in the scientific conduct of the
                      study. Furthermore, the White House later provided guidance directly to
                      the Air Force, requesting that the Advisory Committee ensure that this
                      guidance be implemented. However, the study protocol only authorized the
                      Advisory Committee to provide written comments and recommendations
                      on the conduct of the Ranch Hand study to the White House Office of
                      Science and Technology Policy, not the reverse. The protocol also required
                      that all scientific aspects of the study undergo review by the Advisory
                      Committee. Both Air Force management and the Advisory Committee later
                      expressed concern about the appropriateness of the White House’s
                      communication to the Air Force scientists and sought clarification from the
                      Agent Orange Working Group.


Many Early Problems   In 1985, the Agent Orange Working Group decided that all
Solved by 1990        recommendations intended for the Air Force scientists conducting the
                      study should be communicated first to the Committee. This decision was
                      made primarily in response to concerns about conflicting advice received
                      by the Air Force scientists from bodies such as the White House Office of
                      Science and Technology Policy. We found no evidence that any
                      recommendations made since then have bypassed the Committee. In 1988,
                      Congress mandated that veterans’ organizations be allowed to nominate




                      Page 19                                           GAO/NSIAD-00-31 Agent Orange
                        B-282636




                        three of the Committee’s nine members,10 and we verified the Committee’s
                        compliance with this requirement. In 1989, the Department of Health and
                        Human Services transferred administration of the Committee from the
                        Agent Orange Working Group to the Food and Drug Administration, thus
                        eliminating the Agent Orange Working Group as an intermediary in the
                        chain of communication. Congress passed additional legislation in
                        1990 stating that “no officer or employee of the Federal Government may
                        interfere in or impair direct communication between the Advisory
                        Committee and … [the Air Force] scientists.”11


Remaining Problems in   Although procedural problems in the early years of the study have been
Informing Veterans of   resolved, veterans’ groups and several scientists we spoke with continue to
                        question the credibility of the study. In addition, representatives of
Committee Activities
                        veterans’ groups told us that the Committee’s communication with
                        veterans’ organizations has been intermittent and inconsistent. Two
                        veterans’ groups told us that they had not been notified repeatedly by either
                        the Air Force or the Advisory Committee of public Committee meetings.
                        Although they acknowledged that public Committee meetings are
                        announced in the Federal Register, they stated that similar committees,
                        such as those run by the Department of Veterans Affairs or the National
                        Academy of Sciences, are more proactive in keeping veterans informed of
                        their activities. One group, the Vietnam Veterans of America, which
                        represents veterans with the most interest in herbicides such as Agent
                        Orange, told us that lack of notification persisted even after it had
                        requested directly to the Committee that it be informed of meetings in
                        advance.

                        In addition, the process for soliciting nominations for members of the
                        Advisory Committee has been relatively unstructured and has led to
                        incomplete dissemination of Committee membership openings, as well as
                        inaccuracies in the identification of veterans’ nominees. The Food and
                        Drug Administration is responsible for soliciting nominations from
                        veterans’ organizations. However, openings have not been widely
                        publicized. Rather, the Food and Drug Administration reported using an
                        informal process, which involves asking previous Committee consultants
                        to become nominees and telephone calls to some veterans’ organizations.


                        10
                             P.L. 100-687 section 1205 (1988).
                        11
                             P.L. 101-510 section 1468 (1990).




                        Page 20                                           GAO/NSIAD-00-31 Agent Orange
                             B-282636




                             The Vietnam Veterans of America told us that as a consequence, it has
                             never been asked to nominate Committee members. Our review of
                             Committee documents showed that veterans’ organizations that have
                             nominated Committee members include the American Legion, the Veterans
                             of Foreign Wars, and the Ranch Hand Association. In addition, our review
                             revealed that one current Committee member was incorrectly identified as
                             having been nominated by a veterans’ organization and that another
                             Committee member who was nominated by a veterans’ organization was
                             not identified as such.



Study Has Had Limited        The Ranch Hand study had significant impact on the decision to provide
                             compensation for veterans’ children born with spina bifida. The study has
Impact on                    not contributed to decisions by the Department of Veterans Affairs to
Determination of             compensate veterans for any other diseases. The finding of a possible
                             association between herbicide exposure and diabetes mellitus, first
Diseases for Which           reported by the Ranch Hand study in 1991, has led to greater discussion by
Veterans Are Eligible to     the National Academy of Sciences and further study by Air Force and other
Receive Compensation         scientists, but Vietnam veterans with the disease are not yet eligible for
                             compensation.


Study Contributed to the     The most significant impact of the Ranch Hand study to date has been on a
Decision to Compensate for   decision to allow compensation to Vietnam veterans’ children born with
                             spina bifida. According to a scientist at the National Academy of Sciences,
Spina Bifida
                             because of the high quality of the Ranch Hand birth defect study, the
                             Academy decided in 1996 to upgrade its evaluation of evidence for an
                             association between herbicide exposure and spina bifida from
                             “insufficient/inadequate” to “limited/suggestive.” Subsequently, the
                             Department of Veterans Affairs requested, and Congress approved,
                             legislation allowing the Department of Veterans Affairs to provide
                             compensation to Vietnam veterans’ children with the disease.




                             Page 21                                          GAO/NSIAD-00-31 Agent Orange
                            B-282636




Study Has Not Contributed   The Ranch Hand study has had limited impact on either National Academy
to Decisions to Allow       of Sciences reports or Department of Veterans Affairs decisions concerning
                            associations between herbicide exposure and other diseases.12 Although
Compensation for Other
                            findings from other studies have enabled veterans with any of several
Diseases                    cancers to be eligible for compensation, the Ranch Hand study has had
                            almost no impact on these decisions because of the small size of the Ranch
                            Hand population and the relative rarity of many cancers. In addition,
                            scientific studies such as the Ranch Hand study are involved only in
                            determining for which diseases veterans may be eligible for compensation,
                            not in any other part of the compensation process. There are other
                            requirements, such as obtaining medical diagnosis of a disease and
                            determination of disability, that Vietnam veterans must satisfy in order to
                            receive compensation. Moreover, reports and articles by the Ranch Hand
                            study comprise only a small fraction of the information the National
                            Academy of Sciences reviews and the Department of Veterans Affairs then
                            considers when weighing scientific evidence.

                            A finding of a possible association between herbicide exposure and
                            diabetes mellitus was first suggested in a Ranch Hand study report in 1991.
                            This and subsequent findings by the study have led to increased research
                            on the subject and more reviews by the National Academy of Sciences.
                            Although in its February 1999 report the Academy described the evidence
                            of an association between herbicide exposure and diabetes mellitus as
                            “equivocal,” the Academy still concluded, as it did in its earlier reports, that
                            there was “inadequate/insufficient” evidence for such an association.
                            However, a Task Force within the Department of Veterans Affairs
                            recommended in April 1999 to allow Vietnam veterans with diabetes
                            mellitus to become eligible for compensation. The Task Force wrote that
                            because the weights of positive and negative evidence were approximately
                            equal, the requirement to give veterans the benefit of the doubt (as stated
                            by law) necessitated providing compensation for diabetes mellitus. The
                            Department of Veterans Affairs has asked the Academy to conduct an
                            additional evaluation of the scientific literature on the subject. The
                            evaluation is scheduled to be completed by the end of 1999.




                            12
                               Currently, the Department of Veterans Affairs allows compensation for nine diseases
                            (including six cancers) in veterans and one birth defect in children of veterans.




                            Page 22                                                    GAO/NSIAD-00-31 Agent Orange
                  B-282636




Conclusions       The Air Force has conducted many aspects of the Ranch Hand study in a
                  rigorous manner. However, some past and present problems in the conduct
                  of the study have led some veterans to view the study with suspicion and to
                  express concerns about the study’s conduct. In particular, several veterans’
                  groups and scientists are critical of the study’s methods and results and
                  want greater access to study data. Until all study data is publicly available,
                  people critical of the study cannot fully verify the Air Force’s analyses.
                  Making all study data publicly available would increase the credibility of
                  Ranch Hand study results. In addition, without full access to data, outside
                  scientists cannot perform alternative or additional analyses, which would
                  facilitate open scientific debate on the merits of the Air Force’s
                  methodologies and analyses. The Air Force anticipates making all its study
                  data available to the public by the end of year 2000, but those with interest
                  in the data say that unless the data is available in a more easily accessible
                  format (such as compact disc or the Internet), they would have difficulty in
                  using it.

                  In addition, while communication of study limitations to the public has
                  improved over the years, additional improvements are possible. Lack of
                  sufficient knowledge of the study’s limitations can lead to misleading or
                  incorrect interpretations of the study’s findings. Language used in early
                  reports that may have led to such interpretations is no longer being used,
                  but because of the study’s sensitive and controversial nature, it is important
                  that publicly accessible documents such as executive summaries and press
                  releases provide accurate information on the study’s limitations.

                  Early problems in the Advisory Committee’s composition and
                  communications have been remedied, but problems remain with the
                  Committee’s outreach to veterans. The Committee’s reliance on indirect
                  means, such as the Federal Register, to notify veterans’ organizations of its
                  activities perpetuates the impression that the Committee does not seek
                  veterans’ input. Moreover, better notification of Committee openings would
                  help ensure that veterans’ groups perceive the Committee as fulfilling its
                  role as an independent and unbiased oversight body.



Recommendations   To facilitate public access to study data and more effective communication
                  of study limitations, we recommend that the Secretary of Defense direct
                  the Air Force scientists in charge of the Ranch Hand study, in consultation
                  with the Advisory Committee, to




                  Page 23                                           GAO/NSIAD-00-31 Agent Orange
                  B-282636




                  • establish and publicize a timetable for the release of all study data and
                    provisions to release the data in a format (such as compact disc or the
                    Internet) that is easily accessible to the general public and
                  • include more information on the study’s limitations in press releases and
                    executive summaries, which should address the limited applicability of
                    study results (especially negative results) to other Vietnam veterans and
                    the limited ability of the study to detect small to moderate increases in
                    risks of rare diseases.

                  In addition, to facilitate dissemination of information on the Advisory
                  Committee’s activities to veterans, we recommend that the Secretary of
                  Health and Human Services direct the Committee’s Executive Secretary at
                  the Food and Drug Administration to provide direct and timely notification
                  to veterans’ organizations of scheduled Advisory Committee meetings and
                  of opportunities for veterans to nominate Committee members.



Agency Comments   In written comments on a draft of this report, the Department of Defense
                  concurred with our recommendations and indicated that it is taking steps
                  to address them. The Department also commented on our finding that the
                  publication of study findings in scientific journals was slow to begin,
                  stating that the time required to write and publish journal articles is
                  typically 3-5 years and that work on the papers published in 1990 actually
                  started in 1986. While we recognize that the publication process can be
                  lengthy, it should be noted that the journal articles published in 1990 only
                  included analysis of data collected in 1987 and reported in 1990 in an
                  official Air Force report. No journal articles were published earlier based
                  on data collected in 1982 and 1985 and respectively released in Air Force
                  reports in 1984 and 1987. Other comments from the Department were
                  technical in nature, and we incorporated them where appropriate.
                  Comments from the Department of Defense are reproduced in their
                  entirety in appendix V.

                  The Food and Drug Administration also provided written comments on a
                  draft of this report. It stated that in general it found the report well done
                  and accurate. With regard to our recommendation that the Secretary of
                  Health and Human Services direct the Advisory Committee’s Executive
                  Secretary to facilitate dissemination of information to veterans, it indicated
                  that they are working with the Office of Veterans Affairs and Military
                  Liaison of the Department of Health and Human Services to ensure that
                  veterans’ organizations are notified in a timely manner of Advisory
                  Committee meetings and vacancies. The Food and Drug Administration



                  Page 24                                           GAO/NSIAD-00-31 Agent Orange
B-282636




also provided technical comments that we incorporated where
appropriate. Comments from the Food and Drug Administration are
reproduced in their entirety in appendix VI.

As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from its
issue date. At that time, we will send copies of this report to other
appropriate congressional committees. We will also send copies to the
Honorable William S. Cohen, Secretary of Defense; the Honorable
F. Whitten Peters, Secretary of the Air Force; and the Honorable
Donna E. Shalala, Secretary of Health and Human Services. In addition, we
will make copies available to others upon request.

If you have any questions about this report, please call me at
(202) 512-3652 or John Oppenheim at (202) 512-3111. Weihsueh Chiu was a
key contributor to this report.

Sincerely yours,




Kwai-Cheung Chan
Director
Special Studies and Evaluations




Page 25                                           GAO/NSIAD-00-31 Agent Orange
Appendix I

Objectives, Scope, and Methodology                                                         AA
                                                                                            ppp
                                                                                              ep
                                                                                               ned
                                                                                                 nx
                                                                                                  idx
                                                                                                    eIis




              Our objectives were to (1) assess whether the study’s interim findings and
              data have been properly and promptly reported and disseminated,
              (2) examine the statistical limitations of the study and whether they have
              been adequately reported and communicated, (3) examine the measures
              established to monitor the study’s conduct and to prevent improper
              influence, and (4) assess the impact of the Ranch Hand study on National
              Academy of Sciences reports and Department of Veterans Affairs
              determinations regarding diseases for which Vietnam veterans are eligible
              to receive compensation benefits.

              To assess whether the study’s interim findings and data have been properly
              and promptly reported and disseminated, we reviewed the study protocol
              and memoranda and correspondence by the Air Force and the Advisory
              Committee to determine the schedule for reporting study findings. In
              addition, we reviewed these documents to examine the reasons for any
              delays. We also obtained and reviewed published study reports and
              peer-reviewed articles to determine the actual date of these publications.
              We conducted site visits to Brooks Air Force Base to interview scientists in
              charge of the Ranch Hand study and to review data collection and reporting
              procedures. Finally, we reviewed the current state of public access to study
              data and queried several scientists and veterans’ organizations about their
              ability to access the data.

              To examine the statistical limitations of the study and the reporting and
              communication of these limitations, we reviewed the study protocol, the
              study reports, and the memoranda and correspondence obtained from the
              Air Force and the Advisory Committee. In addition, we reviewed literature
              on epidemiology and communication of health risks. We reviewed study
              reports, executive summaries, and Air Force press releases to assess how
              study limitations have been communicated. We also queried scientists
              about aspects of communicating study limitations. Using the National
              Cancer Institute’s Surveillance, Epidemiological, and End Results database,
              we determined the expected incidence of several cancers in Ranch Hands
              based on the rate in the U.S. population and performed a calculation of the
              statistical power of the study (see app. II).

              To examine measures established to monitor the study’s conduct and to
              prevent improper influence, we reviewed the study protocol, memoranda,
              and correspondence obtained from the Air Force and the Advisory
              Committee and documentation provided by the Food and Drug
              Administration on veterans’ representation on the Committee. We
              documented the roles and activities of key organizations involved in



              Page 26                                          GAO/NSIAD-00-31 Agent Orange
Appendix I
Objectives, Scope, and Methodology




making scientific recommendations to Air Force scientists and compared
them with requirements outlined in the study protocol. We also reviewed
requirements on the selection and composition of the Ranch Hand
Advisory Committee and reporting requirements as stated by law.

To assess the impact of the Ranch Hand study on National Academy of
Sciences reports and Department of Veterans Affairs determinations of
diseases for which veterans are eligible to receive compensation benefits,
we reviewed laws outlining the process to determine for which diseases
Vietnam veterans are eligible for compensation. We then obtained and
reviewed reports by the Academy and the Department of Veterans Affairs.
In addition, we interviewed officials from these organizations to determine
how research results lead to compensation decisions. We also reviewed
those portions of the Federal Register that announce and codify regulations
governing compensation of Vietnam veterans.

In conducting our study, we contacted current and former members,
affiliates, or representatives from the following organizations: Advisory
Committee, Executive Secretary at National Center for Toxicological
Research, Food and Drug Administration; Air Force Research Laboratory,
Brooks Air Force Base, San Antonio, Texas; Air Force Surgeon General,
Bolling Air Force Base, Washington, D.C.; American Legion, Washington,
D.C.; Department of Veterans Affairs, Washington, D.C.; National Academy
of Sciences, Washington, D.C.; Veterans of Foreign Wars, Washington, D.C.;
and Vietnam Veterans of America. In addition, we spoke with several
scientists involved in research on the health effects associated with
exposure to herbicides and dioxin.

We performed our work from January through October 1999 in accordance
with generally accepted government auditing standards.




Page 27                                         GAO/NSIAD-00-31 Agent Orange
Appendix II

Statistical Power of the Ranch Hand Study                                                       Appendx
                                                                                                      iI




               This appendix describes in more detail the limited statistical power of the
               study caused by the small size of the Ranch Hand population. There were a
               total of about 1,200 Ranch Hands in 1982, and there are currently fewer
               than 1,150 still alive. An estimate of the ideal statistical power of the Ranch
               Hand study from 1982 to the last physical examination, to be performed in
               2002, is summarized for several cancers in tables 1-3. The last column of
               each table shows the expected number of cases among Ranch Hands
               necessary for the study to be likely to detect an increased risk of cancer
               with 95-percent significance. Statistical power increases with time because
               cancers, like most health problems, are more common among older people.
               This calculation is “ideal” because it assumes that all Ranch Hands and
               comparisons participate and that none of them die. It also does not account
               for a possible “healthy-soldier effect,” in which military populations are
               healthier than nonmilitary ones because entrance into military service
               requires passing a medical examination that screens out applicants with
               chronic illnesses and because remaining in the service requires maintaining
               good enough health to perform relatively rigorous duties. Because these
               numbers represent what the study could detect under the most ideal
               conditions, the actual power of the study is probably less than indicated
               here.

               As shown in these tables, the study cannot detect a doubling of cancer
               incidence (relative risk of at least 2) until at least 10 years into the study for
               many types of site-specific cancers. For all sites combined, the study can
               detect a doubling in risk.




               Page 28                                              GAO/NSIAD-00-31 Agent Orange
Appendix II
Statistical Power of the Ranch Hand Study




Table 1: Minimum Detectable Effect in 1982
                                                                        Cases expected
                                        Number of                          among Ranch
                                            cases                                  Hands
                       Cumulative        expected         Minimum      corresponding to
                         expected          among      relative riska   minimum relative
                       percentage      comparison       likely to be     risk likely to be
Cancer site            with cancer          group          detected              detected
Liver                        0.012            0.14               42                   6.0
Prostate                     0.088             1.0              7.5                   7.9
Non-Hodgkins
Lymphoma                      0.10             1.2              6.7                   8.3
Melanoma                      0.17             2.0              4.9                   9.7
All anatomical sites           2.2             26               1.7                    43




Table 2: Minimum Detectable Effect in 1992
                                                                        Cases expected
                                         Number of                         among Ranch
                                             cases                                 Hands
                        Cumulative        expected    Minimum          corresponding to
                          expected          among relative riska       minimum relative
                        percentage      comparison likely to be          risk likely to be
Cancer site             with cancer          group    detected                   detected
Liver                         0.053            0.63              11                   7.1
Prostate                         1.1            13              2.0                    27
Non-Hodgkins
Lymphoma                       0.36             4.3             3.1                    14
Melanoma                       0.49             5.9             2.7                    16
All anatomical sites             7.8            94              1.3                   122




Page 29                                                   GAO/NSIAD-00-31 Agent Orange
Appendix II
Statistical Power of the Ranch Hand Study




Table 3: Minimum Detectable Effect in 2002
                                                                                        Cases expected
                                                 Number of                                 among Ranch
                                                     cases                                         Hands
                             Cumulative           expected             Minimum         corresponding to
                               expected             among          relative riska      minimum relative
                             percentage         comparison           likely to be        risk likely to be
Cancer site                  with cancer             group              detected                 detected
Liver                                  0.18                 2.1                4.6                        9.9
Prostate                                5.5                  65                1.4                        90
Non-Hodgkins
Lymphoma                               0.86                  10                2.2                        22
Melanoma                                1.2                  15                1.9                        28
All anatomical sites                     22                258                 1.2                       299


a
  The minimum relative risk is the smallest relative risk that must exist in order for the likelihood that the
study can detect the effect with 95-percent significance to be greater than 50 percent. The last column
is the corresponding minimum number of cases expected among Ranch Hands in order for the study
to detect an adverse effect. Numbers are rounded to two digits unless otherwise shown.
Note: The Ranch Hands used in this analysis were the 1,197 who were identified and alive as of
December 31, 1982. An identical number of comparisons were assumed. They were stratified by birth
in 5-year bins, with blacks and non-blacks considered separately. The age-specific cancer incidence
rates for 1973-96 (from the Surveillance, Epidemiological, and End Results database) were used.
Years before 1973 used the 1973 rates and years after 1996 used the 1996 rates. The cumulative
incidence was calculated assuming all Ranch Hands left Vietnam in 1967.
Sources: GAO analysis of National Cancer Institute and Air Force data.




Page 30                                                                GAO/NSIAD-00-31 Agent Orange
Appendix III

Duties of the Ranch Hand Advisory
Committee                                                                                   Appendx
                                                                                                  Ii




               This appendix discusses the duties of the Advisory Committee in
               overseeing the Ranch Hand study. In general, the Committee conducts
               scientific reviews of the study and provides the Air Force with comments
               and advice. However, this advice, which may be in the form of either formal
               or informal recommendations, is nonbinding.

               The Committee’s primary purpose, as described in the study protocol, is to
               review and assess the study’s conduct. The Committee performs this role
               through both site visits and by reviewing study reports before their release.
               Site visits include visits to Brooks Air Force Base, where the Air Force
               scientists are located, and to locations where physical examinations are
               performed. The Committee also reviews all official Air Force reports on the
               study prior to release, including executive summaries. The Committee does
               not review peer-reviewed journal articles but has consistently encouraged
               the Air Force to publish its results in these journals. The Committee
               convenes meetings when the Air Force has drafted reports to be reviewed
               and has not met on a regular basis.

               According to the study protocol, the Committee should also review
               suggested changes in the data collected and methods of analysis used by
               the study. To this end, the Committee reviews the content of physical
               examinations, along with the statements of work for those examinations. In
               addition, the Air Force has contracted out the overall management,
               logistics, statistical analyses, and writing of Air Force morbidity reports
               subsequent to the first report in 1984. The Committee reviews plans for
               statistical analyses.




               Page 31                                           GAO/NSIAD-00-31 Agent Orange
Appendix IV

Roles of Air Force, National Academy of
Sciences, and Department of Veterans Affairs
in Evaluating Health Effects of Herbicides                                                    Appendx
                                                                                                    IV
                                                                                                     i




               In evaluating the health effects of herbicides, the Air Force, the National
               Academy of Sciences, and the Department of Veterans Affairs have
               different goals, use different criteria, and produce different types of
               products. These differences are illustrated in table 4.

               The goal of the Ranch Hand study is to determine whether health effects
               are attributable to herbicide exposure, while the goals of the Academy and
               the Department of Veterans Affairs emphasize establishing only an
               association. Although statistical methods are the primary tools the Air
               Force uses in the study, the “strength of association” of a result is only one
               of the many criteria Air Force scientists use in evaluating evidence of a
               causal relationship between adverse health and exposure to herbicides.
               The National Academy of Sciences and the Department of Veterans Affairs,
               on the other hand, both look primarily at evidence of a “statistical
               association” between diseases in humans and exposure to herbicides. The
               Academy considers evidence of a causal relationship separately.

               The Ranch Hand study and the reviews by the National Academy of
               Sciences apply a relatively high standard in evaluating evidence. The
               Department of Veterans Affairs, on the other hand, is charged with using a
               specified standard of evidence intended to give veterans the “benefit of the
               doubt.” Specifically, a “positive association” for the Department of Veterans
               Affairs is defined by law as when “the credible evidence for the association
               is equal to or outweighs the credible evidence against the association.”
               Although both the Air Force and the National Academy of Sciences report
               findings of lesser significance, their emphasis is on establishing
               relationships in which they are reasonably confident. Air Force scientists
               noted, for instance, that though not all their criteria need to be satisfied for
               a finding to be reported, the more indicators that point to a causal
               relationship, the stronger the finding. The Academy defines “sufficient”
               evidence as findings in which “a positive association has been observed …
               in studies in which chance, bias, and confounding could be ruled out with
               reasonable confidence.” A second Academy category, one of
               “limited/suggestive” evidence, includes situations in which “evidence is
               suggestive of an association … but is limited because chance, bias, and
               confounding could not be ruled out with confidence.”




               Page 32                                             GAO/NSIAD-00-31 Agent Orange
                                               Appendix IV
                                               Roles of Air Force, National Academy of
                                               Sciences, and Department of Veterans Affairs
                                               in Evaluating Health Effects of Herbicides




Table 4: Summary of Roles and Criteria in Evaluating the Health Effects of Herbicides
             Goal(s)                  Evidence evaluated          Standards/criteria used              Product(s)          Uses of products
Air Force  Existence of health      Data collected by Air         Plausibility.                        Ranch Hand          National Academy
Ranch      effects in Vietnam       Force on Ranch Hands          Internal and external                study official      of Sciences
Hand study veterans attributable to and comparison group.         consistency.                         reports.            reviews.
           exposure to herbicides.                                Strength of association.             Peer-reviewed       Department of
                                                                  Statistical power.                   journal articles.   Veterans Affairs
                                                                  Biological mechanism.                                    compensation
                                                                  Validity or bias.                                        determinations.
                                                                  Statistical significance.                                Scientific base of
                                                                  Statistical assumptions.                                 knowledge.
                                                                  a
National   Existence of statistical   Scientific literature on     Extent to which scientific data     Biennial reports:   Department of
Academy of association between        effects of herbicides in    permit determinations.               Veterans and        Veterans Affairs
                                                                  a
Sciences   disease in Vietnam         humans.                      Strength of evidence.               Agent Orange.       compensation
                                                                  a
           veterans and                                            Appropriateness of methods.                             determinations.
           herbicides.                                                                                                     Recommendations
                                                                                                                           for future studies.
                                                                  a
             Existence of an          Scientific literature on     Extent to which scientific data
             increased risk among     Vietnam veterans and        permit determinations.
             Vietnam veterans.        other human
                                      populations.
                                                                  a
             Existence of a plausible Scientific literature on     Extent to which scientific data
             biological mechanism     effects of herbicides in    permit determinations.
             or other causal          humans and animals.         Validity of animal models.
             relationship.                                        Biochemical origin of dose
                                                                  response.
                                                                  a
Department Existence of a positive    National Academy of          Whether evidence for an             Department Task     Compensation
of Veterans association between       Sciences review of          association equals or outweighs      Force Reports.      determinations.
Affairs     disease in humans and     scientific literature.      evidence against an                  Regulations of      Recommendations
            herbicide exposure.       Other available             association, considering             diseases with       for future studies.
                                      scientific and medical      (a) academy reviews,                 presumption of
                                      information and             (b) statistical significance,        service
                                      analyses.                   (c) capability of replication, and   connection.
                                                                  (d) capability to withstand peer
                                                                  review.


                                               a
                                               Criterion required by law.
                                               Sources: Air Force, Public Law, National Academy of Sciences, Department of Veterans Affairs, and
                                               Federal Register.




                                               Page 33                                                         GAO/NSIAD-00-31 Agent Orange
Appendix V

Comments From the Department of Defense                   Appendx
                                                                V
                                                                i




             Page 34           GAO/NSIAD-00-31 Agent Orange
             Appendix V
             Comments From the Department of Defense




See p. 24.




See p. 24.




See p. 24.




             Page 35                                   GAO/NSIAD-00-31 Agent Orange
Appendix V
Comments From the Department of Defense




Page 36                                   GAO/NSIAD-00-31 Agent Orange
Appendix VI

Comments From the Food and Drug
Administration                                           Appendx
                                                               V
                                                               iI




              Page 37         GAO/NSIAD-00-31 Agent Orange
                         Appendix VI
                         Comments From the Food and Drug
                         Administration




p




    See p. 24.




(713035)         Leter
                         Page 38                           GAO/NSIAD-00-31 Agent Orange
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested