Environmental Protection: Assessing Impacts of EPA's Regulations Through Retrospective Studies

Published by the Government Accountability Office on 1999-09-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting Office

GAO               Report to Congressional Requesters

September 1999
                  Assessing the Impacts
                  of EPA’s Regulations
                  Through Retrospective

      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Resources, Community, and
      Economic Development Division


      September 14, 1999

      The Honorable Bud Shuster
      Committee on Transportation and Infrastructure
      House of Representatives

      The Honorable Frank R. Lautenberg
      United States Senate

      Over the past three decades, the Environmental Protection Agency (EPA)
      has issued and enforced regulations, pursuant to federal statutes, to
      enhance environmental quality. These regulations impose costs, such as
      compliance expenses for the regulated entities, and produce benefits, such
      as improved human health and recreational opportunities. These costs and
      benefits are substantial. In 1997, the Office of Management and Budget
      (OMB) estimated that federal environmental regulations cost about
      $144 billion annually and produced benefits of about $162 billion.1

      Under Executive Order 12866, EPA prepares detailed cost-benefit analyses
      for all economically significant regulations—including, among others,
      those expected to have an annual impact on the economy of $100 million
      or more.2 These economic assessments inform and improve the regulatory
      process by prospectively estimating the future costs and benefits of
      feasible alternatives. Also, under this order, EPA periodically reviews
      existing regulations that are economically significant to determine if they
      can be made more effective or less burdensome. The order, however, does
      not specify the nature or the extent of this review. One tool that can be
      used in considering such regulatory changes is a retrospective study—an
      assessment of the actual costs and benefits.

      Concerns have been raised, however, that agencies have made only limited
      attempts to assess the actual impacts of federal regulations. For example,
      in May 1998, the Senate Committee on Governmental Affairs reported that
      the review of existing rules, as directed by executive order, “has met with
      varying degrees of failure. Clearly, getting agencies to review existing rules

       In a 1998 report, OMB estimated that the costs of federal environmental regulations were between
      $120 billion and $170 billion and the benefits could be as high as $3.3 trillion annually. The controversy
      surrounding this estimate is discussed in our report, Regulatory Accounting: Analysis of OMB’s
      Reports on the Costs and Benefits of Federal Regulation (GAO/GGD-99-59, Apr. 20, 1999).
      The executive order was issued on September 30, 1993. EPA is also directed by the Unfunded
      Mandates Reform Act of 1995 to conduct regulatory cost-benefit analyses under certain circumstances.

      Page 1                            GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                   is much easier said than done.”3 In light of these concerns, you asked us
                   to (1) determine to what extent EPA’s regulations have been the subject of
                   retrospective studies, (2) identify in what ways retrospective studies are
                   viewed as useful or difficult to do, and (3) identify ways to foster such
                   studies in the future.

                   Assessments of the costs and the benefits of EPA’s regulations after they
Results in Brief   have been issued have rarely been done. Of the 101 economically
                   significant regulations issued by EPA from 1981 through 1998, only five
                   were the subject of retrospective studies.4 Various sources—including the
                   Congress, EPA, and academia—have provided the impetus for these studies,
                   which were all completed from 1997 through 1999. These studies covered
                   regulations designed to, among other purposes, control acid rain, phase
                   out chlorofluorocarbons, and reduce air pollution through improved
                   vehicle inspection and maintenance.

                   Authors of retrospective studies, EPA officials, and other experts told us
                   that these studies have been or can be useful but also pose a number of
                   difficulties for researchers. In terms of usefulness, for example,
                   retrospective studies have provided insights into a new, market-based
                   regulatory approach to reduce emissions that cause acid rain. These
                   studies found that the actual costs of reducing emissions were lower than
                   initially estimated. Based on these findings, a legislative proposal to
                   further limit emissions has been introduced. Retrospective studies are
                   viewed as difficult to do because, among other reasons, it is difficult to
                   obtain valid cost data from regulated entities. For example, corporate
                   accounting and financial records do not typically capture the information
                   necessary to determine the incremental costs of complying with federal
                   environmental regulations, and companies have little incentive to isolate
                   and monitor these costs. Also, it is extremely difficult to quantify actual
                   benefits. One reason is that a significant time lag usually exists between
                   the elimination or major reduction of pollutants and any corresponding
                   change in illness rates.

                   We found that a systematic approach by EPA would foster retrospective
                   studies in the future. While all economically significant regulations, by
                   executive order, are subject to a prospective economic assessment, EPA is

                    The Congress has considered a legislative proposal that would require agencies to review or “look
                   back” at existing regulations. To see our testimony on this proposal, see Regulatory Reform:
                   Comments on S. 981—The Regulatory Improvement Act of 1997 (GAO/T-GGD/RCED-97-250, Sept. 12,
                    According to EPA, another 14 economically significant regulations have been issued in 1999.

                   Page 2                           GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                            not expected to calculate the actual costs and benefits for all of these
                            regulations. Accordingly, EPA officials told us that they have used the
                            agency’s limited resources on prospective economic assessments rather
                            than on retrospective studies. Based on discussions with study authors,
                            EPA officials, and other experts, we found that the resource constraints and
                            the difficulties in conducting retrospective studies could be better
                            addressed if EPA were to develop a plan to foster studies of the actual costs
                            and benefits of environmental regulations. We are recommending that EPA
                            develop a plan based on a systematic (1) identification of candidate
                            regulations for study, (2) assessment of the feasibility of conducting
                            retrospective studies of those regulations, and (3) identification of the
                            resources needed.

                            EPA’s regulations have rarely been the subject of a retrospective study.5
Few of EPA’s                According to EPA, the agency issued 101 economically significant
Regulations Have            regulations from 1981 through 1998,6 and five of these regulations have
Been the Subject of         been the subject of retrospective studies. In addition, of the more than
                            2,600 environmental regulations issued during this period that were not
Retrospective Studies       economically significant, 23 were the subject of retrospective studies.

                            As listed below, the retrospective studies we identified ranged from broad
                            studies, such as a report on the impacts of the entire Clean Air Act, to
                            studies of individual regulations, such as the phaseout of
                            chlorofluorocarbons (CFC). All were completed from 1997 through 1999.
                            Most of the studies had prospective cost estimates against which actual
                            costs could be compared. Several also had information on actual benefits.

                        •   Section 812 of the 1990 amendments to the Clean Air Act required EPA to
                            provide information about the economic costs and benefits and the health,
                            welfare, and environmental impacts of the Clean Air Act. EPA issued the
                            Section 812 study in October 1997.7
                        •   The 1990 amendments to the Clean Air Act directed an interagency group
                            to report every 4 years, beginning in 1996, on the costs, the benefits, and
                            the effectiveness of the acid rain program. The group, known as the

                             We considered a study to be a “retrospective” study if, at a minimum, it contained information on the
                            costs of existing regulations and focused on a specific regulation or a set of regulations with common
                             We used 1981 as the starting point because EPA began tabulating the number of its regulations at that
                             When counting the number of regulations that were the subject of retrospective studies, we excluded
                            the regulations covered in the Section 812 study because it aggregated the costs for a number of
                            existing regulations and did not identify the incremental costs of specific regulations.

                            Page 3                           GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

    National Acid Precipitation Assessment Program, issued the first
    retrospective study under this mandate in 1998.8
•   Other retrospective studies involving EPA’s regulations include 23 studies
    on pesticides and individual studies on the phaseout of CFCs, sulfur dioxide
    reductions, vehicle inspection and maintenance, and reformulated
    gasoline. The sulfur dioxide report was the only one of these studies to
    include information on benefits.

    Various sources provided the impetus for the retrospective studies we
    identified. The Congress mandated the Section 812 and acid rain studies.
    The latter study was funded under interagency agreements with EPA and
    with other federal agencies.9 EPA’s Office of Policy signed a research
    cooperative agreement with Resources for the Future (RFF),10 11 which, in
    turn, initiated the 23 pesticide studies and the vehicle inspection and
    maintenance study. EPA’s Office of Air and Radiation’s Acid Rain Division
    partially funded the sulfur dioxide study through cooperative agreements
    with the Massachusetts Institute of Technology and RFF. Two retrospective
    studies were completed without funding from EPA. The CFC phaseout study
    was conducted by a faculty member of Harvard University and funded by
    Harvard University, and the reformulated gasoline study was prepared at
    the researchers’ expense.

    In addition to the retrospective studies that have been completed, others
    are underway. EPA’s Offices of Policy and Ground Water and Drinking
    Water have contracted for a retrospective study of EPA’s drinking water
    standards for atrazine and nitrate.12 EPA expects the study to be completed
    by September 30, 1999. Also, under a cooperative agreement with EPA, RFF
    is conducting another series of retrospective studies on pesticide

     Title IV of the Clean Air Act amendments seeks to reduce the adverse effects of acid rain through
    reductions in annual emissions of sulfur dioxide and nitrogen oxides from utilities burning fossil fuels.
     Agencies in the National Acid Precipitation Assessment Program include EPA; the departments of
    Interior, Energy, and Agriculture; the National Aeronautics and Space Administration; and the National
    Oceanic and Atmospheric Administration.
      Generally, under this type of cooperative agreement, EPA agrees to a research agenda proposed by
    the recipient and provides funds without specifying how the research should be done. Also, the
    recipient shares in the cost of the research.
      RFF is an independent, nonprofit organization engaged in research and public education on natural
    resources and environmental issues.
     Atrazine is a herbicide and nitrate is an inorganic chemical present in fertilizers and animal wastes.
    These chemicals can enter drinking water by runoff into surface water and leaching into groundwater.

    Page 4                            GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                            According to the experts we interviewed, while retrospective studies have
Retrospective Studies       provided valuable insights they also have been difficult to perform.
Are Viewed as Useful
but Also Difficult to
Retrospective Studies Are   Authors of retrospective studies, EPA officials, and other experts we
Seen as Useful              interviewed identified three primary areas where retrospective studies
                            have been or can be useful by providing (1) information for consideration
                            of legislative or regulatory modifications, (2) insights into the need for
                            methodological improvements in prospective economic assessments, and
                            (3) information for EPA’s use in implementing the Government
                            Performance and Results Act (the Results Act).

Modifying Legislation or    Retrospective studies provide information used in considering legislative
Regulations                 and regulatory modifications—especially when a new regulatory approach
                            has been taken. Some time after implementation of a regulation, a
                            comparison of post- and pre-issuance estimates of the costs and the
                            benefits provides insights into whether that regulation is working as
                            planned, whether the burden on regulated entities is appropriate, and
                            whether unanticipated consequences are occurring.

                            For example, title IV of the 1990 amendments to the Clean Air Act
                            authorized EPA to adopt a new regulatory approach to the acid rain
                            problem instead of the traditional “command and control” approach,
                            under which pollution sources are required to install certain pollution
                            control technologies or meet specific performance standards. The new
                            approach provided market-based incentives. It allocated a number of
                            allowances for sulfur dioxide emissions to each utility under the program.
                            Each allowance represents the limited authority of a utility to emit one ton
                            of sulfur dioxide. The Act set an overall goal and capped the number of
                            allowances to ensure an overall reduction in emissions. Each utility may
                            choose to reduce its own emissions or to purchase unused allowances
                            from another utility. Furthermore, a utility may choose to “bank” its
                            allowances for sale at a later date. The allowances trade freely, as stocks

                            Retrospective studies indicate that the actual cost of reducing emissions
                            under title IV has been lower than initially estimated. One reason for the
                            cost overestimation has been the reduced cost of pollution control
                            technologies. Based on these findings and reports that some ecosystems

                            Page 5                   GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                               are not yet recovering from acid rain, a bill (S. 172) was introduced in the
                               U.S. Senate in January 1999 that would, among other things, further limit
                               sulfur dioxide emissions. The retrospective studies also provided
                               information on whether the market-based approach used for addressing
                               acid rain could be applied to other regulatory areas, such as climate
                               change and greenhouse gases.

Improving Methodologies of     Retrospective studies can also provide agencies with insights on how to
Prospective Analyses           better conduct prospective cost-benefit analyses and prevent
                               methodological biases in these analyses. For example, some analysts have
                               suggested that EPA tends to overestimate costs because, among other
                               reasons, it does not anticipate and account for cost-saving technological
                               innovations. However, in its analysis of the accuracy of prospective
                               cost-benefit estimates, RFF found that two studies (on reformulated
                               gasoline and the Dinoseb pesticide) had overestimated costs; two (on the
                               CFC phaseout and sulfur dioxide) had accurately estimated costs; and one
                               (on the Aldicarb pesticide) had underestimated costs.13 Additional
                               retrospective studies would enable EPA to more fully explore concerns
                               about what methodological factors, if any, contribute to the over- or
                               underestimation of costs.

                               According to an EPA official, even the best models for predicting the costs
                               and the benefits could be based on erroneous assumptions because it is
                               difficult to forecast technological improvements and economic changes.
                               The official explained that one conclusion from the retrospective studies’
                               results might be to include a “technology improvement case” when
                               estimating the costs of regulations.

Providing Information for      Retrospective studies of the costs and the benefits can also help EPA in its
Implementing the Results Act   efforts to implement the Results Act. The Results Act requires agencies to
                               set goals for program performance and to measure results. Retrospective
                               studies are one means of measuring results. Such analyses aim to identify
                               relevant results—usually expressed in monetary terms—of existing
                               regulations. The studies that identify actual benefits can provide insights
                               into the outcomes of EPA’s regulations. We reported in 1998 that the
                               Results Act recognizes and encourages the linkage between program
                               evaluations—retrospective studies are one type of such evaluations—and
                               performance measures. The Act directs federal agencies to provide a

                                RFF labeled a prospective study as “accurate” if the estimated costs in the retrospective study were
                               no more than 25 percent higher or no less than 25 percent lower than the prospective estimates.

                               Page 6                           GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                              summary of program evaluation findings along with performance
                              measurement results in the required annual performance reports.14

Retrospective Studies Are     Researchers face a number of difficulties in conducting retrospective
Difficult to Do               studies: (1) determining the baseline against which the changes can be
                              identified and measured, (2) isolating the reasons for the actions taken,
                              (3) obtaining valid cost data from the regulated entities, and
                              (4) quantifying the benefits.

Determining the Baseline      According to OMB’s 1998 Report to the Congress on the Costs and Benefits
                              of Federal Regulations, one of the primary problems with estimating the
                              incremental costs and benefits associated with implementing a federal
                              regulation is determining the baseline against which the costs and the
                              benefits should be compared. A baseline—how things would have been if
                              the regulation had not been issued—is elusive because so many changes
                              could have occurred while the regulation was being implemented. As OMB’s
                              report states, “what would have happened in the absence of regulation can
                              only be an educated guess since it never happened.”

                              In a 1996 report on companies’ regulatory burden, we found that it can be
                              extremely difficult for company officials to determine what actions their
                              company would have taken in the absence of the identified regulations.15
                              EPA and RFF officials took a more optimistic view but contended that
                              modeling would be needed to establish a baseline. Such modeling can be
                              costly and time-consuming and requires making assumptions about
                              economic behavior that may or may not approximate reality.

Determining the Reasons for   According to OMB’s report, it is often difficult to attribute the actions taken
Regulated Entities’ Actions   by regulated entities to specific federal regulations or other external
                              factors. These entities may be responding, for example, to regulations
                              from state and local governments. Voluntary standards from organizations
                              and public pressure also cause firms to protect the public in the absence
                              of federal regulations. In our 1997 report on the governmentwide
                              implementation of the Results Act, we found that regulatory agencies have
                              had difficulty in sorting out the effects of external forces.16 One expert

                               Program Evaluation: Agencies Challenged by New Demand for Information on Program Results
                              (GAO/GGD-98-53, Apr. 24, 1998).
                               Regulatory Burden: Measurement Challenges and Concerns Raised by Selected Companies
                              (GAO/GGD-97-2, Nov. 18, 1996).
                                The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be
                              Uneven (GAO/GGD-97-109, June 2, 1997). Also, see Managing for Results: Regulatory Agencies
                              Identified Significant Barriers to Focusing on Results (GAO/GGD-97-83, June 24, 1997).

                              Page 7                         GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                                 commented that external factors often influence technological innovation
                                 and that isolating the impacts of regulations from these other factors is

                                 In 1997, EPA’s Office of Water found these multiple factors to be a problem
                                 when it attempted to conduct a retrospective study of effluent guidelines.
                                 The Office never implemented the study because it was too difficult to
                                 isolate the costs specifically related to compliance with effluent guidelines
                                 from either self-initiated technological improvements or state and local
                                 government requirements.

Obtaining Valid Cost Data From   A number of studies have reported difficulties in obtaining valid cost data.
Regulated Entities               For example, our 1996 study on regulatory burden also found that
                                 companies’ accounting and financial records did not capture the
                                 information necessary to determine incremental compliance costs and that
                                 there was little incentive to isolate and monitor these costs because such
                                 information had little business value. Furthermore, companies were
                                 concerned about the disclosure of information that is proprietary (e.g.,
                                 information on their operating expenses) or could potentially involve
                                 regulatory enforcement actions. Similarly, in 1996, we found it difficult to
                                 determine the effects of regulatory compliance on research grantees at the
                                 National Institutes of Health.17 Although members of the research
                                 community were confident that compliance costs were high, they were
                                 unable to provide cost data directly attributable to compliance activities.

                                 Other research organizations have also had difficulties in data collection.
                                 According to an expert in cost-benefit analysis at the Massachusetts
                                 Institute of Technology, the university conducted a telephone survey to
                                 collect actual cost data on scrubber technology, but despite having “very
                                 good” industry connections, the survey’s results were somewhat
                                 disappointing because some companies declined to participate. The
                                 Organization for Economic Cooperation and Development—an
                                 international agency sponsored by industrialized countries—is conducting
                                 a business survey to determine the costs associated with implementing
                                 regulations in three general areas: environmental compliance, tax
                                 administration, and employment. Its survey is limited to small- and
                                 medium-sized businesses. According to Organization officials, it is more
                                 difficult for large companies that operate at multiple locations to
                                 determine the cost impacts of regulations on their far-flung enterprises
                                 and to isolate these costs from the costs attributable to other business
                                 decisions. They also told us that asking businesses to self-report capital

                                   Regulatory Compliance for NIH Grantees (GAO/HEHS-96-90R, Mar. 25, 1996).

                                 Page 8                          GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                       costs would not be valid because the data would not be verifiable or
                       consistent. In their view, such data would have to be collected through
                       very detailed structured interviews at selected businesses.

                       Some data on the costs of environmental regulatory compliance were once
                       available. Until 1994, the Department of Commerce administered the
                       Pollution Abatement and Control Expenditures (PACE) survey. When it
                       discontinued the survey to focus its resources on higher priorities,
                       Commerce collected data annually from manufacturers on the aggregate
                       cost of implementing pollution control regulations. Commerce, in
                       collaboration with EPA, is working to resume the PACE survey in 2000.
                       According to EPA, reinstatement of this survey is dependent on the
                       availability of funds. While one expert we interviewed suggested restarting
                       the PACE survey, another raised questions about the data’s validity and
                       usefulness. An official in EPA’s Office of Policy told us that his agency is
                       aware of these questions and, in developing the new survey, will attempt
                       to address them. EPA also pointed out that the agency’s Science Advisory
                       Board and others have recommended reinstatement of the survey.

                       Another attempt at data collection is currently being undertaken. A
                       contractor is collecting cost data for the retrospective study of the
                       drinking water standards for atrazine and nitrate. EPA officials told us that
                       they believe the drinking water treatment facilities will be able to isolate
                       the costs of compliance for these chemicals. They also expected both
                       public and private treatment facilities to cooperate.

Quantifying Benefits   The experts we interviewed agreed that quantifying the benefits, as part of
                       a retrospective study, is extremely difficult. One cited reason is that data
                       on benefits, which generally accrue outside the regulated entities (e.g., to
                       protected species or society at large), are not available. Another reason,
                       which was highlighted by the author of the CFC retrospective study, is the
                       difficulty in assigning the benefits to a particular regulatory action versus
                       other factors. For example, he pointed out that although the benefits from
                       a reduction in occurrences in skin cancer could be attributed to the
                       phaseout of CFCs, other factors could have played a role. Finally,
                       considerable time may elapse—sometimes decades—before many benefits
                       are realized. For example, some diseases, such as cancer, can have latency
                       periods of decades. We emphasized this difficulty in our 1997 report on the
                       Results Act, citing EPA officials who noted that a significant time lag
                       usually exists between the elimination of a chemical hazard and any
                       corresponding change in illness rates. This lag makes it difficult to track
                       the yearly progress of some of the agency’s regulatory actions. EPA’s 1997

                       Page 9                   GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                             Strategic Plan noted how difficult it is to assess the risks from pollutants;
                             measure the consequences of pollution to persons, plants, and animals;
                             and assign a dollar value to the health and environmental effects that are
                             remote in distance and time.

                             Based on discussions with study authors, EPA officials, and other experts,
A Systematic                 we found that a systematic approach by EPA would foster retrospective
Approach Would               studies in the future. The agency, however, has not targeted regulations for
Foster Retrospective         review after systematically (1) identifying candidate regulations for study
                             based on selection criteria, (2) assessing the feasibility of conducting
Studies                      retrospective studies of those regulations, and (3) identifying the
                             resources needed.

Identifying Candidates for   While all economically significant regulations are subject, by executive
Review                       order, to a prospective economic assessment of their costs and benefits
                             before issuance, EPA is not required to calculate the actual costs and
                             benefits retrospectively for all of its rules. Program offices within the
                             agency have, on an ad hoc basis, initiated the sulfur dioxide study and the
                             ongoing study of the drinking water standards. EPA has not, however,
                             reviewed the full array of environmental regulations and, based on criteria,
                             selected certain rules as candidates for study.

                             In identifying candidates for review, one criterion cited by the experts we
                             interviewed is the potential for learning something useful that would
                             improve the efficiency and the effectiveness of a significant regulatory
                             program. For example, the significance of acid rain for the northeastern
                             states and the concern that existing regulations were not adequately
                             controlling the problem played a role when the Congress mandated a
                             retrospective study. Other criteria cited are the potential for gaining
                             insights into a new regulatory approach to environmental protection or the
                             methodology for prospective economic assessments. A retrospective study
                             can, for instance, examine assumptions made in prospective assessments
                             where uncertainty about regulatory costs exists because a new pollution
                             control technology is being used.

                             When identifying candidate regulations, timing is another consideration.
                             EPA would examine whether it would be more useful to study the actual
                             costs and benefits of certain regulations shortly after implementation or to
                             wait until a number of years or even decades have elapsed. A major factor

                             Page 10                  GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                               in the timing decision would be the value of gaining insights into a
                               regulation’s impacts immediately or after a more extended period of time.

Assessing the Feasibility of   The next action would be an assessment of the feasibility of conducting a
a Retrospective Study          retrospective study of the candidate regulations. This assessment would
                               focus on the potential difficulties facing a researcher. For example, to
                               obtain valid cost data, EPA might find it easier to address those regulations
                               that rely on market forces because price and cost information is in the
                               public domain rather than those regulations based on “command and
                               control,” for which collecting the original data is especially challenging. In
                               some command-and-control cases, however, the regulated entities could
                               take steps during the implementation of the regulation to help researchers
                               develop data on compliance costs. These steps could include asking a
                               sample of regulated entities to establish financial and accounting
                               mechanisms to measure the impact of the regulation as it is implemented.
                               In addition, to attribute changes in the actions of regulated entities to
                               specific EPA regulations, RFF officials suggested that researchers employ
                               the same models used in conducting prospective economic assessments
                               and, therefore, make the same assumptions about regulated entities’
                               taking actions in response to the regulations. Of course, the researchers
                               would attempt to re-estimate the costs and the benefits after updating the
                               model, for example, with any data on the actual steps taken by regulated

Identifying Needed             A planned approach would identify the resources needed to support the
Resources                      retrospective study. Funding is, of course, critical. EPA receives funding for
                               analytical work in general. While EPA has occasionally funded
                               retrospective studies, EPA program officials noted that they have used their
                               analytical resources almost entirely for prospective analyses—those that
                               provide information for decisions on proposed regulations. A 1997
                               Congressional Budget Office study found that EPA incurred an average cost
                               of $662,000 and a median cost of $376,000 (in 1995 dollars) on 65 economic
                               assessments.18 The program officials told us that they had limited
                               discretionary funds and resources and needed to use them in developing
                               new regulations. One official said that once a regulation is issued, his
                               office gears up to develop any other regulations required by law.

                                 Regulatory Impact Analyses: Costs at Selected Agencies and Implications for the Legislative Process
                               (Mar. 1997). Costs included time spent by EPA personnel. The majority of the economic assessments
                               in the study were completed between 1990 and 1996. Because the scope and the complexity varied
                               substantially, the costs ranged from as low as $48,000 to as high as $6 million.

                               Page 11                          GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

The funding needed to do retrospective studies would be substantial if a
large proportion of the 101 economically significant regulations issued
since 1981 were the subject of retrospective studies. Where we were able
to obtain information on study costs, they varied substantially—from
$150,000 for the drinking water standards study, to $433,000 for the acid
rain report,19 to $4 million for the Section 812 study.

The method of funding as well as responsibility for a study can have
implications for its credibility. Authors of retrospective studies told us that
the credibility of these studies is enhanced when EPA funds an independent
analysis. The authors stated that it is better not to have the same EPA
offices that were responsible for conducting the prospective cost-benefit
analysis also responsible for conducting the retrospective study. OMB
officials stated that the best estimates of the costs and the benefits of
regulation are likely to be retrospective studies done by individuals who
do not have vested interests but do have reputations as objective analysts
to uphold. One study author believed that the cooperative agreement
arrangement helped ensure that the researcher had freedom to maintain
independence. As discussed earlier, EPA has used cooperative agreements
with organizations, such as RFF and the Massachusetts Institute of
Technology. Another author suggested that if EPA’s program offices were
directly involved in the study, the agency could use some type of advisory
committee or peer review process to oversee the studies and ensure their

Resources also take the form of technical assistance, and EPA has provided
some. For example, the Office of Policy has built a library of EPA’s
economic assessments of major regulations and related reports and
provides a limited number of them on the Internet. Furthermore, EPA is
now soliciting comments on a major revision of its Guidelines for
Performing Economic Assessments.20 The agency first issued these
guidelines in December 1983 and has made a few modifications to them
over the past 15 years. The draft revision provides, among other things, a
summary of analytical methodologies, empirical techniques, and data
sources that can assist analysts in performing economic assessments of
environmental policies. While the guidance is targeted more to prospective
analyses, its principles can also be applied to retrospective studies.

  From 1991 through 1996, the federal government spent $140 million to underwrite the scientific
research contained in the acid rain report. This research included an extensive network of monitoring
stations that provided scientists with data sets.
 The Environmental Economics Advisory Committee of EPA’s Science Advisory Board has a major
role in reviewing these guidelines.

Page 12                          GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

             To foster retrospective studies, the experts we interviewed also suggested
             that the availability and the accessibility of the prospective economic
             assessments and supporting documentation would have to be improved.
             For example, some of these authors who had used the models employed in
             the prospective analyses told us they had difficulty with the availability
             and the usefulness of the models. One researcher wanted to run the model
             used for the prospective economic assessment with different inputs. This
             would have served as a useful way to determine to what extent and why
             the estimate of actual costs and benefits differed from the prospective
             estimate. In conducting his study, he had difficulty obtaining the complete
             model from EPA. He was, however, able to obtain the complete model from
             the contractor who performed the assessment. Another author found that
             although he had access to the original model and supporting
             documentation, it was very difficult to extract information.
             Acknowledging concerns about the accessibility of models, the Office of
             Air and Radiation maintains a web site that has placed all model inputs in
             the public domain.

             Finally, one author focused on the importance of capturing data on the
             incremental costs of regulatory compliance from the regulated entities. He
             noted that EPA would enhance its and others’ capacity to evaluate
             environmental regulatory programs if the agency collected and maintained
             data on the costs immediately after issuing those regulations it has
             targeted for retrospective study.

             EPA’s regulations, issued pursuant to federal statutes, cause this nation to
Conclusion   make a substantial investment in protecting the environment. They also
             result in considerable benefits. While EPA devotes substantial resources to
             cost-benefit analyses when developing new regulations, the agency seldom
             looks back at the actual costs and benefits after those regulations have
             been implemented. An executive order and “good government” principles
             recommend that EPA periodically reassess selected regulations to
             determine if they are still needed and, if so, whether and how they could
             be improved. Retrospective studies are difficult to do, but they are a
             potentially valuable tool for such reassessments. We believe that EPA’s
             current ad hoc approach gives no assurance that economically significant
             regulations will be subjected to review. A systematic, planned approach
             would help EPA to foster retrospective studies in the future.

             Page 13                 GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

                  To help determine if existing environmental regulations need to be
Recommendation    retained or improved, we recommend that the Administrator, EPA,
                  establish a plan to study the actual costs and benefits of such regulations,
                  including when and how to use retrospective studies as an integral part of
                  its approach. The plan should target selected regulations for review based
                  on a systematic identification of candidate regulations, an assessment of
                  the feasibility of studying those regulations retrospectively, and an
                  identification of the resources needed.

                  We provided a draft of this report to EPA for its review and comment. EPA
Agency Comments   commented that the report is accurate, provides a fair characterization of
                  the difficulties in performing retrospective studies, and offers some useful
                  suggestions on how the agency could approach such studies in a more
                  systematic manner.

                  EPA also suggested that the budget implications of conducting
                  retrospective studies need to be emphasized in the report because the
                  scarcity of budget and internal resources limits the number of such studies
                  that the agency could support without the Congress’ providing significant
                  additional resources. In this report, we recognize that EPA concentrates its
                  analytical resources on prospective economic assessments, which EPA is
                  expected to perform, rather than on retrospective studies. Executive
                  Order 12866 states that EPA’s review of existing regulations should be
                  consistent with the agency’s resources and regulatory priorities. We
                  believe that if EPA were to develop a plan, as we are recommending, the
                  agency would be better able to identify the additional resources needed to
                  retrospectively review selected regulations. EPA also provided specific
                  technical comments and clarifications, which we incorporated, as
                  appropriate. EPA’s comments are contained in appendix I.

                  To accomplish our objectives, we interviewed officials in EPA’s Office of
Scope and         Policy; Office of Air and Radiation; Office of Water; Office of Solid Waste
Methodology       and Emergency Response; Office of Prevention, Pesticides, and Toxic
                  Substances; and Office of the Chief Financial Officer and obtained and
                  reviewed related legislation, memorandums, and reports. We also
                  interviewed all of the authors of the retrospective studies identified in this
                  report and several other experts in cost-benefit analysis, including RFF and
                  OMB officials, and representatives from Harvard University, the
                  Massachusetts Institute of Technology, and the Organization for Economic
                  Cooperation and Development. In documenting the difficulties in

                  Page 14                  GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations

obtaining actual cost data from companies, we relied heavily on our
November 1996 report, Regulatory Burden: Measurement Challenges and
Concerns Raised by Selected Companies. To identify the universe of
retrospective studies, we used data developed by RFF for its January 1999
report, On the Accuracy of Regulatory Cost Estimates. We also added the
Section 812 report, the acid rain report, and pesticide studies that RFF
chose to exclude because they met our criteria for containing information
on the actual costs of existing regulations and focusing on a specific
regulation or a set of regulations with common objectives.

We conducted our review from October 1998 through September 1999 in
accordance with generally accepted government auditing standards.

We are sending copies of this report to Senator Fred Thompson and
Senator Joseph Lieberman, as Chairman and Ranking Member,
respectively, of the Senate Committee on Governmental Affairs;
Representative Dan Burton and Representative Henry Waxman, as
Chairman and Ranking Member, respectively, of the House Committee on
Government Reform; Representative James Oberstar as Ranking Member
of the House Committee on Transportation and Infrastructure; the
Honorable Carol Browner, Administrator, EPA; and the Honorable Jacob
Lew, Director of OMB. We will also make copies available to others on

If you have any questions about this report, please contact me on
(202) 512-6111. Major contributors to this report were Jay Cherlow, Ellen
Crocker, Bob Levin, and Bruce Skud.

Peter F. Guerrero
Director, Environmental Protection Issues

Page 15                 GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations
Appendix I

Comments From the Environmental
Protection Agency

             Page 16   GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations
Appendix I
Comments From the Environmental
Protection Agency

Page 17                   GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations
           Appendix I
           Comments From the Environmental
           Protection Agency

(160453)   Page 18                   GAO/RCED-99-250 Retrospective Studies of EPA’s Regulations
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:


or visit GAO’s World Wide Web Home Page at:


United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested