oversight

Peer Review: EPA's Implementation Remains Uneven

Published by the Government Accountability Office on 1997-03-11.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                   United States General Accounting Office

GAO                Testimony
                   Before the House Subcommittee on Energy and
                   Environment, Committee on Science, House of
                   Representatives


For Release
on Delivery
Expected at
                   PEER REVIEW
1 p.m. EST
Tuesday
March 11, 1997
                   EPA’s Implementation
                   Remains Uneven
                   Statement by Stanley J. Czerwinski, Associate Director,
                   Environmental Protection Issues,
                   Resources, Community, and Economic
                   Development Division




GAO/T-RCED-97-95
    Mr. Chairman and Members of the Subcommittee:

    We are pleased to be here today to discuss our recent report on the
    Environmental Protection Agency’s (EPA) implementation of its peer
    review policy.1 As you know, peer review is the critical evaluation of
    scientific and technical work products by independent experts to enhance
    the products’ quality, credibility, and acceptability. Furthermore, peer
    review can actually help avoid costly and time-consuming delays by
    helping to steer product development along the most efficient, effective
    course. The EPA’s current peer review policy—updated in 1994 and
    currently under evaluation by the agency in the light of our report’s
    findings—stresses the importance of such reviews and calls for the peer
    review of all major scientific and technical work products that may
    eventually play an important role in key agency decisions. In the light of
    the critical role that peer review plays in supporting the agency’s
    important decisions, we assessed EPA’s (1) progress in implementing its
    peer review policy and (2) efforts to improve the peer review process.

    In summary, we found that:

•   Despite some recent progress, peer review continues to be implemented
    unevenly. Although we found some cases in which EPA’s peer review policy
    was properly followed, we also found cases in which key aspects of the
    policy were not followed or in which peer review was not conducted at all.
    We believe that two of the primary reasons for this uneven implementation
    are (1) inadequate accountability and oversight to ensure that all relevant
    products are properly peer reviewed and (2) confusion among EPA’s staff
    and management about what peer review is, its importance and benefits,
    and how and when it should be conducted.
•   EPA officials readily acknowledge this uneven implementation and, during
    the course of our work, had a number of efforts under way to improve the
    peer review process. Although we found these efforts to be steps in the
    right direction, we concluded that they were not addressing the underlying
    problems that we had identified. Accordingly, we recommended that EPA
    ensure that (1) upper-level managers have the information they need to
    know whether or not all relevant products have been considered for peer
    review and (2) staff and managers are educated about the need for and
    benefits of peer review and their specific responsibilities in implementing
    the policy. EPA agreed with our recommendations and has several efforts
    under way to implement them. For example, EPA plans to initiate a peer
    review training program for its managers and staff in June 1997. While it is

    1
     Peer Review: EPA’s Implementation Remains Uneven (GAO/RCED-96-236, Sept. 24, 1996).



    Page 1                                                                     GAO/T-RCED-97-95
             still too early to be certain if these efforts will be fully successful, we are
             encouraged by the high-level attention being paid to this very important
             process.


             Peer review is well established as a mechanism for assuring the quality,
Background   credibility, and acceptability of individual and institutional work products.
             This assurance is accomplished by having the products undergo an
             objective, critical review by independent reviewers. Peer review has long
             been used by academia, professional organizations, industry, and
             government. Within EPA, peer review has taken many different forms,
             depending upon the nature of the work product, the relevant statutory
             requirements, and office-specific practices and needs.

             In keeping with scientific custom and/or congressional mandates, several
             offices within EPA have used peer review for many years to enhance the
             quality of science within the agency. In response to a panel of outside
             academicians’ recommendations in 1992,2 EPA issued a policy statement in
             1993 calling for peer review of the major scientific and technical work
             products used to support the agency’s rulemaking and other decisions.
             However, the Congress, GAO,3 and others subsequently raised concerns
             that the policy was not being implemented consistently across the agency.
             In response to these concerns, in 1994 EPA reaffirmed the central role that
             peer review plays in ensuring that the agency’s decisions are based on
             sound science and credible data and revised its 1993 policy.

             The new policy, while retaining the essence of the prior one, was intended
             to expand and improve the use of peer review throughout EPA. The 1994
             policy continued to stress that major products should normally be peer
             reviewed, but it also recognized that statutory and court-ordered
             deadlines, resource limitations, and other constraints might limit or even
             preclude the use of peer review. The policy applied to major work
             products that are primarily scientific or technical in nature and that may
             contribute to the basis for policy or regulatory decisions. In contrast, other
             products used in decision-making are not covered by the policy, nor are
             the ultimate decisions themselves. While peer review can take place at
             several different points along a product’s development, such as during the
             planning stage, it should be applied to a relatively well-developed product.


             2
              Safeguarding the Future: Credible Science, Credible Decisions (EPA/600/9-91/050, Mar. 1992).
             3
              Peer Review: EPA Needs Implementation Procedures and Additional Controls (GAO/RCED-94-89,
             Feb. 22, 1994).



             Page 2                                                                           GAO/T-RCED-97-95
                          The 1994 policy also clarified that peer review is not the same thing as the
                          peer input, stakeholders’ involvement, or public comment—mechanisms
                          used by EPA to develop products, to obtain the views of interested and
                          affected parties, and/or to build consensus among the regulated
                          community. While each of these mechanisms serves a useful purpose, the
                          policy points out that they are not a substitute for peer review because
                          they do not necessarily solicit the same unbiased, expert views that are
                          obtained through peer review.

                          EPA’s policy assigned responsibility to each Assistant and Regional
                          Administrator to develop standard operating procedures and to ensure
                          their use. To help facilitate consistent EPA-wide implementation, EPA’s
                          Science Policy Council—chaired by EPA’s Deputy Administrator—was
                          directed to help the offices and regions develop their procedures and
                          identify products that should be peer reviewed. The Council was also
                          given the responsibility for assessing agencywide progress and developing
                          any needed changes to the policy. However, the ultimate responsibility for
                          implementing the policy was placed with the Assistant and Regional
                          Administrators.


                          We found that—2 years after EPA established its peer review policy—
Implementation            implementation was still uneven. We concluded that EPA’s uneven
Remains Uneven            implementation was primarily due to (1) inadequate accountability and
                          oversight to ensure that all products are properly peer reviewed by
                          program and regional offices and (2) confusion among agency staff and
                          management about what peer review is, what its significance and benefits
                          are, and when and how it should be conducted.

                          According to the Executive Director of the Science Policy Council, the
                          unevenness could be attributed to a number of factors. First, while some
                          offices within EPA—such as the Office of Research and Development
                          (ORD)—have historically used peer review for many years, other program
                          offices and regions have had little prior experience. In addition, the
                          Director and other EPA officials told us that statutory and court-ordered
                          deadlines, budget constraints, and problems in finding and obtaining
                          qualified, independent peer reviewers also contributed to the problem.


Inadequate Oversight to   EPA’s oversight primarily consisted of a two-part reporting scheme that
Ensure Consistent         called for each office and region to annually list (1) the candidate products
Implementation            nominated for peer review during the upcoming year and (2) the status of



                          Page 3                                                      GAO/T-RCED-97-95
the products previously nominated. If a candidate product was no longer
scheduled for peer review, the list had to note this and explain why peer
review was no longer planned.

Although we found this to be an adequate oversight tool for tracking the
status of previously nominated products, we pointed out that it does not
provide upper-level managers with sufficient information to ensure that all
products warranting peer review have been identified. This fact, together
with the misperceptions about what peer review is and the deadlines and
budget constraints that project officers often operate under, has meant
that the peer review program to date has largely been one of
self-identification, allowing some important work products to go unlisted.
According to the Science Policy Council, reviewing officials would be
much better positioned to determine if the peer review policy and
procedures are being properly and consistently implemented if, instead,
EPA’s list contained all major products along with what peer review is
planned and, if none, the reasons why not.

We noted that the need for more comprehensive oversight is especially
important given the policy’s wide latitude in allowing peer review to be
forgone in cases facing time and/or resource constraints. As explained by
the Executive Director of EPA’s Science Policy Council, because so much
of the work that EPA performs is in response to either statutory or
court-ordered mandates and the agency frequently faces budget
uncertainties or limitations, an office under pressure might argue for
nearly any given product that peer review is a luxury the office cannot
afford in the circumstances.

However, as the Executive Director of the Science Advisory Board (SAB)4
told us, not conducting peer review can sometimes be more costly to the
agency in terms of time and resources. He told us of a recent Office of
Solid Waste rulemaking concerning a new methodology for delisting
hazardous wastes in which the Office’s failure to have the methodology
appropriately peer reviewed resulted in important omissions, errors, and
flawed approaches in the methodology; these problems will now take from
1 to 2 years to correct. The SAB also noted that further peer review of the
individual elements of the proposed methodology is essential before the
scientific basis for this rulemaking can be established.




4
The SAB is a legislatively established body of independent experts that provides advice to the EPA
Administrator on scientific and engineering issues.



Page 4                                                                          GAO/T-RCED-97-95
Peer Review Policy and   Although EPA’s policy and procedures provide substantial information
Procedures Not Well      about what peer review entails, we found that some EPA staff and
Understood               managers had misperceptions about what peer review is, what its
                         significance and benefits are, and when and how it should be conducted.
                         Several cases we reviewed illustrate this lack of understanding about what
                         peer review entails. Officials from EPA’s Office of Mobile Sources (OMS)
                         told the House Commerce Committee in August 1995 that they had not had
                         any version of the mobile model5 peer reviewed. Subsequently, in
                         April 1996, OMS officials told us they recognize that external peer review is
                         needed and that EPA planned to have the next iteration of the model so
                         reviewed.

                         We found similar misunderstandings in several other cases we reviewed.
                         EPA regional officials who produced a technical product that assessed the
                         environmental impacts of tributyl tin6 told us that the contractor-prepared
                         product had been peer reviewed. While we found that the draft product
                         did receive some internal review by EPA staff and external review by
                         contributing authors, stakeholders, and the public, it was not reviewed by
                         experts independent of the product itself or of its potential regulatory
                         ramifications. When we pointed out that—according to EPA’s policy and
                         the region’s own peer review procedures—these reviews are not a
                         substitute for peer review, the project director said that she was not aware
                         of these requirements.

                         In two other cases we reviewed, there were misunderstandings about the
                         components of a product that should be peer reviewed. For example, in
                         the Great Waters study—an assessment of the impact of atmospheric
                         pollutants in significant water bodies—the scientific data were subjected
                         to external peer review, but the study’s conclusions that were based on
                         these data were not. Similarly, in the reassessment of dioxin—an
                         examination of the health risks posed by dioxin—the final chapter
                         summarizing and characterizing dioxin’s risks was not as thoroughly peer
                         reviewed.7 In both cases, the project officers did not have the conclusions
                         peer reviewed because they believed that the development of conclusions
                         is an inherently governmental function that should be performed


                         5
                          The mobile model is one of the primary tools used by EPA, states, and localities to calculate the
                         estimated emissions reduction benefits of pollution control activities called for in state
                         implementation plans.
                         6
                          Tributyl tin is a compound that has been used since the 1960s in antifouling paints for boats and large
                         ships.
                         7
                          Although the entire product was reviewed by EPA’s Science Advisory Board, the Board expressed
                         dissatisfaction that the risk characterization chapter did not receive prior peer review.



                         Page 5                                                                            GAO/T-RCED-97-95
                     exclusively by EPA staff. However, some EPA officials with expertise in
                     conducting peer reviews disagreed, maintaining that it is important to have
                     peer reviewers comment on whether or not EPA has properly interpreted
                     the results of the underlying scientific and technical data. EPA’s quality
                     assurance requirements also state that conclusions should be peer
                     reviewed.8


                     During our review, we found that EPA had recently taken a number of steps
EPA’s Actions to     to improve the peer review process. Although we believed that these steps
Improve the Peer     should prove helpful, we concluded that they did not fully address the
Review Process       previously-discussed underlying problems and made some
                     recommendations for improvement. EPA agreed with our findings and
                     recommendations and has recently undertaken steps to implement them.
                     While it is too early to gauge the effectiveness of these efforts, we are
                     encouraged by the attention peer review is receiving by the agency’s
                     upper-level management.


EPA’s Past Efforts   Near the completion of our review, in June 1996, EPA’s Deputy
                     Administrator directed the Science Policy Council’s Peer Review Advisory
                     Group and ORD’s National Center for Environmental Research and Quality
                     Assurance to develop an annual peer review self-assessment and
                     verification process to be conducted by each office and region. The
                     self-assessment was to include information on each peer review
                     completed during the prior year as well as feedback on the effectiveness of
                     the overall process. The verification would consist of the signature of
                     headquarters, laboratory, or regional directors to certify that the peer
                     reviews were conducted in accordance with the agency’s policy and
                     procedures. If the peer review did not fully conform to the policy, the
                     division director or the line manager must explain significant variances
                     and actions needed to limit future significant departures from the policy.
                     The self-assessments and verifications were to be submitted and reviewed
                     by the Peer Review Advisory Group to aid in its oversight responsibilities.
                     According to the Deputy Administrator, this expanded assessment and
                     verification process would help build accountability and demonstrate EPA’s
                     commitment to the independent review of the scientific analyses
                     underlying the agency’s decisions to protect public health and the
                     environment.


                     8
                       EPA Requirements for Quality Management Plans (EPA QA/R-2, Aug. 1994). This document
                     establishes the criteria and mandatory specifications for quality assurance and quality control
                     activities.



                     Page 6                                                                            GAO/T-RCED-97-95
                      During our review, we also found a number of efforts under way within
                      individual offices and regions to improve their implementation of peer
                      review. For example, the Office of Water drafted additional guidance to
                      further clarify the need for, use of, and ways to conduct peer review. The
                      Office of Solid Waste and Emergency Response formed a team to help
                      strengthen the office’s implementation of peer review by identifying ways
                      to facilitate good peer review and addressing barriers to its successful use.
                      Additionally, EPA’s Region 10 formed a Peer Review Group with the
                      responsibility for overseeing the region’s reviews.

                      We concluded that the above efforts should help address the problems we
                      found. However, we also concluded that the efforts aimed at improving the
                      oversight of peer review fell short by not ensuring that all relevant
                      products had been considered for peer review and did not require
                      documenting the reasons why products were not selected. Similarly, we
                      noted that the efforts aimed at better informing staff about the benefits
                      and use of peer review would be more effective if they were done
                      consistently throughout the agency.


EPA’s Current Plans   EPA agreed with our findings and conclusions and has recently undertaken
                      a number of steps to implement our recommendations. On November 5,
                      1996, the Deputy Administrator asked ORD’s Assistant Administrator, in
                      consultation with the other Assistant Administrators, to develop proposals
                      to strengthen the peer review process. In response, ORD’s Assistant
                      Administrator proposed a three-pronged approach consisting of (1) audits
                      of a select number of work products to determine how well the peer
                      review policy was followed; (2) a series of interviews with office and
                      regional staff involved with peer review to determine the processes used
                      to implement the policy; and (3) training to educate and provide help to
                      individuals to improve the implementation of the peer review policy.

                      Significantly, the Deputy Administrator has echoed our message that EPA
                      needs to improve its oversight to ensure that all appropriate products are
                      peer reviewed. In a January 14, 1997, memorandum to the Assistant and
                      Regional Administrators, the Deputy stated, “I want you to ensure that
                      your lists of candidates for peer review are complete.” To help accomplish
                      this goal, each organization is directed to use, among other things, EPA’s
                      regulatory agenda and budget planning documents to help identify
                      potential candidates for peer review.




                      Page 7                                                       GAO/T-RCED-97-95
           While we agree that this should prove to be a useful tool, we continue to
           encourage EPA to expand its existing candidate list to include all major
           work products, along with explanations of why individual products are not
           nominated for peer review. An all-inclusive list such as this will be
           extremely useful to those overseeing the peer review process to determine
           whether or not all products have been appropriately considered for peer
           review.


           In summary, peer review is critical for improving the quality of scientific
           and technical products and for enhancing the credibility and acceptability
           of EPA’s decisions that are based on these products. We are encouraged by
           the renewed attention EPA is giving to improving the peer review process.
           Although it is too early for us to gauge the success of these efforts, the
           involvement of the agency’s upper-level management should go a long way
           to ensure that the problems we identified are resolved. Mr. Chairman, this
           concludes my prepared statement. I will be happy to respond to your
           questions or the questions of Subcommittee members.




(160384)   Page 8                                                     GAO/T-RCED-97-95
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested