oversight

Army Corps of Engineers: Peer Review Process for Civil Works Project Studies Can Be Improved

Published by the Government Accountability Office on 2012-03-08.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States Government Accountability Office

GAO          Report to the Ranking Member,
             Subcommittee on Water Resources and
             Environment, Committee on
             Transportation and Infrastructure,
             House of Representatives
March 2012
             ARMY CORPS OF
             ENGINEERS
             Peer Review Process
             for Civil Works
             Project Studies Can
             Be Improved




GAO-12-352
                                              March 2012

                                              ARMY CORPS OF ENGINEERS
                                              Peer Review Process for Civil Works Project Studies
                                              Can Be Improved
Highlights of GAO-12-352, a report to the
Ranking Member, Subcommittee on Water
Resources and Environment, Committee on
Transportation and Infrastructure, House of
Representatives


Why GAO Did This Study                        What GAO Found
Section 2034 of the Water Resources           Since enactment of the Water Resources Development Act of 2007, 49 project
Development Act of 2007 requires that         studies have undergone peer review but it is unclear how many were performed
certain U.S. Army Corps of Engineers          in response to section 2034 requirements because the Army Corps of Engineers
(Corps) civil works project studies           (Corps) does not make specific determinations or track if a peer review is being
undergo independent external peer             conducted under section 2034. In February 2011, in response to section 2034,
review to assess the adequacy and             the Corps submitted its initial report to Congress summarizing its implementation
acceptability of the methods, models,         of the peer review process. In its report, however, the Corps did not distinguish
and analyses used. In the act,                which studies had been selected for peer review in accordance with section 2034
Congress established a 7-year trial           and therefore, did not provide Congress information that would help decision
period for this requirement and also          makers evaluate the requirements of section 2034 at the end of the trial period.
required the Corps to submit two
reports on its experiences with the           The 49 peer reviews resulted in both direct and indirect costs. Specifically, these
peer review process.                          peer reviews resulted in direct costs of over $9 million in contract costs and fees.
                                              In addition, Corps staff resources were used to manage the reviews, although
GAO was asked to examine (1) the
                                              these costs are not fully quantifiable. Furthermore, the addition of peer review to
number of Corps project studies that
                                              the Corps study process has resulted in indirect costs by altering project study
have undergone independent peer
review in response to section 2034,
                                              schedules to allow for time needed to complete peer reviews. In some cases
(2) the cost of these peer reviews,           where a peer review was not planned during the early stages of the study
(3) the extent to which the Corps’            process, significant delays to project studies occurred while funds were sought to
process for determining if a project          pay for the peer review. In contrast, according to some Corps officials, when
study is subject to peer review is            project managers have built in time and identified funding for peer reviews early,
consistent with section 2034, (4) the         the process has had less of an impact on project study schedules.
process the Corps uses to ensure that         The Corps’ process for determining whether a project study is subject to peer
the contractors it hires and the experts
                                              review is more expansive than section 2034 requirements because it uses
the contractors select to review project
                                              broader criteria, resulting in peer reviews of studies outside the scope of section
studies are independent and free from
conflicts of interest, and (5) the extent     2034. In addition, the process the Corps uses does not include the flexibility
to which peer review recommendations          provided in section 2034, which allows for the exclusion of certain project studies
have been incorporated into project           from peer review. Moreover, some studies are undergoing peer reviews that do
studies. GAO reviewed relevant laws,          not warrant it, according to some Corps officials GAO spoke with.
agency guidance, and documents and            The Corps has a process to review general information on contractors’ conflicts
interviewed Corps officials and               of interest and independence when selecting them to establish peer review
contractors.
                                              panels, but it does not have a process for reviewing project-level information on
What GAO Recommends                           conflicts of interest and independence. As a result, it cannot be assured that
                                              contractors do not have conflicts at the project-level. In contrast, the Corps’
GAO recommends that the Department            contractors do have a process for reviewing information related to conflicts of
of Defense direct the Corps to, among         interest and the independence of experts selected for each peer review panel.
other actions, better track peer review
studies, revise the criteria for              The Corps has adopted and incorporated into its project study reports most of the
determining which studies undergo             peer review recommendations it has received. Doing so has resulted in some
peer review and the timing of these           technical improvements to study reports but generally has not changed the
reviews, and improve its process for          Corps’ decisions about project alternatives, in part because the peer review
ensuring contractor independence. The         process occurs too late in the project study process to affect decision making,
department generally concurred with           according to some Corps officials GAO spoke with. As a result, some
these recommendations.                        recommendations about alternatives may not have been implemented because
                                              the decision on the preferred design had already been made.
View GAO-12-352. For more information,
contact Anu K. Mittal at (202) 512-3841 or
mittala@gao.gov.

                                                                                       United States Government Accountability Office
Contents


Letter                                                                                     1
               Background                                                                  4
               It Is Unclear How Many Peer Reviews Have Been Completed in
                   Response to Section 2034 Requirements                                 15
               Completed Peer Reviews Have Cost Millions of Dollars in Direct
                   and Indirect Costs                                                    16
               The Corps’ Process for Initiating Peer Review Is More Expansive
                   and Less Flexible Than Section 2034 Requirements                      20
               Gaps Exist in the Corps’ Process for Screening Its Contractors Who
                   Are Responsible for Selecting Experts for Peer Review Panels          25
               The Corps Has Adopted Most Peer Review Recommendations,
                   Resulting in Technical Improvements but Generally No Changes
                   in Project Decisions                                                  28
               Conclusions                                                               32
               Recommendations for Executive Action                                      33
               Agency Comments and Our Evaluation                                        33

Appendix I     Objectives, Scope, and Methodology                                        36



Appendix II    Characteristics of Studies That Underwent Peer Review                     39



Appendix III   Comments from the Department of Defense                                   48



Appendix IV    GAO Contact and Staff Acknowledgments                                     52



Tables
               Table 1: Corps EC 209 Guidance on Conducting Peer Review                  20
               Table 2: Characteristics of Studies That Underwent Peer Review            39


Figures
               Figure 1: National Academy of Sciences Criteria for Selecting an
                        Appropriate Level of Review                                       8
               Figure 2: Key Steps in the Feasibility Study Process                      14



               Page i                                      GAO-12-352 Army Corps of Engineers
Abbreviations

EIS               environmental impact statement
NAS               National Academy of Sciences
NEPA              National Environmental Policy Act
OMB               Office of Management and Budget
WRDA              Water Resources Development Act




This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                                               GAO-12-352 Army Corps of Engineers
United States Government Accountability Office
Washington, DC 20548




                                   March 8, 2012

                                   The Honorable Timothy Bishop
                                   Ranking Member
                                   Subcommittee on Water Resources and Environment
                                   Committee on Transportation and Infrastructure
                                   House of Representatives

                                   Dear Mr. Bishop:

                                   Through its civil works program, the U.S. Army Corps of Engineers
                                   (Corps) constructs, operates, and maintains thousands of civil works
                                   projects related to water resources across the United States. 1 These
                                   projects aim to provide safe and reliable waterways; reduce risk to
                                   people, homes, and communities from flooding and coastal storms;
                                   restore and protect the environment; and address water resources
                                   challenges. A Corps civil works project generally starts with a study of a
                                   water resources issue and the development of various alternatives to
                                   address it. Such studies can span the full range of Corps civil works
                                   projects, and can include those that are small and low impact and others
                                   that are large and complex, with potentially significant economic and
                                   environmental impacts. Through its civil works program, the Corps
                                   operates 50 centers of expertise and seven research laboratories that
                                   assist its eight divisions and 38 district offices in the planning, design, and
                                   technical review of civil works projects. 2 Six of these centers are focused
                                   on the quality and effectiveness of water resources planning and are
                                   referred to as “planning centers of expertise.”

                                   Through its civil works projects, the Corps provides vital public
                                   engineering services in peace and war to strengthen the nation’s security,



                                   1
                                    The Corps has both a military and a civil works program. The military program provides,
                                   among other things, engineering and construction services to other federal agencies and
                                   foreign governments, and the civil works program is responsible for investigating,
                                   developing, and maintaining water resources projects. This report discusses only the civil
                                   works program.
                                   2
                                    The centers of expertise are designated individuals or organizations—located either in
                                   district offices, division offices, or research laboratories—with capability or expertise in a
                                   specialized area. The Corps designates employees at various levels within the Corps to
                                   oversee, manage, and coordinate the centers.




                                   Page 1                                                   GAO-12-352 Army Corps of Engineers
energize the economy, and reduce risks from disasters. These projects
involve navigation and flood control activities, environmental restoration,
and emergency response—most recently including emergency response
to Missouri River flooding and rebuilding after Hurricane Katrina.
Technical errors in past studies of Corps projects, however, had raised
concerns about the effectiveness of the Corps’ internal review processes
and the quality of the studies that the Corps used as a basis for its civil
works projects. 3 For example, in March 2006, we reported that certain
studies completed by the Corps from 1992 through 2002 were fraught
with errors, mistakes, and miscalculations and used invalid assumptions
and outdated data. 4 We also reported that these Corps studies
understated costs, overstated benefits, and did not provide a reasonable
basis for decision making. Similar findings have been documented in
reviews by the National Academy of Sciences (NAS) and other
organizations, which concluded that the Corps’ review processes needed
to be strengthened.

In the wake of these reports, Congress passed section 2034 of the Water
Resources Development Act (WRDA) of 2007, which requires that studies
for certain Corps projects undergo independent peer review. 5 To conduct
such peer review, the Corps hires a contractor to select a panel of
independent experts, who assess the adequacy and acceptability of the
economic, engineering, and environmental methods, models, and
analyses used in a Corps’ project study. Upon completion of the peer
review, the Corps is to consider recommendations from the review before
making a final decision on the project. 6




3
 National Research Council, Inland Navigation System Planning: The Upper Mississippi
River-Illinois Waterway (Washington, D.C.: National Academies Press, 2001), and GAO,
Corps of Engineers: Observations on Planning and Project Management Processes for
the Civil Works Program, GAO-06-529T (Washington, D.C.: Mar. 15, 2006).
4
GAO-06-529T. GAO’s review focused on four civil works studies.
5
 Water Resources Development Act of 2007, Pub. L. No. 110-114, § 2034 (Nov. 8, 2007),
codified at 33 U.S.C. § 2343.
6
 The final Corps approval of a project recommendation is signed by the Chief of
Engineers—the Corps’ commanding officer—in what is known as the signed Chief’s
report. This report summarizes the study’s results and includes a final project
recommendation; the report is generally submitted to Congress for authorization of
construction.




Page 2                                              GAO-12-352 Army Corps of Engineers
You asked us to review the efforts the Corps has made to implement and
comply with the independent peer review requirements in section 2034 of
WRDA 2007. This report examines (1) the number of Corps project
studies that have undergone independent external peer review in
response to section 2034, (2) the cost of these peer reviews, (3) the
extent to which the Corps’ process for determining if a project study is
subject to peer review is consistent with section 2034, (4) the process the
Corps uses to ensure that the contractors it hires and the experts the
contractors select to review project studies are independent and free from
conflicts of interest, and (5) the extent to which peer review
recommendations have been incorporated into project studies.

To conduct this work, we reviewed relevant legal requirements, policy
guidance, review plans, and peer review reports for project studies that
were subject to a peer review and for which a report had been completed
since WRDA 2007 was passed. In addition, we selected a
nongeneralizable sample of six peer reviews to examine in greater depth,
to better understand the costs associated with conducting these reviews,
as well as the overall impact of the process on the timeline of the project
study and the study’s outcome. Because this sample was a nonprobability
sample, the information derived from these reviews is not generalizable to
all peer reviews, but the reviews serve as illustrative examples that
provide valuable insights into the Corps’ peer review process; we
selected a review from each planning center of expertise and at least one
for each of the three contractors selected to conduct peer reviews. We
conducted semistructured interviews with officials from Corps
headquarters, the planning centers of expertise involved in managing the
peer reviews, all of the Corps’ eight divisions, and from 10 geographically
dispersed districts that had conducted project studies that underwent peer
review. We also conducted semistructured interviews with the three
contractors, as well as with selected peer review panel members and
local sponsors of Corps civil works projects. 7

To determine the number of studies that underwent peer review and the
cost of these reviews, we reviewed all completed peer review reports and



7
 The Corps also typically receives funds from each project’s local sponsor, which may be
a state, tribal, county, or local agency or government. WRDA of 1986 stipulated that
nonfederal sponsors share the cost of planning and implementing most Corps civil works
projects. The division of federal and nonfederal cost sharing required varies by project
purpose.




Page 3                                               GAO-12-352 Army Corps of Engineers
                          contract award documentation. In addition, for the six peer reviews we
                          examined in depth, we analyzed information on costs associated with
                          managing the review process, including those associated with district and
                          other staff time. To determine the extent to which the Corps’ process for
                          determining if a study is subject to peer review is consistent with section
                          2034, we analyzed the legal requirements and relevant Corps policy
                          guidance for determining when to conduct peer reviews, and we reviewed
                          documentation on Corps decisions. To determine the process the Corps
                          uses to ensure that the contractors it hires and the experts the contractors
                          select are independent and free from conflicts of interest, we analyzed
                          relevant Corps policy guidance and reviewed documentation provided by
                          the contractors to demonstrate that they and the selected experts meet
                          the requirements. To determine the extent to which recommendations are
                          incorporated into project studies, we analyzed relevant policy guidance
                          and summarized peer review recommendations and Corps responses,
                          but we did not assess the quality of the peer review recommendations or
                          the technical sufficiency of the Corps responses to these
                          recommendations. In addition, for the six reviews examined in depth, we
                          analyzed project studies, recommendations, and Corps responses to
                          describe the overall timeline and impact of the peer review process.
                          Appendix I describes our scope and methodology in greater detail.

                          We conducted this performance audit from April 2011 to March 2012 in
                          accordance with generally accepted government auditing standards.
                          Those standards require that we plan and perform the audit to obtain
                          sufficient, appropriate evidence to provide a reasonable basis for our
                          findings and conclusions based on our audit objectives. We believe that
                          the evidence obtained provides a reasonable basis for our findings and
                          conclusions based on our audit objectives.



Background
Corps Process for Civil   The purpose of the Corps’ civil works project study process is to inform
Works Project Studies     federal decision makers whether a water resources project warrants
                          further federal investment. The study process is conducted in two phases:
                          reconnaissance and feasibility. In the reconnaissance phase, the Corps
                          conducts an initial evaluation of potential solutions to a water resources
                          problem. If the Corps determines that a project potentially warrants
                          federal investment, it proceeds to a more detailed feasibility study. The
                          feasibility phase generally begins with the signing of a feasibility cost-
                          share agreement between the Corps and the local project sponsor.


                          Page 4                                       GAO-12-352 Army Corps of Engineers
Feasibility studies are generally prepared by the Corps’ 38 district offices,
with review and oversight provided by the cognizant Corps division office
and by headquarters. During the feasibility phase, the Corps formulates
and evaluates alternative plans for achieving the project’s objectives and
reviews the proposed project to assess whether the benefits of
constructing it outweigh its costs. At the beginning of this phase, a
feasibility scoping meeting is held to bring the Corps, the local sponsor,
and other agencies together to reach agreement on the problems and
solutions to be investigated during the feasibility study and the scope of
the analysis required. The next step includes an alternative formulation
briefing to identify and resolve any legal or policy concerns and to obtain
headquarters approval of the tentatively selected plan and to release the
draft report to the public. Finally, the draft feasibility report—which
presents the study results and findings, including those developed in the
reconnaissance phase—is released to the public. At the conclusion of the
feasibility phase, the Corps selects a recommended plan for proceeding
with the project.

The feasibility report also includes analysis and documentation to meet
the requirements of the National Environmental Policy Act (NEPA). 8
Under NEPA, federal agencies are to assess the effects of major federal
actions, such as Corps construction projects, that significantly affect the
environment and prepare a detailed statement on the environmental
impacts of those actions. NEPA has two principal purposes: (1) to ensure
that an agency carefully considers detailed information concerning
significant environmental impacts and (2) to ensure that this information
will be made available to the public. NEPA requires an agency to prepare
a detailed statement on the environmental impacts of any “major federal
action” significantly affecting the environment. NEPA implementing
regulations generally require an agency to prepare either an
environmental assessment 9 or an environmental impact statement




8
 Pub. L. No. 91-190, 83 Stat. 852 (1970), codified as amended at 42 U.S.C. §§ 4321-4347
(2011).
9
 An environmental assessment is a concise public document that provides sufficient
evidence and analysis for determining whether to prepare an environmental impact
statement or a finding of no significant impact and is to include brief discussions of the
need for the proposal, alternatives, the environmental impacts of the proposed action and
alternatives, and a listing of agencies and persons consulted. 40 C.F.R. § 1508.9 (2011).




Page 5                                                GAO-12-352 Army Corps of Engineers
                              (EIS). 10 NEPA implementing regulations also specify requirements and
                              procedures—such as providing the public with an opportunity to comment
                              on the draft EIS for at least 45 days.

                              Corps project studies have historically been subject to various levels of
                              internal and external review under a number of authorities as well as the
                              Chief of Engineers’ responsibility to ensure the quality of Corps studies.
                              For example, in 1902, Congress created the Board of Engineers for
                              Rivers and Harbors, which was the result of efforts to address
                              inconsistent treatment of proposed Corps projects. The board was made
                              up of Corps staff. Until 1992, when Congress terminated the board, it
                              reviewed thousands of Corps studies for civil works projects and made
                              unfavorable recommendations on more than half. At the time the board
                              was abolished, there was concern that too much duplicative review was
                              occurring between the board and other internal Corps review processes.
                              Subsequently, in the Flood Control Act of 1944, Congress established a
                              mechanism for external review of Corps projects by giving the head of the
                              Department of the Interior and the governors of affected states an
                              opportunity to comment on proposed Corps projects before authorization.
                              Furthermore, starting in 1970, under NEPA, environmental impact
                              statements for Corps projects were required to be sent to the heads of
                              other federal agencies and governors of affected states for comment. As
                              a result of the Flood Control Act of 1970, Congress created the position of
                              Assistant Secretary of the Army for Civil Works, who coordinates the
                              review of Corps studies with the Office of Management and Budget
                              (OMB) before they are submitted to Congress.


Increasing Interest in Peer   Recent congressional interest in establishing an independent external
Review of Corps Project       peer review process for Corps project studies began in the late 1990s,
Studies                       following a series of damaging reports and events, including allegations
                              that the Corps had manipulated information to justify projects.
                              Investigations conducted by NAS and the Army’s Inspector General
                              identified various problems with the Corps internal review process,



                              10
                                An EIS is a more detailed statement than an environmental assessment. An EIS must,
                              among other things, (1) describe the environment that will be affected, (2) identify
                              alternatives to the proposed action and identify the agency’s preferred alternative, (3)
                              present the environmental impacts of the proposed action and alternatives, and (4) identify
                              any adverse environmental impacts that cannot be avoided should the proposed action be
                              implemented. 42 U.S.C. § 4332(c) (2011), 40 C.F.R. §§ 1501.4, 1508.11 (2011).




                              Page 6                                               GAO-12-352 Army Corps of Engineers
including the manipulation of economic analysis and potential institutional
bias toward large construction projects. 11 Around this time, WRDA 2000
required the Corps to contract with NAS to study and make
recommendations concerning the use of peer review for feasibility
reports, including recommending potential criteria to determine how to
apply peer review. 12 In 2002, NAS released its study concluding that the
Corps’ more complex water resources project planning studies should be
subject to external, independent review. 13 The study also found that not
all Corps project studies necessarily require such review, recommending
instead that external peer review be reserved for studies that are
expensive, will affect a large area, are highly controversial, or involve high
levels of risk. The study estimated that about five Corps projects per year
would likely be subject to this level of review. According to the NAS study,
criteria for selecting the appropriate level of review should balance the
risks and consequences of inadequate review against the resources
required for more complex and stringent levels of review. In addition, the
study identified several criteria that should be considered in determining
the appropriate level of review for Corps studies, primarily that as project
magnitude and risks increase, an increasing degree of independence and
scope of review are warranted (see fig. 1).




11
 National Research Council, Review of the U.S. Army Corps of Engineers Restructured
Upper Mississippi River-Illinois Waterway Feasibility Study (Washington, D.C.: National
Academies Press, 2004), and National Research Council, New Directions in Water
Resources Planning for the U.S. Army Corps of Engineers (Washington, D.C.: National
Academies Press, 1999).
12
 Water Resources Development Act of 2000, Pub. L. No. 106-541, § 216(b) (2000).
13
  National Research Council, Review Procedures for Water Resources Project Planning
(Washington, D.C.: National Academies Press, 2002).




Page 7                                              GAO-12-352 Army Corps of Engineers
Figure 1: National Academy of Sciences Criteria for Selecting an Appropriate Level of Review




                                         In addition to recommendations related to the appropriate level of review
                                         for project studies, the 2002 NAS study made several other
                                         recommendations about the Corps’ peer review process. It recommended
                                         that peer review results be presented to the Chief of Engineers before a
                                         final decision on a project study is made, that the Chief of Engineers
                                         respond in writing to each key point of the peer review report, and that
                                         peer review be initiated early enough in the Corps’ study process so that
                                         review results can be meaningfully incorporated into project design.

                                         After NAS published its 2002 study, OMB in December 2004 issued its
                                         Final Information Quality Bulletin for Peer Review citing the Information
                                         Quality Act, 14 as well as its general authorities to oversee the quality of
                                         agency information, analyses, and regulatory actions. This OMB bulletin



                                         14
                                           Pub. L. No. 106-554, App. C, Title V, § 515(a) (2000).




                                         Page 8                                                GAO-12-352 Army Corps of Engineers
                          established governmentwide guidance on enhancing peer review
                          practices and covers what information is subject to peer review, the
                          selection of appropriate peer reviewers, opportunities for public
                          participation, and related issues. 15 The Corps’ Engineering Circular 1105-
                          2-408 (EC 408) was issued in May 2005 and established procedures for
                          ensuring the credibility and quality of Corps documents by supplementing
                          its previous review process, including to add external peer review to its
                          review process in special cases where risk and magnitude warrant this
                          level of review.

                          The Corps faced further criticism after the failure of Corps levees and
                          floodwalls in New Orleans in the wake of Hurricane Katrina in August
                          2005. In 2006, the Corps announced “Twelve Actions for Change,” which
                          included a set of actions intended to transform the Corps’ priorities,
                          processes, and planning and apply lessons learned from Hurricanes
                          Katrina and Rita. Among these actions was to employ independent review
                          for projects with significant consequences, especially the potential for loss
                          of life if the project were to fail.


Section 2034 of WRDA of   In November 2007, Congress passed WRDA 2007 and included section
2007                      2034, which establishes a 7-year trial period for peer reviews of certain
                          studies of civil works projects; this trial generally applies to project studies
                          initiated by the Corps from November 2005 through November 2014. 16
                          The Corps was to provide an initial report to Congress on its
                          implementation of the peer review trial under section 2034 by November
                          2010 and is to provide a final report by November 2013. In February
                          2011, the Corps submitted its initial report to Congress summarizing its




                          15
                            Office of Management and Budget, Final Information Quality Bulletin for Peer Review,
                          M-05-03 (Washington, D.C.: 2004). The OMB guidance for peer review applies to
                          important scientific assessments, referred to as “influential scientific information,” which
                          includes “highly influential scientific assessments,” disseminated by the federal
                          government. The guidance calls for peer review of information that is based on novel
                          methods or presents complex challenges for interpretation, contains precedent-setting
                          methods or models, presents conclusions that are likely to change prevailing practices, or
                          is likely to affect policy decisions that have a significant impact.
                          16
                            Section 2034 applies to (1) project studies initiated from November 2005 through
                          November 2007 and for which the array of alternatives to address the water resources
                          problem had not been identified and to (2) project studies initiated from November 2007
                          through November 2014.




                          Page 9                                                 GAO-12-352 Army Corps of Engineers
experiences implementing the peer review process in response to the
requirement in section 2034 of WRDA 2007.

Section 2034 defines a project study as a feasibility or reevaluation
study—including the EIS for that study—or any other study associated
with a modification of a water resources project that includes an EIS.
Under section 2034, project studies that meet at least one of the following
criteria are required to undergo peer review:

•   The project has an estimated total cost of more than $45 million.

•   The governor of an affected state requests an independent peer
    review.

•   The Chief of Engineers determines that the project study is
    controversial (i.e., significant public dispute exists as to the project’s
    size, nature, or effects or its economic or environmental costs or
    benefits).
In addition, if the head of a federal or state agency charged with reviewing
a project study determines the project is likely to have a significant
adverse impact on environmental, cultural, or other resources, he or she
may request that the Corps consider a peer review by an independent
panel of experts.

WRDA 2007 also provides some instances where exceptions may be
made to peer review. For example, the Corps may exclude from peer
review certain projects having a total estimated cost of more than
$45 million but do not include an EIS, have not been determined by the
Corps to be controversial, and come below specified thresholds of
adverse impacts. The Corps may also exclude other project studies
meeting certain exclusion criteria. For example, the Corps may exclude
studies that involve

•   only the rehabilitation or replacement of existing hydropower turbines,
    lock structures, or flood control gates within the same footprint and for
    the same purpose as an existing water resources project;

•   an activity for which the Corps and industry have ample experience,
    so the activity may be considered routine; and

•   minimal life safety risk.




Page 10                                        GAO-12-352 Army Corps of Engineers
                             Section 2034 also has requirements for the Corps concerning the peer
                             review panel and its independence, as well as for timing the peer review
                             and publishing peer review reports, as described in more detail below.


Contractor Selection,        Under section 2034, the Corps is required to contract with NAS, a similar
Conflicts of Interest, and   independent scientific and technical advisory organization, or an “eligible
Independence                 organization” to establish a panel of experts that will review a project
                             study. Section 2034 defines an eligible organization as having the
                             following five characteristics:

                             •   is a 501(c)(3) tax-exempt organization,

                             •   is independent,

                             •   is free from conflicts of interest,

                             •   does not carry out or advocate for or against federal water resources
                                 projects, and

                             •   has experience in establishing and administering panels.

                             Section 2034 states that when establishing peer review panels,
                             contractors must apply NAS’s policy for selecting committee members to
                             ensure that they also have no conflicts of interest. The NAS Policy on
                             Committee Composition and Balance and Conflicts of Interest outlines
                             several criteria for selecting peer review panel members, including the
                             following:

                             •   All panel members must be highly qualified in terms of knowledge,
                                 training, and experience.

                             •   The knowledge, experience, and perspectives of the panel members
                                 must be thoughtfully and carefully assessed and balanced in terms of
                                 the subtleties and complexities of the particular scientific, technical,
                                 and other issues to be addressed.

                             •   Potential sources of bias must be assessed to determine that the
                                 panel’s report will not be compromised by issues of bias or lack of
                                 objectivity.




                             Page 11                                       GAO-12-352 Army Corps of Engineers
•    Panel members must not have financial interests that could
     significantly impair their objectivity or create an unfair competitive
     advantage for any person or organization.

•    Panel members must not obtain and use, or intend to use, confidential
     information not reasonably available to the public for their own or
     direct and substantial economic benefit.

•    Panel members must not serve as a member on a peer review panel
     that is to review the panel member’s own work.

•    Panel members must not have become committed to a fixed position
     related to the review for which they have a significant directly related
     interest or duty.

•    Persons currently employed by the agency sponsoring the study
     cannot be panel members, except in extremely limited special
     circumstances. 17

Additionally, section 2034 requires that both the experts selected for the
peer review panels and the organizations managing the peer review
selections be independent. Section 2034 does not define the term
independent, but both the 2002 NAS peer review study and OMB’s Final
Information Quality Bulletin for Peer Review regard independent to mean
external to the Corps. 18 Specifically, the NAS study states that a fully
independent review can be accomplished only by reviewers who are free
of conflicts of interest and are appointed by a group external to the Corps.
Similarly, the OMB bulletin states that independent reviewers are
generally not employed by the agency or office producing the document.



17
  In special circumstances and to the extent not prohibited by federal or state laws or
regulations, such an individual may serve as a member of such a committee where the
following requirements are met: (1) the service of the individual on the committee must be
based upon the unique scientific or technical expertise the individual brings to the
committee; (2) the individual must not be involved in any way within the agency in any
deliberative or decision-making process or any policymaking or similar process relating to
the study or other activity or the expected or intended results of the study or other activity;
and (3) it must be specifically determined during the committee appointment process that
service by the individual will not compromise, or appear to compromise, the independence
or objectivity of the particular study or other activity in which the committee is engaged.
18
 National Research Council, Review Procedures (2002), and Office of Management and
Budget, Final Information Quality Bulletin for Peer Review (2004).




Page 12                                                 GAO-12-352 Army Corps of Engineers
Timing of Peer Review and   Section 2034 requires that the peer review be conducted during the
Publishing of Peer Review   period from the signing of the feasibility cost-share agreement between
Reports                     the Corps and the local sponsor and 60 days after the last day of the
                            public comment period for the draft project study. Additionally, section
                            2034 lists three points during the feasibility study process at each of
                            which the Chief of Engineers must consider whether to initiate peer
                            review:

                            •   when the without-project conditions—current and forecasted
                                conditions if the project were not constructed—are identified,

                            •   when the array of alternatives to be considered are identified, and

                            •   when the preferred alternative is identified.

                            Figure 2 shows the key steps in the feasibility study process, including
                            those specified in section 2034. The Corps can conduct peer review at
                            any time during the steps shown highlighted in gray in the figure, but
                            according to Corps officials it generally conducts peer review after the
                            draft feasibility report has been completed. The Washington-level review
                            shown as the final step in the figure concludes with a signed Chief’s
                            report for project studies that will be submitted to Congress for
                            authorization.




                            Page 13                                       GAO-12-352 Army Corps of Engineers
Figure 2: Key Steps in the Feasibility Study Process




                                         Note: The Corps’ independent peer review process can take place at any time during the steps
                                         enclosed by the shaded gray area.
                                         a
                                          The feasibility scoping meeting brings the Corps, local sponsor and relevant government agencies
                                         together to reach agreement on the problems and solutions to be investigated during the feasibility
                                         study. The without-project conditions are identified during the feasibility scoping meeting.
                                         b
                                          The alternative formulation briefing confirms that the plan formulated for the project study and the
                                         definition of federal and nonfederal responsibilities are consistent with applicable laws, statutes,
                                         executive orders, regulations, and current policy guidance. The array of alternatives to be considered
                                         is identified at the alternative formulation briefing.
                                         c
                                          The preferred alternative is identified when the draft feasibility report is released.




                                         Page 14                                                         GAO-12-352 Army Corps of Engineers
                           Section 2034 also requires the Corps to prepare and make publicly
                           available a written response to all completed peer review reports before it
                           finalizes project studies. The Corps must provide Congress with a copy of
                           both the completed peer review report and the Corps’ written response
                           when the signed Chief’s report or other final decision document for the
                           project study is transmitted to Congress.


Corps Guidance for         The Corps’ Engineering Circular 1165-2-209 (EC 209) was issued in
Implementing Peer Review   January 2010 and establishes its civil works review policy, which outlines
                           the processes for implementing product review requirements for Corps
                           civil works projects. EC 209 was developed to include the specific
                           requirements for independent peer review contained in section 2034,
                           OMB’s 2004 peer review guidance, as well as other Corps policy
                           considerations. EC 209 requires that districts, in coordination with the
                           relevant Corps planning center of expertise, prepare review plans for
                           project studies. These review plans are to describe the appropriate levels
                           of potential review that the specific project study will be subject to, such
                           as the district’s quality control procedures, agency technical review, peer
                           review, and policy and legal review. If a project study review plan
                           indicates that a peer review will not be conducted, then the district is
                           required to develop a risk-based recommendation for why the peer review
                           is not required. This recommendation should document, among other
                           things, that the project study is of such limited scope or impact that it
                           would not benefit from a peer review. 19


                           Since enactment of WRDA 2007, 49 Corps civil works project studies
It Is Unclear How          have undergone peer review as of January 2012, but it is unclear how
Many Peer Reviews          many of these reviews were performed in response to the requirements in
                           section 2034. This is because the Corps does not make specific
Have Been Completed        determinations or track whether a peer review is being conducted in
in Response to             response to the requirements of section 2034. Of the 49 project studies
Section 2034               that underwent peer review, the majority were for ecosystem restoration
                           projects, flood risk management projects, or deep draft navigation
Requirements               projects. (App. II lists the 49 project studies that underwent peer review
                           since WRDA 2007 was passed, including information on project and


                           19
                             Earlier Corps guidance related to peer review was issued in August 2008 in Engineering
                           Circular 1105-2-410 (EC 410), Review of Decision Documents, and in May 2005 in
                           Engineering Circular 1105-2-408 (EC 408), Peer Review of Decision Documents.




                           Page 15                                             GAO-12-352 Army Corps of Engineers
                         study type, as well as the district, division, and planning center of
                         expertise associated with each study.)

                         Moreover, it is not possible to determine how many project studies were
                         required to undergo peer review in response to WRDA’s section 2034
                         requirements because the Corps does not centrally track project studies
                         and could not provide us a list of all project studies that fell within the
                         scope of section 2034. Our review of relevant Corps documents for the 49
                         project studies that underwent peer review, such as review plans and
                         completed review reports, found that none of these documents specifies
                         the authority under which peer reviews were conducted. Corps
                         headquarters officials told us that the Corps does not make specific
                         determinations as to whether a peer review is being conducted under
                         section 2034 but instead focuses on ensuring that the peer review is
                         being carried out in compliance with EC 209, which, in their view,
                         complies with section 2034. These officials also told us that, to ensure the
                         quality of Corps project studies, the agency may choose to conduct peer
                         reviews under its other authorities, even if those peer reviews are not
                         required by section 2034.

                         In February 2011, the Corps submitted its initial report to Congress in
                         response to the requirement in section 2034 of WRDA 2007 summarizing
                         its experiences implementing the peer review process. In the report, the
                         Corps noted that the 29 peer reviews that had been completed as of
                         February 2011 followed the procedures described in agency and OMB
                         guidance. The Corps report stated that, in its view, section 2034
                         provisions reinforce and add further definition to the Corps’ process.
                         Nevertheless, because the Corps did not distinguish which studies had
                         been selected for peer review in accordance with section 2034, we
                         believe that it did not provide Congress with the type of information
                         required by section 2034 that would help congressional decision makers
                         evaluate the trial program.


                         The 49 peer reviews conducted by the Corps since November 2007
Completed Peer           resulted in direct costs of about $9 million in contract costs and contract
Reviews Have Cost        administration fees. In addition, Corps staff resources were also used to
                         manage peer reviews, but these costs are not fully quantifiable.
Millions of Dollars in   Furthermore, the addition of peer review to the Corps study process has
Direct and Indirect      resulted in indirect costs by altering project study schedules because of
Costs                    the additional time required to complete the peer review. In some cases
                         where a peer review was not planned for during the early stages of the



                         Page 16                                       GAO-12-352 Army Corps of Engineers
                            study process, significant delays in the project studies have resulted from
                            the addition of the peer review.


Direct Costs Related to     The 49 project studies for which the Corps completed peer reviews since
Peer Reviews Have Totaled   November 2007 cost about $9 million in contract costs and contract
Over $9 Million             administration fees to establish and manage the expert panels for these
                            reviews. In addition, Corps staff resources were also used to manage the
                            peer reviews, but these costs are not fully quantifiable. The Corps used
                            the services of three contractors to manage the peer review process: the
                            nonprofit Battelle Memorial Institute, which managed 46 of the reviews;
                            the nonprofit Noblis, which managed two; and NAS, which managed one.
                            The cost per panel varied considerably. For example, the contracts
                            managed by Battelle cost from about $76,000 to $484,000 for studies that
                            underwent peer review, 20 but the panel managed by NAS cost over
                            $500,000. (See app. II for information on the contract costs for each of
                            the 49 peer reviews.) In addition to the $9 million in contractor costs, the
                            Corps incurred about $109,000 for the administration of the contracts for
                            the 49 peer reviews. Specifically, the Corps used two different entities—
                            the Institute for Water Resources and the Army Research Office—to
                            administer these contracts, and both of these entities charged
                            administration fees. 21 These fees ranged from no fee to 3 percent of the
                            contract cost.

                            Corps staff resources were also used to manage the peer review
                            process—including developing the scope of work for reviews,
                            coordinating establishment of contracts, reviewing contract proposals,
                            and responding to panel comments that the Corps received during a peer
                            review process. Corps district, division, headquarters, and planning-
                            center-of-expertise staff also spent time managing the peer review
                            process. The total of these costs, however, is not fully quantifiable across


                            20
                              In some cases, individual project studies underwent multiple phases of peer review
                            because of revisions made to the study after the initial peer review report, and peer review
                            was completed under multiple reports and contracts. This dollar range does not include
                            one contract that cost about $677,000 because the cost covered peer review for five
                            studies.
                            21
                              The Institute for Water Resources is the Corps center of expertise for integrated water
                            resources management. The U.S. Army Research Laboratory's Army Research Office is
                            the Army's research agency in the engineering, physical, information, and life sciences.
                            These entities’ contracting offices administer the Corps’ contracts for establishing and
                            managing peer review panels.




                            Page 17                                               GAO-12-352 Army Corps of Engineers
                             all Corps districts because not all districts track district staff time spent on
                             peer reviews. For those districts that did track or could estimate district
                             staff time spent on peer review-related activities, we found the following
                             examples of the staff resources that may have been dedicated to
                             managing and responding to peer review activities:

                             •   The Green Bay Dredged Material Management Plan peer review cost
                                 about $101,000 in staff time, according to data provided by district
                                 officials. But district officials involved in this peer review said that the
                                 cost in terms of staff time may have been higher than typical because
                                 this peer review was the first conducted for a study in that district.

                             •   The Chatfield Water Reallocation Study peer review cost about
                                 $20,000 in staff time as of December 2011, but the Corps response
                                 has not been completed for this peer review, and additional staff time
                                 could be involved.

                             •   For the Boston Harbor study, district officials estimated that costs
                                 totaled about $77,000 for district staff time, agency technical review
                                 team labor, and contractor fees for assisting the district with
                                 responding to peer review panel comments.

                             •   For the American River study, district officials estimated that costs
                                 came to about $40,000 for district staff time.

                             Similar to district staff time, other staff time involved in managing the peer
                             review process, including headquarters, division, and some planning
                             centers of expertise time, is also not always tracked and therefore not
                             fully quantifiable: some of these positions are funded with general funds
                             and not project-specific funds, according to Corps officials we spoke with.
                             We did find two examples, however, where planning centers of expertise
                             staff time devoted to peer review related activities was tracked. In these
                             two instances, the cost of planning centers of expertise staff time devoted
                             to peer review activities amounted to about $12,000 for the peer review of
                             the Boston Harbor study and about $32,000 for the American River study.


Addition of Peer Review      The addition of peer review to the Corps study process has also had an
Has Affected Project Study   indirect cost because it has affected project study schedules. Planning
Schedules                    centers of expertise and district officials estimate that obtaining the
                             contract and executing the peer review generally take about a year. The
                             breakdown for the peer review process, according to some of these
                             officials, is about 3 months to initiate the contract; 3 to 6 months for the



                             Page 18                                        GAO-12-352 Army Corps of Engineers
review to be completed; and an additional 3 months to close out the
review, which involves responding to and receiving clarification on panel
comments. Some of these processes occur concurrently with other
aspects of the project study, but some parts of the peer review process,
such as responding to panel comments, may add time to the study
schedule. According to Corps officials, the addition of peer review to the
project schedule adds steps to the review process and takes time away
from other projects. In addition, according to several Corps officials, some
project studies have been delayed because the district did not allocate
funding for the peer review and therefore had to wait until additional
funding was available. In some cases, this delay added significant time to
the schedule. In contrast, according to some Corps division and planning-
center-of-expertise officials, when the project manager had built in time
for the peer review and had identified funding for it early, the peer review
process had much less of an impact on the overall project study
schedule.

Local sponsors are also concerned about the impact that this additional
time is adding to project studies, according to Corps officials and local
sponsors we spoke with. Their concern arises largely because local
sponsors share the cost of the Corps study and depend on its timely
completion. District officials told us that because of the cost-sharing
requirements and the current economic environment demand is greater
from local sponsors for the Corps to finish studies quickly and keep costs
down. Two local sponsors told us that delays negatively affect local
sponsors because they can lose business if a project is not completed in
a timely manner. Similarly, sponsors are accountable to their own local
governments or state legislatures, and additional delays or time required
for peer review can create challenges in getting continued support for a
project. For example, in the case of the Green Bay Dredged Material
Management Plan, Corps officials told us that peer review increased the
cost of the project and caused a 5- to 6-month delay at a time when the
local sponsor was attempting to acquire grant money contingent on
completion of the dredged material management plan.




Page 19                                      GAO-12-352 Army Corps of Engineers
                           The Corps’ process for determining whether a project study is subject to
The Corps’ Process         peer review is more expansive than section 2034 requirements because it
for Initiating Peer        uses broader criteria; this has resulted in peer reviews of studies that are
                           outside the scope of section 2034. In addition, the process the Corps
Review Is More             uses does not include the flexibility provided in section 2034 to exclude
Expansive and Less         certain project studies from peer review. Moreover, some studies are
Flexible Than Section      undergoing peer review that do not warrant it, according to some Corps
                           officials we spoke with.
2034 Requirements
The Corps’ Process for     The Corps’ process for determining whether a project study is subject to
Determining the Need for   peer review uses criteria that are broader than the requirements of
Peer Review Is More        section 2034. As table 1 shows, the Corps relies on its guidance outlined
                           in EC 209 when selecting project studies for peer review, and this
Expansive Than Section     guidance extends beyond section 2034 requirements.
2034 Requirements
                           Table 1: Corps EC 209 Guidance on Conducting Peer Review

                                                                                                           Basis
                                                                                             Section          OMB
                                                                                                                                  a
                               Criteria for conducting peer review                            2034           bulletin     Other
                                                                     b
                               Significant threat to human life                                                               X
                               The total project cost is greater than $45 million                X
                               Request by the governor of an affected state                      X
                               Request by the head of a federal or state agency                  X
                                                                                                  c
                               Significant public dispute as to size, nature, or                 X
                               effects of the project
                               Significant public dispute as to the economic or
                                                                                                  c
                               environmental cost or benefit of the project                      X
                               Information is based on novel methods                                       X
                               Presents complex challenges for interpretation                              X
                               Contains precedent-setting methods or models                                X
                               Presents conclusions that are likely to change                              X
                               prevailing practices
                               Any other circumstance where the Chief of                                                  X
                               Engineers determines peer review is warranted
                           Source: GAO analysis of Corps guidance.

                           a
                            For example, Corps officials noted that the criteria for peer review reflect the agency’s general
                           authority to conduct peer review in association with its responsibility to produce quality projects.
                           b
                            Section 2035 of WRDA 2007 requires that certain projects in design or under construction undergo
                           independent peer review in cases where the Chief of Engineers determines a safety assurance
                           review is necessary, such as where failure of a project would pose a significant threat to human life.




                           Page 20                                                        GAO-12-352 Army Corps of Engineers
c
 Section 2034 criteria include that the Chief of Engineers determines that the study is controversial
considering whether (a) significant public dispute exists as to the size, nature, or effects of the project;
or (b) significant public dispute exists as to the economic or environmental costs or benefits of the
project.

Consequently, the Corps has selected some studies for review based in
part on criteria included in EC 209 that are not required by section 2034,
and others that are outside the scope of section 2034. For example,
according to our analysis of review plans for 44 peer reviewed project
studies, 22 over one-third identified criteria that related to both section
2034 and other authorities. In addition, the Corps process for determining
whether a peer review is required has resulted in 30 project studies
undergoing peer review that were outside the time parameters identified
in section 2034. Based on our analysis of the characteristics of these
studies, the Corps’ process was applied to all studies and reports
regardless of when they were initiated, 23 whereas section 2034 applies to
project studies initiated from November 2005 through November 2014.
Specifically, section 2034 applies to (1) project studies initiated from
November 2005 through November 2007 and for which the array of
alternatives had not been identified, and (2) project studies initiated from
November 2007 through November 2014. As a result, over half (30 of 49)
of the peer reviews conducted since the enactment of WRDA 2007 were
for project studies that did not fall under the scope of section 2034
because the studies were initiated before November 2005. 24

Another reason the Corps’ process for selecting studies for peer review is
more expansive than the scope of section 2034 is that Corps’ guidance


22
  Of the 49 peer reviews that the Corps conducted, 4 review plans for studies that
underwent peer review were not updated to reflect the decision to conduct peer review
and indicate that peer review was not required, and 1 project study that underwent peer
review did not have a review plan completed, so our analysis was based on 44 review
plans.
23
  EC 209 includes an exception for cases where the final decision document package had
been forwarded to headquarters before August 22, 2008.
24
  We requested a list of studies that came within the scope of section 2034 but the Corps
does not track this information, however, and it requested this information from each of the
districts. Through this effort, the Corps identified 134 studies with the caveat that the list
was not complete and 18 of the 49 studies that underwent peer review were included on
this list. In addition, we made our initial request for this list of studies in June and did not
receive it until mid-December. Given the delay and the list’s incompleteness, we did not
evaluate the other 116 studies on the list for whether peer review is required or planned.
We note it is likely that some of the 116 studies may not yet have review plans nor
decisions on whether peer review is to be conducted.




Page 21                                                         GAO-12-352 Army Corps of Engineers
                             does not clearly define “project study.” The guidance refers to a wide
                             range of project studies, decision documents, and work products that may
                             be subject to peer review, whereas section 2034 defines a project study
                             subject to peer review as a feasibility or reevaluation study, including the
                             EIS, or any other study associated with the modification of a water
                             resources project that includes an EIS. According to our analysis of the
                             49 studies that underwent peer review, some of these studies did not fit
                             this definition. Specifically, 34 of the 49 studies that underwent peer
                             review were feasibility or reevaluation studies which are project studies as
                             defined by section 2034 requirements, 8 were other kinds of reports that
                             included an EIS and therefore may have been subject to section 2034
                             requirements, and 7 were neither feasibility nor reevaluation studies and
                             did not include an EIS and therefore did not fit the definition of a project
                             study subject to peer review under section 2034. For more details on
                             each of the studies that underwent peer review, see appendix II.


The Corps Does Not Use       The Corps’ process for determining whether peer review is required for
the Flexibility in Section   project studies does not include the flexibility provided in section 2034 to
2034 to Exclude Studies      exclude certain project studies from otherwise mandatory peer review. EC
                             209 states that most studies should undergo peer review, and the Corps’
from Review                  process requires that for any decision document to forgo a peer review,
                             an exclusion must be requested and approved by headquarters. In
                             addition, guidance provided to Corps staff on how to implement EC 209
                             discourages requests for exclusions, noting that time should not be
                             wasted shopping around for exclusion requests. Furthermore, agency
                             guidance and Corps headquarters officials, including the Director of Civil
                             Works, highlight the value and importance of peer review in achieving the
                             agency’s mission, noting that an extra set of eyes is beneficial. In
                             addition, Corps headquarters officials told us, and agency guidance
                             highlights language from the WRDA 2007 conference report, that
                             “[s]ection 2034 permits the Chief of Engineers to exclude a very limited
                             number of project studies from independent peer review.” 25 We believe,
                             however, that the Corps has misconstrued this statement and overstated
                             its significance. This statement is part of the explanation of the exclusion
                             paragraph (a)(5), and does not apply to the provision as a whole;
                             therefore, this statement pertains to how many studies for which peer




                             25
                              H.R. Conf. Rep. 110-280, at 267.




                             Page 22                                      GAO-12-352 Army Corps of Engineers
review is mandatory would be eligible for exclusion. 26 Furthermore,
another relevant statement in a House committee report on WRDA 2007
suggests that 26 studies over 7 years, or about 4 studies per year, would
be expected to be subject to peer review. Additionally, the 2002 NAS
study—which is prominently mentioned throughout the subsequent
legislative history of WRDA 2007—states that not all Corps water
resources project planning studies will require external, independent
review, but the Corps should institute external review for studies that are
expensive, that will affect a large area, that are highly controversial, or
that involve high levels of risk.

In reviewing the exclusion requests that it receives, Corps headquarters
determines whether the studies meet any of the mandatory requirements
in EC 209 for undergoing peer review. Specifically, the Corps reviews
whether the project has a cost estimate of greater than $45 million,
represents a threat to health and safety, is controversial, and has had a
request for peer review from a governor or the head of a federal or state
agency. If studies do not meet any of these criteria, the Corps generally
approves the study for exclusion from peer review. From our review of 50
studies that had requested exclusion from peer review between 2009 and
2011, we found that the Corps had granted an exclusion for 37 studies
because they did not meet any of the criteria in EC 209 for studies that
must undergo peer review except for one study, which did not fit the
definition of a project study in section 2034. 27

Under section 2034, however, the Corps also has the flexibility to exclude
studies from peer review that exceed the $45 million threshold if they: do
not have an EIS; are not controversial; are expected to have negligible
adverse impacts on scarce or unique cultural, historic, or tribal resources;
have no substantial impacts on fish and wildlife species or their habitats;
and have no more than negligible impacts on threatened or endangered
species or their critical habitat. Similarly, under section 2034, the Corps



26
  H. Rep. 110-80, at 132 (discussing H.R. 1495 § 2037) (Mar. 29, 2007).
27
  This study exceeded the threshold of $45 million, but Corps headquarters’ approval of
the request for exclusion said it was excluded because it did not meet any of the other
mandatory criteria for peer review. It was not a feasibility study or a reevaluation study and
did not include an EIS. Of the 50 studies that went through the exclusion process as of
November 2011, the Corps excluded 37. The Corps decided not to exclude 7 of these
studies and determined that 4 did not need to request an exclusion. A decision is pending
for 1 study and has been deferred for another study.




Page 23                                                GAO-12-352 Army Corps of Engineers
may exclude studies from peer review that involve (1) only the
rehabilitation or replacement of existing hydropower turbines, lock
structures, or flood control gates within the same footprint and for the
same purpose as an existing water resources project; (2) an activity for
which there is ample experience within the Corps and industry to treat the
activity as being routine; and (3) minimal risk to human life and safety.
Nevertheless, according to our analysis of exclusion request documents
and headquarters’ responses to these requests, as of November 2011,
the Corps had not granted an exclusion based on any of the flexibilities
included in section 2034.

Several Corps officials expressed concerns about the Corps exclusion
process. Specifically, some officials told us that they were concerned
about the cost and time involved and said that the exclusion of projects
that do not meet any of the mandatory criteria should be delegated to the
division offices. In their opinion such delegation would help streamline the
process. Moreover, some of the studies that underwent or are currently
undergoing peer review did not warrant it, according to some Corps
officials we spoke with. Specifically, we found the following examples of
studies that may not have warranted a peer review:

•   Two dredged material management plans underwent peer review. For
    example, the Green Bay Dredged Material Management Plan
    underwent peer review but is not a project study as defined by section
    2034. Officials we spoke with said that such plans should not
    generally require peer review because any significant impacts would
    be addressed under NEPA and because the Corps has sufficient
    expertise in the area of dredging.

•   The Chacon Creek study in southern Texas underwent peer review
    but should not have, according to some Corps officials we spoke with.
    This study was for a project that would remove houses from a
    floodplain, but officials said it should not require peer review because
    there are no structural components and it did not exceed the $45
    million threshold. Corps headquarters denied the request for exclusion
    and stated that flood studies warrant peer review because of the
    nature of the hazard and the need to assess the extent and treatment
    of risk. Headquarters officials highlighted the importance of assessing
    and addressing such risks in light of Hurricane Katrina and said that
    flood studies such as Chacon Creek require peer review because of
    the importance of assessing and decreasing risks associated with
    flooding.




Page 24                                      GAO-12-352 Army Corps of Engineers
                        •   The Yuba River General Reevaluation study is undergoing peer
                            review but does not warrant it according to some Corps officials.
                            District officials told us that the study does not warrant peer review
                            because the construction work involved has already been completed,
                            and the purpose of the study is to determine the amount the local
                            sponsor should be reimbursed by the Corps.

                        Officials from several districts, divisions, and planning centers of expertise
                        we spoke with told us that peer review should be focused on larger and
                        more complex or controversial projects and should not be the default
                        approach. Two Corps officials described the Corps’ peer review policy as
                        a one-size-fits-all approach, and one of these officials stated that it is
                        inflexible and risk averse.


                        The Corps has a process to review general information on contractors’
Gaps Exist in the       conflicts of interest and independence during its contractor selection
Corps’ Process for      process, but it does not have a process for reviewing project-specific
                        information provided by contractors to determine if conflicts of interest
Screening Its           and independence exist at the project level. The Corps’ contractors,
Contractors Who Are     however, have a process for reviewing the appropriate information related
                        to the conflicts of interest and independence of the experts selected for
Responsible for         peer review panels at the project level.
Selecting Experts for
Peer Review Panels      For its initial peer reviews, the Corps relied on Battelle to establish and
                        manage the peer review panels. From 2007 to 2009, Battelle managed 15
                        independent peer reviews for the Corps. The Corps had identified Battelle
                        as a potential contractor for managing its peer review panels as early as
                        August 2007, when WRDA 2007 was being considered. To ensure that
                        Battelle could meet the section 2034 independence requirements,
                        according to Corps officials, the Director of Civil Works and the Chief of
                        Planning and Policy held discussions with Battelle, and officials from the
                        Corps’ Institute for Water Resources met with Battelle to discuss
                        Battelle’s existing review process and the independent peer review
                        requirements of WRDA 2007. Battelle informed the Corps that it met all
                        WRDA 2007 requirements for an eligible organization, and Battelle
                        identified its existing contract with the Army Research Office as a vehicle
                        for employing Battelle to establish and manage peer review panels under
                        section 2034.

                        During that time, NAS also conducted one independent peer review for
                        the Corps on the Louisiana Coastal Protection and Restoration Program,
                        which charged the Corps with developing a full range of flood control,


                        Page 25                                       GAO-12-352 Army Corps of Engineers
coastal restoration, and hurricane protection measures for South
Louisiana. According to Corps planning-center-of-expertise officials,
because of the extensive scope and breadth of the project, NAS was
chosen instead of Battelle to conduct that peer review. But Corps
headquarters and planning-center-of-expertise officials told us that, over
the course of NAS’ review, they realized that NAS would not be the
appropriate organization for reviewing individual projects studies because
its process was too time-consuming and expensive. A member of that
NAS peer review panel also told us that while he would recommend NAS
review for larger projects, in his opinion NAS might not be the appropriate
organization for reviewing smaller Corps projects.

In 2009, the Corps sought additional contractors to establish and manage
peer review panels and began its contractor selection process by putting
out a request for proposals. This solicitation included as contract
requirements the section 2034 criteria that the organizations establishing
and managing peer review panels be independent and free from conflicts
of interest. The Corps received six proposals, including one from Battelle,
and each of these proposals was then evaluated by a three-person
review panel. The panel chairperson told us that the section 2034 criteria
that eligible organizations be independent and free from conflicts of
interest were considered as minimum qualifications for screening and
selection. 28 As a result of this process, the Corps awarded a contract to
Battelle—this contract was in addition to the existing contract Battelle
already had with the Corps through the Army Research Office—and one
to Noblis. 29 The Corps determines which of the two contractors it will use
to manage individual peer reviews on the basis of the contractors’
responses to specific project study scope of work requests, described
below.

Although the Corps’ contractor screening and selection process identifies
general contractor independence and areas of conflicts of interest, the
Corps does not have a process for reviewing the selected contractors’


28
  The summary evaluation of contract proposals does not discuss the section 2034
eligibility criteria; although, the proposals from both Battelle and Noblis address the criteria
and state that those organizations meet all section 2034 criteria for eligible organizations.
29
  Initially, one contract was awarded to Battelle and one to a second organization whose
contract was terminated 6 months later because of internal management issues at that
organization. In November 2010, on the basis of Noblis’s proposal from the original panel
evaluation, the Corps selected Noblis to replace this organization.




Page 26                                                  GAO-12-352 Army Corps of Engineers
project-specific independence and freedom from conflicts of interest. For
each project study undergoing a peer review, the Corps sends both
contractors a “scope of work” document, which describes the project
study and lists the required contractor qualifications. These qualifications
include independence and freedom from conflicts of interest related to the
specific project study being reviewed. In response, the contractors send
the Corps their proposals for conducting the peer review, which generally
include statements that they are independent and free from project-
specific conflicts. 30 Nevertheless, we identified a number of weaknesses
in the Corps’ approach for reviewing and corroborating this information,
including the following:

•    The Corps’ planning centers of expertise are expected to review the
     contractors’ overall proposals, but the Corps does not require the
     centers to ensure that contractors’ statements of independence within
     the proposals are reviewed and corroborated for each individual
     project. Although planning-center-of-expertise officials told us that
     they review the overall proposals, some of these officials also stated
     that they did not believe that the statements required review because
     Corps headquarters had already prescreened the contractors during
     the initial contractor screening and selection process. Furthermore,
     the Corps has not provided any guidance to the planning centers of
     expertise or other Corps offices that specifies how those officials
     should review the contractors’ project-specific statements at the
     proposal stage and ensure that they are accurate and that the
     contractors are in fact independent and free from conflicts of interest.
     Absent such guidance, the Corps cannot ensure that its contractors
     are independent and free from conflicts of interest at the project level.

•    The Corps neither conducts any internal conflicts-of-interest checks
     nor asks contractors for documentation about potential conflicts of
     interest so that it can determine whether a conflict exists; rather, the
     Corps allows the contractors to make that determination on their own.
     As a result, if the contractors do not provide this information to the
     Corps, the agency does not have a process for otherwise obtaining
     this information.




30
 Prior to the selection of Noblis as a contractor in 2010, the scope of work went solely to
Battelle. Battelle was still required to provide the Corps with a proposal describing how it
would conduct the peer review and responding to the contractor requirements.




Page 27                                                GAO-12-352 Army Corps of Engineers
                         Unlike the Corps’ review of contractor independence, the Corps’
                         contractors do solicit and review information on panel members’
                         independence and conflicts of interest at the project level. The contractors
                         gather information about prospective panel members using screening
                         questions developed from the scope of work for each peer review. These
                         questions cover issues described in the NAS policy on committee
                         composition and conflicts of interest, such as financial and employment
                         interests and public statements and positions. The peer review reports
                         from both Battelle and Noblis state that they follow both the OMB
                         guidance on peer review and the NAS policy when selecting panel
                         members. According to contractors and Corps officials, district and
                         planning-center-of-expertise officials review the contractors’ screening
                         questions, as well as the resumes of selected experts, and can provide
                         the contractors with additional information about potential conflicts of
                         interest, such as previous work a particular expert may have done for the
                         Corps. Corps officials told us that the contractors follow up on such
                         information where appropriate, but the contractors and a Corps official we
                         spoke to said that it is the contractors who ultimately select the panel
                         members and ensure their independence.


                         The Corps has adopted and incorporated most of the peer review
The Corps Has            recommendations it has received. Adoption of these recommendations
Adopted Most Peer        has resulted in some technical improvements to project study reports but
                         generally has not changed the Corps’ decisions in selecting preferred
Review                   project designs. According to some Corps officials we spoke with, this is
Recommendations,         the result of the review occurring too late in the process to effect a
Resulting in Technical   change in decision making.

Improvements but
Generally No Changes
in Project Decisions
The Corps Has Adopted    Of the 49 project studies that have undergone peer review, the Corps has
and Incorporated Most    provided a final written response for 17. The Corps has adopted 231 of
Peer Review              274 recommendations, partially adopted 31, and rejected 12 for these 17
                         peer reviews. Several Corps district officials told us that they make every
Recommendations,         effort not to reject peer review recommendations and that headquarters
Resulting in Technical   has directed them to adopt recommendations whenever possible. In fact,
Improvements             some district officials told us that they felt pressure from headquarters to
                         adopt peer review recommendations even when the recommendations



                         Page 28                                      GAO-12-352 Army Corps of Engineers
would not affect the study outcome and would be burdensome to
implement.

The Corps’ adoption of peer review recommendations has improved the
technical quality of its project study reports, according to Corps officials
and panel members we spoke with. Corps officials complimented the
quality and technical competence of panel members and stated that the
panels’ recommendations have been helpful in clarifying and
strengthening the arguments presented in the studies. Most of the
recommendations either requested that the Corps add to or clarify the
study report or stated that the study report did not sufficiently address
certain issues. The Corps addressed these issues in almost all instances
(193 of 201 recommendations) within its written responses to completed
peer review reports. 31

A smaller number of recommendations addressed the underlying
assumptions and inputs to the project studies’ economic, engineering,
and environmental analyses. 32 We identified four instances in which the
Corps revised portions of its analysis on the basis of these kinds of
recommendations. In none of these cases did the Corps indicate that the
revised analyses would change the study decisions. In one case,
according to Corps documents, the revised analysis served to strengthen
arguments in favor of its recommended plan. In response to a
recommendation concerning an environmental analysis from the peer
review of the Mid-Chesapeake Bay Islands Ecosystem Restoration
project study, the Corps conducted additional analyses to justify its
calculations of environmental benefits. The Corps reported that the
additional analyses led to the determination that the selected plan was
appropriate but that by considering the ecosystem impacts of the project
in a more detailed fashion, justification of the recommended plan was
strengthened.




31
  Forty-nine studies underwent peer review, but 3 of these studies resulted in 2 peer
review reports, so the total number of reports was 52. For the 52 completed peer review
reports, panel members made a total of 910 recommendations. Most of the
recommendations (672 of 910) either requested that the Corps add to or clarify the study
report or stated that the study report did not sufficiently address certain issues.
32
  Specifically, 34 out of the total 910 peer review recommendations indicated a problem
with the economic analysis, 24 indicated a problem with the engineering analysis, and 19
indicated a problem with the environmental analysis.




Page 29                                              GAO-12-352 Army Corps of Engineers
                              Nevertheless, despite these technical improvements, some Corps officials
                              have questioned the benefit of peer review, given the significant amount
                              of time that district staff have to spend managing the process and
                              responding to recommendations. The process for responding to
                              recommendations begins with district officials drafting a written response,
                              which they provide to the panel. The Corps’ response to the peer review
                              recommendations includes a detailed description of the steps that the
                              Corps has taken or will take to incorporate the recommendations into the
                              project study. The contractor then convenes a teleconference at which
                              district officials discuss the draft response with panel members. After this
                              discussion, the panel members provide written feedback—“backcheck
                              responses”—to the Corps stating whether they agree with the district’s
                              response. 33 The district then finalizes its response to the
                              recommendations and forwards the response to its division office. After its
                              review, the division forwards the response to headquarters, where the
                              response is finalized. The final written response is generally published at
                              the same time as the final decision document for the project study. The
                              time between completion of the peer review report and public availability
                              of Corps’ written response therefore varies greatly depending on the
                              individual project. In one case it was 3 months, while in other cases peer
                              review reports have been completed for more than 3 years without a final
                              response from the Corps having been made public.


For Most Studies, Peer        Corps officials we spoke with told us that peer review recommendations
Review Occurs Too Late in     have generally had no impact on the Corps’ decision making process.
the Study Process to Affect   These Corps officials were not aware of any project studies for which the
                              study outcome changed as a result of peer review. Corps headquarters
Decision Making               officials told us that one reason for the lack of impact of peer review on
                              decision-making is because the Corps’ internal review process is
                              identifying the same issues as peer review. Another reason cited by
                              Corps officials for the lack of impact on decision making is the fact that
                              peer review is occurring at the end of the study process. Peer review
                              generally occurs concurrently with the public comment period for the draft
                              study report, which comes after the preferred design has been selected.
                              As a result, some recommendations about alternatives may not have


                              33
                                 The backcheck responses are not part of the panel’s final peer review report which
                              contains only the panel’s recommendations. The backcheck responses are given after the
                              final peer review report is submitted to ensure that the panel’s opinion and objectivity are
                              not influenced by the Corps.




                              Page 30                                               GAO-12-352 Army Corps of Engineers
been implemented because the decision on the preferred design had
already been made. Selecting a different preferred design at that stage
would require the Corps to revisit an already completed selection analysis
and decision. For example, in the peer review report on the Cedar River-
Cedar Rapids Iowa Flood Risk Management project study, the review
panel recommended that the Corps further investigate one of the non-
selected design alternatives, because panel members felt that the
alternative might achieve project objectives better than the preferred
design. The Corps, however, had already selected its design and decided
to proceed. The Corps did not adopt this recommendation, stating that it
believed its analysis of alternatives was sound and that there was no
reasonable expectation that a more detailed analysis of the alternative
would result in finding that it had greater net economic benefits than the
preferred design.

In contrast, when the Corps has conducted a peer review earlier in the
process, opportunities have arisen for positive impacts on a study
decision. For example, the American River Common Features project
study peer review was conducted early in the study process. According to
Corps division and planning-center-of-expertise officials, they conducted
peer review early to obtain external input on defining the problem and to
inform decision-making due to the complexity of the project. As a result,
the peer review began before the alternative formulation briefing, when
the without-project conditions were being identified. By employing this
approach, the Corps received feedback from the review panel before
selecting the preferred design. The panel’s recommendations included
three suggested changes to the Corps’ analyses and model calibrations,
which the Corps had time to incorporate before conducting the alternative
analysis and selection. According to the contractor that managed the peer
review, the panel members involved in the American River Common
Features peer review also found the timing of the review to be beneficial
and suggested that the Corps conduct peer review earlier for other project
studies.

The timing of peer review was also addressed in the 2002 NAS study on
peer review. NAS recommended that the Corps initiate peer reviews early
enough in the study process so that the review results could be
meaningfully incorporated into the study or project design and stated that
conducting peer review before selecting a recommended plan is essential
if the Corps is to benefit from the review. Corps officials nevertheless told
us that they have generally chosen to conduct peer reviews later in the
process to minimize effects on project study schedules. Corps
headquarters officials noted that, for many studies, peer review occurred


Page 31                                      GAO-12-352 Army Corps of Engineers
              late in the process because the studies were under way at the time the
              Corps began requiring peer review. These officials also noted that it
              would be challenging to assemble a peer review panel to conduct a
              review early in the study process and retain the same panel to complete
              this review at the end of the study. Furthermore, Corps headquarters
              officials noted that a further challenge is implementing section 2033
              WRDA requirements along with section 2034. Section 2033 generally
              requires the Corps to complete feasibility studies within 2 years.
              According to Corps officials, there is tension between these requirements
              and it may be challenging to include peer review throughout the study
              process without altering project study schedules.


              Section 2034 established a trial to look at the cost and impact of
Conclusions   conducting peer review for controversial and costly projects over a 7-year
              period. After the trial period, based on information provided by the Corps,
              Congress could reconsider whether to retain or revise section 2034 or
              allow it to lapse. Because the Corps generally does not specify the
              authority under which peer review was conducted, however, it has not
              provided Congress with the information needed to evaluate the merits of
              the section 2034 requirements. In addition, the Corps’ implementation of
              peer review has not focused on the larger, more complex, and
              controversial projects that were contemplated when section 2034 was
              enacted and as recommended by NAS a decade ago. As a result, project
              studies are being selected to undergo peer review that may not be
              warranted and may thereby be increasing project costs and schedules
              needlessly. Further, essential to the integrity of the peer review process is
              the assurance that the Corps has effective processes not only to ensure
              overall contractor independence and freedom from conflicts of interest but
              also to ensure project-level independence and freedom from conflicts of
              interest. The Corps’ current process, however, has a number of
              weaknesses with respect to ensuring no conflicts of interest exist at the
              project level. Finally, with peer review generally occurring late in the
              Corps’ project study process, peer review serves more to strengthen the
              Corps’ presentation of its decisions than to influence its decision making.
              This effect runs counter to what NAS recommended in 2002, that
              realizing the benefits of peer review requires the results to be used as
              inputs in the decision-making process. By choosing to apply peer review
              late in the project study process, the Corps has effectively chosen to not
              use the results of peer review to enhance its decision-making process
              and ensure selection of the most effective project alternatives.




              Page 32                                      GAO-12-352 Army Corps of Engineers
                      We recommend that the Secretary of Defense direct the Chief of
Recommendations for   Engineers and Commanding General of the U.S. Army Corps of
Executive Action      Engineers to take the following three actions:

                      To facilitate congressional evaluation of the 7-year trial period outlined in
                      section 2034, the Corps should:

                      •   Identify for each past and future peer review the specific statutory
                          authority under which the peer review was conducted and the criteria
                          triggering peer review under the Corps’ civil works review policy.
                      To better reflect section 2034 and provide more effective stewardship of
                      public resources and ensure efficient and effective operations, the Corps
                      should:

                      •   Revise the criteria in the Corps’ process for conducting peer review to
                          focus on larger, more complex, and controversial projects; to
                          encourage peer review to occur earlier in the study process; and to
                          include exclusions to peer review that align with section 2034.

                      •   Develop a documented process to ensure that contractors are
                          independent and free from conflicts of interest on a project-specific
                          basis.

                      We provided a draft of this report to the Department of Defense for review
Agency Comments       and comment. In its written comments, reprinted in appendix III, the
and Our Evaluation    department generally concurred with our recommendations. Specifically,
                      in response to our first recommendation, the department agreed that the
                      Corps should, and stated that it will, identify for each past and future peer
                      review the specific statutory authority under which the peer review was
                      conducted and the criteria triggering peer review under the Corps’ civil
                      works review policy. In response to our second recommendation, the
                      department partially concurred, stating that it agreed that peer review
                      should be focused on studies that will significantly benefit from peer
                      review and that initiating reviews early is advantageous. Nevertheless,
                      the department noted that early involvement must be balanced with
                      having sufficient data and analysis available for review and also
                      highlighted work under way at the agency to overhaul its planning
                      processes, which includes efforts to better align product reviews for
                      greater effectiveness. In response to our third recommendation, the
                      department agreed that the Corps should develop a documented process
                      to ensure that contractors are independent and free from conflicts of
                      interest on a project-specific basis.



                      Page 33                                       GAO-12-352 Army Corps of Engineers
Although the department generally concurred with our recommendations,
it disagreed with our report’s finding that the Corps’ process does not use
the flexibility provided in section 2034, and it disagreed that some studies
have undergone review that did not warrant it. The department stated that
the Corps has carefully deliberated in support of the agency decision to
conduct peer review on the three studies noted in our report and also
stated that the Corps stands by all of its decisions to date to grant or deny
exclusions from peer review. Nevertheless, the department stated that as
part of the Corps’ ongoing review of the civil works review policy, it will
assess the effectiveness of its criteria and how the criteria are applied to
determine which studies should be considered for exclusion. In addition,
the department expressed concern about the level of weight given in the
report to anecdotal remarks from field-level officials, who in the
department’s opinion may not have had the benefit of the corporate vision
supporting the Army Civil Works Program. We disagree with the
department’s characterization of our methodology. As clearly described in
the scope and methodology section of this report, we interviewed officials
who had a corporate-level perspective, as well as those who had a
project-level perspective. Specifically, to obtain a corporate-level view, we
interviewed senior level officials from Corps headquarters, the Institute for
Water Resources, and the planning centers of expertise involved in
managing the peer reviews. In addition, to get a project-level perspective
and to assess the impact of peer review on division and district offices,
we interviewed officials in all of the Corps’ eight divisions, and from 10
geographically dispersed Corps districts that had conducted studies that
underwent peer review. We also interviewed the three contractors and
selected peer review panel members and local sponsors of Corps civil
works projects. We believe that the report provides a balanced
perspective from both the headquarters and field levels.


As agreed with your office, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from the
report date. At that time, we will send copies of this report to the
appropriate congressional committees, the Secretary of Defense, the
Chief of Engineers and Commanding General of the U.S. Army Corps of
Engineers, and other interested parties. This report will be available at no
charge on the GAO website at http://www.gao.gov.




Page 34                                      GAO-12-352 Army Corps of Engineers
If you or your staff members have any questions about this report, please
contact me at (202) 512-3841 or mittala@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made key contributions to this
report are listed in appendix IV.

Sincerely yours,




Anu K. Mittal
Director, Natural Resources and Environment




Page 35                                     GAO-12-352 Army Corps of Engineers
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              Our objectives for this work were to examine (1) the number of Corps
              project studies that have undergone independent peer review in response
              to section 2034 of the Water Resources and Development Act (WRDA) of
              2007, (2) the cost of these peer reviews, (3) the extent to which the U.S.
              Army Corps of Engineers’ (Corps) process for determining if a project
              study is subject to peer review is consistent with section 2034, (4) the
              process the Corps uses to ensure that the contractors it hires and the
              experts the contractors select to review project studies are independent
              and free from conflicts of interest, and (5) the extent to which
              recommendations from peer reviews have been incorporated into project
              studies. We focused on peer reviews for which reports had been
              completed since WRDA 2007 was enacted.

              To address all of these objectives, we reviewed relevant legal
              requirements, policy guidance, review plans, and peer review reports for
              project studies that were subject to a peer review and for which a peer
              review report had been completed since WRDA 2007 was enacted. In
              addition, we selected a nongeneralizable sample of six peer reviews to
              examine in greater depth to better understand the costs associated with
              conducting these reviews, as well as the overall impact of the process on
              the timeline of the project study and the study outcome. We chose these
              reviews as illustrative examples and selected one from each of the Corps’
              planning centers of expertise and at least one for each of the three
              contractors the Corps has used to manage peer reviews since enactment
              of WRDA 2007. Although the information derived from analysis of these
              case studies cannot be generalized, these examples provide valuable
              insights into the peer review process. We conducted semistructured
              interviews with officials from Corps headquarters, the planning centers of
              expertise involved in managing the peer reviews, all of the Corps’ eight
              divisions, and from 10 geographically dispersed Corps districts that had
              conducted studies that underwent peer review. We also conducted
              semistructured interviews with the three contractors, as well as selected
              peer review panel members and local sponsors of Corps civil works
              projects.

              To determine the number of studies that have undergone peer review in
              response to section 2034 of WRDA, we reviewed all completed peer
              review reports, plus Corps reports and information on completed peer
              reviews. We reviewed information on completed peer reviews obtained
              from headquarters, the planning centers of expertise, divisions, and
              selected districts. We also reviewed information on completed peer
              reviews obtained from the contractors that established the peer review



              Page 36                                     GAO-12-352 Army Corps of Engineers
Appendix I: Objectives, Scope, and
Methodology




panels and the entities the Corps used to administer these contracts: the
Institute for Water Resources and the Army Research Office.

To determine the cost of these reviews, we reviewed contract award
documents and information on contract costs from the contractors.
Generally, we relied on the contract award amounts reported in the
contracts to determine the cost of the contracts awarded for establishing
review panels. For four contract awards, the contract work included
establishing a peer review panel and additional work. For these awards,
we therefore relied on information provided by the contractor on the
portion of the contract cost that was for the peer review. For the contract
award for peer review of a local sponsor-led study, we relied on
information from the local sponsor and the contractor on the cost of the
award. In addition, for the six case study peer reviews, we analyzed
information on costs associated with managing the review process,
including cost data and estimates provided by districts with regard to
district and other staff time involved in peer review. In cases where we
reported cost data including staff time associated with completing peer
review, we asked knowledgeable officials about the data system and the
quality of the data and determined that they were sufficiently reliable for
our purposes. In cases where we reported estimates of these costs, we
asked officials about how these estimates were developed and
determined that they were sufficiently reliable for our purposes.

To determine the extent to which the Corps’ process for determining if a
study is subject to peer review is consistent with section 2034, we
analyzed the legal requirements and relevant policy guidance for
determining when to conduct peer review. We also reviewed
documentation on decisions to conduct peer review included in review
plans and documents requesting exclusion from peer review. In addition,
we reviewed information on the characteristics of studies that underwent
peer review, including date initiated, whether an environmental impact
statement was included, and the type of study. We identified this
information in review plans, study drafts, signed Chief’s reports, and other
Corps study-related documents; Corps officials from relevant districts or
divisions reviewed this information.

To determine the process the Corps uses to ensure that the contractors it
hires and the experts the contractors select are independent and free
from conflicts of interest, we reviewed information on contractor selection
obtained from Corps headquarters and the Institute for Water Resources.
We also reviewed documentation from the contractors that outlined
contractor and reviewer qualifications, as well as the National Academy of


Page 37                                      GAO-12-352 Army Corps of Engineers
Appendix I: Objectives, Scope, and
Methodology




Sciences Policy on Committee Composition and Balance and Conflicts of
Interest.

To determine the extent to which peer review recommendations are
incorporated into project studies, we reviewed information obtained from
headquarters, the planning centers of expertise, divisions, and selected
districts on how the Corps responds to peer review recommendations.
We also reviewed all peer review recommendations contained in
completed peer review reports, as well as all responses to peer review
recommendations contained in the Corps’ published responses to the
completed peer review reports.

We conducted this performance audit from April 2011 to March 2012 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.




Page 38                                    GAO-12-352 Army Corps of Engineers
Appendix II: Characteristics of Studies That
                                                Appendix II: Characteristics of Studies That
                                                Underwent Peer Review



Underwent Peer Review

                                                Forty-six of the 49 peer reviews completed in table 2 below were
                                                conducted by Battelle Memorial Institute; Noblis completed the Green Bay
                                                Dredged Material Management Plan and the Wood River Levee System
                                                General Reevaluation Report peer reviews, and the National Academy of
                                                Sciences conducted the Louisiana Coastal Protection and Restoration
                                                peer review. As table 2 shows, the studies that underwent peer review
                                                came under the areas of ecosystem restoration (19 of 49), flood risk
                                                management (15 of 49), deep draft navigation (7 of 49), coastal storm
                                                damage reduction (5 of 49), inland navigation (2 of 49), and water
                                                management and reallocation (1 of 49). According to our analysis of
                                                Corps documents, 32 of the 49 studies included an environmental impact
                                                statement (EIS), 19 of 49 were initiated after November 2005, and 42 of
                                                49 had an estimated total project cost greater than $45 million.

Table 2: Characteristics of Studies That Underwent Peer Review

Dollars in thousands
                                                          Corps            Date of      Feasibility Initiated   Estimated            Cost of
                                                          planning         peer             or        after        cost                peer
Study name                                                center of        review      reevaluation   Nov.      exceeding            review
and type               Corps district Corps division      expertise        report(s)      studya      2005      $45 million   EIS   contract
Alton to Gale       St. Louis         Mississippi         Flood risk       Aug. 18,                                 √                  $145
Organized Levee                       Valley              management       2010
Districts
(Continuing,
Deficiency
Corrections) Letter
Report
American River     Sacramento         South               Flood risk       June 10,            √        √           √         √         458
Draft Natomas                         Pacific             management       2009 &
Post-Authorization                                                         Sept. 16,
Change Report                                                              2010
             b
and Draft EIS
Barataria Basin        New Orleans    Mississippi         Ecosystem        Oct. 3,                      √           √         √         141
Barrier Shoreline                     Valley              restoration      2011
Restoration Draft
Construction
Report and Draft
EIS
Biscayne Bay           Jacksonville   South Atlantic      Ecosystem        Dec. 1,             √                    √         √         164
Coastal Wetlands                                          restoration      2009
Project
Implementation
Report




                                                Page 39                                                GAO-12-352 Army Corps of Engineers
                                             Appendix II: Characteristics of Studies That
                                             Underwent Peer Review




Dollars in thousands
                                                       Corps            Date of      Feasibility Initiated   Estimated            Cost of
                                                       planning         peer             or        after        cost                peer
Study name                                             center of        review      reevaluation   Nov.      exceeding            review
and type               Corps district Corps division   expertise        report(s)      studya      2005      $45 million   EIS   contract
Boston Harbor     New England        North Atlantic    Deep draft       June 3,             √                    √         √         159
Navigation                                             navigation       2008
Improvement Draft
Feasibility Study
and Draft
Supplemental EIS
Brevard County     Jacksonville      South Atlantic    Coastal storm    Dec. 9,             √                    √         √         173
Mid-Reach                                              damage           2009
Shoreline                                              reduction
Protection Project
Draft Integrated
General Re-
evaluation Report
and Supplemental
EIS
Calcasieu River  New Orleans         Mississippi       Deep draft       Aug. 29,                                 √         √         135
and Pass Dredged                     Valley            navigation       2008
Material
Management Plan
and Supplemental
EIS
Cedar Rapids           Rock Island   Mississippi       Flood risk       Oct. 28,            √        √           √                   122
Flood Risk                           Valley            management       2010
Management
Feasibility Study
with Integrated
Environmental
Assessment
Chacon Creek,          Fort Worth    Southwestern      Flood Risk       Nov. 17,            √                                        137
Rio Grande Draft                                       management       2010
Feasibility Report
and Integrated
Environmental
Assessment
Chatfield Storage Omaha              Northwestern      Water            Oct. 25,                                 √         √         134
Reallocation Study                                     management/      2011
and EIS                                                reallocation
Clear Creek Risk       Galveston     Southwestern      Flood risk       May, 12,            √                    √         √         290
Management                                             management       2009
General
Reevaluation
Report and
Preliminary Draft
   b,c
EIS




                                             Page 40                                                GAO-12-352 Army Corps of Engineers
                                              Appendix II: Characteristics of Studies That
                                              Underwent Peer Review




Dollars in thousands
                                                        Corps            Date of      Feasibility Initiated   Estimated            Cost of
                                                        planning         peer             or        after        cost                peer
Study name                                              center of        review      reevaluation   Nov.      exceeding            review
and type               Corps district Corps division    expertise        report(s)      studya      2005      $45 million   EIS   contract
Columbia River at      Portland       Northwestern      Deep draft       Mar. 9,                                  √                   200
the Mouth Major                                         navigation       2011
Rehabilitation
Evaluation Report
East Branch Dam, Pittsburgh           Great Lakes    Flood risk          July 2,                      √           √                   105
Clarion River, Elk                    and Ohio River management          2010
County, Dam
Safety
Modification
Report
East St. Louis    St. Louis           Mississippi       Flood risk       Aug. 3,             √        √           √                   159
Flood Protection                      Valley            management       2010
Limited
Reevaluation
Report and
Environmental
Assessment on
Design Deficiency
Corrections
                          d
Environmental          N/A            Great Lakes    Ecosystem           Dec. 7,                      √                               151
DNA (eDNA)                            and Ohio River restoration         2010
Science and
Methodology
Fargo-Moorhead       St. Paul         Mississippi       Flood risk       May 17,             √        √           √         √         282
Flood Risk                            Valley            management       2010 &
Management                                                               July 7,
                 b,e
Feasibility Study                                                        2011
Freeport Harbor        Galveston      Southwestern      Deep draft       Aug. 20,            √                    √         √         214
Draft Feasibility                                       navigation       2008
                 e
Report and EIS
Green Bay        Detroit              Great Lakes    Inland              June 27,                     √           √                   101
Dredged Material                      and Ohio River navigation          2011
Management Plan
Jamaica Bay,           New York       North Atlantic    Ecosystem        Dec. 8,             √                    √                   164
Marine Park, and                                        restoration      2010
Plumb Beach
Draft Interim
Feasibility Report
Kissimmee River        Jacksonville   South Atlantic    Ecosystem        Oct. 13,            √        √           √                    98
Restoration Post-                                       restoration      2010
Authorization
Change Limited
Reevaluation
Report




                                              Page 41                                                GAO-12-352 Army Corps of Engineers
                                             Appendix II: Characteristics of Studies That
                                             Underwent Peer Review




Dollars in thousands
                                                       Corps            Date of      Feasibility Initiated   Estimated            Cost of
                                                       planning         peer             or        after        cost                peer
Study name                                             center of        review      reevaluation   Nov.      exceeding            review
and type               Corps district Corps division   expertise        report(s)      studya      2005      $45 million   EIS   contract
L-31N Seepage    Jacksonville        South Atlantic    Ecosystem        Mar. 10,                                                     102
Management Pilot                                       restoration      2009
Project Draft
Integrated Pilot
Project Design
Report and
Environmental
Assessment
                                                                                                                                           f
Louisiana Coastal New Orleans        Mississippi       Ecosystem        June 23,            √        √                     √         677
Area Amite River                     Valley            restoration      2010
Diversion Canal
Modification
Integrated
Feasibility Study
and Supplemental
EIS
                                                                                                                                           f
Louisiana Coastal      New Orleans   Mississippi       Ecosystem        June 25,            √        √           √         √         677
Area Convey                          Valley            restoration      2010
Atchafalaya River
Water to Northern
Terrebonne
Marshes,
Lafourche
Terrebonne, St.
Mary Parish,
Integrated
Feasibility Study
and EIS
                                                                                                                                           f
Louisiana Coastal New Orleans        Mississippi       Ecosystem        June 23,            √        √           √         √         677
Area Medium                          Valley            restoration      2010
Diversion at White
Ditch,
Plaquemines
Parish, Integrated
Feasibility Study
and Supplemental
EIS
                                                                                                                                           f
Louisiana Coastal      New Orleans   Mississippi       Ecosystem        June 22,            √        √           √         √         677
Area Small                           Valley            restoration      2010
Diversion at
Convent/Blind
River, St. James
Parish, Integrated
Feasibility Study
and EIS




                                             Page 42                                                GAO-12-352 Army Corps of Engineers
                                             Appendix II: Characteristics of Studies That
                                             Underwent Peer Review




Dollars in thousands
                                                       Corps            Date of      Feasibility Initiated   Estimated            Cost of
                                                       planning         peer             or        after        cost                peer
Study name                                             center of        review      reevaluation   Nov.      exceeding            review
and type               Corps district Corps division   expertise        report(s)      studya      2005      $45 million   EIS   contract
                                                                                                                                           f
Louisiana Coastal      New Orleans   Mississippi       Ecosystem        June 25,            √                    √         √         677
Area Terrebonne                      Valley            restoration      2010
Basin Barrier
Shoreline
Restoration,
Terrebonne
Parish, Integrated
Feasibility Study
and EIS
Louisiana Coastal      New Orleans   Mississippi       Coastal storm    2009                         √          N/A        √         595
Protection and                       Valley            damage
Restoration                                            reduction
Program Draft
Final Technical
       g
Report
Marlinton Detailed Huntington        Great Lakes    Flood risk          Nov. 16,            √                    √         √         191
Project Report and                   and Ohio River management          2010
EIS
Melvin Price Wood St. Louis          Mississippi       Flood risk       Apr. 6,             √        √                               130
River                                Valley            management       2011
Underseepage
Limited
Reevaluation
Report and
Environmental
Assessment on
Design Deficiency
Corrections
Middle             Baltimore         North Atlantic    Ecosystem        Jan. 23,            √                    √         √         121
Chesapeake Bay                                         restoration      2008
Islands Ecosystem
Restoration Final
Integrated
Feasibility Report
and EIS and
supporting
documentation
Mississippi            Mobile        South Atlantic    Coastal storm    Nov. 7,                      √           √         √         161
Coastal                                                damage           2008
Improvements                                           reduction
Program
Comprehensive
Plan




                                             Page 43                                                GAO-12-352 Army Corps of Engineers
                                              Appendix II: Characteristics of Studies That
                                              Underwent Peer Review




Dollars in thousands
                                                        Corps            Date of      Feasibility Initiated   Estimated            Cost of
                                                        planning         peer             or        after        cost                peer
Study name                                              center of        review      reevaluation   Nov.      exceeding            review
and type               Corps district Corps division    expertise        report(s)      studya      2005      $45 million   EIS   contract
Mississippi River      New Orleans    Mississippi       Ecosystem        June 3,             √        √           √         √         373
Gulf Outlet                           Valley            restoration      2011
Ecosystem
Restoration Plan
Feasibility Study
and EIS
Mohawk Dam             Huntington     Great Lakes    Flood risk          Apr. 22,                     √           √                   240
Major                                 and Ohio River management          2011
Rehabilitation
Report
Navigation and   Rock Island          Mississippi       Ecosystem        Dec. 17,            √                    √                   141
Ecosystem                             Valley            restoration      2009
Sustainability
Program, Project
P2, Lock and Dam
22 Fish Passage
Improvement
Project
Implementation
Report
Olmsted Post-          Louisville     Great Lakes    Inland              Nov. 15,            √                    √         √         201
Authorization                         and Ohio River navigation          2010 &
              b,e
Change Report                                                            Nov. 4,
                                                                         2011
Port Everglades        Jacksonville   South Atlantic    Ecosystem        Aug. 17,            √                    √         √          76
Harbor Science                                          restoration      2011
Reports for the
Feasibility Study
and EIS
Programmatic EIS Omaha                Northwestern      Ecosystem        Mar. 17,                                 √         √         120
for the Mechanical                                      restoration      2010
Creation and
Maintenance of
Emergent
Sandbar Habitat
on the Upper
Missouri River
Sabine Neches      Galveston          Southwestern      Deep draft       Dec. 13,            √                    √         √         313
Waterway                                                navigation       2007
Channel
Improvement Plan
Draft Feasibility
Report, Draft EIS,
and Supporting
                 e
Documentation




                                              Page 44                                                GAO-12-352 Army Corps of Engineers
                                              Appendix II: Characteristics of Studies That
                                              Underwent Peer Review




Dollars in thousands
                                                        Corps            Date of      Feasibility Initiated   Estimated            Cost of
                                                        planning         peer             or        after        cost                peer
Study name                                              center of        review      reevaluation   Nov.      exceeding            review
and type               Corps district Corps division    expertise        report(s)      studya      2005      $45 million   EIS   contract
Sacramento River San                  South Pacific     Deep draft       Sep. 30,            √                    √         √         164
Deep Water Ship Francisco                               navigation       2011
Channel Limited
Reevaluation
Study and
Supplemental EIS
San Clemente           Los Angeles    South Pacific     Coastal storm    July 23,            √                              √         145
Storm Damage                                            damage           2010
and Shoreline                                           reduction
Protection
Feasibility Study
Savannah Harbor Savannah              South Atlantic    Deep draft       Feb. 11,            √                    √         √         484
Expansion Project                                       navigation       2011
General
Reevaluation
       e
Report
St. Johns Bayou        Memphis        Mississippi       Flood risk       Oct. 23,                                 √         √         453
and New Madrid                        Valley            management       2009
Floodway
Consolidated
NEPA Document
and Work Plan
         h
(Phase I)
Success Dam      Sacramento           South Pacific     Flood risk       Mar. 24,                                 √         √         285
Remediation Dam                                         management       2011
Safety Assurance
Program Letter
Report
Surf City and          Wilmington     South Atlantic    Coastal storm    Apr. 16,            √                    √         √         187
North Topsail                                           damage           2010
Beach Draft                                             reduction
Integrated
Feasibility Report
and EIS
Tamiami Trail          Jacksonville   South Atlantic    Ecosystem        June 2,             √                    √                   142
Limited                                                 restoration      2008
Reevaluation
Report
Tres Rios del       Los Angeles       South Pacific     Ecosystem        Sep. 26,            √                    √         √         159
Norte, Pima                                             restoration      2011
County, Feasibility
Study




                                              Page 45                                                GAO-12-352 Army Corps of Engineers
                                              Appendix II: Characteristics of Studies That
                                              Underwent Peer Review




Dollars in thousands
                                                           Corps                 Date of      Feasibility Initiated     Estimated                Cost of
                                                           planning              peer             or        after          cost                    peer
Study name                                                 center of             review      reevaluation   Nov.        exceeding                review
and type               Corps district Corps division       expertise             report(s)      studya      2005        $45 million     EIS     contract
Western C-111          Jacksonville   South Atlantic       Ecosystem             Oct. 30,         √                          √            √          184
Spreader Canal                                             restoration           2009
Project
Implementation
Report
White Oak Bayou        Galveston -   Southwestern          Flood risk            Jan. 11,         √                          √                       124
Flood Damage           Local sponsor                       management            2011
Reduction Plan         led
General
Reevaluation
Report
Wood River Levee St. Louis            Mississippi          Flood risk            Aug. 24,         √            √             √                         79
System Limited                        Valley               management            2011
Reevaluation
Report for Design
Deficiency
Corrections
                                              Source: GAO analysis of Corps documents.

                                              Note: In addition to the 49 peer reviews included on this list, the Louisiana Coastal Area Beneficial
                                              Use of Dredged Material Program Preliminary Study Report— which addressed the methodology to
                                              select and prioritize projects under the program—underwent peer review in July 2008. Section
                                              7006(d) of WRDA 2007 authorized this $100 million program for the beneficial use of dredged
                                              material. The review was overseen by the U.S. Geological Survey and conducted by the Louisiana
                                              Coastal Area Science Board, and about $12,000 in Corps funds were used to support its completion.
                                              However, because the completion of this review was overseen by the U.S. Geological Survey, and
                                              did not involve the Corps’ typical contract process or costs, it was not included in our list or analysis of
                                              completed peer reviews.
                                              a
                                               Postauthorization change reports are generally considered reevaluation reports, and project
                                              implementation reports are generally considered feasibility studies.
                                              b
                                                  Peer review included multiple reports.
                                              c
                                               As part of this contract, a follow up report was completed in July 2010 to determine whether
                                              recommendations from the peer review report were addressed. This follow up report was not included
                                              in our analysis because it was not provided until our review was complete.
                                              d
                                               The eDNA science and methodology work was done by the University of Notre Dame, and the study
                                              is part of the Great Lakes Regional Initiative.
                                              e
                                               Peer review was completed under multiple contracts, and the contract cost provided is the sum of
                                              the cost of those contracts.
                                              f
                                              Multiple peer reviews were conducted under one contract for five of the Louisiana Coastal Area
                                              studies.
                                              g
                                               This review does not include a preliminary report done by Battelle before WRDA 2007. Also,
                                              according to officials, the estimated cost of the project for this study is not applicable (N/A) because
                                              the study made no recommendations for construction.




                                              Page 46                                                         GAO-12-352 Army Corps of Engineers
Appendix II: Characteristics of Studies That
Underwent Peer Review




h
 As part of this contract, a report on Phase I and Phase II of the project were completed. The Phase II
report was completed in April 2010 with an addendum in November 2010 but was not included in our
analysis because it was not provided until our review was complete.




Page 47                                                      GAO-12-352 Army Corps of Engineers
Appendix III: Comments from the
              Appendix III: Comments from the Department
              of Defense



Department of Defense




              Page 48                                      GAO-12-352 Army Corps of Engineers
Appendix III: Comments from the Department
of Defense




Page 49                                      GAO-12-352 Army Corps of Engineers
Appendix III: Comments from the Department
of Defense




Page 50                                      GAO-12-352 Army Corps of Engineers
Appendix III: Comments from the Department
of Defense




Page 51                                      GAO-12-352 Army Corps of Engineers
Appendix IV: GAO Contact and Staff
                  Appendix IV: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Anu K. Mittal, (202) 512-3841 or mittala@gao.gov
GAO Contact
                  In addition to the individual listed above, Vondalee R. Hunt, Assistant
Staff             Director; Darnita Akers; Elizabeth Beardsley; and Janice Ceperich made
Acknowledgments   significant contributions to this report. Ellen Wo Chu, Cindy Gilbert,
                  Richard P. Johnson, Ben Shouse, and Kiki Theodoropoulos also made
                  key contributions.




(361279)
                  Page 52                                    GAO-12-352 Army Corps of Engineers
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.