oversight

Highway Infrastructure: FHWA Has Acted to Disclose the Limitations of Its Environmental Review Analysis

Published by the Government Accountability Office on 2003-01-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

United States General Accounting Office
Washington, DC 20548




          January 16, 2003

          The Honorable Don Young
          Chairman, Committee on Transportation
           and Infrastructure
          House of Representatives

          Subject: Highway Infrastructure: FHWA Has Acted to Disclose the Limitations of
                    Its Environmental Review Analysis

          Dear Mr. Chairman:

          Environmental reviews of complex federally funded highway construction projects
          may take years.1 The Congress has an interest in identifying and, if necessary,
          addressing the reasons to expedite highway projects. To better understand these
          reasons, the Federal Highway Administration (FHWA) surveyed its 55 division offices
          to determine why the environmental review of certain highway projects took more
          than 5 years. In September 2000, FHWA made the results of its analysis of 89 projects
          it identified available to the Congress, and in December 2000, it posted the results on
          its Web site. As the reauthorization of the Transportation Equity Act for the 21st
          Century approaches, transportation stakeholders have used the results of this
          analysis in discussions about whether legislative reforms are needed for
          environmental reviews of federally funded highway projects.

          You requested that we determine (1) the reasonableness of FHWA’s survey
          methodology and offer suggestions for improvement, if appropriate, and (2) the
          feasibility of resurveying the same projects to gain an additional understanding about
          why environmental reviews took more than 5 years. To carry out this work, we
          interviewed FHWA officials about how they selected projects, surveyed their division
          offices, and analyzed the responses they obtained. In addition, we applied the
          lessons learned from FHWA’s original approach to assess the feasibility of
          resurveying the same projects.




          1
           Under the National Environmental Policy Act of 1969, the consequences of proposed transportation
          projects and alternative choices on the environment and on historic properties, if any, must be
          identified and assessed. Federally funded highway project construction cannot begin until
          environmental and historic preservation issues are addressed, if needed.


                                              GAO-03-338R Federal Highway Environmental Analysis
Results in Brief

Aspects of FHWA’s methodology, such as relying on its division offices for
information rather than on a larger set of stakeholders, were reasonable, given the
agency’s desire for a quick exploration of the subject. However, several other aspects
of FHWA’s methodology lead us to question the usefulness of the results. FHWA’s
reliance on narrative responses (rather than multiple choice questions, for example)
produced results that are not particularly useful because the answers are general and
typically do not delineate the underlying reasons why environmental reviews took
more than 5 years. In addition, when multiple reasons were cited, many responses
did not specify which reason or reasons were primary. In these instances, the FHWA
staff member chose only one reason as the main reason, basing her selection, in large
part, on which reason was listed first. FHWA’s results would have been more useful
if it had asked respondents to provide enough information so that underlying reasons
could be understood and to indicate relative importance of reasons when more than
one reason was cited. Finally, FHWA did not provide enough detail about how it
conducted its work to allow potential users to assess its reliability.

Resurveying the 89 projects is feasible and could provide more useful and reliable
information on why environmental reviews took more than 5 years. However, several
issues would have to be addressed before undertaking any such effort. These issues
include (1) verifying that the 14 FHWA division offices (25 percent) that did not
respond to the agency’s original survey did not have projects in environmental review
for 5 years or more and (2) determining whether a newer data set (e.g., projects with
environmental reviews lasting more than 5 years at the time of the resurvey) might
provide more current and reliable information. The costs and benefits of redoing the
survey would also have to be weighed against the expected results of several current
FHWA initiatives that are intended to shed light on how environmental reviews of
federally funded highway projects are completed. For example, FHWA has hired a
contractor to review about 250 projects from the 1990s to identify variables that
correlate with time these projects spent in environmental reviews. FHWA is also
conducting a Gallup poll that surveys stakeholders’ views on the environmental
review process. The results for FHWA’s efforts are expected by spring 2003.

The Department of Transportation offered no specific comments on a draft of this
report. However, in response to our findings, FHWA revised its Web site in a manner
that adequately discloses the issues discussed in this report.

Background

FHWA provides financial assistance to states to build and improve highways and
roads and provides transportation engineering services (such as planning and design)
for highway and bridges that provide access to federally owned lands. FHWA relies
on its 52 division offices nationwide and 3 Federal Lands Highway Division offices to
carry out this mission. For fiscal year 2003, FHWA expects to fund about $20 billion
in highway infrastructure improvements and congestion mitigations. The
responsibility for designing, planning, and awarding contracts for federally funded


Page 2                            GAO-03-338R Federal Highway Environmental Analysis
highway projects generally rests with state departments of transportation and local
planning organizations.

Before a federally funded highway project can be constructed, it must comply with
the requirements of the National Environmental Policy Act of 1969, among other
things. Under the act, the consequences, if any, of proposed transportation projects
and alternative choices (such as alternative routings) on the natural and human (e.g.,
health) environment and on historic properties must be identified and assessed. For
federally funded highway projects that will have a significant impact on the
environment, the state department of transportation prepares an environmental
impact statement, which FHWA must approve before the project can be built. The
environmental impact statement must describe the project, characterize the
surrounding environment, analyze the environmental effects of a range of reasonable
project alternatives, and indicate plans for complying with environmental laws and
mitigating environmental damage, if any. Other federal agencies (called resource
agencies), such as the Environmental Protection Agency, Army Corps of Engineers,
and Fish and Wildlife Service, often participate in the preparation and review of the
environmental impact statements for highway projects because of their
responsibilities under federal laws, such as the Clean Air Act, Clean Water Act, and
Endangered Species Act. Projects requiring an environmental impact statement are
usually complex and affect sensitive populations. According to FHWA, only about 3
percent of all federally funded highway projects have a significant enough impact on
the environment to require preparation of an environmental impact statement. In
2001, FHWA reported that the average time to prepare an environmental impact
statement had increased from 2.2 years in the 1970s to 5.0 years in the 1990s because
requirements for environmental impact statements have become more extensive and
more complex over time. Environmental review now takes as much as 25 percent of
the total time needed to plan, design, gain approval for, and construct a complex
federally funded highway project.2

Following a March 8, 2000, hearing, the Subcommittee on Ground Transportation,
House Committee on Transportation and Infrastructure, requested that FHWA
provide information about active highway projects requiring an environmental impact
statement whose federal environmental review had not been completed within 5
years.3 Specifically, the subcommittee requested that FHWA identify the location,
scope, duration of the federal environmental review, and specific steps that FHWA
had taken with the resource agencies and states to streamline the approval of these
particular projects. In response, FHWA surveyed its 52 division offices and its 3
Federal Land Highways Division offices to obtain this information. In addition,

2
 Federal Highway Administration, Evaluating the Performance of Environmental Streamlining:
Development of a NEPA Baseline for Measuring Continuous Performance (Washington, D.C.: May 8,
2001) and U.S. General Accounting Office, Highway Infrastructure: Preliminary Information on the
Timely Completion of Highway Construction Projects, GAO-02-1067T (Washington, D.C.:
Sept. 19, 2002). Highway projects with no significant impact on the environment do not require
preparation of an environmental impact statement.
3
 This subcommittee is now the Subcommittee on Highways and Transit.


Page 3                               GAO-03-338R Federal Highway Environmental Analysis
FHWA asked its division staff to provide a brief narrative on reasons why the projects
                                                                    4
had been in the environmental review phase for more than 5 years. The survey
identified 89 active highway projects that had been undergoing federal environmental
review for more than 5 years. According to an FHWA official, the survey and
subsequent analysis were not intended to be scientifically rigorous; rather, they were
intended to provide a quick exploration of the subject so that it could begin to think
about this issue and help begin to inform the public debate.

FHWA provided the preliminary results of its analysis to the Congress in September
2000 and the final results 3 months later. FHWA posted its final results on its Web
page.5 FHWA concluded that the top three reasons why highway projects were in the
environmental review phase for more than 5 years were (1) lack of funding or low
priority (33 percent), (2) local controversy (16 percent), and (3) complexity of the
projects (13 percent). The analytical results presented by FHWA were primarily in
the form of a pie chart, using the characterizations shown below. (See table 1.) The
analysis has been widely used by FHWA and by groups in commenting on whether
legislative changes are needed to environmental review requirements when the
                                      st
Transportation Equity Act for the 21 Century is reauthorized.

Table 1: FHWA’s Characterization of Why Environmental Reviews of 89 Active Highway
Projects Took More Than 5 Years

Reason cited                                                    Percent of total projects
Lack of funding                                                                              18
Local controversy                                                                            16
Low priority                                                                                 15
Complex project                                                                              13
Resource agency review                                                                        8
Change in scope                                                                               8
Fish and Wildlife Service/Endangered Species Act                                              7
Section 106 of the National Historic Preservation Act of 1966                                 6
Wetlands                                                                                      4
Law suits                                                                                     3
Hazardous materials                                                                           2
Total                                                                                       100

Source: FHWA.

A More Refined and Transparent Methodology Would
Have Been Likely to Produce More Useful Results

FHWA’s reliance on its divisions for information was reasonable given the agency’s
desire for a quick exploration of the subject. However, its methodology did not yield
particularly useful or necessarily reliable results. FHWA did not provide enough


4
 According to FHWA, in some cases, division offices consulted with state departments of
transportation.
5
 The results of the analysis can be found at www.fhwa.dot.gov/environment/strmlng/eisdelay.htm.


Page 4                                  GAO-03-338R Federal Highway Environmental Analysis
detail on its methodology to allow potential users to assess the extent that they
should rely on FHWA’s results.

Use of Multiple-choice Questions Would Have
Been Likely to Produce More Useful Results

FHWA asked its division offices for a brief narrative on why the identified projects
were in environmental review for more than 5 years but did not provide any guidance
for preparing the narrative. As a result, (1) the responses were general and did not
provide enough information for FHWA to report on the underlying causes and (2)
when multiple responses were given, the FHWA staff member who conducted the
effort had to decide which reason was the primary reason.

The difficulty in determining the underlying causes of why environmental review
lasted more than 5 years is apparent in the response categories that FHWA ultimately
reported. For example, FHWA reported that local controversy was the second most
prevalent reason given (16 percent of respondents). However, the response category
is not specific enough to determine the reason for the controversy. The local
controversy could have existed either for environmental or nonenvironmental
reasons. For example, some in local communities may have believed the project was
not needed, or they may have believed the project was needed but objected to its
scope or its impact on wetlands or historic properties. Other response categories
were similarly uninformative. For example, the resource agency review, Fish and
Wildlife Service/Endangered Species Act, wetlands, and the section 106 of the historic
preservation act categories are not specific enough to determine the underlying
nature of the problem (e.g., availability of staff to conduct environmental and historic
property reviews, issues involving working relationships among stakeholders,
differences in interpretations of requirements).

Because FHWA provided no guidance to its division offices on differentiating primary
reasons from other reasons, it had no means of determining which reason or reasons
were important and which were not when the respondent cited more than one reason
and did not indicate their relative importance. As a result, FHWA cannot assure that
respondents would have agreed with FHWA’s judgment as to which reasons were
important. We found that many of the narrative responses contained more than one
reason for the time spent in environmental review and the main reason was
unambiguous in only a few responses. When the narrative responses included
multiple reasons, the FHWA staff member chose only one reason as the main reason,
basing her selection, in large part, on which reason was listed first. For example, one
narrative response cited “[c]ontroversy, lawsuits (Consent Decree), funding/cost, and
replenishment housing.” The FHWA staff member recalled that she chose to
categorize “controversy” as this response’s main reason because it came first in the
                                                                     6
list, even though it was not clear that this was the primary reason.
6
 The FHWA staff member said that, for several projects with which she was familiar, she used her
knowledge to choose which was the main reason of several reasons given. The staff member kept no
records of the category to which she assigned each response.



Page 5                             GAO-03-338R Federal Highway Environmental Analysis
The use of multiple-choice questions would probably have made the results of
FHWA’s survey more useful. We recognize that developing clear multiple-choice
questions with informative answers that cover most anticipated responses can be
time consuming. In addition, FHWA officials told us that it did not have enough
knowledge at the outset of its efforts to prepare multiple-choice questions. However,
this approach could have provided clearer, more informative response categories that
would not have required subjective determinations when results were compiled and it
could have allowed respondents to indicate the relative importance of multiple
reasons why environmental review took more than 5 years. A reasonable alternative
to multiple choice questions, within the framework of a quick exploration of the
topic, would have been for FHWA to ask its division offices to provide more
information so that underlying reasons and their relative importance could be
understood.

Effects of Possible Nonresponse Error and the Lack of
Verification of How Responses Were Categorized Are Unknown

To identify active projects whose environmental reviews lasted more than 5 years,
FHWA e-mailed a list of 70 projects that it had identified as being in environmental
review for more than 5 years to each of its 55 division and Federal Land Highways
Division offices and asked them to add to the list as warranted. It followed up with
an e-mail reminder for divisions that did not respond. Ultimately, 41 of these offices
                          7
(75 percent) responded. The FHWA staff member who conducted this effort said she
assumed that the nonresponding division offices had no projects meeting the criteria.
While 75 percent was a substantial response rate, it is possible that the lack of data
from nonrespondents (called “nonresponse error”) may have negatively affected the
accuracy of FHWA’s analyses in two ways. First, FHWA may not have identified all
(or nearly all) active highway projects whose environmental reviews lasted more than
5 years. Second, to the extent that such highway projects existed in nonresponding
jurisdictions, the reasons that the nonresponding division offices might have cited
could have been different from the results that FHWA reported. A higher response
rate would have reduced the likelihood of either or both of these possible
nonresponse errors. However, because we did not contact the nonrespondents to
determine whether their jurisdiction included projects that met the survey criteria,
we cannot determine what effect, if any, a higher response rate might have had.
Achieving a very high response rate is often difficult in survey research. However,
the authority that FHWA headquarters exercises over its division offices might be
expected to produce a response rate nearing 100 percent.

According to FHWA officials, they did not perform quality control to ensure that the
FHWA staff member reasonably and accurately assigned survey responses to the
individual categories. According to the FHWA staff member who performed this
7
 FHWA might have elected to survey other transportation stakeholders in order to obtain additional
perspectives. However, doing so would have substantially increased the effort needed. We believe
that FHWA’s decision to limit the scope of its survey to its division offices was reasonable given its
desire for a quick exploration of the subject.


Page 6                                    GAO-03-338R Federal Highway Environmental Analysis
analysis, no one either independently categorized the narrative responses or checked
her work. We did not attempt to independently categorize responses—in part
because we would have had no basis for selecting primary reasons when respondents
cited multiple reasons and did not identify the primary reasons--and therefore cannot
say whether a similar effort by FHWA at the time it was compiling results would have
affected the results it reported.

FHWA Could Have Better Disclosed How It
Conducted Its Survey and Subsequent Analysis

On its Web page, FHWA reported that “[d]uring the spring and summer of 2000…[it]
surveyed the Division offices and the Regional Federal Land Highways Divisions…to
gather information to respond to a question from [a House subcommittee].” It also
listed the information it sought from its divisions. However, FHWA did not provide
any further information on the quality of its data, how it resolved data issues, and
                                                     8
how it made judgments about the data it collected. Given the significant weaknesses
in this effort, such disclosure would have helped potential users assess the rigor of
the analysis and the extent to which they should rely on it. In response to our
findings, FHWA revised its Web site in a manner that adequately discloses the issues
discussed in this report.

Issues to Address if the 89 Projects Are Resurveyed

It would be feasible to resurvey the 89 projects using the lessons learned from our
review to attempt to obtain more useful information on why the environmental
review for each lasted more than 5 years. However, other approaches, such as
reviewing a newer set of projects, might be more useful. The costs and benefits of a
resurvey would have to be weighed against the expected results of several FHWA
initiatives that are intended to shed light on how environmental reviews of federally
funded highway projects are completed.

Projects Could Be Resurveyed but Other
Approaches and Issues Would Have to Be Addressed

It would certainly be feasible for FHWA to resurvey the 89 projects that it analyzed in
2000. Even within the context of a quick exploration, FHWA might be able to obtain
more useful information if it employed a methodology that allowed it to better
identify the underlying causes—and their relative importance--than it did in its
original analysis. However, assessing the desirability of such an endeavor would
involve several considerations:



8
 In addition, FHWA did not indicate whether the data obtained were based on recollection of
respondents or from the respondents’ reviews of documentation. Although documentary evidence is
generally preferable to recall, sometimes documentary evidence is either not available or time-
consuming to review. The use of recall may be appropriate in these instances, but disclosure of its use
is important.


Page 7                               GAO-03-338R Federal Highway Environmental Analysis
•   The 14 division offices that did not respond to the original survey would have to
    be contacted to verify that, in 2000, no active highway projects under their
    jurisdiction had been in environmental review for more than 5 years.

•   To the extent that FHWA division staff based their original responses on recall,
    the accuracy of their responses to a resurvey may be reduced because 2 years
    have elapsed since the 2000 survey. In addition, some staff with the best
    knowledge of events in 2000 might have moved on and documentation, if used
    before, might be more difficult to locate.

•   A newer data set--for example, highway projects that were in environmental
    review for more than 5 years as of the date of any resurvey--could provide better
    information about current conditions.

•   Surveying projects whose environmental review took less than 5 years could
    provide additional perspective about FHWA’s findings in its original analysis. For
    example, environmental issues (e.g., resource agency review, endangered species,
    historic preservation) would likely have been present in projects where
    environmental review lasted less than 5 years by the very fact that an
    environmental impact statement was required. In addition this approach might
    provide some insights into actions by transportation stakeholders that have
    reduced the time taken for environmental review.

•   Obtaining the views of other transportation stakeholders--such as state
    departments of transportation, federal and state resource agencies, community
    action agencies, and others--would better ensure that the reasons cited
    represented all viewpoints. However, because many stakeholders are involved in
    complex projects, identifying them and obtaining their views would add
    substantially to the complexity of the effort.

FHWA Initiatives May Provide Information about
Reasons for Time Spent in Environmental Review

FHWA has several initiatives under way aimed at improving its understanding of the
reasons for the time projects spent in environmental review. In particular, FHWA has
two studies that follow up on a study completed in May 2001 looking for variables
that correlate with time spent in environmental review.9 In the first study, a
contractor is reviewing about 250 projects from the 1990s whose environmental
reviews have been completed. Like the May 2001 study, this study is looking for
variables that correlate with time spent in environmental review, such as whether
controversy existed regarding wetlands or endangered species. FHWA expects that
the results of this study will be available around March 2003. In the second initiative,
FHWA is reviewing 8 highway projects whose environmental reviews took less than 5

9
Federal Highway Administration, Evaluating the Performance of Environmental Streamlining:
Development of a NEPA Baseline for Measuring Continuous Performance (Washington, D.C.: May 8,
2001).


Page 8                                GAO-03-338R Federal Highway Environmental Analysis
years and 8 highway projects whose environmental reviews took more than 5 years to
complete, to identify factors that contribute to time spent in environmental review.
FHWA expects the results of its analysis of projects taking less than 5 years to
complete environmental review to be issued by the end of January 2003. FHWA does
not yet have an estimate for when the remaining portion of the initiative will be
complete. Finally, FHWA has commissioned a Gallup poll of federal and a limited
number of non-federal stakeholders to survey their perceptions of the environmental
review process and of how well stakeholders work together. FHWA expects the
results of this poll to be completed by spring 2003. We did not assess whether these
initiatives are likely to provide useful results.

As you know, we are assessing for this committee the reasons that environmental
review of federally funded highway projects take years. We expect to report our
results in spring 2003.

Conclusions

By relying on often general narrative responses without directing its division offices
to indicate the relative importance of responses, FHWA limited its ability to provide
useful insights on why environmental reviews of certain federally funded highway
projects took more than 5 years. Furthermore, by publishing its survey results and its
subsequent analysis in reports and on its Web site without describing its
methodology, FHWA limited the ability of the Congress and the public to evaluate the
strengths and weaknesses of its work in order to understand its reliability. While
some weaknesses, such as vague categories used to categorize reasons why
environmental reviews took more than 5 years, are evident without explanation,
other weaknesses, such as FHWA’s inability to separate primary from minor reasons,
are not. If the Congress and the public are not aware of these weaknesses, they may
place undue confidence in FHWA’s results to the extent that they use these results to
evaluate the need for legislative reforms. FHWA’s actions to revise its Web site in
response to our concerns address these disclose concerns.

Agency Comments and Our Evaluation

The Department of Transportation did not offer specific comments on a draft of this
report. Our draft report contained a proposed recommendation that the Secretary of
Transportation direct the FHWA Administrator to describe how its survey and
analysis were conducted in such a way that their strengths and limitations can be
understood and that this description should be included in any public use of the
survey and analysis, including their posting on FHWA’s Web site. As discussed
earlier, FHWA revised its Web site to address our concerns. As a result, we deleted
the recommendation from this report.




Page 9                         GAO-03-338R Federal Highway Environmental Analysis
Scope and Methodology

To analyze FHWA’s methodology, we interviewed officials in FHWA’s Office of
National Environmental Policy Act Facilitation to determine how they carried out and
reported the results of their analysis and assessed their approach using basic
analytical approaches. Specifically, we asked them to explain how they selected
respondents, identified projects, provided instructions on how to complete the
survey, categorized narrative responses, and performed quality control. We then
reviewed FHWA’s methodology to determine how measurement error and bias were
            10
minimized. To assess potential sources of measurement error, we analyzed FHWA’s
methodology for determining and categorizing the main reason for time in
environmental review from the narrative responses it received. This included
reviewing whether categories were clear and whether coding of the main response
was determined through use of consistent rules and verified by a second coder. To
assess potential sources of bias, we reviewed FHWA’s methodology for identifying
projects to be included in the survey, its identification of the appropriate respondents
to the survey, efforts to minimize nonresponse from among those surveyed, and the
sources of data used by respondents when providing narrative responses. Although
we did review the data that FHWA obtained, we did not attempt to determine
whether FHWA categorized all narrative responses correctly. Furthermore, we did
not attempt to determine the validity of the answers provided by contacting the
FHWA officials who completed the survey or through other independent verification.

To assess whether it was feasible to resurvey the same projects about why
environmental reviews took more than 5 years, we considered the age of the data and
its sources, as well as the limitations that we observed in FHWA’s survey and
subsequent analysis. Finally, we identified initiatives under way at FHWA to provide
information on environmental reviews of federally funded highway projects. We did
not assess whether these initiatives are likely to provide useful results.

We conducted our work from October 2002 through January 2003 in accordance with
generally accepted government auditing standards.

                                             _ _ _ _ _

As arranged with your office, unless you publicly announce its contents earlier, we
plan no further distribution of this report until 30 days after the date of this letter. At
that time, we will send copies of this report to congressional committees with
responsibilities for highway and environmental issues; the Secretary of
Transportation; the Administrator, Federal Highway Administration; and the Director,
Office of Management and Budget. We will also make copies available to others upon
request. This report will be available at no charge on our home page at
http://www.gao.gov.


10
 Measurement error results in inaccuracy in quantifying a variable. Bias results in a misrepresentation
of what is being measured.


Page 10                                  GAO-03-338R Federal Highway Environmental Analysis
If you or your staff have any questions about this report, please contact either James
Ratzenberger at ratzenbergerj@gao.gov or me at siggerudk@gao.gov. Alternatively,
we may be reached at (202) 512-2834. Key contributors to this report were Gail
Marnik, Kristen Massey, SaraAnn Moessbauer, and James Ratzenberger.

Sincerely yours,




Katherine Siggerud
Acting Director, Physical Infrastructure Issues




(542015)


Page 11                        GAO-03-338R Federal Highway Environmental Analysis