oversight

Pipeline Safety: Systematic Process Needed to Evaluate Outcomes of Research and Development Program

Published by the Government Accountability Office on 2003-06-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to Congressional Subcommittees




June 2003
             PIPELINE SAFETY
             Systematic Process
             Needed to Evaluate
             Outcomes of Research
             and Development
             Program




GAO-03-746
             a
                                                June 2003


                                                PIPELINE SAFETY

                                                Systematic Process Needed to Evaluate
Highlights of GAO-03-746, a report to           Outcomes of Research and Development
congressional subcommittees
                                                Program



From 1998 through 2002, a total of              OPS distributes its R&D budget among four main areas. For example, in
1,770 pipeline accidents occurred,              fiscal year 2003, the office plans to allocate its $8.7 million budget as follows:
resulting in 100 fatalities and                 •   46 percent ($4.0 million) to developing new technologies to prevent
$621 million in property damage.                    damage to pipelines and prevent leaks;
The Office of Pipeline Safety (OPS)             •   21 percent ($1.9 million) to improving technologies for operating,
within the Department of
Transportation operates a research
                                                    controlling, and monitoring the condition of pipelines;
and development (R&D) program                   •   19 percent ($1.7 million) to improved pipeline materials, such as
aimed at advancing the most                         materials that are resistant to damage and defects; and
promising technologies for                      •   14 percent ($1.2 million) to efforts to improve data on the location and
ensuring the safe operation of                      safety performance of pipelines.
pipelines. In fiscal year 2003, OPS             On the basis of our work, we believe that OPS’s R&D funding is generally
received $8.7 million for its R&D               aligned with its mission and pipeline safety goals. OPS has taken a number
program, a sevenfold increase                   of steps to ensure this alignment. For example, it obtained the views of a
since fiscal year 1998. In response             variety of experts and stakeholders in deciding on its R&D priorities and has
to a directive from the House                   described in various plans how its R&D efforts can lead to new and
Committee on Appropriations,                    improved technologies that can help achieve its safety performance goals,
GAO (1) assessed OPS’s
distribution of funding among
                                                such as reducing the impacts of pipeline accidents.
various areas of R&D and the
alignment of this funding with its              The pipeline safety R&D priorities of the experts we surveyed are generally
mission and goals, (2) surveyed                 consistent with OPS’s R&D priorities. For example, most assigned a high
experts to obtain their views on                priority to the two areas of R&D that receive the highest amount of funding
R&D priorities, and (3) determined              from OPS.
how OPS evaluates R&D outcomes.
                                                OPS’s efforts to evaluate the outcomes of its R&D have been limited. The
                                                agency has taken some preliminary steps toward developing an evaluation
                                                process for its R&D program, such as identifying possible measures of
To better determine the                         program results. Leading research organizations, the Office of Management
effectiveness of its R&D program,               and Budget, and GAO have identified a number of best practices for
GAO recommends that OPS                         systematically evaluating the outcomes of federal R&D programs, such as
develop a systematic process for                setting clear R&D goals, measuring progress toward goals, and reporting
evaluating program outcomes,                    periodically on evaluation results. These best practices can help OPS to
using recognized best practices,                determine the effectiveness of its R&D program in achieving desired
and include the results of R&D                  outcomes, such as the development and use of new and improved
evaluations in its annual reports to            technologies that can enhance pipeline safety.
Congress.

OPS officials told us that they
generally agreed with the report’s
findings and will follow our
recommendations as they continue
to develop an evaluation process
for their R&D program.
www.gao.gov/cgi-bin/getrpt?GAO-03-746.

To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Peter Guerrero
at (202) 512-2834 or guerrerop@gao.gov.
Contents



Letter                                                                                                 1
                             Results in Brief                                                          2
                             Background                                                                5
                             OPS’s R&D Funding Is Aligned with Its Mission and Pipeline Safety
                               Goals                                                                  10
                             Experts Generally Support OPS’s R&D Priorities                           19
                             OPS Lacks a Systematic Process for Evaluating R&D Outcomes               24
                             Conclusions                                                              31
                             Recommendations for Executive Action                                     32
                             Agency Comments                                                          32


Appendixes
              Appendix I:    Experts’ Views on R&D Priorities and OPS’s R&D Funding, by
                             Type of R&D                                                              34
             Appendix II:    Scope and Methodology                                                    38
             Appendix III:   GAO Contacts and Staff Acknowledgments                                   42
                             GAO Contacts                                                             42
                             Acknowledgments                                                          42


Tables                       Table 1: Major Categories of R&D Related to Pipeline Safety              20
                             Table 2: Views of Experts from Three Subgroups on Pipeline Safety
                                      R&D Priorities                                                  24


Figures                      Figure 1: OPS’s R&D Budget, Fiscal Years 1998-2003                        9
                             Figure 2: OPS’s Planned Allocation of R&D Funding for Fiscal Year
                                       2003                                                           12
                             Figure 3: OPS’s R&D Funding by Area of R&D, Fiscal Years
                                       2001-03                                                        14
                             Figure 4: Expert Ratings of Categories of Pipeline Safety R&D            21




                             Page i                                        GAO-03-746 Pipeline Safety R&D
Contents




Abbreviations

DOE          Department of Energy
DOT          Department of Transportation
MMS          Minerals Management Service
NIH          National Institutes of Health
OPS          Office of Pipeline Safety
R&D          research and development
RSPA         Research and Special Programs Administration

 This is a work of the U.S. government and is not subject to copyright protection in the
 United States. It may be reproduced and distributed in its entirety without further
 permission from GAO. However, because this work may contain copyrighted images or
 other material, permission from the copyright holder may be necessary if you wish to
 reproduce this material separately.




Page ii                                                   GAO-03-746 Pipeline Safety R&D
A
United States General Accounting Office
Washington, D.C. 20548



                                    June 30, 2003                                                                    Lert




                                    The Honorable Ernest Istook, Jr.
                                    Chairman
                                    The Honorable John Olver
                                    Ranking Minority Member
                                    Subcommittee on Transportation,
                                     Treasury, and Independent Agencies
                                    Committee on Appropriations
                                    House of Representatives

                                    The Honorable Richard C. Shelby
                                    Chairman
                                    The Honorable Patty Murray
                                    Ranking Minority Member
                                    Subcommittee on Transportation,
                                     Treasury, and General Government
                                    Committee on Appropriations
                                    United States Senate

                                    Pipelines transport nearly all of the natural gas and nearly two-thirds of the
                                    crude oil and refined oil products in the United States. Although pipelines
                                    have a better safety record than other modes of freight transportation, their
                                    cargo is dangerous and leaks or ruptures can have serious consequences,
                                    including fatalities, harm to the environment, and property damage. For
                                    example, pipeline ruptures in Bellingham, Washington, in 1999 and in
                                    Carlsbad, New Mexico, in 2000 together resulted in a total of 15 deaths and
                                    property and other damages totaling about $46 million. Investigators have
                                    determined that one of the probable causes of the Bellingham accident was
                                    excavation damage and that the cause of the Carlsbad accident was severe
                                    internal corrosion.

                                    The Office of Pipeline Safety (OPS), within the Department of
                                    Transportation’s (DOT) Research and Special Programs Administration, is
                                    responsible for pipeline safety regulation and research. The agency
                                    operates a research and development (R&D) program aimed at enhancing
                                    the safety and reducing the potential environmental impacts of
                                    transporting natural gas and hazardous liquids through pipelines.
                                    Specifically, the program seeks to advance the most promising
                                    technological solutions to problems that impede pipeline safety, such as
                                    damage to pipelines from excavation or corrosion. From fiscal years 2001
                                    through 2003, the budget of OPS’s R&D program more than tripled, from



                                    Page 1                                            GAO-03-746 Pipeline Safety R&D
                   $2.8 million to $8.7 million, partly as a result of congressional interest in
                   achieving technological advances that can improve pipeline safety.

                   In House Report 107-722, which accompanied the Department of
                   Transportation and Related Agencies Appropriations Bill for fiscal year
                   2003, the House Appropriations Committee raised concerns regarding the
                   effective management and utilization of these significant increases in
                   funding for the department’s pipeline safety R&D program. The committee
                   directed GAO to review the effectiveness of the program. In subsequent
                   discussions with staff of the Subcommittee on Transportation, Treasury,
                   and Independent Agencies of the House Appropriations Committee, we
                   agreed to determine (1) OPS’s distribution of funding among various areas
                   of pipeline safety R&D since fiscal year 2001 and the extent to which this
                   funding is aligned with the agency’s mission and pipeline safety goals,
                   (2) the views of experts on pipeline safety R&D priorities, and (3) how OPS
                   evaluates the outcomes of the pipeline safety R&D it funds.

                   To carry out this work, we reviewed legislation and agency documents
                   pertaining to the R&D program and interviewed agency officials
                   responsible for this program. We also interviewed key experts and
                   stakeholders regarding their views on R&D priorities and gaps and on
                   OPS’s management of its R&D program, including the alignment of the
                   agency’s research agenda with its mission and goals. We identified best
                   practices for evaluating the outcomes of R&D through a review of relevant
                   literature. In addition, we sent a questionnaire to selected experts to obtain
                   their views on pipeline safety R&D priorities. We selected experts who are
                   informed about pipeline safety or the development of new pipeline safety
                   technologies, including representatives of federal and state agencies,
                   pipeline safety advocacy groups, industry associations, pipeline
                   companies, technical and consulting organizations, and research institutes.
                   We received responses from 49 of 55 experts we contacted, for a response
                   rate of 89 percent. Our results pertaining to experts’ views on R&D
                   priorities represent the views of only the experts who responded to our
                   questionnaire and cannot be generalized to a broader population. (See app.
                   II for additional details on our scope and methodology.)



Results in Brief   OPS distributes its R&D budget to three major areas involving the research
                   and development of pipeline safety technologies as well as to a fourth
                   area—efforts to improve the agency’s pipeline mapping and information
                   systems. For example, in fiscal year 2003, OPS plans to allocate its
                   $8.7 million R&D budget as follows:



                   Page 2                                             GAO-03-746 Pipeline Safety R&D
• $4.0 million (46 percent) to developing new technologies for preventing
  damage to pipelines and detecting leaks,

• $1.9 million (21 percent) to improving technologies for operating,
  controlling, and monitoring the condition of pipelines,

• $1.7 million (19 percent) to improving pipeline materials, and

• $1.2 million (14 percent) on efforts to improve pipeline mapping data
  and data on the safety performance of pipelines.1

On the basis of our work, we believe that OPS’s R&D funding is generally
aligned with its mission and pipeline safety goals. The agency has obtained
the views of external experts and stakeholders in determining what types
of R&D are aligned with its mission of ensuring the safe, reliable, and
environmentally sound operation of the nation’s pipeline transportation
system. OPS has also recently improved coordination with other federal
agencies that fund pipeline R&D in order to avoid overlap between their
R&D programs. Both expert review and coordination among agencies are
recognized as best practices that help ensure that federal agencies’ R&D
activities are relevant to their missions and do not overlap. OPS has also
described, in various plans, how its R&D efforts can lead to new and
improved technologies that can help achieve its performance goals of
reducing the impacts of pipeline incidents, including fatalities and injuries,
and reducing spills of hazardous material. Key experts and stakeholders
we contacted generally told us that, in their view, the agency has chosen
appropriate R&D areas to fund.

The pipeline safety R&D priorities of the experts who completed our
questionnaire are generally consistent with OPS’s R&D priorities. The
ranking of the major R&D areas based on the responses to our
questionnaire is similar to the relative levels of funding OPS has assigned to
these areas:

• 92 percent (45 of 49) of the experts assigned a high priority to the
  development of new technologies for preventing damage to pipelines
  and detecting leaks,




1
Figures do not add to total due to rounding.




Page 3                                            GAO-03-746 Pipeline Safety R&D
• 80 percent (39 of 49) assigned a high priority to improvements in
  technologies for operating, controlling, and monitoring the condition of
  pipelines, and

•   31 percent (15 of 49) assigned a high priority to improvements in
    pipeline materials.

However, the experts’ level of support for improvements in pipeline
materials was much lower than that for the other two main R&D areas that
OPS is funding and this level of support differed across different groups of
experts. Although 70 percent (7 of 10) of experts from research
organizations indicated that this area should receive high priority, only 21
percent (8 of 39) of the remaining experts—from government, public
interest, industry, and technical and consulting organizations—indicated
that it should receive high priority. OPS officials told us that they are
currently updating their research agenda, using the input of experts and
stakeholders, and that they will consider our questionnaire results in this
process.

Despite the significant growth in its R&D budget since fiscal year 2001, OPS
has not developed a systematic process for evaluating the outcomes of the
R&D it funds. For example, the agency tracks and disseminates
information on the progress of individual R&D projects but has not
developed a process for assessing and reporting on the results of its R&D
program as a whole. Without such a process, OPS cannot determine and
demonstrate the progress of its R&D program in achieving intended results,
such as the development and use of new and improved technologies that
can enhance pipeline safety. The agency has taken some preliminary steps
toward developing an evaluation process for its R&D program, such as
identifying possible measures of program results, and could benefit from
adopting identified best practices for systematically evaluating the
outcomes of federal R&D programs. Leading research organizations, the
Office of Management and Budget, and GAO have identified a number of
such practices, including setting clear R&D goals and measuring progress
toward these goals, using expert review to evaluate the quality of research
outcomes, and reporting periodically on evaluation results. The results of
evaluations can be used to refocus R&D priorities periodically, as
necessary, to ensure that program resources are most effectively utilized.
The Pipeline Safety Improvement Act of 2002 requires that, starting in
December 2003, DOT, the Department of Energy (DOE), and the National
Institute of Standards and Technology jointly provide annual reports to
Congress on their pipeline R&D efforts but does not fully specify what



Page 4                                           GAO-03-746 Pipeline Safety R&D
             types of information should be included in these reports. This requirement
             provides an opportunity for OPS to keep Congress informed about the
             results of evaluations of its R&D program.

             To improve OPS’s ability to demonstrate the effectiveness of its R&D
             program and to make the most effective use of program resources, we are
             recommending that the agency develop a systematic process for evaluating
             program outcomes, using identified best practices, and include the results
             of R&D evaluations in the required annual reports to Congress on pipeline
             R&D. We provided DOT with a draft of this report for its review and
             comment. DOT officials generally agreed with the report’s findings and
             conclusions. They emphasized that they have started to develop a
             framework for evaluating the effectiveness of their pipeline safety R&D
             program and that they intend to follow our recommendations as they move
             forward in developing and implementing this framework.



Background   Three primary types of pipelines form a 2.2 million-mile network across the
             nation.

             • Natural gas transmission pipelines transport natural gas over long
               distances from sources to communities.

             • Natural gas distribution pipelines continue to transport natural gas from
               transmission lines to consumers.

             • Hazardous liquid pipelines transport crude oil to refineries and refined
               oil products, such as gasoline, to product terminals.




             Page 5                                          GAO-03-746 Pipeline Safety R&D
OPS, within DOT’s Research and Special Programs Administration (RSPA),
is responsible for enhancing the safety of and reducing the potential
environmental impacts of transporting natural gas and hazardous liquids
through pipelines. The agency primarily carries out this responsibility
through regulation, oversight, enforcement, and R&D. OPS sets and
enforces regulations that pipeline operators must follow in designing,
constructing, maintaining, and operating pipelines. State agencies
responsible for overseeing pipeline safety help OPS to enforce its
regulations.2 In December 2000, it began implementing a new risk-based
regulatory approach, called “integrity management.” Under this approach,
operators are required, in addition to meeting minimum safety standards,
to better protect pipeline segments where a leak or rupture could have
significant consequences, such as near highly populated areas, by
conducting new tests of these segments, completing repairs according to
specified schedules, and developing comprehensive plans for addressing
the range of risks facing these segments.3 The agency’s R&D program is
aimed at advancing the most promising technologies for ensuring the safe
operations of pipelines. For example, current R&D projects seek to
develop new and improved techniques for assessing the condition of
pipelines and detecting anomalies—such as leaks, corrosion, and damage
from excavators—that can lead to pipeline accidents. From 1998 through
2002, a total of 1,770 pipeline accidents occurred, resulting in 100 fatalities
and $621 million in property damage.4




2
 In general, OPS retains full responsibility for inspecting and enforcing regulations on
interstate pipelines but certifies states to perform these functions for intrastate pipelines. In
2003, 49 state agencies, the District of Columbia, and Puerto Rico were certified for
inspecting and enforcing regulations on intrastate pipelines. In addition, OPS has
agreements with 11 states to inspect segments of interstate pipelines within their
boundaries.
3
 We have previously reported on OPS’s implementation of this new regulatory approach.
See U.S. General Accounting Office, Pipeline Safety and Security: Improved Workforce
Planning and Communication Needed, GAO-02-785 (Washington, D.C.: Aug. 26, 2002), and
Pipeline Safety: The Office of Pipeline Safety Is Changing How It Oversees the Pipeline
Industry, GAO/RCED-00-128 (Washington, D.C.: May 15, 2000).
4
 These figures are based on accidents reported to OPS. For hazardous liquid pipelines, they
include accidents involving any fatality or injury, a fire or explosion, total costs of $50,000 or
more, or releases of 50 or more barrels of hazardous liquids or 5 or more barrels of highly
volatile liquids. For natural gas pipelines, they include accidents involving any fatality or
injury, total costs of $50,000 or more, or the emergency shutdown of a liquified natural gas
facility, as well as any accidents considered to be significant by the pipeline operator.




Page 6                                                         GAO-03-746 Pipeline Safety R&D
OPS’s R&D program has undergone major changes in the last several years.
In particular, the agency has developed a new agenda for its R&D program,
using the input of key experts and stakeholders, and has received
significant increases in funding for this program.

• Until 2001, most of the research funded by OPS was aimed at helping the
  agency perform its regulatory function or was in response to an accident
  investigation or congressional direction. In November 2001, the agency
  held an R&D planning workshop to gain the perspectives of a variety of
  experts and stakeholders on areas of R&D that have the most potential
  for enhancing pipeline safety. Attendees included representatives of
  federal and state agencies, research organizations, industry groups,
  pipeline companies, and technical organizations that set industry safety
  standards. OPS used the R&D priorities identified in this workshop to
  develop a new agenda for its R&D program, focusing on three main
  areas: (1) developing new technologies for preventing damage and
  detecting leaks, (2) improving technologies for operating, controlling,
  and monitoring the condition of pipelines, and (3) improving pipeline
  materials. From March through December 2002, the agency issued
  announcements requesting project proposals in these areas, asking that
  prospective funding recipients provide at least 50 percent of the
  proposed project’s cost. As of May 2003, it had funded 10 R&D
  proposals it received in response to these announcements.5 In addition,
  after its November 2001 R&D workshop, OPS established a Web site on
  its R&D program in order to improve communications with experts,
  stakeholders, and the public about its R&D agenda and activities.




5
 In March 2002, OPS requested proposals related to damage prevention and leak detection.
It received 82 proposals in response and, in November 2002, funded 7 of them. In June 2002,
the agency requested proposals related to enhanced pipeline operations, controls, and
monitoring. It received 57 proposals in response and, in February 2003, funded 3 of them,
based on the availability of funding. OPS intends to fund 3 more of these proposals in June
2003. OPS has provided approximately 50 percent of the cost of the projects to awardees.
In December 2002, the agency requested proposals related to improved performance of
pipeline materials and other pipeline safety improvements. It expects to make funding
decisions about these proposals in summer 2003.




Page 7                                                    GAO-03-746 Pipeline Safety R&D
• OPS’s budget for its R&D program has risen more than sevenfold since
  fiscal year 1998, with the most significant increases occurring since
  fiscal year 2001. Figure 1 shows the agency’s budgeted amounts for
  R&D from fiscal years 1998 through 2003.6 OPS’s budget for R&D rose
  steadily from fiscal year 1998 to fiscal year 2001, from $1.3 million to
  $2.8 million. In fiscal year 2002, the agency received $4.8 million for its
  R&D program, which was $2 million more than RSPA had requested for
  the program. Agency officials attribute this funding increase to
  increased concerns for pipeline safety within Congress following the
  tragic pipeline accidents in Bellingham, Washington (1999), and
  Carlsbad, New Mexico (2000), which together caused 15 fatalities. For
  fiscal year 2003, RSPA requested and received about $4 million in
  additional funding for the program, for a total of $8.7 million. OPS
  officials told us that this requested increase was a response to
  heightened congressional interest in achieving technological solutions
  to pipeline safety, as evidenced by legislative proposals that called for
  increased attention to this area.7 RSPA is proposing funding for OPS’s
  R&D program of $9.2 million in fiscal year 2004, an increase of about
  $0.5 million above the fiscal year 2003 amount. OPS officials explained
  that they intend to use most of this increase for a study, required by the
  Pipeline Safety Improvement Act of 2002, to assess the performance of
  controllers who monitor pipeline operations. Overall, agency officials
  also attribute recent increases in funding for OPS’s pipeline safety R&D
  program to a recognition of the challenges posed by the agency’s new
  integrity management regulatory approach and the criticality of the
  nation’s pipeline infrastructure, in the aftermath of the terrorist attacks
  of September 11, 2001.




6
 These figures have been adjusted to account for inflation. They are in constant fiscal year
2003 dollars.
7
 In addition, RSPA’s budget submission for fiscal year 2003 noted that the proposed pipeline
safety R&D budget would consolidate into RSPA a pipeline infrastructure R&D program
operated by DOE. However, according to DOE and OPS officials, no transfer of funding or
projects between the two programs actually took place.




Page 8                                                      GAO-03-746 Pipeline Safety R&D
Figure 1: OPS’s R&D Budget, Fiscal Years 1998-2003




Note: Figures are in constant fiscal year 2003 dollars.


OPS’s pipeline safety R&D program is continuing to evolve in response to
new directives in the Pipeline Safety Improvement Act of 2002 for the
planning and reporting of federal pipeline R&D efforts. The act, which
became law in December 2002, assigned the Secretary of Transportation
responsibility for developing a 5-year plan for pipeline R&D and
transmitting the plan to Congress by December 2003, in coordination with
DOE and the National Institute of Standards and Technology. (OPS
officials told us that the Secretary has delegated this responsibility to OPS.)

DOE operates an R&D program that is focused on developing future
technologies to improve the integrity, reliability, and security of the natural
gas infrastructure, including pipelines and storage facilities. In
comparison with OPS’s R&D program, which focuses on the development
of quick-to-market technologies that could become available in the short
term (1-3 years) or midterm (3-5 years), DOE’s program focuses on
technologies that could become available in the midterm (3-5 years) or
longer term (5-8 years). The National Institute of Standards and
Technology does not operate an R&D program focused on pipelines, but,




Page 9                                                    GAO-03-746 Pipeline Safety R&D
                       reflecting its expertise in materials research, the act assigns it a key role in
                       planning future pipeline R&D.

                       The Department of the Interior’s Minerals Management Service (MMS),
                       although not assigned an R&D planning role in the act, funds pipeline R&D,
                       including research on offshore pipeline safety. Consequently, OPS plans to
                       include that agency in efforts to develop a 5-year plan for pipeline R&D.
                       The act requires the heads of DOT, DOE, and the National Institute of
                       Standards and Technology to jointly report annually to Congress, beginning
                       in December 2003, on the status and results of implementation of the plan.



OPS’s R&D Funding Is   Since fiscal year 2001, OPS has allocated its rising R&D funding to three
                       main areas of pipeline safety R&D that were identified at its 2001
Aligned with Its       workshop: (1) developing new technologies for preventing damage to
Mission and Pipeline   pipelines and detecting leaks, (2) improving technologies for operating,
                       controlling, and monitoring the condition of pipelines, and (3) improving
Safety Goals           the performance of pipeline materials. The agency has also allocated some
                       R&D funding to a fourth area, efforts to improve the agency’s mapping and
                       information systems.

                       On the basis of our work, we believe that the agency’s R&D funding is
                       generally aligned with its mission and pipeline safety goals. The agency has
                       obtained the views of external experts and stakeholders in determining
                       what types of R&D are aligned with its mission of ensuring the safe,
                       reliable, and environmentally sound operation of the nation’s pipeline
                       transportation system. The agency has also recently improved
                       coordination with other federal agencies that fund pipeline R&D in order to
                       avoid overlap between their R&D programs. Both of these practices have
                       been recommended by leading organizations that conduct scientific and
                       engineering research. OPS has also linked its R&D efforts with its
                       performance goals of reducing the impacts of pipeline incidents, including
                       fatalities and injuries, and reducing spills of hazardous material. In its
                       plans, the agency has described how new and improved technologies
                       resulting from its R&D funding can help achieve these performance goals.
                       Finally, a number of key experts and stakeholders told us that, in their view,
                       the agency has chosen appropriate R&D areas to fund.




                       Page 10                                            GAO-03-746 Pipeline Safety R&D
OPS Allocates Pipeline       OPS allocates its R&D budget to three major areas involving the research
Safety R&D Funding to Four   and development of pipeline safety technologies as well as to a fourth
                             area—efforts to improve the agency’s pipeline mapping and information
Major Areas                  systems—that does not involve such research and development. Figure 2
                             shows how the agency plans to distribute its fiscal year 2003 R&D budget of
                             $8.7 million among these areas.8

                             OPS plans to spend the largest share of its R&D budget, 46 percent, or
                             $4.0 million, on the area of Damage Prevention and Leak Detection, which
                             includes the development of new technologies to prevent damage to
                             pipelines, detect pipeline defects, and quickly and accurately locate and
                             control pipeline leaks. Damage to pipelines from “third parties,” such as
                             companies performing excavation work, is the leading cause of pipeline
                             failures and can lead to property damage and injuries or fatalities. 9

                             OPS plans to allocate 21 percent of its R&D budget, $1.9 million, to the area
                             of Enhanced Operations, Controls, and Monitoring, which includes
                             improvements in technologies for operating, controlling, and monitoring
                             the integrity of pipelines to help identify and prioritize pipeline safety
                             problems and solutions.

                             The agency intends to spend a slightly lesser amount, 19 percent of its R&D
                             budget, or $1.7 million, on the area of Improved Materials Performance,
                             which includes improvements in pipeline materials in order to extend the
                             integrity and lifetime of installed pipelines and their various components.

                             Finally, the agency plans to allocate the smallest portion of its R&D budget,
                             14 percent, or $1.2 million, to the area of Mapping and Information
                             Systems, which includes efforts to improve the collection, integration, and
                             analysis of data on the location and safety performance of pipelines. These
                             efforts make pipeline mapping information available to federal, state, and



                             8
                              These amounts represent OPS’s planned expenditures in each area. However, the agency’s
                             actual expenditures in an area depend on the approval of R&D proposals received and may
                             therefore differ from planned expenditures. Figures do not add to total due to rounding.
                             9
                              “Third parties” are people or companies not associated with a pipeline company or its
                             contractors. Damage to pipelines can result from such people or companies digging in the
                             vicinity of buried pipelines without realizing that the pipelines are there. For example,
                             excavating equipment can accidentally strike a pipeline and cause a leak or rupture, either
                             immediately or over time, which poses a hazard to life and property.




                             Page 11                                                    GAO-03-746 Pipeline Safety R&D
local officials and support pipeline inspection activities of OPS and its state
partners.



Figure 2: OPS’s Planned Allocation of R&D Funding for Fiscal Year 2003




Note: Shaded areas represent the major pipeline safety R&D areas funded by OPS. Dollar figures
have been rounded.


Since fiscal year 2001, OPS’s allocation of funding to each of the three main
areas of pipeline safety R&D—Damage Prevention and Leak Detection;
Enhanced Operations, Controls, and Monitoring; and Improved Materials
Performance—has risen significantly, while its allocation to Mapping and
Information Systems efforts has remained level. The tripling of the
agency’s R&D budget—from $2.8 million in fiscal year 2001 to $8.7 million
in fiscal year 2003—has enabled it to increase funding for these three R&D
areas. Specifically, OPS has increased funding for R&D efforts in Damage
Prevention and Leak Detection from $1.3 million in fiscal year 2001 to
$4.0 million in fiscal year 2003, an increase of over 200 percent. The agency
has increased funding for Enhanced Operations, Controls, and Monitoring
from $309,000 in fiscal year 2001 to $1.9 million in fiscal year 2003, an
increase of more than 500 percent. OPS started funding Improved
Materials Performance research in fiscal year 2002, increasing funding in
this area to a level of $1.7 million in fiscal year 2003.



Page 12                                                       GAO-03-746 Pipeline Safety R&D
Agency officials explained to us that they allocated funding to these three
R&D areas in fiscal years 2002 and 2003 based on the results of their 2001
R&D planning workshop.10 For example, they added Improved Materials
Performance to their R&D agenda because it was identified as a priority
area at the workshop. They have also considered other factors in deciding
how to allocate funding. For example, the agency significantly increased
funding for R&D in the areas of Damage Prevention and Leak Detection
and Enhanced Operations, Controls, and Monitoring because of a great
need for improved performance in these areas. OPS officials explained
that, because the agency’s new risk-based regulatory approach requires
pipeline operators to assess and mitigate risks to pipeline segments where
a leak or rupture could have significant consequences, these operators
need better tools and methods for monitoring pipelines and making
necessary repairs. They also noted that OPS’s R&D results assist in the
creation of industry standards on the appropriate use of new technologies.
In addition, officials explained that they decided to allocate a significant
portion of their R&D budget to the area of Improved Materials Performance
because, on the basis of current information on the development of
pipeline technologies, they believed that advances in this area held much
promise for improving pipeline safety.

Finally, OPS has allocated about $1.2 million per year to the Mapping and
Information Systems area since fiscal year 2001 in order to maintain efforts
to improve these systems.11 (See fig. 3.)




10
 Another area of pipeline R&D—the development of technologies to support Arctic and
offshore pipeline operations—was identified as a main area of R&D at OPS’s 2001
workshop. However, the agency did not include this as a main area of funding in its R&D
agenda because it was not identified as a high-priority area at the workshop and because the
Department of the Interior’s MMS funds some R&D in this area. OPS has recently cofunded
with MMS several projects and a workshop in this area, at a cost of almost $148,000.
11
 Figures have been adjusted to account for inflation. They are in constant fiscal year 2003
dollars.




Page 13                                                    GAO-03-746 Pipeline Safety R&D
Figure 3: OPS’s R&D Funding by Area of R&D, Fiscal Years 2001-03




Note: Figures are in constant fiscal year 2003 dollars and represent the agency’s budgeted amounts
for each area. In some cases, OPS spends less than the budgeted amount in a fiscal year. The
agency has been allowed up to 3 years to spend amounts appropriated for R&D.


OPS has provided $3.0 million in funding to 10 projects related to Damage
Prevention and Leak Detection since fiscal year 2001. Examples of funded
projects include the following:

• OPS provided $0.6 million in funding to five projects focused on
  improving in-line inspection techniques, including “smart pigs” and
  other technologies, for detecting damage and defects in pipe walls.12
  Such improved techniques can help to prevent pipeline leaks or ruptures


12
 Smart pigs are devices that run inside a pipeline to detect anomalies, such as corrosion,
metal loss, or damage from excavation.




Page 14                                                         GAO-03-746 Pipeline Safety R&D
     by making possible the early detection and repair of damage and
     defects.

• In partnership with the U.S. Air Force, OPS provided $1.2 million in
  funding to a project focused on developing an approach for detecting
  pipeline leaks using an airborne laser system that measures levels of
  chemicals in the atmosphere just above the earth’s surface.

OPS has provided $0.9 million in funding to six projects related to
Enhanced Pipeline Operations, Controls, and Monitoring since fiscal year
2001. Most of this funding—$0.6 million—has been allocated to two
projects to improve alternative inspection techniques, called direct
assessment, for identifying internal and external corrosion and other
defects in pipelines that cannot accommodate smart pigs.13 This is a
significant issue for natural gas pipelines. One industry association
estimates that only about 35 percent of the total natural gas pipeline
mileage can accommodate smart pigs, which are typically used to assess
the condition of liquid pipelines. OPS officials told us that they are
planning to fund three additional R&D projects in this area in June 2003.

As of May 2003, OPS has provided $0.1 million in funding to one project in
the area of Improved Materials Performance. This project seeks to develop
a “smart” composite pipe that will allow for real-time monitoring of the
condition of the pipe through a remote monitoring system. The agency
requested proposals in this area in December 2002 and expects to start
funding some of these proposals in the summer of 2003. Among the types
of proposals that OPS has requested are proposals to develop

• materials that better withstand third-party damage, corrosion, and
  cracking;

• higher grade/strength steels; and

• materials that facilitate the operation of pipelines at higher design
  pressures.


13
 Direct assessment involves several steps, including digging holes at intervals along a
pipeline to examine suspected problem areas. In a notice of proposed rulemaking, OPS has
proposed integrity management regulations for gas transmission pipelines that would allow
operators to use direct assessment techniques. See 68 Fed. Reg. 4278, 4318 (Jan. 28, 2003).
We have previously reported on the challenges faced by OPS in ensuring that operators use
these techniques appropriately. See GAO-02-785.




Page 15                                                   GAO-03-746 Pipeline Safety R&D
                             Finally, of the roughly $1.2 million that OPS has allocated each year since
                             fiscal year 2001 to the Mapping and Information Systems area, it spent or
                             plans to spend

                             • about $800,000 each year for efforts to improve the National Pipeline
                               Mapping System, which depicts the location of pipelines in relation to
                               areas that are populated or environmentally sensitive, and

                             • about $400,000 each year for efforts to integrate information systems the
                               agency uses in overseeing pipeline safety in cooperation with the states.

                             The agency expects to continue funding this area at this level for the
                             foreseeable future in order to improve and update these systems
                             continually. OPS officials explained that these mapping and information
                             systems assist OPS inspectors and state and local officials in their efforts to
                             oversee pipelines and protect the community and environment from
                             pipeline leaks or ruptures.



Expert Review and            OPS’s mission is to ensure the safe, reliable, and environmentally sound
Coordination Help OPS        operation of the nation’s pipeline transportation system. It has indicated in
                             its budget and plans that its R&D program supports this broad mission as
Align Its R&D Funding with
                             well as the following more specific performance goals: (1) to reduce
Its Mission and Goals        deaths, injuries, property damage, and economic disruptions resulting from
                             pipeline incidents and (2) to reduce the amount of oil and other hazardous
                             liquids spilled from pipelines. The agency has described how new and
                             improved technologies resulting from its R&D funding can help achieve
                             these performance goals. For example, the number of pipeline incidents
                             and the amount of hazardous material spilled could be reduced through the
                             use of improved technologies for detecting third-party damage, corrosion,
                             and defects and the use of improved pipeline materials that can better
                             withstand damage and corrosion.




                             Page 16                                            GAO-03-746 Pipeline Safety R&D
The Committee on Science, Engineering, and Public Policy—a joint
committee of the National Academy of Sciences, the National Academy of
Engineering, and the Institute of Medicine—has recommended the use of
expert review to determine whether a research program is focused on the
subjects most relevant to an agency’s mission.14 Under this form of review,
experts in related fields as well as potential users of the research evaluate
the relevance of research to an agency’s mission and goals and its potential
value to intended users.

OPS has used expert review to help it develop a research agenda that is
aligned with its mission and goals. At its November 2001 R&D planning
workshop, it asked a variety of experts as well as potential users of
research to identify the types of R&D that would be most likely to enhance
pipeline safety. Participants included representatives from federal and
state agencies with pipeline responsibilities, pipeline companies and their
associations, research groups, and technical organizations that set industry
safety standards for pipelines. The agency subsequently used the results
of this workshop in developing its research agenda, guided by an R&D
planning panel composed of key experts from such groups.

OPS has also used peer review, a form of expert review, in deciding which
R&D proposals to fund, a practice that is recommended by the Committee
on Science, Engineering, and Public Policy. OPS’s review panels have
included representatives from other federal agencies that conduct pipeline
R&D, industry associations, and associations of state agencies with
pipeline safety responsibilities.

The Pipeline Safety Improvement Act, enacted in December 2002, requires
that the Secretary of Transportation consult with a variety of groups in
preparing a 5-year plan for pipeline safety R&D, which must be provided to
Congress by December 2003. In response, OPS is continuing to involve
various experts and stakeholders in its R&D planning. Agency officials
have told us that, in preparation for developing this 5-year plan, they are in
the process of obtaining updated external views in order to reassess
research priorities. This has involved participating in the pipeline R&D
planning efforts of industry associations and research organizations,
discussing R&D priorities with state agency officials, and reconvening their


14
 Committee on Science, Engineering, and Public Policy, Evaluating Federal Research
Programs: Research and the Government Performance and Results Act (Washington, D.C.:
National Academy of Sciences, February 1999).




Page 17                                               GAO-03-746 Pipeline Safety R&D
R&D planning panel of outside experts. In developing the plan, agency
officials also plan to consult with OPS’s two technical advisory committees.
Finally, OPS plans to hold another R&D workshop during the winter of
2003-04.

Coordination among federal agencies that conduct related research helps
to avoid duplication and ensure that each agency performs research that is
aligned with its particular mission and goals. The Committee on Science,
Engineering, and Public Policy has recommended that agencies establish a
formal process for coordinating similar fields of research, in order to
improve collaboration, help keep important questions from being
overlooked, and avoid duplication of effort.15 Since 2001, OPS has
increased efforts to coordinate pipeline R&D with DOE and the
Department of the Interior’s MMS, both of which also conduct research
related to pipelines. This increased coordination has taken the form of
mutual participation in panels that review R&D proposals and workshops
to plan R&D activities. According to OPS officials, officials of these
agencies have used these opportunities to communicate about their
respective pipeline R&D efforts and avoid duplication. However, these
agencies have not had a formal mechanism in place that defines each
agency’s responsibilities for pipeline R&D.

The Pipeline Safety Improvement Act requires that the heads of DOT, DOE,
and the National Institute of Standards and Technology develop a
memorandum of understanding to formally coordinate pipeline R&D
efforts. (Although the institute does not operate an R&D program focused
on pipelines, the act assigned it a key role in pipeline R&D based on its
expertise in materials research.) In response, OPS, DOE, and the institute
have developed such a memorandum and are in the process of finalizing
it.16 The Pipeline Safety Improvement Act also requires that DOT
coordinate with DOE and the National Institute of Standards and
Technology in developing a 5-year plan for pipeline R&D. In response, OPS
is involving DOE and the institute, as well as MMS, in efforts to develop


15
 Committee on Science, Engineering, and Public Policy, Evaluating Federal Research
Programs: Research and the Government Performance and Results Act, 11.
16
 DOT and the Department of the Interior have a memorandum of understanding in place to
coordinate their regulatory efforts regarding outer continental shelf pipelines; this
memorandum states that that the two departments will coordinate their respective R&D
projects concerning these pipelines. In addition, OPS and MMS have an interagency
agreement to jointly fund R&D projects related to offshore pipelines.




Page 18                                                 GAO-03-746 Pipeline Safety R&D
                    such a plan. These agencies are also considering holding joint workshops
                    on pipeline R&D in the future. In addition, OPS and the National Institute
                    of Standards and Technology have started to participate in each others’
                    proposal review panels and are discussing entering into an agreement to
                    have the institute conduct some research on pipeline materials.

                    We asked a number of key experts and stakeholders for their views on the
                    extent to which OPS’s R&D agenda is aligned with its mission and goals.
                    These individuals included officials in DOE and MMS, representatives of
                    four industry associations, a former head of a state agency that regulates
                    gas pipelines, the heads of two leading pipeline R&D organizations, two
                    foremost technical experts in pipeline safety, and an environmentalist
                    active in pipeline safety. Six of these individuals have been or are members
                    of OPS advisory committees or R&D planning or review panels. They
                    generally told us that, in their view, the agency has chosen to fund
                    appropriate areas.



Experts Generally   The pipeline safety R&D priorities of the experts who completed our
                    questionnaire are generally consistent with OPS’s R&D priorities. Of the
Support OPS’s R&D   three main R&D areas that OPS is currently funding, Damage Prevention
Priorities          and Leak Detection received the most scores of high or very high funding
                    priority; Enhanced Operations, Controls, and Monitoring received the
                    second highest number of such scores; and Improved Materials
                    Performance received the third highest number. This ranking corresponds
                    to the relative levels of funding OPS has assigned to these areas, as
                    described in the previous section. However, the experts’ level of support
                    for Improved Materials Performance was much lower than that for the
                    other two main R&D areas that OPS is funding. OPS officials told us that
                    they are currently updating their research agenda, using the input of
                    experts and stakeholders, and that they will consider our questionnaire
                    results in this process.

                    To obtain the views of experts on pipeline safety R&D priorities, we asked
                    55 experts to complete a questionnaire indicating the funding priority they
                    would assign to various types of pipeline safety R&D, using categories
                    identified as part of OPS’s 2001 R&D planning workshop. Table 1 provides
                    a description of the main categories of R&D we asked experts to prioritize.
                    The first three categories correspond to the main areas of R&D that OPS is
                    currently funding. Although the fourth category—Arctic and Offshore
                    Technologies—was identified as a main area of pipeline R&D at its
                    workshop, OPS decided not to include it as a main area in its R&D agenda.



                    Page 19                                          GAO-03-746 Pipeline Safety R&D
Agency officials told us that they made this decision because R&D related
to Arctic and Offshore Technologies was not considered to be a high
priority by participants at its workshop and because MMS funds some R&D
in this area and is the primary offshore regulator. We did not include
Mapping and Information Systems—an area that OPS is currently funding
from its R&D budget—as a category for the experts to rate because it was
not identified as a main category of R&D at the 2001 workshop.



Table 1: Major Categories of R&D Related to Pipeline Safety

Category of R&D                            Description
Damage Prevention and                      Develop new technologies to prevent third-party damage,
Leak Detection                             detect pipeline defects, and quickly and accurately locate
                                           and control pipeline leaks.
Enhanced Operations,                       Improve technology for operating, controlling, and
Controls, and Monitoring                   monitoring the integrity of pipelines to help identify and
                                           prioritize pipeline safety problems and solutions.
Improved Materials                         Improve pipeline materials to extend the integrity and
Performance                                lifetime of installed pipelines and their various
                                           components.
Arctic and Offshore                        Develop safer, more cost-effective materials and
Technologies                               procedures to support Arctic and offshore pipeline
                                           applications.
Sources: Materials from OPS’s 2001 R&D planning workshop and other OPS documents related to pipeline safety R&D.


Figure 4 shows how the 49 experts who completed our questionnaire rated
the four categories of pipeline safety R&D. We also asked experts to rate
specific types of R&D within each category. (See app. I for how the experts
rated specific types of R&D within these main categories and for
information on the agency’s funding of these specific types of R&D. See
app. II for information on our methodology for selecting experts and
obtaining their views.)




Page 20                                                                          GAO-03-746 Pipeline Safety R&D
The experts who completed our questionnaire strongly supported the
Damage Prevention and Leak Detection and Enhanced Operations,
Controls, and Monitoring categories of R&D as important areas for OPS to
fund. Ninety-two percent of the experts (45 of 49) indicated that the
Damage Prevention and Leak Detection category should receive high or
very high funding priority.17 Within this category, experts assigned the most
scores of high or very high funding priority to the following types of R&D:
improvements in the ability of in-line inspection tools, such as “smart pigs,”
to detect damage and defects (39 of 49), and the development of new
technologies, such as the innovative application of ultrasonics, that can be
used for inspecting pipelines (38 of 49). Several experts we interviewed
highlighted the need to improve methods for detecting damage to pipelines,
citing the fact that third-party damage is the leading cause of pipeline
accidents. According to both liquid and gas pipeline associations, current
inspection tools cannot reliably detect such damage to pipelines.



Figure 4: Expert Ratings of Categories of Pipeline Safety R&D




Note: Percentages are based on 49 respondents.


17
  Experts assigned a funding priority to each category and specific type of R&D using the
following scale: 1=little or no funding, 2=some funding priority, 3=moderate funding
priority, 4=high funding priority, and 5=very high funding priority. Experts could also
indicate that they did not know or had no basis to judge the funding priority for a particular
R&D category.




Page 21                                                     GAO-03-746 Pipeline Safety R&D
Eighty percent of the experts (39 of 49) indicated that the Enhanced
Operations, Controls, and Monitoring category should receive high or very
high funding priority. Within this category, the type of R&D that received
the most scores of high or very high funding priority (37 of 49) was the
improvement of alternative inspection techniques, called direct
assessment, to identify corrosion and other defects in pipelines that cannot
accommodate in-line inspection devices known as smart pigs. This is a
significant issue for natural gas pipelines because the majority of these
pipelines cannot currently accommodate smart pigs, which are typically
used to assess the condition of liquid pipelines.

In contrast to the experts’ views on the importance of these first two
categories, less than one-third of the experts considered the remaining two
categories of R&D, Improved Materials Performance and Arctic and
Offshore Technologies, to be a high priority for OPS to fund. Thirty-one
percent of the experts (15 of 49) assigned scores of high or very high
funding priority to the Improved Materials Performance category, and 20
percent (10 of 49) assigned such scores to the Arctic and Offshore
Technologies category. However, within the category of Improved
Materials Performance, about half (25 of 49) of the experts indicated that
the type of R&D aimed at developing damage- and defect-resistant
materials should receive high or very high funding priority. Such materials
could be used in the replacement of existing pipe or in the installation of
new pipe. One researcher we interviewed noted that such materials are
particularly important for the gas pipeline industry, which is expanding its
infrastructure in response to increased demands for natural gas. One
industry association estimates that the natural gas industry will need to
install about 49,500 miles of transmission pipeline from 2001 through 2015
to meet increased demand in the United States.




Page 22                                          GAO-03-746 Pipeline Safety R&D
Some differences exist in the views of experts from the following three
subgroups: (1) federal and state government and public interest
organizations, (2) pipeline industry and technical and consulting
organizations, and (3) research organizations.18 As shown in table 2,
experts from all three subgroups generally gave the category of Damage
Prevention and Leak Detection the highest ranking, followed by the
category of Enhanced Operations, Controls, and Monitoring. However,
experts from research organizations considered the categories of Improved
Materials Performance and Arctic and Offshore Technologies to be more
important for OPS to fund than did experts from the other two subgroups.
For example, 70 percent of experts from research organizations (7 of 10)
rated Improved Materials Performance as a high or very high priority
compared with 19 percent of experts from government and public interest
organizations (3 of 16) and 22 percent of experts from pipeline industry and
technical and consulting organizations (5 of 23). In addition, 60 percent of
the researchers (6 of 10) rated Arctic and Offshore Technologies as a high
or very high priority for OPS compared with 19 percent of experts from
government and public interest organizations (3 of 16) and only 4 percent
of experts from pipeline industry and technical and consulting
organizations (1 of 23).




18
  We also examined results for experts from those organizations that had bid on OPS R&D
funding in fiscal year 2002 to see how they compared to those of other experts who
completed our questionnaire. Seven of the experts who completed our questionnaire are
from organizations that had bid on OPS R&D funding within this time frame. Of these, all
seven assigned scores of high or very high funding priority to the Damage Prevention and
Leak Detection category; six assigned such scores to the Enhanced Operations, Controls,
and Monitoring category; three assigned such scores to the Improved Materials
Performance category; and four assigned such scores to the Arctic and Offshore
Technologies category.




Page 23                                                  GAO-03-746 Pipeline Safety R&D
                         Table 2: Views of Experts from Three Subgroups on Pipeline Safety R&D Priorities

                                                            Number of experts who assigned a high or very
                                                                 high funding priority to category
                                                                               Experts from
                                                              Experts from         pipeline
                                                               government      industry and
                                                                and public    technical and     Experts from
                                                                   interest      consulting        research
                                                             organizations    organizations    organizations
                         Category of R&D                               (16)             (23)             (10)
                         Damage Prevention and Leak                     16               20                 9
                         Detection
                         Enhanced Operations, Controls,                 13               18                 8
                         and Monitoring
                         Improved Materials Performance                  3                5                 7
                         Arctic and Offshore Technologies                3                1                 6
                         Source: GAO.


                         An OPS official told us that he believed that researchers rated the Improved
                         Materials Performance category more highly than other experts did
                         because researchers have the best and most current information about the
                         “state of the art” in technology development and are more aware of
                         opportunities in this area. A leading expert from a pipeline research
                         organization noted that the foundation of pipeline R&D has been the
                         development of defect-resistant steels and that, as a consequence,
                         researchers in this area are very interested in R&D that will lead to further
                         improvements in the performance of pipeline materials. He also explained
                         that researchers may have rated the Arctic and Offshore Technologies
                         category more highly than the other types of experts who completed our
                         questionnaire because researchers may be more aware of the need for such
                         R&D to support the construction of new pipelines in these areas in order to
                         reach new energy supplies.



OPS Lacks a              Although OPS has received significant increases in funding for its R&D
                         program in recent years, the agency has not developed a systematic
Systematic Process for   process for evaluating the effectiveness of its R&D program. For example,
Evaluating R&D           the agency tracks and disseminates information on the progress of
                         individual R&D projects but has not developed a process for assessing and
Outcomes                 reporting on the results of its R&D program as a whole. Such a process is
                         needed to demonstrate the program’s progress toward achieving its



                         Page 24                                              GAO-03-746 Pipeline Safety R&D
                            objectives, such as the development and use of new technologies that can
                            improve pipeline safety. OPS has taken some preliminary steps toward
                            developing an evaluation process for its R&D program and could benefit
                            from adopting identified best practices for systematically evaluating the
                            outcomes of federal R&D programs. Leading research organizations, the
                            Office of Management and Budget, and GAO have identified a number of
                            such practices, including setting clear R&D goals and measuring progress
                            toward these goals, using expert review to evaluate the quality of research
                            outcomes, and reporting periodically on evaluation results. The results of
                            evaluations can be used to refocus the direction of R&D programs
                            periodically, as necessary, to ensure that resources are most effectively
                            utilized.



OPS’s Efforts to Evaluate   Although OPS has funded R&D to develop pipeline safety technologies
Research Outcomes Have      since the mid-1990s, the agency’s efforts to evaluate the outcomes of this
                            R&D have been limited and have focused on individual projects. 19 OPS’s
Been Limited                R&D contracts define project goals and require research performers to
                            meet specific milestones for the development of a technology. Contracts
                            also require research performers to report quarterly and at the end of the
                            project on results, including milestones achieved and patents applied for
                            and received. OPS has made some efforts to disseminate the results to date
                            of individual R&D projects. For example, it has started to put “success
                            stories” on its Web site that describe achievements in ongoing projects,
                            such as the development of product prototypes. These success stories help
                            to communicate the results of individual projects to industry and other
                            interested parties.

                            At the program level, OPS has not yet established specific quantifiable
                            goals for its R&D program or a method for measuring progress toward
                            these goals. OPS has indicated, in various planning documents, that its
                            R&D program will help achieve its performance goals of reducing the
                            impacts of pipeline incidents, including fatalities and injuries, and reducing
                            spills of hazardous material. However, agency officials have acknowledged
                            that it is difficult to show the effect of the R&D program on these
                            performance goals. A more immediate objective of the program, according


                            19
                               We have recently reported that RSPA has not fulfilled a DOT requirement for overseeing
                            and developing ways to improve research evaluation efforts throughout the department.
                            See U.S. General Accounting Office, Transportation Research: Action Needed to Improve
                            Coordination and Evaluation of Research, GAO-03-500 (Washington, D.C.: May 1, 2003).




                            Page 25                                                   GAO-03-746 Pipeline Safety R&D
to agency plans, is to promote the transfer of new and improved pipeline
safety technologies to the market in the near term.

In deciding which R&D proposals to fund, OPS gives preference to those
that plan to bring a new product to market within 5 years. In addition,
agency officials told us that OPS plans to promote the use of new
technologies by providing information to potential users and its state
partners about them and, when appropriate, by encouraging their use
through regulation.20 Agency officials told us that the R&D program aims
to have 80 percent of its projects result in products on the market within 5
years. Such an objective is specific and measurable, but OPS has not
formally established it as a goal in any plan or developed a method for
measuring progress toward achieving it. Furthermore, since the agency
has not yet established specific goals or outcome measures for its R&D
program, it does not have a process for documenting and reporting on the
extent to which this program is achieving its goals.

OPS officials explained that they have not yet developed a process for
evaluating the outcomes of the agency’s R&D program because, prior to
2001, the program’s budget was relatively low and, since restructuring the
program in 2001, they have focused program efforts on building a process
for setting research priorities. However, officials do recognize the need for
evaluating R&D outcomes and have taken some preliminary steps toward
developing an evaluation process for their R&D program.




20
 For example, in a notice of proposed rulemaking, OPS proposed integrity management
regulations for natural gas transmission pipelines that would allow pipeline operators to
assess the integrity (structural soundness) of their pipelines using a new technique called
direct assessment. See 68 Fed. Reg. 4278, 4318 (Jan. 28, 2003). OPS has funded and is
currently funding R&D to develop and validate this assessment method.




Page 26                                                     GAO-03-746 Pipeline Safety R&D
OPS is considering some possible measures of the outcomes of its R&D
program as a whole, such as the number of new patents resulting from
R&D efforts. In addition, agency officials told us that, although tracking
the transfer to the market of new pipeline safety technologies can be
challenging, OPS intends to track the use of new technologies in the future
through its process for inspecting operators’ “integrity management”
programs.21 For example, OPS inspectors could document the use of new
or improved technologies by companies to evaluate the condition of their
pipelines. Agency officials noted that the agency will develop inspection
protocols that require inspectors to collect data on the use of new
technologies after their proposed integrity management rule for natural gas
transmission pipelines is finalized.

OPS is also considering the number of documented R&D “success
stories”—summaries of the accomplishments of individual R&D projects—
as a possible measure of program results. However, in previous reviews of
R&D programs operated by other federal agencies, we have found that the
success story approach is selective and does not adequately assess
programwide performance.22

In early June 2003, OPS presented a potential set of performance measures
for its R&D program to its R&D planning panel of outside experts in order
to obtain their views on these measures. This panel includes
representatives of DOE, MMS, the National Institute of Standards and
Technology, pipeline industry associations, state agencies with pipeline
responsibilities, and a key pipeline research organization. OPS intends to
refine its set of measures based on comments received from this panel and
to continue obtaining the views of this panel as it moves forward in
developing an evaluation process for its R&D program.



21
  OPS has issued requirements for hazardous liquid pipeline operators to develop such
programs, which are aimed at assessing the integrity (structural soundness) of pipelines and
identifying and addressing risks to segments where a leak or rupture could have significant
consequences, such as near highly populated areas. See 49 CFR § 195.452. In a notice of
proposed rulemaking, the agency has proposed such requirements for operators of natural
gas transmission pipelines. See 68 Fed. Reg. 4278 (Jan. 28, 2003).
22
 U.S. General Accounting Office, Highway Research: DOT’s Actions to Implement Best
Practices for Setting Research Agendas and Evaluating Outcomes, GAO-03-640T
(Washington, D.C.: Apr. 10, 2003); Highway Research: Systematic Selection and Evaluation
Processes Needed for Research Program, GAO-02-573 (Washington, D.C.: May 24, 2002); and
DOE’s Success Stories Report, GAO/RCED-96-120R (Washington, D.C.: Apr. 15, 1996).




Page 27                                                    GAO-03-746 Pipeline Safety R&D
                          Finally, OPS officials also told us that the agency intends to obtain the
                          views of experts on its R&D outcomes as well as on its future R&D
                          priorities at its next R&D workshop, scheduled for the winter of 2003-04.
                          However, OPS is in the beginning stages of planning this workshop and has
                          not defined a process for using experts’ views to evaluate the outcomes of
                          its R&D program.

                          OPS officials told us that they are considering including information on the
                          effectiveness of the agency’s R&D program in the annual reports to
                          Congress on pipeline R&D that the agency is required to submit, starting in
                          December 2003. The Pipeline Safety Improvement Act requires that DOT,
                          DOE, and the National Institute of Standards and Technology jointly
                          provide these annual reports to Congress, but does not fully specify what
                          types of information should be included in these reports.



Best Practices Help       Since OPS is in the beginning stages of developing an evaluation process
Agencies Systematically   for its R&D program, it could benefit from adopting best practices for
                          systematically evaluating federal R&D programs. Leading organizations
Evaluate Research
                          that conduct scientific and engineering research, the Office of Management
Outcomes                  and Budget, and GAO have identified a number of these best practices.
                          Although the uncertain nature of research outcomes over time can make it
                          challenging to demonstrate the results of such R&D programs, these
                          practices are designed to enable agencies to systematically assess and
                          report on these results regularly in accordance with the Government
                          Performance and Results Act of 1993.23 These assessments can be used to
                          refocus the direction of R&D programs periodically, as necessary, to ensure
                          that resources are most effectively utilized. Identified best practices are
                          discussed in the following sections.




                          23
                           The Government Performance and Results Act requires all federal agencies to measure
                          and report on the results of their activities annually.




                          Page 28                                                 GAO-03-746 Pipeline Safety R&D
Setting Clear, Quantifiable Goals   We have previously reported that, to be effective, any R&D program must
and Measuring Progress toward       be directed toward a clear, measurable goal.24 Such goals help ensure a
These Goals                         direct linkage between R&D program efforts and an agency’s overall
                                    performance goals and mission. Applied research programs, such as OPS’s
                                    R&D program, are directed toward achieving specific useful outcomes,
                                    such as the development of new technologies, which can help accomplish
                                    agency performance goals. The Committee on Science, Engineering, and
                                    Public Policy recommended in a 1999 report that agencies operating
                                    applied research programs measure progress toward practical outcomes
                                    and noted that such measurement can usually be performed annually using
                                    milestones.25

                                    Similarly, in May 2002 the Office of Management and Budget established
                                    investment criteria for federal R&D programs that require these programs
                                    to clearly define goals and track progress toward these goals using
                                    appropriate outcome measures and interim milestones. Indicators that
                                    have been used to measure the outcomes of R&D include the achievement
                                    of specific targets for developing new or improved technologies and patent
                                    applications filed and granted.26 However, measuring research outcomes
                                    can be challenging. For example, outcomes may not occur for a number of
                                    years and may be difficult to track.




                                    24
                                     U.S. General Accounting Office, Research and Development: Lessons Learned from
                                    Previous Research Could Benefit FreedomCAR Initiative, GAO-02-810T (Washington, D.C.:
                                    June 6, 2002).
                                    25
                                     Committee on Science, Engineering, and Public Policy. Evaluating Federal Research
                                    Programs: Research and the Government Performance and Results Act.
                                    26
                                     See U.S. General Accounting Office, Measuring Performance: Strengths and Limitations
                                    of Performance Indicators, GAO/RCED-97-91 (Washington, D.C.: Mar. 21, 1997), and
                                    Intellectual Property: Federal Agency Efforts in Transferring and Reporting New
                                    Technology, GAO-03-47 (Washington, D.C.: Oct. 31, 2002). Also, see Committee on Science,
                                    Engineering, and Public Policy, Evaluating Federal Research Programs: Research and the
                                    Government Performance and Results Act.




                                    Page 29                                                  GAO-03-746 Pipeline Safety R&D
Using Expert Review to Evaluate   In its 1999 report and again in 2001, the Committee on Science,
the Quality of Research           Engineering, and Public Policy recommended the use of expert review,
Outcomes                          supplemented by quantitative methods, to evaluate research regularly.27
                                  Expert review can be a useful addition to performance measures because
                                  of the value of the reviewers’ deep knowledge of the field. Such review can
                                  be performed on a somewhat longer term basis, rather than annually, and
                                  does not require that the final impact of the research be known. Peer
                                  review, a form of expert review, includes an independent assessment of the
                                  technical and scientific merit or quality of research by peers with essential
                                  subject matter expertise and perspective equal to that of the researchers.
                                  In 1999, we reported that some federal agencies, such as the Department of
                                  Agriculture, the National Institutes of Health (NIH), and DOE, use peer
                                  review to help them evaluate the performance of programs and determine
                                  whether to continue or renew research projects.28

                                  The Committee on Science, Engineering, and Public Policy reported in 2001
                                  on the use of expert review, including peer review, by NIH, DOE, the
                                  National Science Foundation, the Department of Defense, and the National
                                  Aeronautics and Space Administration to evaluate the quality of their
                                  research programs. These agencies used varying methods for carrying out
                                  this review, including convening panels of experts who use defined
                                  evaluation processes and obtaining the views of external advisory
                                  committees. The Committee on Science, Engineering, and Public Policy
                                  has also noted that expert evaluation of applied research programs requires
                                  the input of potential users of the results of the research, since the ultimate
                                  usability of these results is an important factor in determining the worth of
                                  the research. Similarly, key experts and stakeholders we interviewed noted
                                  that the degree to which new technologies are actually used would be a
                                  good indication of the effectiveness of OPS’s R&D program. One industry
                                  association representative we interviewed noted that a “constant theme”
                                  raised by pipeline companies is the need for R&D efforts to produce new
                                  technologies that they can actually use in operating their pipelines.




                                  27
                                   Committee on Science, Engineering, and Public Policy, Evaluating Federal Research
                                  Programs: Research and the Government Performance and Results Act, and
                                  Implementing the Government Performance and Results Act: A Status Report
                                  (Washington, D.C.: National Academy of Sciences, 2001).
                                  28
                                   U.S. General Accounting Office, Federal Research: Peer Review Practices at Federal
                                  Science Agencies Vary, GAO/RCED-99-99 (Washington, D.C.: Mar. 17, 1999).




                                  Page 30                                                  GAO-03-746 Pipeline Safety R&D
Reporting Periodically on   Periodic reporting by applied research programs on results can help keep
Evaluation Results          key stakeholders—including oversight organizations and potential users of
                            new technologies—up-to-date on program accomplishments. According to
                            the Committee on Science, Engineering, and Public Policy, applied
                            research programs can usually report annually on progress in meeting
                            milestones. In addition, a retrospective analysis over several years is
                            needed to evaluate outcomes that take more than 1 year to emerge. The
                            committee also has recommended that agencies demonstrate the value of
                            their review processes by publicly describing them to oversight groups, the
                            potential users of research results, and the general public. One expert we
                            interviewed stressed the importance of periodic public reporting by OPS on
                            research goals and outcomes and on the method for evaluating outcomes,
                            in order to disseminate research results and build support for its R&D
                            program.



Conclusions                 OPS has made significant progress in establishing a pipeline safety
                            research agenda that is aligned with its mission and goals and that
                            incorporates the views of experts and stakeholders. However, without a
                            systematic process for evaluating the outcomes of its R&D program, the
                            agency is not able to demonstrate that it is effectively using its increased
                            resources for R&D to foster new and improved technologies that can
                            enhance pipeline safety. Identified best practices for evaluating federal
                            R&D programs—including setting clear quantifiable R&D goals and
                            measuring progress toward these goals, using expert review to evaluate the
                            quality of research outcomes, and reporting periodically on evaluation
                            results—can guide OPS as it moves forward in developing an evaluation
                            process for its program. By following such practices, the agency can help
                            ensure that it develops a systematic evaluation process that will enable it to
                            determine and demonstrate the results of its investment in pipeline safety
                            R&D. OPS could use such an evaluation process to periodically refocus the
                            direction of its program in order to make the most effective use of
                            resources.

                            Furthermore, although the Pipeline Safety Improvement Act’s requirement
                            for annual reports on pipeline R&D, starting in December 2003, does not
                            specify in detail what information should be included in these reports, this
                            requirement provides an opportunity for the agency to keep Congress
                            informed about the results of evaluations of its R&D program. In addition,
                            such reporting, along with other communication methods already in use by
                            the agency, can keep other interested parties—including the pipeline
                            industry, state pipeline safety agencies, pipeline safety advocates, and



                            Page 31                                           GAO-03-746 Pipeline Safety R&D
                      researchers—up-to-date on the program’s progress in advancing the most
                      promising pipeline safety technologies.



Recommendations for   To improve OPS’s ability to demonstrate the effectiveness of its R&D
                      program and make the most effective use of program funds, we
Executive Action      recommend that the Secretary of Transportation direct OPS to

                      • develop a systematic process for evaluating the outcomes of its R&D
                        program that incorporates identified best practices and

                      • include in the annual reports to Congress, which are required by the
                        Pipeline Safety Improvement Act, information on the results of R&D
                        evaluations.



Agency Comments       We provided DOT with a draft of this report for review and comment. DOT
                      officials, including OPS’s Director of Program Development, provided oral
                      comments on the draft on June 13, 2003. The officials generally agreed
                      with the report’s findings and conclusions. They emphasized that they are
                      starting to develop a framework for evaluating the effectiveness of their
                      pipeline safety R&D program and that they intend to finalize this
                      framework by December 2003 by documenting it in the 5-year plan and first
                      annual report on pipeline R&D that DOT is required to submit to Congress,
                      jointly with DOE and the National Institute of Standards and Technology.
                      They also noted that they agree with and intend to implement our
                      recommendations and provided some technical clarifications, which we
                      have incorporated as appropriate.


                      We are sending copies of this report to the Secretary of Transportation, the
                      Administrator of RSPA, RSPA’s Associate Administrator for Pipeline Safety,
                      the Director of the Office of Management and Budget, and appropriate
                      congressional committees. We will make copies available to others upon
                      request. In addition, this report will be available at no charge on the GAO
                      Web site at http://www.gao.gov.




                      Page 32                                          GAO-03-746 Pipeline Safety R&D
If you or your staffs have any questions about this report, please contact
me at (202) 512-2834 or guerrerop@gao.gov. Individuals making key
contributions to this report are listed in appendix III.




Peter Guerrero
Director, Physical Infrastructure Issues




Page 33                                          GAO-03-746 Pipeline Safety R&D
Appendix I

Experts’ Views on R&D Priorities and OPS’s                                                                                              Append
                                                                                                                                             xeis




R&D Funding, by Type of R&D                                                                                                              AppenIx
                                                                                                                                               di




                                                   We asked selected experts to review the following descriptions of specific
                                                   types of pipeline safety research and development (R&D) and assign a
                                                   funding priority to each, based on its importance in achieving the Office of
                                                   Pipeline Safety’s (OPS) mission of ensuring the safe, reliable, and
                                                   environmentally sound operation of the nation’s pipeline transportation
                                                   system. Experts used the following scale: 1=little or no funding, 2=some
                                                   funding priority, 3=moderate funding priority, 4=high funding priority, and
                                                   5=very high funding priority. Experts could also indicate that they did not
                                                   know or had no basis to judge the funding priority for a particular type of
                                                   R&D. The following table shows, for each type of R&D, the number of
                                                   experts who assigned it a high or very high funding priority and OPS’s
                                                   current and planned allocation of funding to it. A total of 49 experts
                                                   completed our questionnaire.




                                                      Number of experts who
                                                        assigned it a high or
                                                          very high funding       OPS’s current and planned allocation of funding
Type of R&D                                                           priority    to this type of R&Da
Damage Prevention and Leak Detection
In-line inspection for damage and defects:                                 39     Allocated $592,500 to five projects in November
Improve in-line inspection techniques,                                            2002 for periods of 9 to 24 months.
including “smart pigs” and other technologies,
for detecting and measuring damage, cracking,
and defects in pipe walls
Nondestructive evaluation: Develop new                                     38     Allocated $500,000 to one project in November
approaches or technologies, such as the                                           2002 for a period of 24 months.
innovative application of ultrasonics, that can
be used for the nondestructive evaluation of
operational pipelines
Real-time monitoring using sensors                                         27     Allocated $182,000 to one project in April 2001 for
attached to pipe: Develop and test real-time                                      period of 12 months. Requested proposals in
sensors applied or attached to the pipe that                                      March 2002 but did not fund any of those received.
can detect possible third-party contact, leaks,                                   Requested additional proposals in December 2002
or other signs of damage                                                          and plans to make funding decisions in July 2003.
Small leak detection: Improve technologies                                 22     Requested proposals in December 2002 and plans
for quickly detecting small pipeline leaks                                        to make funding decisions in July 2003.
Pipe location: Develop better techniques or                                17     Allocated $534,521 to two projects in July 2002 for
materials to locate steel and plastic pipelines,                                  periods of 23 to 24 months.
including determining their depth
Encroachment monitoring using satellites:                                  16     Requested proposals in March 2002 but did not
Develop satellite monitoring for encroachment                                     fund any of those received. Requested additional
and ground movement                                                               proposals in December 2002 and plans to make
                                                                                  funding decisions in July 2003.




                                                   Page 34                                             GAO-03-746 Pipeline Safety R&D
                                                   Appendix I
                                                   Experts’ Views on R&D Priorities and OPS’s
                                                   R&D Funding, by Type of R&D




                                                      Number of experts who
                                                        assigned it a high or
                                                          very high funding            OPS’s current and planned allocation of funding
Type of R&D                                                           priority         to this type of R&Da
Improved directional drilling: Improve                                       14        Requested proposals in March 2002 but did not
directional drilling techniques to avoid                                               fund any of those received. Requested additional
accidental damage to other underground                                                 proposals in December 2002 and plans to make
utilities                                                                              funding decisions in July 2003.
Real-time right-of-way monitoring without                                    12        Requested proposals in March 2002 but did not
pipe contact: Develop fiber optic lines that can                                       fund any of those received. Requested additional
be buried above or alongside pipeline to detect                                        proposals in December 2002 and plans to make
nearby movement                                                                        funding decisions in July 2003.
Airborne chemical mapping: Develop                                           11        Allocated $600,000 to one project in April 2001 for a
approaches using aerial surveillance with                                              period of 12 months. Allocated an additional
optical technologies for right-of-way monitoring                                       $600,000 to this project in April 2002 for an
or other pipeline safety concerns                                                      additional 12 months. Plans to allocate an
                                                                                       additional $600,000 to this project in May 2003 for
                                                                                       an additional 12 months.b
Enhanced Operations, Controls, and Monitoring
Direct assessment: Improve alternative                                       37        Allocated $572,000 to two projects in January 2003
inspection techniques for “unpiggable”                                                 for periods of 12 to 26 months.
pipelines to identify internal and external
corrosion, third-party damage, and other pipe
defects
External corrosion control: Improve                                          30        Allocated $297,000 to one project in January 2003
techniques for characterizing, detecting, and                                          for a period of 26 months.
preventing external corrosion damage
Internal corrosion control: Improve                                          30        Allocated $275,000 to one project in January 2003
techniques for characterizing, detecting, and                                          for a period of 12 months.
preventing internal corrosion damage
Stress corrosion cracking detection:                                         25        Allocated $675,281 to four projects in May and July
Improve techniques for characterizing,                                                 2002 for periods of 12 to 24 months.
detecting, and preventing stress corrosion
cracking
Enhanced repair techniques: Develop                                          25
enhanced repair techniques that can be
implemented without shutdown of pipeline
Risk assessment: Enhance techniques to                                       25        Allocated $97,737 to three projects in May 2002 for
integrate and evaluate risk data to define pipe                                        a period of 12 months. Allocated $70,000 to an
susceptibility to various threats                                                      additional project in January 2003 for a period of 24
                                                                                       months. Requested additional proposals in
                                                                                       December 2002 and plans to make funding
                                                                                       decisions in July 2003.
Pipe strength: Improve methods for                                           17        Requested proposals in December 2002 and plans
characterizing remaining pipe strength                                                 to make funding decisions in July 2003.
Human factors: Study human factors, such as                                   7        Requested proposals in June 2002 but did not fund
operator fatigue, that influence pipeline                                              any of those received. Requested additional
integrity and develop technologies or                                                  proposals in December 2002 and plans to make
procedures to minimize operator error                                                  funding decisions in July 2003.




                                                   Page 35                                                  GAO-03-746 Pipeline Safety R&D
                                                 Appendix I
                                                 Experts’ Views on R&D Priorities and OPS’s
                                                 R&D Funding, by Type of R&D




                                                    Number of experts who
                                                      assigned it a high or
                                                        very high funding            OPS’s current and planned allocation of funding
Type of R&D                                                         priority         to this type of R&Da
Trenchless pipe installation: Develop                                       5
trenchless pipe installation and replacement
techniques, including techniques that use
directional drilling or robotics
Improved Materials Performance
Damage- and defect-resistant materials:                                    25        Requested proposals in December 2002 and plans
Develop materials that better withstand third-                                       to make funding decisions in July 2003.
party damage, corrosion, and cracking
Pipe coatings: Develop enhanced field- and                                 15        Requested proposals in December 2002 and plans
factory-applied coatings, methods for testing                                        to make funding decisions in July 2003.
coatings, and methods to improve coating
choices
Higher grade/strength steels: Develop higher                               14        Requested proposals in December 2002 and plans
grade/strength steels, evaluate their                                                to make funding decisions in July 2003.
performance, and develop methods for
determining when to use them
Welding and joining: Develop enhanced                                      13        Requested proposals in December 2002 and plans
welding and joining techniques and improved                                          to make funding decisions in July 2003.
methods for assessing performance of welds
and joints
Higher design pressure: Develop materials                                  11        Requested proposals in December 2002 and plans
that facilitate pipelines operating at higher                                        to make funding decisions in July 2003.
design pressures and methods for determining
when to use higher pressure designs
Composite pipe: Develop pipe made of, or                                   11        Allocated $98,680 to one project in November 2002
layered with, materials other than steel that                                        for a period of 6 months. Requested additional
may exceed current performance standards or                                          proposals in December 2002 and plans to make
allow greater flexibility or lower cost in                                           funding decisions in July 2003.
challenging installation conditions
Plastic pipe: Develop new or improved plastic                              11        Requested proposals in December 2002 and plans
pipe for local distribution company systems                                          to make funding decisions in July 2003.
Arctic and Offshore Technologies
Leak detection: Develop approaches to                                      17        Allocated $7,781 to one project in May 2002 for a
detect, verify, and respond to leaks                                                 period of 12 months.
Inspection and maintenance procedures:                                      8        Allocated $50,000 to one project in May 2001 for a
Develop alternative inspection and                                                   period of 12 months.
maintenance technologies and procedures
Enhanced performance: Develop materials                                     8        Allocated $59,955 to one project in May 2002 for a
and fabrication techniques to enhance low                                            period of 12 months.
temperature performance
Site evaluation: Develop improved techniques                                5
for site evaluation




                                                 Page 36                                                  GAO-03-746 Pipeline Safety R&D
                                                   Appendix I
                                                   Experts’ Views on R&D Priorities and OPS’s
                                                   R&D Funding, by Type of R&D




                                                       Number of experts who
                                                         assigned it a high or
                                                           very high funding                OPS’s current and planned allocation of funding
Type of R&D                                                            priority             to this type of R&Da
Other Pipeline Safety Improvements
Inspection tools: Evaluate and quantify,                                        28
where possible, the strengths, limits, and
performance of current inspection tools
Pipeline modeling enhancements: Develop                                         24          Requested proposals in December 2002 and plans
better mathematical or computational modeling                                               to make funding decisions in July 2003.
techniques to improve ability to detect defects,
including growth defects and small leaks
Impact of multiple utilities: Characterize                                      18
impact of multiple utilities in common right-of-
way on integrity management practices, such
as cathodic protection
Higher stress levels: Evaluate potential for                                    17
current piping to operate at higher stress levels
Reduction of rupture impact: Explore means                                      15
to reduce the impact of a pipeline rupture and
explosion, for example, through additives to
gas/liquid or enhanced shutoff capability
Impact of past releases: Research the impact                                      7
of past pipeline releases on their surrounding
areas and provide information that could be
used to support local zoning decisions
Sources: OPS data; GAO analysis.

                                                   Note: The pipeline safety R&D categories were identified as part of OPS’s 2001 R&D Planning
                                                   Workshop. The descriptions of the categories are based on materials from this workshop as well as
                                                   OPS’s 2002 and 2003 announcements soliciting R&D proposals. The information on OPS’s funding of
                                                   each category is based on GAO’s analysis of information provided by OPS.
                                                   a
                                                    This column describes OPS’s funding of R&D projects, by category, in fiscal years 2001 and 2002 and
                                                   planned allocation of funding in fiscal year 2003. Some projects that are applicable to more than one
                                                   category of R&D appear more than once.
                                                   b
                                                    In conference reports accompanying appropriations bills for fiscal years 2001, 2002, and 2003,
                                                   Congress expressed its intent that the Research and Special Programs Administration devote
                                                   $600,000 of its pipeline safety R&D budget to this project in each of these fiscal years.




                                                   Page 37                                                          GAO-03-746 Pipeline Safety R&D
Appendix II

Scope and Methodology                                                                          AppenIx
                                                                                                     di




              To perform our work, we reviewed Office of Pipeline Safety (OPS)
              documentation on its research and development (R&D) funding and
              analyzed this information to identify trends; reviewed pertinent legislation
              and agency documents pertaining to the R&D program; and interviewed
              OPS officials regarding their R&D funding, agenda-setting processes, and
              processes for evaluating the outcomes of their R&D program. We also
              interviewed key experts and stakeholders concerning OPS’s management
              of its R&D program, including the alignment of the agency’s research
              agenda with its mission and goals, and their views on R&D priorities and
              gaps. These individuals included officials of the Department of Energy
              (DOE) and the Department of the Interior’s Minerals Management Service
              (MMS) who are responsible for pipeline R&D; representatives of pipeline
              industry associations and leading pipeline research organizations; and
              several key experts in pipeline safety. Also, we identified best practices for
              evaluating the outcomes of federal R&D through a review of relevant
              literature and compared the agency’s processes with these practices.

              To determine the views of experts on pipeline safety R&D priorities, we
              sought to identify experts considered to be very knowledgeable about the
              development of new pipeline safety technologies or pipeline safety issues.
              To identify appropriate experts, we obtained recommendations on
              individuals to contact from key organizations, contacted those individuals,
              and obtained further recommendations from them on additional
              individuals to contact. We identified initial individuals to contact through
              prior work on pipeline safety issues or through recommendations from
              OPS. These initial contacts included officials in DOE and MMS,
              representatives of four industry associations, a former head of a state
              agency that regulates gas pipelines, the heads of two leading pipeline R&D
              organizations, two technical experts in pipeline safety, and an
              environmentalist active in pipeline safety. Six of these individuals have
              been or are members of OPS advisory committees or R&D planning or
              review panels. We obtained recommendations from these individuals on
              experts who could provide us with views on pipeline safety R&D priorities.

              We based our final selection of experts on the criteria of knowledge,
              balance, and independence. We considered indications of their extent of
              knowledge of pipeline safety R&D, as evidenced by the number of times
              they had been recommended, their participation in OPS’s R&D planning
              and review activities, or other relevant factors. We included individuals
              from a variety of groups in order to achieve a balanced representation of
              experts, including some who are relatively independent of OPS and the
              pipeline industry. We included individuals from federal and state agencies,



              Page 38                                           GAO-03-746 Pipeline Safety R&D
Appendix II
Scope and Methodology




pipeline safety advocacy groups, industry associations, pipeline
companies, technical and consulting organizations, and research
organizations. We also provided our list of identified experts to the
National Academy of Sciences and OPS for their review and comment.

We contacted 55 individuals whom we had identified as appropriate
experts for our review and asked them to complete a questionnaire
indicating their views on pipeline safety R&D priorities. Forty-nine
individuals responded, for an 89 percent response rate. Our results
pertaining to experts’ views on R&D priorities represent the views of only
the experts who responded to our questionnaire. In a number of cases,
these individuals collaborated with others in their organizations in
completing their questionnaires. Listed below are the organizational
affiliations of experts who completed our questionnaire. 1

Government and Public Interest Organizations

Federal Agencies
Federal Energy Regulatory Commission
Minerals Management Service, Department of the Interior
National Institute of Standards and Technology, Department of Commerce
National Transportation Safety Board
Office of Fossil Energy, Department of Energy

State Agencies and Associations
National Association of Pipeline Safety Representatives
National Association of Regulatory Utility Commissioners
New York State Department of Public Service
Railroad Commission of Texas
Virginia State Corporation Commission
Washington Utilities and Transportation Commission

Pipeline Safety Advocacy Groups
Common Ground Alliance
Cook Inlet Keeper
Safe Bellingham




1
Two of the individuals who responded are former officials of these organizations.




Page 39                                                   GAO-03-746 Pipeline Safety R&D
Appendix II
Scope and Methodology




Pipeline Industry and Technical/Consulting Organizations

Industry Associations
American Gas Association
American Petroleum Institute
Association of Oil Pipelines
Interstate Natural Gas Association of America
Offshore Operators Committee

Pipeline Companies
BP Pipelines, North America
ConocoPhillips
CMS Panhandle Companies
Duke Energy
El Paso Corporation
Enbridge Pipelines
Enron
Explorer Pipeline Company
ExxonMobil Pipeline Company
KeySpan Energy
Peoples Energy
Shell Pipeline Company

Technical/Consulting Organizations
Accufacts, Inc.
Batten and Associates, Inc.
Duckworth Pipeline Integrity Services, Inc.
HSB Solomon
Kiefner and Associates, Inc.
National Association of Corrosion Engineers

Research Organizations

Advantica, Inc.
Battelle
CFER Technologies
Edison Welding Institute
Gas Technology Institute
Ohio State University, Fontana Corrosion Center
Pipeline Research Council International, Inc.
Southwest Research Institute




Page 40                                         GAO-03-746 Pipeline Safety R&D
Appendix II
Scope and Methodology




Texas A&M University, Department of Mechanical Engineering
University of Florida, Department of Chemical Engineering

In the questionnaire, we asked respondents to review descriptions of
various main categories of pipeline safety R&D as well as specific types of
R&D within these main categories and indicate what funding priority they
would assign to each.2 (See table 1 for descriptions of the main R&D
categories. See app. I for descriptions of the types of R&D within these
main categories.) We based the R&D categories and descriptions on
materials prepared as part of an R&D planning workshop held by OPS in
2001, in which a variety of experts and stakeholders participated; on
announcements the agency subsequently issued soliciting proposals for
R&D in various areas; and on other OPS documents related to pipeline
safety R&D.

We compiled the scores obtained from the questionnaires to produce a
ranking of R&D priorities representing the views of the experts who
completed our survey. We also analyzed our results to determine whether
any differences existed in the responses of experts from the three
subgroups: government and public interest organizations, industry and
technical and consulting organizations, and research organizations. In
addition, we identified organizations that had bid on R&D funding from
OPS in fiscal year 2002 and conducted a separate analysis of the responses
of experts from these organizations to determine how they compared with
those of other experts who completed our questionnaire. Seven of the
experts who completed our questionnaire are from organizations that had
bid on OPS R&D funding within this time frame. 3

We conducted our work from January 2003 through June 2003 in
accordance with generally accepted government auditing standards.

2
 Experts assigned a funding priority to each category and subcategory of R&D using the
following scale: 1=little or no funding, 2=some funding priority, 3=moderate funding
priority, 4=high funding priority, and 5=very high funding priority. We also asked
respondents if they wished to identify any additional R&D categories and, if so, what score
they would assign to these categories.
3
 Of the 49 experts who completed our questionnaire, we identified 7 from organizations that
had submitted R&D proposals in response to announcements issued by OPS in March and
June 2002, based on information provided by OPS. Of these 7, 5 were from organizations
that received funding from OPS. In December 2002, OPS issued another announcement
soliciting R&D proposals. However, because OPS has not yet made funding decisions about
proposals received in response to this announcement, officials preferred not to provide us
with information about these proposals.




Page 41                                                    GAO-03-746 Pipeline Safety R&D
Appendix III

GAO Contacts and Staff Acknowledgments                                                       Appen
                                                                                                 Ix
                                                                                                  di




GAO Contacts      Peter Guerrero, (202) 512-2834
                  Susan Fleming, (202) 512-4431



Acknowledgments   In addition to those named above, Sharon Dyer, Etana Finkler, Judy
                  Guilliams-Tapia, Brandon Haller, Bert Japikse, Nancy Kingsbury, Donna
                  Leiss, Gary Stofko, Ron Stouffer, and Stacey Thompson made key
                  contributions to this report.




(545025)          Page 42                                       GAO-03-746 Pipeline Safety R&D
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300
Address Service Requested