United States General Accounting Office GAO Testimony Before the Committee on Science, Subcommittee on Environment, Technology and Standards, House of Representatives For Release on Delivery Expected at 10:00 a.m. EDT Thursday, April 10, 2003 HIGHWAY RESEARCH DOT's Actions to Implement Best Practices for Setting Research Agendas and Evaluating Outcomes Statement of Katherine Siggerud, Acting Director Physical Infrastructure Team GAO-03-640T This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. April 10, 2003 HIGHWAY RESEARCH DOT's Actions to Implement Best Highlights of GAO-03-640T, a report to House Committee on Science, Management Practices for Setting Subcommittee on Environment, Technology and Standards Research Agendas and Evaluating Outcomes Improvement and innovation based Leading organizations, federal agencies, and experts that conduct scientific on highway research have long and engineering research use best practices designed to ensure that research been important to the highway objectives are related to the areas of greatest interest to research users and system. The Federal Highway that research is evaluated according to these objectives. Of the specific best Administration (FHWA) is the practices recommended by experts—such as the Committee on Science, primary federal agency involved in highway research. Throughout the Engineering, and Public Policy and the National Science Foundation—GAO past decade, FHWA received identified the following practices as particularly relevant for FHWA: (1) hundreds of millions of dollars for developing research agendas in consultation with external stakeholders to its surface transportation research identify high-value research and (2) using a systematic approach to evaluate program, including nearly half of research through such techniques as peer review. the Department of Transportation’s approximate $1 billion budget for FHWA’s processes for developing its research agendas do not always research in fiscal year 2002. Given consistently include stakeholder involvement. External stakeholder the expectations of highway involvement is important for FHWA because its research is to be used by research and the level of resources others that manage and construct transportation systems. FHWA dedicated to it, it is important to acknowledges that its approach for developing research agendas lacks a know that FHWA is conducting high quality research that is systematic process to ensure that external stakeholders are involved. In relevant and useful. In May 2002, response to GAO’s recommendation, FHWA has drafted plans that take the GAO issued a report on these necessary steps toward developing a systematic process for involving issues and made recommendations external stakeholders. While the plans appear responsive to GAO’s to FHWA, which the agency agreed recommendation, as shown in the table below, GAO cannot evaluate their with, aimed at improving its effectiveness until they are implemented. processes for setting research agendas and evaluating its research FHWA does not have a systematic process that incorporates techniques such efforts. as peer review for evaluating research outcomes. Instead, the agency primarily uses a “success story” approach to communicate about those GAO was asked to testify on (1) research projects that have positive impacts. As a result, it is unclear the best practices for developing research agendas and evaluating extent to which all research projects have achieved their objectives. FHWA research outcomes for federal acknowledges that it must do more to measure the performance of its research programs; (2) how research program, however, it is still in the process of developing a FHWA's processes for developing framework for this purpose. While FHWA’s initial plans appear responsive research agendas align with these to GAO’s recommendation, GAO cannot evaluate their effectiveness until best practices; and (3) how they are implemented. FHWA's processes for evaluating FHWA’s Response to GAO’s Recommendations research outcomes align with these Does FHWA have plans Has FHWA best practices. that are responsive to implemented the GAO’s Recommendation the recommendation? recommendation? Develop a systematic approach for obtaining input from external stakeholders in ● ○ determining the research program’s agendas. Develop a systematic process for evaluating significant ongoing and completed research that includes peer review or other best ● ○ practices in use at federal research agencies. www.gao.gov/cgi-bin/getrpt? GAO-03-640T. Develop specific plans for implementing these To view the full report, including the scope recommendations, including time frames and ◒ ◒ and methodology, click on the link above. estimates of their costs. For more information, contact Katherine Siggerud at (202) 512-2834 or Source: GAO. email@example.com Note: ●=yes; ◒=partially; and ○=no. Mr. Chairman and Members of the Committee: We appreciate the opportunity to testify today on the Federal Highway Administration’s (FHWA) surface transportation research and technology program. Change, improvement, and innovation based on highway research have long been important to the highway system. While this research is a shared responsibility among FHWA, state departments of transportation, and private organizations, we focused on FHWA’s important leadership role as the primary federal agency involved in highway research. Throughout the past decade, FHWA has received hundreds of millions of dollars for its surface transportation research and technology program, including nearly half of the Department of Transportation’s (DOT) approximate $1 billion budget for research, development, and technology in fiscal year 2002. Given the important expectations of highway research and the significant level of resources dedicated to it, it is important for the Congress and the American people to know that the agency is conducting research that is relevant and useful to stakeholders and that is of high quality. In May 2002 we issued a report on these issues and made recommendations to FHWA, which the agency agreed with, aimed at improving its processes for setting research agendas and evaluating its research efforts.1 As it considers reauthorizing FHWA’s research and technology program, Congress will be making decisions about the structure of the program. Accordingly, my testimony today will discuss (1) best practices for developing research agendas and evaluating research outcomes for federal research programs; (2) the extent to which FHWA’s processes for developing research agendas align with the best practices for similar federal research programs; and (3) the extent to which FHWA’s processes for evaluating research outcomes align with these best practices. My statement is based in part on our May 2002 report, which focused primarily on those activities funded by the surface transportation research and technology deployment funding categories identified in the Transportation Equity Act for the 21st Century. In developing that report, we held discussions with FHWA officials and reviewed relevant program documents, legislation, and publications on best practices in federal research from the Transportation Research Board, the Committee on Science, Engineering, and Public Policy and others. In preparing for this 1 Highway Research: Systematic Selection and Evaluation Processes Needed for Research Program (GAO-02-573, May 2002). Page 1 GAO-03-640T Highway Research hearing, we also updated FHWA’s activities in response to our findings and recommendations. • Leading organizations that conduct scientific and engineering research, In summary: other federal agencies with research programs, and experts in research and technology have identified and use best practices designed to ensure that research objectives are related to the areas of greatest interest and concern to research users and that research is evaluated according to these objectives. Specific best practices in these areas used in other federal research programs or recommended by experts—such as the Committee on Science, Engineering, and Public Policy2, the Environmental Protection Agency, the National Science Foundation, and the Office of Management and Budget—include: (1) developing research agendas in consultation with external stakeholders to identify high-value research and (2) using a systematic approach to evaluate ongoing and completed research through such techniques as peer review. • As we reported last year, FHWA’s processes for developing research agendas for its research and technology program do not always consistently include stakeholder involvement. External stakeholder involvement is important for FHWA because its research is expected to be used by others, such as state departments of transportation, which manage and construct transportation systems. FHWA acknowledges that its approach for developing research agendas lacks a consistent, transparent, and systematic process to ensure that external stakeholders are involved. Instead, the agency expects each program office to determine how or whether to involve external stakeholders in the agenda setting process. As a result, this approach is used inconsistently. To improve its program and in response to our recommendations, FHWA has drafted plans that seem to take the necessary steps toward developing a systematic process for involving external stakeholders in the agenda setting process. FHWA’s plans have not been finalized, and we cannot comment on the potential effectiveness of these plans. • We reported last year that FHWA does not have a systematic process that incorporates techniques such as peer review for evaluating research 2 Committee on Science, Engineering, and Public Policy, Evaluating Federal Research Programs: Research and the Government Performance and Results Act (Washington, DC: Feb. 1999). The Committee on Science, Engineering, and Public Policy is a joint committee of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. Page 2 GAO-03-640T Highway Research outcomes. Instead, the agency primarily uses a “success story” approach to evaluate and communicate its research outcomes. While this approach illustrates some benefits of the agency’s research, it cannot be used as the primary method to evaluate the outcomes of the research against intended results because these stories represent only a fraction of the program’s completed research projects. As a result, it is unclear whether the organization is selecting research projects that have the highest potential value, or the extent to which these projects have achieved their objectives. We recommended that FHWA develop a systematic approach to evaluating its research program, and noted peer review as a best practice for doing so. In response, FHWA agreed that the agency must do a better job to measure the performance of its research and technology program. However, currently it is still in the process of developing, defining, and adopting a framework for measuring performance. Therefore, we cannot yet comment on FHWA’s efforts to evaluate research outcomes. FHWA is the DOT agency responsible for federal highway programs— Background including distributing billions of dollars in federal highway funds to the states—and developing federal policy regarding the nation’s highways. The agency provides technical assistance to improve the quality of the transportation network, conducts transportation research, and disseminates research results throughout the country. FHWA’s program offices conduct these activities through its Research and Technology Program, which includes “research” (conducting research activities), “development” (developing practical applications or prototypes of research findings), and “technology” (communicating research and development knowledge and products to users). FHWA maintains a highway research facility in McLean, Virginia. This facility, known as the Turner-Fairbank Highway Research Center, has over 24 indoor and outdoor laboratories and support facilities. Approximately 300 federal employees, on-site contract employees, and students are currently engaged in transportation research at the center. FHWA’s research and technology program is based on the research and technology needs of each of its program offices such as the Offices of Infrastructure, Safety, or Policy. Each of the program offices is responsible for identifying research needs, formulating strategies to address transportation problems, and setting goals for research and technology activities that support the agency’s strategic goals. (See Appendix I for examples of research that these offices undertake.) One program office that is located at FHWA’s research facility provides support for administering the overall program and conducts some of the research. The Page 3 GAO-03-640T Highway Research agency’s leadership team, consisting of the associate administrators of the program offices and other FHWA offices, provides periodic oversight of the overall program. In 2002 FHWA appointed the Director of its Office of Research, Development, and Technology as the focal point for achieving the agency’s national performance objective of increasing the effectiveness of all FHWA program offices, as well as its partners and stakeholders, in determining research priorities and deploying technologies and innovation. In addition to the research activities within FHWA, the agency collaborates with other DOT agencies to conduct research and technology activities. For example, FHWA works with DOT’s Research and Special Programs Administration to coordinate efforts to support key research identified in the department’s strategic plan.3 Other nonfederal research and technology organizations also conduct research funded by FHWA related to highways and bridges. Among these are state research and technology programs that address technical questions associated with the planning, design, construction, rehabilitation, and maintenance of highways. In addition, the National Cooperative Highway Research Program conducts research on acute problems related to highway planning, design, construction, operation, and maintenance that are common to most states. Private organizations, including companies that design and construct highways and supply highway-related products, national associations of industry components, and engineering associations active in construction and highway transportation, also conduct or sponsor individual programs. Universities receive funding for research on surface transportation from FHWA, the states, and the private sector. 3 st As required by the Transportation Equity Act for the 21 Century, DOT annually develops the department wide “Research, Development, and Technology Plan.” This plan, drafted by the Research and Special Programs Administration and funded in part by FHWA, provides program-level detail on the directions that DOT’s research will take. This plan is used by the individual operating administrations, such as FHWA and the Research and Special Programs Administration, as a resource document to develop their subsequent program proposals for inclusion in their administration budgets. Page 4 GAO-03-640T Highway Research Leading organizations that conduct scientific and engineering research, Research Community other federal agencies with research programs, and experts in research Promotes Use of Best and technology have identified and use best practices for developing research agendas and evaluating research outcomes. Although the Practices for uncertain nature of research outcomes over time makes it difficult to set Developing Research specific, measurable program goals and evaluate results, the best practices we identified are designed to ensure that the research objectives are Agendas and related to the areas of greatest interest and concern to research users and Evaluating Research that research is evaluated according to these objectives. These practices Outcomes include (1) developing research agendas through the involvement of external stakeholders and (2) evaluation of research using techniques such as expert review of the quality of research outcomes. Developing Research External stakeholder involvement is particularly important for FHWA Agendas Through the because its research is expected to improve the construction, safety, and Involvement of External operation of transportation systems that are primarily managed by others, such as state departments of transportation. According to the Stakeholders Transportation Research Board’s Research and Technology Coordinating Committee,4 research has to be closely connected to its stakeholders to help ensure relevance and program support, and stakeholders are more likely to promote the use of research results if they are involved in the research process from the start.5 The committee also identified merit review of research proposals by independent technical experts based on technical criteria as being necessary to help ensure the most effective use of federal research funds. In 1999, we reported that other federal science agencies—such as the Environmental Protection Agency and the National Science Foundation—used such reviews to varying degrees to assess the merits of competitive and noncompetitive research proposals.6 In April 4 The Research and Technology Coordinating Committee was convened in 1991 by the Transportation Research Board of the National Academies to provide a continuing, independent assessment of FHWA’s research and technology program. FHWA provides funding for the committee. 5 Transportation Research Board, The Federal Role in Highway Research and Technology (Washington, D.C.: National Academy Press, 2001), p. 76. For surface transportation research, potential stakeholders include state and local highway agencies that own and operate the nation’s highways; highway users; the companies that furnish the products, services, and equipment needed to build, operate, and maintain the highway system; and the people and communities that benefit from and are affected by the system. 6 Federal Research: Peer Review Practices at Federal Science Agencies Vary (GAO/RCED-99-99, Mar. 1999), p. 2. Page 5 GAO-03-640T Highway Research 2002, the Office of Management and Budget issued investment criteria for federal research and technology program budgets that urge these agencies to put into place processes to assure the relevance, quality and performance of their programs. For example, the guidance requires these programs to have agendas that are assessed prospectively and retrospectively through external review to ensure that funds are being expended on quality research efforts. Evaluation of Research The Committee on Science, Engineering, and Public Policy reported in Using Systematic 1999 that federal agencies that support research in science and Approach to Review the engineering have been challenged to find the most useful and effective ways to evaluate the performance and results of the research programs Quality of Research they support. Nevertheless, the committee found that research programs, Outcomes no matter what their character and goals, can be evaluated meaningfully on a regular basis and in accordance with the Government Performance and Results Act. Similarly, in April 2002 the Office of Management and Budget issued investment criteria for federal research and technology program budgets that require these programs to define appropriate outcome measures and milestones that can be used to track progress toward goals and assess whether funding should be enhanced or redirected. In addition, program quality should be assessed periodically in relation to these criteria through retrospective expert review. The Committee on Science, Engineering, and Public Policy also emphasized that the evaluation methods must match the type of research and its objectives, and it concluded that expert or peer review is a particularly effective means to evaluate federally funded research. Peer review is a process that includes an independent assessment of the technical and scientific merit or quality of research by peers with essential subject area expertise and perspective equal to that of the researchers. Peer review does not require that the final impact of the research be known. In 1999, we reported that federal agencies, such as the Department of Agriculture, the National Institutes of Health, and the Department of Energy, use peer review to help them (1) determine whether to continue or renew research projects, (2) evaluate the results of research prior to publication of those results, and (3) evaluate the performance of programs and scientists.7 In its 1999 report, the Committee on Science, Engineering, and Public Policy also stated that expert review is widely used to evaluate: 7 GAO/RCED-99-99. Page 6 GAO-03-640T Highway Research (1) the quality of current research as compared with other work being conducted in the field, (2) the relevance of research to the agency’s goals and mission, and (3) whether the research is at the “cutting edge.” Although FHWA engages external stakeholders in elements of its research External and technology program, the agency currently does not follow the best Stakeholders’ practice of engaging external stakeholders on a consistent and transparent basis in setting its research agendas. The agency expects each program Involvement in office to determine how or whether to involve external stakeholders in the Developing FHWA’s agenda setting process. As we reported in May 2002, FHWA acknowledges that its approach to preparing research agendas is inconsistent and that Research Agendas the associate administrators of FHWA’s program offices primarily use Has Been Limited input from the agency’s program offices, resource centers, and division offices.8 Although agency officials told us that resource center and division office staff provide the associate administrators with input based on their interactions with external stakeholders, to the extent that external stakeholder input into developing research agendas occurs, it is usually ad hoc and provided through technical committees and professional societies. For example, the agency’s agenda for environmental research was developed with input from both internal sources (including DOT’s and FHWA’s strategic plans and staff) and external sources (including the Transportation Research Board’s reports on environmental research needs and clean air, environmental justice leaders, planners, civil rights advocates, and legal experts). In our May 2002 report we recommended that FHWA develop a systematic approach for obtaining input from external stakeholders in determining its research and technology program’s agendas. FHWA concurred with our recommendation and has taken steps to develop such an approach. FHWA formed a planning group consisting of internal stakeholders as well as representatives from the Research and Special Programs Administration and the Pennsylvania Department of Transportation to determine how to implement our recommendation. This planning group prepared a report analyzing the approaches that four other federal agencies are taking to involve external stakeholders in setting their research and technology program agendas. Using the lessons learned from reviewing these other agencies’ activities, FHWA has drafted a Corporate Master Plan for 8 FHWA has 4 resource centers throughout the country, and division offices in each state, Puerto Rico and District of Columbia. Page 7 GAO-03-640T Highway Research Research and Deployment of Technology & Innovation. Under the draft plan, the agency would be required to establish specific steps for including external stakeholders in the agenda setting process for all areas of research throughout the agency’s research and technology program by fiscal year 2004. In drafting this plan, FHWA officials obtained input from internal stakeholders as well as external stakeholders, including state departments of transportation, academia, consultants, and members of the Transportation Research Board. It appears that FHWA has committed to taking the necessary steps to adopt the best practice of developing a systematic process for involving external stakeholders in the agenda setting process. The draft plan invites external stakeholders to assist FHWA with such activities as providing focus and direction to the research and technology program and setting the program’s agendas and priorities. However, because FHWA’s plan has not been finalized, we cannot comment on its potential effectiveness in involving external stakeholders. As we reported last year, FHWA does not have an agency wide systematic FHWA Lacks a process to evaluate whether its research projects are achieving intended Systematic Approach results that uses such techniques as peer review. Although the agency’s program offices may use methods such as obtaining feedback from to Evaluating customers and evaluating outputs or outcomes versus milestones, they all Research Outcomes use success stories as the primary method to evaluate and communicate research outcomes. According to agency officials, success stories are examples of research results adopted or implemented by such stakeholders as state departments of transportation. These officials told us that success stories can document the financial returns on investment and nonmonetary benefits of research and technology efforts. However, we raised concerns that success stories are selective and do not cover the breadth of FHWA’s research and technology program. In 2001, the Transportation Research Board’s Research and Technology Coordinating Committee concluded that peer or expert review is an appropriate way to evaluate FHWA’s surface transportation research and technology program. Therefore, the committee recommended a variety of actions, including a systematic evaluation of outcomes by panels of external stakeholders and technical experts to help ensure the maximum return on investment in research. Agency officials told us that increased stakeholder involvement and peer review will require significant additional expenditures for the program. However, a Transportation Research Board official told us that the cost of obtaining expert assistance could be relatively low because the time needed to provide input would be Page 8 GAO-03-640T Highway Research minimal and could be provided by such inexpensive methods as electronic mail. In our May 2002 report, we recommended that FHWA develop a systematic process for evaluating significant ongoing and completed research that incorporates peer review or other best practices in use at federal agencies that conduct research.9 While FHWA has concurred that the agency must measure the performance of its research and technology program, it has not developed, defined or adopted a framework for measuring performance. FHWA’s report on efforts of other federal agencies that conduct research, discussed above, analyzed the approaches that four other federal agencies are taking to evaluate their research and technology programs using these best practices. According to FHWA’s assistant director for Research, Technology, and Innovation Deployment, the agency is using the results of this report to develop its own systematic approach for evaluating its research and technology program. However, this official noted that FHWA has been challenged to find the most useful and effective ways to evaluate the performance and results of the agency’s research and technology program. According to FHWA’s draft Corporate Master Plan for Research and Deployment of Technology & Innovation, FHWA is committed to developing a systematic method of evaluating its research and technology program that includes the use of a merit review panel. This panel would conduct evaluations and reviews in collaboration with representatives from FHWA staff, technical experts, peers, special interest groups, senior management, and contracting officers. According to the draft plan, these merit reviews would be conducted on a periodic basis for program-level and agency-level evaluations, while merit reviews at the project level would depend on the project’s size and complexity. FHWA is still in the process of developing, defining, and adopting a framework for measuring performance. Therefore, we cannot yet comment on how well FHWA’s efforts to evaluate research outcomes will follow established best practices. Mr. Chairman, this concludes my prepared statement. I would be pleased to answer any questions that you or Members of the Committee may have. 9 GAO-02-573. Page 9 GAO-03-640T Highway Research For further information on this testimony, please contact Katherine Contacts and Siggerud at (202) 512-2834 or firstname.lastname@example.org. Deena Richart made key Acknowledgments contributions to this testimony. Page 10 GAO-03-640T Highway Research Appendix I: Roles of Program Offices in FHWA’s Research and Technology Program FHWA’s research and technology program is based on the research and technology needs of each of its program offices such as the Offices of Infrastructure, Safety, and Policy. Each of the program offices is responsible for identifying research needs, formulating strategies to address transportation problems, and setting goals for research and technology activities that support the agency’s strategic goals. (See table 1.) Table 1: Roles of Program Offices in Research and Technology Program office name Role in research and technology Examples of research and technology projects Federal Lands Highway Development of applied research and Road Surface Analyzer (ROSAN) measurement of technology applicable to transportation pavement smoothness. systems serving federal lands. Infrastructure Development of research and technology in Long-term pavement performance. the areas of highway construction and physical maintenance, pavements, and Concrete research and technology. structures. Innovative bridge technology. Operations Development of research and technology Research into advanced traffic simulation modeling. program plans for the Intelligent Transportation Systems program, as well as Prediction tools and research into advanced, adaptive operation of the transportation system and traffic signal control strategies. management of freight transportation. Analysis of critical intermodal freight corridors and facilities. Work zone best practices guide and program support. Planning and Environment Development of research and technology in Workshops, synthesis materials, and case studies of the areas of planning, environment, and state consultation practices with rural officials. property acquisition. Statewide planning and travel forecasting training. Research on the contribution of transportation to air pollution and on strategies to reduce transportation effects. Highway noise barrier design handbook. Policy Development of analytical tools and data National personal transportation survey. systems for policy development and studies; conducting analysis and studies to support Highway cost allocation study. the formulation of transportation policy and legislative initiatives; and preparation of Production of biennial report, “Status of the Nation’s major reports to Congress on highway policy Highways, Bridges, and Transit: Condition and issues. Performance.” Safety Leading in development of research and Interactive highway safety design model for two-lane technology activities in the areas of roads. Intersections; Pedestrian and Bicyclist Safety; Roadside Safety; Run-Off-Road Pedestrian safety countermeasure selection system. Safety; and Speed Management. Page 11 GAO-03-640T Highway Research Program office name Role in research and technology Examples of research and technology projects Education and community programs for pedestrian/bicyclist safety. Analysis of intersection safety issues. Red-light running prevention. Speed limit setting and enforcement. Variable speed limits. Research, Development, Support of all other business units in the Research activities to support Infrastructure, and Technology development and delivery of new Operations, and Safety units. technologies. Source: GAO’s presentation of information provided by FHWA. (542023) Page 12 GAO-03-640T Highway Research
Highway Research: DOT's Actions to Implement Best Practices for Setting Research Agendas and Evaluating Outcomes
Published by the Government Accountability Office on 2003-04-10.
Below is a raw (and likely hideous) rendition of the original report. (PDF)