United States General Accounting Office GAO Testimony Before the Subcommittee on Technology, Committee on Science, House of Representatives For Release on Delivery Expected at MEASURING 10 a.m. EDT Thursday PERFORMANCE April 10, 1997 Challenges in Evaluating Research and Development Statement by Allen Li, Associate Director, Energy, Resources, and Science Issues, Resources, Community, and Economic Development Division GAO/T-RCED-97-130 Madame Chair and Members of the Subcommittee: We are pleased to be here today to discuss our work related to two programs at the National Institute of Standards and Technology (NIST): the Advanced Technology Program (ATP) and the Manufacturing Extension Partnership (MEP) Program. In addition, we will be discussing our work performed in response to the Subcommittee’s request regarding research and development (R&D) performance measures. This work resulted in our report entitled Performance Measurement: Strengths and Limitations of Research Indicators (GAO/RCED-97-91), which is being released today in conjunction with this hearing. Our report on measuring R&D performance highlights the difficulty in measuring the impact of technology programs such as ATP and MEP. Our work on these programs was performed in response to specific congressional concerns. In the case of ATP, we were asked to assess the impact of the program. In the case of MEP, we obtained manufacturers’ views on (1) the impact of the services provided by the program, as well as other manufacturing extension programs, on their business performance and (2) the factors affecting the impact of these services. In the case of our report on performance measurement, this Subcommittee asked us to evaluate various indicators that are used to measure the results of R&D. In summary, our work showed that: • The Advanced Technology Program has funded research projects that would have been funded by the private sector as well as those that would not.1 The award recipients were nearly evenly divided when asked if they would have pursued their projects if they had not received such funding. We also found that in most cases, the participants in our survey did not look for funding from other sources, private or public, before trying to obtain funding from the Advanced Technology Program. About half of the 45 applicants that tried to find funding elsewhere before turning to the Advanced Technology Program were told by prospective funders that their projects were either too risky or “precompetitive”2—characteristics that fulfill the aims of the Advanced Technology Program. • Manufacturers viewed the manufacturing extension programs’ services positively, as was demonstrated in our national survey of manufacturers 1 Measuring Performance: The Advanced Technology Program and Private Sector Funding (GAO/RCED-96-47, Jan. 11, 1996). 2 Precompetitive refers to the stage during R&D at which a preliminary assessment of a technology’s commercial potential can be made but before commercial prototypes are developed. Page 1 GAO/T-RCED-97-130 who had received substantive services from the programs in 1993.3 Most manufacturers responding to our questionnaire—about 73 percent—reported that they believed that the type of assistance they had received from these programs had positively affected their overall business performance. About 15 percent of the respondents reported that they believed the programs’ assistance had not affected their overall business performance.4 • The amount of money spent on research and development, the primary indicator of research investment, is useful as an input measure of how much research is being performed. However, the level of spending is not a reliable indicator of research results. We found that there is no primary indicator of research and development results. The companies that we spoke with collect data on various output indicators, such as return on investment and patents granted, but in general make limited use of them in their investment decisions. Instead, they emphasized that research and development contributes directly to the bottom line. Because companies are profit oriented, many of the indicators tracked by the private sector cannot be directly applied to the federal government. Determining the specific outcomes resulting from federal research and development is a challenge that will not be easily resolved. However, in response to recent legislation requiring agencies to report on program results, some progress is being made in measuring the impacts of research. ATP’s mission is to stimulate economic growth in the United States through Background technology development. The program seeks to accomplish that mission by sharing the cost of R&D projects with private industry. The projects selected by ATP for funding are characterized by the program as having “a potential broad-based economic impact but a relatively high technical risk and a long time horizon.” ATP’s program guidance has stated that if the technical risk associated with a project is very low, federal funding should not be necessary. In addition, when submitting a research proposal, applicants must sign a form stating that “this proposal is not requesting funding for existing or planned research programs that would be conducted in the same time period in the absence of financial assistance under the ATP.” This wording suggests that ATP should not fund projects that other sources would have funded or, 3 Manufacturing Extension Programs: Manufacturers’ Views About Delivery and Impact of Services (GAO/GGD-96-75, Mar. 14, 1996) and Manufacturing Extension Programs: Manufacturers’ Views of Services (GAO/GGD-95-216BR, Aug. 7, 1995). 4 Of the remaining 12 percent, 8 percent of the respondents said that it was too early to determine if there was an effect, while 4 percent said they had no basis to estimate an effect. Page 2 GAO/T-RCED-97-130 when ATP does fund such projects, that ATP funds should enable applicants to complete their projects in a shorter time. Manufacturing extension programs offer manufacturers assistance in modernizing or upgrading their operations, often with state and federal funding. NIST manages federal funding of this type of program through its Manufacturing Extension Partnership Program, or MEP. In our prior reports, we used MEP to collectively refer to all state, federal, and university manufacturing extension programs.5 The primary mission of manufacturing extension programs is to give “hands-on” technical assistance to small- and medium-sized manufacturers trying to improve their operations through the use of appropriate technologies.6 These programs engage in a variety of activities to assist small- and medium-sized manufacturers, often in partnership with other business assistance providers, such as Small Business Development Centers, community colleges, and federal laboratories. The programs offer a wide range of business services, including helping companies (1) solve individual manufacturing problems, (2) obtain training for their workers, (3) create marketing plans, and (4) upgrade their equipment and computers. The assistance focuses on small- and medium-sized manufacturers because research by the National Research Council and others has indicated that these companies lack the resources necessary to improve their manufacturing performance. In our work on ATP, our objective was to examine, as one way to assess the The Advanced program’s impact, whether research projects would have been funded by Technology Program the private sector if they had not received funds from ATP.7 We also examined ATP’s impact in terms of other goals of the program, such as aiding the formation of joint ventures. We focused on two groups of ATP applicants, which we called “winners” and “near winners.” Both groups submitted proposals that were rated highest during ATP’s review, but the near winners did not ultimately receive ATP funding. We surveyed all applicants that qualified as winners or near 5 Manufacturing Extension Programs (GAO/GGD-96-75, Mar. 14, 1996) and Manufacturing Extension Programs (GAO/GGD-95-216BR, Aug. 7, 1995). 6 The Small Business Administration generally defines a small business as having fewer than 500 employees. Some experts have further divided small manufacturers into small firms with fewer than 100 employees and medium-sized firms with from 100 to 499 employees. 7 Measuring Performance (GAO/RCED-96-47, Jan. 11, 1996). Page 3 GAO/T-RCED-97-130 winners during ATP’s first 4 years (1990-93). We achieved a 100-percent response rate from the 123 respondents that we included in our analysis (89 winners and 34 near winners). We found that ATP had funded research projects that would have been funded by the private sector as well as those that would not. The winners were nearly evenly divided when asked if they would have pursued their projects even if they had not received ATP funding. Half of the near winners continued their projects without relying on ATP funding, while the other half discontinued their projects for various reasons. Almost all the near winners that continued their projects did so on a modified schedule, meeting the projects’ milestones later than they had scheduled in their proposals to ATP. Of the 123 applicants we surveyed, 77,8 or 63 percent, did not look for funding from other sources before requesting it from ATP. Those applicants that did look for funding looked for a long time and made many attempts to find funding, on average. Seven applicants turned down offers from private sources because they could not reach an acceptable funding arrangement. We also found that ATP had other effects. More than three-fourths of the joint-venture applicants indicated that they had come together solely to pursue an ATP project, thus satisfying ATP’s goal of serving as a catalyst for the formation of joint ventures. Furthermore, of the 45 applicants that tried to find funding elsewhere before turning to ATP, about half were told by prospective funders that their projects were either too risky or precompetitive—characteristics that fulfill the aims of ATP funding. We surveyed 766 U.S. manufacturers that had completed at least 40 hours Manufacturing of manufacturing extension program assistance, including NIST’s MEP, and Extension Program received 551 responses.9 We obtained respondents’ views on the impact of Services these services on their business performance and on the factors affecting the impact of these services. We did not verify either the positive or negative impacts reported by manufacturers, nor did we evaluate the operations or management of specific federal or state programs. We also 8 One applicant did not know. 9 Thirteen of these manufacturing extension programs received NIST funding in fiscal year 1994. These 13 programs accounted for 36 percent of the 551 total respondents to our survey. See Manufacturing Extension Programs (GAO/GGD-96-75, Mar. 14, 1996) and Manufacturing Extension Programs (GAO/GGD-95-216BR, Aug. 7, 1995). Page 4 GAO/T-RCED-97-130 obtained the views of other manufacturers that had little or no experience with these programs to determine why they made little or no use of them. Most manufacturers responding to our questionnaire—about 73 percent—reported that they believed the programs’ assistance had positively affected their overall business performance. About 15 percent of the respondents reported that they believed the assistance had not affected their overall business performance. Approximately 8 percent said that it was too early to determine the effect, and another 4 percent said they had had no basis to estimate the effect. In addition, most respondents reported that the assistance had positively affected their use of technology in the workplace (about 63 percent), the quality of their product (about 61 percent), and the productivity of their workers (about 56 percent). Between about 44 percent and 63 percent of the respondents reported that the programs’ assistance had positively affected certain specific indicators of their business performance, such as customer satisfaction, profits, and the ability to meet production schedules. Of those respondents not reporting a positive impact on specific indicators of their business performance, most said the programs’ assistance had not had any impact. Two percent or fewer of the respondents reported a negative impact on each specific performance indicator. Among the factors that manufacturers said had affected the impact of MEP services was their own companies’ input. The companies that had committed their own financial resources to implement the programs’ recommendations reported greater benefits from the assistance relative to other survey respondents. Of those 322 respondents who had made a financial investment, 86 percent said that the programs’ assistance had positively affected their business performance. However, 54 percent of those who had not made a financial investment also reported an overall positive impact. Other factors, according to the respondents, that influenced the effectiveness of the programs’ services were the expertise and experience of the programs’ staff and the affordability of the assistance. In our related telephone survey of 200 additional manufacturers who were not extensive users of the programs’ services, about 82 percent reported that they had not used the services because they were unaware of these programs. About 10 percent said that although they knew about these programs, they had not used them because they believed the assistance Page 5 GAO/T-RCED-97-130 would not be necessary. The companies we interviewed said that other sources of modernization assistance besides these programs were their customers, vendors and/or suppliers, industry associations, and consultants. The report we are releasing today on performance measurement shows Measuring the that there is no single indicator or evaluation method that adequately Impacts of Federal captures the results of R&D. However, indicators do provide helpful Research Programs information for making decisions about R&D. Whether the focus is on basic research, applied research, or development, the amount of money spent in that area is taken as an indication of how much research is being performed. The major advantages of using expenditure data as an indicator are that they are easily understandable, readily available, and have been, in general, consistently gathered over time. In addition, spending on different projects in different research areas can be measured according to the same unit, dollars, making comparisons between projects straightforward. The amount of funding, however, does not provide a good indication of research results. Companies told us that they are switching their spending to more short-term R&D projects rather than long-term projects. However, the impacts of that change are unclear. The reduced funding levels for long-term projects may not reflect the fact that the R&D efforts can be performed with greater efficiency. For example, one way in which the federal government and the private sector have tried to use R&D resources more efficiently and effectively is through consortia with universities or other companies. By combining their research activities, companies attempt to avoid expensive duplication and learn from each other. We also found that because of the difficulties in identifying the impacts of research, quantitative and qualitative indicators have been developed as proxies to assess R&D results. The strengths and limitations are evident in both types of indicators. Quantitative indicators focus mainly on return on investment, patenting rates, and bibliometrics—the study of publication-based data. While implying a degree of precision, these indicators were not originally intended to measure long-term R&D results. Qualitative assessment such as peer review provides detailed information, but it relies on the judgments of experts and may be expensive. Because of these difficulties, the companies we interviewed stressed marketplace results rather than R&D output indicators. While varying in Page 6 GAO/T-RCED-97-130 the types of indicators they collect, they emphasized the difficulties in measuring R&D’s specific contribution to a company’s overall performance. For example, one company stated that because so many people have been involved in a product’s evolution, it is difficult to separate the contribution of the research unit from that of other units. All of the companies interviewed have increased their expectation that R&D contribute directly to their profitability. However, instead of increasing their efforts at measuring R&D results, they have shifted the responsibility for making R&D decisions to the business units. For example, if the business units believe that a particular R&D project would increase their profits, the firm would budget for that R&D. Many of the R&D output measures tracked by the private sector do not apply directly to the federal government. In particular, while facing the same increasing cost pressures as the private sector, the federal government cannot rely on the profit motive to guide its decisions. This discussion of performance measures for R&D is particularly relevant because of the current emphasis on the Government Performance and Results Act (GPRA). In response to questions about the value and effectiveness of federal programs, GPRA seeks to shift federal agencies’ focus away from such traditional concerns as staffing, activity levels, and tasks completed toward a focus on program outcomes. GPRA incorporates performance measurement as one of its most important features. Under this act, executive branch agencies are required to develop annual performance plans that use performance measurement to reinforce the connection between the long-term strategic goals outlined in their strategic plans and the day-to-day activities of their managers and staff. However, the very nature of the innovative process makes measuring the performance of science-related projects difficult. For example, a wide range of factors determines if and when a particular R&D project will result in commercial or other benefits. It can also take many years for a research project to achieve results. Experiences from recent GPRA pilot efforts reinforce the fact that output measures are highly specific to the management and mission of each federal agency and that no single indicator exists to measure the results of research. The Army Research Laboratory, which was designated as a pilot project for performance measurement under the act, has developed a multifaceted approach using quantitative indicators, peer review, and customer feedback to evaluate the results of R&D. Although this is not the only approach that can be taken, this response to the challenges in Page 7 GAO/T-RCED-97-130 measuring the impacts of research shows that some progress is being made in response to GPRA. Madame Chair, this concludes my prepared remarks. I would be happy to respond to any questions you may have. (141046) Page 8 GAO/T-RCED-97-130 Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Measuring Performance: Challenges in Evaluating Research and Development
Published by the Government Accountability Office on 1997-04-10.
Below is a raw (and likely hideous) rendition of the original report. (PDF)