United States General Accounting Office GAO Report to Congressional Requesters March 1997 MEASURING PERFORMANCE Strengths and Limitations of Research Indicators GAO/RCED-97-91 United States GAO General Accounting Office Washington, D.C. 20548 B-275241 March 21, 1997 The Honorable Constance A. Morella Chairwoman The Honorable Bart Gordon Ranking Minority Member Subcommittee on Technology Committee on Science House of Representatives The Honorable John S. Tanner House of Representatives This report responds to your request for information on the indicators used to evaluate the results of research and development (R&D). The report discusses the relative strengths and limitations of the input and output indicators used by the federal and private sectors to measure the results of R&D. The report also provides a historical perspective on research spending. As agreed with your offices, we plan no further distribution of this report until 30 days from its date of issue, unless you publicly announce its contents earlier. We will then send copies to interested parties, and we will also make copies available to others upon request. If you have any questions, I can be reached at (202) 512-3600. Major contributors to this report are listed in appendix II. Allen Li Associate Director, Energy, Resources, and Science Issues Executive Summary American taxpayers invested more than $60 billion in federal funds in Purpose military and civilian research and development (R&D) efforts in 1996. The private sector invested more than $110 billion that same year. The technological advancements resulting from these efforts are a critical factor in improving the productivity of American workers and, correspondingly, the nation’s standard of living. However, while the contribution of R&D to technological advancement is widely recognized, there is no widely accepted method of measuring the results of that research. To facilitate discussions of the adequacy of the funding and of the results of the R&D, the Subcommittee on Technology, House Committee on Science, asked GAO to evaluate the various indicators that are used to measure the results of R&D. Specifically, this report discusses the strengths and limitations of the input and output indicators used by the federal and private sectors to measure the results of R&D. This report also provides a historical perspective on spending for research. The commitment to reduce the federal deficit is forcing the Congress to Background reexamine the value of programs across the federal government. Although scientific research is often considered to be intrinsically valuable to society, there is pressure on all federal agencies, including science agencies, to demonstrate that they are making effective use of the taxpayers’ dollars. This greater emphasis on results is evident in the passage of the Government Performance and Results Act of 1993 (GPRA). The act fundamentally seeks to shift the focus of federal management and accountability from a preoccupation with staffing, activity levels, and tasks completed to a focus on results—that is, the real difference that federal programs make in people’s lives. The experts in research measurement have tried for years to develop indicators that would provide a measure of the results of R&D. However, the very nature of the innovative process makes measuring the performance of science-related projects difficult. For example, a wide range of factors determine if and when a particular R&D project will result in commercial or other benefits. It can also take many years for a research project to achieve results. The amount of money spent on research and development, the primary Results in Brief indicator of the investment in research, is useful as a measure of how Page 2 GAO/RCED-97-91 Measuring Performance Executive Summary much research is being performed. Having been refined over many years, these data are generally available for the research efforts in both the public and private sectors. However, the level of spending is not a reliable indicator of the level of results achieved by research. Unlike the situation with the input measures of research and development, there is no primary indicator of the outputs. Output indicators include quantitative analyses of return on investment, patents granted, and other outputs as well as qualitative assessments based on peer review. The companies that GAO spoke with collect data on various output indicators but, in general, make limited use of them in their investment decisions. Instead, the companies emphasized that research and development contribute directly to their “bottom line.” Because companies are profit-oriented, many of the indicators tracked by the private sector cannot be directly applied to the federal government. Experiences from pilot efforts made under the Government Performance and Results Act have reinforced the finding that output measures are highly specific to the management and mission of each federal agency and that no single indicator exists to measure the results of research. Principal Findings Funding Indicates Funding has been used as the primary input indicator for decades. Research Activity but Does Whether a policymaker is interested in basic research, applied research, or Not Measure the Results of development, the amount of money spent in that area is taken as an indication of how much research is being performed. The major Research advantages of using expenditure data as an indicator are that they are easily understandable, readily available, and have been, in general, consistently gathered over time. In addition, spending on different projects in different research areas can be measured according to the same unit, dollars, making comparisons between projects straightforward. The amount of funding, however, does not provide a good indication of the results of research. Companies told GAO that they are focusing more of their spending on short-term R&D projects than on long-term projects. However, the impacts of that change in focus are unclear. The reduced funding levels may not reflect the fact that the R&D efforts are being performed with greater efficiency. For example, one way in which the federal government and the private sector have tried to use R&D resources Page 3 GAO/RCED-97-91 Measuring Performance Executive Summary more efficiently and effectively is through consortia with universities or other companies. By combining their research activities, the companies attempt to avoid expensive duplication and learn from each other. R&D Output Indicators Because of the difficulties in identifying the impacts of research, Can Provide Limited decisionmakers have developed quantitative and qualitative indicators as Information About the proxies to assess the results of R&D activity. The strengths and limitations are evident in both types of indicators. The current quantitative indicators Results of R&D focus mainly on return on investment, patenting rates, and bibliometrics—the study of published data. While implying a degree of precision, these indicators were not originally intended to measure the long-term results of R&D. Qualitative assessment provides detailed information, but it relies on the judgments of experts and may be expensive. Because of these difficulties, the companies interviewed by GAO stressed marketplace results rather than R&D output indicators. While varying in the types of indicators they collect, they emphasized the difficulties in measuring R&D’s specific contribution to a company’s overall performance. For example, one company stated that because so many people have been involved in a product’s evolution, it is difficult to separate the contribution of the research unit from that of other units. All of the companies interviewed have increased their expectation that R&D contribute directly to their profitability, but instead of increasing their efforts at measuring R&D results, they have shifted the responsibility for R&D decisions to the business units. However, many of the R&D output measures tracked by the private sector do not apply directly to the federal government. In particular, while facing the same increasing cost pressures as the private sector, the federal government cannot rely on the profit motive to guide its decisions. The GPRA requires the executive agencies to develop their annual R&D plans with suitable performance measures. The Research Roundtable, a group of federal researchers and managers representing a cross-section of R&D departments and agencies, warned about the difficulties of quantifying the results of R&D and the potential for incorrect application with subsequent harm to scientific endeavors. The Army Research Laboratory, which was designated a pilot project for performance measurement under the act, has developed a multifaceted approach using quantitative indicators, peer review, and customer feedback to evaluate the results of R&D. Page 4 GAO/RCED-97-91 Measuring Performance Executive Summary This report contains no recommendations. Recommendations Because this report focuses broadly on the R&D of both the federal and Agency Comments private sectors, and not on the effort of individual agencies, GAO did not submit a draft of this report to federal agencies for their review and comment. Page 5 GAO/RCED-97-91 Measuring Performance Contents Executive Summary 2 Chapter 1 8 The Process of Innovation and the Use of R&D Indicators 8 Introduction Past Efforts to Evaluate R&D Impacts 10 The Government Performance and Results Act 11 Objectives, Scope, and Methodology 12 Chapter 2 14 R&D Spending Gives an Indication of Innovative Effort 14 R&D Spending Data R&D Spending Is Not a Good Indicator of R&D Results 15 Provide Some Spending Patterns Show a Greater Emphasis on Short-Term 17 Research Information About Innovative Activity but Not About R&D Results Chapter 3 18 Current Indicators Have Strengths and Limitations 18 R&D Output The Private Sector’s Emphasis on Marketplace Results Limits 22 Indicators Can Lessons for the Federal Government Federal Science Agencies Are Still Exploring Ways to Measure 26 Provide Limited the Impacts of R&D Information About R&D Results Appendixes Appendix I: Historical Perspective on Research Spending 28 Appendix II: Major Contributors to This Report 34 Figure Figure I.1: U.S. R&D Spending, 1953-96 30 Page 6 GAO/RCED-97-91 Measuring Performance Contents Abbreviations ARPA Advanced Research Projects Agency DOD Department of Defense GAO General Accounting Office GPRA Government Performance and Results Act IRI Industrial Research Institute NIH National Institutes of Health NSF National Science Foundation NASA National Aeronautics and Space Administration OECD Office of Economic Cooperation and Development OTA Office of Technology Assessment R&D research and development Page 7 GAO/RCED-97-91 Measuring Performance Chapter 1 Introduction Over $180 billion was spent on research and development (R&D) in the United States in 1996.1 Most of that amount was spent by industry and the federal government—$113 billion and $62 billion, respectively; the balance was spent by universities and other nonprofit organizations. The leading experts in the study of research indicators agree that R&D has a significant, positive effect on economic growth and the overall standard of living. However, because of the complexity of linking the results of R&D to its economic impacts, there is no widely accepted method of measuring the results of R&D spending in either industry or the federal government. The commitment to reduce the federal budget deficit is forcing the Congress and the executive branch to undertake a basic reexamination of the value of programs across the federal government. It is also placing pressure on all federal agencies, including the civilian science agencies, to clearly demonstrate that they are making effective use of the taxpayers’ dollars. The Government Performance and Results Act of 1993 (GPRA) provides a legislative vehicle for the agencies to use as they seek to demonstrate and improve their effectiveness. Equally important, if successfully implemented, GPRA should help the Congress make the difficult funding, policy, and program decisions that the current budget environment demands. Researchers have developed their own terminology for describing the The Process of process of transforming R&D into economic results. In its simplest form, Innovation and the the theory underlying both public and private decision-making has been Use of R&D Indicators that innovative activity positively affects economic performance. Innovation can be thought of as the development and application of a new product, process, or service. It can include the use of an existing product in a new application or the development of a new device for an existing application. Innovation encompasses many activities, including scientific, technical, and market research; product, process, or service development; and manufacturing and marketing to the extent that they support the dissemination and application of the invention. Innovation is a combination of invention and commercialization. Invention describes the initial conception of a new product, process, or service, but not the act of putting it to use. Commercialization refers to the attempt to profit from innovation through the sale or use of new products, processes, and services. 1 These figures are based on preliminary 1996 statistics reported by R&D performers to the National Science Foundation. Page 8 GAO/RCED-97-91 Measuring Performance Chapter 1 Introduction This description as well as the traditional views of innovation has been strongly influenced by the linear model of innovation, which says that innovation proceeds sequentially through the stages of basic research, applied research, development, manufacturing, and marketing.2 This model assumes that basic research serves as the source of innovation and that new scientific knowledge initiates a chain of events culminating in the development and sale of a new product, process, or service. In this model, basic research is the most uncertain part of the process; once basic research is conducted, innovation and commercialization can proceed. The model suggests that the firms with the best technology will likely be the first to market and win the lion’s share of profits. The simplicity of this model makes it particularly useful in policy discussions. Other models may be more accurate, but they provide a more complex explanation of the relationship between science and the commercialization of new technology. These models, such as the “chain-linked model,” include feedback loops that allow for interaction among the different stages of the linear model. These models also reflect the fact that the ideas for new inventions or changes to existing products often arise from the recognition of new market opportunities, advances in manufacturing capabilities, or advances in technology independent of progress in the underlying science. Real-world examples show that technological breakthroughs can precede as well as follow basic research. In many cases, science is not the source of innovation. The Wright brothers, for example, developed the first airplane without an understanding of aerodynamic theory, and Chester Carlson developed the first xerographic copier without a thorough understanding of photoconductive materials. These inventions have resulted in considerable research into aerodynamic theory and materials science, respectively, as scientists and engineers attempted to improve the original invention. In studying complex processes or concepts such as innovation, it is not always possible to measure them directly. As a result, researchers turn to the use of “indicators.” Indicators point to or illustrate the process or 2 The National Science Foundation uses the following definitions in its resource surveys: Basic research has as its objective to gain a more comprehensive knowledge or understanding of the subject under study, without specific applications in mind. Applied research is aimed at gaining knowledge or understanding to determine the means by which a specific, recognized need may be met. Development is the systematic use of the knowledge or understanding gained from research directed toward the production of useful materials, devices, systems, or methods. The Foundation recognizes the limitations of this classification scheme but continues to use these categories to maintain historical consistency, among other reasons. Page 9 GAO/RCED-97-91 Measuring Performance Chapter 1 Introduction concept in question but do not directly measure it. For example, in order to determine an object’s temperature, one can use a thermometer to measure it directly. However, when trying to measure something as complex as the health of a country’s economy, one relies on different indicators, such as unemployment rates, stock market averages, or trade balances that do not provide a direct measurement of economic health but do give an indication of its status. In the case of innovation, various “input” and “output” indicators are used that are based on the linear model of innovation. The following chapters of this report are broken down according to these two sets of indicators. Chapter 2 is concerned with R&D spending, or expenditure data, which is the most widely used input indicator of innovation, and chapter 3 focuses on some widely used output indicators. For almost two decades, numerous reports have documented the Past Efforts to difficulties of quantifying the results of R&D. As noted above, the Evaluate R&D identification of the economic and social effects of research involves Impacts complex issues of measurement, analysis, and interpretation. The following efforts may help to give perspective to the present concerns about measuring R&D results. • In 1979, we reported on a wide range of factors that make the measurement of R&D results difficult.3 The report noted that R&D expenditures are undertaken for a variety of reasons. Some attempt to develop new knowledge; others are directed at meeting needs, such as national defense, for which there is no commercial market; and still others are directed at lowering the cost of products. Furthermore, some projects produce revolutions; others produce nothing. Most important, R&D is only one input into a complex process. Thus, we concluded that there is no possibility of eliminating the role that judgment plays in the allocation of federal R&D resources. • In 1986, the Office of Technology Assessment (OTA) issued a detailed report that questioned the utility of the effort to quantify R&D returns.4 According to OTA, the fundamental stumbling block to placing an economic value on federal R&D is that improving productivity or producing an economic return is not the primary justification for most federal R&D programs. The report added that the attempts to measure the economic return to federal R&D are flawed because many of the research outputs, 3 Assessing the “Output” of Federal Commercially Directed R&D (GAO/PAD-79-69, Aug. 1979). 4 Research Funding as an Investment: Can We Measure the Returns? Office of Technology Assessment, 72 pp. (Apr. 1986). Page 10 GAO/RCED-97-91 Measuring Performance Chapter 1 Introduction such as national defense, cannot be assigned an economic value. The report also noted that in industry, where one might expect quantitative techniques to prevail because of the existence of a well-defined economic objective, OTA found a reliance on subjective judgment and good communications between R&D, management, and marketing staffs. • In 1996, responding to concerns about the implementation of GPRA, the National Science and Technology Council issued a report that stressed the limited role of quantification in measuring the results of R&D.5 It stated that the insufficiency of quantitative measures per se is one reason why other sources of evidence, such as merit review of past performance, narrative discussion, and descriptions of outstanding accomplishments and more typical levels of achievement, should be included in annual performance reports. The report concluded that the cornerstone of world-class science will continue to be merit review with peer evaluation, while the science community works to develop the measurement tools and other methods needed to assess the contributions of fundamental science. • At the international level, the Organization of Economic Cooperation and Development (OECD) is also grappling with these questions. Beginning with the first edition of the Standard Practice for Surveys of R&D (the “Frascati Manual”) in the 1960s, OECD has been developing international frameworks for the measurement of R&D inputs. In connection with its more recent effort to develop new output indicators (the “Oslo Manual”), OECD stated that the new indicators and the underlying statistics usually take two or more decades to reach general acceptance and regular collection and publication. The participants at a 1996 OECD conference extensively discussed the organization’s efforts to improve the quality of innovation indicators. At this stage, these indicators appear to be most useful in helping researchers study and describe the process of innovation. In response to questions about the value and effectiveness of federal The Government programs, GPRA seeks to shift federal agencies’ focus away from such Performance and traditional concerns as staffing, activity levels, and tasks completed Results Act toward a focus on program outcomes—that is, the real difference the federal program makes in people’s lives. In the GPRA context, an “outcome measure” assesses the results of a program activity compared to its intended purpose. An “output measure,” according to GPRA, tabulates, calculates, or records the level of activity or effort and can be expressed in a quantitative or qualitative manner. The output indicators discussed in our report could be considered either outcome measures or output measures, depending on the context. For example, return-on-investment 5 Assessing Fundamental Science, National Science and Technology Council (July 1996). Page 11 GAO/RCED-97-91 Measuring Performance Chapter 1 Introduction calculations could be outcome measures in a business context since gaining the maximum return on investment is the intended purpose of a business. Patenting rates would be output measures in some businesses because patents could serve as one measure of the level of activity of a research unit. The act recognizes how difficult it is to state the goals and measure the results of some programs. While the law encourages the use of objective measures of performance, it authorizes agencies—with the approval of the Office of Management and Budget—to use alternative, subjective measures of performance. Also, instead of having GPRA take effect immediately after its passage in 1993, the Congress allowed for a period of time for the government to learn how to implement the act. As part of the learning process, the act called for the use of pilot projects in performance measurement. Among the approximately 70 agencies or parts of agencies that participated in pilot projects, one addresses scientific research. Upon the full implementation of the act, the executive branch agencies are required to devise plans that are outcome-oriented. The act calls for the agencies to develop three documents: a strategic plan, an annual performance plan, and an annual performance report. In response to the Subcommittee on Technology’s request, our objective Objectives, Scope, was to review various indicators that are used to measure the results of and Methodology R&D. Specifically, this report discusses the relative strengths and limitations of the input and output indicators used by the federal and private sectors to measure the results of R&D as well as the claim that industry focuses on short-term profitability rather than long-term R&D needs. This report also provides a historical perspective on research spending. (See app. I.) The impacts of innovation are widely studied in the public and private sectors as well as in academia. Our work relied on a limited number of experts in each of these sectors to provide us with the prevailing understanding of and latest developments in the area. As a result, our review does not provide an exhaustive examination of R&D measures nor does it answer the question of how R&D should be measured. It does, however, as agreed with your offices, consist of information on the strengths and limitations in the use of these indicators as well as anecdotal information based on interviews with leading R&D companies. We also reviewed the relevant literature. Page 12 GAO/RCED-97-91 Measuring Performance Chapter 1 Introduction We interviewed a number of experts, including a former IBM Vice President for Research and National Science Foundation (NSF) Director, a former Executive Director of the Manufacturing Forum, a former Associate Director of the White House Office of Science and Technology Policy, the Director of the Special Projects Office for the Army Research Laboratory, and the former Chief Financial Officer for Apple Computer. We also conducted six teleconferences with company officials who were typically at the director level. We interviewed representatives of General Electric, Lucent Technologies (formerly Bell Labs), Dow Chemical, Eastman-Kodak, IBM, and Microsoft Corporation. These companies spent from $5 billion to $800 million on R&D in 1995. We chose companies having among the largest total R&D budgets, in terms of dollars spent, because we believed that their experiences would be the most relevant to the needs of the federal government. Our work also covered the available data on R&D spending and output indicators provided by the leading data-gathering organizations in this field—the OECD, NSF, and the Industrial Research Institute (IRI). Our collaboration with the OECD included participation in an OECD conference on science and technology indicators. Throughout the course of this work, we also interviewed the NSF staff responsible for publishing the Science and Engineering Indicators. We performed our audit work from June 1996 through February 1997 in accordance with generally accepted government auditing standards. Page 13 GAO/RCED-97-91 Measuring Performance Chapter 2 R&D Spending Data Provide Some Information About Innovative Activity but Not About R&D Results R&D spending data provide an indication of how much research is being performed but do not provide a measure of the impacts of that spending. Spending data have been used for budgeting purposes; however, they have also been used as an indicator of the level of innovative activity within a nation or company. The use of R&D spending data as an indicator has a number of advantages. For example, it reduces the innovation process to a single figure for the purposes of discussion. In addition, the data-gathering methods have been refined over many years and are generally reliable over time. However, the level of spending is not a reliable indicator of the level of research results. For example, companies told us that they are focusing more of their spending on short-term R&D projects than on long-term projects, but the impacts of that change in emphasis are unclear. The use of spending data is more appropriate for discussions of R&D spending priorities than of the effectiveness and impacts of R&D spending levels. Traditionally, R&D expenditures have been taken to indicate the “amount” R&D Spending Gives of innovative activity that is occurring within a country or a firm. One of an Indication of the advantages of using expenditure data in this way is that it simplifies Innovative Effort the discussion of the complex process of innovation to a single unit of measurement. Another advantage is that the use of dollars as the unit of measurement enables direct comparisons to be made. In addition, the gathering of spending data has been refined over many years, increasing the data’s reliability and relevance to policy-making. This straightforward rationale—the more R&D spending, the more innovative activity—is the primary advantage to using expenditure data in policy discussions. Its simplicity and close ties to the linear model of innovation allow it to be readily understood by those with little specialized knowledge, making it appealing in policy discussions. These same simplifying characteristics may have led to its use in other areas. In some contexts, countries and companies are categorized according to their technological sophistication on the basis of their R&D spending levels; little attention is given to other factors. Another advantage arises from the common use of “dollars” in the R&D spending data. This usage enables the spending in different research areas to be compared according to the same units. These straightforward comparisons are useful in demonstrating the priorities of the nation at large. For example, recent U.S. R&D expenditure data show that a reduction in defense-related R&D was somewhat counterbalanced by an increase in federal support for civilian R&D programs, including those Page 14 GAO/RCED-97-91 Measuring Performance Chapter 2 R&D Spending Data Provide Some Information About Innovative Activity but Not About R&D Results aimed at improving the diagnosis and treatment of disease, cleaning up the environment, and enhancing technological competitiveness and economic prosperity. In addition, converting different foreign currencies to dollars allows for international comparisons of research. After allowing for the variation of inflation over time, funding data depict historical patterns of real expenditures. The accuracy and policy relevance of spending data are also important advantages to their use as indicators. The typical source of these data is the widely cited Science and Engineering Indicators document published periodically by the National Science Board. The Science Resources Studies Division of NSF has been gathering data to use in Indicators for many years, and many improvements have been made in the accuracy of the data.6 For example, NSF has sponsored the Survey of Industrial R&D since 1953 as a source of data for Indicators. Recent improvements to this survey include selecting samples annually (rather than less frequently) and increasing the sample size from approximately 14,000 to nearly 24,000 firms. NSF took these steps to account more accurately for the establishment of R&D-performing entities in the survey universe and to survey more fully and accurately the R&D performed by nonmanufacturing firms. In addition, NSF is constantly searching for ways to make Indicators more responsive to policymakers’ needs. NSF began to make adjustments in its surveys when it recognized that there was a need to supply more information on the service sector to policymakers because the survey historically had focused on the manufacturing industries in which R&D performance had been heavily concentrated in the past. The use of spending data is limited in its relevance to the impacts of R&D. R&D Spending Is Not There is some correlation between the level of R&D spending and a Good Indicator of innovative success. For example, if fewer research projects are performed, R&D Results then companies and countries forgo the potential benefits of the research. However, spending alone does not guarantee innovative success because many additional factors figure into the innovation process and have important effects on the resulting outputs. The reality of the process of innovation is much more complex than expenditure data alone can reveal. The usefulness of R&D spending data as an indicator of results is limited because the data measure the amount of resources a firm or a nation 6 Some problems with the data may still exist, however. For example, NSF itself notes that the data on trends in the private sector’s basic research contain anomalous spending spikes in 1986 and 1991. The inconsistencies in the data appear to derive from changes made in the NSF survey in those years. For example, in 1991 the survey was expanded to cover a broader array of nonmanufacturing firms. Page 15 GAO/RCED-97-91 Measuring Performance Chapter 2 R&D Spending Data Provide Some Information About Innovative Activity but Not About R&D Results dedicates to innovation, but not its ability to convert that effort into successful products, processes, and services. Accounts in the press have noted that companies are proud of high R&D spending levels because well-focused R&D generally pays off in long-term revenue growth and profits. However, more is not always better, as shown by the companies that ranked among the leaders for R&D spending and disappeared shortly thereafter. For example, two companies called Xonics and Transitron Electronic ranked as the top companies in R&D spending per employee for 1984 and 1986, respectively; in the same years that they achieved their top ranking, one company was forced to file for Chapter 11 protection and the other company dissolved. There is great uncertainty in research and development investments. The processes leading to commercially viable and socially useful technologies are complex and involve substantial non-R&D factors. One company official told us that it was impossible to determine which business function was most important to the success of a new product—research, marketing, or sales. While there is some correlation between the level of R&D spending and innovative success, spending alone does not guarantee success. One official from a high-technology company told us that his company is constantly evaluating research projects with respect to their targeted markets and estimating the expected return on investment. However, in one case the company lost approximately $100 million when the market would not support a newly developed product at the price the company expected. The usefulness of R&D spending indicators is also limited because the way in which innovative activities are structured and managed can be as significant as the amounts of resources devoted to them in determining their outcomes and effects on performance. Those nations or firms with extremely efficient innovation systems can outperform those that use greater R&D resources inefficiently. As a former director of NSF pointed out to us, streamlining research efforts could reduce both bureaucratic and financial overhead costs and make up for spending reductions. In addition, falling spending levels may hide the greater efficiencies that might be realized by leveraging research. One company official told us that although his company was cutting back on long-term research, it was relying more on its relationships with federal and university laboratories for similar work. Collaboration between firms—through joint ventures, consortia, and contracting—has recently been on the rise as firms attempt to efficiently distribute risk, pool their resources, and tap into external Page 16 GAO/RCED-97-91 Measuring Performance Chapter 2 R&D Spending Data Provide Some Information About Innovative Activity but Not About R&D Results expertise. The magnitude of alliance formation is difficult to gauge, as are the implications for the innovation and commercialization of new technologies in the United States. However, one academic told us that if the rhetoric about the efficiencies coming from cooperative R&D is true, then no one should be too disappointed with recent drops in R&D spending. Federal and private R&D spending patterns reflect the changing conditions Spending Patterns in the country and the world at large. The share of basic research Show a Greater supported by private sources peaked in the early post-World War II period, Emphasis on driven by thriving domestic and international markets. However, recent surveys and anecdotal evidence suggest that increased international Short-Term Research competitive pressures have forced companies to emphasize short-term development over long-term research. Company officials echoed this shift and suggested that the focus on short-term research was part of a push toward more relevancy in their R&D departments. They pointed out that their business units are determining their research needs to a greater extent today than they have in the past. Some companies told us that this strategy could be viewed as short-sighted. Others defended this strategy. For example, one company official told us that his company emphasizes short-term research because “competition is short-term.” He stated that in today’s competitive environment, once a market is lost, it is gone forever. Page 17 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results Quantitative and qualitative indicators have been developed to evaluate R&D activities and their results, but both types of indicators have strengths and limitations. Our interviews with a number of companies showed that the private sector stresses marketplace results rather than relying on output indicators. Because of the companies’ profit orientation, many of the indicators tracked by the private sector cannot be directly applied to the federal government. In response to the GPRA, the federal science agencies are exploring new ways to quantify the impacts of research. However, it is too early to tell whether new performance measures can be developed and whether they will meet the needs of the Congress. Because of the difficulties in identifying the impacts of research, Current Indicators decisionmakers in the public and private sectors typically have chosen to Have Strengths and measure outcomes using a variety of proxies. These quantitative and Limitations qualitative indicators have strengths and limitations. To illustrate these strengths and limitations, we looked at three of the most frequently cited quantitative indicators: return on investment, patents issued, and bibliometrics. While these indicators imply a degree of precision, they were generally not designed to measure the long-term results of R&D programs, nor are they easily adaptable to such a purpose. Qualitative assessment provides detailed, descriptive information, but it depends on subjective judgments and may be costly. Return on Investment This indicator aims at measuring the sales and profits resulting from investments in R&D; as such, it addresses one of the fundamental concerns about the value of such investments. However, a variety of factors, such as the complexity of the innovation process and its inherently long time frames, pose serious obstacles to the calculation of these returns. The literature dealing with return on investment is replete with words of caution against quantifying R&D results. NSF’s Science and Engineering Indicators 1996 pointed out that not only is much of this information unobtainable or ambiguous, but many of the gains from research are simply monetarily intangible. Experts on the R&D process have stated frequently that the long time periods and multiple inputs involved make the task of calculating the return on basic research especially difficult. Productivity growth may lag 20 years behind the first appearance of research in the scientific community, and the lag for interindustry effects may be 30 years. A more serious impediment, however, is the fact that outcomes are often not Page 18 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results directly traceable to specific inputs or may result from a combination of such inputs. The National Aeronautics and Space Administration (NASA) and the National Bureau of Standards (now the National Institute for Standards and Technology) attempted to measure the economic impacts and benefits of certain of their technologies in the 1970s and early 1980s. The studies at the Bureau were discontinued, according to staff, because of serious theoretical and methodological problems. As with the Bureau’s studies, NASA’s studies evoked serious criticisms and were likewise discontinued. Despite the difficulties in calculating return on investment, the leading researchers in this field agree that R&D offers high private and social returns in terms of high productivity. One recent survey of 63 studies found that R&D activity achieves, on average, a 20- to 30-percent annual return on private (industrial) investments.7 Patents Patents show certain strengths as useful indicators in measuring technical change and inventive input and output over time. For example, they can reveal a variety of trends involving ownership and levels of activity in technical areas. According to NSF, the data concerning ownership show that the federal share of patents averaged 3.5 percent of the total number of U.S. patents during 1963 through 1980 but declined thereafter. In addition, the data concerning a country’s distribution of patents by technical area have proved to be a reliable indicator of a nation’s technological strengths as well as its direction in product development. For example, the three most emphasized U.S. patent categories for inventors show specific contrasts between U.S. and foreign patents. U.S. inventors obtained most of their patents in wells, mineral oils, and certain areas of surgery. Japanese inventors focused their efforts on certain areas of communications, organic compounds, and refrigeration. Patent activity can be used to pinpoint potentially important changes. In 1980 through 1987, U.S. inventors led all other foreign inventors in radio and television patents, but in 1987 the United States lost its front position to Japanese inventors in this area. Despite their usefulness as indicators of broad national and international trends in various industries and areas of research, patents also have several intrinsic drawbacks. Inconsistency across industries in the number of patents granted is a major limitation that results from the wide 7 Nadiri, M.I. Innovations and Technological Spillovers (1993). Page 19 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results variations in the propensity to patent inventions. Consequently, according to NSF, it is not advisable to compare patenting rates between different technologies or industries. Inconsistency in quality is a second drawback. The aggregated patent statistics do not distinguish between those that led to major innovations and those that led to minor improvements. Incompleteness is a further limitation. Many inventions are not patented at all, and trade secrecy may become a preferred alternative to patenting. Another limitation is that patents do not lend themselves to the evaluation of the most significant results achieved by an R&D program. They can provide intermediate measures of progress, but they are not usually the purpose for which the research was undertaken. In addition, the use of patents as a measure of federal R&D effectiveness may be hampered by their limited relevance. The 1996 report entitled Assessing Fundamental Science by the National Science and Technology Council noted that any use of patent counts should be undertaken only with a full awareness of their limitations. A recent academic study of patents and the evaluation of R&D also commented on the extraordinarily limited applicability of patent evaluation to government-performed R&D. Both studies pointed to the relatively low level of federal patenting activity. The academic study noted that most government laboratories are granted only one or two patents per year, and only a few of them patent extensively. It concluded that for government laboratories, one may question the overall wisdom of evaluating public R&D with private techniques.8 Bibliometrics A third area of effort in developing quantitative measurements involves bibliometrics, or the study of published data. Bibliometrics counts citations in an attempt to address questions of productivity, influence, and the transfer of knowledge. Its most appropriate use is in quantifying the work of researchers whose results are published. Thus, it may be especially applicable in areas such as basic research where the results are more often published than protected by firms. However, its usefulness as a measure of research results remains somewhat controversial. We believe that the use of bibliometrics as a source of information on the quality of the publications or the citations being counted needs to be 8 In its October 11, 1996 issue, Science magazine noted the French government’s announcement that patent records will form a part of the evaluation of publicly funded researchers. The new proposal, according to the magazine, is controversial because it might upset the proper balance between basic and applied research by favoring those who work in fields that are immediately applicable at the expense of people doing basic research. Page 20 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results approached with caution. Although bibliometric indicators can be weighted by publication or other quality measure, the frequency of citation, for example, provides no indication of the level of research innovation. Another limitation is the problems that arise in interdisciplinary comparisons of results. Some critics have gone so far as to say that bibliometric findings should not be used in science policy work until the problems with the analysis of citations are addressed. In addition, the relevance of bibliometric analysis to decision-making by the federal government appears very limited. One expert noted that a recent comprehensive review of bibliometrics shows the sparsity of bibliometric studies for evaluations of the impact of research reported by the federal government. Another pointed out that few federal agencies use bibliometric analysis as an evaluative tool. One of the few is the National Institutes of Health (NIH), which uses this method to evaluate the effectiveness of its different institutes, the comparative productivity of NIH-sponsored research and similar international programs, and the responsiveness of NIH’s research programs to their congressional mandate. Peer Review Recognizing the limitations of quantitative indicators, the National Science and Technology Council concluded that it makes sense to track relevant measures but that they cannot supplant the essential element of expert judgment. Peer review, the most important form of qualitative assessment, uses technical experts to judge R&D results on the basis of the expert’s evaluation of the quality of research. However, peer review has serious shortcomings; it generally depends on criteria that are inherently difficult to measure and on subjective judgment that is vulnerable to bias. Peer review has been used extensively in the selection of proposed research projects. To a lesser extent, it has also been used to evaluate R&D impacts. Peer review has come to be viewed by some observers as the best assurance that quality criteria will prevail over social, economic, and political considerations, while others view it as an element of elitism in science that tends to discount such concerns as economic significance. Its major strength is its ability to bring together the leading experts in the area of concern. Most peer review procedures require a minimum of three reviewers; if the review involves a more ambitious scope of coverage (such as an entire agency), dozens of reviewers may be involved. The process of selecting the peer reviewers varies. One of the chief responsibilities of the professional staff in science agencies such as NSF Page 21 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results and NIH is to stay in touch with a specialized community of scientists who are qualified to judge the agency’s activities. However, others rely on in-house managers who are not active researchers. The major limitations of peer review are twofold. First, the perception of quality depends largely on the expertise of the panel members. It is based on the judgment of experts about a proposal or a set of research-related results. Generally, a final judgment will depend on the collective weight of the different opinions. Frequently, a numerical rating scale—such as 1 for poor through 5 for excellent—is used. Despite the appearance of precision conferred by a specific number, the numbers represent the best of sometimes widely differing judgments. Consequently, although peer review has been a mainstay in judging science for over three centuries, questions remain about ways of improving it. For instance, according to one academic, to improve “validity and reliability,” research needs to be done on the optimal numbers of reviewers and on the advisability of training people to perform peer reviews. Second, peer review evaluation of completed or ongoing R&D projects is a more thorough and expensive process than peer review for the purpose of selecting proposals for funding. According to one study, the cost of a 2-day, 10-person, face-to-face NSF merit review panel is in the neighborhood of $20,000. Another study concluded that if this method were applied annually to all federal research programs, the cost in reviewer time alone would be enormous. For example, the Army Research Laboratory has contracted for a peer review of its activities; the contract calls for a 3-year review directed by the National Research Council at approximately $650,000 per year. The private-sector companies we interviewed varied in terms of the types The Private Sector’s of quantitative and qualitative R&D indicators that they collect, but in Emphasis on general they made limited use of these indicators in their decisions. Many Marketplace Results companies stressed the difficulties involved in measuring the contribution of R&D to the firm’s overall performance using return on investment, Limits Lessons for the patents issued, and other R&D output measures. All of the firms mentioned Federal Government that they were increasing R&D that contributes directly to the bottom line of the firm. Thus, they shifted the responsibility for R&D decisions to the business-unit level so that the R&D would be tied more directly to the profits of those units. The private sector’s experience offers general lessons to the federal government in terms of ensuring that the R&D Page 22 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results contributes directly to the mission of the organization, although the specific output measures do not apply directly to federal R&D efforts. Companies Cited Companies told us that measuring the return on R&D investment is very Difficulties in Measuring difficult. Companies stated that one factor making measurement more the Results of R&D difficult is the long time lag between the research and any revenue that might be earned. Companies also stated that because so many people have been involved in a product’s evolution, it is difficult to separate the contribution of the research unit from that of other units. Companies also mentioned difficulties with some of the other indicators that they track. For example, one indicator was alternatively labeled a “vitality index” or “product innovation metric,” which reflected the share of the firm’s products that could be considered new products. This measure provided an indication of how rapidly the company was incorporating new ideas and research into its products. Several company officials mentioned that this measure had to be applied carefully because of the problem of defining a “new” product; some products are completely different from their predecessors, while others might incorporate cosmetic changes. Firms Increased Their One issue that was mentioned by all of the firms in our discussions was Attention to R&D’s the increasing emphasis on the relevance of R&D to business needs and the Contribution to the Bottom “bottom line.” These comments came up in a number of contexts. For example, some firms mentioned that they look for research with a Line shorter-term payoff in order to have a greater impact on getting new products into the marketplace. Other firms cited the increased emphasis on applied R&D, or the small amount of research that they perform that could be called basic research. A common element was that these firms were attempting to reduce what might be called benevolent research: projects that benefit the industry or the nation but do not have a payoff to the firm. However, a number of the firms noted that this emphasis on the bottom line does not necessarily create the opportunities for breakthrough products. All of the firms either reserved a certain fraction of their research funding for these types of projects or developed processes that made it possible to continue some level of research in areas that might not be directly aligned with any particular product line. In this context, one Page 23 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results firm mentioned the importance of cooperation with the federal government and universities in the pursuit of fundamental research. Firms Have Reduced While this increased emphasis on getting the most out of R&D might be Emphasis on R&D Output expected to lead to greater efforts to measure the results of research, most Measures of the firms that we spoke to responded by changing the organization so that measuring R&D outputs was no longer so important. By shifting the responsibility for research decisions to the business units in the firm that make use of the research outputs, the companies have sidestepped the need for centralized indicators of the quality of research. If the business units believe that a particular R&D project would increase their profits, the firm would budget for that R&D. If the business units do not perform up to expectations in terms of their profitability, the entire unit would be responsible. For example, one firm shifted from a policy of centrally directed research to a policy in which the individual business units make the decisions on the appropriate research projects. Under the previous arrangement, the various business units were assessed to support the central R&D efforts; under the new arrangement, the units pay only for those projects that they think are valuable. This shift was designed to make the research more responsive to the needs of the business units, in that these units do not pay for the research unless they find it useful. This shift also greatly reduced the emphasis on developing R&D output indicators in the central laboratory because each of the business units would be reviewed on the basis of its profitability. The reasoning is that if these units were making poor decisions on R&D projects, the unit’s overall profitability would decrease. Private Sector’s Measures Our interviews with private-sector firms suggest that many of the R&D Do Not Directly Apply to output measures tracked by the private sector do not apply directly to the the Federal Government federal government. Many of these measures are directly related to the contribution of the R&D to the bottom line profitability of the firm. However, federal agencies do not operate in order to make a profit, but to accomplish a variety of other missions and goals. For example, agencies conduct R&D to support a variety of missions, such as maintaining national security or improving citizens’ health. R&D in these areas can contribute greatly to the quality of life in the United States, even if it has a negative Page 24 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results return on investment.9 Given these very different missions, there is also no reason to believe that any single measure is appropriate for different public-sector agencies. Despite the lack of specific measures that can be translated from the private to the public sector, there are general lessons to be learned from the private sector’s experience. Possibly the most important is the recognition that as the pressures on costs increased at many of the firms that we interviewed, the firms made significant efforts to ensure that R&D contributed directly to the bottom line. The federal government likewise faces increasing pressure on costs such as R&D expenditures but, unlike the private sector, cannot rely on the marketplace to ensure that the R&D contributes to the agency’s goals. Federal R&D is undertaken to support a variety of agency missions, and producing an economic return is not the primary justification for most federal R&D programs. In fact, the purpose of federal R&D is to promote research that is socially beneficial but may not be profitable for the private sector to pursue. Without this competitive marketplace to ensure the relevance of R&D, the federal agencies will continue to be challenged to develop better measures of the outputs of their R&D. The literature confirms this general finding of our discussions. For example, in 1995 the National Research Council reported on the results of a workshop on what the federal government might learn from the corporate experience in research restructuring and assessment.10 The Council invited senior corporate research managers from IBM, AT&T, Ford, and Xerox to discuss their experiences in this area. The report concluded that developing useful metrics and using them appropriately is a difficult problem, but it is not impossible. In addition, the participants were not successful in translating their private-sector experience into specific lessons about what can be measured and what makes sense to measure for the federal government. 9 The return-on-investment measure that is appropriate for the federal government would compare the total “social benefits” to the nation with the costs of the initial investment. Computing this measure has many of the difficulties of computing the return on investment for private firms, with the added complications associated with placing a value on national security, improved quality of life, and other intangible qualities. 10 Research Restructuring and Assessment: Can We Apply the Corporate Experience to Government Agencies?, 72 pp., National Research Council (1995). Page 25 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results Our July 1996 testimony on the implementation of GPRA noted that the Federal Science Congress recognized that successful implementation will not come quickly Agencies Are Still or easily for many agencies.11 To help address the challenges of Exploring Ways to “measuring” the results of R&D programs, the Research Roundtable, a group of federal researchers and managers representing a cross-section of Measure the Impacts departments and agencies, has met periodically to share ideas and of R&D approaches for implementing GPRA. The Army Research Laboratory has also begun to address this issue in a pilot project for performance measurement under GPRA. The Research Roundtable has considered the extent to which R&D agencies can and should adopt a common approach to measuring performance. In 1995, it issued a paper based on 6 months of discussions on the development of performance measures for research programs. Although the Roundtable stated that the results of a research program’s performance can be measured, it cautioned that at the same time, it is important to recognize the complexity of the cause-effect relationship between R&D and its results. It added that this complexity makes it difficult to establish quantifiable measures that consistently measure program performance. It also noted that such measures create a potential for incorrect application, which could lead subsequently to a detrimental effect on scientific endeavors. It warned that quantitative empirical demonstrations of such cause-effect relationships should not be required and are often not even possible. The Army Research Laboratory was designated a pilot project for performance measurement under GPRA. Of the more than 70 such pilot projects governmentwide, the laboratory was the only pilot project that addresses scientific research. As such, it attempted to break new ground in both the planning and the evaluation of basic and applied research. The Chief of the Army Research Laboratory’s Special Projects Office, who is mainly responsible for designing the laboratory’s approach to implementing GPRA, submitted a case study to the Office of Management and Budget in 1996.12 The case study outlines an approach that makes use of “three pillars:” metrics, peer review, and customer feedback. In the case study, the laboratory identified about 60 metrics, most of which measure input using fiscal, facilities, and personnel data. Some of the 11 Managing for Results: Key Steps and Challenges in Implementing GPRA in Science Agencies (GAO/T-GGD/RCED-96-214, July 10, 1996). 12 Brown, Edward A. “Applying the Principles of the Government Performance and Results Act to the Research and Development Function” (1996). Page 26 GAO/RCED-97-91 Measuring Performance Chapter 3 R&D Output Indicators Can Provide Limited Information About R&D Results metrics, such as tasks completed, patents awarded, and articles published, measure output; none measure outcome. The laboratory views the measures as useful tools for understanding the functional health of the organization and the management of the laboratory but cautions that the information will not enable it to determine the real quality and impact of its R&D. The laboratory is relying more heavily on peer review of its research and on customer surveys for information about quality. According to the case study, the laboratory has contracted with the National Research Council of the National Academies of Science and Engineering to conduct a retrospective peer review of research over a 2- or 3-year period. The Council is to assemble a Technical Assessment Board that consists of six panels with 8 to 10 people, each of whom is of high repute within the technical community. These panels will appraise the quality of the laboratory’s technical and scientific efforts and, to a limited extent, productivity. The case study also described the use of customer feedback. The laboratory has identified a number of internal and external customers. For those customers to whom it delivers specific items, it uses a series of feedback questionnaires to determine their degree of satisfaction on a 1 to 5 scale in terms of the quality, timeliness, and utility of the deliverable. For those customers who do not receive a specific, identifiable product, the laboratory is developing a Stakeholders’ Advisory Board of senior leadership and user representatives to provide first-hand guidance and feedback. Page 27 GAO/RCED-97-91 Measuring Performance Appendix I Historical Perspective on Research Spending Since World War II, U.S. public and private research and development (R&D) spending patterns have reflected changing priorities as well as reactions to the changing national and international economies. Some of the more prominent events that have shaped spending in both sectors have been the Cold War and the recent international competitive pressures. Declines in spending in both sectors have been less frequent than increases. In his July 1945 report, Science—The Endless Frontier, Vannevar Bush, Federal Spending who headed the U.S. R&D effort during World War II, provided the rationale for federal support of both basic research and research related to national security, industry, and human health and welfare. His plan contributed to the legislation adopted in 1950 that established the National Science Foundation (NSF). By that time, however, the National Institutes of Health (NIH) already had control over most health-related research; the Office of Naval Research had taken on a major role in supporting academic research in the physical sciences; and the new Atomic Energy Commission had been assigned control of the R&D on nuclear weapons and nuclear power. NSF’s mission focused on supporting fundamental research and related educational activities. Its annual budget was less than $10 million until the late 1950s. In contrast, NIH’s annual budget, which had been less than $3 million at the end of the war, grew to more than $50 million by 1950. The scope of federal R&D support grew in the decade after World War II. Anxiety over the Cold War and the loss in 1949 of the U.S. monopoly in nuclear weapons led to expanded R&D programs in the Army and in the newly established Air Force and to a continuing buildup in support for nuclear weapons R&D in the Atomic Energy Commission. On the civilian side, R&D programs were established or expanded in fields with direct practical importance, such as aeronautics technology, water desalinization, and atmospheric disturbances and weather. The launch of Sputnik by the Soviet Union in 1957 led to immediate efforts to expand U.S. R&D, science and engineering education, and technology deployment. Within months of the Sputnik launch, the National Aeronautics and Space Administration (NASA) and the Advanced Research Projects Agency (ARPA) in the Department of Defense (DOD) were established. NASA’s core included the aeronautics programs of the National Advisory Committee on Aeronautics and some of the space activities of DOD; ARPA’s purpose was to enable DOD to conduct advanced R&D to meet military needs. Page 28 GAO/RCED-97-91 Measuring Performance Appendix I Historical Perspective on Research Spending Federal appropriations for R&D and for mathematics and science education in NSF and other government agencies rose rapidly over the next decade, often at double-digit rates in real terms. (See fig. I.1.) During the early 1950s, the growth in federal funding for health research slowed considerably from its fast pace in the immediate postwar years. However, in the late 1950s federal support for health research accelerated rapidly. From 1966 to 1975, federal support for nondefense R&D dropped nearly 22 percent in real terms. The costs of the Vietnam War put downward pressure on nondefense R&D along with other nondefense discretionary spending. The conclusion of NASA’s Apollo program contributed to the decline in federal R&D funding during that period, also. Until recently, this was the only period in which federal funds for R&D were reduced substantially. Page 29 GAO/RCED-97-91 Measuring Performance Appendix I Historical Perspective on Research Spending Figure I.1: U.S. R&D Spending, 1953-96 Constant 1987 dollars (billions) 160 Total U.S. spending 140 120 100 Industry 80 60 Federal 40 20 0 1954 1956 1958 1960 1962 1964 1966 1968 1970 1972 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 Year Source: National Science Foundation, Science Resources Studies Division. In the 1970s, new R&D-intensive agencies addressed environmental and energy issues. Both the environmental movement and the energy crisis of the 1970s, according to some analysts, led to increased public and private spending on environmental and energy R&D. The Environmental Protection Agency was established in 1970. In 1977, the Energy Research and Development Administration and other federal energy-related activities were combined to form the Department of Energy, which was given major new responsibilities to fund energy-related R&D. Page 30 GAO/RCED-97-91 Measuring Performance Appendix I Historical Perspective on Research Spending In the 1980s, the federal role in R&D expanded to enable the United States to compete in a global market. Out of this atmosphere, several programs were initiated to provide financial and other incentives for industrial R&D and for industrially related R&D conducted at universities or federal laboratories. Two major factors have influenced federal support for R&D funding in the 1990s. These factors have been the efforts to reduce the budget deficit and the defense drawdown. The Department of Defense and the Department of Energy, two of the four largest sources of federal R&D support, have had constant-dollar reductions in R&D obligations during the 1990s. In fiscal year 1995, Defense accounted for roughly half of all federal R&D obligations, down from nearly two-thirds of the total in 1986 at the height of the defense build-up that occurred during the Reagan administration. At the same time, military-related R&D spending was being curtailed, while federal investment in selected civilian R&D activities increased, including support for research aimed at improving health and the environment and for technology advancement. The Department of Health and Human Services, which is a distant second to Defense in terms of total R&D support, had the largest absolute increase—$3 billion—in federal R&D obligations during the 1990s. The proportion of all U.S. R&D devoted to health-related projects has been increasing continuously for nearly a decade. The Commerce Department has registered the largest percentage increase in federal R&D obligations during the 1990s so far. In addition, the federal government, which supplies about three-fifths of all funds used to perform R&D on college and university campuses, has been increasing its support of academic research continuously since 1982, even after adjusting for inflation.13 During the early post-World War II period, thriving domestic and Private-Sector international markets supported large profits and the rapid expansion of Funding R&D in both the central laboratories and the divisional laboratories of large companies. Central R&D facilities focused increasingly on fundamental research in many of these large firms, leaving the development and application of new technologies, as well as the improvement of established products and processes, to the divisional laboratories. The data on basic research for 1953 through 1960 are less reliable than those for later years but suggest that the share of total U.S. basic research financed by industry 13 Much of this section is based on Allocating Federal Funds for Science and Technology by the Committee on Criteria for Federal Support of Research and Development of the National Academy of Sciences, National Academy Press (1995). Page 31 GAO/RCED-97-91 Measuring Performance Appendix I Historical Perspective on Research Spending during the postwar period may well have been at its peak during the 1950s and early 1960s. Severe competitive pressures from foreign firms, increases in the real cost of capital, and a slowdown in the growth rate of the domestic economy in the 1970s have been cited as causes for the apparent decline in the returns to R&D investment during the mid-1970s, and the rate of growth in real industry expenditures on R&D declined. Industry’s funding of basic research shrank, and many of the central research facilities of the giant corporations entered a period of budgetary austerity or cutbacks. After a resurgence in the early 1980s, the rate of growth in industry-funded R&D declined in the late 1980s. Industry’s funding of R&D was generally flat between 1991 and 1994; several reasons have been cited for the lack of growth in some companies’ R&D programs during this time. These include corporate downsizing, decentralization (i.e., a shift of R&D activity from corporate laboratories to individual business units), and increasing collaboration among industrial firms and with partners in academia, in government, in the nonprofit sector, and in other countries. The preliminary data for 1995 and 1996 indicate a slight upswing in industrial R&D funding. According to NSF, the most striking recent trend in industrial R&D performance has been the increase in the proportion of total R&D funded by the companies classified as nonmanufacturing industries. Prior to 1983, nonmanufacturing industries accounted for less than 5 percent of the industry’s total R&D. That share grew steadily during the next decade so that in 1993, nonmanufacturing firms represented more than 25 percent of all industrial R&D performed in the United States. Part of this reported growth, however, reflects improvements in the NSF’s survey efforts to measure R&D in nonmanufacturing industries, in addition to actual growth in service sector R&D spending. Between 1984 and 1994, some significant changes occurred among the 100 largest publicly held R&D funding companies, although the four leading firms were the same in both 1984 and 1994. During the decade, the number of pharmaceutical and computer hardware and software companies among the largest R&D funders rose. In contrast, the number of large defense contractors and chemical and petroleum companies among the largest R&D funders fell. Page 32 GAO/RCED-97-91 Measuring Performance Appendix I Historical Perspective on Research Spending U.S. firms have begun to alter their R&D patterns in response to increasing competitive pressures. Firms have shifted a greater portion of their R&D resources away from long-term investments and toward shorter-term projects. U.S. companies now allocate 22 percent of their R&D spending to long-term projects when compared with their Japanese counterparts, who devote 50 percent. Increasingly, firms are emphasizing short-term R&D for immediate problem-solving or near-term development over basic research; and basic research is being directed toward the needs of product development and manufacturing teams. Many central research laboratories at large companies—such as AT&T, IBM, General Electric, Kodak, and Xerox—have been downsized and work more closely with product development divisions. They now receive a larger share of their operating funds from individual business units rather than general corporate funds. Page 33 GAO/RCED-97-91 Measuring Performance Appendix II Major Contributors to This Report Victor S. Rezendes, Director Resources, Robin M. Nazzaro, Assistant Director Community, and Andrew J. Vogelsang, Evaluator-in-Charge Economic Dennis Carroll, Senior Evaluator Development Division, Washington, D.C. Loren Yager, Assistant Director Office of the Chief Economist (307738) Page 34 GAO/RCED-97-91 Measuring Performance Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Measuring Performance: Strengths and Limitations of Research Indicators
Published by the Government Accountability Office on 1997-03-21.
Below is a raw (and likely hideous) rendition of the original report. (PDF)