oversight

Measuring Performance: Strengths and Limitations of Research Indicators

Published by the Government Accountability Office on 1997-03-21.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to Congressional Requesters




March 1997
                 MEASURING
                 PERFORMANCE
                 Strengths and
                 Limitations of
                 Research Indicators




GAO/RCED-97-91
      United States
GAO   General Accounting Office
      Washington, D.C. 20548




      B-275241

      March 21, 1997

      The Honorable Constance A. Morella
      Chairwoman
      The Honorable Bart Gordon
      Ranking Minority Member
      Subcommittee on Technology
      Committee on Science
      House of Representatives

      The Honorable John S. Tanner
      House of Representatives

      This report responds to your request for information on the indicators used to evaluate the
      results of research and development (R&D). The report discusses the relative strengths and
      limitations of the input and output indicators used by the federal and private sectors to measure
      the results of R&D. The report also provides a historical perspective on research spending.

      As agreed with your offices, we plan no further distribution of this report until 30 days from its
      date of issue, unless you publicly announce its contents earlier. We will then send copies to
      interested parties, and we will also make copies available to others upon request.

      If you have any questions, I can be reached at (202) 512-3600. Major contributors to this report
      are listed in appendix II.




      Allen Li
      Associate Director, Energy,
        Resources, and Science Issues
Executive Summary


                   American taxpayers invested more than $60 billion in federal funds in
Purpose            military and civilian research and development (R&D) efforts in 1996. The
                   private sector invested more than $110 billion that same year. The
                   technological advancements resulting from these efforts are a critical
                   factor in improving the productivity of American workers and,
                   correspondingly, the nation’s standard of living. However, while the
                   contribution of R&D to technological advancement is widely recognized,
                   there is no widely accepted method of measuring the results of that
                   research.

                   To facilitate discussions of the adequacy of the funding and of the results
                   of the R&D, the Subcommittee on Technology, House Committee on
                   Science, asked GAO to evaluate the various indicators that are used to
                   measure the results of R&D. Specifically, this report discusses the strengths
                   and limitations of the input and output indicators used by the federal and
                   private sectors to measure the results of R&D. This report also provides a
                   historical perspective on spending for research.


                   The commitment to reduce the federal deficit is forcing the Congress to
Background         reexamine the value of programs across the federal government. Although
                   scientific research is often considered to be intrinsically valuable to
                   society, there is pressure on all federal agencies, including science
                   agencies, to demonstrate that they are making effective use of the
                   taxpayers’ dollars. This greater emphasis on results is evident in the
                   passage of the Government Performance and Results Act of 1993 (GPRA).
                   The act fundamentally seeks to shift the focus of federal management and
                   accountability from a preoccupation with staffing, activity levels, and
                   tasks completed to a focus on results—that is, the real difference that
                   federal programs make in people’s lives.

                   The experts in research measurement have tried for years to develop
                   indicators that would provide a measure of the results of R&D. However,
                   the very nature of the innovative process makes measuring the
                   performance of science-related projects difficult. For example, a wide
                   range of factors determine if and when a particular R&D project will result
                   in commercial or other benefits. It can also take many years for a research
                   project to achieve results.


                   The amount of money spent on research and development, the primary
Results in Brief   indicator of the investment in research, is useful as a measure of how



                   Page 2                                    GAO/RCED-97-91 Measuring Performance
                             Executive Summary




                             much research is being performed. Having been refined over many years,
                             these data are generally available for the research efforts in both the
                             public and private sectors. However, the level of spending is not a reliable
                             indicator of the level of results achieved by research.

                             Unlike the situation with the input measures of research and development,
                             there is no primary indicator of the outputs. Output indicators include
                             quantitative analyses of return on investment, patents granted, and other
                             outputs as well as qualitative assessments based on peer review. The
                             companies that GAO spoke with collect data on various output indicators
                             but, in general, make limited use of them in their investment decisions.
                             Instead, the companies emphasized that research and development
                             contribute directly to their “bottom line.” Because companies are
                             profit-oriented, many of the indicators tracked by the private sector
                             cannot be directly applied to the federal government. Experiences from
                             pilot efforts made under the Government Performance and Results Act
                             have reinforced the finding that output measures are highly specific to the
                             management and mission of each federal agency and that no single
                             indicator exists to measure the results of research.



Principal Findings

Funding Indicates            Funding has been used as the primary input indicator for decades.
Research Activity but Does   Whether a policymaker is interested in basic research, applied research, or
Not Measure the Results of   development, the amount of money spent in that area is taken as an
                             indication of how much research is being performed. The major
Research                     advantages of using expenditure data as an indicator are that they are
                             easily understandable, readily available, and have been, in general,
                             consistently gathered over time. In addition, spending on different projects
                             in different research areas can be measured according to the same unit,
                             dollars, making comparisons between projects straightforward.

                             The amount of funding, however, does not provide a good indication of the
                             results of research. Companies told GAO that they are focusing more of
                             their spending on short-term R&D projects than on long-term projects.
                             However, the impacts of that change in focus are unclear. The reduced
                             funding levels may not reflect the fact that the R&D efforts are being
                             performed with greater efficiency. For example, one way in which the
                             federal government and the private sector have tried to use R&D resources




                             Page 3                                    GAO/RCED-97-91 Measuring Performance
                        Executive Summary




                        more efficiently and effectively is through consortia with universities or
                        other companies. By combining their research activities, the companies
                        attempt to avoid expensive duplication and learn from each other.


R&D Output Indicators   Because of the difficulties in identifying the impacts of research,
Can Provide Limited     decisionmakers have developed quantitative and qualitative indicators as
Information About the   proxies to assess the results of R&D activity. The strengths and limitations
                        are evident in both types of indicators. The current quantitative indicators
Results of R&D          focus mainly on return on investment, patenting rates, and
                        bibliometrics—the study of published data. While implying a degree of
                        precision, these indicators were not originally intended to measure the
                        long-term results of R&D. Qualitative assessment provides detailed
                        information, but it relies on the judgments of experts and may be
                        expensive.

                        Because of these difficulties, the companies interviewed by GAO stressed
                        marketplace results rather than R&D output indicators. While varying in the
                        types of indicators they collect, they emphasized the difficulties in
                        measuring R&D’s specific contribution to a company’s overall performance.
                        For example, one company stated that because so many people have been
                        involved in a product’s evolution, it is difficult to separate the contribution
                        of the research unit from that of other units. All of the companies
                        interviewed have increased their expectation that R&D contribute directly
                        to their profitability, but instead of increasing their efforts at measuring
                        R&D results, they have shifted the responsibility for R&D decisions to the
                        business units. However, many of the R&D output measures tracked by the
                        private sector do not apply directly to the federal government. In
                        particular, while facing the same increasing cost pressures as the private
                        sector, the federal government cannot rely on the profit motive to guide its
                        decisions.

                        The GPRA requires the executive agencies to develop their annual R&D plans
                        with suitable performance measures. The Research Roundtable, a group of
                        federal researchers and managers representing a cross-section of R&D
                        departments and agencies, warned about the difficulties of quantifying the
                        results of R&D and the potential for incorrect application with subsequent
                        harm to scientific endeavors. The Army Research Laboratory, which was
                        designated a pilot project for performance measurement under the act, has
                        developed a multifaceted approach using quantitative indicators, peer
                        review, and customer feedback to evaluate the results of R&D.




                        Page 4                                     GAO/RCED-97-91 Measuring Performance
                  Executive Summary




                  This report contains no recommendations.
Recommendations
                  Because this report focuses broadly on the R&D of both the federal and
Agency Comments   private sectors, and not on the effort of individual agencies, GAO did not
                  submit a draft of this report to federal agencies for their review and
                  comment.




                  Page 5                                    GAO/RCED-97-91 Measuring Performance
Contents



Executive Summary                                                                                2


Chapter 1                                                                                        8
                      The Process of Innovation and the Use of R&D Indicators                    8
Introduction          Past Efforts to Evaluate R&D Impacts                                      10
                      The Government Performance and Results Act                                11
                      Objectives, Scope, and Methodology                                        12

Chapter 2                                                                                       14
                      R&D Spending Gives an Indication of Innovative Effort                     14
R&D Spending Data     R&D Spending Is Not a Good Indicator of R&D Results                       15
Provide Some          Spending Patterns Show a Greater Emphasis on Short-Term                   17
                        Research
Information About
Innovative Activity
but Not About R&D
Results
Chapter 3                                                                                       18
                      Current Indicators Have Strengths and Limitations                         18
R&D Output            The Private Sector’s Emphasis on Marketplace Results Limits               22
Indicators Can          Lessons for the Federal Government
                      Federal Science Agencies Are Still Exploring Ways to Measure              26
Provide Limited         the Impacts of R&D
Information About
R&D Results
Appendixes            Appendix I: Historical Perspective on Research Spending                   28
                      Appendix II: Major Contributors to This Report                            34

Figure                Figure I.1: U.S. R&D Spending, 1953-96                                    30




                      Page 6                                   GAO/RCED-97-91 Measuring Performance
Contents




Abbreviations

ARPA       Advanced Research Projects Agency
DOD        Department of Defense
GAO        General Accounting Office
GPRA       Government Performance and Results Act
IRI        Industrial Research Institute
NIH        National Institutes of Health
NSF        National Science Foundation
NASA       National Aeronautics and Space Administration
OECD       Office of Economic Cooperation and Development
OTA        Office of Technology Assessment
R&D        research and development


Page 7                                GAO/RCED-97-91 Measuring Performance
Chapter 1

Introduction


                        Over $180 billion was spent on research and development (R&D) in the
                        United States in 1996.1 Most of that amount was spent by industry and the
                        federal government—$113 billion and $62 billion, respectively; the balance
                        was spent by universities and other nonprofit organizations. The leading
                        experts in the study of research indicators agree that R&D has a significant,
                        positive effect on economic growth and the overall standard of living.
                        However, because of the complexity of linking the results of R&D to its
                        economic impacts, there is no widely accepted method of measuring the
                        results of R&D spending in either industry or the federal government. The
                        commitment to reduce the federal budget deficit is forcing the Congress
                        and the executive branch to undertake a basic reexamination of the value
                        of programs across the federal government. It is also placing pressure on
                        all federal agencies, including the civilian science agencies, to clearly
                        demonstrate that they are making effective use of the taxpayers’ dollars.
                        The Government Performance and Results Act of 1993 (GPRA) provides a
                        legislative vehicle for the agencies to use as they seek to demonstrate and
                        improve their effectiveness. Equally important, if successfully
                        implemented, GPRA should help the Congress make the difficult funding,
                        policy, and program decisions that the current budget environment
                        demands.


                        Researchers have developed their own terminology for describing the
The Process of          process of transforming R&D into economic results. In its simplest form,
Innovation and the      the theory underlying both public and private decision-making has been
Use of R&D Indicators   that innovative activity positively affects economic performance.
                        Innovation can be thought of as the development and application of a new
                        product, process, or service. It can include the use of an existing product
                        in a new application or the development of a new device for an existing
                        application. Innovation encompasses many activities, including scientific,
                        technical, and market research; product, process, or service development;
                        and manufacturing and marketing to the extent that they support the
                        dissemination and application of the invention. Innovation is a
                        combination of invention and commercialization. Invention describes the
                        initial conception of a new product, process, or service, but not the act of
                        putting it to use. Commercialization refers to the attempt to profit from
                        innovation through the sale or use of new products, processes, and
                        services.




                        1
                         These figures are based on preliminary 1996 statistics reported by R&D performers to the National
                        Science Foundation.



                        Page 8                                                  GAO/RCED-97-91 Measuring Performance
Chapter 1
Introduction




This description as well as the traditional views of innovation has been
strongly influenced by the linear model of innovation, which says that
innovation proceeds sequentially through the stages of basic research,
applied research, development, manufacturing, and marketing.2 This
model assumes that basic research serves as the source of innovation and
that new scientific knowledge initiates a chain of events culminating in the
development and sale of a new product, process, or service. In this model,
basic research is the most uncertain part of the process; once basic
research is conducted, innovation and commercialization can proceed.
The model suggests that the firms with the best technology will likely be
the first to market and win the lion’s share of profits.

The simplicity of this model makes it particularly useful in policy
discussions. Other models may be more accurate, but they provide a more
complex explanation of the relationship between science and the
commercialization of new technology. These models, such as the
“chain-linked model,” include feedback loops that allow for interaction
among the different stages of the linear model. These models also reflect
the fact that the ideas for new inventions or changes to existing products
often arise from the recognition of new market opportunities, advances in
manufacturing capabilities, or advances in technology independent of
progress in the underlying science.

Real-world examples show that technological breakthroughs can precede
as well as follow basic research. In many cases, science is not the source
of innovation. The Wright brothers, for example, developed the first
airplane without an understanding of aerodynamic theory, and Chester
Carlson developed the first xerographic copier without a thorough
understanding of photoconductive materials. These inventions have
resulted in considerable research into aerodynamic theory and materials
science, respectively, as scientists and engineers attempted to improve the
original invention.

In studying complex processes or concepts such as innovation, it is not
always possible to measure them directly. As a result, researchers turn to
the use of “indicators.” Indicators point to or illustrate the process or

2
 The National Science Foundation uses the following definitions in its resource surveys: Basic research
has as its objective to gain a more comprehensive knowledge or understanding of the subject under
study, without specific applications in mind. Applied research is aimed at gaining knowledge or
understanding to determine the means by which a specific, recognized need may be met. Development
is the systematic use of the knowledge or understanding gained from research directed toward the
production of useful materials, devices, systems, or methods. The Foundation recognizes the
limitations of this classification scheme but continues to use these categories to maintain historical
consistency, among other reasons.



Page 9                                                  GAO/RCED-97-91 Measuring Performance
                      Chapter 1
                      Introduction




                      concept in question but do not directly measure it. For example, in order
                      to determine an object’s temperature, one can use a thermometer to
                      measure it directly. However, when trying to measure something as
                      complex as the health of a country’s economy, one relies on different
                      indicators, such as unemployment rates, stock market averages, or trade
                      balances that do not provide a direct measurement of economic health but
                      do give an indication of its status. In the case of innovation, various “input”
                      and “output” indicators are used that are based on the linear model of
                      innovation. The following chapters of this report are broken down
                      according to these two sets of indicators. Chapter 2 is concerned with R&D
                      spending, or expenditure data, which is the most widely used input
                      indicator of innovation, and chapter 3 focuses on some widely used output
                      indicators.


                      For almost two decades, numerous reports have documented the
Past Efforts to       difficulties of quantifying the results of R&D. As noted above, the
Evaluate R&D          identification of the economic and social effects of research involves
Impacts               complex issues of measurement, analysis, and interpretation. The
                      following efforts may help to give perspective to the present concerns
                      about measuring R&D results.

                  •   In 1979, we reported on a wide range of factors that make the
                      measurement of R&D results difficult.3 The report noted that R&D
                      expenditures are undertaken for a variety of reasons. Some attempt to
                      develop new knowledge; others are directed at meeting needs, such as
                      national defense, for which there is no commercial market; and still others
                      are directed at lowering the cost of products. Furthermore, some projects
                      produce revolutions; others produce nothing. Most important, R&D is only
                      one input into a complex process. Thus, we concluded that there is no
                      possibility of eliminating the role that judgment plays in the allocation of
                      federal R&D resources.
                  •   In 1986, the Office of Technology Assessment (OTA) issued a detailed
                      report that questioned the utility of the effort to quantify R&D returns.4
                      According to OTA, the fundamental stumbling block to placing an economic
                      value on federal R&D is that improving productivity or producing an
                      economic return is not the primary justification for most federal R&D
                      programs. The report added that the attempts to measure the economic
                      return to federal R&D are flawed because many of the research outputs,

                      3
                       Assessing the “Output” of Federal Commercially Directed R&D (GAO/PAD-79-69, Aug. 1979).
                      4
                        Research Funding as an Investment: Can We Measure the Returns? Office of Technology Assessment,
                      72 pp. (Apr. 1986).



                      Page 10                                               GAO/RCED-97-91 Measuring Performance
                      Chapter 1
                      Introduction




                      such as national defense, cannot be assigned an economic value. The
                      report also noted that in industry, where one might expect quantitative
                      techniques to prevail because of the existence of a well-defined economic
                      objective, OTA found a reliance on subjective judgment and good
                      communications between R&D, management, and marketing staffs.
                  •   In 1996, responding to concerns about the implementation of GPRA, the
                      National Science and Technology Council issued a report that stressed the
                      limited role of quantification in measuring the results of R&D.5 It stated that
                      the insufficiency of quantitative measures per se is one reason why other
                      sources of evidence, such as merit review of past performance, narrative
                      discussion, and descriptions of outstanding accomplishments and more
                      typical levels of achievement, should be included in annual performance
                      reports. The report concluded that the cornerstone of world-class science
                      will continue to be merit review with peer evaluation, while the science
                      community works to develop the measurement tools and other methods
                      needed to assess the contributions of fundamental science.
                  •   At the international level, the Organization of Economic Cooperation and
                      Development (OECD) is also grappling with these questions. Beginning with
                      the first edition of the Standard Practice for Surveys of R&D (the “Frascati
                      Manual”) in the 1960s, OECD has been developing international frameworks
                      for the measurement of R&D inputs. In connection with its more recent
                      effort to develop new output indicators (the “Oslo Manual”), OECD stated
                      that the new indicators and the underlying statistics usually take two or
                      more decades to reach general acceptance and regular collection and
                      publication. The participants at a 1996 OECD conference extensively
                      discussed the organization’s efforts to improve the quality of innovation
                      indicators. At this stage, these indicators appear to be most useful in
                      helping researchers study and describe the process of innovation.


                      In response to questions about the value and effectiveness of federal
The Government        programs, GPRA seeks to shift federal agencies’ focus away from such
Performance and       traditional concerns as staffing, activity levels, and tasks completed
Results Act           toward a focus on program outcomes—that is, the real difference the
                      federal program makes in people’s lives. In the GPRA context, an “outcome
                      measure” assesses the results of a program activity compared to its
                      intended purpose. An “output measure,” according to GPRA, tabulates,
                      calculates, or records the level of activity or effort and can be expressed in
                      a quantitative or qualitative manner. The output indicators discussed in
                      our report could be considered either outcome measures or output
                      measures, depending on the context. For example, return-on-investment

                      5
                       Assessing Fundamental Science, National Science and Technology Council (July 1996).



                      Page 11                                               GAO/RCED-97-91 Measuring Performance
                     Chapter 1
                     Introduction




                     calculations could be outcome measures in a business context since
                     gaining the maximum return on investment is the intended purpose of a
                     business. Patenting rates would be output measures in some businesses
                     because patents could serve as one measure of the level of activity of a
                     research unit.

                     The act recognizes how difficult it is to state the goals and measure the
                     results of some programs. While the law encourages the use of objective
                     measures of performance, it authorizes agencies—with the approval of the
                     Office of Management and Budget—to use alternative, subjective
                     measures of performance. Also, instead of having GPRA take effect
                     immediately after its passage in 1993, the Congress allowed for a period of
                     time for the government to learn how to implement the act. As part of the
                     learning process, the act called for the use of pilot projects in performance
                     measurement. Among the approximately 70 agencies or parts of agencies
                     that participated in pilot projects, one addresses scientific research. Upon
                     the full implementation of the act, the executive branch agencies are
                     required to devise plans that are outcome-oriented. The act calls for the
                     agencies to develop three documents: a strategic plan, an annual
                     performance plan, and an annual performance report.


                     In response to the Subcommittee on Technology’s request, our objective
Objectives, Scope,   was to review various indicators that are used to measure the results of
and Methodology      R&D. Specifically, this report discusses the relative strengths and
                     limitations of the input and output indicators used by the federal and
                     private sectors to measure the results of R&D as well as the claim that
                     industry focuses on short-term profitability rather than long-term R&D
                     needs. This report also provides a historical perspective on research
                     spending. (See app. I.)

                     The impacts of innovation are widely studied in the public and private
                     sectors as well as in academia. Our work relied on a limited number of
                     experts in each of these sectors to provide us with the prevailing
                     understanding of and latest developments in the area. As a result, our
                     review does not provide an exhaustive examination of R&D measures nor
                     does it answer the question of how R&D should be measured. It does,
                     however, as agreed with your offices, consist of information on the
                     strengths and limitations in the use of these indicators as well as anecdotal
                     information based on interviews with leading R&D companies. We also
                     reviewed the relevant literature.




                     Page 12                                   GAO/RCED-97-91 Measuring Performance
Chapter 1
Introduction




We interviewed a number of experts, including a former IBM Vice President
for Research and National Science Foundation (NSF) Director, a former
Executive Director of the Manufacturing Forum, a former Associate
Director of the White House Office of Science and Technology Policy, the
Director of the Special Projects Office for the Army Research Laboratory,
and the former Chief Financial Officer for Apple Computer. We also
conducted six teleconferences with company officials who were typically
at the director level. We interviewed representatives of General Electric,
Lucent Technologies (formerly Bell Labs), Dow Chemical,
Eastman-Kodak, IBM, and Microsoft Corporation. These companies spent
from $5 billion to $800 million on R&D in 1995. We chose companies having
among the largest total R&D budgets, in terms of dollars spent, because we
believed that their experiences would be the most relevant to the needs of
the federal government.

Our work also covered the available data on R&D spending and output
indicators provided by the leading data-gathering organizations in this
field—the OECD, NSF, and the Industrial Research Institute (IRI). Our
collaboration with the OECD included participation in an OECD conference
on science and technology indicators. Throughout the course of this work,
we also interviewed the NSF staff responsible for publishing the Science
and Engineering Indicators.

We performed our audit work from June 1996 through February 1997 in
accordance with generally accepted government auditing standards.




Page 13                                  GAO/RCED-97-91 Measuring Performance
Chapter 2

R&D Spending Data Provide Some
Information About Innovative Activity but
Not About R&D Results
                     R&D  spending data provide an indication of how much research is being
                     performed but do not provide a measure of the impacts of that spending.
                     Spending data have been used for budgeting purposes; however, they have
                     also been used as an indicator of the level of innovative activity within a
                     nation or company. The use of R&D spending data as an indicator has a
                     number of advantages. For example, it reduces the innovation process to a
                     single figure for the purposes of discussion. In addition, the data-gathering
                     methods have been refined over many years and are generally reliable over
                     time. However, the level of spending is not a reliable indicator of the level
                     of research results. For example, companies told us that they are focusing
                     more of their spending on short-term R&D projects than on long-term
                     projects, but the impacts of that change in emphasis are unclear. The use
                     of spending data is more appropriate for discussions of R&D spending
                     priorities than of the effectiveness and impacts of R&D spending levels.


                     Traditionally, R&D expenditures have been taken to indicate the “amount”
R&D Spending Gives   of innovative activity that is occurring within a country or a firm. One of
an Indication of     the advantages of using expenditure data in this way is that it simplifies
Innovative Effort    the discussion of the complex process of innovation to a single unit of
                     measurement. Another advantage is that the use of dollars as the unit of
                     measurement enables direct comparisons to be made. In addition, the
                     gathering of spending data has been refined over many years, increasing
                     the data’s reliability and relevance to policy-making.

                     This straightforward rationale—the more R&D spending, the more
                     innovative activity—is the primary advantage to using expenditure data in
                     policy discussions. Its simplicity and close ties to the linear model of
                     innovation allow it to be readily understood by those with little specialized
                     knowledge, making it appealing in policy discussions. These same
                     simplifying characteristics may have led to its use in other areas. In some
                     contexts, countries and companies are categorized according to their
                     technological sophistication on the basis of their R&D spending levels; little
                     attention is given to other factors.

                     Another advantage arises from the common use of “dollars” in the R&D
                     spending data. This usage enables the spending in different research areas
                     to be compared according to the same units. These straightforward
                     comparisons are useful in demonstrating the priorities of the nation at
                     large. For example, recent U.S. R&D expenditure data show that a
                     reduction in defense-related R&D was somewhat counterbalanced by an
                     increase in federal support for civilian R&D programs, including those



                     Page 14                                    GAO/RCED-97-91 Measuring Performance
                      Chapter 2
                      R&D Spending Data Provide Some
                      Information About Innovative Activity but
                      Not About R&D Results




                      aimed at improving the diagnosis and treatment of disease, cleaning up the
                      environment, and enhancing technological competitiveness and economic
                      prosperity. In addition, converting different foreign currencies to dollars
                      allows for international comparisons of research. After allowing for the
                      variation of inflation over time, funding data depict historical patterns of
                      real expenditures.

                      The accuracy and policy relevance of spending data are also important
                      advantages to their use as indicators. The typical source of these data is
                      the widely cited Science and Engineering Indicators document published
                      periodically by the National Science Board. The Science Resources
                      Studies Division of NSF has been gathering data to use in Indicators for
                      many years, and many improvements have been made in the accuracy of
                      the data.6 For example, NSF has sponsored the Survey of Industrial R&D
                      since 1953 as a source of data for Indicators. Recent improvements to this
                      survey include selecting samples annually (rather than less frequently) and
                      increasing the sample size from approximately 14,000 to nearly 24,000
                      firms. NSF took these steps to account more accurately for the
                      establishment of R&D-performing entities in the survey universe and to
                      survey more fully and accurately the R&D performed by nonmanufacturing
                      firms. In addition, NSF is constantly searching for ways to make Indicators
                      more responsive to policymakers’ needs. NSF began to make adjustments
                      in its surveys when it recognized that there was a need to supply more
                      information on the service sector to policymakers because the survey
                      historically had focused on the manufacturing industries in which R&D
                      performance had been heavily concentrated in the past.


                      The use of spending data is limited in its relevance to the impacts of R&D.
R&D Spending Is Not   There is some correlation between the level of R&D spending and
a Good Indicator of   innovative success. For example, if fewer research projects are performed,
R&D Results           then companies and countries forgo the potential benefits of the research.
                      However, spending alone does not guarantee innovative success because
                      many additional factors figure into the innovation process and have
                      important effects on the resulting outputs. The reality of the process of
                      innovation is much more complex than expenditure data alone can reveal.

                      The usefulness of R&D spending data as an indicator of results is limited
                      because the data measure the amount of resources a firm or a nation

                      6
                       Some problems with the data may still exist, however. For example, NSF itself notes that the data on
                      trends in the private sector’s basic research contain anomalous spending spikes in 1986 and 1991. The
                      inconsistencies in the data appear to derive from changes made in the NSF survey in those years. For
                      example, in 1991 the survey was expanded to cover a broader array of nonmanufacturing firms.



                      Page 15                                                 GAO/RCED-97-91 Measuring Performance
Chapter 2
R&D Spending Data Provide Some
Information About Innovative Activity but
Not About R&D Results




dedicates to innovation, but not its ability to convert that effort into
successful products, processes, and services. Accounts in the press have
noted that companies are proud of high R&D spending levels because
well-focused R&D generally pays off in long-term revenue growth and
profits. However, more is not always better, as shown by the companies
that ranked among the leaders for R&D spending and disappeared shortly
thereafter. For example, two companies called Xonics and Transitron
Electronic ranked as the top companies in R&D spending per employee for
1984 and 1986, respectively; in the same years that they achieved their top
ranking, one company was forced to file for Chapter 11 protection and the
other company dissolved.

There is great uncertainty in research and development investments. The
processes leading to commercially viable and socially useful technologies
are complex and involve substantial non-R&D factors. One company official
told us that it was impossible to determine which business function was
most important to the success of a new product—research, marketing, or
sales. While there is some correlation between the level of R&D spending
and innovative success, spending alone does not guarantee success. One
official from a high-technology company told us that his company is
constantly evaluating research projects with respect to their targeted
markets and estimating the expected return on investment. However, in
one case the company lost approximately $100 million when the market
would not support a newly developed product at the price the company
expected.

The usefulness of R&D spending indicators is also limited because the way
in which innovative activities are structured and managed can be as
significant as the amounts of resources devoted to them in determining
their outcomes and effects on performance. Those nations or firms with
extremely efficient innovation systems can outperform those that use
greater R&D resources inefficiently. As a former director of NSF pointed out
to us, streamlining research efforts could reduce both bureaucratic and
financial overhead costs and make up for spending reductions.

In addition, falling spending levels may hide the greater efficiencies that
might be realized by leveraging research. One company official told us that
although his company was cutting back on long-term research, it was
relying more on its relationships with federal and university laboratories
for similar work. Collaboration between firms—through joint ventures,
consortia, and contracting—has recently been on the rise as firms attempt
to efficiently distribute risk, pool their resources, and tap into external



Page 16                                     GAO/RCED-97-91 Measuring Performance
                      Chapter 2
                      R&D Spending Data Provide Some
                      Information About Innovative Activity but
                      Not About R&D Results




                      expertise. The magnitude of alliance formation is difficult to gauge, as are
                      the implications for the innovation and commercialization of new
                      technologies in the United States. However, one academic told us that if
                      the rhetoric about the efficiencies coming from cooperative R&D is true,
                      then no one should be too disappointed with recent drops in R&D spending.


                      Federal and private R&D spending patterns reflect the changing conditions
Spending Patterns     in the country and the world at large. The share of basic research
Show a Greater        supported by private sources peaked in the early post-World War II period,
Emphasis on           driven by thriving domestic and international markets. However, recent
                      surveys and anecdotal evidence suggest that increased international
Short-Term Research   competitive pressures have forced companies to emphasize short-term
                      development over long-term research. Company officials echoed this shift
                      and suggested that the focus on short-term research was part of a push
                      toward more relevancy in their R&D departments. They pointed out that
                      their business units are determining their research needs to a greater
                      extent today than they have in the past.

                      Some companies told us that this strategy could be viewed as
                      short-sighted. Others defended this strategy. For example, one company
                      official told us that his company emphasizes short-term research because
                      “competition is short-term.” He stated that in today’s competitive
                      environment, once a market is lost, it is gone forever.




                      Page 17                                     GAO/RCED-97-91 Measuring Performance
Chapter 3

R&D Output Indicators Can Provide Limited
Information About R&D Results

                       Quantitative and qualitative indicators have been developed to evaluate
                       R&D activities and their results, but both types of indicators have strengths
                       and limitations. Our interviews with a number of companies showed that
                       the private sector stresses marketplace results rather than relying on
                       output indicators. Because of the companies’ profit orientation, many of
                       the indicators tracked by the private sector cannot be directly applied to
                       the federal government. In response to the GPRA, the federal science
                       agencies are exploring new ways to quantify the impacts of research.
                       However, it is too early to tell whether new performance measures can be
                       developed and whether they will meet the needs of the Congress.


                       Because of the difficulties in identifying the impacts of research,
Current Indicators     decisionmakers in the public and private sectors typically have chosen to
Have Strengths and     measure outcomes using a variety of proxies. These quantitative and
Limitations            qualitative indicators have strengths and limitations. To illustrate these
                       strengths and limitations, we looked at three of the most frequently cited
                       quantitative indicators: return on investment, patents issued, and
                       bibliometrics. While these indicators imply a degree of precision, they
                       were generally not designed to measure the long-term results of R&D
                       programs, nor are they easily adaptable to such a purpose. Qualitative
                       assessment provides detailed, descriptive information, but it depends on
                       subjective judgments and may be costly.


Return on Investment   This indicator aims at measuring the sales and profits resulting from
                       investments in R&D; as such, it addresses one of the fundamental concerns
                       about the value of such investments. However, a variety of factors, such as
                       the complexity of the innovation process and its inherently long time
                       frames, pose serious obstacles to the calculation of these returns. The
                       literature dealing with return on investment is replete with words of
                       caution against quantifying R&D results. NSF’s Science and Engineering
                       Indicators 1996 pointed out that not only is much of this information
                       unobtainable or ambiguous, but many of the gains from research are
                       simply monetarily intangible.

                       Experts on the R&D process have stated frequently that the long time
                       periods and multiple inputs involved make the task of calculating the
                       return on basic research especially difficult. Productivity growth may lag
                       20 years behind the first appearance of research in the scientific
                       community, and the lag for interindustry effects may be 30 years. A more
                       serious impediment, however, is the fact that outcomes are often not



                       Page 18                                    GAO/RCED-97-91 Measuring Performance
          Chapter 3
          R&D Output Indicators Can Provide Limited
          Information About R&D Results




          directly traceable to specific inputs or may result from a combination of
          such inputs. The National Aeronautics and Space Administration (NASA)
          and the National Bureau of Standards (now the National Institute for
          Standards and Technology) attempted to measure the economic impacts
          and benefits of certain of their technologies in the 1970s and early 1980s.
          The studies at the Bureau were discontinued, according to staff, because
          of serious theoretical and methodological problems. As with the Bureau’s
          studies, NASA’s studies evoked serious criticisms and were likewise
          discontinued.

          Despite the difficulties in calculating return on investment, the leading
          researchers in this field agree that R&D offers high private and social
          returns in terms of high productivity. One recent survey of 63 studies
          found that R&D activity achieves, on average, a 20- to 30-percent annual
          return on private (industrial) investments.7


Patents   Patents show certain strengths as useful indicators in measuring technical
          change and inventive input and output over time. For example, they can
          reveal a variety of trends involving ownership and levels of activity in
          technical areas. According to NSF, the data concerning ownership show
          that the federal share of patents averaged 3.5 percent of the total number
          of U.S. patents during 1963 through 1980 but declined thereafter.

          In addition, the data concerning a country’s distribution of patents by
          technical area have proved to be a reliable indicator of a nation’s
          technological strengths as well as its direction in product development.
          For example, the three most emphasized U.S. patent categories for
          inventors show specific contrasts between U.S. and foreign patents. U.S.
          inventors obtained most of their patents in wells, mineral oils, and certain
          areas of surgery. Japanese inventors focused their efforts on certain areas
          of communications, organic compounds, and refrigeration. Patent activity
          can be used to pinpoint potentially important changes. In 1980 through
          1987, U.S. inventors led all other foreign inventors in radio and television
          patents, but in 1987 the United States lost its front position to Japanese
          inventors in this area.

          Despite their usefulness as indicators of broad national and international
          trends in various industries and areas of research, patents also have
          several intrinsic drawbacks. Inconsistency across industries in the number
          of patents granted is a major limitation that results from the wide

          7
           Nadiri, M.I. Innovations and Technological Spillovers (1993).



          Page 19                                                  GAO/RCED-97-91 Measuring Performance
                Chapter 3
                R&D Output Indicators Can Provide Limited
                Information About R&D Results




                variations in the propensity to patent inventions. Consequently, according
                to NSF, it is not advisable to compare patenting rates between different
                technologies or industries. Inconsistency in quality is a second drawback.
                The aggregated patent statistics do not distinguish between those that led
                to major innovations and those that led to minor improvements.
                Incompleteness is a further limitation. Many inventions are not patented at
                all, and trade secrecy may become a preferred alternative to patenting.
                Another limitation is that patents do not lend themselves to the evaluation
                of the most significant results achieved by an R&D program. They can
                provide intermediate measures of progress, but they are not usually the
                purpose for which the research was undertaken.

                In addition, the use of patents as a measure of federal R&D effectiveness
                may be hampered by their limited relevance. The 1996 report entitled
                Assessing Fundamental Science by the National Science and Technology
                Council noted that any use of patent counts should be undertaken only
                with a full awareness of their limitations. A recent academic study of
                patents and the evaluation of R&D also commented on the extraordinarily
                limited applicability of patent evaluation to government-performed R&D.
                Both studies pointed to the relatively low level of federal patenting
                activity. The academic study noted that most government laboratories are
                granted only one or two patents per year, and only a few of them patent
                extensively. It concluded that for government laboratories, one may
                question the overall wisdom of evaluating public R&D with private
                techniques.8


Bibliometrics   A third area of effort in developing quantitative measurements involves
                bibliometrics, or the study of published data. Bibliometrics counts
                citations in an attempt to address questions of productivity, influence, and
                the transfer of knowledge. Its most appropriate use is in quantifying the
                work of researchers whose results are published. Thus, it may be
                especially applicable in areas such as basic research where the results are
                more often published than protected by firms. However, its usefulness as a
                measure of research results remains somewhat controversial.

                We believe that the use of bibliometrics as a source of information on the
                quality of the publications or the citations being counted needs to be

                8
                  In its October 11, 1996 issue, Science magazine noted the French government’s announcement that
                patent records will form a part of the evaluation of publicly funded researchers. The new proposal,
                according to the magazine, is controversial because it might upset the proper balance between basic
                and applied research by favoring those who work in fields that are immediately applicable at the
                expense of people doing basic research.



                Page 20                                                 GAO/RCED-97-91 Measuring Performance
              Chapter 3
              R&D Output Indicators Can Provide Limited
              Information About R&D Results




              approached with caution. Although bibliometric indicators can be
              weighted by publication or other quality measure, the frequency of
              citation, for example, provides no indication of the level of research
              innovation. Another limitation is the problems that arise in
              interdisciplinary comparisons of results. Some critics have gone so far as
              to say that bibliometric findings should not be used in science policy work
              until the problems with the analysis of citations are addressed.

              In addition, the relevance of bibliometric analysis to decision-making by
              the federal government appears very limited. One expert noted that a
              recent comprehensive review of bibliometrics shows the sparsity of
              bibliometric studies for evaluations of the impact of research reported by
              the federal government. Another pointed out that few federal agencies use
              bibliometric analysis as an evaluative tool. One of the few is the National
              Institutes of Health (NIH), which uses this method to evaluate the
              effectiveness of its different institutes, the comparative productivity of
              NIH-sponsored research and similar international programs, and the
              responsiveness of NIH’s research programs to their congressional mandate.


Peer Review   Recognizing the limitations of quantitative indicators, the National Science
              and Technology Council concluded that it makes sense to track relevant
              measures but that they cannot supplant the essential element of expert
              judgment. Peer review, the most important form of qualitative assessment,
              uses technical experts to judge R&D results on the basis of the expert’s
              evaluation of the quality of research. However, peer review has serious
              shortcomings; it generally depends on criteria that are inherently difficult
              to measure and on subjective judgment that is vulnerable to bias.

              Peer review has been used extensively in the selection of proposed
              research projects. To a lesser extent, it has also been used to evaluate R&D
              impacts. Peer review has come to be viewed by some observers as the best
              assurance that quality criteria will prevail over social, economic, and
              political considerations, while others view it as an element of elitism in
              science that tends to discount such concerns as economic significance.

              Its major strength is its ability to bring together the leading experts in the
              area of concern. Most peer review procedures require a minimum of three
              reviewers; if the review involves a more ambitious scope of coverage
              (such as an entire agency), dozens of reviewers may be involved. The
              process of selecting the peer reviewers varies. One of the chief
              responsibilities of the professional staff in science agencies such as NSF



              Page 21                                     GAO/RCED-97-91 Measuring Performance
                         Chapter 3
                         R&D Output Indicators Can Provide Limited
                         Information About R&D Results




                         and NIH is to stay in touch with a specialized community of scientists who
                         are qualified to judge the agency’s activities. However, others rely on
                         in-house managers who are not active researchers.

                         The major limitations of peer review are twofold. First, the perception of
                         quality depends largely on the expertise of the panel members. It is based
                         on the judgment of experts about a proposal or a set of research-related
                         results. Generally, a final judgment will depend on the collective weight of
                         the different opinions. Frequently, a numerical rating scale—such as 1 for
                         poor through 5 for excellent—is used. Despite the appearance of precision
                         conferred by a specific number, the numbers represent the best of
                         sometimes widely differing judgments. Consequently, although peer
                         review has been a mainstay in judging science for over three centuries,
                         questions remain about ways of improving it. For instance, according to
                         one academic, to improve “validity and reliability,” research needs to be
                         done on the optimal numbers of reviewers and on the advisability of
                         training people to perform peer reviews.

                         Second, peer review evaluation of completed or ongoing R&D projects is a
                         more thorough and expensive process than peer review for the purpose of
                         selecting proposals for funding. According to one study, the cost of a
                         2-day, 10-person, face-to-face NSF merit review panel is in the
                         neighborhood of $20,000. Another study concluded that if this method
                         were applied annually to all federal research programs, the cost in
                         reviewer time alone would be enormous. For example, the Army Research
                         Laboratory has contracted for a peer review of its activities; the contract
                         calls for a 3-year review directed by the National Research Council at
                         approximately $650,000 per year.


                         The private-sector companies we interviewed varied in terms of the types
The Private Sector’s     of quantitative and qualitative R&D indicators that they collect, but in
Emphasis on              general they made limited use of these indicators in their decisions. Many
Marketplace Results      companies stressed the difficulties involved in measuring the contribution
                         of R&D to the firm’s overall performance using return on investment,
Limits Lessons for the   patents issued, and other R&D output measures. All of the firms mentioned
Federal Government       that they were increasing R&D that contributes directly to the bottom line
                         of the firm. Thus, they shifted the responsibility for R&D decisions to the
                         business-unit level so that the R&D would be tied more directly to the
                         profits of those units. The private sector’s experience offers general
                         lessons to the federal government in terms of ensuring that the R&D




                         Page 22                                     GAO/RCED-97-91 Measuring Performance
                             Chapter 3
                             R&D Output Indicators Can Provide Limited
                             Information About R&D Results




                             contributes directly to the mission of the organization, although the
                             specific output measures do not apply directly to federal R&D efforts.


Companies Cited              Companies told us that measuring the return on R&D investment is very
Difficulties in Measuring    difficult. Companies stated that one factor making measurement more
the Results of R&D           difficult is the long time lag between the research and any revenue that
                             might be earned. Companies also stated that because so many people have
                             been involved in a product’s evolution, it is difficult to separate the
                             contribution of the research unit from that of other units.

                             Companies also mentioned difficulties with some of the other indicators
                             that they track. For example, one indicator was alternatively labeled a
                             “vitality index” or “product innovation metric,” which reflected the share
                             of the firm’s products that could be considered new products. This
                             measure provided an indication of how rapidly the company was
                             incorporating new ideas and research into its products. Several company
                             officials mentioned that this measure had to be applied carefully because
                             of the problem of defining a “new” product; some products are completely
                             different from their predecessors, while others might incorporate cosmetic
                             changes.


Firms Increased Their        One issue that was mentioned by all of the firms in our discussions was
Attention to R&D’s           the increasing emphasis on the relevance of R&D to business needs and the
Contribution to the Bottom   “bottom line.” These comments came up in a number of contexts. For
                             example, some firms mentioned that they look for research with a
Line                         shorter-term payoff in order to have a greater impact on getting new
                             products into the marketplace. Other firms cited the increased emphasis
                             on applied R&D, or the small amount of research that they perform that
                             could be called basic research. A common element was that these firms
                             were attempting to reduce what might be called benevolent research:
                             projects that benefit the industry or the nation but do not have a payoff to
                             the firm.

                             However, a number of the firms noted that this emphasis on the bottom
                             line does not necessarily create the opportunities for breakthrough
                             products. All of the firms either reserved a certain fraction of their
                             research funding for these types of projects or developed processes that
                             made it possible to continue some level of research in areas that might not
                             be directly aligned with any particular product line. In this context, one




                             Page 23                                     GAO/RCED-97-91 Measuring Performance
                            Chapter 3
                            R&D Output Indicators Can Provide Limited
                            Information About R&D Results




                            firm mentioned the importance of cooperation with the federal
                            government and universities in the pursuit of fundamental research.


Firms Have Reduced          While this increased emphasis on getting the most out of R&D might be
Emphasis on R&D Output      expected to lead to greater efforts to measure the results of research, most
Measures                    of the firms that we spoke to responded by changing the organization so
                            that measuring R&D outputs was no longer so important. By shifting the
                            responsibility for research decisions to the business units in the firm that
                            make use of the research outputs, the companies have sidestepped the
                            need for centralized indicators of the quality of research. If the business
                            units believe that a particular R&D project would increase their profits, the
                            firm would budget for that R&D. If the business units do not perform up to
                            expectations in terms of their profitability, the entire unit would be
                            responsible.

                            For example, one firm shifted from a policy of centrally directed research
                            to a policy in which the individual business units make the decisions on
                            the appropriate research projects. Under the previous arrangement, the
                            various business units were assessed to support the central R&D efforts;
                            under the new arrangement, the units pay only for those projects that they
                            think are valuable. This shift was designed to make the research more
                            responsive to the needs of the business units, in that these units do not pay
                            for the research unless they find it useful. This shift also greatly reduced
                            the emphasis on developing R&D output indicators in the central laboratory
                            because each of the business units would be reviewed on the basis of its
                            profitability. The reasoning is that if these units were making poor
                            decisions on R&D projects, the unit’s overall profitability would decrease.


Private Sector’s Measures   Our interviews with private-sector firms suggest that many of the R&D
Do Not Directly Apply to    output measures tracked by the private sector do not apply directly to the
the Federal Government      federal government. Many of these measures are directly related to the
                            contribution of the R&D to the bottom line profitability of the firm.
                            However, federal agencies do not operate in order to make a profit, but to
                            accomplish a variety of other missions and goals. For example, agencies
                            conduct R&D to support a variety of missions, such as maintaining national
                            security or improving citizens’ health. R&D in these areas can contribute
                            greatly to the quality of life in the United States, even if it has a negative




                            Page 24                                     GAO/RCED-97-91 Measuring Performance
Chapter 3
R&D Output Indicators Can Provide Limited
Information About R&D Results




return on investment.9 Given these very different missions, there is also no
reason to believe that any single measure is appropriate for different
public-sector agencies.

Despite the lack of specific measures that can be translated from the
private to the public sector, there are general lessons to be learned from
the private sector’s experience. Possibly the most important is the
recognition that as the pressures on costs increased at many of the firms
that we interviewed, the firms made significant efforts to ensure that R&D
contributed directly to the bottom line. The federal government likewise
faces increasing pressure on costs such as R&D expenditures but, unlike
the private sector, cannot rely on the marketplace to ensure that the R&D
contributes to the agency’s goals. Federal R&D is undertaken to support a
variety of agency missions, and producing an economic return is not the
primary justification for most federal R&D programs. In fact, the purpose of
federal R&D is to promote research that is socially beneficial but may not
be profitable for the private sector to pursue. Without this competitive
marketplace to ensure the relevance of R&D, the federal agencies will
continue to be challenged to develop better measures of the outputs of
their R&D.

The literature confirms this general finding of our discussions. For
example, in 1995 the National Research Council reported on the results of
a workshop on what the federal government might learn from the
corporate experience in research restructuring and assessment.10 The
Council invited senior corporate research managers from IBM, AT&T, Ford,
and Xerox to discuss their experiences in this area. The report concluded
that developing useful metrics and using them appropriately is a difficult
problem, but it is not impossible. In addition, the participants were not
successful in translating their private-sector experience into specific
lessons about what can be measured and what makes sense to measure for
the federal government.




9
 The return-on-investment measure that is appropriate for the federal government would compare the
total “social benefits” to the nation with the costs of the initial investment. Computing this measure
has many of the difficulties of computing the return on investment for private firms, with the added
complications associated with placing a value on national security, improved quality of life, and other
intangible qualities.
10
 Research Restructuring and Assessment: Can We Apply the Corporate Experience to Government
Agencies?, 72 pp., National Research Council (1995).



Page 25                                                  GAO/RCED-97-91 Measuring Performance
                      Chapter 3
                      R&D Output Indicators Can Provide Limited
                      Information About R&D Results




                      Our July 1996 testimony on the implementation of GPRA noted that the
Federal Science       Congress recognized that successful implementation will not come quickly
Agencies Are Still    or easily for many agencies.11 To help address the challenges of
Exploring Ways to     “measuring” the results of R&D programs, the Research Roundtable, a
                      group of federal researchers and managers representing a cross-section of
Measure the Impacts   departments and agencies, has met periodically to share ideas and
of R&D                approaches for implementing GPRA. The Army Research Laboratory has
                      also begun to address this issue in a pilot project for performance
                      measurement under GPRA.

                      The Research Roundtable has considered the extent to which R&D agencies
                      can and should adopt a common approach to measuring performance. In
                      1995, it issued a paper based on 6 months of discussions on the
                      development of performance measures for research programs. Although
                      the Roundtable stated that the results of a research program’s
                      performance can be measured, it cautioned that at the same time, it is
                      important to recognize the complexity of the cause-effect relationship
                      between R&D and its results. It added that this complexity makes it difficult
                      to establish quantifiable measures that consistently measure program
                      performance. It also noted that such measures create a potential for
                      incorrect application, which could lead subsequently to a detrimental
                      effect on scientific endeavors. It warned that quantitative empirical
                      demonstrations of such cause-effect relationships should not be required
                      and are often not even possible.

                      The Army Research Laboratory was designated a pilot project for
                      performance measurement under GPRA. Of the more than 70 such pilot
                      projects governmentwide, the laboratory was the only pilot project that
                      addresses scientific research. As such, it attempted to break new ground
                      in both the planning and the evaluation of basic and applied research. The
                      Chief of the Army Research Laboratory’s Special Projects Office, who is
                      mainly responsible for designing the laboratory’s approach to
                      implementing GPRA, submitted a case study to the Office of Management
                      and Budget in 1996.12 The case study outlines an approach that makes use
                      of “three pillars:” metrics, peer review, and customer feedback.

                      In the case study, the laboratory identified about 60 metrics, most of which
                      measure input using fiscal, facilities, and personnel data. Some of the

                      11
                       Managing for Results: Key Steps and Challenges in Implementing GPRA in Science Agencies
                      (GAO/T-GGD/RCED-96-214, July 10, 1996).
                      12
                       Brown, Edward A. “Applying the Principles of the Government Performance and Results Act to the
                      Research and Development Function” (1996).



                      Page 26                                              GAO/RCED-97-91 Measuring Performance
Chapter 3
R&D Output Indicators Can Provide Limited
Information About R&D Results




metrics, such as tasks completed, patents awarded, and articles published,
measure output; none measure outcome. The laboratory views the
measures as useful tools for understanding the functional health of the
organization and the management of the laboratory but cautions that the
information will not enable it to determine the real quality and impact of
its R&D. The laboratory is relying more heavily on peer review of its
research and on customer surveys for information about quality.

According to the case study, the laboratory has contracted with the
National Research Council of the National Academies of Science and
Engineering to conduct a retrospective peer review of research over a 2-
or 3-year period. The Council is to assemble a Technical Assessment
Board that consists of six panels with 8 to 10 people, each of whom is of
high repute within the technical community. These panels will appraise
the quality of the laboratory’s technical and scientific efforts and, to a
limited extent, productivity.

The case study also described the use of customer feedback. The
laboratory has identified a number of internal and external customers. For
those customers to whom it delivers specific items, it uses a series of
feedback questionnaires to determine their degree of satisfaction on a 1 to
5 scale in terms of the quality, timeliness, and utility of the deliverable. For
those customers who do not receive a specific, identifiable product, the
laboratory is developing a Stakeholders’ Advisory Board of senior
leadership and user representatives to provide first-hand guidance and
feedback.




Page 27                                     GAO/RCED-97-91 Measuring Performance
Appendix I

Historical Perspective on Research
Spending

                   Since World War II, U.S. public and private research and development
                   (R&D) spending patterns have reflected changing priorities as well as
                   reactions to the changing national and international economies. Some of
                   the more prominent events that have shaped spending in both sectors have
                   been the Cold War and the recent international competitive pressures.
                   Declines in spending in both sectors have been less frequent than
                   increases.


                   In his July 1945 report, Science—The Endless Frontier, Vannevar Bush,
Federal Spending   who headed the U.S. R&D effort during World War II, provided the rationale
                   for federal support of both basic research and research related to national
                   security, industry, and human health and welfare. His plan contributed to
                   the legislation adopted in 1950 that established the National Science
                   Foundation (NSF). By that time, however, the National Institutes of Health
                   (NIH) already had control over most health-related research; the Office of
                   Naval Research had taken on a major role in supporting academic
                   research in the physical sciences; and the new Atomic Energy Commission
                   had been assigned control of the R&D on nuclear weapons and nuclear
                   power. NSF’s mission focused on supporting fundamental research and
                   related educational activities. Its annual budget was less than $10 million
                   until the late 1950s. In contrast, NIH’s annual budget, which had been less
                   than $3 million at the end of the war, grew to more than $50 million by
                   1950.

                   The scope of federal R&D support grew in the decade after World War II.
                   Anxiety over the Cold War and the loss in 1949 of the U.S. monopoly in
                   nuclear weapons led to expanded R&D programs in the Army and in the
                   newly established Air Force and to a continuing buildup in support for
                   nuclear weapons R&D in the Atomic Energy Commission. On the civilian
                   side, R&D programs were established or expanded in fields with direct
                   practical importance, such as aeronautics technology, water
                   desalinization, and atmospheric disturbances and weather. The launch of
                   Sputnik by the Soviet Union in 1957 led to immediate efforts to expand
                   U.S. R&D, science and engineering education, and technology deployment.
                   Within months of the Sputnik launch, the National Aeronautics and Space
                   Administration (NASA) and the Advanced Research Projects Agency (ARPA)
                   in the Department of Defense (DOD) were established. NASA’s core included
                   the aeronautics programs of the National Advisory Committee on
                   Aeronautics and some of the space activities of DOD; ARPA’s purpose was to
                   enable DOD to conduct advanced R&D to meet military needs.




                   Page 28                                  GAO/RCED-97-91 Measuring Performance
Appendix I
Historical Perspective on Research
Spending




Federal appropriations for R&D and for mathematics and science education
in NSF and other government agencies rose rapidly over the next decade,
often at double-digit rates in real terms. (See fig. I.1.) During the early
1950s, the growth in federal funding for health research slowed
considerably from its fast pace in the immediate postwar years. However,
in the late 1950s federal support for health research accelerated rapidly.

From 1966 to 1975, federal support for nondefense R&D dropped nearly
22 percent in real terms. The costs of the Vietnam War put downward
pressure on nondefense R&D along with other nondefense discretionary
spending. The conclusion of NASA’s Apollo program contributed to the
decline in federal R&D funding during that period, also. Until recently, this
was the only period in which federal funds for R&D were reduced
substantially.




Page 29                                    GAO/RCED-97-91 Measuring Performance
                                                          Appendix I
                                                          Historical Perspective on Research
                                                          Spending




Figure I.1: U.S. R&D Spending, 1953-96

      Constant 1987 dollars (billions)
160


                                                                                                                                   Total U.S. spending
140




120




100

                                                                                                                                                 Industry

 80




 60
                                                                                                                                                     Federal

 40




 20




  0
         1954   1956   1958   1960   1962   1964   1966   1968   1970   1972   1974   1976   1978   1980   1982   1984   1986   1988   1990   1992   1994   1996

      Year
                                                          Source: National Science Foundation, Science Resources Studies Division.




                                                          In the 1970s, new R&D-intensive agencies addressed environmental and
                                                          energy issues. Both the environmental movement and the energy crisis of
                                                          the 1970s, according to some analysts, led to increased public and private
                                                          spending on environmental and energy R&D. The Environmental Protection
                                                          Agency was established in 1970. In 1977, the Energy Research and
                                                          Development Administration and other federal energy-related activities
                                                          were combined to form the Department of Energy, which was given major
                                                          new responsibilities to fund energy-related R&D.




                                                          Page 30                                                         GAO/RCED-97-91 Measuring Performance
                 Appendix I
                 Historical Perspective on Research
                 Spending




                 In the 1980s, the federal role in R&D expanded to enable the United States
                 to compete in a global market. Out of this atmosphere, several programs
                 were initiated to provide financial and other incentives for industrial R&D
                 and for industrially related R&D conducted at universities or federal
                 laboratories.

                 Two major factors have influenced federal support for R&D funding in the
                 1990s. These factors have been the efforts to reduce the budget deficit and
                 the defense drawdown. The Department of Defense and the Department of
                 Energy, two of the four largest sources of federal R&D support, have had
                 constant-dollar reductions in R&D obligations during the 1990s. In fiscal
                 year 1995, Defense accounted for roughly half of all federal R&D
                 obligations, down from nearly two-thirds of the total in 1986 at the height
                 of the defense build-up that occurred during the Reagan administration.

                 At the same time, military-related R&D spending was being curtailed, while
                 federal investment in selected civilian R&D activities increased, including
                 support for research aimed at improving health and the environment and
                 for technology advancement. The Department of Health and Human
                 Services, which is a distant second to Defense in terms of total R&D
                 support, had the largest absolute increase—$3 billion—in federal R&D
                 obligations during the 1990s. The proportion of all U.S. R&D devoted to
                 health-related projects has been increasing continuously for nearly a
                 decade. The Commerce Department has registered the largest percentage
                 increase in federal R&D obligations during the 1990s so far. In addition, the
                 federal government, which supplies about three-fifths of all funds used to
                 perform R&D on college and university campuses, has been increasing its
                 support of academic research continuously since 1982, even after
                 adjusting for inflation.13


                 During the early post-World War II period, thriving domestic and
Private-Sector   international markets supported large profits and the rapid expansion of
Funding          R&D in both the central laboratories and the divisional laboratories of large
                 companies. Central R&D facilities focused increasingly on fundamental
                 research in many of these large firms, leaving the development and
                 application of new technologies, as well as the improvement of established
                 products and processes, to the divisional laboratories. The data on basic
                 research for 1953 through 1960 are less reliable than those for later years
                 but suggest that the share of total U.S. basic research financed by industry

                 13
                  Much of this section is based on Allocating Federal Funds for Science and Technology by the
                 Committee on Criteria for Federal Support of Research and Development of the National Academy of
                 Sciences, National Academy Press (1995).



                 Page 31                                              GAO/RCED-97-91 Measuring Performance
Appendix I
Historical Perspective on Research
Spending




during the postwar period may well have been at its peak during the 1950s
and early 1960s.

Severe competitive pressures from foreign firms, increases in the real cost
of capital, and a slowdown in the growth rate of the domestic economy in
the 1970s have been cited as causes for the apparent decline in the returns
to R&D investment during the mid-1970s, and the rate of growth in real
industry expenditures on R&D declined. Industry’s funding of basic
research shrank, and many of the central research facilities of the giant
corporations entered a period of budgetary austerity or cutbacks. After a
resurgence in the early 1980s, the rate of growth in industry-funded R&D
declined in the late 1980s.

Industry’s funding of R&D was generally flat between 1991 and 1994;
several reasons have been cited for the lack of growth in some companies’
R&D programs during this time. These include corporate downsizing,
decentralization (i.e., a shift of R&D activity from corporate laboratories to
individual business units), and increasing collaboration among industrial
firms and with partners in academia, in government, in the nonprofit
sector, and in other countries. The preliminary data for 1995 and 1996
indicate a slight upswing in industrial R&D funding.

According to NSF, the most striking recent trend in industrial R&D
performance has been the increase in the proportion of total R&D funded
by the companies classified as nonmanufacturing industries. Prior to 1983,
nonmanufacturing industries accounted for less than 5 percent of the
industry’s total R&D. That share grew steadily during the next decade so
that in 1993, nonmanufacturing firms represented more than 25 percent of
all industrial R&D performed in the United States. Part of this reported
growth, however, reflects improvements in the NSF’s survey efforts to
measure R&D in nonmanufacturing industries, in addition to actual growth
in service sector R&D spending.

Between 1984 and 1994, some significant changes occurred among the 100
largest publicly held R&D funding companies, although the four leading
firms were the same in both 1984 and 1994. During the decade, the number
of pharmaceutical and computer hardware and software companies
among the largest R&D funders rose. In contrast, the number of large
defense contractors and chemical and petroleum companies among the
largest R&D funders fell.




Page 32                                    GAO/RCED-97-91 Measuring Performance
Appendix I
Historical Perspective on Research
Spending




U.S. firms have begun to alter their R&D patterns in response to increasing
competitive pressures. Firms have shifted a greater portion of their R&D
resources away from long-term investments and toward shorter-term
projects. U.S. companies now allocate 22 percent of their R&D spending to
long-term projects when compared with their Japanese counterparts, who
devote 50 percent. Increasingly, firms are emphasizing short-term R&D for
immediate problem-solving or near-term development over basic research;
and basic research is being directed toward the needs of product
development and manufacturing teams. Many central research
laboratories at large companies—such as AT&T, IBM, General Electric,
Kodak, and Xerox—have been downsized and work more closely with
product development divisions. They now receive a larger share of their
operating funds from individual business units rather than general
corporate funds.




Page 33                                  GAO/RCED-97-91 Measuring Performance
Appendix II

Major Contributors to This Report


                        Victor S. Rezendes, Director
Resources,              Robin M. Nazzaro, Assistant Director
Community, and          Andrew J. Vogelsang, Evaluator-in-Charge
Economic                Dennis Carroll, Senior Evaluator

Development
Division, Washington,
D.C.
                        Loren Yager, Assistant Director
Office of the Chief
Economist




(307738)                Page 34                                GAO/RCED-97-91 Measuring Performance
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested