oversight

Measuring Performance: Challenges in Evaluating Research and Development

Published by the Government Accountability Office on 1997-04-10.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                    United States General Accounting Office

GAO                 Testimony
                    Before the Subcommittee on Technology, Committee on
                    Science,
                    House of Representatives


For Release
on Delivery
Expected at
                    MEASURING
10 a.m. EDT
Thursday            PERFORMANCE
April 10, 1997



                    Challenges in Evaluating
                    Research and Development
                    Statement by Allen Li,
                    Associate Director, Energy, Resources, and Science Issues,
                    Resources, Community, and Economic
                    Development Division




GAO/T-RCED-97-130
    Madame Chair and Members of the Subcommittee:

    We are pleased to be here today to discuss our work related to two
    programs at the National Institute of Standards and Technology (NIST): the
    Advanced Technology Program (ATP) and the Manufacturing Extension
    Partnership (MEP) Program. In addition, we will be discussing our work
    performed in response to the Subcommittee’s request regarding research
    and development (R&D) performance measures. This work resulted in our
    report entitled Performance Measurement: Strengths and Limitations of
    Research Indicators (GAO/RCED-97-91), which is being released today in
    conjunction with this hearing.

    Our report on measuring R&D performance highlights the difficulty in
    measuring the impact of technology programs such as ATP and MEP. Our
    work on these programs was performed in response to specific
    congressional concerns. In the case of ATP, we were asked to assess the
    impact of the program. In the case of MEP, we obtained manufacturers’
    views on (1) the impact of the services provided by the program, as well as
    other manufacturing extension programs, on their business performance
    and (2) the factors affecting the impact of these services. In the case of our
    report on performance measurement, this Subcommittee asked us to
    evaluate various indicators that are used to measure the results of R&D.

    In summary, our work showed that:

•   The Advanced Technology Program has funded research projects that
    would have been funded by the private sector as well as those that would
    not.1 The award recipients were nearly evenly divided when asked if they
    would have pursued their projects if they had not received such funding.
    We also found that in most cases, the participants in our survey did not
    look for funding from other sources, private or public, before trying to
    obtain funding from the Advanced Technology Program. About half of the
    45 applicants that tried to find funding elsewhere before turning to the
    Advanced Technology Program were told by prospective funders that their
    projects were either too risky or “precompetitive”2—characteristics that
    fulfill the aims of the Advanced Technology Program.
•   Manufacturers viewed the manufacturing extension programs’ services
    positively, as was demonstrated in our national survey of manufacturers

    1
     Measuring Performance: The Advanced Technology Program and Private Sector Funding
    (GAO/RCED-96-47, Jan. 11, 1996).
    2
     Precompetitive refers to the stage during R&D at which a preliminary assessment of a technology’s
    commercial potential can be made but before commercial prototypes are developed.



    Page 1                                                                        GAO/T-RCED-97-130
                 who had received substantive services from the programs in 1993.3 Most
                 manufacturers responding to our questionnaire—about
                 73 percent—reported that they believed that the type of assistance they
                 had received from these programs had positively affected their overall
                 business performance. About 15 percent of the respondents reported that
                 they believed the programs’ assistance had not affected their overall
                 business performance.4
             •   The amount of money spent on research and development, the primary
                 indicator of research investment, is useful as an input measure of how
                 much research is being performed. However, the level of spending is not a
                 reliable indicator of research results. We found that there is no primary
                 indicator of research and development results. The companies that we
                 spoke with collect data on various output indicators, such as return on
                 investment and patents granted, but in general make limited use of them in
                 their investment decisions. Instead, they emphasized that research and
                 development contributes directly to the bottom line. Because companies
                 are profit oriented, many of the indicators tracked by the private sector
                 cannot be directly applied to the federal government. Determining the
                 specific outcomes resulting from federal research and development is a
                 challenge that will not be easily resolved. However, in response to recent
                 legislation requiring agencies to report on program results, some progress
                 is being made in measuring the impacts of research.


                 ATP’s mission is to stimulate economic growth in the United States through
Background       technology development. The program seeks to accomplish that mission
                 by sharing the cost of R&D projects with private industry. The projects
                 selected by ATP for funding are characterized by the program as having “a
                 potential broad-based economic impact but a relatively high technical risk
                 and a long time horizon.”

                 ATP’s program guidance has stated that if the technical risk associated with
                 a project is very low, federal funding should not be necessary. In addition,
                 when submitting a research proposal, applicants must sign a form stating
                 that “this proposal is not requesting funding for existing or planned
                 research programs that would be conducted in the same time period in the
                 absence of financial assistance under the ATP.” This wording suggests that
                 ATP should not fund projects that other sources would have funded or,



                 3
                  Manufacturing Extension Programs: Manufacturers’ Views About Delivery and Impact of Services
                 (GAO/GGD-96-75, Mar. 14, 1996) and Manufacturing Extension Programs: Manufacturers’ Views of
                 Services (GAO/GGD-95-216BR, Aug. 7, 1995).
                 4
                  Of the remaining 12 percent, 8 percent of the respondents said that it was too early to determine if
                 there was an effect, while 4 percent said they had no basis to estimate an effect.


                 Page 2                                                                           GAO/T-RCED-97-130
                     when ATP does fund such projects, that ATP funds should enable applicants
                     to complete their projects in a shorter time.

                     Manufacturing extension programs offer manufacturers assistance in
                     modernizing or upgrading their operations, often with state and federal
                     funding. NIST manages federal funding of this type of program through its
                     Manufacturing Extension Partnership Program, or MEP. In our prior
                     reports, we used MEP to collectively refer to all state, federal, and
                     university manufacturing extension programs.5

                     The primary mission of manufacturing extension programs is to give
                     “hands-on” technical assistance to small- and medium-sized manufacturers
                     trying to improve their operations through the use of appropriate
                     technologies.6 These programs engage in a variety of activities to assist
                     small- and medium-sized manufacturers, often in partnership with other
                     business assistance providers, such as Small Business Development
                     Centers, community colleges, and federal laboratories. The programs offer
                     a wide range of business services, including helping companies (1) solve
                     individual manufacturing problems, (2) obtain training for their workers,
                     (3) create marketing plans, and (4) upgrade their equipment and
                     computers. The assistance focuses on small- and medium-sized
                     manufacturers because research by the National Research Council and
                     others has indicated that these companies lack the resources necessary to
                     improve their manufacturing performance.


                     In our work on ATP, our objective was to examine, as one way to assess the
The Advanced         program’s impact, whether research projects would have been funded by
Technology Program   the private sector if they had not received funds from ATP.7 We also
                     examined ATP’s impact in terms of other goals of the program, such as
                     aiding the formation of joint ventures.

                     We focused on two groups of ATP applicants, which we called “winners”
                     and “near winners.” Both groups submitted proposals that were rated
                     highest during ATP’s review, but the near winners did not ultimately receive
                     ATP funding. We surveyed all applicants that qualified as winners or near


                     5
                      Manufacturing Extension Programs (GAO/GGD-96-75, Mar. 14, 1996) and Manufacturing Extension
                     Programs (GAO/GGD-95-216BR, Aug. 7, 1995).
                     6
                      The Small Business Administration generally defines a small business as having fewer than 500
                     employees. Some experts have further divided small manufacturers into small firms with fewer than
                     100 employees and medium-sized firms with from 100 to 499 employees.
                     7
                      Measuring Performance (GAO/RCED-96-47, Jan. 11, 1996).



                     Page 3                                                                       GAO/T-RCED-97-130
                    winners during ATP’s first 4 years (1990-93). We achieved a 100-percent
                    response rate from the 123 respondents that we included in our analysis
                    (89 winners and 34 near winners).

                    We found that ATP had funded research projects that would have been
                    funded by the private sector as well as those that would not. The winners
                    were nearly evenly divided when asked if they would have pursued their
                    projects even if they had not received ATP funding. Half of the near winners
                    continued their projects without relying on ATP funding, while the other
                    half discontinued their projects for various reasons. Almost all the near
                    winners that continued their projects did so on a modified schedule,
                    meeting the projects’ milestones later than they had scheduled in their
                    proposals to ATP.

                    Of the 123 applicants we surveyed, 77,8 or 63 percent, did not look for
                    funding from other sources before requesting it from ATP. Those applicants
                    that did look for funding looked for a long time and made many attempts
                    to find funding, on average. Seven applicants turned down offers from
                    private sources because they could not reach an acceptable funding
                    arrangement.

                    We also found that ATP had other effects. More than three-fourths of the
                    joint-venture applicants indicated that they had come together solely to
                    pursue an ATP project, thus satisfying ATP’s goal of serving as a catalyst for
                    the formation of joint ventures. Furthermore, of the 45 applicants that
                    tried to find funding elsewhere before turning to ATP, about half were told
                    by prospective funders that their projects were either too risky or
                    precompetitive—characteristics that fulfill the aims of ATP funding.


                    We surveyed 766 U.S. manufacturers that had completed at least 40 hours
Manufacturing       of manufacturing extension program assistance, including NIST’s MEP, and
Extension Program   received 551 responses.9 We obtained respondents’ views on the impact of
Services            these services on their business performance and on the factors affecting
                    the impact of these services. We did not verify either the positive or
                    negative impacts reported by manufacturers, nor did we evaluate the
                    operations or management of specific federal or state programs. We also


                    8
                     One applicant did not know.
                    9
                     Thirteen of these manufacturing extension programs received NIST funding in fiscal year 1994. These
                    13 programs accounted for 36 percent of the 551 total respondents to our survey. See Manufacturing
                    Extension Programs (GAO/GGD-96-75, Mar. 14, 1996) and Manufacturing Extension Programs
                    (GAO/GGD-95-216BR, Aug. 7, 1995).



                    Page 4                                                                        GAO/T-RCED-97-130
obtained the views of other manufacturers that had little or no experience
with these programs to determine why they made little or no use of them.

Most manufacturers responding to our questionnaire—about
73 percent—reported that they believed the programs’ assistance had
positively affected their overall business performance. About 15 percent of
the respondents reported that they believed the assistance had not
affected their overall business performance. Approximately 8 percent said
that it was too early to determine the effect, and another 4 percent said
they had had no basis to estimate the effect.

In addition, most respondents reported that the assistance had positively
affected their use of technology in the workplace (about 63 percent), the
quality of their product (about 61 percent), and the productivity of their
workers (about 56 percent). Between about 44 percent and 63 percent of
the respondents reported that the programs’ assistance had positively
affected certain specific indicators of their business performance, such as
customer satisfaction, profits, and the ability to meet production
schedules. Of those respondents not reporting a positive impact on
specific indicators of their business performance, most said the programs’
assistance had not had any impact. Two percent or fewer of the
respondents reported a negative impact on each specific performance
indicator.

Among the factors that manufacturers said had affected the impact of MEP
services was their own companies’ input. The companies that had
committed their own financial resources to implement the programs’
recommendations reported greater benefits from the assistance relative to
other survey respondents. Of those 322 respondents who had made a
financial investment, 86 percent said that the programs’ assistance had
positively affected their business performance. However, 54 percent of
those who had not made a financial investment also reported an overall
positive impact. Other factors, according to the respondents, that
influenced the effectiveness of the programs’ services were the expertise
and experience of the programs’ staff and the affordability of the
assistance.

In our related telephone survey of 200 additional manufacturers who were
not extensive users of the programs’ services, about 82 percent reported
that they had not used the services because they were unaware of these
programs. About 10 percent said that although they knew about these
programs, they had not used them because they believed the assistance



Page 5                                                    GAO/T-RCED-97-130
                     would not be necessary. The companies we interviewed said that other
                     sources of modernization assistance besides these programs were their
                     customers, vendors and/or suppliers, industry associations, and
                     consultants.


                     The report we are releasing today on performance measurement shows
Measuring the        that there is no single indicator or evaluation method that adequately
Impacts of Federal   captures the results of R&D. However, indicators do provide helpful
Research Programs    information for making decisions about R&D. Whether the focus is on
                     basic research, applied research, or development, the amount of money
                     spent in that area is taken as an indication of how much research is being
                     performed. The major advantages of using expenditure data as an
                     indicator are that they are easily understandable, readily available, and
                     have been, in general, consistently gathered over time. In addition,
                     spending on different projects in different research areas can be measured
                     according to the same unit, dollars, making comparisons between projects
                     straightforward.

                     The amount of funding, however, does not provide a good indication of
                     research results. Companies told us that they are switching their spending
                     to more short-term R&D projects rather than long-term projects. However,
                     the impacts of that change are unclear. The reduced funding levels for
                     long-term projects may not reflect the fact that the R&D efforts can be
                     performed with greater efficiency. For example, one way in which the
                     federal government and the private sector have tried to use R&D resources
                     more efficiently and effectively is through consortia with universities or
                     other companies. By combining their research activities, companies
                     attempt to avoid expensive duplication and learn from each other.

                     We also found that because of the difficulties in identifying the impacts of
                     research, quantitative and qualitative indicators have been developed as
                     proxies to assess R&D results. The strengths and limitations are evident in
                     both types of indicators. Quantitative indicators focus mainly on return on
                     investment, patenting rates, and bibliometrics—the study of
                     publication-based data. While implying a degree of precision, these
                     indicators were not originally intended to measure long-term R&D results.
                     Qualitative assessment such as peer review provides detailed information,
                     but it relies on the judgments of experts and may be expensive.

                     Because of these difficulties, the companies we interviewed stressed
                     marketplace results rather than R&D output indicators. While varying in



                     Page 6                                                     GAO/T-RCED-97-130
the types of indicators they collect, they emphasized the difficulties in
measuring R&D’s specific contribution to a company’s overall
performance. For example, one company stated that because so many
people have been involved in a product’s evolution, it is difficult to
separate the contribution of the research unit from that of other units. All
of the companies interviewed have increased their expectation that R&D
contribute directly to their profitability. However, instead of increasing
their efforts at measuring R&D results, they have shifted the responsibility
for making R&D decisions to the business units. For example, if the
business units believe that a particular R&D project would increase their
profits, the firm would budget for that R&D.

Many of the R&D output measures tracked by the private sector do not
apply directly to the federal government. In particular, while facing the
same increasing cost pressures as the private sector, the federal
government cannot rely on the profit motive to guide its decisions.

This discussion of performance measures for R&D is particularly relevant
because of the current emphasis on the Government Performance and
Results Act (GPRA). In response to questions about the value and
effectiveness of federal programs, GPRA seeks to shift federal agencies’
focus away from such traditional concerns as staffing, activity levels, and
tasks completed toward a focus on program outcomes. GPRA incorporates
performance measurement as one of its most important features. Under
this act, executive branch agencies are required to develop annual
performance plans that use performance measurement to reinforce the
connection between the long-term strategic goals outlined in their
strategic plans and the day-to-day activities of their managers and staff.
However, the very nature of the innovative process makes measuring the
performance of science-related projects difficult. For example, a wide
range of factors determines if and when a particular R&D project will
result in commercial or other benefits. It can also take many years for a
research project to achieve results.

Experiences from recent GPRA pilot efforts reinforce the fact that output
measures are highly specific to the management and mission of each
federal agency and that no single indicator exists to measure the results of
research. The Army Research Laboratory, which was designated as a pilot
project for performance measurement under the act, has developed a
multifaceted approach using quantitative indicators, peer review, and
customer feedback to evaluate the results of R&D. Although this is not the
only approach that can be taken, this response to the challenges in



Page 7                                                     GAO/T-RCED-97-130
           measuring the impacts of research shows that some progress is being
           made in response to GPRA.

           Madame Chair, this concludes my prepared remarks. I would be happy to
           respond to any questions you may have.




(141046)   Page 8                                                  GAO/T-RCED-97-130
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested