oversight

Federal Research: Evaluation of Small Business Innovation Research Can Be Strengthened

Published by the Government Accountability Office on 1999-06-17.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                    United States General Accounting Office

GAO                 Testimony
                    Before the Subcommittee on Technology,
                    Committee on Science
                    House of Representatives


For Release
on Delivery
Expected at
                    FEDERAL RESEARCH
1 p.m. EDT
Thursday
June 17, 1999
                    Evaluation of Small
                    Business Innovation
                    Research Can Be
                    Strengthened
                    Statement of Susan D. Kladiva, Associate Director,
                    Energy, Resources, and Science Issues,
                    Resources, Community, and Economic
                    Development Division




GAO/T-RCED-99-198
    Madam Chair and Members of the Subcommittee:

    We are here today to present the results of our review1 of the Small
    Business Innovation Research (SBIR) program, which we performed for
    your subcommittee. As a nation competing in a global economy, the
    United States depends heavily on innovation through research and
    development. The Small Business Innovation Development Act of 1982,
    which authorized the SBIR program, directs agencies with sizeable external
    research and development budgets to set aside a portion of these budgets
    for projects by eligible small businesses. It emphasizes the benefits of
    technological innovation and the ability of small businesses to transform
    the results of research into new products. In reauthorizing the program in
    1992, the Congress stated its intention to expand and improve the
    program, emphasize the program’s goal of increasing the private sector’s
    commercialization of technology developed through federal research and
    development, and improve the federal government’s dissemination of
    information on the program.

    In its 16 years, the program has provided over 45,000 awards worth
    $8.4 billion in 1998 dollars to thousands of small high-technology
    companies. In the 1990s, congressional concern has focused on the
    companies’ ability to commercialize the results of their research and on
    the concentration of awards in certain states and companies—commonly
    known as “frequent winners.” Concern about frequent winners has arisen,
    in part, because studies conducted by us and the Department of Defense
    indicate that frequent winners generally achieve lower levels of
    commercialization than companies winning fewer awards.

    The report we are releasing today discusses

•   the distribution of awards by company and geographic area, with special
    emphasis on the share of awards received by the 25 most frequent
    winners;
•   the extent to which federal agencies are considering commercial potential
    and the program’s other goals in making their awards; and
•   previous evaluations of the SBIR program to identify an opportunity to
    improve measurements of the program’s outcomes.

    Our statement today highlights the message of our review. In summary,
    Madam Chair:

    1
     Federal Research: Evaluation of Small Business Innovation Research Can Be Strengthened,
    (GAO/RCED-99-114, June 4, 1999).



    Page 1                                                                      GAO/T-RCED-99-198
The 25 most frequent winners, which represent fewer than 1 percent of the
companies in the program, received about 11 percent of the program’s
awards from fiscal year 1983 through fiscal year 1997. These companies
accumulated over $900 million in total awards; the leading frequent winner
received over $108 million. However, one-third of the companies receiving
awards from fiscal year 1993 through fiscal year 1997 were first-time
winners, indicating that the program is attracting an average of 750 new
companies annually. In our view, this level of participation by first-time
winners is indicative of a substantial capacity to attract new participants
each year.

In response to the 1992 reauthorization, agencies are considering
commercial potential as a criterion when evaluating proposals and
collecting data on commercialization by frequent winners. However, the
reauthorization does not clarify how much weight should be given to the
commercialization record as part of the goal of commercialization and
how much weight should be given to the program’s other goals, such as
technological innovation or importance to an agency’s mission. This lack
of clarity has led to differences in agencies’ evaluation approaches. For
example, using an approach shared by none of the other agencies, the
Department of Defense planned to give significantly lower scores to
companies perceived as poor commercializers, but we found that this
approach would penalize companies with relatively few awards and no
sales but would not penalize frequent winners with limited sales. Defense
has revised its approach to avoid these unintended consequences. Our
report raises as a matter for congressional consideration how the
commercialization record as part of the goal of commercialization should
be balanced against the program’s other goals in evaluations of proposals.

Federal agencies and others have relied on various methods to evaluate
the program’s commercial outcomes. These methods have used
“snapshots” of sales, data on additional developmental funding for
projects, “success stories,” and other indicators of success. However, they
become quickly outdated and do not provide an ongoing, consistent, and
programwide record. The use of a single method with uniform criteria for
success focusing on commercial and other outcomes would help to satisfy
the requirements of the Government Performance and Results Act.2 The
Small Business Administration (SBA) is currently developing a new
database called Tech-Net, which is scheduled for implementation in 1999.
Tech-Net affords an opportunity to maintain current, consistent

2
 In December 1997, the Congress specified that information on the SBIR program must be included by
each federal agency in the updates or revisions to its strategic plan required by the Results Act. 15
U.S.C. 638(t).



Page 2                                                                         GAO/T-RCED-99-198
                        information on commercial outcomes and other indicators of success in
                        response to the Results Act. Our report recommends that the SBA
                        Administrator develop standard criteria for measuring the commercial and
                        other outcomes of the SBIR program and incorporate these criteria into the
                        new Tech-Net database. Let me be more specific about each of these
                        issues.


                        To analyze the distribution of SBIR awards by company, we divided the
Frequent and            winners of phase II awards3 into three groups: the 25 companies with the
Infrequent Winners      most awards, the companies with between 1 and 4 awards, and a middle
Are Major SBIR          group of companies with between 5 and 27 awards. Figure 1 presents the
                        distribution of phase II awards to these three groups from fiscal year 1984,
Players, but Frequent   when the first phase II awards were made, through fiscal year 1997, the
Winners and Certain     latest year for which complete data are available.
States Have Won a
Large Share of the
Program’s Resources




                        3
                        Phase II awards follow Phase I awards and are designed to further develop the scientific and technical
                        merit and the feasibility of research ideas.



                        Page 3                                                                         GAO/T-RCED-99-198
Figure 1: Percentage of Phase II Awards Won by Various Groups of Participants, Fiscal Years 1984-97




                 7




                                          Source: GAO’s analysis of data from SBA’s SBIR database.




                                          This figure bears out concerns about the concentration of awards,
                                          showing that nearly half of the program’s awards have gone to just over a
                                          tenth of the participants represented by the top and middle groups.
                                          (Additional information about the 25 most frequent winners is provided in
                                          appendix I.) However, the figure also indicates wide participation,
                                          showing that thousands of other companies have received only one or just
                                          a few awards. Data for fiscal years 1993 through 1997 further indicate that
                                          the program is attracting an average of over 750 new companies each year.

                                          Concern about the concentration of awards has also focused on their
                                          geographic distribution. SBA has found that the distribution of SBIR awards
                                          generally resembles the distribution of non-SBIR expenditures for research
                                          and development, venture capital investments, and academic research
                                          funds. Companies in a small number of states, especially California and
                                          Massachusetts, have won the majority of awards, largely because they
                                          have submitted the most proposals. Data for fiscal year 1998 show that
                                          proposals from companies in states with historically lesser amounts of



                                          Page 4                                                      GAO/T-RCED-99-198
                        federal research funding won awards at almost the same rate as proposals
                        from companies in other states.

                        To encourage greater participation by companies in states with fewer
                        awards, the National Science Foundation has used a program it
                        established about 20 years ago to support research in states with
                        historically lesser amounts of federal research funding. The Foundation’s
                        Experimental Program to Stimulate Competitive Research (EPSCoR) began
                        in 1981 and was funded at about $49 million in fiscal year 1999. The
                        Foundation has used this program to increase the number of SBIR awards
                        to small businesses in states that have received lesser amounts of federal
                        R&D funding. Eighteen states and the Commonwealth of Puerto Rico
                        participate in the program.4 Since 1994, EPSCoR has awarded 82 phase I SBIR
                        grants valued at over $7 million. Other agencies also have such programs
                        but have not used them to assist their SBIR participants. Several agencies
                        are considering such an initiative to increase their outreach efforts in the
                        SBIR program.



                        In reauthorizing the program in 1992, the Congress emphasized
Agencies Are            commercialization, requiring agencies to consider commercial potential in
Considering             making awards and to collect data on commercialization by companies
Commercial Potential    that have received 15 or more phase II awards during the preceding 5
                        years. We found that, in response, agencies are weighing the commercial
in Making Awards, but   potential of all proposals and collecting data on commercialization by
the Emphasis on         companies, including frequent winners. However, neither the 1992
                        reauthorization nor a 1993 SBA policy directive on implementing the
Commercialization       legislation clarifies for the agencies what weight they are to assign to
Raises Questions        commercial potential relative to the program’s other goals in evaluating
About the Relative      proposals or how they are to use the data they collect on companies’
                        commercialization results. Without clarification, the agencies are
Importance of the       developing different evaluation approaches, at least one of which would
Program’s Other         have had unintended consequences.
Goals




                        4
                         The states are Alabama, Arkansas, Idaho, Kansas, Kentucky, Louisiana, Maine, Mississippi, Montana,
                        Nebraska, Nevada, North Dakota, Oklahoma, South Carolina, South Dakota, Vermont, West Virginia,
                        and Wyoming.



                        Page 5                                                                        GAO/T-RCED-99-198
Agencies Are Considering    As described in the 1992 act, commercial potential has four indicators, one
Commercial Potential to     of which is a company’s commercialization record.5 Our review showed
Varying Degrees in Making   that agencies are considering commercial potential, including a company’s
                            commercialization record and the other indicators. Together, these four
Awards                      indicators can account for as much as one-third of a proposal’s total score.
                            However, the commercialization record alone has played a limited role
                            because it is only one indicator of commercial potential that, in turn, is
                            one among several criteria such as technical merit considered when
                            evaluating proposals. At the Department of Defense, for example, the
                            commercialization record currently accounts for about one-fourth of the
                            commercial potential score and about one-twelfth of the total score for a
                            proposal; at the Department of Energy, it accounts for about
                            one-eighteenth of the total score. Thus, even if a company has a poor
                            commercialization record, this factor has exercised only a limited
                            influence on agencies’ evaluations of proposals to date.

                            According to SBA’s Assistant Administrator for Technology, the 1992
                            reauthorization directs participating agencies to collect information on
                            commercialization by companies with 15 or more phase II awards during
                            the previous 5 years but does not clarify how they are supposed to use it.
                            In response, the agencies have collected data on commercialization by
                            companies, including frequent winners. Because the 1992 act and the SBA
                            policy directive do not address how to use the information on
                            commercialization records, differences among the agencies have emerged.
                            The Department of Defense, for example, recently planned to implement a
                            new evaluation approach designed to give greater prominence to the
                            commercialization records of companies with 5 or more phase II awards.
                            This approach would have led to significantly lower scores for companies
                            with relatively few awards and no sales but would not have penalized
                            frequent winners with only modest sales. The Department has since
                            revised its approach to avoid these unintended consequences by taking
                            into account the concept of statistical significance as it relates to
                            companies with widely varying numbers of awards.




                            5
                             Under the 1992 reauthorization, commercial potential was evidenced by “(i) the small business
                            concern’s record of successfully commercializing SBIR or other research; (ii) the existence of second
                            phase funding commitments from private sector or non-SBIR funding sources; (iii) the existence of
                            third phase, follow-on commitments for the subject of the research; and (iv) the presence of other
                            indicators of the commercial potential of the idea.” 15 U.S.C. 638(e)(4)(B).



                            Page 6                                                                         GAO/T-RCED-99-198
The Emphasis on            Despite the greater emphasis on commercialization since 1992, the
Commercialization Raises   program’s other goals, such as innovation and responsiveness to an
Questions About the Role   agency’s needs, remain important to the agencies when evaluating a
                           company’s accomplishments and subsequent proposals. According to
of Other Goals in          some of the program managers, a relatively low level of commercialization
Evaluating Companies’      may not signal failure because a company may have achieved other goals.
Performance                The difficulty for agencies of using any particular goal as a key criterion
                           for selecting future proposals for funding stems from their not having (1) a
                           clear definition of the program’s goals, (2) information on the relative
                           weight that should be given to these potential goals, and (3) criteria for
                           judging whether the goals have been achieved.

                           Finding practical ways to define and measure the SBIR program’s goals in
                           order to evaluate proposals has been difficult. For example, efforts to
                           define and measure technological innovation have posed a challenge.
                           Because technological innovation occurs in many different ways, no one
                           indicator is an accurate measure of it. In addition, according to SBA’s
                           Assistant Administrator for Technology, the 1992 reauthorization lacks a
                           clear definition of “commercialization,” and he has sometimes differed
                           with agencies on its meaning. This absence of a definition makes it more
                           difficult, in his view, to determine when a frequent winner is “failing” to
                           achieve a sufficient level of commercialization.

                           The relative weight that should be given to the goals when evaluating
                           proposals remains unclear. Innovation and responsiveness to an agency’s
                           needs, for example, may compete with commercialization. In the view of
                           many program managers, innovation involves a willingness to undertake
                           R&D with a higher element of risk and a greater chance that it may not
                           lead to a commercial product; responsiveness to an agency’s needs
                           involves R&D that may be aimed at special niches with likewise limited
                           commercial potential. According to the program managers, the challenge
                           lies in striking the right balance between encouraging new, unproven
                           technologies and achieving commercial sales.

                           Agencies have also not agreed on criteria for “success” in meeting the
                           program’s goals. In general, it is difficult to determine the appropriate mix
                           of higher-risk projects that lead less frequently to commercial outcomes
                           and of lower-risk projects that lead more frequently to successful
                           products. The difficulty caused by not having criteria is compounded by
                           the fact that only a handful of projects in the program achieve substantial
                           commercial success. Only about 1.5 percent of the projects account for
                           about half of the sales, and 4 percent of the projects account for about



                           Page 7                                                      GAO/T-RCED-99-198
                             75 percent of the sales. When the great majority of projects achieve no
                             sales or only very limited sales, evaluating subsequent proposals from
                             individual companies becomes more difficult if commercialization is
                             considered the primary goal.


                             Commercialization is only one of the program’s objectives but has become
SBA Has an                   the main outcome for measuring its effectiveness. Studies of
Opportunity to               commercialization have proliferated as agencies have tried to obtain data
Standardize                  on commercial activity. Although these studies rely on different
                             approaches, they contain some common criteria for success, such as levels
Evaluations of the           of sales and developmental funding. Expanding the SBIR database at SBA to
Program’s Outcomes           include information on commercial outcomes and other indicators of
                             success, such as savings to agencies resulting from SBIR projects, would
                             help to standardize evaluations of the program. As SBA develops a new
                             database, called Tech-Net, which is scheduled for full implementation in
                             1999, it has an opportunity to include outcome-related measures that can
                             be used to track commercialization and other indicators of success.


Various Methods With         Our 1992 report on the SBIR program6 responded to a congressional
Similar Criteria Have Been   mandate that we assess the commercial outcomes of the program. We
Used in Attempting to        surveyed companies that had won phase II awards from 1984 through
                             1987. Our survey asked, for example, “Has the technology associated with
Measure Outcomes             this project led to additional developmental funding and/or sales, and is
                             further work on this technology under way?” This approach was also used
                             in later surveys of the program, including those conducted by a support
                             contractor for DOD in 1996 and for SBA in 1998. Although the Department of
                             Energy used a different evaluation approach, it also focused on outcomes,
                             seeking information on the results of SBIR technology rather than on
                             individual awards.

                             Another common evaluation approach, used by the National Science
                             Foundation, DOD, the National Aeronautics and Space Administration, and
                             other agencies, relies on success stories stemming from the agencies’
                             awards. Although the agencies have used these stories to document the
                             most significant results of their SBIR awards and to help companies market
                             their technologies, they do not follow a consistent method and they ignore
                             less successful projects, biasing the results of this approach. In our view,
                             the approach is “open-ended,” meaning that it can be used to develop a

                             6
                              Federal Research: Small Business Innovation Research Shows Success but Can Be Strengthened
                             (GAO/RCED-92-37, Mar. 30, 1992).



                             Page 8                                                                     GAO/T-RCED-99-198
                           detailed story for an individual company but does not lend itself to
                           systematization.


A Standard Approach        The evaluation methods we identified do not provide consistent
Involves the Use of        information across agencies on the program’s results. The use of a single
Uniform Criteria for       method with uniform criteria focusing on outcomes would produce such
                           information, enabling SBA and the agencies to satisfy the requirements of
Success and Improvements   the Results Act.
in SBA’s New Database
                           As the central administrative agency for the program, SBA has maintained a
                           governmentwide database that brings together the data submitted by the
                           individual agencies participating in the program. This database has had
                           two major shortcomings. First, because it was developed long before the
                           Results Act emphasized the measurement of outcomes, the database
                           reflects the earlier attention given to inputs, such as the name of each
                           company and the amount of funding it has received. In general, companies
                           have not provided information on the actual results of their research.
                           Second, the database has contained unreliable information because it has
                           lacked a unique identifying code for each company. Slight variations in the
                           spelling of a company’s name have created difficulty because the database
                           has counted each separate spelling as a separate company.

                           In June 1998, SBA announced the introduction of a new database called
                           Tech-Net. SBA database managers and SBIR program managers are
                           optimistic about their ability to expand Tech-Net to capture
                           outcome-related data. Tech-Net will enable agencies to update their
                           information on SBIR awards and companies to update key information
                           about their activities. As a result, companies will be able to provide more
                           information on the actual results of their SBIR awards. In addition, SBA
                           plans to assign each company a unique identification number to eliminate
                           the confusion about the identity of participants.


                           Madam Chair, this concludes my statement. I would be happy to answer
                           any questions you or the other Members of the Subcommittee may have.




                           Page 9                                                     GAO/T-RCED-99-198
Appendix I

An Overview of the Top 25 Frequent
Winners, Fiscal Years 1983-97


              Dollars in millions
                                                                 Percentage of
                                                                annual revenue
                                    Phase II     Total   Dollar     from SBIR
              Company                awards    awards    value           (1998)
              Foster Miller             147       573    108.2              20
              Physical Optics            96       377     71.2              68
              Creare                     87       281     61.4              64
              Physical Sciences          76       290     57.2              42
              Spire                      75       351     59.4              26
              Radiation
              Monitoring
              Devices                    59       187     43.3              38
              Bend Research              58       166     34.3              23
              EIC Laboratories           53       188     38.1              33
              Mission Research           50       196     39.8               8
              Science
              Research
              Laboratory                 49       147     33.4              76
              Advanced
              Technology
              Materials                  48       208     38.4              10
              Advanced Fuel
              Research                   42       154     27.8              52
              Ultramet                   38       140     28.4              37
              Aerodyne
              Research                   35       134     27.5              36
              CFD Research               35       107     24.7              52
                                                                              a
              Sparta                     35       162     28.3
              TDA Research               35       127     19.5              70
                                                                              a
              Thermacore                 35       102     25.8
              American
              Research Corp.
              of Virginia                34       102     19.3              80
              Waterjet
                                                                              a
              Technology                 34       102     21.5
              Scientific
              Research
                                                                              a
              Associates                 33       113     24.0
              Giner                      30       110     22.1              70
              Schwartz
              Electro-optics             30       104     20.1               6
                                                                    (continued)




              Page 10                                       GAO/T-RCED-99-198
           Appendix I
           An Overview of the Top 25 Frequent
           Winners, Fiscal Years 1983-97




           Dollars in millions
                                                                              Percentage of
                                                                             annual revenue
                                            Phase II         Total    Dollar     from SBIR
           Company                           awards        awards     value           (1998)
           Bio-Metric
                                                                                           a
           Systems                               29              89    18.5
           Satcon
           Technology                            28             119    22.2              44

           a
               Information was not available.

           Source: GAO’s analysis of data from SBA’s SBIR database.




(141335)   Page 11                                                       GAO/T-RCED-99-198
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested