United States General Accounting Office GAO Testimony Before the Subcommittee on Technology, Committee on Science House of Representatives For Release on Delivery Expected at FEDERAL RESEARCH 1 p.m. EDT Thursday June 17, 1999 Evaluation of Small Business Innovation Research Can Be Strengthened Statement of Susan D. Kladiva, Associate Director, Energy, Resources, and Science Issues, Resources, Community, and Economic Development Division GAO/T-RCED-99-198 Madam Chair and Members of the Subcommittee: We are here today to present the results of our review1 of the Small Business Innovation Research (SBIR) program, which we performed for your subcommittee. As a nation competing in a global economy, the United States depends heavily on innovation through research and development. The Small Business Innovation Development Act of 1982, which authorized the SBIR program, directs agencies with sizeable external research and development budgets to set aside a portion of these budgets for projects by eligible small businesses. It emphasizes the benefits of technological innovation and the ability of small businesses to transform the results of research into new products. In reauthorizing the program in 1992, the Congress stated its intention to expand and improve the program, emphasize the program’s goal of increasing the private sector’s commercialization of technology developed through federal research and development, and improve the federal government’s dissemination of information on the program. In its 16 years, the program has provided over 45,000 awards worth $8.4 billion in 1998 dollars to thousands of small high-technology companies. In the 1990s, congressional concern has focused on the companies’ ability to commercialize the results of their research and on the concentration of awards in certain states and companies—commonly known as “frequent winners.” Concern about frequent winners has arisen, in part, because studies conducted by us and the Department of Defense indicate that frequent winners generally achieve lower levels of commercialization than companies winning fewer awards. The report we are releasing today discusses • the distribution of awards by company and geographic area, with special emphasis on the share of awards received by the 25 most frequent winners; • the extent to which federal agencies are considering commercial potential and the program’s other goals in making their awards; and • previous evaluations of the SBIR program to identify an opportunity to improve measurements of the program’s outcomes. Our statement today highlights the message of our review. In summary, Madam Chair: 1 Federal Research: Evaluation of Small Business Innovation Research Can Be Strengthened, (GAO/RCED-99-114, June 4, 1999). Page 1 GAO/T-RCED-99-198 The 25 most frequent winners, which represent fewer than 1 percent of the companies in the program, received about 11 percent of the program’s awards from fiscal year 1983 through fiscal year 1997. These companies accumulated over $900 million in total awards; the leading frequent winner received over $108 million. However, one-third of the companies receiving awards from fiscal year 1993 through fiscal year 1997 were first-time winners, indicating that the program is attracting an average of 750 new companies annually. In our view, this level of participation by first-time winners is indicative of a substantial capacity to attract new participants each year. In response to the 1992 reauthorization, agencies are considering commercial potential as a criterion when evaluating proposals and collecting data on commercialization by frequent winners. However, the reauthorization does not clarify how much weight should be given to the commercialization record as part of the goal of commercialization and how much weight should be given to the program’s other goals, such as technological innovation or importance to an agency’s mission. This lack of clarity has led to differences in agencies’ evaluation approaches. For example, using an approach shared by none of the other agencies, the Department of Defense planned to give significantly lower scores to companies perceived as poor commercializers, but we found that this approach would penalize companies with relatively few awards and no sales but would not penalize frequent winners with limited sales. Defense has revised its approach to avoid these unintended consequences. Our report raises as a matter for congressional consideration how the commercialization record as part of the goal of commercialization should be balanced against the program’s other goals in evaluations of proposals. Federal agencies and others have relied on various methods to evaluate the program’s commercial outcomes. These methods have used “snapshots” of sales, data on additional developmental funding for projects, “success stories,” and other indicators of success. However, they become quickly outdated and do not provide an ongoing, consistent, and programwide record. The use of a single method with uniform criteria for success focusing on commercial and other outcomes would help to satisfy the requirements of the Government Performance and Results Act.2 The Small Business Administration (SBA) is currently developing a new database called Tech-Net, which is scheduled for implementation in 1999. Tech-Net affords an opportunity to maintain current, consistent 2 In December 1997, the Congress specified that information on the SBIR program must be included by each federal agency in the updates or revisions to its strategic plan required by the Results Act. 15 U.S.C. 638(t). Page 2 GAO/T-RCED-99-198 information on commercial outcomes and other indicators of success in response to the Results Act. Our report recommends that the SBA Administrator develop standard criteria for measuring the commercial and other outcomes of the SBIR program and incorporate these criteria into the new Tech-Net database. Let me be more specific about each of these issues. To analyze the distribution of SBIR awards by company, we divided the Frequent and winners of phase II awards3 into three groups: the 25 companies with the Infrequent Winners most awards, the companies with between 1 and 4 awards, and a middle Are Major SBIR group of companies with between 5 and 27 awards. Figure 1 presents the distribution of phase II awards to these three groups from fiscal year 1984, Players, but Frequent when the first phase II awards were made, through fiscal year 1997, the Winners and Certain latest year for which complete data are available. States Have Won a Large Share of the Program’s Resources 3 Phase II awards follow Phase I awards and are designed to further develop the scientific and technical merit and the feasibility of research ideas. Page 3 GAO/T-RCED-99-198 Figure 1: Percentage of Phase II Awards Won by Various Groups of Participants, Fiscal Years 1984-97 7 Source: GAO’s analysis of data from SBA’s SBIR database. This figure bears out concerns about the concentration of awards, showing that nearly half of the program’s awards have gone to just over a tenth of the participants represented by the top and middle groups. (Additional information about the 25 most frequent winners is provided in appendix I.) However, the figure also indicates wide participation, showing that thousands of other companies have received only one or just a few awards. Data for fiscal years 1993 through 1997 further indicate that the program is attracting an average of over 750 new companies each year. Concern about the concentration of awards has also focused on their geographic distribution. SBA has found that the distribution of SBIR awards generally resembles the distribution of non-SBIR expenditures for research and development, venture capital investments, and academic research funds. Companies in a small number of states, especially California and Massachusetts, have won the majority of awards, largely because they have submitted the most proposals. Data for fiscal year 1998 show that proposals from companies in states with historically lesser amounts of Page 4 GAO/T-RCED-99-198 federal research funding won awards at almost the same rate as proposals from companies in other states. To encourage greater participation by companies in states with fewer awards, the National Science Foundation has used a program it established about 20 years ago to support research in states with historically lesser amounts of federal research funding. The Foundation’s Experimental Program to Stimulate Competitive Research (EPSCoR) began in 1981 and was funded at about $49 million in fiscal year 1999. The Foundation has used this program to increase the number of SBIR awards to small businesses in states that have received lesser amounts of federal R&D funding. Eighteen states and the Commonwealth of Puerto Rico participate in the program.4 Since 1994, EPSCoR has awarded 82 phase I SBIR grants valued at over $7 million. Other agencies also have such programs but have not used them to assist their SBIR participants. Several agencies are considering such an initiative to increase their outreach efforts in the SBIR program. In reauthorizing the program in 1992, the Congress emphasized Agencies Are commercialization, requiring agencies to consider commercial potential in Considering making awards and to collect data on commercialization by companies Commercial Potential that have received 15 or more phase II awards during the preceding 5 years. We found that, in response, agencies are weighing the commercial in Making Awards, but potential of all proposals and collecting data on commercialization by the Emphasis on companies, including frequent winners. However, neither the 1992 reauthorization nor a 1993 SBA policy directive on implementing the Commercialization legislation clarifies for the agencies what weight they are to assign to Raises Questions commercial potential relative to the program’s other goals in evaluating About the Relative proposals or how they are to use the data they collect on companies’ commercialization results. Without clarification, the agencies are Importance of the developing different evaluation approaches, at least one of which would Program’s Other have had unintended consequences. Goals 4 The states are Alabama, Arkansas, Idaho, Kansas, Kentucky, Louisiana, Maine, Mississippi, Montana, Nebraska, Nevada, North Dakota, Oklahoma, South Carolina, South Dakota, Vermont, West Virginia, and Wyoming. Page 5 GAO/T-RCED-99-198 Agencies Are Considering As described in the 1992 act, commercial potential has four indicators, one Commercial Potential to of which is a company’s commercialization record.5 Our review showed Varying Degrees in Making that agencies are considering commercial potential, including a company’s commercialization record and the other indicators. Together, these four Awards indicators can account for as much as one-third of a proposal’s total score. However, the commercialization record alone has played a limited role because it is only one indicator of commercial potential that, in turn, is one among several criteria such as technical merit considered when evaluating proposals. At the Department of Defense, for example, the commercialization record currently accounts for about one-fourth of the commercial potential score and about one-twelfth of the total score for a proposal; at the Department of Energy, it accounts for about one-eighteenth of the total score. Thus, even if a company has a poor commercialization record, this factor has exercised only a limited influence on agencies’ evaluations of proposals to date. According to SBA’s Assistant Administrator for Technology, the 1992 reauthorization directs participating agencies to collect information on commercialization by companies with 15 or more phase II awards during the previous 5 years but does not clarify how they are supposed to use it. In response, the agencies have collected data on commercialization by companies, including frequent winners. Because the 1992 act and the SBA policy directive do not address how to use the information on commercialization records, differences among the agencies have emerged. The Department of Defense, for example, recently planned to implement a new evaluation approach designed to give greater prominence to the commercialization records of companies with 5 or more phase II awards. This approach would have led to significantly lower scores for companies with relatively few awards and no sales but would not have penalized frequent winners with only modest sales. The Department has since revised its approach to avoid these unintended consequences by taking into account the concept of statistical significance as it relates to companies with widely varying numbers of awards. 5 Under the 1992 reauthorization, commercial potential was evidenced by “(i) the small business concern’s record of successfully commercializing SBIR or other research; (ii) the existence of second phase funding commitments from private sector or non-SBIR funding sources; (iii) the existence of third phase, follow-on commitments for the subject of the research; and (iv) the presence of other indicators of the commercial potential of the idea.” 15 U.S.C. 638(e)(4)(B). Page 6 GAO/T-RCED-99-198 The Emphasis on Despite the greater emphasis on commercialization since 1992, the Commercialization Raises program’s other goals, such as innovation and responsiveness to an Questions About the Role agency’s needs, remain important to the agencies when evaluating a company’s accomplishments and subsequent proposals. According to of Other Goals in some of the program managers, a relatively low level of commercialization Evaluating Companies’ may not signal failure because a company may have achieved other goals. Performance The difficulty for agencies of using any particular goal as a key criterion for selecting future proposals for funding stems from their not having (1) a clear definition of the program’s goals, (2) information on the relative weight that should be given to these potential goals, and (3) criteria for judging whether the goals have been achieved. Finding practical ways to define and measure the SBIR program’s goals in order to evaluate proposals has been difficult. For example, efforts to define and measure technological innovation have posed a challenge. Because technological innovation occurs in many different ways, no one indicator is an accurate measure of it. In addition, according to SBA’s Assistant Administrator for Technology, the 1992 reauthorization lacks a clear definition of “commercialization,” and he has sometimes differed with agencies on its meaning. This absence of a definition makes it more difficult, in his view, to determine when a frequent winner is “failing” to achieve a sufficient level of commercialization. The relative weight that should be given to the goals when evaluating proposals remains unclear. Innovation and responsiveness to an agency’s needs, for example, may compete with commercialization. In the view of many program managers, innovation involves a willingness to undertake R&D with a higher element of risk and a greater chance that it may not lead to a commercial product; responsiveness to an agency’s needs involves R&D that may be aimed at special niches with likewise limited commercial potential. According to the program managers, the challenge lies in striking the right balance between encouraging new, unproven technologies and achieving commercial sales. Agencies have also not agreed on criteria for “success” in meeting the program’s goals. In general, it is difficult to determine the appropriate mix of higher-risk projects that lead less frequently to commercial outcomes and of lower-risk projects that lead more frequently to successful products. The difficulty caused by not having criteria is compounded by the fact that only a handful of projects in the program achieve substantial commercial success. Only about 1.5 percent of the projects account for about half of the sales, and 4 percent of the projects account for about Page 7 GAO/T-RCED-99-198 75 percent of the sales. When the great majority of projects achieve no sales or only very limited sales, evaluating subsequent proposals from individual companies becomes more difficult if commercialization is considered the primary goal. Commercialization is only one of the program’s objectives but has become SBA Has an the main outcome for measuring its effectiveness. Studies of Opportunity to commercialization have proliferated as agencies have tried to obtain data Standardize on commercial activity. Although these studies rely on different approaches, they contain some common criteria for success, such as levels Evaluations of the of sales and developmental funding. Expanding the SBIR database at SBA to Program’s Outcomes include information on commercial outcomes and other indicators of success, such as savings to agencies resulting from SBIR projects, would help to standardize evaluations of the program. As SBA develops a new database, called Tech-Net, which is scheduled for full implementation in 1999, it has an opportunity to include outcome-related measures that can be used to track commercialization and other indicators of success. Various Methods With Our 1992 report on the SBIR program6 responded to a congressional Similar Criteria Have Been mandate that we assess the commercial outcomes of the program. We Used in Attempting to surveyed companies that had won phase II awards from 1984 through 1987. Our survey asked, for example, “Has the technology associated with Measure Outcomes this project led to additional developmental funding and/or sales, and is further work on this technology under way?” This approach was also used in later surveys of the program, including those conducted by a support contractor for DOD in 1996 and for SBA in 1998. Although the Department of Energy used a different evaluation approach, it also focused on outcomes, seeking information on the results of SBIR technology rather than on individual awards. Another common evaluation approach, used by the National Science Foundation, DOD, the National Aeronautics and Space Administration, and other agencies, relies on success stories stemming from the agencies’ awards. Although the agencies have used these stories to document the most significant results of their SBIR awards and to help companies market their technologies, they do not follow a consistent method and they ignore less successful projects, biasing the results of this approach. In our view, the approach is “open-ended,” meaning that it can be used to develop a 6 Federal Research: Small Business Innovation Research Shows Success but Can Be Strengthened (GAO/RCED-92-37, Mar. 30, 1992). Page 8 GAO/T-RCED-99-198 detailed story for an individual company but does not lend itself to systematization. A Standard Approach The evaluation methods we identified do not provide consistent Involves the Use of information across agencies on the program’s results. The use of a single Uniform Criteria for method with uniform criteria focusing on outcomes would produce such information, enabling SBA and the agencies to satisfy the requirements of Success and Improvements the Results Act. in SBA’s New Database As the central administrative agency for the program, SBA has maintained a governmentwide database that brings together the data submitted by the individual agencies participating in the program. This database has had two major shortcomings. First, because it was developed long before the Results Act emphasized the measurement of outcomes, the database reflects the earlier attention given to inputs, such as the name of each company and the amount of funding it has received. In general, companies have not provided information on the actual results of their research. Second, the database has contained unreliable information because it has lacked a unique identifying code for each company. Slight variations in the spelling of a company’s name have created difficulty because the database has counted each separate spelling as a separate company. In June 1998, SBA announced the introduction of a new database called Tech-Net. SBA database managers and SBIR program managers are optimistic about their ability to expand Tech-Net to capture outcome-related data. Tech-Net will enable agencies to update their information on SBIR awards and companies to update key information about their activities. As a result, companies will be able to provide more information on the actual results of their SBIR awards. In addition, SBA plans to assign each company a unique identification number to eliminate the confusion about the identity of participants. Madam Chair, this concludes my statement. I would be happy to answer any questions you or the other Members of the Subcommittee may have. Page 9 GAO/T-RCED-99-198 Appendix I An Overview of the Top 25 Frequent Winners, Fiscal Years 1983-97 Dollars in millions Percentage of annual revenue Phase II Total Dollar from SBIR Company awards awards value (1998) Foster Miller 147 573 108.2 20 Physical Optics 96 377 71.2 68 Creare 87 281 61.4 64 Physical Sciences 76 290 57.2 42 Spire 75 351 59.4 26 Radiation Monitoring Devices 59 187 43.3 38 Bend Research 58 166 34.3 23 EIC Laboratories 53 188 38.1 33 Mission Research 50 196 39.8 8 Science Research Laboratory 49 147 33.4 76 Advanced Technology Materials 48 208 38.4 10 Advanced Fuel Research 42 154 27.8 52 Ultramet 38 140 28.4 37 Aerodyne Research 35 134 27.5 36 CFD Research 35 107 24.7 52 a Sparta 35 162 28.3 TDA Research 35 127 19.5 70 a Thermacore 35 102 25.8 American Research Corp. of Virginia 34 102 19.3 80 Waterjet a Technology 34 102 21.5 Scientific Research a Associates 33 113 24.0 Giner 30 110 22.1 70 Schwartz Electro-optics 30 104 20.1 6 (continued) Page 10 GAO/T-RCED-99-198 Appendix I An Overview of the Top 25 Frequent Winners, Fiscal Years 1983-97 Dollars in millions Percentage of annual revenue Phase II Total Dollar from SBIR Company awards awards value (1998) Bio-Metric a Systems 29 89 18.5 Satcon Technology 28 119 22.2 44 a Information was not available. Source: GAO’s analysis of data from SBA’s SBIR database. (141335) Page 11 GAO/T-RCED-99-198 Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Federal Research: Evaluation of Small Business Innovation Research Can Be Strengthened
Published by the Government Accountability Office on 1999-06-17.
Below is a raw (and likely hideous) rendition of the original report. (PDF)