United States GAO General Accounting Washington, Office D.C. 20548 Resources, Community, and Economic Development Division B-283229 August 3,1999 The Honorable F. James Sensenbrenner, Jr. Chairman, Committee on Science House of Representatives Subject: Federal Research Information on the Advanced Technologv Program’s Award Selection This report responds to your request concerning the Advanced Technology Program (ATP), which is administered by the National Institute of Standards and Technology within the Department of Commerce. ATP’s mission is to stimulate U.S. economic growth by developing high-risk and enabling technologies through industry-driven cost- shared partnerships. ATP carries out competitions each year to select research and development projects for support. The program’s fiscal year 1999 budget is $203.5 million; the President’s fiscal year 2000 budget request seeks $238.7 million. A July 1997 Commerce Department report states that “[plroject proposals are carefully scrutinized during the review process to ensure that ATP funding is in fact necessary. The ATP frequently rejects projects when it concludes that the applicants could probably find funding elsewhere or that a delay in [the project’s] progress would not be a serious national economic concernV1 In light of that report and recent changes to the ATP Proposal Preparation Kit, you asked us to review whether ATP has used these selection criteria in its funding decisions for proposals. Specifically, you asked us (1) to describe how ATP determines that a delay in a project’s progress would be “a serious national economic concern” and (2) to identify the number of ATP applications that have been rejected since July 1997 because “a delay in progress would not be a serious national economic concern” or because the applicants could probably find funding elsewhere. Results in Brief While ATP collects a great deal of economic information from applicants during the proposal process, it does not specifically assess whether the nonselection of projects would pose a “serious national economic concern.” Agency officials said that the issue of a serious national economic concern influenced the development of the program and remains a basis for the program. The selection process itself focuses on evaluating the ‘Stren.s%heninethe Commerce Demxrtment’s Advanced Technolornr Promam: An Action Plan, U.S. Department of Commerce, Technology Administration (July 1997). /%m--- GAOLRCED-99-2588 Advanced Technology Program B-283229 potential broad-based economic benefits of proposed projects, including an assessment of the timeliness of introducing the technology to the market. Since ATP does not assess whether the nonselection of projects would pose a serious national economic concern, no proposals were rejected specifically because “a delay in progress would not be a serious national economic concern.” ATP officials said that at various stages of the review process, proposals are often rejected for a combination of factors, including a determination that applicants could probably find funding elsewhere. While the proposal reviewers’ comments are documented, ATP does not have a database that identifies the reasons that proposals have not been selected for funding. As a result, we could not readily determine how many proposals have been rejected because the applicants could probably find funding elsewhere. Background Only project proposals submitted in response to a form&l competition are considered for program funding. Since July 1997, ATP has announced 10 competitions. ATP’s fiscal year 1998 competitions consisted of one general competition open to all technology areas, and eight focused program competitions in the following areas: photonics manufacturing, premium power technologies, digital video for information networks, catalysis and biocatalysis technologies, microelectronics manufacturing technologies, selective-membrane platforms, tools for DNA diagnostics, and adaptive learning systems. ATP’s fiscal year 1998 budget was $192.5 million. However, funds for new awards were capped at $82 million. A total of 79 industry-generated projects were selected out of 502 proposals; industry’s cost share was $224 million, and ATP’s investment was $236 million over the life of the projects. Currently, ATP is in the process of evaluating 434 proposals received in response to the fiscal year 1999 competition-a single competition that was open to proposals from any area of technology. The ATP Proposal Preparation Kit that applicants use to prepare project proposals contains background material on the program, all of the required forms, and guidance for preparing the application, including a narrative section that addresses the ATP selection criteria. The kit used for the fiscal year 1998 competitions assigned various weights to the following five selection criteria: (1) scientific and technical merit (30 percent); (2) the potential net broad-based economic benefits, including the need for ATP funding (20 percent); (3) the adequacy of plans for eventual commercialization (20 percent); (4) the level of commitment and organizational structure (20 percent); and (5) experience and qualifications (10 percent). In November 1998, ATP published a new Proposal Preparation Kit for the fiscal year 1999 competition that combined the five selection criteria into two equally weighted criteria-scientific and technical merit (50 percent) and the potential for broad-based economic benefits (50 percent). The latter criterion includes the adequacy of commercialization plans, the level of commitment and organizational structure, experience and qualifications, and the need for ATP funding. To address the need for ATP funding, applicants are expected to answer questions such as, “Why is full private funding not available or not possible? What makes this project special and deserving of public support?” 2 GAOLRCED-99-258R Advanced Technology Program B-283229 ATP’s Efforts to Determine Whether Projects May Pose a Serious National Economic Concern ATP focuses on the potential broad-based economic benefits that may result from making an award. In regard to the phrase “serious national economic concern,” ATP officials stated that this language influenced the development of the program at its outset and remains a basis for the program. However, ATP does not specifically consider “serious national economic concern” in the proposal review process. In testimony before the Subcommittee on Technology, House Committee on Science, on February 26,1998, the National Institute of Standards and Technology’s Director stated that “serious [national] economic concern” is not a criterion [that] ATP uses in the selection process. Rather, the selection process focuses on evaluating the potential broad-based economic benefits of proposed projects, including an assessment of the timeliness of introducing the technology to the market and the benefits of the project for the nation. ATP’s Proposal Preparation Kits used by applicants in the fiscal year 1998 and fiscal 1999 competitions instructed the applicants to answer a number of questions related to the broad-based economic benefits of the project and the need for ATP funding. For example, applicants were directed to describe the size of the affected markets and the impact that the project could have on these markets and were asked how different the time scale of the project would be without ATP funding. They were asked why full private funding was not available or not possible. Applicants were asked whether they would pursue the project without ATP funding and, if not, why not. The Proposal Preparation Kit used for the fiscal year 1998 competitions specifically stated, “Even though broad-based economic benefits may be great, you will not be scored high and the project will not be funded if those benefits can be expected to be realized without ATP support.” ATP’s proposal review criteria indicate that ATP should not fund projects unless there is strong evidence that the funding can bring about important national economic benefits beyond what would likely result without the program’s involvement. The business reviewers’ work sheets contain sections for commenting on the potential broad-based economic benefits for a proposed project. Reviewers are asked to evaluate the proposed project in terms of the economic benefits; the need for ATP funding; and the pathway to the economic benefits, including commercialization plans and the potential for spillover benefits. ATP Applications Rejected Since July 1997 Since July 1997, ATP has not rejected any proposals for funding specifically because “a delay in progress would not be a serious national economic concern.” As discussed above, ATP’s selection criteria do not include an assessment of whether the nonselection of projects would pose a serious national economic concern; rather, the selection criteria include projects’ potential broad-based economic benefits. According to program officials, ATP has rejected proposals when a determination was made that applicants could probably find funding elsewhere. For the fiscal year 1998 competitions, a new section 16 was added to the application form that instructs applicants to “[dIescribe what efforts were made, prior to applying for ATP funding, to 3 GAO/RCED-99-258R Advanced Technology Program B-283229 secure private capital to support this project wholly.” For fiscal year 1999, ATP added guidance directing the applicants to describe why full private funding is not available or not possible and to describe their efforts to secure internal funding as well as external private funds. However, program officials stated that ATP would not immediately d.isqualIfy an applicant if the applicant did not complete the new section 16 as long as the rest of the application was in order. ATP officials said that there are opportunities throughout the review process to assess the proposal’s need for ATP funding. For example, the need for ATP support must be addressed by all semifinalists during the oral review stage of the review process. All ATP proposals undergo a multiphase review process. Proposals are first screened for conformance to the ATP regulations and the annual program announcement. According to the November 1998 Proposal Preparation Kit, about 10 percent of the proposals have been rejected at the preliminary screening stage since the program held its first competition in 1990. The proposals are then evaluated by external technical and business reviewers to assess the proposed technology’s scientific and technical merit and its potential for yielding broad-based economic benefits to the nation. The peer reviewers’ comments are documented and summarized, and a recommendation is made to a Source Evaluation Board. This board assesses the relative merit of the proposals with regard to the selection criteria and identifies a list of semifinalists. ATP invites the semifinahsts to the National Institute of Standards and Technology for oral reviews. The oral reviews focus on detailed technical and business questions. In past competitions, about 15 percent of the proposals have reached the oral review stage. Following the oral reviews, the board ranks the semifinalists’ proposals and recommends proposals for funding, taking into account the information contained in the proposals, the reviewers’ comments, and the information presented at the oral reviews. Final decisions regarding the awards are made by the Source Selecting Official on the basis of the board’s ranked list of proposals and available funding. In past competitions, ATP funded one-half to two-thirds of the proposals that underwent oral reviews. ATP officials stated that deficient proposals are typically deficient in more than one area and that a proposal may be rejected for a variety of reasons at any stage in the review process. Thus, no one factor generally determines whether a proposal will be rejected. As a proposal moves through the review process, additional weaknesses may be identified, which would result in the proposal’s rejection. As a result, ATP does not prioritize the principal reasons for rejecting a proposal nor does it have a database that identifies the reasons that proposals have not been selected for funding. Thus, we could not readily determine how many proposals have been rejected since July 1997 because ATP had determined that the applicants could probably find funding elsewhere. Objectives, Scope, and Methodology To describe how ATP determines that a delay in progress would be “a serious national economic concern” and to identify the number of ATP applications that have been rejected since July 1997 because “a delay in progress would not be a serious national economic concern” or because the applicants could probably find funding elsewhere, we met with ATP program officials. We also reviewed the ATP Proposal Preparation Kits for the fiscal year 1998 and fiscal 1999 competitions and the guidance that the technical 4 GAO/RCED-99-258R Advanced Technology Program B-283229 and business reviewers were provided. However, because of time constraints, we did not review the assessment work sheets prepared by the reviewers. We focused our review on the time period since July 1997, which included the completed fiscal year 1998 competitions and the ongoing fiscal 1999 competition. We conducted our review from June through August 1999 in accordance with generally accepted government auditing standards. Agency Comments and Our Evaluation We provided the National Institute of Standards and Technology with a draft of this report for review and comment. The Director responded that the Institute generally concurs with the report; however, he provided further information on how the agency reviews an applicant’s need for ATP funding. We included additional information on the review of the applicant’s need for ATP funding in our report. The comments appear as enclosure I to this report. We are sending copies of this report to other interested congressional members. We are also sending copies to the Honorable William M. Daley, Secretary of Commerce; Raymond G. Rammer, Director, National Institute of Standards and Technology; Lura J. Powell, Director, Advanced Technology Program; Johnnie E. Frazier, Inspector General, Department of Commerce; and the Honorable Jacob J. Lew, Director, Office of Management and Budget. We will make copies available to others on request. If you or your staff have any questions regarding this report, please call either Robin M. Nazzaro or me at (202) 512-3841. A key contributor to this report was Katherine L. Hale. Sincerely yours, I Susan D. Kladiva Associate Director, Energy, Resources, and Science Issues Enclosure 5 GAO/RCED-99-258R Advanced Technology Program Enclosure I Comments From the National Institute of Standards and Technology UNED STZi>S DEPAKFNlENF OF COMMERCE #+~~el lnsntuta of Standards and T=hnalogy Gatiersburg. Marjand 20999-0X1 OFFICEOF THE DIREC-IW? My 30.1999 Ms. Susm D. KIadiva &xc&t= Dhct0r, Energy, Resour::s, 2nd sciclcc Issues United States Gexral Accounting Office W&ingm, D.C. 20548 T&k you far the opportunity to review 2ud provide comxnea~~on the draft GAO report e&led m hfmm2tion on tht Advansed Technoloe, Prom’s Award Sele&xi (GAOLKED-99-ZSSR). WC gencrsIly concur tih be mpor; howevef, would like to elaborate fur&z OAthe disrzssion on p2ge 5, fist incomplete pu3@apir -rb.is par,gK@l ates: “For fiscal year 1999. ATP added gukIanca directing the applicants to destibe why &II p&at= funding is xot available or not possiile and to describe their efrbks to seme iubzr12I funding 2.~well as extd private fimds. However, prom officials stated that ATP would not immediately disqu2li& an applicvlt if the apphnt did not complete the new section 16 as long as the reS: of the 2pplication was in order. ’ ATP officials said that thre F opportcnities fhmu&out the review pmcess to assess the proposd’s need for Al? kndhg.” : .is P.TP of~55& advised GAO of%&s at the exit conferer~cc 2nd as stated in the above paragraph if an spplica;lt did not complete section 16 of the proposal cover sheet (Form MST-1262 or 1263). the need for A?? support must sti?! be addresied in the pmposal nanative as required in the ATP Proposai E5zgaratiox1 Kit, page 28. Additicnally, 52 izYorrzaiorr must be addrxssc5 252in by all semifinalists dmiig the maI review stage of tie review mcess. F-~rtbmore, the ATP r&ations found at I5 CER 295, section 6, Qittia for selection, -es tie following: . . . kidhionally, no zroposJ. wiII be frnded tbt does not require Federal support [emphasis added], tiar is product dmelo-pment rather than high risk R&D, that does not display an 2ppmpriate level. of _ cornmknent tioom ffie proposer, or does riot have an adaqua*& tech1ic21 and commercialhtion ph. Consistent with tke re&ations, if m applicant fails to adequately addre5.sthis issue, the proposal 4 not be fcndedd. Again. thank you for the oppomrnity to provide comments. R3ymond G. lbnnxr Dirmtor (141353) 6 GAO/RCED-99-258R Advanced Technology Program Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and Mastercard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu wilI provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: infoQwww.gao.gov or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States General Accounting Office Washington, D.C. 20548-0001 Official Business Penalty for Private Use $300 Address Correction Requested
Federal Research: Information on the Advanced Technology Program's Award Selection
Published by the Government Accountability Office on 1999-08-03.
Below is a raw (and likely hideous) rendition of the original report. (PDF)