oversight

Federal Research: Information on the Advanced Technology Program's Award Selection

Published by the Government Accountability Office on 1999-08-03.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

      United States
GAO   General Accounting
      Washington,
                            Office
                    D.C. 20548

      Resources, Community,         and
      Economic Development          Division


      B-283229


      August 3,1999


      The Honorable F. James Sensenbrenner, Jr.
      Chairman, Committee on Science
      House of Representatives

      Subject: Federal Research Information on the Advanced Technologv Program’s Award
               Selection

      This report responds to your request concerning the Advanced Technology Program
      (ATP), which is administered by the National Institute of Standards and Technology
      within the Department of Commerce. ATP’s mission is to stimulate U.S. economic
      growth by developing high-risk and enabling technologies through industry-driven cost-
      shared partnerships. ATP carries out competitions each year to select research and
      development projects for support. The program’s fiscal year 1999 budget is $203.5
      million; the President’s fiscal year 2000 budget request seeks $238.7 million.

      A July 1997 Commerce Department report states that “[plroject proposals are carefully
      scrutinized during the review process to ensure that ATP funding is in fact necessary.
      The ATP frequently rejects projects when it concludes that the applicants could
      probably find funding elsewhere or that a delay in [the project’s] progress would not be
      a serious national economic concernV1 In light of that report and recent changes to the
      ATP Proposal Preparation Kit, you asked us to review whether ATP has used these
      selection criteria in its funding decisions for proposals. Specifically, you asked us (1) to
      describe how ATP determines that a delay in a project’s progress would be “a serious
      national economic concern” and (2) to identify the number of ATP applications that
      have been rejected since July 1997 because “a delay in progress would not be a serious
      national economic concern” or because the applicants could probably find funding
      elsewhere.

      Results    in Brief

      While ATP collects a great deal of economic information from applicants during the
      proposal process, it does not specifically assess whether the nonselection of projects
      would pose a “serious national economic concern.” Agency officials said that the issue
      of a serious national economic concern influenced the development of the program and
      remains a basis for the program. The selection process itself focuses on evaluating the

      ‘Stren.s%heninethe Commerce Demxrtment’s Advanced Technolornr Promam: An Action Plan, U.S. Department of
      Commerce, Technology Administration (July 1997).

                                                     /%m---
                                                          GAOLRCED-99-2588        Advanced    Technology    Program
B-283229


potential broad-based economic benefits of proposed projects, including an assessment
of the timeliness of introducing the technology to the market.

Since ATP does not assess whether the nonselection of projects would pose a serious
national economic concern, no proposals were rejected specifically because “a delay in
progress would not be a serious national economic concern.” ATP officials said that at
various stages of the review process, proposals are often rejected for a combination of
factors, including a determination that applicants could probably find funding
elsewhere. While the proposal reviewers’ comments are documented, ATP does not
have a database that identifies the reasons that proposals have not been selected for
funding. As a result, we could not readily determine how many proposals have been
rejected because the applicants could probably find funding elsewhere.

Background
Only project proposals submitted in response to a form&l competition are considered for
program funding. Since July 1997, ATP has announced 10 competitions. ATP’s fiscal
year 1998 competitions consisted of one general competition open to all technology
areas, and eight focused program competitions in the following areas: photonics
manufacturing, premium power technologies, digital video for information networks,
catalysis and biocatalysis technologies, microelectronics manufacturing technologies,
selective-membrane platforms, tools for DNA diagnostics, and adaptive learning
systems. ATP’s fiscal year 1998 budget was $192.5 million. However, funds for new
awards were capped at $82 million. A total of 79 industry-generated projects were
selected out of 502 proposals; industry’s cost share was $224 million, and ATP’s
investment was $236 million over the life of the projects. Currently, ATP is in the
process of evaluating 434 proposals received in response to the fiscal year 1999
competition-a single competition that was open to proposals from any area of
technology.

The ATP Proposal Preparation Kit that applicants use to prepare project proposals
contains background material on the program, all of the required forms, and guidance
for preparing the application, including a narrative section that addresses the ATP
selection criteria. The kit used for the fiscal year 1998 competitions assigned various
weights to the following five selection criteria: (1) scientific and technical merit (30
percent); (2) the potential net broad-based economic benefits, including the need for
ATP funding (20 percent); (3) the adequacy of plans for eventual commercialization (20
percent); (4) the level of commitment and organizational structure (20 percent); and (5)
experience and qualifications (10 percent). In November 1998, ATP published a new
Proposal Preparation Kit for the fiscal year 1999 competition that combined the five
selection criteria into two equally weighted criteria-scientific and technical merit (50
percent) and the potential for broad-based economic benefits (50 percent). The latter
criterion includes the adequacy of commercialization plans, the level of commitment
 and organizational structure, experience and qualifications, and the need for ATP
 funding. To address the need for ATP funding, applicants are expected to answer
 questions such as, “Why is full private funding not available or not possible? What
 makes this project special and deserving of public support?”




2                                      GAOLRCED-99-258R   Advanced   Technology   Program
B-283229

ATP’s Efforts to Determine     Whether    Projects   May Pose a Serious      National
Economic Concern

ATP focuses on the potential broad-based economic benefits that may result from
making an award. In regard to the phrase “serious national economic concern,” ATP
officials stated that this language influenced the development of the program at its
outset and remains a basis for the program. However, ATP does not specifically
consider “serious national economic concern” in the proposal review process. In
testimony before the Subcommittee on Technology, House Committee on Science, on
February 26,1998, the National Institute of Standards and Technology’s Director stated
that “serious [national] economic concern” is not a criterion [that] ATP uses in the
selection process. Rather, the selection process focuses on evaluating the potential
broad-based economic benefits of proposed projects, including an assessment of the
timeliness of introducing the technology to the market and the benefits of the project for
the nation.

ATP’s Proposal Preparation Kits used by applicants in the fiscal year 1998 and fiscal
1999 competitions instructed the applicants to answer a number of questions related to
the broad-based economic benefits of the project and the need for ATP funding. For
example, applicants were directed to describe the size of the affected markets and the
impact that the project could have on these markets and were asked how different the
time scale of the project would be without ATP funding. They were asked why full
private funding was not available or not possible. Applicants were asked whether they
would pursue the project without ATP funding and, if not, why not. The Proposal
Preparation Kit used for the fiscal year 1998 competitions specifically stated, “Even
though broad-based economic benefits may be great, you will not be scored high and the
project will not be funded if those benefits can be expected to be realized without ATP
support.”

ATP’s proposal review criteria indicate that ATP should not fund projects unless there is
strong evidence that the funding can bring about important national economic benefits
beyond what would likely result without the program’s involvement. The business
reviewers’ work sheets contain sections for commenting on the potential broad-based
economic benefits for a proposed project. Reviewers are asked to evaluate the
proposed project in terms of the economic benefits; the need for ATP funding; and the
pathway to the economic benefits, including commercialization plans and the potential
for spillover benefits.

ATP Applications    Rejected   Since July 1997
Since July 1997, ATP has not rejected any proposals for funding specifically because “a
delay in progress would not be a serious national economic concern.” As discussed
above, ATP’s selection criteria do not include an assessment of whether the
nonselection of projects would pose a serious national economic concern; rather, the
selection criteria include projects’ potential broad-based economic benefits.

According to program officials, ATP has rejected proposals when a determination was
made that applicants could probably find funding elsewhere. For the fiscal year 1998
competitions, a new section 16 was added to the application form that instructs
applicants to “[dIescribe what efforts were made, prior to applying for ATP funding, to


3                                      GAO/RCED-99-258R   Advanced   Technology   Program
B-283229


secure private capital to support this project wholly.” For fiscal year 1999, ATP added
guidance directing the applicants to describe why full private funding is not available or
not possible and to describe their efforts to secure internal funding as well as external
private funds. However, program officials stated that ATP would not immediately
d.isqualIfy an applicant if the applicant did not complete the new section 16 as long as
the rest of the application was in order. ATP officials said that there are opportunities
throughout the review process to assess the proposal’s need for ATP funding. For
example, the need for ATP support must be addressed by all semifinalists during the
oral review stage of the review process.

All ATP proposals undergo a multiphase review process. Proposals are first screened
for conformance to the ATP regulations and the annual program announcement.
According to the November 1998 Proposal Preparation Kit, about 10 percent of the
proposals have been rejected at the preliminary screening stage since the program held
its first competition in 1990. The proposals are then evaluated by external technical and
business reviewers to assess the proposed technology’s scientific and technical merit
and its potential for yielding broad-based economic benefits to the nation. The peer
reviewers’ comments are documented and summarized, and a recommendation is made
to a Source Evaluation Board. This board assesses the relative merit of the proposals
with regard to the selection criteria and identifies a list of semifinalists. ATP invites the
semifinahsts to the National Institute of Standards and Technology for oral reviews.

The oral reviews focus on detailed technical and business questions. In past
competitions, about 15 percent of the proposals have reached the oral review stage.
Following the oral reviews, the board ranks the semifinalists’ proposals and
recommends proposals for funding, taking into account the information contained in the
proposals, the reviewers’ comments, and the information presented at the oral reviews.
Final decisions regarding the awards are made by the Source Selecting Official on the
basis of the board’s ranked list of proposals and available funding. In past competitions,
ATP funded one-half to two-thirds of the proposals that underwent oral reviews.

ATP officials stated that deficient proposals are typically deficient in more than one area
and that a proposal may be rejected for a variety of reasons at any stage in the review
process. Thus, no one factor generally determines whether a proposal will be rejected.
As a proposal moves through the review process, additional weaknesses may be
identified, which would result in the proposal’s rejection. As a result, ATP does not
prioritize the principal reasons for rejecting a proposal nor does it have a database that
identifies the reasons that proposals have not been selected for funding. Thus, we could
not readily determine how many proposals have been rejected since July 1997 because
ATP had determined that the applicants could probably find funding elsewhere.

Objectives,   Scope, and Methodology

To describe how ATP determines that a delay in progress would be “a serious national
economic concern” and to identify the number of ATP applications that have been
rejected since July 1997 because “a delay in progress would not be a serious national
economic concern” or because the applicants could probably find funding elsewhere, we
met with ATP program officials. We also reviewed the ATP Proposal Preparation Kits
for the fiscal year 1998 and fiscal 1999 competitions and the guidance that the technical


 4                                       GAO/RCED-99-258R    Advanced   Technology   Program
    B-283229


    and business reviewers were provided. However, because of time constraints, we did
    not review the assessment work sheets prepared by the reviewers. We focused our
    review on the time period since July 1997, which included the completed fiscal year 1998
    competitions and the ongoing fiscal 1999 competition. We conducted our review from
    June through August 1999 in accordance with generally accepted government auditing
    standards.

    Agency Comments      and Our Evaluation

    We provided the National Institute of Standards and Technology with a draft of this
    report for review and comment. The Director responded that the Institute generally
    concurs with the report; however, he provided further information on how the agency
    reviews an applicant’s need for ATP funding. We included additional information on the
    review of the applicant’s need for ATP funding in our report. The comments appear as
    enclosure I to this report.




    We are sending copies of this report to other interested congressional members. We are
    also sending copies to the Honorable William M. Daley, Secretary of Commerce;
    Raymond G. Rammer, Director, National Institute of Standards and Technology; Lura J.
    Powell, Director, Advanced Technology Program; Johnnie E. Frazier, Inspector General,
    Department of Commerce; and the Honorable Jacob J. Lew, Director, Office of
    Management and Budget. We will make copies available to others on request.

    If you or your staff have any questions regarding this report, please call either Robin M.
    Nazzaro or me at (202) 512-3841. A key contributor to this report was Katherine L. Hale.

    Sincerely yours,



I
    Susan D. Kladiva
    Associate Director, Energy,
     Resources, and Science Issues

    Enclosure




    5                                     GAO/RCED-99-258R   Advanced   Technology   Program
Enclosure I

                 Comments         From the National            Institute       of Standards          and Technology



                                                                      UNED        STZi>S      DEPAKFNlENF      OF COMMERCE
                                                                      #+~~el       lnsntuta    of Standards    and T=hnalogy
                                                                      Gatiersburg. Marjand 20999-0X1
                                                                      OFFICEOF THE DIREC-IW?


                                                        My 30.1999



      Ms. Susm D. KIadiva
      &xc&t= Dhct0r, Energy, Resour::s, 2nd
        sciclcc Issues
      United States Gexral Accounting Office
      W&ingm,      D.C. 20548



      T&k you far the opportunity to review 2ud provide comxnea~~on the draft GAO report e&led m
       hfmm2tion on tht Advansed Technoloe, Prom’s       Award Sele&xi  (GAOLKED-99-ZSSR).        WC gencrsIly
      concur tih be mpor; howevef, would like to elaborate fur&z OAthe disrzssion on p2ge 5, fist incomplete
      pu3@apir -rb.is par,gK@l ates:

              “For fiscal year 1999. ATP added gukIanca directing the applicants to destibe why &II p&at= funding is
              xot available or not possiile and to describe their efrbks to seme iubzr12I funding 2.~well as extd
              private fimds. However, prom        officials stated that ATP would not immediately disqu2li& an applicvlt
              if the apphnt did not complete the new section 16 as long as the reS: of the 2pplication was in order.             ’
              ATP officials said that thre F opportcnities fhmu&out the review pmcess to assess the proposd’s need
              for Al? kndhg.”                                             :

      .is P.TP of~55& advised GAO of%&s at the exit conferer~cc 2nd as stated in the above paragraph if an spplica;lt
      did not complete section 16 of the proposal cover sheet (Form MST-1262 or 1263). the need for A?? support must
      sti?! be addresied in the pmposal nanative as required in the ATP Proposai E5zgaratiox1 Kit, page 28. Additicnally,
      52 izYorrzaiorr must be addrxssc5 252in by all semifinalists dmiig the maI review stage of tie review mcess.

      F-~rtbmore, the ATP r&ations       found at I5 CER 295, section 6, Qittia      for selection, -es   tie following:

             . . . kidhionally, no zroposJ. wiII be frnded tbt does not require Federal support [emphasis added],
             tiar is product dmelo-pment rather than high risk R&D, that does not display an 2ppmpriate level. of
           _ cornmknent tioom ffie proposer, or does riot have an adaqua*& tech1ic21 and commercialhtion   ph.

      Consistent with tke re&ations,   if m applicant fails to adequately addre5.sthis issue, the proposal 4     not be
      fcndedd.

      Again. thank you for the oppomrnity to provide comments.




                                                       R3ymond G. lbnnxr
                                                       Dirmtor




  (141353)


  6
                                                                 GAO/RCED-99-258R             Advanced    Technology       Program
Ordering     Information

The first copy of each GAO report and testimony is free.
Additional   copies are $2 each. Orders should be sent to the
following  address, accompanied by a check or money order
made out to the Superintendent     of Documents, when
necessary. VISA and Mastercard      credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted    25 percent.

Orders by mail:

U.S. General Accounting     Office
P.O. Box 37050
Washington,  DC 20013

or visit:

Room 1100
700 4th St. NW (corner     of 4th and G Sts. NW)
U.S. General Accounting     Office
Washington,  DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony.   To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone   phone. A recorded menu wilI provide information       on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

infoQwww.gao.gov

or visit    GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States
General Accounting    Office
Washington,   D.C. 20548-0001


Official   Business
Penalty    for Private   Use $300
Address    Correction    Requested