oversight

Department of Energy: Advanced Research Projects Agency-Energy Could Benefit from Information on Applicants' Prior Funding

Published by the Government Accountability Office on 2012-01-13.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States Government Accountability Office

GAO            Report to Congressional Requesters




January 2012
               DEPARTMENT OF
               ENERGY
               Advanced Research
               Projects Agency-
               Energy Could Benefit
               from Information on
               Applicants’ Prior
               Funding




GAO-12-112
                                            January 2012

                                            DEPARTMENT OF ENERGY
                                            Advanced Research Projects Agency-Energy Could
                                            Benefit from Information on Applicants' Prior
                                            Funding
Highlights of GAO-12-112, a report to
congressional requesters




Why GAO Did This Study                      What GAO Found
The Department of Energy’s (DOE)            ARPA-E uses four selection criteria, such as the potential impact of the proposed
Advanced Research Projects Agency-          technology relative to the state of the art, and other considerations in awarding
Energy's (ARPA-E) purpose is to             funds. Other considerations include balancing a variety of technology
overcome long-term and high-risk            approaches and the likelihood the technology would be brought to market. GAO
technological barriers in the               identified 18 out of 121 award winners through ARPA-E’s first three funding
development of energy technologies.         rounds that had received some prior private sector investment, and ARPA-E took
Since 2009, ARPA-E has awarded              steps to identify and understand how this funding was related to proposed
$521.7 million to universities, public      projects. Beginning with the third funding round, ARPA-E began requiring that
and private companies, and national         applicants explain why private investors were not willing to fund proposed
laboratories to fund energy research
                                            projects. However, ARPA-E did not provide applicants with guidance, such as a
projects.
                                            sample response, to assist them in completing this requirement, and responses
GAO was asked to examine                    were generally limited. Some applicants provided general information about prior
(1) ARPA-E’s use of criteria and other      research but did not specifically explain why private investors would not support
considerations for making awards and        their projects. When applicants provided little prior funding information, ARPA-E’s
the extent to which applicants identify     program directors spent time and resources to determine the extent of such
and explain other private funding           funding for proposed ARPA-E projects. One applicant included a letter from its
information, (2) the extent to which        venture capital investor to explain why the investor was not willing to fund the
ARPA-E-type projects could have been        work proposed to ARPA-E, an approach the National Institute of Standards and
funded through the private sector, and      Technology uses as a check in its funding applications for advanced research but
(3) the extent to which ARPA-E              that ARPA-E currently does not use. Also, ARPA-E officials said that they have
coordinates with other DOE program          considered but have not used venture capital data to identify applicants with prior
offices to avoid duplicating efforts.
                                            private investors. Examining such data allowed GAO to quickly cross-check
GAO interviewed ARPA-E program
                                            applicants’ prior private funding.
directors, award winners, and
nonwinners with characteristics similar     GAO’s review suggests that most ARPA-E projects could not have been funded
to those of award winners. GAO also         solely by private investors. Private venture capital firms told GAO that, among
analyzed private venture capital            other considerations, they generally do not fund projects that rely on unproven
funding data and spoke with venture         technologies and tend to invest in projects that can be commercialized in less
capital firms.                              than 3 years. Data from ARPA-E on award winners show that 91 out of 121
What GAO Recommends                         ARPA-E projects from the first three funding rounds had technological concepts
                                            that had not yet been proven in a laboratory setting. Also, nearly all of the ARPA-
GAO recommends that ARPA-E                  E award winners and applicants GAO spoke with estimated that their projects
consider providing applicants guidance      were at least 3 years away from potential commercialization. In addition, GAO
with a sample response explaining           found that few eligible applicants that were not selected for an award later
prior sources of funding, requiring         secured private funding.
applicants to provide letters from
investors explaining why they are not       ARPA-E officials have taken steps to coordinate with other DOE offices to avoid
willing to fund proposed projects, and      duplication. For example, ARPA-E program directors told GAO they engage in
using third-party venture capital data to   outreach with officials from related DOE offices in advance of funding
identify applicants' prior funding.         announcements to identify funding gaps in research. In addition, program
ARPA-E commented on a draft of this         directors have recruited officials from other DOE offices and the Department of
report and concurred with key findings      Defense (DOD) to review ARPA-E applications. This cross-agency interaction
and recommendations.                        may also reduce the potential for overlap in funding.



View GAO-12-112 or key components. For
more information, contact Frank Rusco at
(202) 512-3841 or ruscof@gao.gov.

                                                                                    United States Government Accountability Office
Contents


Letter                                                                                  1
               Background                                                               5
               In Addition to Its Selection Criteria, ARPA-E Also Considers
                  Applicants’ Prior Sources of Private Funding; However, Most
                  Award Winners We Reviewed Did Not Explain This Information           10
               Most ARPA-E Projects Likely Could Not Have Been Funded Solely
                  by Private Investors                                                 14
               ARPA-E Officials Have Taken Steps to Coordinate with Other
                  Department of Energy Offices in Advance of Awarding Funds            19
               Conclusions                                                             21
               Recommendations for Executive Action                                    22
               Agency Comments and Our Evaluation                                      22

Appendix I     Scope and Methodology                                                   24



Appendix II    ARPA-E Program Technology Areas                                         28



Appendix III   Description of ARPA-E Award Winners with Prior Private Investment       31



Appendix IV    Comments from ARPA-E                                                    34



Appendix V     GAO Contact and Staff Acknowledgments                                   36



Table
               Table 1: ARPA-E Funding Announcement Program Technology Areas            9


Figures
               Figure 1: ARPA-E’s Described Role within DOE and the Private
                        Sector                                                          6
               Figure 2: Current Battery Range for Electric Vehicles and Goals of
                        ARPA-E Research                                                 7



               Page i                                                    GAO-12-112 ARPA-E
Abbreviations

ADEPT                 Agile Delivery of Electrical Power Technology
America               America Creating Opportunities to
 COMPETES             Meaningfully Promote Excellence in
 Act                  Technology, Education, and Science Act
ARPA-E                Advanced Research Projects Agency-Energy
BEEST                 Batteries for Electrical Energy Storage in
                      Transportation
BEETIT                Building Energy Efficiency Through Innovative
                      Thermodevices
DARPA                 Defense Advanced Research Projects Agency
DOD                   Department of Defense
DOE                   Department of Energy
EERE                  Office of Energy Efficiency and Renewable Energy
GENI                  Green Electricity Network Integration
GRIDS                 Grid-Scale Rampable Intermittent Dispatchable
                      Storage
HEATS                 High Energy Advanced Thermal Storage
HVAC                  heating, ventilating, and air conditioning
IMPACCT               Innovative Materials and Processes for Advanced
                      Carbon Capture Technologies
NIST                  National Institute of Standards and Technology
NVCA                  National Venture Capital Association
PASTA                 Panel of Senior Technical Advisors
PETRO                 Plants Engineered To Replace Oil
REACT                 Rare Earth Alternatives in Critical Technologies for
                      Energy
PHEV                  plug-in hybrid electric vehicle
SBIR                  Small Business Innovation Research
TRL                   technology readiness level


This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                                                               GAO-12-112 ARPA-E
United States Government Accountability Office
Washington, DC 20548




                                   January 13, 2012

                                   The Honorable Ralph M. Hall
                                   Chairman
                                   Committee on Science, Space, and Technology
                                   House of Representatives

                                   The Honorable Paul Broun, M.D.
                                   Chairman
                                   Subcommittee on Investigations and Oversight
                                   Committee on Science, Space, and Technology
                                   House of Representatives

                                   U.S. energy consumption has increased by 20 percent over the past 20
                                   years and is projected to continue to grow. 1 Volatile prices, global supply
                                   disruptions, and the impacts of energy use on climate and the
                                   environment have driven interest in reducing energy demand, improving
                                   energy efficiency, and expanding supplies with both renewable and
                                   traditional energy sources. In 2005, members of Congress asked the
                                   National Academies what actions federal policymakers could take to
                                   enhance the nation’s science and technology enterprise so that the
                                   United States could successfully compete, prosper, and be secure in the
                                   global community of the 21st century. 2 The National Academies compiled
                                   their findings and recommendations in a report that identified two key
                                   challenges: (1) creating high-quality jobs for Americans, and
                                   (2) responding to the nation’s need for clean, affordable, and reliable
                                   energy. 3 The report also highlighted the idea that scientific and technical
                                   innovations are key drivers of economic growth in the United States.
                                   Among the report’s recommendations was the creation of an organization
                                   within DOE to sponsor energy research that industry by itself cannot or




                                   1
                                    U.S. Energy Information Administration, Annual Energy Review 2009. (Washington, D.C.:
                                   2009).
                                   2
                                    The National Academies comprise four organizations: the National Academy of Sciences,
                                   the National Academy of Engineering, the Institute of Medicine, and the National
                                   Research Council; they advise policymakers on scientific and technical matters.
                                   3
                                    The National Academies, Rising Above the Gathering Storm: Energizing and Employing
                                   America for a Brighter Economic Future (Washington, D.C.; 2007).




                                   Page 1                                                            GAO-12-112 ARPA-E
will not support and in which risk may be high but success would provide
dramatic benefits for the nation in meeting long-term energy challenges.

In 2007, the America Creating Opportunities to Meaningfully Promote
Excellence in Technology, Education, and Science Act (America
COMPETES) established the Advanced Research Projects Agency-
Energy (ARPA-E) within the Department of Energy (DOE) to overcome
the long-term and high-risk technological barriers in the development of
energy technologies. 4 ARPA-E borrows from the Defense Advanced
Research Projects Agency (DARPA) model, an agency created within the
Department of Defense (DOD) in 1958 to direct and perform advanced
research and development projects. As specified in statute, ARPA-E’s
program goals are to enhance U.S. economic and energy security
through the development of certain energy technologies and to ensure
that the United States maintains a technological lead in developing and
deploying advanced energy technologies.

Since first receiving an appropriation in 2009 in the American Recovery
and Reinvestment Act of 2009, ARPA-E has awarded $521.7 million to
universities, public and private companies, and national laboratories to
fund 181 projects that attempt to make transformational—rather than
incremental––advances to a variety of energy technologies, including
high-energy batteries and renewable fuels. 5 Award winners must meet
cost share requirements, through either in-kind contributions or outside
funding sources. 6

ARPA-E is required by statute to achieve its goals through energy
technology projects that, among other things, accelerate transformational
technological advances in areas that industry by itself is not likely to



4
 Pub. L. No. 110-69, § 5012 (2007).
5
 ARPA-E generally uses cooperative agreements to make funding awards, which involve
the transfer of a thing of value to the recipient to carry out a public purpose authorized by
law. Cooperative agreements differ from grants because substantial involvement is
expected between ARPA-E and the recipient. ARPA-E uses similar funding agreements
for national laboratories.
6
 The cost share requirement for award winners is generally at least 20 percent of total
allowable costs, although under section 988(b)(3) of the Energy Policy Act of 2005, ARPA-
E has reduced the cost share requirement for certain applicants, such as universities, to 5
percent or 10 percent for all of the funding rounds except the first. Award winners’ cost
share must be provided by a nonfederal source.




Page 2                                                                   GAO-12-112 ARPA-E
undertake because of technical and financial uncertainty. At the same
time, the Director of ARPA-E is required to ensure, to the maximum
extent practicable, that ARPA-E’s activities are coordinated with, and do
not duplicate the efforts of, programs and laboratories within DOE and
other relevant research agencies. This report responds to your request
that we examine ARPA-E awards. Our objectives were to examine (1)
ARPA-E’s use of criteria and other considerations for making awards, and
the extent to which applicants identify and explain other private funding
information; (2) the extent to which ARPA-E-type projects could have
been funded through the private sector; and (3) the extent to which
ARPA-E coordinates with other DOE program offices to avoid duplicating
efforts.

To address these three objectives, we reviewed ARPA-E applications and
conducted interviews with applicants, award winners, representatives of
venture capital firms and other experts, and DOE and other federal
agency officials. ARPA-E has released a total of four funding
announcements—meaning the agency was accepting project proposals
for a set period of time—in April 2009, December 2009, March 2010, and
April 2011. Our review focused on ARPA-E’s first three funding rounds,
which had closed prior to the start of our review. The fourth funding round
did not close until September 2011. 7 ARPA-E awarded funds for 121
projects out of 4,788 applicants across the three funding rounds we
examined. To address these three objectives, we reviewed ARPA-E
applications and conducted interviews with applicants, award winners,
DOE officials, and representatives of venture capital firms, among other
activities. Specifically,

•   To examine ARPA-E’s use of criteria and other considerations for
    making awards, we selected a nonprobability sample of 20
    applications from the 4,788 applications in the first three funding
    rounds and reviewed these 20 applications. To examine the extent to
    which applicants identify and explain other private funding information,
    we searched for evidence of prior private funding for all 121 award




7
In the fourth round of funding, ARPA-E awarded 60 projects out of 427 applicants.




Page 3                                                              GAO-12-112 ARPA-E
     winners in VentureDeal, a venture capital database. 8 As a result of
     our search, we identified 18 award winners that had some prior
     private venture capital funding from the 121 award winners. We then
     reviewed the applications of these 18 award winners and interviewed
     their representatives. 9

•    To analyze the extent to which ARPA-E projects could have been
     funded through the private sector, we analyzed data on the state of
     technology and potential time to commercialization for the 121 award
     winners from ARPA-E’s first three funding rounds. In addition to
     analyzing data for the 121 award winners, we conducted structured
     interviews with 22 of 33 “contingently selected” applicants chosen by
     ARPA-E during its second and third funding rounds. Contingently
     selected applicants are those applicants that met ARPA-E’s selection
     criteria but were ultimately not awarded funds. 10 We also conducted
     structured interviews with a nonprobability sample of 13 award
     winners selected from ARPA-E’s first three funding rounds and we
     spoke with the 18 ARPA-E award winners mentioned above that we
     identified through the VentureDeal database to discuss key
     differences between their prior research and their ARPA-E-funded
     projects. 11 We also conducted interviews with a variety of companies
     and individuals knowledgeable about research associated with ARPA-
     E-type projects, including six venture capital firms.

•    To examine the extent to which ARPA-E coordinates with other DOE
     programs to avoid duplicating efforts, we spoke with the ARPA-E
     program directors as well as officials from other DOE program offices,
     DARPA, and the DOE Office of Inspector General. We also asked


8
 We were not able to verify the completeness of VentureDeal data, and there may have
been ARPA-E award winners that had prior private funding that did not appear in these
data. To compile data on venture capital funding, VentureDeal uses (1) Securities and
Exchange Commission regulatory filings, (2) survey information collected directly from
venture capital firms, (3) financial news media announcements and press releases from
venture capital firms or recipient companies, and (4) local business journals.
9
 Eight of these 18 companies also appeared in the sample of 20 applicants that we
selected to examine ARPA-E's use of criteria and other considerations for making awards.
10
  According to ARPA-E officials with whom we spoke, these applicants would have been
selected for an award had additional funds been available. The remaining 11 contingently
selected applicants did not respond to our requests for an interview.
11
  Four of these 18 award winners also appeared in our nonprobability sample of 13 award
winners.




Page 4                                                               GAO-12-112 ARPA-E
                  award winners and contingently selected applicants to discuss their
                  understanding of other potential sources of DOE funding for their
                  projects.

             We provide a more in-depth discussion of our methods in appendix I.

             We conducted this performance audit from November 2010 to December
             2011 in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objectives. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objectives.


             In 2005, the National Academies recommended to Congress the creation
Background   of an organization within DOE like DARPA. In 2007, the America
             COMPETES Act created a new agency within DOE called ARPA-E. In
             line with the National Academies’ recommendation, the America
             COMPETES Act as amended directs ARPA-E to achieve its goals by
             identifying and promoting revolutionary advances in fundamental and
             applied sciences, translating scientific discoveries and cutting-edge
             inventions into technological innovations, and accelerating
             transformational technological advances in areas that industry by itself is
             not likely to undertake because of technical and financial uncertainty. As
             such, ARPA-E officials told us that ARPA-E was designed to sponsor
             research beyond basic science, yet riskier than what the private sector
             alone or DOE’s applied offices would support. 12 (See fig. 1.) The National
             Academies recommended that ARPA-E should not perform research and
             development itself, but should fund it to be conducted by universities and
             others in the private sector. In 2009, the American Recovery and
             Reinvestment Act of 2009 appropriated $400 million for ARPA-E.



             12
               DOE funds the development of energy technologies ranging from basic and applied
             energy research to loan guarantees for clean energy generation facilities. For example,
             DOE’s Office of Science supports basic scientific research, including chemistry, biology,
             and materials sciences, as foundational research for a number of energy technologies.
             DOE’s Office of Energy Efficiency and Renewable Energy applies established research to
             alternative and clean energy technologies, such as improving existing lithium ion batteries
             for use in electric or hybrid vehicles. DOE’s Loan Programs Office offers loans to
             commercialize clean energy projects, such as wind farms.




             Page 5                                                                 GAO-12-112 ARPA-E
Figure 1: ARPA-E’s Described Role within DOE and the Private Sector




Note: Private sector groups are in oval boxes and DOE offices are in rectangular boxes. According to
ARPA-E officials, higher-risk research is less likely to be successful in being brought to market but
may have greater potential benefits in increasing energy supplies and creating jobs. Technology
readiness levels are used by DOE to categorize research according to its proximity to basic science
or large-scale deployment.


ARPA-E is an agency with fewer than 30 federal employees, and its eight
program directors, who are generally scientists and engineers, create and
manage funding programs for the agency. ARPA-E’s program development
and award selection process takes 6 to 8 months from start to finish,
beginning when the agency hires a program director for a 3-year term and
tasks the program director with identifying a gap in energy technology
research and developing a program to fill that gap. For example, ARPA-E’s
batteries for transportation program, called the Batteries for Electrical Energy
Storage in Transportation (BEEST) program, was established to fill a gap in
existing federal research programs on batteries for electric vehicles.
Identifying these gaps and designing the program involves research;
consultation with scientific experts, including a workshop with outside
experts; and internal discussion at ARPA-E. From this process, program
directors develop funding announcements that describe the technical
requirements specific to each program’s technology area that applicants




Page 6                                                                        GAO-12-112 ARPA-E
have to meet and the four criteria that ARPA-E uses in its selection
process. 13 The four criteria are the
•    Impact of the proposed technology relative to the state of the art. The
     applicant must demonstrate the potential for a transformational—not
     incremental––advancement over current technologies. (See fig. 2.)
     More specifically, the applicant must demonstrate an awareness of
     competing commercial and emerging technologies and identify how its
     proposed concept/technology provides significant improvement over
     these other solutions.
•    Overall scientific and technical merit. The applicant must demonstrate
     that the work is unique and innovative. The applicant must also
     demonstrate a sound technical approach to accomplish the proposed
     research and development objectives. The outcome and deliverables
     of the program, if successful, should be clearly defined. Specific
     technical requirements that are unique to each individual ARPA-E
     program funding announcement must also be addressed.
•    Qualifications, experience, and capabilities. The applicant must
     demonstrate that it has the expertise and experience to accomplish the
     proposed project. In addition, the applicant must have access to all
     facilities required to accomplish the research and development effort.
•    Sound management plan. The applicant must have a workable plan to
     manage people and resources. Major technical research and
     development risks should be identified. The schedule and budget
     should be reasonable.

Figure 2: Current Battery Range for Electric Vehicles and Goals of ARPA-E Research




13
  ARPA-E program directors developed funding announcements for six program technology
areas that made up the agency's second and third funding rounds. (See table 1.)




Page 7                                                            GAO-12-112 ARPA-E
ARPA-E employs the following three-stage application process:

•   Concept paper. Applicants initially submit a 5- to 7-page abstract of
    their projects. Scientific experts from industry, government, and
    academia serve as reviewers.

•   Full application. After reviews of the concept paper, ARPA-E encourages
    some applicants to submit full applications using ARPA-E’s online
    application system. ARPA-E’s current instructions request that applicants
    provide, among other things, information about other prior, current, and
    pending public and private sources of funding, as well as why other
    funding sources are not willing to fund the projects. Full applications are
    then reviewed by leading scientific experts in the field, who evaluate
    them against the four criteria and assign numerical scores.

•   Reply to reviewer comments. After assessing the full applications,
    reviewers provide comments and questions to the applicants, who
    then have the opportunity to respond.

The applications with the reviewers’ comments are forwarded to a three-
person panel beginning the next three phases of ARPA-E’s award funding
process, which are as follows:

•   Selection. The three-person panel, usually chaired by the relevant
    program director, considers the reviewers’ comments and numerical
    scores, and recommends applications to award. The final decisions
    on which applicants to select are made by the selecting official, which
    is usually the ARPA-E Director.
•   Award negotiations. Negotiations proceed for approximately 2
    months. Program directors work closely with the award winners to set
    up a project plan with technical milestones that are to be met during
    the funding of the award, which are planned to last between 2 and 3
    years. Funds are awarded following the negotiations.

•   Monitoring. ARPA-E monitors and supports the project through
    quarterly reviews and site visits. After about 1 year, the agency
    decides whether to continue or terminate the project if the agreed-to
    milestones are not met.

In April 2009, ARPA-E started its funding award process by releasing a
funding announcement soliciting proposals for all energy ideas and
technologies. Following the review process, 36 projects were awarded
funds after being selected from 3,700 applications that spanned the



Page 8                                                       GAO-12-112 ARPA-E
                                                  technology areas of 10 programs. ARPA-E released additional funding
                                                  announcements in December 2009, March 2010, and April 2011. (See
                                                  table 1.) Money appropriated by the American Recovery and
                                                  Reinvestment Act of 2009 funded ARPA-E’s first three funding rounds.
                                                  After receiving an appropriation in DOE’s fiscal year 2011 appropriations
                                                  act, ARPA-E announced a fourth round of funding in April 2011.

Table 1: ARPA-E Funding Announcement Program Technology Areas

(Dollars in millions)
                                                                            Funding announcements
                                      1                                       2                           3                            4
                                 April 2009                        December 2009                   March 2010                     April 2011
Program (number Biomass energy (5)                       Batteries for transportation (10) Grid-scale electricity        Advanced thermal
of projects)                                                                               storage (12)                  storage (15)
                        Building efficiency (3)          Materials for carbon capture        Building efficiency (16) Electricity network
                                                         (15)                                                         integration (14)
                        Carbon capture (5)               Electrofuels (13)                   Electrical power            Plants engineered to
                                                                                             electronics (14)            replace oil (10)
                        Conventional energy (1)                                                                          Rare earth alternatives for
                                                                                                                         energy technologies (14)
                        Direct solar fuels (4)                                                                           Solar electrical power
                                                                                                                         technology (7)
                        Energy storage (6)
                        Energy-efficient water
                        purification (1)
                        Renewable power (4)
                        Vehicle technologies (5)
                        Waste heat capture (2)
Total funding           $150                             $113                                $94                         $156
                                                  Source: GAO analysis of ARPA-E data.

                                                  Note: The table shows 116 total projects funded by ARPA-E in its first three funding rounds. In August
                                                  and September 2010, ARPA-E selected 5 additional projects for funding, raising total funded projects
                                                  over these three rounds to 121. These 5 projects were in the following program technology areas:
                                                  building efficiency, vehicle technologies, renewable power, and energy storage. Funds awarded
                                                  across the four funding rounds totaled $521.7 million. For further details on ARPA-E’s program
                                                  technology areas, see appendix II.




                                                  Page 9                                                                         GAO-12-112 ARPA-E
                           In addition to applying its four criteria, ARPA-E gives program directors
In Addition to Its         discretion to use additional considerations to award funds to projects,
Selection Criteria,        including whether ARPA-E applicants received private funding. Most
                           ARPA-E award winners did not receive prior private funding, but for those
ARPA-E Also                that did, most award winners we reviewed did not explain these funds.
Considers Applicants’
Prior Sources of
Private Funding;
However, Most Award
Winners We Reviewed
Did Not Explain This
Information

ARPA-E Program Directors   Of the 20 applications we reviewed for award selection criteria, all
Use the Agency’s Four      contained supporting information addressing the agency’s four criteria. In
Selection Criteria and     our analysis of the ARPA-E reviewers’ evaluations from these 20
                           applications, we noted regular assignment of numerical scores rating
Other Considerations to    applicants on the extent to which they met the criteria. All eight ARPA-E
Select Projects            program directors told us they considered or, if they were recently hired,
                           will consider all four criteria, but several focused more heavily on two
                           criteria—the impact of the proposed technology relative to the state of the
                           art and its overall scientific and technical merit.

                           In addition to basing the numerical scores applicants receive on the
                           extent to which they meet the four selection criteria, program directors
                           told us the agency gives them the ability to take other qualitative
                           considerations into account when awarding funds. One of those
                           considerations is to fund a broad range of potential technological
                           solutions with varying levels of risk in solving a given technical problem.
                           Two program directors selected projects to reflect a variety of
                           technologies, and they told us they believe that this approach increases
                           their programs’ overall chances of success. Specifically, one program
                           director told us he chose projects that employed a variety of new battery
                           technologies, a strategy that should increase the likelihood that at least
                           one of them will work. This program director also chose some battery
                           projects with much higher potential storage capacity but with a lower




                           Page 10                                                    GAO-12-112 ARPA-E
probability of success in achieving project milestones and in ultimately
being brought to market. In those cases, ARPA-E provided smaller
awards to the projects with the lower probability of success. 14

Several program directors also told us that during the selection process,
they considered the applicants’ projects’ proposed project scope and
duration, requested funding levels, and technical milestones and
negotiated to revise these, if necessary, to better align applicants’
projects with ARPA-E’s program goals. According to our review of ARPA-
E data from the first three rounds of funding, the agency reduced
requested award amounts by 5 percent or more on 31 out of 121 projects,
for a total of $59 million below total requested award amounts for these
rounds. When ARPA-E makes these kinds of adjustments, the agency
may also reduce the proposed project scope to fund only what the
program directors consider to be the transformational part of the project
and to avoid funding applied research or development work that would be
outside ARPA-E’s program goals. For example, the agency reduced the
award amount and proposed project scope for an energy storage
technology project designed to improve energy storage on the electrical
grid. The project proposal initially requested nearly $5 million to
demonstrate the technology at nearly full scale. During award
negotiations, ARPA-E reduced this amount to $750,000 to focus the
project only on smaller-scale development and testing of the technology.
ARPA-E officials told us the larger-scale demonstration could likely be
funded by the private sector. 15




14
  All the ARPA-E program directors who had finished their selection process at the time
we spoke with them told us that for a project to have a high risk level is not one of their
other considerations when evaluating projects, but that the nature of the agency’s work
has the effect of ultimately funding high-risk projects. Several program directors said that a
high risk level is inherent in projects with high technological impact.
15
  Alternatively, ARPA-E can consider only funding a specific part of research that must be
proven before additional work is funded. For example, one applicant’s carbon capture
project involved a new means of solidifying carbon dioxide. The program director told us
he wanted the applicant to first demonstrate that the approach would work, so he reduced
the initial amount of funding by over 60 percent.




Page 11                                                                 GAO-12-112 ARPA-E
ARPA-E Also Considers       When making award decisions or adjusting the scope of proposed
Applicants’ Prior Sources   projects, ARPA-E program directors may also consider the identification
of Private Funding, but     in applications of sources of private funding and the extent to which that
                            funding might support the proposed projects. This information can help
Most Award Winners We       provide program directors with assurance that ARPA-E funds do not
Reviewed Did Not Explain    overlap with private investment. During the first two funding rounds,
This Information            ARPA-E required that applicants identify relevant private investors if the
                            applicant believed these funds were related to the proposed project. Of
                            the 18 applications we reviewed from award winners that we identified as
                            having received private venture capital, 14 applied during ARPA-E’s first
                            and second funding rounds. Most of these award winners did not explain
                            why investors were not willing to fund proposed work. ARPA-E program
                            directors and an ARPA-E official, speaking on behalf of the agency, told
                            us they took additional steps to clarify outstanding prior funding questions
                            when ARPA-E was aware that applicants had received private sector
                            funding. For example, one applicant we reviewed from the first funding
                            round had previously received substantial private funding for work that
                            appeared very similar to its proposed ARPA-E project. ARPA-E officials
                            told us they were initially unable to determine why the private investor
                            was not willing to also fund the proposed ARPA-E project and that the
                            company’s application did not include an explanation. ARPA-E officials
                            told us that getting this information required them to draft a series of direct
                            and detailed questions that elicited several pages of written responses
                            from the applicant. ARPA-E officials also told us they conducted multiple
                            rounds of written and oral follow-up with the applicant and the private
                            investor. Through these efforts, ARPA-E determined that the
                            technological risks of key parts of the project were too high for the private
                            investor and therefore decided to fund the research. 16

                            Because ARPA-E officials recognized the need for applicants to provide
                            better prior funding explanations, beginning with its third funding
                            announcement, the agency required applicants to explain why proposed
                            work was not sponsored internally if the applicant was a large company,
                            or why private investors were not willing to support the project if the
                            applicant was a small business or start-up company. ARPA-E did not
                            provide guidance on how applicants should respond to this additional
                            requirement by, for example, providing a sample response. Of the 18


                            16
                              ARPA-E program directors told us they also conducted review efforts in a number of
                            other cases where it was initially unclear why other entities would not provide additional
                            funds to ARPA-E award winners.




                            Page 12                                                                 GAO-12-112 ARPA-E
ARPA-E award-winning companies, 4 applied during the third funding
round, and these companies provided a range of information in response
to this new requirement in their funding applications. Two explained how
ARPA-E funds would allow them to go beyond currently funded work but
did not provide reasons why investors were not willing to support the
proposed work. Another wrote only that the ARPA-E research was too
risky for the company’s private investors. One application contained an
explanation outlining the specific research its private investors were and
were not willing to fund. This applicant explained that private funds were
directed toward lower-risk and higher-cost technologies. 17 This application
also included a letter from the company’s venture capital investors that
explained which parts of its research the investors were planning to
continue funding and which research was too risky for them, although not
requested by ARPA-E. This letter provided additional third-party support
for the funding information in the application. Officials from the National
Institute of Standards and Technology’s (NIST) Technology Innovation
Program told us they request that applicants provide letters from private
investors to document why applicants’ projects could not be privately
funded. 18

When we followed up with the 18 companies, they were generally able to
explain to us why their private investors were not willing to undertake the
additional risk and uncertainty associated with the proposed projects. 19
When we examined the data in the VentureDeal database for a number of
applicants, the data allowed us to quickly cross-check the names of prior
private investors that applicants reported to ARPA-E. ARPA-E officials
said that they have not used such data for these purposes but that they
have considered doing so. Without an examination of outside venture
capital data on its applicants, the agency may be missing a time-saving
opportunity to check information on private funding provided in
applications, especially in instances where applicants may not have been
thorough in their explanations. We found a number of readily available
subscription-based venture capital data services that provided company


17
  ARPA-E officials told us that they funded only the parts of this project that were too risky
for the private investors.
18
  According to NIST, the Technology Innovation Program funds advanced research in
areas of critical national need including health care, robotics, and civil infrastructure.
19
  Most of these explanations related to the risk or uncertainty associated with the
proposed projects.




Page 13                                                                  GAO-12-112 ARPA-E
                        names, transaction amounts, and funding purposes. We found that the
                        web-based VentureDeal database matched formats and data available
                        from other venture capital data services.


                        Our review suggests that most ARPA-E-type projects could not be funded
Most ARPA-E Projects    solely by private investors. Private venture capital firms told us that,
Likely Could Not        among other considerations, they generally do not invest in projects that
                        cannot be commercialized in less than 3 years. Nearly all of the 13
Have Been Funded        ARPA-E award winners and most of the 22 of the contingently selected
Solely by Private       applicants we spoke with estimated that their projects were 3 or more
Investors               years away from a potential market-ready product (i.e.,
                        commercialization). In addition, we found that only 2 of the 22
                        contingently selected applicants we spoke with that met ARPA-E’s
                        selection criteria but were not selected for an award subsequently
                        secured private funding.


Venture Capitalists     The representatives we spoke with from six venture capital firms identified
Generally Do Not Fund   three factors that limit the general availability of venture capital funding for
Projects That ARPA-E    new energy technologies. These factors were consistent with data we
                        analyzed for the 121 award winners from ARPA-E’s first three funding
Looks to Fund           rounds, the sample of 13 award winners we interviewed from these
                        funding rounds, and the 22 contingently selected applicants we
                        interviewed.

                        First, venture capital firms generally do not fund projects that rely on
                        unproven technological concepts or lack working prototypes
                        demonstrating the technology. A number of venture capital firm
                        representatives told us that they are generally not willing to fund the
                        applied scientific research sometimes required by ARPA-E-type projects.
                        Projects they fund generally focus on developing technologies based on
                        known scientific principles. Data from ARPA-E on award winners show
                        that 91 out of 121 ARPA-E projects from the first three funding rounds
                        had technological concepts that had not yet been demonstrated in a
                        laboratory setting. 20 According to a recent report from the American
                        Energy Innovation Council, private investors consider these projects too


                        20
                          These data showed that most ARPA-E award winners were at or below technology
                        readiness level (TRL) 3. TRL 3 represents a level where potential technologies are still
                        unproven.




                        Page 14                                                                GAO-12-112 ARPA-E
high risk for investment, even for concepts with promising technological
potential. 21 Most of the contingently selected applicants we spoke with––
17 out of the 22––told us they were unlikely to receive funding from other
sources for their proposed projects because of high levels of scientific
uncertainty, an unavailable or undeveloped market, or a lack of a working
prototype. For example, one such applicant said that he only had a
computer model suggesting that his high-efficiency air conditioning device
would work, which was insufficient to convince potential private investors.
In addition, many of the ARPA-E award winners we surveyed also
recognized the inherent uncertainty in their research; 5 of the 13 told us
that their projects had a fairly low probability of success. 22

Second, venture capital firms seek more rapid returns on investment and
closely analyze a project’s potential return on investment over time, a
factor that influences their decisions to invest in projects that are in later
stages of development and closer to commercialization. Venture capital
firm officials told us that they focused closely on the timeliness of
investment returns, with one firm noting that the industry tended to invest
in technologies that could be commercialized in less than 3 years and that
would potentially exhibit exponential market growth in approximately 5 to
7 years. However, we found that nearly all of the ARPA-E award winners
and most contingently selected applicants we spoke with estimated that
their projects were 3 or more years away from potential
commercialization. 23 For example, 12 out of 13 ARPA-E award winners
estimated that it would take at least 3 years for their ARPA-E projects to
reach the commercialization stage with ARPA-E funding. 24 Had they not



21
  American Energy Innovation Council, Catalyzing American Ingenuity: The Role of
Government in Energy Innovation (Sept. 2011). The American Energy Innovation Council
is a bipartisan group of American business leaders.
22
  The 18 award winners we identified as having received prior private venture capital told
us they were generally able to pursue the development of energy technologies with
greater scientific or technical uncertainty with the ARPA-E funding than what they were
working on with their private funding. See appendix III for more detail on these companies'
prior privately funded research.
23
  On the basis of the initial testing of our questions, we determined that these estimates
may be optimistic, given that respondents are invested in attempting to bring a technology
to market as soon as possible.
24
  For the 12 award winners, 6 estimated their projects would take 3 to 5 years to reach
commercialization, and the remaining 6 estimated their projects would take more than 5
years to do so.




Page 15                                                               GAO-12-112 ARPA-E
received ARPA-E funding, most of these award winners––10 out of 13––
told us they either would not have pursued their ARPA-E project or that
they would not have been able to develop a commercial product in less
than 10 years. At the same time, 18 out of 22 ARPA-E contingently
selected applicants estimated it would take at least 3 years for their
projects to reach commercialization if they had been able to secure
funding for the proposal they submitted to ARPA-E.

Third, venture capital firms may not be comfortable investing in new
energy technologies, noting the historical lack of successful venture
capital investments in these types of projects. Venture representatives
said that venture firms were more comfortable investing in software
companies or other businesses with higher potential profit margins and
less costly product development than new energy technologies. One
venture representative noted that his firm looked to invest in products with
potential gross profit margins of 50 percent or more. In addition, these
representatives noted that it is difficult for new advanced energy
technologies to compete with well-established and low-margin traditional
sources of energy like natural gas. Venture representatives also noted
that venture firms had become more risk averse and reluctant to fund new
energy technologies after lackluster investment returns have made the
venture industry more aware of the challenges associated with investing
in unproven energy technologies.

While venture capital firms generally do not fund projects that ARPA-E
looks to fund, our work suggests that receiving ARPA-E project funding
may have a positive effect on some award winners’ ability to attract
follow-on funding from the private sector for their ARPA-E work. For
example, ARPA-E’s data indicate that 18 out of 121 ARPA-E award
winners from ARPA-E’s first three rounds of funding had received private
sector funding totaling $318 million after receiving ARPA-E funding. 25 In
some cases, award winners received private follow-on funding
immediately after receiving ARPA-E funding. A number of the award
winners we spoke with stated that, given the highly competitive nature of
the program, receiving ARPA-E funding served as a “stamp of approval”
to venture capital or other private firms. 26 These award winners told us


25
  A number of these 18 award winners with follow-on funding were part of the 18
companies we identified as having received private funding prior to ARPA-E.
26
 These award winners included those that had prior venture funding as well as the others
we interviewed.




Page 16                                                             GAO-12-112 ARPA-E
                            that an ARPA-E award served as a signal of scientific and financial
                            approval for potential investors. Economists call this rapid follow-on
                            private funding a certification effect, which may explain the experiences of
                            some of these award winners. 27 This effect suggests that public awards
                            address information gaps that might have otherwise precluded private
                            investment. Some award winners and economists we spoke with told us
                            that the government was suited to identifying technical risks because of
                            its ability to draw on the expertise of many scientific reviewers, while
                            venture firms may not have the scientific expertise on hand to fully
                            understand potential investments. Furthermore, economic literature
                            suggests that the certification effect may be particularly relevant in the
                            high-technology industries, where the venture capital community plays an
                            important role and in which traditional financial measures of risk and
                            returns on investments may prove insufficient. 28 Appendix III has more
                            information on the difference between research funded by selected award
                            winners’ prior investors and ARPA-E funded work.


Few Contingently Selected   Eighteen of the 22 ARPA-E contingently selected applicants we
Applicants Found Funding    interviewed sought funding after being turned down for ARPA-E funds. Of
from Private Investors or   the 18 that sought funding elsewhere, 13 submitted project proposals to
                            government sources, such as other DOE offices, the National Science
Public Sources              Foundation, or nonprofit academic research institutes, and the remaining
                            5 submitted proposals to private investors such as venture capital firms. 29
                            As of September 2011, we found that 2 out of the 22 contingently
                            selected applicants secured funding from venture capital firms for work
                            that was very similar to their ARPA-E project proposals. 30 We also found
                            that 4 contingently selected applicants secured funding from a


                            27
                              See Andrew A. Toole, Calum Turvey. “How Does Initial Public Financing Influence
                            Private Incentives for Follow-on Investment in Early-Stage Technologies?” Journal of
                            Technology Transfer, 34: 43-58 (2007). See also Josh Lerner. “The Government as
                            Venture Capitalist: The Long-Run Impact of the SBIR Program,” Journal of Business, vol.
                            72, no.3, 285-318 (1999).
                            28
                              Lerner, “The Government as Venture Capitalist: The Long-Run Impact of the SBIR
                            Program.”
                            29
                              One of the 5 contingently selected applicants that sought funding from a private investor
                            also sought public funding.
                            30
                              In addition, our review of venture capital funding data for the other 11 contingently
                            selected applicants with whom we did not speak did not show that any had received
                            venture capital funding since not being awarded ARPA-E funds.




                            Page 17                                                                 GAO-12-112 ARPA-E
government or nonprofit source for their projects. 31 In addition, we found
that most contingently selected applicants modified their ARPA-E project
proposals to attract subsequent funding for their projects by reducing the
scope of their proposals or by focusing on more basic science research.
For example, the 4 contingently selected applicants that secured funding
from a government or nonprofit source modified their ARPA-E proposals
to be more focused on basic science research, rather than on developing
a commercial technology. In addition, 1 of these applicants told us that
the funding will allow it to continue exploring fundamental materials
science rather than developing a product. Also, many contingently
selected applicants and award winners said that other government
sources were limited. Some noted that Small Business Innovation
Research (SBIR) grants would not allow them to make as much progress
as larger ARPA-E awards. 32 Finally, a number of others mentioned that
military funding agencies were not as focused on developing low-cost
technologies with broader market appeal, because aerospace or military
applications do not need to achieve the same low costs and market
appeal as consumer or commercial applications.




31
  Three contingently selected applicants that submitted proposals to government or
nonprofit sources were still awaiting responses.
32
  Federal agencies that have budgets in excess of $100 million for research conducted by
others (extramural research) are required to use 2.5 percent of these budgets to establish
and operate a Small Business Innovation Research program. The program is intended to
stimulate technological innovation, use small businesses to meet federal research and
development needs, foster and encourage participation by minority and disadvantaged
persons in technological innovation, and increase private sector commercialization of
innovations derived from federal research and development. Eleven federal agencies
participate in the SBIR program, with $16 billion awarded to date. Initial SBIR awards
normally do not exceed $150,000 and are usually for a period of 6 months.




Page 18                                                               GAO-12-112 ARPA-E
                      According to ARPA-E officials and documents, agency officials have
ARPA-E Officials      taken steps to coordinate with other DOE offices in advance of awarding
Have Taken Steps to   ARPA-E funds to help avoid duplication of efforts. These coordination
                      efforts can be categorized into three areas:
Coordinate with
Other Department of   Prefunding coordination. ARPA-E officials told us that program directors
Energy Offices in     engage with officials from related DOE offices in advance of announcing
                      the availability of ARPA-E funds. ARPA-E program directors told us that
Advance of Awarding   early in the development of a funding announcement, they conduct
Funds                 outreach with industry, academic, and government officials both inside
                      and outside of DOE in an attempt to identify funding gaps related to the
                      technology they wish to develop. For example, by doing such outreach,
                      one program director determined that there had been little funding at DOE
                      or elsewhere for lithium air or lithium sulfur batteries, which have the
                      potential to last significantly longer than existing lithium ion batteries.
                      Program directors also hold workshops and invite relevant participants,
                      including those from other DOE offices and from other federal agencies,
                      to identify technologies that have little to no existing research funding but
                      that have transformational potential. ARPA-E officials told us that
                      directors use the workshops and other meetings to identify research
                      areas that other DOE offices are not working on, and the other DOE
                      officials provide insights on funding areas where they are not active. For
                      example, one of these ARPA-E program directors told us that he met with
                      officials from DOE’s Office of Electricity Delivery and Energy Reliability
                      and the Solar Energy Technologies Program within the Office of Energy
                      Efficiency and Renewable Energy (EERE) before announcing available
                      funds for the electrical power electronics funding announcement. 33
                      According to this program director, this coordination helped him identify
                      that there had been little funding for the development of magnetic devices
                      for use in electrical power electronics. He ultimately designed the ARPA-
                      E electrical power electronics funding announcement to focus, in part, on
                      the development of improved magnetic devices because of the lack of
                      funding elsewhere.

                      In addition to inviting officials from other DOE offices to ARPA-E
                      workshops, program directors told us they also engage with other DOE
                      officials in other ways, both formally and informally. The program director



                      33
                        Electrical power electronics modify electrical energy (i.e., change its voltage, current, or
                      frequency) and can be found in applications like lighting, motors, and electric vehicles.




                      Page 19                                                                  GAO-12-112 ARPA-E
responsible for ARPA-E’s work on advanced batteries said that he was a
member of DOE’s Energy Storage Technology Development Team and
regularly met with other officials who are engaged in applied battery
research. This director said that it had become clear that DOE’s Vehicle
Technologies Program will continue to focus on incremental
improvements to existing lithium ion battery technologies that are
currently on the market, while ARPA-E will fund newer, alternative battery
technologies. Other program directors told us that they have regular
discussions with counterparts within DOE to avoid duplicating efforts,
although through other means than a formal committee.

Coordination of application reviews. Some ARPA-E program directors told
us that they have recruited officials from other DOE offices to review
applications submitted to ARPA-E and that these officials made up as
many as one-third of the reviewers for one director. These application
reviewers rate and recommend proposals for potential ARPA-E funding.
ARPA-E program directors told us that these DOE reviewers help them
stay aware of the types of projects that other DOE offices are funding. For
example, according to one program director, DOE reviewers indicated on
a number of occasions that an ARPA-E advanced battery applicant would
be better suited for funding under DOE’s Vehicle Technologies Program
because it was for a more developed technology. ARPA-E has also used
application reviewers from other federal agencies, such as the
Department of Defense. One program director told us that these
reviewers have also helped avoid funding projects similar to those
potentially funded elsewhere.

Official DOE coordination groups. ARPA-E is also a participant in DOE’s
SunShot Initiative within the Solar Energy Technologies Program. The
SunShot Initiative is an effort to coordinate solar energy research across
DOE’s Office of Science, four national laboratories, the National Science
Foundation, and ARPA-E, with the goal of achieving costs of $1 per watt
for solar-generated electricity. One ARPA-E program director is a
member of the SunShot Initiative advisory board and therefore able to
coordinate ARPA-E solar-related activities with other SunShot Initiative
members. SunShot Initiative officials told us that DOE plans to make it a




Page 20                                                   GAO-12-112 ARPA-E
              model for DOE’s internal coordination efforts and that DOE hopes to
              expand the approach to other research areas. 34

              Additionally, the ARPA-E Director created the Panel of Senior Technical
              Advisors (PASTA), which is a group of high-level DOE managers that
              meet periodically to discuss current and future DOE research efforts.
              ARPA-E officials told us that PASTA is an attempt to avoid duplicating
              efforts within DOE. PASTA meeting attendees have included officials
              from DOE’s applied and basic science offices.

              We were not able to directly evaluate the effectiveness of ARPA-E’s
              efforts to coordinate with other DOE offices. Nevertheless, we found that
              on the basis of our interviews with ARPA-E award winners and
              contingently selected applicants, four award winners and two contingently
              selected applicants had received prior funding from other DOE offices. 35
              According to these award winners and contingently selected applicants,
              the prior funding was either for more proven technologies or was focused
              on more basic or foundational research than was the ARPA-E funded
              project.


              ARPA-E recognizes the need to ensure that the agency is not funding
Conclusions   projects that would be otherwise funded by the private sector, and has
              taken steps to get information from applicants on their other sources of
              funding. The agency has also taken steps to coordinate with other DOE
              offices in advance of awarding ARPA-E funds. However, for the
              applications we reviewed, we found that ARPA-E’s current funding
              announcements have generally yielded limited information from
              applicants that had prior sources of private funding. Where applicants
              provided little information, ARPA-E’s program directors spent time and
              resources to determine the extent of such funding for projects related to
              or similar to the applicants’ proposed ARPA-E projects. The agency’s



              34
                In addition, SunShot Initiative program officials told us they are trying to share elements
              of ARPA-E’s selection process with other offices within DOE—current DOE funding
              announcements generally take 15 months to carry out, compared with 6 to 8 months for
              ARPA-E. For example, ARPA-E officials also told us that DOE’s Energy Efficiency and
              Renewable Energy office is adopting their online application system because of its
              advantages over paper-based systems.
              35
                These award winners included those in our nonprobability sample of 13, as well as the
              18 we identified with VentureDeal data.




              Page 21                                                                  GAO-12-112 ARPA-E
                      requirements for information on private sector funding could be improved.
                      For example, ARPA-E does not provide guidance to applicants, such as a
                      sample response, on how to meet its information requirement on prior
                      private funding. An approach used by another federal program that funds
                      advanced research is for applicants to provide letters from private
                      investors to document why their projects could not be privately funded.
                      This approach was used by one ARPA-E award winner, who included a
                      letter from the company’s venture capital investors to explain why the
                      investors were not willing to fund the project proposed to ARPA-E. Also,
                      ARPA-E officials said that they have not used venture capital data to
                      identify applicants with prior private investors and to check information
                      applicants provide to them, but that they have considered doing so.
                      Examining such data allowed us to quickly cross-check applicants’ self-
                      reported prior private funding. Without additional tools to better
                      understand prior private funding, ARPA-E program directors will continue
                      to spend time and agency resources taking additional steps to clarify prior
                      private funding and may miss opportunities to avoid duplication with
                      private investors.


                      To ensure that ARPA-E uses a more complete range of methods to
Recommendations for   ensure that limited federal funds are targeted appropriately, we
Executive Action      recommend that the Secretary of Energy consider taking the following
                      three actions:

                      •   provide guidance with a sample response to assist applicants in
                          providing information on sources of private funding for proposed
                          ARPA-E projects,
                      •   require that applicants provide letters or other forms of documentation
                          from private investors that explain why investors are not willing to fund
                          the projects proposed to ARPA-E, and
                      •   use venture capital funding databases to help identify applicants with
                          prior private investors and to help check information applicants
                          provide on their applications.


                      We provided a copy of our draft report to ARPA-E for review and
Agency Comments       comment. ARPA-E concurred with key findings and our recommendations
and Our Evaluation    in its written comments, which are reproduced in appendix IV. In its
                      comments, ARPA-E outlined the steps that the agency plans to take to
                      address our recommendations. ARPA-E also provided additional
                      clarifying comments, which we incorporated.




                      Page 22                                                    GAO-12-112 ARPA-E
As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from the
report date. At that time, we will send copies of this report to the
appropriate congressional committees, the Secretary of Energy, and
other interested parties. In addition, the report will be available at no
charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact
me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices
of Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made major contributions to this report
are listed in appendix V.




Frank Rusco
Director, Natural Resources and Environment




Page 23                                                   GAO-12-112 ARPA-E
Appendix I: Scope and Methodology
             Appendix I: Scope and Methodology




             To examine the Advanced Research Projects Agency-Energy’s (ARPA-E)
             use of criteria and other considerations for making awards and the extent
             to which applicants identify and explain other private funding information,
             we reviewed 20 applications drawn from a nonprobability sample of the
             4,788 applicants ARPA-E received during its first three funding rounds.
             We selected applications from a range of ARPA-E technology program
             areas to which the applications were submitted and applicant institution
             types (e.g., small company or university). Because we selected a
             nonprobability sample of applications to review, information we collected
             cannot be generalized to all applicants; however, it provided us with an
             understanding of ARPA-E’s criteria and other considerations for making
             an award. We also interviewed and reviewed the applications from our
             sample of 18 award winners, which were private companies that we
             identified as having received funding from private investors prior to
             receiving an ARPA-E award. We identified these 18 companies by
             searching for evidence of prior private funding for the 121 award winners
             in the VentureDeal venture capital database. 1 In our review of these
             applications, we focused on the extent to which applicants disclosed prior
             private funding. 2 We also spoke with all eight ARPA-E program directors
             to discuss ARPA-E’s process for making awards and managing projects
             of award winners.

             To analyze the extent to which ARPA-E projects could have been funded
             through the private sector, we conducted three sets of interviews with
             ARPA-E applicants and award winners. Specifically,

             •   We conducted structured interviews with 22 of the 33 contingently
                 selected applicants that ARPA-E encouraged to submit full
                 applications during its second and third funding rounds. 3 Each of the


             1
              VentureDeal is a company that maintains data on private venture capital funding. The
             VentureDeal database only includes businesses and does not include data on venture
             capital funding secured by universities or national laboratories. However, according to a
             VentureDeal official, universities and national laboratories are not likely to directly receive
             venture funding. In addition, we were not able to verify the completeness of VentureDeal
             data, and there may have been ARPA-E award winners that had prior private funding that
             did not appear in these data. We identified 1 of these award winners through an
             examination of ARPA-E data.
             2
              Eight of these 18 companies also appeared in the sample of 20 applicants that we
             selected to examine ARPA-E's criteria for making awards.
             3
              The remaining 11 contingently selected applicants did not respond to our requests for an
             interview.




             Page 24                                                                   GAO-12-112 ARPA-E
Appendix I: Scope and Methodology




    contingently selected applicants fulfilled ARPA-E’s selection criteria,
    had the same characteristics as ARPA-E award winners, and,
    according to ARPA-E officials with whom we spoke, would have been
    selected for an award had additional funds been available. 4 This
    approach allowed us to consider the potential of ARPA-E-type
    projects to receive private funding.

•   We conducted structured interviews with a nonprobability sample of
    13 award winners selected from ARPA-E’s first three funding rounds. 5
    We selected subjects for this sample across a range of ARPA-E
    award winner characteristics, including the technology program area
    for which an award winner received funding, the stage of development
    of an award winner’s project, and an award winner’s type of institution
    (e.g., small company or university). Because this was a nonprobability
    sample, the information from these structured interviews cannot be
    generalized to all award winners but can provide examples about
    award winners’ experiences. We conducted content analyses 6 of the
    award winners’ and contingently selected applicants’ interview
    responses to quantify issues such as the ability of each group to
    secure private sector funding for ARPA-E-type projects. 7

•   Third, we spoke with the 18 ARPA-E award winners we identified
    through the VentureDeal database to discuss key differences between
    their prior research and their ARPA-E-funded projects. 8




4
 We spoke with 1 applicant twice because the applicant applied for two different projects.
For each of the 33 contingently selected applicants, we also searched for entries in the
VentureDeal database to determine if any had received private venture capital funding
prior to and following ARPA-E's selection process.
5
 We selected 15 award winners as part of our nonprobability sample but 2 award winners
did not respond to our interview requests.
6
 Content analysis is a systematic research method for analyzing textual information in a
standardized way that allows evaluators to make inferences about that information.
7
 We conducted pretests of our structured interview questions with 3 award winners and 3
contingently selected applicants to ensure that the questions were understandable and
unbiased.
8
 Four of these 18 award winners also appeared in our nonprobability sample of 13 award
winners.




Page 25                                                               GAO-12-112 ARPA-E
Appendix I: Scope and Methodology




We also conducted interviews with a variety of companies and individuals
knowledgeable about research associated with ARPA-E-type projects,
including six venture capital firms 9 and the National Venture Capital
Association (NVCA), a trade association, to determine the availability of
private capital for ARPA-E-type projects and the criteria venture capital
firms apply in making their investment decisions; two additional public
companies that were awarded ARPA-E funding to discuss the ability of a
public company to internally fund research; 10 and three economists to
discuss the role and effectiveness of government-funded research and
development of technology. 11

To examine the extent to which ARPA-E coordinates with other DOE
programs to avoid duplicating efforts, we spoke with the ARPA-E program
directors as well as officials from other DOE program offices including the
Office for Energy Efficiency and Renewable Energy (EERE) and the
Office of Science. In addition, we met with officials from the SunShot
Initiative, which is a collaboration among EERE, the Office of Science,
and ARPA-E to make solar energy technologies cost-competitive with
other forms of energy. We also spoke with officials from the Defense
Advanced Research Projects Agency (DARPA) and the Department of
Energy (DOE) Office of Inspector General. During our interviews with the
award winners and contingently selected applicants previously
mentioned, we asked them to discuss their understanding of other
potential sources of DOE funding for their projects.

To assess the reliability of data from ARPA-E and VentureDeal, we
reviewed relevant documentation and interviewed key data system
officials at ARPA-E and VentureDeal and determined that the data were
sufficiently reliable for the purposes of this report.

We conducted this performance audit from November 2010 to December
2011 in accordance with generally accepted government auditing



9
 These firms were Khosla Ventures; Kleiner, Perkins, Caufield, and Byers; U.S. Venture
Partners; Rockport Capital; Mohr-Davidow Ventures; and Polaris Ventures. We selected
these venture capital firms because they were knowledgeable about ARPA-E-funded
companies according to NVCA.
10
 These two companies did not appear in our earlier samples.
11
  We selected these economists based on their publication of literature on the role and
effectiveness of government-funded research.




Page 26                                                               GAO-12-112 ARPA-E
Appendix I: Scope and Methodology




standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 27                                                  GAO-12-112 ARPA-E
Appendix II: ARPA-E Program Technology
             Appendix II: ARPA-E Program Technology
             Areas



Areas

             Funding Round 1
             •   Biomass energy. Biomass energy projects focus on means to convert
                 crops, along with plant waste from other industrial processes, into
                 energy through chemical, biological, or thermal techniques.
             •   Building efficiency. Building efficiency projects focus on technologies
                 that heat, power, and maintain buildings.
             •   Carbon capture. Carbon capture and sequestration projects seek to
                 create new methods to prevent the release of carbon dioxide into the
                 atmosphere from traditional fossil fuel sources such as coal, natural
                 gas, and petroleum.
             •   Conventional energy. Conventional energy projects seek to
                 significantly increase the efficiency of traditional fossil fuel power
                 production and reduce waste generated from this use.
             •   Direct solar fuels. Direct solar fuel projects seek to utilize
                 photosynthetic microorganisms to produce liquid fuels and fuel
                 precursors directly from solar energy.
             •   Energy storage. Energy storage projects seek to revolutionize battery,
                 capacitor, and other energy storage methods for significantly
                 improved efficiency.
             •   Energy-efficient water purification. Water technology projects seek to
                 reduce the water intensity of the electricity and fuel sectors and,
                 reciprocally, to reduce the energy intensity of the water sector.
             •   Renewable power. Renewable power projects focus on innovative
                 technologies in several sustainable energy areas such as extremely
                 efficient photovoltaic solar collectors, wind turbines, and geothermal
                 energy.
             •   Vehicle technologies. Vehicle technology projects seek to advance
                 efficiency in vehicles through technologies like new hybrid engines to
                 those that convert on-board waste heat to electricity.
             •   Waste heat capture. Waste heat capture projects seek to use thermal
                 energy expelled by traditional industrial processes, such as coal
                 smokestacks, and efficiently convert that heat into electricity.

             Funding Round 2
             •   Batteries for transportation. Batteries for Electrical Energy Storage in
                 Transportation (BEEST) projects seek to develop batteries for plug-in
                 hybrid electric vehicles (PHEV) and electric vehicles (EV) that can
                 make a 300- to 500-mile-range electric car.



             Page 28                                                       GAO-12-112 ARPA-E
Appendix II: ARPA-E Program Technology
Areas




•   Materials for carbon capture. Innovative Materials and Processes for
    Advanced Carbon Capture Technologies (IMPACCT) projects seek to
    reduce the cost of carbon capture significantly through a combination
    of new materials, improvements to existing processes, and
    demonstration of new capture processes.
•   Electrofuels. Electrofuels projects intend to explore new paradigms for
    the production of renewable liquid fuels that are compatible with
    today’s infrastructure. They seek to use microorganisms to harness
    chemical or electrical energy to convert carbon dioxide into liquid fuels
    without using petroleum or biomass.

Funding Round 3
•   Grid-scale electricity storage. Grid-Scale Rampable Intermittent
    Dispatchable Storage (GRIDS) projects seek to develop new energy
    storage technologies that are comparable in reliability and cost to
    pumped hydropower and that are modular and can be deployed in
    any location in the country.
•   Building efficiency. Building Energy Efficiency Through Innovative
    Thermodevices (BEETIT) projects focus on developing new
    approaches and technologies for cooling equipment used in heating,
    ventilating, and air conditioning (HVAC) systems in buildings, as well
    as in refrigeration.
•   Electrical power electronics. Agile Delivery of Electrical Power
    Technology (ADEPT) projects strive to reinvent the basic building
    blocks of circuits from transistors, inductors, and transformers to
    capacitors for a broad spectrum of power applications. ADEPT
    focuses on two areas: (1) creating the world’s first kilovolt-scale
    integrated circuits, and (2) developing transistor switches operating at
    grid-level voltages that would exceed 13 kilovolts.

Funding Round 4
•   Advanced thermal storage. High Energy Advanced Thermal Storage
    (HEATS) projects seek to develop revolutionary cost-effective thermal
    energy storage technologies. HEATS focuses on three areas: (1)
    high-temperature storage systems to deliver solar electricity more
    efficiently around the clock to allow nuclear and fossil base load
    resources the flexibility to meet peak demand, (2) fuel produced from
    the sun’s heat, and (3) HVAC systems that use thermal storage to
    dramatically improve the driving range of electric vehicles.




Page 29                                                    GAO-12-112 ARPA-E
Appendix II: ARPA-E Program Technology
Areas




•   Electricity network integration. Green Electricity Network Integration
    (GENI) projects focus on innovative control software and high-voltage
    hardware to reliably control the grid network. GENI focuses on two
    areas: (1) cost-optimizing controls to manage sporadically available
    sources, such as wind and solar, alongside coal and nuclear, and (2)
    resilient power flow control hardware to enable automated, real-time
    control of grid components.
•   Plants Engineered To Replace Oil (PETRO). PETRO projects seek to
    advance technologies that optimize the biochemical processes of
    energy capture and conversion in plants to develop farm-ready crops
    that deliver more energy per acre with less processing.
•   Rare earth alternatives for energy technologies. Rare Earth
    Alternatives in Critical Technologies for Energy (REACT) projects
    work on early-stage technology alternatives that reduce or eliminate
    dependence on rare earth materials that may jeopardize the
    widespread adoption of many critical energy solutions by developing
    substitutes in two key areas: electric vehicle motors and wind
    generators.
•   Solar electrical power technology. Solar Agile Delivery of Electrical
    Power Technology (Solar ADEPT) projects focus on integrating
    advanced power electronics into solar panels to extract and deliver
    energy more efficiently. Solar ADEPT projects are centered on
    advances in magnetics, semiconductor switches, and charge storage.




Page 30                                                   GAO-12-112 ARPA-E
Appendix III: Description of ARPA-E Award
              Appendix III: Description of ARPA-E Award
              Winners with Prior Private Investment



Winners with Prior Private Investment

              The 18 award winners we identified as having received prior private
              venture capital told us that with the ARPA-E funding, they were generally
              able to pursue the development of energy technologies with greater
              scientific or technical uncertainty than they had when they were working
              with their private funding. About two-thirds of these award winners told us
              that the ARPA-E funding is allowing them to develop prototypes or to
              prove basic technology concepts on more advanced ideas than their prior
              work—6 of these award winners said this was for completely new
              research and 7 said it was for major advancements to prior research. A
              few of these award winners also told us they were able to work on
              projects with outstanding scientific research questions that private
              investors would not have funded. Five of these award winners reported
              that they would likely have been able to pursue some research similar to
              their ARPA-E projects, but it would have taken years longer without
              ARPA-E funding.

              The following three examples reflect in more detail much of what we
              heard from these 18 award winners regarding the distinction between
              research funded by their prior investors and ARPA-E funded work:

              •   Sun Catalytix. Sun Catalytix was founded by a professor at the
                  Massachusetts Institute of Technology to commercialize a set of
                  catalysts to split water into hydrogen and oxygen gases. This reaction
                  allows these gases to be cheaply produced for a variety of purposes,
                  including renewable energy. Sun Catalytix was initially funded by a
                  Boston area venture capital firm to develop a product based on these
                  catalysts. According to a representative from this firm, the venture
                  capital funding allows Sun Catalytix to attempt to develop a product
                  that would potentially earn the venture firm a return on investment in a
                  reasonable amount of time. At the time of the ARPA-E award, Sun
                  Catalytix representatives told us they were still some years away from
                  a commercial product using this new technology. According to these
                  representatives, had the firm not won an ARPA-E award, further
                  venture capital might not have been available to develop an initial
                  version of their products. Sun Catalytix representatives told us that
                  ARPA-E funds allowed the company to conduct additional applied
                  scientific research that led to their discovery of a new platinum-free
                  and therefore lower-cost catalyst with much wider market potential,
                  including renewable energy applications. A representative from the
                  venture firm told us that the firm would not have funded the additional
                  advanced scientific research needed to develop the new, cheaper
                  catalyst.




              Page 31                                                    GAO-12-112 ARPA-E
Appendix III: Description of ARPA-E Award
Winners with Prior Private Investment




•   Agrivida. This small biotechnology company based in the Boston area
    is developing genetically modified sorghum, corn, and switchgrass
    crops for use in biofuel production. Agrivida representatives explained
    that the goal of their ARPA-E project is to generate crops capable of
    producing enzymes within the plant itself to internally break down the
    plant’s own cellulose after harvest. This technology would significantly
    lower the costs of cellulosic biofuel production, because enzyme
    treatments are currently a large part of the costs of current production
    methods. Before winning an award from ARPA-E, Agrivida had
    received venture capital funding to develop the technology. We spoke
    with a representative of the venture firm that funded Agrivida, who told
    us that this venture funding was only for research on the corn crop
    enzymes; the firm was not willing to fund additional research on other
    crops because the amount of funding it could provide to any one
    company in the early stages of research was limited. Agrivida officials
    told us that the ARPA-E award allowed them to expand the scope of
    their work and conduct additional research on switchgrass, which may
    have potential to become a major biofuel crop. They said that the
    ARPA-E funds have enabled rapid progress, allowing them to
    complete laboratory work in 1 year that would have otherwise taken 5
    years. Officials from Agrivida said they hope to have made enough
    progress by the time they complete their ARPA-E research to be able
    to attract additional investors and secure commercialization partners.

•   24M. This is a startup company that is developing flow batteries for
    use in transportation and electrical grid applications. Unlike normal
    batteries, flow batteries generate electrical current by internally
    circulating electrically active liquids, which allows for much lower costs
    than traditional batteries. However, flow batteries do not exist for use
    in tight spaces like cars where their cost advantages could allow for
    significant improvements to electric vehicles. In 2010, concurrent with
    its ARPA-E award, 24M received $10 million from two venture capital
    firms to develop flow batteries for consumer and commercial
    applications. A representative from one of these venture firms told us
    that his firm would not have been confident in funding the 24M project
    without ARPA-E involvement. Representatives from 24M said that the
    ARPA-E award was critical to their ability to secure private investment
    and to launch the company and that they now expect to have a
    working prototype by the end of their ARPA-E project.

In addition, the two public companies we spoke with that were awarded
ARPA-E money told us that although their companies had internal
resources devoted to research and development, they were not able to



Page 32                                                     GAO-12-112 ARPA-E
Appendix III: Description of ARPA-E Award
Winners with Prior Private Investment




internally fund the projects they proposed to ARPA-E. They told us there
were two reasons for this. First, the companies said that existing product
lines placed heavy demands on their internal research and development
budgets, and that there is continuous pressure from existing customers
and competitors to improve existing products; since ARPA-E projects
were still a number of years away from a return on investment, these
investments could not be justified. Second, these companies told us that
internal investments had to meet minimum investment return thresholds,
and that ARPA-E-type projects were not able to meet these thresholds.
Officials from one company told us that the rate of return on investment
required by its management was at least 20 percent per year.




Page 33                                                   GAO-12-112 ARPA-E
Appendix IV: Comments from ARPA-E
             Appendix IV: Comments from ARPA-E




             Page 34                             GAO-12-112 ARPA-E
Appendix IV: Comments from ARPA-E




Page 35                             GAO-12-112 ARPA-E
Appendix V: GAO Contact and Staff
                  Appendix V: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Frank Rusco, (202) 512-3841 or ruscof@gao.gov
GAO Contact
                  In addition to the individual named above, Tim Minelli (Assistant Director),
Staff             Paola Bobadilla, Cindy Gilbert, Robert Marek, Justin Mausel, Alison
Acknowledgments   O’Neill, Jeanette Soares, and Franklyn Yao made important contributions
                  to this report.




(361246)
                  Page 36                                                    GAO-12-112 ARPA-E
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov,
Congressional         (202) 512-4400
Relations             U.S. Government Accountability Office, 441 G Street NW, Room 7125
                      Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548



                        Please Print on Recycled Paper.