oversight

    National Science Foundation      • 4201 Wilson Boulevard •             Arlington, Virginia 22230
                                    Office of the Inspector General

MEMORANDUM

DATE:           February 20, 2013

TO:             Susan Singer
                Director, Division of Undergraduate Education

FROM:           Ken Chason /s/
                Assistant IG for Legal, Legislative & External Affairs

SUBJECT:        Evaluation of Results for Division of Undergraduate Education Programs


Background

The initial objective of our evaluation was to assess whether final project reports pertaining to
Division of Undergraduate Education-funded programs contained quantitative outputs/results.
This Division administers programs to strengthen Science, Technology, Engineering, and
Mathematics (STEM) education at two and four-year institutions through improved curricula and
instruction and expanding student and faculty diversity, among other things. We chose to focus
on DUE because its programs readily lend themselves to a lay-person review of project reports to
assess whether they contained expected quantitative output data to demonstrate project
accomplishments, the ease of finding this data, and our ability to match the goals of each selected
grant with reported quantitative output results. 1

During our evaluation, we became aware of a new reporting requirement for Project Outcomes
Reports (PORs) to be posted publicly on the Research.gov website. PORs, which are discussed
more fully in the sections that follow, are intended to “provide the general public with a
complete picture of the results of the funded research.” We revised our objective to focus more
squarely on Project Outcomes Reports (PORs) instead of final project reports. We are providing
the following observations to advance transparency in PORs so the public can understand the
point of the financial investment and the extent to which the projects were successful. In tough


1
  For this evaluation, the OIG examined grants awarded under three DUE programs: the Robert Nonce Teacher
Scholarship Program, the Advanced Technological Education Program, and Transforming Undergraduate Education
in Science Program.

                                                    1
economic times, federal programs must make every dollar count and the public should be able to
see that funded programs are meeting their intended goals.

Project Reporting Requirements

Historically, all NSF grantees have been, and will continue to be, required to complete final
project reports (in addition to annual progress reports) at the completion of their funding. These
reports are submitted to the relevant NSF program official to document that the work on the
project has been accomplished. They are not made available to the public.

The America Competes Act of 2007 added a public dimension to project reporting by requiring
that both research outcomes of NSF-funded research and citations of published documents
resulting from that research for all awards receiving funding after January 4, 2010, be available
to the public in electronic format. To implement this, NSF instituted the Project Outcomes
Report. The POR is in addition to the final project report and is intended to “provide the general
public with a complete picture of the results of the funded research.” According to NSF
guidance, the report is expected to be a brief summary (200-800 words) “of the nature and
outcomes of the project.” 2

Observations

Because the new POR requirement has been in effect for such a short period of time, our limited
review of select DUE programs included very few grants with PORs. 3 Nevertheless, given the
public audience for these reports, DUE’s suitability towards output/quantitative results, and our
lay-person perspective, we offer the following observations.

As mentioned, we initially focused on final DUE project reports to see if they were adequately
reporting outcomes. We found some significant shortcomings in the final reports we reviewed,
which could spill-over into PORs and undermine their effectiveness as a means of transmitting
clear outcomes to the public. Namely, some final project reports were very lengthy and it was
difficult for a lay-person to understand what the project accomplished. Also, it was challenging
to match program results to objectives in the abstracts. We recognize that the more technical
information in final reports is appropriate for NSF program officials, but our concern is that
absent clear instructions in program solicitations emphasizing that PORs are intended for the
public, grantees may rely upon their final project reports for their PORs.

2
 Frequently Asked Questions (FAQs) on Project Outcomes Report for the General Public (POR), November 30,
2010, http://www.nsf.gov/pubs/policydocs/porfaqs.jsp
3
  Most of those grants included in our limited review that did have PORs were found to be more short-term funding
for conferences and did not lend themselves to the type of outcomes that would be expected in a more typical DUE
grant. However, we did find some traditional grants (all within the ATE program) that had PORs and these included
useful quantitative data in their reports.

                                                       2
When the public accesses a POR for a specific grant within Research.gov, it will be found at the
bottom of a page, after scrolling past the project “Abstract at Time of Award.” Project abstracts
state what the project plans to achieve at the outset of the funding. The POR should provide a
visible link between the project’s objectives and what was ultimately accomplished toward those
objectives. A transparent connection between stated goals and eventual outcomes is important
insofar as it shows how well the grantee institution has served as a steward of the taxpayer funds
the project received.

To illustrate the point about linkage, an abstract in one of the awards in our limited review states
that the program “is increasing the number of science and engineering welding technicians to
meet workforce demands. The Center furthers comprehensive reform in welding education by
providing technologically current educational materials and professional development
opportunities to two-year colleges and other educational institutions. The focus is on welding
technician education at community colleges, but secondary and university education are being
advanced.”

One would naturally expect the Center’s POR to show: the extent to which the number of
technicians increased; how many educational materials were provided; and how many
professional developmental opportunities were provided. We were able to find some of this
information within the POR. For instance, the Center identified a core curriculum and four
training modules it produced. In addition, 217 instructors and faculty participated in and
completed professional development courses during the course of the grant. However, there was
no evidence identifying an increase in the number of technicians produced.

To help ensure that project results are presented to the public in a clear and understandable
manner, NSF solicitations should explain that to the extent that objectives are listed in the
abstract, the POR should describe how the objectives were achieved.

The three DUE programs within our limited review contained a standard reference to the POR
requirement in each of the current program solicitations, which states:

       The project outcomes report must be prepared and submitted using Research.gov.
       This report serves as a brief summary, prepared specifically for the public, of the
       nature and outcomes of the project. This report will be posted on the NSF website
       exactly as it is submitted by the PI.

There is no mention of the need for a demonstrable link between stated objectives and ultimate
outcomes. As noted, we are concerned that without this kind of specific, up-front, guidance,
grantees (perhaps for expedience) may be tempted to simply take portions of their final reports,


                                                 3
which they have prepared for the NSF program officials, and use them to populate the POR,
which should be written for the public. As we found when we conducted our lay-person review
of the final reports, these documents are not easy for lay people to decipher. We also found very
little in the way of concrete quantitative data that supported the specific goals of these DUE
programs, even though these educational programs are often centered around measurable goals,
e.g., training teachers, educating students, developing curricula.

We add that it is not inappropriate to tailor the solicitation to the needs of the program – in fact,
immediately following the above-quoted statement is the following program-specific
requirement:

       All projects will be required to participate in program monitoring and evaluation
       activities conducted by a third party as part of the Directorate for Education and
       Resources' program evaluation efforts that will require annual data collection.

By defining POR expectations with greater precision in the program solicitation, DUE can alert
its awardees to this important new reporting requirement and ensure that results are
communicated in a meaningful way to the public and other stakeholders. Although we recognize
that DUE programs lend themselves especially to measurable outcomes, our observations are
nonetheless applicable to other programs NSF funds. For that reason, we are also copying NSF’s
policy office on this memorandum.

Thank you for the assistance provided during this review. If you have any questions, please do
not hesitate to contact me.

cc: Clifford J. Gabriel
    Jean Feldman




                                                 4