National Science Foundation • 4201 Wilson Boulevard • Arlington, Virginia 22230 Office of the Inspector General MEMORANDUM DATE: February 20, 2013 TO: Susan Singer Director, Division of Undergraduate Education FROM: Ken Chason /s/ Assistant IG for Legal, Legislative & External Affairs SUBJECT: Evaluation of Results for Division of Undergraduate Education Programs Background The initial objective of our evaluation was to assess whether final project reports pertaining to Division of Undergraduate Education-funded programs contained quantitative outputs/results. This Division administers programs to strengthen Science, Technology, Engineering, and Mathematics (STEM) education at two and four-year institutions through improved curricula and instruction and expanding student and faculty diversity, among other things. We chose to focus on DUE because its programs readily lend themselves to a lay-person review of project reports to assess whether they contained expected quantitative output data to demonstrate project accomplishments, the ease of finding this data, and our ability to match the goals of each selected grant with reported quantitative output results. 1 During our evaluation, we became aware of a new reporting requirement for Project Outcomes Reports (PORs) to be posted publicly on the Research.gov website. PORs, which are discussed more fully in the sections that follow, are intended to “provide the general public with a complete picture of the results of the funded research.” We revised our objective to focus more squarely on Project Outcomes Reports (PORs) instead of final project reports. We are providing the following observations to advance transparency in PORs so the public can understand the point of the financial investment and the extent to which the projects were successful. In tough 1 For this evaluation, the OIG examined grants awarded under three DUE programs: the Robert Nonce Teacher Scholarship Program, the Advanced Technological Education Program, and Transforming Undergraduate Education in Science Program. 1 economic times, federal programs must make every dollar count and the public should be able to see that funded programs are meeting their intended goals. Project Reporting Requirements Historically, all NSF grantees have been, and will continue to be, required to complete final project reports (in addition to annual progress reports) at the completion of their funding. These reports are submitted to the relevant NSF program official to document that the work on the project has been accomplished. They are not made available to the public. The America Competes Act of 2007 added a public dimension to project reporting by requiring that both research outcomes of NSF-funded research and citations of published documents resulting from that research for all awards receiving funding after January 4, 2010, be available to the public in electronic format. To implement this, NSF instituted the Project Outcomes Report. The POR is in addition to the final project report and is intended to “provide the general public with a complete picture of the results of the funded research.” According to NSF guidance, the report is expected to be a brief summary (200-800 words) “of the nature and outcomes of the project.” 2 Observations Because the new POR requirement has been in effect for such a short period of time, our limited review of select DUE programs included very few grants with PORs. 3 Nevertheless, given the public audience for these reports, DUE’s suitability towards output/quantitative results, and our lay-person perspective, we offer the following observations. As mentioned, we initially focused on final DUE project reports to see if they were adequately reporting outcomes. We found some significant shortcomings in the final reports we reviewed, which could spill-over into PORs and undermine their effectiveness as a means of transmitting clear outcomes to the public. Namely, some final project reports were very lengthy and it was difficult for a lay-person to understand what the project accomplished. Also, it was challenging to match program results to objectives in the abstracts. We recognize that the more technical information in final reports is appropriate for NSF program officials, but our concern is that absent clear instructions in program solicitations emphasizing that PORs are intended for the public, grantees may rely upon their final project reports for their PORs. 2 Frequently Asked Questions (FAQs) on Project Outcomes Report for the General Public (POR), November 30, 2010, http://www.nsf.gov/pubs/policydocs/porfaqs.jsp 3 Most of those grants included in our limited review that did have PORs were found to be more short-term funding for conferences and did not lend themselves to the type of outcomes that would be expected in a more typical DUE grant. However, we did find some traditional grants (all within the ATE program) that had PORs and these included useful quantitative data in their reports. 2 When the public accesses a POR for a specific grant within Research.gov, it will be found at the bottom of a page, after scrolling past the project “Abstract at Time of Award.” Project abstracts state what the project plans to achieve at the outset of the funding. The POR should provide a visible link between the project’s objectives and what was ultimately accomplished toward those objectives. A transparent connection between stated goals and eventual outcomes is important insofar as it shows how well the grantee institution has served as a steward of the taxpayer funds the project received. To illustrate the point about linkage, an abstract in one of the awards in our limited review states that the program “is increasing the number of science and engineering welding technicians to meet workforce demands. The Center furthers comprehensive reform in welding education by providing technologically current educational materials and professional development opportunities to two-year colleges and other educational institutions. The focus is on welding technician education at community colleges, but secondary and university education are being advanced.” One would naturally expect the Center’s POR to show: the extent to which the number of technicians increased; how many educational materials were provided; and how many professional developmental opportunities were provided. We were able to find some of this information within the POR. For instance, the Center identified a core curriculum and four training modules it produced. In addition, 217 instructors and faculty participated in and completed professional development courses during the course of the grant. However, there was no evidence identifying an increase in the number of technicians produced. To help ensure that project results are presented to the public in a clear and understandable manner, NSF solicitations should explain that to the extent that objectives are listed in the abstract, the POR should describe how the objectives were achieved. The three DUE programs within our limited review contained a standard reference to the POR requirement in each of the current program solicitations, which states: The project outcomes report must be prepared and submitted using Research.gov. This report serves as a brief summary, prepared specifically for the public, of the nature and outcomes of the project. This report will be posted on the NSF website exactly as it is submitted by the PI. There is no mention of the need for a demonstrable link between stated objectives and ultimate outcomes. As noted, we are concerned that without this kind of specific, up-front, guidance, grantees (perhaps for expedience) may be tempted to simply take portions of their final reports, 3 which they have prepared for the NSF program officials, and use them to populate the POR, which should be written for the public. As we found when we conducted our lay-person review of the final reports, these documents are not easy for lay people to decipher. We also found very little in the way of concrete quantitative data that supported the specific goals of these DUE programs, even though these educational programs are often centered around measurable goals, e.g., training teachers, educating students, developing curricula. We add that it is not inappropriate to tailor the solicitation to the needs of the program – in fact, immediately following the above-quoted statement is the following program-specific requirement: All projects will be required to participate in program monitoring and evaluation activities conducted by a third party as part of the Directorate for Education and Resources' program evaluation efforts that will require annual data collection. By defining POR expectations with greater precision in the program solicitation, DUE can alert its awardees to this important new reporting requirement and ensure that results are communicated in a meaningful way to the public and other stakeholders. Although we recognize that DUE programs lend themselves especially to measurable outcomes, our observations are nonetheless applicable to other programs NSF funds. For that reason, we are also copying NSF’s policy office on this memorandum. Thank you for the assistance provided during this review. If you have any questions, please do not hesitate to contact me. cc: Clifford J. Gabriel Jean Feldman 4
Evaluation of Results for Division of Undergraduate Education Programs
Published by the National Science Foundation, Office of Inspector General on 2013-02-20.
Below is a raw (and likely hideous) rendition of the original report. (PDF)