United States GAO General Accounting Office Washington, D.C. 20548 Resources, Community, and Economic Development Division B-277424 July 11, 1997 The Honorable Richard K. Armey Majority Leader House of Representatives The Honorable John Kasich Chairman, Committee on the Budget House of Representatives The Honorable Dan Burton Chairman, Committee on Government Reform and Oversight House of Representatives The Honorable Bob Livingston Chairman, Committee on Appropriations House of Representatives Subject: Results Act: Observations on the National Science Foundation’s Draft Strategic Plan On June 12, 1997, you asked us to review the draft strategic plans submitted by the Cabinet departments and selected major agencies for consultation with the Congress as required by the Government Performance and Results Act of 1993 (the Results Act). This report is our response to that request concerning the draft strategic plan for the National Science Foundation (NSF). We agreed to review NSF’s draft plan and assess (1) whether it fulfills the Objectives, Scope, requirements of the Results Act and to provide our views on its overall and Methodology quality; (2) whether it reflects NSF’s key statutory authorities; (3) whether it reflects interagency coordination for crosscutting programs, activities, or functions that are similar or complementary to other federal agencies’; and (4) whether NSF’s data and information systems are providing adequate information for measuring results. You also asked us to assess whether the draft plan addresses the management problems we have previously identified. Because we have not reported on NSF’s management problems in the past, this report will not address that issue. Page 1 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 We reviewed the most recent draft strategic plan—dated June 9, 1997—provided to congressional committees. Our overall assessment of NSF’s draft strategic plan was generally based on our knowledge of NSF’s programs and operations, our discussions with NSF’s Assistant to the Director for Science Policy and Planning and the Deputy Director, Office of Legislative and Public Affairs, and other existing information available at the time of our assessment. Specifically, the criteria we used to determine whether NSF’s draft strategic plan complies with the requirements of the Results Act were the Results Act itself and the Office of Management and Budget’s (OMB) guidance on developing the plans (Circular A-11, Part 2). To make judgments about the overall quality of the plan and its components, we used our May 1997 guidance for congressional review of the plans (GAO/GGD-10.1.16) as a tool. To determine whether the plan contains information on interagency coordination, we relied on our general knowledge of federal science agencies’ operations and programs and the results of our previous reports. In determining whether NSF’s draft strategic plan reflects its major statutory responsibilities, as you requested, we coordinated our review with the Congressional Research Service. To determine whether NSF has adequate systems in place to provide reliable information on performance, we relied on information provided by the Foundation’s Office of Inspector General (OIG). It is also important to recognize that NSF’s final plan is not due to the Congress and OMB until September 1997. Furthermore, the Results Act anticipated that it may take several planning cycles to perfect the process and that the final plan will continue to be refined as future planning cycles occur. Thus, our comments are a snapshot of the plan at this time. We recognize that developing a strategic plan is a dynamic process and that NSF is continuing to work to revise the draft with input from OMB, congressional staff, and other stakeholders. Our work was performed in June and July 1997. We obtained comments on a draft of this report from NSF. Its comments are in enclosure I. NSF is an independent federal agency created by the National Science Background Foundation Act of 1950 (P.L. 81-507). Its aim is to promote and advance scientific and engineering progress in the United States. The idea of such a foundation was an outgrowth of the important contributions made by science and technology during World War II. Since that time, NSF has been Page 2 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 responsible for the overall well-being of science and engineering across all disciplines. In contrast, other federal agencies support research focused on specific missions, such as health or defense. NSF is also committed to help ensure the nation’s supply of scientists, engineers, and science educators through the financial support of education and research. NSF funds research and education in the areas of science and engineering through grants, contracts, and cooperative agreements to more than 2,000 colleges, universities, and other research institutions. NSF receives about 53,000 requests for funding (both new and renewal projects) each year and makes about 20,000 awards. The agency operates no laboratories itself but does support National Research Centers, certain oceanographic vessels, and Antarctic research stations. It also supports cooperative research between universities and industry and U.S. participation in international scientific efforts. NSF has been involved in strategic planning efforts since 1992, when the National Science Board Commission on the Future of the NSF was established. The board established national science and technology goals that later became the basis for NSF’s 1994 strategic plan. The plan was developed with input and support from in-house staff and advisory bodies. NSF views the Results Act plan as its implementation strategy for the goals set forth in its 1995 strategic plan. The Results Act requires that an agency’s strategic plan contain the following six critical elements: (1) a comprehensive mission statement; (2) agencywide long-term goals and objectives for all major functions and operations; (3) approaches (or strategies) and the various resources needed to achieve the goals and objectives; (4) the relationship between the long-term goals and objectives and the annual performance goals; (5) an identification of key factors, external to the agency and beyond its control, that could significantly affect the achievement of the strategic goals; and (6) a description of how program evaluations were used to establish or revise strategic goals and a schedule for future program evaluations. NSF’s draft strategic plan is incomplete and not specific enough to allow Results in Brief the Congress to evaluate whether the agency’s goals are achievable. The draft strategic plan addresses aspects of five of six required elements. However, three of the five elements are not yet complete—goals and objectives, strategies for achieving goals, and how program evaluation was Page 3 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 used. Furthermore, NSF did not include one key element—external factors that could affect the achievement of the plan’s goals. Because the plan is incomplete, the Congress is missing critical information for its consultations with NSF. NSF’s draft strategic plan appears to reflect the consideration of its key statutory authority, the National Science Foundation Act of 1950, as amended. We note, however, that NSF is subject to other statutes related to its core functions; these other statutes broaden the scope of its responsibilities. NSF could provide useful information by describing its responsibilities under these other statutes when its plan includes goals and objectives based on them. NSF’s draft strategic plan acknowledges the crosscutting nature of its work. However, the draft plan does not show evidence of interagency coordination. While the strategic plan emphasizes the importance of NSF’s many partners in the research and education enterprise, it does not identify who these partners are or provide sufficient information to determine the extent to which NSF’s and its partners’ functions are duplicative or overlapping. While we have not analyzed NSF’s data and information systems, inadequacies in both financial information and information technology at NSF have been identified by NSF’s Office of Inspector General. NSF’s OIG and an independent public accounting firm completed the first audit of NSF’s consolidated, agencywide fiscal year 1996 Statement of Financial Position (balance sheet). Except for inadequate documentation to support the reported amounts for property, plant, and equipment, the auditors concluded that NSF’s assets, liabilities, and net position are reliable. (No audit was performed on the 1996 Statement of Operations and Changes in Net Position. Therefore, the reliability of NSF’s revenue and expense information is uncertain. It is our understanding that this information will be audited for fiscal year 1997.) In addition, the auditors found that NSF had not yet met the requirements in the Chief Financial Officers (CFO) Act to develop an integrated agency accounting and financial management system that provides for reporting cost information and the systematic measurement of performance. Furthermore, linkages between NSF’s technology and its programs’ missions and goals were not included in its strategic plan. Page 4 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 While five of the six required elements are addressed, at least four of them NSF’s Strategic Plan need further development, and the sixth element—key external factors—is Partly Fulfills the not included in the current draft. NSF’s draft plan is not specific enough to Requirements of the allow the Congress to evaluate whether NSF’s goals and objectives are achievable. In its plan, the Foundation describes itself as an investment Results Act agent, setting its mission, goals, and supporting strategies toward the performance of its investment portfolios through grants, contracts, and cooperative agreements. The draft plan indicates that NSF’s annual performance goals for results will appear as descriptive standards, as allowed by the Results Act’s option to set performance goals in an alternative format. The draft plan also indicates that the performance goals for process will be largely quantitative. A brief section on the use of program evaluation in establishing strategic goals is included. NSF also included a section in its plan that discusses the processes and essential coordination functions managed by NSF staff that must operate effectively if NSF’s outcome goals are to be met. In this section, the Foundation addresses other factors critical to the successful management of the agency. NSF Addresses NSF’s mission statement focuses on strengthening the nation’s potential for Requirements for Its research and education in science and engineering. Included with its Mission Statement mission statement are those areas initiated and supported by NSF pursuant to its core statutory responsibilities. These are: basic scientific and engineering research; programs strengthening scientific and engineering research potential; science and engineering education; and an information and policy base for science and engineering. The section on NSF’s mission statement and the supporting text is comprehensive and results-oriented and fulfills public needs and statutory responsibilities. Although brief, it defines the basic purpose of the agency, focusing particularly on its core programs and activities. Goals and Objectives Are The draft plan sets out three overarching goals intended to guide NSF’s Defined in the Draft Plan strategic direction, as well as four broadly worded outcome goals. The overarching goals do not represent the unique functions and operations of NSF. The draft plan states that these goals cannot be achieved unilaterally. According to NSF’s Deputy Director, Office of Legislative and Public Affairs, the overarching goals were developed by the National Science Board 3 years ago in the context of a broader effort to establish goals for all science agencies. Page 5 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 The outcome goals set out the long-term programmatic, policy, and management goals to be accomplished through NSF’s program office investments. Because many of these goals are not expressed in a measurable form, it is unclear whether the Foundation and the Congress will be able to assess whether the goals are achieved. While in some cases these goals provide the immediate context for NSF’s investment decisions, in other cases they do not. For example, one goal is to encourage “improved achievement in the essential mathematics and science skills needed by all Americans.” The achievement of this goal is targeted at results over which NSF has a reasonable degree of influence—instructional materials developed through NSF awards, for example. However, another stated goal seeks “discoveries at and across the frontier of science and engineering.” It is unclear how NSF will be able to assess whether this goal is achieved. When goals are defined in a way that precludes a direct, future determination of achievement, the performance goals and indicators in the annual performance plan should be used to provide the basis for the assessment, according to OMB Circular A-11. Therefore, NSF may want to elaborate on how it is achieving certain goals in its performance plan. Strategies to Achieve NSF’s NSF’s plan provides some general dates for achieving its goals but does not Goals Lack Precision provide the underlying assumptions, projections, or a schedule for initiating or completing significant actions. Also lacking is the process for communicating goals and objectives throughout the agency and for assigning accountability to managers and staff for the achievement of goals. Each of NSF’s four outcome goals is supported by three to four investment strategies. For example, the first goal is to encourage “discoveries at and across the frontier of science and engineering.” The draft plan identifies four key investment strategies to meet this goal. The first strategy is to “seek out the most innovative ideas, actively shaping the portfolio in ways that influence capabilities for the future.” NSF officials agree that it would be helpful if the language were more specific. One way to achieve specificity may be to link the stated outcome goals to the relevant statutory objective. Although the Results Act does not require such a linkage, including such information may help NSF management to better formulate its goals. It could also facilitate constructive consultation between the agency and the congressional oversight committees about the agency’s goals and its priorities for achieving them. Page 6 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 Although not discussed in detail in NSF’s plan, a single investment can work toward many of the outcome goals articulated. For example, goal three is to achieve “a diverse, productive globally-oriented workforce of scientists and engineers.” NSF’s Faculty Early Career Development (Career) program for junior-level faculty members at colleges and universities encourages them to contribute to research and education early in their careers. The Career program supports research that leads to linking discovery and learning, one of the supporting strategies for NSF’s first goal. Moreover, according to NSF, frequently the most original and innovative ideas come from young scientists whose ideas are often of interest to industry and to other agencies. Thus, investments made through this program may also support the outcome goals for discoveries and connections. The Plan Addresses the As required by OMB’s guidance, NSF’s draft plan outlines the type of Relationship Between performance goals to be included in its performance plan. Performance NSF’s Long-Term Goals goals for NSF will include (1) quantitative goals measuring the process of investing in or facilitating research and education projects, as well as and Annual Performance performance goals for facilities operations, and (2) descriptive standards Goals for assessing the results of NSF’s investments at an aggregate level. The relationship between the performance goals and NSF’s outcome goals is described briefly as is the relevance and use of performance goals in helping to determine the achievement of general goals and objectives. In addition, the plan addresses the linkage between NSF’s budget and its annual performance plan. Performance plans for an upcoming year are to be developed in the light of the analysis of past performance, an assessment of how recent or projected changes in the investment portfolio will influence future performance, and how the portfolio fits with the outcome goals identified in the strategic plan. As we have reported in the past, the very nature of the innovative process makes measuring the performance of science-related activities difficult, since outcomes may not be seen for many years.1 An NSF official told us that the alternative form of performance assessment for the agency’s annual performance plan should provide some basis for assessing whether NSF’s goals have been met. According to information on a 1995 NSF discussion paper,2 the ultimate outcomes of NSF’s programs, such as new 1 Managing for Results: Key Steps and Challenges in Implementing GPRA in Science Agencies (GAO/T-GGD/RCED-96-214, July 10, 1996). 2 National Science Foundation Case Study: Development and Use of Outcome Information by the National Science Foundation (Oct. 25, 1996). Page 7 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 technologies or improved quality of life, are too far from NSF’s sphere of influence to be used for setting performance goals. As a result, the discussion paper argues that NSF would need to manage toward intermediate outcomes, such as major new conceptual frameworks, enduring partnerships, and cadres of trained technical talent, all of which involve qualitative elements not lending themselves to quantitative indicators. NSF has focused its efforts on developing outcome indicators, as encouraged by the Results Act. NSF’s Plan Does Not NSF’s draft plan neither addresses key external factors nor describes how Discuss External Factors achieving particular goals could be affected by external factors. Such That Could Affect Goal factors as the extent to which schools and universities emphasize mathematics and science or subsidize faculty research are influences Achievement outside of NSF’s control. These influences could affect NSF’s realization of its goal aimed at improving the achievement of mathematics and science skills needed by all Americans, or its goal of making discoveries at and across the frontier of science and engineering. Therefore, the identification of these factors and related actions that could reduce or ameliorate their potential impact could be useful in the Congress’s review of and NSF’s implementation of the strategic plan. NSF’s Plan Lacks a NSF’s draft plan does not discuss how the agency used specific program Complete Description of evaluations to develop its strategic goals or the other components of the How Program Evaluations plan. However, beginning in fiscal year 1998, the plan indicates that the Foundation will structure its internal and external assessment processes Will Be Used and a using this strategic plan and the performance plan and that performance Schedule for Future reports will be requested annually from all NSF units. Further details are Program Evaluations needed on a schedule for future evaluations, the scope of and methodology for future evaluations, and how the findings could be useful in assessing NSF’s goals and performance plans. NSF’sdraft plan highlights the agency’s core responsibilities under the Statutory National Science Foundation Act of 1950 (Public Law 81-507), as amended. Responsibilities Are However, the draft plan does not mention additional authorities given to Generally Reflected in NSF under the Science and Engineering Equal Opportunities Act, Title I of the Education for Economic Security Act, or other related legislation, such NSF’s Strategic Plan as that concerning polar research and conservation. Although these mandates are not specifically mentioned, the core responsibilities appear general enough to encompass the additional responsibilities. Nonetheless, Page 8 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 it might be helpful if NSF provided a list, as a supplement to its plan, of its statutory authorities and the major responsibilities that flow from such legislation. NSF’s draft plan does not identify specific programs and activities that are Crosscutting crosscutting or similar to those of other federal agencies. The draft plan Functions Are Not does briefly describe the role NSF plays in the federal science and Adequately engineering enterprise. Although NSF captures investment opportunities across the spectrum of science, mathematics, and engineering to influence Recognized in the the nation’s capabilities in all aspects of these endeavors, the plan Strategic Plan provides little evidence to suggest that interagency coordination occurred to address the potential issues of duplication and overlap. Because overlapping and fragmented programs can waste scarce funds, confuse and frustrate program customers, and limit the overall effectiveness of the federal effort, it is important for NSF to address crosscutting programs in its plan. In the science and technology area, where the federal government spent $60 billion in fiscal year 1996 and the potential for unnecessary overlap is particularly high, close coordination is essential. The Foundation’s mission includes promoting the progress of science, and one of its overarching goals is to enable the United States to uphold a position of world leadership in all aspects of science, mathematics, and engineering. However, our review of other agencies’ draft strategic plans identified the following examples of agencies with missions that could potentially overlap NSF’s mission: • The Department of Energy’s (DOE) science mission is to maintain leadership in basic research and to advance scientific knowledge. • The Department of Commerce’s mission includes keeping America competitive with cutting-edge science and technology. There are additional examples of potential overlap among federal agencies. NSF’s authorizing legislation directs it to initiate and support science and engineering education programs at all levels and in all fields of science and engineering. Similarly, DOE’s draft strategic plan states that it will use its laboratories and the nation’s universities to contribute to the nation’s science and mathematics education. According to NSF’s Assistant to the Director for Science Policy and Planning, the Foundation participates in a number of groups such as the Page 9 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 National Science and Technology Council, the Committee on Fundamental Science, and the Research Roundtable. While one purpose of the groups is to coordinate, NSF’s draft plan neither mentions these groups nor the frequency of their discussions. However, relating NSF’s strategic plan to other agencies’ crosscutting programs would enhance the Congress’s ability to assess any concerns about potential overlap and duplication. While NSF’s draft plan recognizes the importance of information NSF’s Plan DOEs Not technology, the draft plan could benefit from a clearer and more detailed Fully Address Its discussion of how it specifically intends to use information technology to Capacity to Provide improve performance, to reduce costs, and ultimately to achieve its missions, goals, and objectives. Furthermore, NSF’s draft plan does not Reliable Information address the “year 2000 problem” (which requires that computer systems be on the Achievement of changed to accommodate dates beyond the year 1999) or significant weaknesses in information security—two issues that we have identified as Strategic Goals high risk across government. Recent information technology reform legislation, including the Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996, set forth requirements that promote more efficient and effective use of information technology in support of agencies’ missions and of improved program performance. Under these acts, agencies are to better link their technology plans and information technology use to their programs’ missions and goals. According to officials in NSF’s OIG, one of NSF’s primary financial management challenges is to prepare and audit consolidated, agencywide financial statements, as required by the Chief Financial Officers (CFO) Act and the Government Management Reform Act of 1994. Financial statements are required to be prepared and audited to instill greater accountability and to provide reliable financial information for formulating budgets, managing government and program operations, and making difficult policy decisions. NSF received a qualified opinion on its fiscal year 1996 statement of financial position because of inadequate documentation for property, plant, and equipment in the possession of its contractors and grantees. (No audit was performed on the 1996 Statement of Operations and Changes in Net Position. Therefore, the reliability of NSF’s revenue and expense information is uncertain. It is our understanding that this information will be audited for fiscal year 1997.) Reliable information on investments in the property assets used to carry out agency’s mission is an essential part of performance measurement. Page 10 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 In addition, to support the Results Act’s implementation and to ensure that NSF is complying with federal cost-accounting system standards, the Foundation will need to relate costs to financial and program performance data. Key requirements of the CFO Act are the development of cost information to enable the systematic measurement of performance and the integrations of systems (i.e., program, accounting, and budget systems). NSF plans to modify its cost-accounting system to support reporting on the Results Act once its strategic plan is approved. We provided a draft of this report to the National Science Foundation for Agency Comments its review and comment. (NSF’s comments appear in enc. I.) In general, NSF and Our Evaluation agreed that additional information on two of the required elements would be useful and provided additional context to explain NSF’s approach to meeting the remaining requirements of the Results Act. NSF stated its intention to update its draft plan by providing additional detail on the external factors that could influence its ability to meet its goals; current and future program evaluation efforts and their contributions to the formulation of investment strategies; and how NSF addresses issues of potential duplication and overlap and its appropriate role in interagency activities. In regard to information technology, we believe that the implementation of the tenets of the Clinger-Cohen and Paperwork Reduction Acts is essential to effectively using information technology to improve performance and carry out an agency’s mission, goals, and objectives. Successfully addressing “year 2000” problems is also critical. It is therefore important for NSF, in its strategic plan, to articulate how it plans to use information technology in meeting these key challenges. While we questioned the measurability of NSF’s outcome goals, NSF believes that by using expert judgment (peer review) to assess a range of factors about a project’s results, it can appraise whether the observed outcomes meet the stated goals. We agree that quantitative and qualitative indicators are widely used as proxies to assess research and development (R&D) results because of the difficulties in identifying the impacts of research. Yet, while implying a degree of precision, these indicators were not originally intended to measure long-term R&D results. Furthermore, while peer review provides detailed information, it relies on the subjective decisions of individuals and can be expensive.3 3 Measuring Performance: Challenges in Evaluating Research and Development (GAO/T-RCED-97-130, Apr. 10, 1997). Page 11 GAO/RCED-97-203R NSF’s Draft Strategic Plan B-277424 As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will send copies of this report to the Minority Leader of the House of Representatives; the Ranking Minority Members of your Committees and the Chairmen and Ranking Minority Members of the other Committees that have jurisdiction over NSF; the Director of NSF; and the Director, Office of Management and Budget. We will send copies to others on request. Please call me at (202) 512-3841 if you or your staff have any questions concerning this letter. Victor S. Rezendes Director, Energy, Resources, and Science Issues Enclosure Page 12 GAO/RCED-97-203R NSF’s Draft Strategic Plan Page 13 GAO/RCED-97-203R NSF’s Draft Strategic Plan Enclosure I Comments From the National Science Foundation Page 14 GAO/RCED-97-203R NSF’s Draft Strategic Plan Enclosure I Comments From the National Science Foundation Now on pp. 5 & 6. See p. 8. See p. 8. See p. 8. Page 15 GAO/RCED-97-203R NSF’s Draft Strategic Plan Enclosure I Comments From the National Science Foundation See pp. 9 & 10. See pp. 10 & 11. (141077) Page 16 GAO/RCED-97-203R NSF’s Draft Strategic Plan Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Results Act: Observations on the National Science Foundation's Draft Strategic Plan
Published by the Government Accountability Office on 1997-07-11.
Below is a raw (and likely hideous) rendition of the original report. (PDF)