United States General Accounting Office GAO Testimony Before the Senate Committee on Commerce, Science, and Transportation U.S. Senate For Release on Delivery Expected at FEDERAL MANAGEMENT 9:30 a.m., EDT Thursday, July 24, 1997 Observations on the National Science Foundation Statement of Susan D. Kladiva, Acting Associate Director, Energy, Resources, and Science Issues, Resources, Community and Economic Development Division GAO/T-RCED/AIMD-97-168 Mr. Chairman and Members of the Committee: We are pleased to be here today to discuss a number of management and program issues at the National Science Foundation (NSF) as recently reported by the NSF Office of Inspector General (OIG) and GAO. Many of the management issues have been acknowledged by the agency’s senior management and are receiving varying levels of attention. We have not specifically reviewed NSF’s management and program areas, other than its implementation of the Government Performance and Results Act (Results Act). Therefore, as agreed with the Committee, my statement will focus on steps to address some of the systemic problems that are based on similar concerns across the federal government. Specifically, I will address (1) duplicate funding in the Small Business Innovation Research (SBIR) program at NSF and elsewhere; (2) challenges for NSF in implementing the Results Act; (3) the status of NSF’s efforts to prepare and have audited agencywide financial statements as required by the Chief Financial Officers (CFO) Act as amended; and (4) a framework for assessing NSF’s readiness to meet the computer challenges that will arise in the year 2000. In summary, Mr. Chairman: • Among the management issues identified by NSF’s OIG most in need of reform at NSF is the SBIR program. Regarding the SBIR Program, we have reported that duplicate funding of similar or even identical SBIR proposals submitted to more than one agency has occurred at NSF, NASA, and DOD.1 According to agency officials, a few companies received funding for the same proposals twice, three times, and even five times before agencies became aware of the duplication. Several factors contribute to this problem, including (1) the evasion of certification procedures, so that companies fail to identify similar proposals submitted to other agencies, (2) the lack of a consensus on what constitutes a duplicate proposal, and (3) the general lack of interagency access to and exchange of current information about recent awards by other agencies. NSF has revised its certification form to require applicants to certify, under criminal penalties for perjury, exactly what, if any, applications for similar research were pending in other agencies. In response to our recommendations, the Administrator of the Small Business Administration (SBA) has taken steps to address all three issues. 1 Federal Research: Interim Report on the Small Business Innovation Research Program (GAO/RCED-95-59, Mar. 8, 1995). Page 1 GAO/T-RCED/AIMD-97-168 • In implementing the Results Act, NSF, like most agencies, is still struggling to develop the mission based goals and the performance measurement requirements of the Act. We recently found that NSF’s draft strategic plan is incomplete and not specific enough to allow the Congress to evaluate whether the agency’s goals are achievable.2 This is to be expected during the initial efforts of such a challenging management reform effort. Measuring the performance of science-related projects can be extremely difficult because a wide range of factors determine if and how a particular research and development (R&D) project will result in a commercial application or have other benefits, and it can take years for a research project to realize a successful outcome. Moreover, we recently reported that there is no single indicator or evaluation method that adequately captures the results of R&D.3 As a result, determining the specific outcomes resulting from federal R&D is a challenge that will not easily be resolved. The Army Research Laboratory, which was designated as a pilot project for performance measurement under the Results Act, has developed a multifaceted approach using quantitative indicators, peer review, and customer feedback to evaluate the results of R&D. This response to the challenges in measuring the impacts of research shows that some progress is being made in response to the Results Act and may provide useful guidance to other science agencies. • As required by the CFO Act as amended by the Government Management Reform Act (GMRA), one of NSF’s primary financial management challenges is to prepare and audit consolidated agencywide financial statements. Financial statements are required to be prepared and audited to instill greater accountability; to provide reliable financial information for managing government and program operations and for making difficult policy decisions. NSF received its first-time audit of its agencywide financial statements for fiscal year (FY) 1996. NSF’s auditor concluded that the reported property, plant, and equipment account balance is unreliable. Because about 99 percent of NSF’s property is in the custody of R&D contractors and grantees, NSF is considering corrective actions that will include a change in accounting treatment as well as adequate inventory controls. Also, in response to its auditor’s concerns, NSF is in the process of developing a framework for identifying and developing performance measures in response to the CFO Act’s and GPRA’s requirements. 2 Results Act: Observations on the National Science Foundation’s Draft Strategic Plan (GAO/RCED-97-203R, July 11, 1997). 3 Measuring Performance: Strengths and Limitations of Research Indicators (GAO/RCED-97-91, Mar. 21, 1997). Page 2 GAO/T-RCED/AIMD-97-168 • The year 2000 problem is caused by systems that typically use two digits to represent years thereby making the year 2000 indistinguishable from 1900, 2001 indistinguishable from 1901, and so on. NSF like other federal agencies that have system or application programs that use dates to perform calculations, comparisons, or sorting may generate incorrect results when working with years after 1999. Converting systems to a four-digit year can be a massive undertaking for agencies such as NSF because it involves identifying computer systems, developing conversion strategies and plans, and dedicating sufficient resources to converting and adequately testing their computer systems and programs before January 1, 2000. GAO has developed an assessment guide that provides a structured approach for assessing the readiness of agency’s year 2000 programs.4 In recent years, the Congress has put in place a statutory framework for Background improving management in the federal government, and for helping the Congress and the executive branch make the difficult trade-offs that the current budget environment demands. This framework includes as its essential elements the Chief Financial Officers Act; information technology reform legislation; and the Government Performance and Results Act. We have begun a body of work on each of these management areas which provides some broad guidance of use to this Committee in its current review of NSF. Because of the mismanagement of funds, the OIG identified SBIR as one Duplicate Funding of program most in need of reform. Duplicate funding of similar or even SBIR Proposals identical proposals submitted to more than one agency has occurred in NSF, NASA, and DOD; however, steps have been taken to address the issue. Several factors have contributed to the problem of duplicate funding. First, the companies proposing projects have failed to identify identical proposals they have made to other agencies, thereby evading the certification procedure that requires them to provide such information. In response to this problem, NSF’s OIG officials were concerned about the need for more complete certification requirements and recommended that the NSF certification form be revised and strengthened. NSF has implemented this recommendation. Second, the lack of definitions and guidelines for the key terms, such as “similar” or “overlapping” research, has resulted in disagreement about what constitutes duplicate research. SBA’s prior policy directives and the 4 Year 2000 Computing Crisis: An Assessment Guide (GAO/AIMD-10.1.14, Feb. 1997). Page 3 GAO/T-RCED/AIMD-97-168 individual agencies’ solicitations did not define key terms and thus provided little or no guidance in avoiding the risk of duplicate funding. In general, the absence of substantive definitions for key terms placed the burden of judgment on the company. In some cases, as we stated in our March 1995 report, the appropriate certification may be difficult to determine and can lead to conflicts of opinion that may harm the SBIR program as well as the individual company. In response to this problem, officials in NSF’s OIG defined “overlapping” research and noted that the scope and funding of some SBIR awards in 1994 were reduced to eliminate overlapping work. Third, the agencies lacked interagency access to and exchange of current information about recent awards that might help to prevent or detect duplicate awards. At the time of our report, SBA maintained a database for the SBIR program that it used primarily to produce its annual report to the Congress regarding the program. The information had a “time lag” of about 9 months because it was first processed by each agency and then forwarded to SBA. Some SBIR officials, however, believed that the existing methods might not be adequate for detecting duplication when the agencies were receiving 20,000 proposals annually. In response to these three issues, SBA generally agreed with our concerns and modified its SBIR policy directive in July 1995 to address these problems. SBA refined the certification statement on possible duplication, clarified the definition of similar awards, and began work on developing a computer system that would enable all agencies to access timely information on awards being made by other agencies. The system is now operational, but, according to an SBIR program official at SBA, further work is under way to strengthen it. Under the Results Act, federal agencies are to set strategic and annual Implementation of the goals, measure performance, and report on the degree to which goals are Results Act in Science met. The Results Act requires that an agency’s strategic plan contain the Agencies following six critical elements: a mission statement; agencywide goals and objectives; strategies and resources needed to achieve the goals and objectives; the relationship between the long-term goals and objectives and the annual performance goals; key external factors that could affect the achievement of goals; and a description of how program evaluations were and will be used to establish or revise strategic goals. Page 4 GAO/T-RCED/AIMD-97-168 While NSF addressed five of the six required elements in its draft strategic plan, at least four of them need further development, and one element—key external factors—is not included in the current draft. NSF’s outcome goals set out the long-term programmatic, policy, and management goals to be accomplished through its program office investments; but many of these goals are not expressed in a measurable form. NSF’s plan provides some general dates for achieving its goals but does not provide the underlying assumptions, projections, or a schedule for initiating or completing significant actions. The strategies for achieving NSF’s goals lack precision, making it unclear whether the Foundation and the Congress will be able to assess whether the goals are achieved. Also unclear is the process for communicating goals and objectives throughout the agency and for assigning accountability to managers and staff for achieving the goals. Finally, NSF’s draft strategic plan does not discuss how the agency used specific program evaluations to develop its strategic goals or the other components of the plan. We recognize that NSF is currently revising its draft strategic and performance plans based on comments received by various parties. Nonetheless, as expected during the initial efforts of such a challenging management reform effort, most science agencies like NSF, are still struggling to develop mission based goals and performance measures. Determining the specific outcomes resulting from federal R&D is a challenge that will not easily be resolved. The experts in research measurement have tried for years to develop indicators that would provide a measure of the results of R&D. However, the very nature of the innovative process makes measuring the performance of science-related projects difficult. For example, a wide range of factors determines if and how a particular R&D project will result in a commercial application or have other benefits. It can also take many years for a research project to achieve results. Because of these difficulties, there is no single indicator or evaluation method that adequately captures the results of R&D. Decisionmakers have developed quantitative and qualitative indicators as proxies to assess the results of R&D activity. Our March 1997 report discusses the strengths and limitations in both types of indicators. The amount of money spent on R&D, the primary indicator of research investment, is useful as a measure of how much research is being performed. Having been refined over many years, these data are generally available for both the public and private sectors. However, the level of spending is not a reliable indicator of the level of research results. Page 5 GAO/T-RCED/AIMD-97-168 Quantitative output indicators focus mainly on return on investment, patenting rates, and bibliometrics—the study of publication-based data. While implying a degree of precision, these indicators were not originally intended to measure long-term R&D results. Qualitative assessment, such as peer review, provides detailed information, but depends on criteria that are inherently difficult to measure and on subjective judgment that is vulnerable to bias. Science agencies, like other agencies, must guard against the understandable tendency to overly rely on goals and measures that are easily quantifiable, such as the numbers of research grants provided and completed, at the expense of what is truly important but more difficult to measure, such as the difference the program makes in people’s lives. However, the legislative history that accompanied the Results Act states that agencies should not trivialize measurement by seeking to measure performance in a forced or artificial way simply to present quantifiable measures. The Army Research Laboratory, which was designated as a pilot project for performance measurement under the Act, developed a multifaceted approach using quantitative indicators, peer review, and customer feedback to evaluate the results of R&D. Although this is not the only approach that can be taken, this response to the challenges in measuring the impacts of research shows that some progress is being made in response to the Results Act and may provide useful guidance to other science agencies. One of NSF’s primary financial management challenges is to prepare and Implementation of the have its consolidated agencywide financial statements audited, as required CFO Act by the Chief Financial Officers Act of 1990, as amended by the Government Management Reform Act (GMRA) of 1994. Financial statements are required to be prepared and audited to instill greater accountability and provide reliable financial information for managing government and program operations and making difficult policy decisions. For FY 1991 through FY 1995, NSF had only prepared and had audited the financial statements for its Donations Account (Trust Fund). To meet the challenge of preparing and auditing consolidated agencywide financial statements, NSF’s CFO contracted with an independent public accounting firm to prepare proforma consolidated financial statements for FY 1994 and FY 1995. The CFO also contracted for assistance in preparing agencywide FY 1996 financial statements. Page 6 GAO/T-RCED/AIMD-97-168 In preparation for the FY 1996 agencywide financial statement audits, the OIG assessed NSF’s financial and administrative controls, including property controls, and its general ledger accounting system, the effectiveness of electronic data processing controls, and adherence to core financial system requirements. Because NSF’s OIG lacked adequate audit resources, the IG contracted with a separate independent public accounting firm for the audit of NSF’s FY 1996 Statement of Financial Position (balance sheet). The IG and NSF’s contract auditor issued a qualified opinion on NSF’s FY 1996 balance sheet due to inadequate supporting documentation and lack of a system to confirm account balances for approximately 99 percent of NSF’s property, plant and equipment (PP&E), which is held by NSF’s contractors and grantees. In its efforts to resolve these accounting problems, NSF management is considering the appropriate accounting treatment under the new federal accounting standards. NSF management believes that NSF has only a reversionary interest5 in PP&E held by its contractors and grantees. Under the new federal accounting standards, where this is the case, the assets are not to be reported as PP&E in federal agency accounts unless the assets actually revert to federal ownership. The new accounting standards provide for such assets to be accounted for and reported as R&D investment. Under the standard, NSF would account for the acquisition cost of PP&E for its contractors and grantees as an annual investment (expense). As a result, NSF would not be required to report related PP&E asset values in its financial statements. While PP&E held by contractors and grantees would still be subjected to inventory controls and audit, NSF’s OIG is concerned about the adequacy of its contractors and grantees auditors’ scope for federal financial statement audit purposes. NSF has requested formal approval of the accounting changes from OMB and FASAB. Upon approval by OMB and FASAB, NSF plans to implement the R&D investment standard for its FY 1997 financial statements, one year ahead of the implementation date in the standard. In addition, the IG and NSF’s contract auditor reported that NSF has not yet met the CFO Act requirement for systematic measurement of performance. NSF is in the process of developing a framework for identifying and developing performance measures in response to CFO Act and GPRA requirements. 5 A reversionary interest arises when NSF reserves the right to take back at the end of the grant or contract term, assets that it funded for use by a contractor or grantee. Page 7 GAO/T-RCED/AIMD-97-168 Further, to support GPRA implementation and to ensure that NSF is complying with federal cost accounting standards, the CFO is considering hiring an independent public accounting firm to evaluate its current cost accounting system capability against the requirements in the new federal cost accounting standards. NSF’s current financial system is capable of capturing direct costs by organization and program activities and allocating indirect costs. However, when NSF determines what performance measures will be used to capture program and financial outputs and outcomes, it will need to modify the system to relate costs to financial and program performance data. Along these lines, NSF plans to have an activity-based accounting system module operational by the end of FY 1998. Also, NSF plans to report performance measures related to R&D investment, including program outputs and outcomes. The year 2000 computing problem is rooted in the way dates are recorded Year 2000 Computer and computed in many computer systems. For the past several decades, Challenges systems have typically used two digits to represent the year such as “97” representing 1997, in order to conserve on electronic data storage and reduce operating costs. With this two-digit format, however, the year 2000 is indistinguishable from 1900, 2001 from 1901 and so on. As a result of this ambiguity, system or application programs that use dates to perform calculations, comparisons, or sorting may generate incorrect results when working with years after 1999. Because converting systems to a four-digit year can be a massive undertaking, agencies need to identify their inventories of mission-critical computer systems, develop conversion strategies and plans, and dedicate sufficient resources to converting and adequately testing their computer systems and programs before January 1, 2000. Although we have not evaluated NSF’s efforts, the agency reported in June that it had assessed which systems to change, replace or discard. To assist agencies in achieving year 2000 compliance, we developed an assessment guide which provides a framework and a checklist for assessing the readiness of federal agencies. The guide addresses issues that will be common to most year 2000 needs. It provides information on the scope of the challenge, and offers a structured approach for reviewing the adequacy and agency planning and management of the year 2000 program. This approach includes five phases addressing awareness, assessment, renovation, validation and implementation and are supported by program and project management activities. However, because each Page 8 GAO/T-RCED/AIMD-97-168 agency is different, there is no single, cookie cutter approach. Each agency must tailor its year 2000 program in response to its unique needs. In summary, a statutory framework exists for improving management in the federal government. Fully resolving the problems, however, will require sustained management attention. Congressional oversight, such as is being given in this instance by the Chairman and the Committee, is also key. This concludes my statement. I would be happy to respond to any questions you or the Members of the Committee may have. (141063) Page 9 GAO/T-RCED/AIMD-97-168 Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: info@www.gao.gov or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Federal Management: Observations on the National Science Foundation
Published by the Government Accountability Office on 1997-07-24.
Below is a raw (and likely hideous) rendition of the original report. (PDF)