United States General Accounting Office
GAO Testimony
Before the Senate Committee on Commerce, Science,
and Transportation
U.S. Senate
For Release
on Delivery
Expected at
FEDERAL MANAGEMENT
9:30 a.m., EDT
Thursday,
July 24, 1997
Observations on the
National Science
Foundation
Statement of Susan D. Kladiva,
Acting Associate Director,
Energy, Resources, and Science Issues,
Resources, Community and Economic
Development Division
GAO/T-RCED/AIMD-97-168
Mr. Chairman and Members of the Committee:
We are pleased to be here today to discuss a number of management and
program issues at the National Science Foundation (NSF) as recently
reported by the NSF Office of Inspector General (OIG) and GAO. Many of the
management issues have been acknowledged by the agency’s senior
management and are receiving varying levels of attention. We have not
specifically reviewed NSF’s management and program areas, other than its
implementation of the Government Performance and Results Act (Results
Act). Therefore, as agreed with the Committee, my statement will focus on
steps to address some of the systemic problems that are based on similar
concerns across the federal government. Specifically, I will address
(1) duplicate funding in the Small Business Innovation Research (SBIR)
program at NSF and elsewhere; (2) challenges for NSF in implementing the
Results Act; (3) the status of NSF’s efforts to prepare and have audited
agencywide financial statements as required by the Chief Financial
Officers (CFO) Act as amended; and (4) a framework for assessing NSF’s
readiness to meet the computer challenges that will arise in the year 2000.
In summary, Mr. Chairman:
• Among the management issues identified by NSF’s OIG most in need of
reform at NSF is the SBIR program. Regarding the SBIR Program, we have
reported that duplicate funding of similar or even identical SBIR proposals
submitted to more than one agency has occurred at NSF, NASA, and DOD.1
According to agency officials, a few companies received funding for the
same proposals twice, three times, and even five times before agencies
became aware of the duplication. Several factors contribute to this
problem, including (1) the evasion of certification procedures, so that
companies fail to identify similar proposals submitted to other agencies,
(2) the lack of a consensus on what constitutes a duplicate proposal, and
(3) the general lack of interagency access to and exchange of current
information about recent awards by other agencies. NSF has revised its
certification form to require applicants to certify, under criminal penalties
for perjury, exactly what, if any, applications for similar research were
pending in other agencies. In response to our recommendations, the
Administrator of the Small Business Administration (SBA) has taken steps
to address all three issues.
1
Federal Research: Interim Report on the Small Business Innovation Research Program
(GAO/RCED-95-59, Mar. 8, 1995).
Page 1 GAO/T-RCED/AIMD-97-168
• In implementing the Results Act, NSF, like most agencies, is still struggling
to develop the mission based goals and the performance measurement
requirements of the Act. We recently found that NSF’s draft strategic plan is
incomplete and not specific enough to allow the Congress to evaluate
whether the agency’s goals are achievable.2 This is to be expected during
the initial efforts of such a challenging management reform effort.
Measuring the performance of science-related projects can be extremely
difficult because a wide range of factors determine if and how a particular
research and development (R&D) project will result in a commercial
application or have other benefits, and it can take years for a research
project to realize a successful outcome. Moreover, we recently reported
that there is no single indicator or evaluation method that adequately
captures the results of R&D.3 As a result, determining the specific outcomes
resulting from federal R&D is a challenge that will not easily be resolved.
The Army Research Laboratory, which was designated as a pilot project
for performance measurement under the Results Act, has developed a
multifaceted approach using quantitative indicators, peer review, and
customer feedback to evaluate the results of R&D. This response to the
challenges in measuring the impacts of research shows that some progress
is being made in response to the Results Act and may provide useful
guidance to other science agencies.
• As required by the CFO Act as amended by the Government Management
Reform Act (GMRA), one of NSF’s primary financial management challenges
is to prepare and audit consolidated agencywide financial statements.
Financial statements are required to be prepared and audited to instill
greater accountability; to provide reliable financial information for
managing government and program operations and for making difficult
policy decisions. NSF received its first-time audit of its agencywide
financial statements for fiscal year (FY) 1996. NSF’s auditor concluded that
the reported property, plant, and equipment account balance is unreliable.
Because about 99 percent of NSF’s property is in the custody of R&D
contractors and grantees, NSF is considering corrective actions that will
include a change in accounting treatment as well as adequate inventory
controls. Also, in response to its auditor’s concerns, NSF is in the process
of developing a framework for identifying and developing performance
measures in response to the CFO Act’s and GPRA’s requirements.
2
Results Act: Observations on the National Science Foundation’s Draft Strategic Plan
(GAO/RCED-97-203R, July 11, 1997).
3
Measuring Performance: Strengths and Limitations of Research Indicators (GAO/RCED-97-91, Mar. 21,
1997).
Page 2 GAO/T-RCED/AIMD-97-168
• The year 2000 problem is caused by systems that typically use two digits to
represent years thereby making the year 2000 indistinguishable from 1900,
2001 indistinguishable from 1901, and so on. NSF like other federal
agencies that have system or application programs that use dates to
perform calculations, comparisons, or sorting may generate incorrect
results when working with years after 1999. Converting systems to a
four-digit year can be a massive undertaking for agencies such as NSF
because it involves identifying computer systems, developing conversion
strategies and plans, and dedicating sufficient resources to converting and
adequately testing their computer systems and programs before January 1,
2000. GAO has developed an assessment guide that provides a structured
approach for assessing the readiness of agency’s year 2000 programs.4
In recent years, the Congress has put in place a statutory framework for
Background improving management in the federal government, and for helping the
Congress and the executive branch make the difficult trade-offs that the
current budget environment demands. This framework includes as its
essential elements the Chief Financial Officers Act; information
technology reform legislation; and the Government Performance and
Results Act. We have begun a body of work on each of these management
areas which provides some broad guidance of use to this Committee in its
current review of NSF.
Because of the mismanagement of funds, the OIG identified SBIR as one
Duplicate Funding of program most in need of reform. Duplicate funding of similar or even
SBIR Proposals identical proposals submitted to more than one agency has occurred in
NSF, NASA, and DOD; however, steps have been taken to address the issue.
Several factors have contributed to the problem of duplicate funding.
First, the companies proposing projects have failed to identify identical
proposals they have made to other agencies, thereby evading the
certification procedure that requires them to provide such information. In
response to this problem, NSF’s OIG officials were concerned about the
need for more complete certification requirements and recommended that
the NSF certification form be revised and strengthened. NSF has
implemented this recommendation.
Second, the lack of definitions and guidelines for the key terms, such as
“similar” or “overlapping” research, has resulted in disagreement about
what constitutes duplicate research. SBA’s prior policy directives and the
4
Year 2000 Computing Crisis: An Assessment Guide (GAO/AIMD-10.1.14, Feb. 1997).
Page 3 GAO/T-RCED/AIMD-97-168
individual agencies’ solicitations did not define key terms and thus
provided little or no guidance in avoiding the risk of duplicate funding. In
general, the absence of substantive definitions for key terms placed the
burden of judgment on the company. In some cases, as we stated in our
March 1995 report, the appropriate certification may be difficult to
determine and can lead to conflicts of opinion that may harm the SBIR
program as well as the individual company. In response to this problem,
officials in NSF’s OIG defined “overlapping” research and noted that the
scope and funding of some SBIR awards in 1994 were reduced to eliminate
overlapping work.
Third, the agencies lacked interagency access to and exchange of current
information about recent awards that might help to prevent or detect
duplicate awards. At the time of our report, SBA maintained a database for
the SBIR program that it used primarily to produce its annual report to the
Congress regarding the program. The information had a “time lag” of about
9 months because it was first processed by each agency and then
forwarded to SBA. Some SBIR officials, however, believed that the existing
methods might not be adequate for detecting duplication when the
agencies were receiving 20,000 proposals annually.
In response to these three issues, SBA generally agreed with our concerns
and modified its SBIR policy directive in July 1995 to address these
problems. SBA refined the certification statement on possible duplication,
clarified the definition of similar awards, and began work on developing a
computer system that would enable all agencies to access timely
information on awards being made by other agencies. The system is now
operational, but, according to an SBIR program official at SBA, further work
is under way to strengthen it.
Under the Results Act, federal agencies are to set strategic and annual
Implementation of the goals, measure performance, and report on the degree to which goals are
Results Act in Science met. The Results Act requires that an agency’s strategic plan contain the
Agencies following six critical elements: a mission statement; agencywide goals and
objectives; strategies and resources needed to achieve the goals and
objectives; the relationship between the long-term goals and objectives
and the annual performance goals; key external factors that could affect
the achievement of goals; and a description of how program evaluations
were and will be used to establish or revise strategic goals.
Page 4 GAO/T-RCED/AIMD-97-168
While NSF addressed five of the six required elements in its draft strategic
plan, at least four of them need further development, and one
element—key external factors—is not included in the current draft. NSF’s
outcome goals set out the long-term programmatic, policy, and
management goals to be accomplished through its program office
investments; but many of these goals are not expressed in a measurable
form. NSF’s plan provides some general dates for achieving its goals but
does not provide the underlying assumptions, projections, or a schedule
for initiating or completing significant actions. The strategies for achieving
NSF’s goals lack precision, making it unclear whether the Foundation and
the Congress will be able to assess whether the goals are achieved. Also
unclear is the process for communicating goals and objectives throughout
the agency and for assigning accountability to managers and staff for
achieving the goals. Finally, NSF’s draft strategic plan does not discuss how
the agency used specific program evaluations to develop its strategic goals
or the other components of the plan.
We recognize that NSF is currently revising its draft strategic and
performance plans based on comments received by various parties.
Nonetheless, as expected during the initial efforts of such a challenging
management reform effort, most science agencies like NSF, are still
struggling to develop mission based goals and performance measures.
Determining the specific outcomes resulting from federal R&D is a
challenge that will not easily be resolved. The experts in research
measurement have tried for years to develop indicators that would provide
a measure of the results of R&D. However, the very nature of the innovative
process makes measuring the performance of science-related projects
difficult. For example, a wide range of factors determines if and how a
particular R&D project will result in a commercial application or have other
benefits. It can also take many years for a research project to achieve
results.
Because of these difficulties, there is no single indicator or evaluation
method that adequately captures the results of R&D. Decisionmakers have
developed quantitative and qualitative indicators as proxies to assess the
results of R&D activity. Our March 1997 report discusses the strengths and
limitations in both types of indicators. The amount of money spent on R&D,
the primary indicator of research investment, is useful as a measure of
how much research is being performed. Having been refined over many
years, these data are generally available for both the public and private
sectors. However, the level of spending is not a reliable indicator of the
level of research results.
Page 5 GAO/T-RCED/AIMD-97-168
Quantitative output indicators focus mainly on return on investment,
patenting rates, and bibliometrics—the study of publication-based data.
While implying a degree of precision, these indicators were not originally
intended to measure long-term R&D results. Qualitative assessment, such as
peer review, provides detailed information, but depends on criteria that
are inherently difficult to measure and on subjective judgment that is
vulnerable to bias.
Science agencies, like other agencies, must guard against the
understandable tendency to overly rely on goals and measures that are
easily quantifiable, such as the numbers of research grants provided and
completed, at the expense of what is truly important but more difficult to
measure, such as the difference the program makes in people’s lives.
However, the legislative history that accompanied the Results Act states
that agencies should not trivialize measurement by seeking to measure
performance in a forced or artificial way simply to present quantifiable
measures.
The Army Research Laboratory, which was designated as a pilot project
for performance measurement under the Act, developed a multifaceted
approach using quantitative indicators, peer review, and customer
feedback to evaluate the results of R&D. Although this is not the only
approach that can be taken, this response to the challenges in measuring
the impacts of research shows that some progress is being made in
response to the Results Act and may provide useful guidance to other
science agencies.
One of NSF’s primary financial management challenges is to prepare and
Implementation of the have its consolidated agencywide financial statements audited, as required
CFO Act by the Chief Financial Officers Act of 1990, as amended by the
Government Management Reform Act (GMRA) of 1994. Financial
statements are required to be prepared and audited to instill greater
accountability and provide reliable financial information for managing
government and program operations and making difficult policy decisions.
For FY 1991 through FY 1995, NSF had only prepared and had audited the
financial statements for its Donations Account (Trust Fund). To meet the
challenge of preparing and auditing consolidated agencywide financial
statements, NSF’s CFO contracted with an independent public accounting
firm to prepare proforma consolidated financial statements for FY 1994
and FY 1995. The CFO also contracted for assistance in preparing
agencywide FY 1996 financial statements.
Page 6 GAO/T-RCED/AIMD-97-168
In preparation for the FY 1996 agencywide financial statement audits, the
OIG assessed NSF’s financial and administrative controls, including property
controls, and its general ledger accounting system, the effectiveness of
electronic data processing controls, and adherence to core financial
system requirements. Because NSF’s OIG lacked adequate audit resources,
the IG contracted with a separate independent public accounting firm for
the audit of NSF’s FY 1996 Statement of Financial Position (balance sheet).
The IG and NSF’s contract auditor issued a qualified opinion on NSF’s FY
1996 balance sheet due to inadequate supporting documentation and lack
of a system to confirm account balances for approximately 99 percent of
NSF’s property, plant and equipment (PP&E), which is held by NSF’s
contractors and grantees. In its efforts to resolve these accounting
problems, NSF management is considering the appropriate accounting
treatment under the new federal accounting standards. NSF management
believes that NSF has only a reversionary interest5 in PP&E held by its
contractors and grantees. Under the new federal accounting standards,
where this is the case, the assets are not to be reported as PP&E in federal
agency accounts unless the assets actually revert to federal ownership.
The new accounting standards provide for such assets to be accounted for
and reported as R&D investment. Under the standard, NSF would account
for the acquisition cost of PP&E for its contractors and grantees as an
annual investment (expense). As a result, NSF would not be required to
report related PP&E asset values in its financial statements. While PP&E held
by contractors and grantees would still be subjected to inventory controls
and audit, NSF’s OIG is concerned about the adequacy of its contractors and
grantees auditors’ scope for federal financial statement audit purposes.
NSF has requested formal approval of the accounting changes from OMB and
FASAB. Upon approval by OMB and FASAB, NSF plans to implement the R&D
investment standard for its FY 1997 financial statements, one year ahead
of the implementation date in the standard.
In addition, the IG and NSF’s contract auditor reported that NSF has not yet
met the CFO Act requirement for systematic measurement of performance.
NSF is in the process of developing a framework for identifying and
developing performance measures in response to CFO Act and GPRA
requirements.
5
A reversionary interest arises when NSF reserves the right to take back at the end of the grant or
contract term, assets that it funded for use by a contractor or grantee.
Page 7 GAO/T-RCED/AIMD-97-168
Further, to support GPRA implementation and to ensure that NSF is
complying with federal cost accounting standards, the CFO is considering
hiring an independent public accounting firm to evaluate its current cost
accounting system capability against the requirements in the new federal
cost accounting standards. NSF’s current financial system is capable of
capturing direct costs by organization and program activities and
allocating indirect costs. However, when NSF determines what
performance measures will be used to capture program and financial
outputs and outcomes, it will need to modify the system to relate costs to
financial and program performance data. Along these lines, NSF plans to
have an activity-based accounting system module operational by the end
of FY 1998. Also, NSF plans to report performance measures related to R&D
investment, including program outputs and outcomes.
The year 2000 computing problem is rooted in the way dates are recorded
Year 2000 Computer and computed in many computer systems. For the past several decades,
Challenges systems have typically used two digits to represent the year such as “97”
representing 1997, in order to conserve on electronic data storage and
reduce operating costs. With this two-digit format, however, the year 2000
is indistinguishable from 1900, 2001 from 1901 and so on. As a result of this
ambiguity, system or application programs that use dates to perform
calculations, comparisons, or sorting may generate incorrect results when
working with years after 1999.
Because converting systems to a four-digit year can be a massive
undertaking, agencies need to identify their inventories of mission-critical
computer systems, develop conversion strategies and plans, and dedicate
sufficient resources to converting and adequately testing their computer
systems and programs before January 1, 2000. Although we have not
evaluated NSF’s efforts, the agency reported in June that it had assessed
which systems to change, replace or discard.
To assist agencies in achieving year 2000 compliance, we developed an
assessment guide which provides a framework and a checklist for
assessing the readiness of federal agencies. The guide addresses issues
that will be common to most year 2000 needs. It provides information on
the scope of the challenge, and offers a structured approach for reviewing
the adequacy and agency planning and management of the year 2000
program. This approach includes five phases addressing awareness,
assessment, renovation, validation and implementation and are supported
by program and project management activities. However, because each
Page 8 GAO/T-RCED/AIMD-97-168
agency is different, there is no single, cookie cutter approach. Each agency
must tailor its year 2000 program in response to its unique needs.
In summary, a statutory framework exists for improving management in
the federal government. Fully resolving the problems, however, will
require sustained management attention. Congressional oversight, such as
is being given in this instance by the Chairman and the Committee, is also
key.
This concludes my statement. I would be happy to respond to any
questions you or the Members of the Committee may have.
(141063) Page 9 GAO/T-RCED/AIMD-97-168
Ordering Information
The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.
Orders by mail:
U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015
or visit:
Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC
Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.
For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:
info@www.gao.gov
or visit GAO’s World Wide Web Home Page at:
http://www.gao.gov
PRINTED ON RECYCLED PAPER
United States Bulk Rate
General Accounting Office Postage & Fees Paid
Washington, D.C. 20548-0001 GAO
Permit No. G100
Official Business
Penalty for Private Use $300
Address Correction Requested
Federal Management: Observations on the National Science Foundation
Published by the Government Accountability Office on 1997-07-24.
Below is a raw (and likely hideous) rendition of the original report. (PDF)