oversight

Federal Management: Observations on the National Science Foundation

Published by the Government Accountability Office on 1997-07-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                         United States General Accounting Office

GAO                      Testimony
                         Before the Senate Committee on Commerce, Science,
                         and Transportation
                         U.S. Senate


For Release
on Delivery
Expected at
                         FEDERAL MANAGEMENT
9:30 a.m., EDT
Thursday,
July 24, 1997
                         Observations on the
                         National Science
                         Foundation
                         Statement of Susan D. Kladiva,
                         Acting Associate Director,
                         Energy, Resources, and Science Issues,
                         Resources, Community and Economic
                         Development Division




GAO/T-RCED/AIMD-97-168
    Mr. Chairman and Members of the Committee:

    We are pleased to be here today to discuss a number of management and
    program issues at the National Science Foundation (NSF) as recently
    reported by the NSF Office of Inspector General (OIG) and GAO. Many of the
    management issues have been acknowledged by the agency’s senior
    management and are receiving varying levels of attention. We have not
    specifically reviewed NSF’s management and program areas, other than its
    implementation of the Government Performance and Results Act (Results
    Act). Therefore, as agreed with the Committee, my statement will focus on
    steps to address some of the systemic problems that are based on similar
    concerns across the federal government. Specifically, I will address
    (1) duplicate funding in the Small Business Innovation Research (SBIR)
    program at NSF and elsewhere; (2) challenges for NSF in implementing the
    Results Act; (3) the status of NSF’s efforts to prepare and have audited
    agencywide financial statements as required by the Chief Financial
    Officers (CFO) Act as amended; and (4) a framework for assessing NSF’s
    readiness to meet the computer challenges that will arise in the year 2000.

    In summary, Mr. Chairman:

•   Among the management issues identified by NSF’s OIG most in need of
    reform at NSF is the SBIR program. Regarding the SBIR Program, we have
    reported that duplicate funding of similar or even identical SBIR proposals
    submitted to more than one agency has occurred at NSF, NASA, and DOD.1
    According to agency officials, a few companies received funding for the
    same proposals twice, three times, and even five times before agencies
    became aware of the duplication. Several factors contribute to this
    problem, including (1) the evasion of certification procedures, so that
    companies fail to identify similar proposals submitted to other agencies,
    (2) the lack of a consensus on what constitutes a duplicate proposal, and
    (3) the general lack of interagency access to and exchange of current
    information about recent awards by other agencies. NSF has revised its
    certification form to require applicants to certify, under criminal penalties
    for perjury, exactly what, if any, applications for similar research were
    pending in other agencies. In response to our recommendations, the
    Administrator of the Small Business Administration (SBA) has taken steps
    to address all three issues.




    1
     Federal Research: Interim Report on the Small Business Innovation Research Program
    (GAO/RCED-95-59, Mar. 8, 1995).



    Page 1                                                               GAO/T-RCED/AIMD-97-168
•   In implementing the Results Act, NSF, like most agencies, is still struggling
    to develop the mission based goals and the performance measurement
    requirements of the Act. We recently found that NSF’s draft strategic plan is
    incomplete and not specific enough to allow the Congress to evaluate
    whether the agency’s goals are achievable.2 This is to be expected during
    the initial efforts of such a challenging management reform effort.
    Measuring the performance of science-related projects can be extremely
    difficult because a wide range of factors determine if and how a particular
    research and development (R&D) project will result in a commercial
    application or have other benefits, and it can take years for a research
    project to realize a successful outcome. Moreover, we recently reported
    that there is no single indicator or evaluation method that adequately
    captures the results of R&D.3 As a result, determining the specific outcomes
    resulting from federal R&D is a challenge that will not easily be resolved.
    The Army Research Laboratory, which was designated as a pilot project
    for performance measurement under the Results Act, has developed a
    multifaceted approach using quantitative indicators, peer review, and
    customer feedback to evaluate the results of R&D. This response to the
    challenges in measuring the impacts of research shows that some progress
    is being made in response to the Results Act and may provide useful
    guidance to other science agencies.

•   As required by the CFO Act as amended by the Government Management
    Reform Act (GMRA), one of NSF’s primary financial management challenges
    is to prepare and audit consolidated agencywide financial statements.
    Financial statements are required to be prepared and audited to instill
    greater accountability; to provide reliable financial information for
    managing government and program operations and for making difficult
    policy decisions. NSF received its first-time audit of its agencywide
    financial statements for fiscal year (FY) 1996. NSF’s auditor concluded that
    the reported property, plant, and equipment account balance is unreliable.
    Because about 99 percent of NSF’s property is in the custody of R&D
    contractors and grantees, NSF is considering corrective actions that will
    include a change in accounting treatment as well as adequate inventory
    controls. Also, in response to its auditor’s concerns, NSF is in the process
    of developing a framework for identifying and developing performance
    measures in response to the CFO Act’s and GPRA’s requirements.



    2
     Results Act: Observations on the National Science Foundation’s Draft Strategic Plan
    (GAO/RCED-97-203R, July 11, 1997).
    3
     Measuring Performance: Strengths and Limitations of Research Indicators (GAO/RCED-97-91, Mar. 21,
    1997).



    Page 2                                                                  GAO/T-RCED/AIMD-97-168
                       •   The year 2000 problem is caused by systems that typically use two digits to
                           represent years thereby making the year 2000 indistinguishable from 1900,
                           2001 indistinguishable from 1901, and so on. NSF like other federal
                           agencies that have system or application programs that use dates to
                           perform calculations, comparisons, or sorting may generate incorrect
                           results when working with years after 1999. Converting systems to a
                           four-digit year can be a massive undertaking for agencies such as NSF
                           because it involves identifying computer systems, developing conversion
                           strategies and plans, and dedicating sufficient resources to converting and
                           adequately testing their computer systems and programs before January 1,
                           2000. GAO has developed an assessment guide that provides a structured
                           approach for assessing the readiness of agency’s year 2000 programs.4


                           In recent years, the Congress has put in place a statutory framework for
Background                 improving management in the federal government, and for helping the
                           Congress and the executive branch make the difficult trade-offs that the
                           current budget environment demands. This framework includes as its
                           essential elements the Chief Financial Officers Act; information
                           technology reform legislation; and the Government Performance and
                           Results Act. We have begun a body of work on each of these management
                           areas which provides some broad guidance of use to this Committee in its
                           current review of NSF.


                           Because of the mismanagement of funds, the OIG identified SBIR as one
Duplicate Funding of       program most in need of reform. Duplicate funding of similar or even
SBIR Proposals             identical proposals submitted to more than one agency has occurred in
                           NSF, NASA, and DOD; however, steps have been taken to address the issue.
                           Several factors have contributed to the problem of duplicate funding.
                           First, the companies proposing projects have failed to identify identical
                           proposals they have made to other agencies, thereby evading the
                           certification procedure that requires them to provide such information. In
                           response to this problem, NSF’s OIG officials were concerned about the
                           need for more complete certification requirements and recommended that
                           the NSF certification form be revised and strengthened. NSF has
                           implemented this recommendation.

                           Second, the lack of definitions and guidelines for the key terms, such as
                           “similar” or “overlapping” research, has resulted in disagreement about
                           what constitutes duplicate research. SBA’s prior policy directives and the

                           4
                            Year 2000 Computing Crisis: An Assessment Guide (GAO/AIMD-10.1.14, Feb. 1997).



                           Page 3                                                              GAO/T-RCED/AIMD-97-168
                         individual agencies’ solicitations did not define key terms and thus
                         provided little or no guidance in avoiding the risk of duplicate funding. In
                         general, the absence of substantive definitions for key terms placed the
                         burden of judgment on the company. In some cases, as we stated in our
                         March 1995 report, the appropriate certification may be difficult to
                         determine and can lead to conflicts of opinion that may harm the SBIR
                         program as well as the individual company. In response to this problem,
                         officials in NSF’s OIG defined “overlapping” research and noted that the
                         scope and funding of some SBIR awards in 1994 were reduced to eliminate
                         overlapping work.

                         Third, the agencies lacked interagency access to and exchange of current
                         information about recent awards that might help to prevent or detect
                         duplicate awards. At the time of our report, SBA maintained a database for
                         the SBIR program that it used primarily to produce its annual report to the
                         Congress regarding the program. The information had a “time lag” of about
                         9 months because it was first processed by each agency and then
                         forwarded to SBA. Some SBIR officials, however, believed that the existing
                         methods might not be adequate for detecting duplication when the
                         agencies were receiving 20,000 proposals annually.

                         In response to these three issues, SBA generally agreed with our concerns
                         and modified its SBIR policy directive in July 1995 to address these
                         problems. SBA refined the certification statement on possible duplication,
                         clarified the definition of similar awards, and began work on developing a
                         computer system that would enable all agencies to access timely
                         information on awards being made by other agencies. The system is now
                         operational, but, according to an SBIR program official at SBA, further work
                         is under way to strengthen it.


                         Under the Results Act, federal agencies are to set strategic and annual
Implementation of the    goals, measure performance, and report on the degree to which goals are
Results Act in Science   met. The Results Act requires that an agency’s strategic plan contain the
Agencies                 following six critical elements: a mission statement; agencywide goals and
                         objectives; strategies and resources needed to achieve the goals and
                         objectives; the relationship between the long-term goals and objectives
                         and the annual performance goals; key external factors that could affect
                         the achievement of goals; and a description of how program evaluations
                         were and will be used to establish or revise strategic goals.




                         Page 4                                                GAO/T-RCED/AIMD-97-168
While NSF addressed five of the six required elements in its draft strategic
plan, at least four of them need further development, and one
element—key external factors—is not included in the current draft. NSF’s
outcome goals set out the long-term programmatic, policy, and
management goals to be accomplished through its program office
investments; but many of these goals are not expressed in a measurable
form. NSF’s plan provides some general dates for achieving its goals but
does not provide the underlying assumptions, projections, or a schedule
for initiating or completing significant actions. The strategies for achieving
NSF’s goals lack precision, making it unclear whether the Foundation and
the Congress will be able to assess whether the goals are achieved. Also
unclear is the process for communicating goals and objectives throughout
the agency and for assigning accountability to managers and staff for
achieving the goals. Finally, NSF’s draft strategic plan does not discuss how
the agency used specific program evaluations to develop its strategic goals
or the other components of the plan.

We recognize that NSF is currently revising its draft strategic and
performance plans based on comments received by various parties.
Nonetheless, as expected during the initial efforts of such a challenging
management reform effort, most science agencies like NSF, are still
struggling to develop mission based goals and performance measures.
Determining the specific outcomes resulting from federal R&D is a
challenge that will not easily be resolved. The experts in research
measurement have tried for years to develop indicators that would provide
a measure of the results of R&D. However, the very nature of the innovative
process makes measuring the performance of science-related projects
difficult. For example, a wide range of factors determines if and how a
particular R&D project will result in a commercial application or have other
benefits. It can also take many years for a research project to achieve
results.

Because of these difficulties, there is no single indicator or evaluation
method that adequately captures the results of R&D. Decisionmakers have
developed quantitative and qualitative indicators as proxies to assess the
results of R&D activity. Our March 1997 report discusses the strengths and
limitations in both types of indicators. The amount of money spent on R&D,
the primary indicator of research investment, is useful as a measure of
how much research is being performed. Having been refined over many
years, these data are generally available for both the public and private
sectors. However, the level of spending is not a reliable indicator of the
level of research results.



Page 5                                                 GAO/T-RCED/AIMD-97-168
                        Quantitative output indicators focus mainly on return on investment,
                        patenting rates, and bibliometrics—the study of publication-based data.
                        While implying a degree of precision, these indicators were not originally
                        intended to measure long-term R&D results. Qualitative assessment, such as
                        peer review, provides detailed information, but depends on criteria that
                        are inherently difficult to measure and on subjective judgment that is
                        vulnerable to bias.

                        Science agencies, like other agencies, must guard against the
                        understandable tendency to overly rely on goals and measures that are
                        easily quantifiable, such as the numbers of research grants provided and
                        completed, at the expense of what is truly important but more difficult to
                        measure, such as the difference the program makes in people’s lives.
                        However, the legislative history that accompanied the Results Act states
                        that agencies should not trivialize measurement by seeking to measure
                        performance in a forced or artificial way simply to present quantifiable
                        measures.

                        The Army Research Laboratory, which was designated as a pilot project
                        for performance measurement under the Act, developed a multifaceted
                        approach using quantitative indicators, peer review, and customer
                        feedback to evaluate the results of R&D. Although this is not the only
                        approach that can be taken, this response to the challenges in measuring
                        the impacts of research shows that some progress is being made in
                        response to the Results Act and may provide useful guidance to other
                        science agencies.


                        One of NSF’s primary financial management challenges is to prepare and
Implementation of the   have its consolidated agencywide financial statements audited, as required
CFO Act                 by the Chief Financial Officers Act of 1990, as amended by the
                        Government Management Reform Act (GMRA) of 1994. Financial
                        statements are required to be prepared and audited to instill greater
                        accountability and provide reliable financial information for managing
                        government and program operations and making difficult policy decisions.
                        For FY 1991 through FY 1995, NSF had only prepared and had audited the
                        financial statements for its Donations Account (Trust Fund). To meet the
                        challenge of preparing and auditing consolidated agencywide financial
                        statements, NSF’s CFO contracted with an independent public accounting
                        firm to prepare proforma consolidated financial statements for FY 1994
                        and FY 1995. The CFO also contracted for assistance in preparing
                        agencywide FY 1996 financial statements.



                        Page 6                                               GAO/T-RCED/AIMD-97-168
In preparation for the FY 1996 agencywide financial statement audits, the
OIG assessed NSF’s financial and administrative controls, including property
controls, and its general ledger accounting system, the effectiveness of
electronic data processing controls, and adherence to core financial
system requirements. Because NSF’s OIG lacked adequate audit resources,
the IG contracted with a separate independent public accounting firm for
the audit of NSF’s FY 1996 Statement of Financial Position (balance sheet).

The IG and NSF’s contract auditor issued a qualified opinion on NSF’s FY
1996 balance sheet due to inadequate supporting documentation and lack
of a system to confirm account balances for approximately 99 percent of
NSF’s property, plant and equipment (PP&E), which is held by NSF’s
contractors and grantees. In its efforts to resolve these accounting
problems, NSF management is considering the appropriate accounting
treatment under the new federal accounting standards. NSF management
believes that NSF has only a reversionary interest5 in PP&E held by its
contractors and grantees. Under the new federal accounting standards,
where this is the case, the assets are not to be reported as PP&E in federal
agency accounts unless the assets actually revert to federal ownership.
The new accounting standards provide for such assets to be accounted for
and reported as R&D investment. Under the standard, NSF would account
for the acquisition cost of PP&E for its contractors and grantees as an
annual investment (expense). As a result, NSF would not be required to
report related PP&E asset values in its financial statements. While PP&E held
by contractors and grantees would still be subjected to inventory controls
and audit, NSF’s OIG is concerned about the adequacy of its contractors and
grantees auditors’ scope for federal financial statement audit purposes.

NSF has requested formal approval of the accounting changes from OMB and
FASAB. Upon approval by OMB and FASAB, NSF plans to implement the R&D
investment standard for its FY 1997 financial statements, one year ahead
of the implementation date in the standard.

In addition, the IG and NSF’s contract auditor reported that NSF has not yet
met the CFO Act requirement for systematic measurement of performance.
NSF is in the process of developing a framework for identifying and
developing performance measures in response to CFO Act and GPRA
requirements.




5
 A reversionary interest arises when NSF reserves the right to take back at the end of the grant or
contract term, assets that it funded for use by a contractor or grantee.



Page 7                                                                    GAO/T-RCED/AIMD-97-168
                     Further, to support GPRA implementation and to ensure that NSF is
                     complying with federal cost accounting standards, the CFO is considering
                     hiring an independent public accounting firm to evaluate its current cost
                     accounting system capability against the requirements in the new federal
                     cost accounting standards. NSF’s current financial system is capable of
                     capturing direct costs by organization and program activities and
                     allocating indirect costs. However, when NSF determines what
                     performance measures will be used to capture program and financial
                     outputs and outcomes, it will need to modify the system to relate costs to
                     financial and program performance data. Along these lines, NSF plans to
                     have an activity-based accounting system module operational by the end
                     of FY 1998. Also, NSF plans to report performance measures related to R&D
                     investment, including program outputs and outcomes.


                     The year 2000 computing problem is rooted in the way dates are recorded
Year 2000 Computer   and computed in many computer systems. For the past several decades,
Challenges           systems have typically used two digits to represent the year such as “97”
                     representing 1997, in order to conserve on electronic data storage and
                     reduce operating costs. With this two-digit format, however, the year 2000
                     is indistinguishable from 1900, 2001 from 1901 and so on. As a result of this
                     ambiguity, system or application programs that use dates to perform
                     calculations, comparisons, or sorting may generate incorrect results when
                     working with years after 1999.

                     Because converting systems to a four-digit year can be a massive
                     undertaking, agencies need to identify their inventories of mission-critical
                     computer systems, develop conversion strategies and plans, and dedicate
                     sufficient resources to converting and adequately testing their computer
                     systems and programs before January 1, 2000. Although we have not
                     evaluated NSF’s efforts, the agency reported in June that it had assessed
                     which systems to change, replace or discard.

                     To assist agencies in achieving year 2000 compliance, we developed an
                     assessment guide which provides a framework and a checklist for
                     assessing the readiness of federal agencies. The guide addresses issues
                     that will be common to most year 2000 needs. It provides information on
                     the scope of the challenge, and offers a structured approach for reviewing
                     the adequacy and agency planning and management of the year 2000
                     program. This approach includes five phases addressing awareness,
                     assessment, renovation, validation and implementation and are supported
                     by program and project management activities. However, because each



                     Page 8                                                GAO/T-RCED/AIMD-97-168
           agency is different, there is no single, cookie cutter approach. Each agency
           must tailor its year 2000 program in response to its unique needs.


           In summary, a statutory framework exists for improving management in
           the federal government. Fully resolving the problems, however, will
           require sustained management attention. Congressional oversight, such as
           is being given in this instance by the Chairman and the Committee, is also
           key.

           This concludes my statement. I would be happy to respond to any
           questions you or the Members of the Committee may have.




(141063)   Page 9                                               GAO/T-RCED/AIMD-97-168
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested