oversight

Audit of NSF's Evaluation and Assessment Capability Section's Use and Oversight of Contracts

Published by the National Science Foundation, Office of Inspector General on 2020-03-23.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

Audit of NSF’s Evaluation and
Assessment Capability Section’s
Use and Oversight of Contracts




 March 23, 2020
 OIG 20-2-005
NSF     NATIONAL SCIENCE FOUNDATION
        OFFICE OF INSPECTOR GENERAL


 AT A GLANCE
 Audit of NSF’s Evaluation and Assessment Capability Section’s
 Use and Oversight of Contracts
 Report No. OIG 20-2-005
 March 23, 2020

  WHY WE DID THIS AUDIT
  NSF uses evaluations, among other methods, to inform its strategic planning, assess progress on
  strategic objectives, and examine program effectiveness. In 2014, NSF established the Evaluation and
  Assessment Capability (EAC) Section, which assists directorates and program offices commissioning
  external evaluations of programs and investments. We conducted this audit to determine whether
  (1) NSF follows Federal and NSF contracting policies and procedures for its evaluation contracts and
  (2) NSF uses EAC’s contracted evaluations for policy decision-making and planning.

  WHAT WE FOUND
  EAC generally followed Federal and NSF contracting policies and procedures for the contracts in our
  audit sample; however, it could improve the planning of its contracted evaluations. Specifically, NSF
  may have opportunities to use strategic sourcing and firm-fixed-price contracts for its evaluation
  contracts. Further, although some EAC evaluations resulted in positive change, NSF has not always
  used evaluation results to inform decision-making and strategic planning nor ensured the results were
  publicly disseminated. In addition, evaluations were not always completed timely, in part due to
  program office delays in reviewing key contract deliverables. At the time of our audit, NSF also did
  not have an agency-wide evaluation plan to help focus evaluation resources on high priority issues,
  but it designated a senior official responsible for the plan’s development. EAC has an opportunity to
  further develop policies and procedures to guide planning and communication between EAC and
  program offices.

  WHAT WE RECOMMEND
  We made four recommendations that, if implemented, should enhance EAC’s policies and procedures
  for evaluations and NSF’s evidence-based planning and policymaking.

  AGENCY RESPONSE
  NSF agreed with our recommendations. NSF’s response is included in its entirety in Appendix A.

  FOR FURTHER INFORMATION, CONTACT US AT OIGPUBLICAFFAIRS@NSF.GOV.
                          National Science Foundation • Office of Inspector General
                             2415 Eisenhower Avenue, Alexandria, Virginia 22314

MEMORANDUM

DATE:                 March 23, 2020

TO:                   Fleming Crim
                      Chief Operating Officer

                      Suzanne Iacono
                      Office Head
                      Office of Integrative Activities

FROM:                 Mark Bell
                      Assistant Inspector General
                      Office of Audits

SUBJECT:              Audit Report No. 20-2-005, Audit of NSF’s Evaluation and Assessment Capability
                      Section’s Use and Oversight of Contracts

Attached is the final report on the subject audit. We have included NSF’s response to the draft report as
an appendix.

This report contains four recommendations aimed at enhancing NSF’s evidence-based planning and
policymaking and the Evaluation and Assessment Capability Section’s policies and procedures. NSF
concurred with all of our recommendations. In accordance with Office of Management and Budget
Circular A-50, Audit Followup, please provide a written corrective action plan to address the report
recommendations. In addressing the report’s recommendations, this corrective action plan should detail
specific actions and associated milestone dates. Please provide the action plan within 60 calendar days.

We appreciate the courtesies and assistance NSF staff provided during the audit. If you have questions,
please contact Elizabeth Kearns, Director of Audit Execution, at 703.292.7100 or
oigpublicaffairs@nsf.gov.


cc:            Christina Sarris       Keith Boyea            Elizabeth Kearns
               Clemencia Cosentino    Allison Lerner         Melissa Prunchak
               Cynthia Phillips       Lisa Vonder Haar       Vashti Young
               Teresa Grancorvitz     Dan Buchtel            Ashley Lippolis Aviles
               Patrick Breen
NSF          NATIONAL SCIENCE FOUNDATION
             OFFICE OF INSPECTOR GENERAL



 TABLE OF CONTENTS
 Background ......................................................................................................................................... 1
 Results of Audit................................................................................................................................... 1
   NSF May Have Opportunities to Improve Contracting for Evaluations ................................... 1
   NSF Used Some EAC Evaluations to Drive Positive Change
   But Has Not Always Used Evaluation Results ............................................................................ 3
   Evaluations Were Not Always Completed Timely ...................................................................... 4
   NSF Has Not Shared All Evaluation Results ............................................................................... 5
   EAC Can Further Develop Policies and Procedures ................................................................... 5
   New Act Requires an Agency-wide Evaluation Plan .................................................................. 6
 Recommendations ............................................................................................................................. 6
 OIG Evaluation of Agency Response ............................................................................................... 7
 Appendix A: Agency Response ......................................................................................................... 8
 Appendix B: Objectives, Scope, and Methodology ...................................................................... 10




 ABBREVIATIONS
 COR                    Contracting Officer’s Representative
 DACS                   Division of Acquisition and Cooperative Support
 EAC                    Evaluation and Assessment Capability
 OMB                    Office of Management and Budget
 SOW                    Statement of Work
NSF       NATIONAL SCIENCE FOUNDATION
          OFFICE OF INSPECTOR GENERAL


 Background
 The National Science Foundation is an independent Federal agency created by Congress in 1950 “to
 promote the progress of science; to advance the national health, prosperity, and welfare; to secure the
 national defense....” NSF uses evaluations, 1 among other methods, to inform its strategic planning,
 assess progress on strategic objectives, and examine program effectiveness.

 In 2014, NSF established the Evaluation and Assessment Capability (EAC) Section within its Office of
 Integrative Activities. EAC assists directorates and program offices commissioning external evaluations
 of programs and investments. It also conducts its own evaluations of NSF programs and helps oversee
 evaluation contracts. EAC provides centralized support and resources for data collection, analytics, and
 the design of evaluation studies and surveys. These activities are intended to enable NSF to more
 consistently evaluate the impacts of its investments, to make more data-driven decisions, and to establish
 a culture of evidence-based planning and policymaking. EAC operates with a $3 million budget and four
 staff members. EAC conducts its own evaluations and works to enhance NSF capabilities for evaluation,
 knowledge management, and decision-making. Our audit focused on EAC’s portfolio of contracted
 evaluations; please see Appendix A for our objectives, scope, and methodology.

 The objectives of this audit were to determine whether (1) NSF follows Federal and NSF contracting
 policies and procedures for its evaluation contracts and (2) NSF uses EAC’s contracted evaluations for
 policy decision-making and planning.

 Results of Audit
 EAC generally followed Federal and NSF contracting policies and procedures for the contracts in our
 audit sample; however, it could improve the planning of its contracted evaluations. Specifically, NSF
 may have opportunities to use strategic sourcing and firm-fixed-price contracts for its evaluation
 contracts. Further, although some EAC evaluations resulted in positive change, NSF has not always used
 evaluation results to inform decision-making and strategic planning nor ensured the results were
 publicly disseminated. In addition, evaluations were not always completed timely, in part due to
 program office delays in reviewing key contract deliverables. At the time of our audit, NSF also did not
 have an agency-wide evaluation plan to help focus evaluation resources on high priority issues, but it
 designated a senior official responsible for the plan’s development. EAC has an opportunity to further
 develop policies and procedures to guide planning and communication between EAC and program
 offices.

 NSF May Have Opportunities to Improve Contracting for Evaluations
 For our audit sample, we judgmentally selected 4 of 12 of EAC’s complete and ongoing evaluation
 contracts; see Table 1. As this is a judgmental sample, we cannot project our findings across the total
 population of EAC’s evaluation contracts.
 1
  The Foundations for Evidence-Based Policymaking Act of 2018 (Pub. L. No. 115-435) defines an evaluation as “an
 assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to
 assess their effectiveness and efficiency.”

                                                              1
NSF        NATIONAL SCIENCE FOUNDATION
           OFFICE OF INSPECTOR GENERAL


      Table 1. Judgmentally Selected Evaluation Contracts
                                                       Award Effective       Contract Ceiling         Final Amount
                    Evaluation Title
                                                           Date                 Amount                  Expended
       Evaluation of the NSF’s Science,
       Engineering, and Education for                       4/3/2014             $1,750,259             $1,502,867
       Sustainability Portfolio of Programs
       Evaluation of the Innovation Corps
       Teams Program                                       9/24/2015             $1,128,409            $1,298,556*
       Evaluation of the NSF Inclusion across
       the Nation of Communities of Learners
                                                           9/26/2016             $3,425,426            $3,420,719**
       of Underrepresented Discoverers in
       Engineering and Science Program
       Evaluation of the NSF Geoscience
                                                           9/18/2017              $626,282              $625,213**
       Education and Diversity Portfolio
      Source: NSF OIG-generated based on evaluation contract information.
      *
        NSF increased the contract ceiling amount by about $170,000 to complete the task identified due to NSF delays.
      **
         NSF should close these contracts within 20 months from receipt of the final deliverable. As a result, this amount may
      increase.

 EAC and the Division of Acquisition and Cooperative Support (DACS) generally followed Federal
 and agency regulations when awarding the four contracts in our sample and managed these contracts
 as required. DACS also ensured the contracting officer’s representatives (COR) assigned to the
 evaluation contracts were properly trained and certified. However, EAC may have opportunities to
 strategically source its contracts and negotiate firm-fixed-price or hybrid contracts where appropriate.

 EAC May Have Opportunities to Strategically Source its Contracts

 According to NSF’s Acquisition Manual, NSF must use “collaborative strategic sourcing plans and
 implementation strategies for sourcing goods and services more effectively” and to “achieve
 improvements in price, performance, total cost of ownership, and overall business efficiency….” In
 addition, one of the President’s Management Agenda 2018 goals is to increase the Government’s
 strategic buying of goods and services. 2

 Contracting officers did not use strategic sourcing for the four contracts in our audit sample because of
 the unique nature of the requirements. However, a senior DACS official said DACS may be able to
 design a strategic sourcing plan that would allow NSF to procure evaluation services for multiple
 evaluations if EAC provided their requirements for several evaluations at once. Better communication
 with DACS regarding ongoing evaluation needs could help ensure EAC does not miss the opportunity
 to develop a strategic sourcing plan with DACS. For example, NSF may be able to realize cost savings
 by procuring evaluation services for multiple program offices through a Blanket Purchase Agreement,
 which can be used to fill repetitive needs for supplies or services.

 2
  “meaning that they share in contracts… [that provide savings] for taxpayers, in order to buy common goods and services as
 an enterprise.”

                                                                2
NSF       NATIONAL SCIENCE FOUNDATION
          OFFICE OF INSPECTOR GENERAL

 EAC May Have Opportunities to Use Firm-Fixed-Price Contracts

 Of EAC’s 12 contracted evaluations, 11 used time-and-materials contracts and one used a hybrid of
 firm-fixed-price for the first half of the contract and time-and-materials for the second half. A time-
 and-materials contract “provides for acquiring supplies or services” based on “[d]irect labor hours at
 specified fixed hourly rates that include wages, overhead, general and administrative expenses, and
 profit” and the “[a]ctual cost for materials[.]” 3 As the Federal Acquisition Regulation explains, such a
 contract “may be used only when” an agency cannot clearly define its needs. 4 NSF’s Acquisition
 Manual moreover, deems time-and-materials contracts as “high risk” or “medium risk” — meaning
 that there is a “strong” or at least “some likelihood that the technical, cost and/or schedule risk(s) to
 NSF over the life of the acquisition could result in a negative impact on the NSF mission.” 5 By
 contrast, a firm-fixed-price contract “provides maximum incentive for the contractor to control costs
 and perform effectively....” 6 An agency can also combine contract types if firm-fixed-price cannot be
 used for the entire contract. 7

 A former EAC official and a current DACS official we interviewed both said that NSF may be able to
 use a firm fixed-price contract for part, or all, of the contract if EAC clearly defines its needs.
 However, DACS staff told us that EAC and the program offices do not invest enough time during the
 planning phase to develop Statements of Work (SOW) that would allow them to use firm-fixed-price
 contracts. Better coordination with the program offices and DACS to develop detailed SOWs could
 allow NSF to take advantage of firm-fixed-price or hybrid contracts if appropriate.

 NSF Used Some EAC Evaluations to Drive Positive Change But Has Not Always
 Used Evaluation Results

 NSF has used evaluation results to drive positive change when EAC coordinated with the program
 offices and ensured the evaluation was well planned. For example, program staff reported using the two
 most recent evaluations in our sample — which EAC helped design and manage in their entirety —
 more than the prior two evaluations, which NSF initiated before fully staffing EAC. Additionally, for
 one of the more recent contracted evaluations, EAC and the program office were involved in annual
 summary report review meetings and discussions on how they could use and implement the evaluation
 results. In this instance, the program office said the evaluation helped to bring positive change to the
 program and had an impact on program outcomes.

 However, EAC has not always ensured NSF uses evaluation results to inform decision-making and
 strategic planning or to assess the performance of NSF processes and programs. For one evaluation in
 our sample, EAC did not have additional plans beyond the contractor’s final presentation to help the
 program office apply the evaluation results. EAC noted the evaluation had taken a long time to execute
 and the program office experienced high staff turnover, which negatively impacted evaluation support.

 3
   48 CFR § 16.601(b)
 4
   48 CFR § 16.601(c)
 5
   NSF’s Acquisition Manual, Subpart 2542.302-70(d)(2), (3)
 6
   48 CFR § 16.202-1
 7
   48 CFR § 16.104(e)

                                                              3
NSF          NATIONAL SCIENCE FOUNDATION
             OFFICE OF INSPECTOR GENERAL

 Program office staff told us the timing of the final evaluation report did not align with program decisions
 or next steps and could not be used because the program had changed during the time taken to complete
 the evaluation.

 Evaluations Were Not Always Completed Timely
 According to NSF’s Evaluation Principles, dated August 2016, “[e]valuations and assessments are
 deemed relevant when they produce practical, timely, and actionable evidence.” We found that for each
 of the four contracts in our sample, NSF did not receive the final evaluation report by the due date
 originally established in the SOW, which details contract requirements. The most significant delay —
 27 months — resulted in a contract cost increase of more than $170,000; see Table 2.

 Table 2. Evaluation Contract Delays
                                                      Award            Final        Final Report      Delay of
                                                                                                                     Cost of
                 Evaluation Title                    Effective       Report Due       Received         Final
                                                                                                                     Delay
                                                       Date             Date            Date          Report
     Evaluation of the NSF’s Science,
                                                                                                        17
     Engineering, and Education for                   4/3/2014        4/3/2017        8/24/2018                        $0
                                                                                                       months
     Sustainability Portfolio of Programs
     Evaluation of the Innovation Corps                                                                 27
                                                     9/24/2015       3/24/2017        6/28/2019                     $170,147
     Teams Program                                                                                     months
     Evaluation of the NSF Inclusion across
     the Nation of Communities of Learners
                                                     9/26/2016       9/23/2019       11/21/2019       2 months         $0*
     of Underrepresented Discoverers in
     Engineering and Science Program
     Evaluation of the NSF Geoscience
                                                     9/18/2017       2/28/2018        8/27/2018       6 months         $0*
     Education and Diversity Portfolio
 Source: NSF OIG-generated based on evaluation contract information.
 *
   NSF should close these contracts within 20 months from receipt of the final deliverable. As a result, this amount may
 increase.

 The contracted evaluation planning process has many steps, which may include identifying the
 evaluation need, drafting a SOW, procuring evaluation services, developing the evaluation plan, and
 obtaining approval from the Office of Management and Budget (OMB) to conduct surveys. EAC
 acknowledged contractors have needed more time to complete the planning phase than EAC projected
 and to which the contractor agreed. We found one contractor took 6 months longer than expected to
 draft the evaluation plan and another took 5 months longer than expected to submit a final evaluation
 plan. EAC extended contracts in our sample due to NSF-caused delays, including program offices being
 slow to review contractor deliverables.

 In some cases, these delays also contributed to delays in receiving OMB approval. Under the Paperwork
 Reduction Act of 1995 (PRA), 8 Federal agencies must obtain OMB clearance to survey (using identical
 questions) 10 or more non-Federal individuals or entities, and OMB can take up to 60 days to approve


 8
     Pub. L. No. 104-13, codified at 44 U.S.C. 3501 et seq.

                                                                 4
NSF         NATIONAL SCIENCE FOUNDATION
            OFFICE OF INSPECTOR GENERAL

 survey requests. 9 Three of the four contracts in our sample required PRA approval from OMB. For two
 of these contracts, NSF delays in providing feedback on deliverables resulted in delays in obtaining
 OMB approval and NSF extending the periods of performance. In one instance, OMB approval was
 delayed nearly 2 years as NSF redesigned the original study, conducted exploratory testing, restarted the
 PRA clearance process, and did not review and finalize the contractor’s OMB package in a timely
 manner.

 NSF Has Not Shared All Evaluation Results
 According to NSF’s Evaluation Principles, “[r]esults of evaluations will be made public, unless
 specifically prohibited.” For one evaluation contract in our sample, NSF publicly disseminated results
 from the annual summary reports required by the contract. However, NSF did not publicly release the
 results of three evaluations in our sample. NSF also does not have a central repository where it can share
 evaluation results internally. Consequently, NSF staff may not always be aware of previous or ongoing
 evaluations or be able to easily locate evaluation results. For example, a contracted evaluation was
 previously completed for a program office, but program staff could not find the evaluation results after a
 key staff member left NSF.

 NSF’s June 2018 Sharing of Non-public NSF Information — Interim Guidance allows contractors and
 evaluators to publish results if “(a) the scope of the data analysis and dissemination plans are articulated
 in the contract, which would be approved by relevant management, and (b) the draft publication has
 been approved for release by the cognizant [Assistant Director/Office Head] or Deputy [Assistant
 Director/Office Head].” EAC has not always included dissemination plans in its evaluation contracts. A
 former EAC official told us EAC releases information on a case-by-case basis and does not have a
 formal process for determining what to publish. NSF staff also told us they were open to publishing
 these evaluation results but faced obstacles, such as not knowing how best to release the reports on
 NSF’s website, or not having a version of the evaluation report that can be released to the public.

 EAC Can Further Develop Policies and Procedures
 Although EAC has some policies and procedures, it did not have sufficient policies and procedures to
 guide planning and communication between EAC and program offices. For example, EAC has
 procedures to assist program offices with establishing an evaluation strategy and SOW and provides
 contractors with checklists to establish evaluation standards. EAC also established evaluation principles
 and roles and responsibilities. However, EAC did not have an evaluation guide or handbook that would
 inform program offices of their evaluation process, or policies and procedures that describe how
 evaluation results will be used or implemented. EAC also had not established guidelines on the amount
 of time a program office has to complete deliverable reviews or expectations for communication
 between the two offices.

 The U.S. Government Accountability Office’s Standards for Internal Control in the Federal
 Government discusses developing and implementing policies and procedures to assist in ensuring an


 9
     See 44 U.S.C. § 3507(c)(2); 5 C.F.R. § 1320.10.

                                                       5
NSF       NATIONAL SCIENCE FOUNDATION
          OFFICE OF INSPECTOR GENERAL

 agency meets its objectives. Without sufficient policies and procedures, NSF’s contracted evaluations
 may continue to take longer and cost more to complete than expected.

 New Act Requires an Agency-wide Evaluation Plan
 According to a 2013 OMB memorandum, having an “agency-wide evaluation plan developed with
 senior policy and program officials can focus evaluation resources on high priority issues.” 10 At the time
 of our audit, NSF did not have an agency-wide evaluation plan. Although EAC included its evaluation
 plan in its budget request to Congress for FYs 2014–2021, until recently, it did not have the mandate or
 authority to establish an evaluation policy or plan for the entire Foundation.

 During our audit, the Foundations for Evidence-Based Policymaking Act of 2018 (Pub. L. No. 115-435)
 was enacted on January 14, 2019, and OMB subsequently issued implementing guidance to agencies on
 July 10, 2019. 11 The Act requires, in part, that an agency designate an Evaluation Officer to establish
 and implement an agency-wide evaluation plan. 12 According to OMB’s implementing guidance, each
 agency is required to submit an evaluation plan that covers evaluation activities planned for FY 2022,
 concurrent with the submission of its FY 2022 Annual Performance Plan by September 2020. Further,
 this concurrent submission must be repeated each year thereafter, with the annual evaluation plan being
 published along with the publicly disseminated annual performance plan.

 In July 2019, NSF designated EAC’s Section Head the agency Evaluation Officer, who is now
 responsible for the development of a comprehensive, agency-wide evaluation plan. This will help NSF
 consistently evaluate the impacts of its investments and continue to establish a culture of evidence-based
 planning and policymaking.



 Recommendations
 We recommend the Chief Operating Officer, National Science Foundation:

      1. Ensure the Evaluation Officer has the authority and resources necessary to implement applicable
         requirements in the Foundations for Evidence-Based Policymaking Act of 2018.

 We recommend the Evaluation Officer, National Science Foundation:

      2. Ensure NSF’s implementation plan meets the requirements of the Foundations for Evidence-
         Based Policymaking Act of 2018 and includes guidelines for internal and public dissemination
         and archiving of evaluation results.



 10
    OMB M-13-17, Next Steps in the Evidence and Innovation Agenda, July 26, 2013
 11
    OMB M-19-23, Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning
 Agendas, Personnel, and Planning Guidance
 12
    Not all requirements of the Foundations for Evidence-Based Policymaking Act of 2018 and its Implementation Guidance
 were in effect during the time of our audit; however, we aligned our recommendations with these new requirements.

                                                            6
NSF       NATIONAL SCIENCE FOUNDATION
          OFFICE OF INSPECTOR GENERAL

      3. Continue to develop policies and procedures, including:

             a. Expectations for program offices, including timeframes for program office staff to review
                deliverables and respond to information requests, and timelines for review of
                documentation for Paperwork Reduction Act clearance;
             b. Guidance for the level of support program offices will receive analyzing evaluation
                results and implementing recommendations; and
             c. Guidance for the dissemination of results and implementation of recommendations.

      4. Develop and implement contract planning procedures to ensure DACS and EAC fully consider
         the potential to use strategic sourcing and firm-fixed-price contracts.



 OIG Evaluation of Agency Response
 NSF agreed with our recommendations. NSF’s response is included in its entirety in Appendix A.




                                                     7
NSF   NATIONAL SCIENCE FOUNDATION
      OFFICE OF INSPECTOR GENERAL


 Appendix A: Agency Response




                         National Science Foundation
                         Office of the Director



       MEMORANDUM
                             MAR 0 9 2020
       Date:
       To:              Mark Bell, Assistant Inspector General, Office of         ~d~v
       From:            F. Fleming Crim , Chi ef Operating Officer, NSF
                        Suzanne Iacono, Office Head, Office oflntegrative Activities
       Subject:         NSF Response to the OIG's Official Draft Report for its Audit ofNSF 's
                        Evaluation and Assessment Capability Section 's Use and Oversight of Contracts
       NSF appreciates the opportunity to review and respond to the OIG's Official Draft Report for its
       Audit ofNSF 's Evaluation and Assessment Capability Section 's Use and Oversight of Contracts.
       NSF thanks the OIG for its thoughtful, iterative process in producing this draft report, which
       integrates input from different stakeholders at NSF. This approach has resulted in a repo1i with
       recommendations designed to bolster NSF' s ongoing efforts to strengthen the Evaluation and
       Assessment Capability Section (EAC) in the Office oflntegrative Activities (OIA) and support
       its important work in leading a robust Agency-wide response to the Foundations for Evidence-
       Based Policymaking Act of2018 (Evidence Act). This report also comes at an opportune time,
       as EAC transitions from its early inception to full implementation of evaluation services.
       NSF agrees with the OIG's four reconunendations and will pursue several actions aligned with
       these recommendations. These include:

             1. Continuing to socialize the requirements of, and oppo1iunities created by, the Evidence
                Act to focus Agency attention on the development and coordinated implementation of an
                Agency-wide evaluation plan.

             2. Developing new policies (such as an evaluation policy for the Agency as required by the
                Evidence Act) and enhancing existing policies to promote timely completion of
                evaluations, release of evaluation findings and data, and support for the interpretation and
                use of evaluation results to inform deci sion-making.
             3. Refining EAC's FY 2020 contracting and contract administration strategies. NSF will
                seek to maximize the use of common contract solutions in accordance with 0MB
                Memorandum M-19-13, including the use of solutions identified as "Best In Class" or
                solutions managed at the Government-wide level (e.g., Tier 3 and Tier 2 Spend Under
                Management designated solutions). Further, when conducting acquisition planning, EAC
                and the Division of Acquisition and Cooperative Support (DACS) in the Office of
                Budget, Finance and Award Management wi ll seek to maximize the use of firm-fixed-


                                  2415 Eise11ho1verAven11e, Suite 19100Alexa11dda, VA 2231./




                                                                          8
NSF   NATIONAL SCIENCE FOUNDATION
      OFFICE OF INSPECTOR GENERAL




              price (FFP) contracts, where practicable. Contract type selection and administration will
              be thoroughly documented in the Acquisition Plan and by required determinations and
              findings.
       On behalf of NSF staff pai.ticipating in this engagement, we further acknowledge the OIG's
       diligence and commitment to understanding the use and oversight of contracts by EAC. We look
       forwai.·d to receiving the final report. If you have any concerns, please contact Clemencia
       Cosentino, Section Head, EAC at ccosenti@nsf.gov or 703-292-4394.


       cc:    France Cordova, Director, NSF
              Diane Souvaine, Chair, NSB
              Anneila Sai.·gent, Chair, Committee on Oversight, NSB




                                                    2 of2




                                                                9
NSF       NATIONAL SCIENCE FOUNDATION
          OFFICE OF INSPECTOR GENERAL


 Appendix B: Objectives, Scope, and Methodology
 The objectives of this audit were to determine whether (1) NSF follows Federal and NSF contracting
 policies and procedures for its evaluation contracts and (2) NSF uses EAC’s contracted evaluations for
 policy decision-making and planning.

 While EAC conducts their own evaluations, we only focused on EAC’s contracted evaluations to
 determine how evaluation contracts are planned, monitored, and closed and how evaluations are used to
 make decisions. During our audit, EAC’s portfolio included five completed and seven ongoing
 evaluation contracts. We judgmentally selected four evaluation contracts for our sample and selected
 contracts that EAC initiated in FYs 2014 through 2017, that would be complete in FY 2019, and that had
 a variety of collaborating directorates. As this is a judgmental sample, we cannot project our findings
 across the total population of EAC’s evaluation contracts.

 Through research and information-gathering meetings, we identified applicable Federal requirements
 and reviewed a variety of resources on “best practices” related to evaluations, Federal contracting, and
 records management, including:

      •   American Evaluation Association Guiding Principles
      •   Council of the Inspectors General on Integrity and Efficiency Checklist for Review of Monitoring
          of Audit Work Performed by an Independent Public Accounting Firm
      •   Federal Acquisition Regulation
      •   Handbook of Practical Program Evaluation
      •   Miron Evaluation Report Checklist
      •   National Academies Measuring the Impacts of Federal Investments in Research: A Workshop
          Summary
      •   National Academies Advancing Concepts and Models for Measuring Innovation: Proceedings of
          a Workshop
      •   National Institutes of Health Partnerships for Environmental Public Health Evaluation Metrics
          Manual
      •   NSF Acquisition Manual
      •   NSF Contract Closeout Procedures Desk Guide
      •   NSF Contract Type Selection Guide
      •   NSF Contracting Officer's Representative Handbook
      •   NSF Evaluation Principles
      •   NSF Strategic Plan for Fiscal Years 2018–2022
      •   OMB M-13-17, Next Steps in the Evidence and Innovation Agenda
      •   OMB M-19-21, Transition to Electronic Records
      •   President’s Management Agenda (2018)
      •   Stufflebeam Evaluation Plans and Operations Checklist
      •   Stufflebeam Program Evaluations Metaevaluation Checklist
      •   U.S. Agency for International Development Evaluation Policy and Evaluation Toolkit
      •   U.S. Department of Energy Overview of Evaluation Methods for R&D Programs

                                                     10
NSF       NATIONAL SCIENCE FOUNDATION
          OFFICE OF INSPECTOR GENERAL

      •   U.S. General Services Administration Office of Evaluation Sciences Better Government Through
          Testing and Learning Workshop
      •   U.S. Government Accountability Office, GAO-12-208G, Designing Evaluations; and
      •   Urban Institute Making Evidence Relevant to Government.

 We assessed compliance with Federal and NSF policies and procedures, as well as assessed consistency
 with evaluation best practices. In addition, we reviewed documentation from the contracting officer’s
 and COR administration files. We assessed data reliability to ensure accuracy of the data used in our
 review. We held information-gathering meetings with staff in the Office of Integrative Activities,
 including in EAC, and the Office of Budget, Finance and Award Management, including DACS; we
 also met with program staff and CORs.

 We conducted this performance audit between August 2018 and January 2020 in accordance with the
 Inspector General Act of 1978, as amended, and Generally Accepted Government Auditing Standards.
 Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to
 provide a reasonable basis for our findings and conclusions, based on our audit objectives. We believe
 that the evidence obtained provides a reasonable basis for our findings and conclusions.

 Major contributors to this report include Elizabeth Kearns, Director, Audit Execution; Melissa
 Prunchak, Audit Manager; Vashti Young, Senior Management Analyst; Ashley Lippolis Aviles,
 Management Analyst, Elizabeth Argeris Lewis, Executive Officer and Communications Analyst; and
 Ruth Gonzalez, Independent Report Referencer.




                                                    11
NSF      NATIONAL SCIENCE FOUNDATION
         OFFICE OF INSPECTOR GENERAL


 About NSF OIG
 We promote effectiveness, efficiency, and economy in administering the Foundation’s programs; detect
 and prevent fraud, waste, and abuse within NSF or by individuals who receive NSF funding; and
 identify and help to resolve cases of research misconduct. NSF OIG was established in 1989, in
 compliance with the Inspector General Act of 1978, as amended. Because the Inspector General reports
 directly to the National Science Board and Congress, the Office is organizationally independent from the
 Foundation.

 Obtaining Copies of Our Reports
 To view this and any of our other reports, please visit our website at www.nsf.gov/oig.

 Connect with Us
 For further information or questions, please contact us at OIGpublicaffairs@nsf.gov or 703.292.7100.
 Follow us on Twitter at @nsfoig. Visit our website at www.nsf.gov/oig.

 Report Fraud, Waste, Abuse, or Whistleblower Reprisal
    • File online report: https://www.nsf.gov/oig/report-fraud/form.jsp
    • Anonymous Hotline: 1.800.428.2189
    • Email: oig@nsf.gov
    • Mail: 2415 Eisenhower Avenue, Alexandria, VA 22314 ATTN: OIG HOTLINE




                                                    12
e   NATIONAL SCIENCE FOUNDATION
    OFFICE OF INSPECTOR GENERAL




           13