oversight

Audit of NSF's Process for Evaluating the Operations and Maintenance Cost Proposal for the Ocean Observatories Initiative

Published by the National Science Foundation, Office of Inspector General on 2020-04-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

Audit of NSF’s Process for Evaluating
the Operations and Maintenance Cost
Proposal for the Ocean Observatories
Initiative




April 14, 2020
OIG 20-2-004
AT A GLANCE
Audit of NSF’s Process for Evaluating the Operations and Maintenance
Proposal for the Ocean Observatories Initiative
Report No. OIG 20-2-004
April 14, 2020

WHY WE DID THIS AUDIT
In September 2018, NSF awarded a $220 million, 5-year cooperative agreement to Woods Hole
Oceanographic Institution (WHOI) for operating the Ocean Observatories Initiative (OOI). To stay
within the annual budget of $44 million, WHOI plans to implement efficiencies to absorb inflation or
reduce science support (the scientific activities included in the agreed-upon scope of work). We
evaluated whether NSF has taken steps to ensure that WHOI’s proposed cost containment measures are
achievable within the proposed budget.

WHAT WE FOUND
NSF did not ensure WHOI’s proposal to operate OOI within the proposed $220 million budget
adequately addressed inflation and risks. Specifically, WHOI’s initial proposal did not address its
ability to absorb expected inflation into the fixed budget, and its supplementary inflation absorption
plan did not provide the basis of estimate or sufficient details regarding how WHOI would achieve cost
saving efficiencies. As a result, NSF could not have confidence that WHOI’s proposed cost savings
would compensate for expected inflation and allow WHOI to operate within budget without reducing
the proposed level of science support or increasing funding. Further, at the time of our audit, NSF did
not provide sufficient guidance on how to conduct risk and uncertainty analyses for operations
proposals and did not require it.
As a result of our audit, NSF developed new solicitation language to increase the quality of cost
estimates for all future operations awards and developed additional guidance in the Major Facilities
Guide on the use of risk/uncertainty and sensitivity analyses for future operations proposals. For
example, future cost estimates must contain escalation factors including inflation, and new proposals
must include additional information on risk.

WHAT WE RECOMMEND
We recommend that NSF analyze WHOI’s plans to determine if it is achieving its proposed efficiency
cost savings and ensure that future major facility operations proposals include inflation factors, as well
as risk and mitigation strategies.

AGENCY RESPONSE
NSF agreed with our recommendations. NSF’s response is included in its entirety in Appendix A.
FOR FURTHER INFORMATION, CONTACT US AT OIGPUBLICAFFAIRS@NSF.GOV.
                          National Science Foundation • Office of Inspector General
                             2415 Eisenhower Avenue, Alexandria, Virginia 22314


MEMORANDUM


DATE:                 April 14, 2020


TO:                   Teresa Grancorvitz
                      Office Head and Chief Financial Officer
                      Office of Budget, Finance and Award Management

FROM:                 Mark Bell
                      Assistant Inspector General
                      Office of Audits

SUBJECT:              Audit Report No. 20-2-004, Audit of NSF’s Process for Evaluating the
                      Operations and Maintenance Cost Proposal for the Ocean Observatories
                      Initiative

Attached is the final report on the subject audit. We have included NSF’s response to the draft report as
an appendix.

This report contains three recommendations aimed at improving NSF’s review and oversight of the
Ocean Observatories Initiative (OOI) award and other NSF major facilities in the operations phase. NSF
concurred with all of our recommendations. In accordance with Office of Management and Budget
Circular A-50, Audit Followup, please provide a written corrective action plan to address the report
recommendations. In addressing the report’s recommendations, this corrective action plan should detail
specific actions and associated milestone dates. Please provide the action plan within 60 calendar days.

We appreciate the courtesies and assistance NSF staff provided during the audit. If you have questions,
please contact Elizabeth Kearns, Director of Audit Execution, at 703.292.7100 or
oigpublicaffairs@nsf.gov.


cc:            Christina Sarris        Elizabeth Kearns      Matthew Hawkins
               Fleming Crim            Patrick Breen         Janis Coughlin-Piester
               Allison Lerner          James Ulvestad        Eddie Whitehurst
               Lisa Clough             Bauke Houtman         Dan Buchtel
               Anneila Sargent         Jeanette Hyatt        Louise Nelson
               William Easterling      Linnea Avallone       Karen Scott
               Scott Borg              Heather Gallagher
TABLE OF CONTENTS
Background .................................................................................................................................................... 1
Results of Audit.............................................................................................................................................. 2
     NSF Did Not Ensure WHOI’s Proposal Adequately Addressed Inflation and Risks .................... 2
     NSF Did Not Require Major Facilities Operations Proposals to Include a Risk Analysis ........... 4
Recommendations ........................................................................................................................................ 5
OIG Evaluation of Agency Response .......................................................................................................... 5
Appendix A: Agency Response .................................................................................................................... 6
Appendix B: Objective, Scope, and Methodology ..................................................................................... 8
Appendix C: OIG Analysis of WHOI’s Inflation Absorption Plan ............................................................ 10




ABBREVIATIONS
COL                    Consortium for Ocean Leadership
CPRD                   Cost Proposal Review Document
GAO                    U.S. Government Accountability Office
LFO                    Large Facilities Office
NSB                    National Science Board
OOI                    Ocean Observatories Initiative
UNOLS                  University-National Oceanographic Laboratory System
WHOI                   Woods Hole Oceanographic Institute
Background
As part of its mission, the National Science Foundation funds the design, construction, and operation of
major multi-user research facilities (major facilities), which are shared-use infrastructure such as
telescopes and research ships accessible to a broad community of researchers and educators. NSF’s
major facilities typically have construction costs ranging from one hundred to several hundred million
dollars and may operate for 20 to 40 years. NSF makes awards to external recipient entities (recipient) to
undertake major facility construction, operations, and maintenance, typically through cooperative
agreements, a type of Federal assistance award. As of July 2019, NSF had 23 major facilities including
the Ocean Observatories Initiative (OOI), an ocean research observatory that includes seven research
arrays comprising the world’s largest network of ocean and seafloor sensors, as shown in Figure 1.

In 2009, the National Science Board 1 (NSB)
approved funding for OOI, and NSF subsequently
awarded a $769 million cooperative agreement to the
Consortium for Ocean Leadership (COL) to
construct and operate the ocean observatory over the
next 8 years. 2 OOI began initial operations in 2010,
and COL completed construction in June 2016.

NSF originally budgeted funding for OOI operations
at $67.9 million in 2016 and $72.6 million in 2017.
However, in 2013, NSF decided to cap OOI’s budget
at $55 million per year beginning in fiscal year 2016,
the first year of full operations, due to a constrained
budget environment. In 2015, the National Research
Council 3 recommended that NSF make an
“immediate” reduction in funding for OOI
operations by 20 percent. In its response, dated May
11, 2015, NSF supported the recommendation, and
described its long-term plan for reducing costs by
20 percent, including its intent to solicit a new
cooperative agreement (to replace the existing
agreement set to expire in April 2017), with a
                                                                     Figure 1. Location of OOI Arrays
reduced annual budget of no more than $45 million
                                                                     Source: Woods Hole Oceanographic Institution


1
  The NSB establishes policies for NSF within the framework of applicable policies set forth by the President and Congress.
In this capacity, the NSB identifies issues that are critical to NSF’s future and approves new major programs and awards,
unless it has delegated such award approval authority to the Director. The NSB also serves as an independent body of
advisors to the President and Congress, providing reports every other year on the state of science and engineering in the U.S.,
as well as other reports as needed or by request, on specific, individual policy matters within NSF’s authority.
2
  The award included $386 million for construction and $383 million for operations.
3
  At NSF’s request, the National Research Council evaluated NSF's priorities for ocean science and issued recommendations
in Sea Change: 2015-2025 Decadal Survey of Ocean Sciences (The National Academies Press, 2015).



1      NSF.GOV/OIG | OIG 20-2-004-
per year. In November 2016, NSF issued a new solicitation for a cooperative agreement to operate OOI
with a reduced annual budget of $44 million.

In September 2018, NSF awarded a cooperative agreement to Woods Hole Oceanographic Institution
(WHOI) to operate OOI and approved funding of $220 million for a 5-year period. OOI operations
transitioned from COL to WHOI with the new award. WHOI’s award is flat funded at $44 million for
each of the 5 years included in the award. To stay within the annual budgeted amount determined by
NSF’s Division of Ocean Sciences, WHOI plans to implement efficiencies to absorb inflation or reduce
the level of science support (the scientific activities included in the agreed-upon scope of work). If
WHOI cannot absorb inflation or costs increase beyond the $44 million per year, NSF also has the
option to supplement OOI funds through its Research and Related Activities account.

The objective of this audit was to determine whether NSF has taken steps to ensure that WHOI’s
proposed cost containment measures are achievable, allowing OOI to function within the 5-year,
$220 million proposed budget.

Results of Audit
NSF did not ensure WHOI’s proposal to operate OOI within the proposed $220 million budget
adequately addressed inflation and risks. Specifically, WHOI’s initial proposal did not address its ability
to absorb expected inflation into the fixed budget, and its supplementary inflation absorption plan did
not provide sufficient details of the basis of estimate or how it would achieve cost saving efficiencies.
As a result, NSF could not have confidence that the proposed cost savings would compensate for
expected inflation and allow WHOI to operate within budget without reducing the proposed level of
science support or increasing funding. Further, at the time of our audit, NSF did not provide sufficient
guidance on how to conduct risk and uncertainty analyses for operations proposals and did not require it.

NSF Did Not Ensure WHOI’s Proposal Adequately Addressed Inflation and Risks
The proposal WHOI submitted in April 2017 to operate and manage OOI within a $44 million annual
budget for a period of 5 years did not address its ability to absorb expected inflation into the fixed
budget determined by NSF’s Division of Ocean Sciences. NSF’s June 2015 and March 2017 Large
Facilities Manual requires recipients to prepare proposals in accordance with the best practices included
in the GAO [U.S. Government Accountability Office] Cost Estimating and Assessment Guide.
According to the Large Facilities Manual and the GAO Cost Estimating and Assessment Guide, the
impact of inflation should have been included in WHOI’s cost estimate, and NSF should have reviewed
it to make a conclusion on the quality of those estimates.

NSF contracted an independent company to perform an Independent Cost Assessment of WHOI’s cost
proposal. In February 2018, the contractor recommended that NSF obtain further information on
WHOI’s inflation estimates and management, as well as a risk and uncertainty analysis. Subsequently,
on March 29, 2018, WHOI provided supplemental information, including a document titled WHOI
Management of Inflation Versus Fixed Program Budget (inflation absorption plan) and a cost sensitivity




2     NSF.GOV/OIG | 20-2-004
analysis, which described the areas of the budget that were most uncertain and may pose challenges over
the 5-year award period.

Based on WHOI’s inflation absorption plan, NSF concluded “[t]he estimated cost escalation and the
proposed approach to off-set the costs and mitigate the risks is appropriate and reasonable, given this is a
flat-funded award and the scope and budget will be continually refined through the Annual Work Plan
process.” However, NSF could not analyze WHOI’s estimates in accordance with GAO’s cost
estimating guidelines because WHOI’s inflation absorption plan did not provide the basis of estimate
and calculations for the cost savings. WHOI’s inflation absorption plan also lacked sufficient details
regarding how WHOI would achieve cost saving efficiencies. Therefore, NSF could not have confidence
that the proposed cost savings would be enough to cover inflation.

WHOI’s Planned Efficiencies May Not Be Enough to Manage Inflation

According to its inflation absorption plan, WHOI expected inflation to increase costs by an average of
   percent each year in years 2 through 5, totaling $      million during the life of the award. 4 WHOI
proposed the following three categories of efficiencies in years 2 through 5 to offset expected inflation:

    •    Reductions in refurbishment labor costs;
    •    Reductions to cruise lengths with improved weather prediction; and
    •    Reduction in refurbishment materials costs.

However, based on our analysis of WHOI’s inflation absorption plan (see Appendix C), expected cost
savings from these efficiencies may not be large enough to cover anticipated inflation costs over the
5-year award period. To remain within the flat budget, WHOI also proposed




WHOI’s Cost Sensitivity Analysis Sometimes Conflicted with Its Inflation Absorption Plan

As previously described, WHOI’s cost sensitivity analysis described the areas of the budget that are
most uncertain and may pose challenges over the 5-year award period. For example, the analysis
identified several routine annual cost increases, such as fuel and deployments lost at sea. WHOI did not
specifically identify these challenges in its original budget proposal.




4
  WHOI expects the projected inflation rate for the next 5 years to average about   percent per year. Applying the effects of
this increase alone, the program cost in Year 2 is $      million, Year 3 is $    million, Year 4 is $     million, and Year
5 is $      million.




3       NSF.GOV/OIG | 20-2-004
NSF’s Cost Proposal Review Document (CPRD), dated June 11, 2018, contains NSF’s evaluation of
WHOI’s proposal. Although NSF attached WHOI’s cost sensitivity analysis to the CPRD, it did not
include agency conclusions on the analysis or its potential impact on the budget. Instead, in its
evaluation in the CPRD, NSF stated, “WHOI provided … a summary discussion of its risk analysis
process, all of which will be further refined during the transition phase and incorporated in the first
Annual Work Plan for NSF approval.” A more thorough analysis of the supplemental information would
have provided more realistic expectations of the level of science support that can be maintained with
$44 million in funding.

NSF Did Not Require Major Facilities Operations Proposals to Include a Risk
Analysis

At the time of our audit, NSF did not require recipients to complete a risk analysis for operations awards
at proposal submission because it believed the award structure allowed sufficient flexibility to handle
unexpected costs during operations. For example, if risks arose the recipient could re-budget per the
award terms and conditions, reduce the level of science support with NSF’s approval if significant, or
request additional funding from NSF. NSF specified in its WHOI CPRD that its “… current policy
regarding [operations and maintenance] budgets does not require sensitivity and risk analysis to the
extent required for construction awards.”

Requiring proposing organizations to articulate risks — including estimated cost and likelihood,
mitigation strategies, and whether to accept each risk — could allow the proposing organization and
NSF to identify and better understand the likelihood of outcomes such as re-budgeting, reducing the
level of science support, or requesting supplemental funding during the operations stage.

NSF Corrective Actions
As a result of our audit, NSF has begun taking steps to improve the quality of future proposals for major
facility operations. For example, in December 2019, NSF revised its standard template language for
major facility operation solicitations. The new template clarifies that any proposed NSF funding level is
for planning purposes only and the actual funding amount will be based on the detailed cost estimate,
NSF's cost analysis, and the availability of funding. The template also requires cost estimating plans to
articulate escalation factors used, including inflation, and address risk.

Additionally, NSF revised its Major Facilities Guide (formerly the Large Facilities Manual) in
September 2019 to include strategies for handling risks during operations awards. However, the guide
does not require a risk and uncertainty analysis for operations proposals.




4     NSF.GOV/OIG | 20-2-004
Recommendations
We recommend that Office Head and Chief Financial Officer, Office of Budget, Finance and Award
Management, National Science Foundation:

    1. Perform an analysis of WHOI’s current budgetary estimates and constraints to determine if
       WHOI is achieving the proposed efficiency cost savings and, if necessary, determine the
       reductions to the level of science support that will need to occur to remain within the $44 million
       per year budget.

    2. Ensure that inflation factors are included in future major facility operations proposals and are
       reviewed and evaluated.

    3. NSF should develop internal and external guidance to ensure all operations proposals include an
       evaluation of key operational risks, their potential cost and scientific impacts, and mitigation
       strategies. The guidance should include instructions on determining whether to conduct a risk
       and uncertainty analysis or a sensitivity analysis, and how to document that analysis.


OIG Evaluation of Agency Response
NSF agreed with our recommendations. NSF’s response is included in its entirety in Appendix A.




5     NSF.GOV/OIG | 20-2-004
Appendix A: Agency Response




6   NSF.GOV/OIG | 20-2-004
7   NSF.GOV/OIG | 20-2-004
Appendix B: Objective, Scope, and Methodology
The objective of this performance audit was to determine whether NSF has taken steps to ensure that
WHOI’s proposed cost containment measures are achievable, allowing OOI to function within the
5-year, $220 million proposed budget.

To accomplish our objective, we:

    •    Determined NSF’s requirements for evaluating WHOI’s OOI proposal by analyzing NSF
         policies and procedures, including NSF's Large Facilities Manual, NSF 17-066, March 2017;
         NSF’s Major Facilities Guide, Draft for Public Comment, December 2018; NSF Proposal and
         Award Policies and Procedures Guide, Version NSF 17-1, January 30, 2017; NSF Standard
         Operating Guidelines; and the GAO Cost Estimating and Assessment Guide, GAO-09-3SP,
         March 2009, which NSF policies reference.
    •    Reviewed WHOI’s proposal (1743430) submitted April 17, 2017, for the Operations and
         Management of OOI including its supplemental documentation outlining proposed cost saving
         efficiencies to cover inflation throughout the award period.
    •    Evaluated NSF’s review of WHOI’s proposal by reviewing the Independent Cost Assessment
         (ICA) of Ocean Observatories Initiative (OOI) Project Operations,
                          (February 23, 2018) and the Cost Proposal Review Document: Management and
         Operation of Ocean Observatories Initiative (001), 2018-2023, Woods Hole Oceanographic
         Institute, June 11, 2018.
    •    Conducted interviews with the Programs Officers, Grants Officers, and officials from the Large
         Facilities Office.
    •    Reviewed open NSB meetings from 2000 to 2015 to identify NSB discussions on OOI budget
         decisions.
    •    Reviewed the August 15, 2017, February 21, 2018, and May 2, 2018 NSB Committee on Awards
         and Facilities meetings to determine if NSF presented information to the NSB on the risk
         associated with the OOI budget constraints.
    •    Reviewed prior reports and studies of the OOI project, including Sea Change: 2015-2025
         Decadal Survey of Ocean Sciences, National Academy of Sciences, (2015) and NSF’s response;
         An Ocean Blueprint for the 21st Century, U.S. Commission on Ocean Policy, 2004; Ocean
         Observatories Initiative (OOI) Scientific Objectives and Network Design: A Closer Look,
         Construction Final Report, Consortium for Ocean Leadership, September 2016; and Illuminating
         the Hidden Planet: The Future of Seafloor Observatory Science, National Academy Press, 2000.

To ensure that NSF applied appropriate internal controls in its review of WHOI’s proposal, we
compared the requirements in the Major Facilities A-123 Oversight Process, June 2017, and NSF’s
policies to NSF’s documentation of its review. We made recommendations in the report where NSF
could strengthen internal controls. We did not identify any instances of fraud, illegal acts, violations, or
abuse. We did not rely on computer-processed data to complete the audit.




8       NSF.GOV/OIG | 20-2-004
We conducted this performance audit between February 2019 and February 2020, in accordance with
Generally Accepted Government Auditing Standards. Those standards require that we plan and perform
the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and
conclusions, based on our audit objectives. We believe that the evidence obtained provides a reasonable
basis for our findings and conclusions.

Major contributors to this report include Dan Buchtel, Deputy Assistant Inspector General; Elizabeth
Kearns, Director, Audit Execution; Holly Snow, Audit Manager; Jeanette Hyatt, Senior Auditor;
Heather Gallagher, Senior Auditor; and Brittany Moon, Independent Report Referencer.




9     NSF.GOV/OIG | 20-2-004
Appendix C: OIG Analysis of WHOI’s Inflation Absorption Plan
Based on our analysis of WHOI’s inflation absorption plan, expected cost savings may not cover
anticipated inflation costs over the 5-year award period. As a result, the proposed level of scientific
support may need to be reduced or funding for OOI may need to be increased to cover expected inflation
costs.

WHOI proposed the following three categories of cost saving efficiencies in years 2 through 5 to absorb
inflation and stay within the annual budget:

     •    Efficiencies and Reductions in Refurbishment Labor Costs
     •    Reductions to Cruise Lengths with Improved Weather Prediction
     •    Reduction in Refurbishment Materials Costs

Efficiencies and Reductions in Refurbishment Labor Costs

Refurbishment labor includes efforts to clean, disassemble, and replace parts and equipment, and to
reassemble, integrate, and test moorings at array sites. According to the document WHOI Management
of Inflation Versus Fixed Program Budget, WHOI estimated annual costs for refurbishment labor would
decrease by




5
  Our analysis assumes that WHOI will achieve the maximum estimated efficiency of percent of refurbishment labor costs
included in WHOI’s Year 1 annual workplan, which is a savings of $        per year, or approximately $       over the
remaining 4 years of the cooperative agreement.




10       NSF.GOV/OIG | 20-2-004
Figure 2. WHOI’s Proposed Labor Refurbishment Cost Vs. Calculated Labor Included
Efficiency Savings and Inflation*




     Source: NSF OIG
     *
       The green bars depict WHOI’s proposed costs for refurbishment labor over the 5-year award. The blue bars depict total
     costs after accounting for estimated labor efficiencies and anticipated inflation,


 Reductions to Cruise Lengths with Improved Weather Prediction

 To refurbish the arrays, WHOI travels by boat to the array site to deploy and recover the arrays (turn
 cruises) one to four times each year. The cruise lengths range from approximately 19 to 42 days for a
 total of 174 days per year. Poor weather can increase the length of the turn cruises or require additional
 cruises if sea conditions prevent WHOI from deploying or recovering the 5-ton surface moorings.
 WHOI continuously gathers weather data to attempt to predict weather patterns to determine the ideal
 travel times.

 WHOI estimated it could save            percent of yearly cruise costs by gathering weather data to predict
 future weather patterns and plan shorter turn cruises. Based on WHOI’s annual operation plan for year 1,
 total cruise costs are just over $ million or an average of $           per day. If WHOI achieves savings
 of percent, this would equate to approximately fewer days of travel and a total savings of
 approximately $             per year. 6

 However, WHOI’s inflation absorption plan did not provide the basis of its estimate and calculations for
 the cost savings and did not include sufficient details regarding how WHOI would achieve the
 efficiencies. For example, WHOI plans to use several research ships from the Academic Research Fleet
 6
  Our analysis assumes that WHOI will achieve the maximum estimated efficiency of percent of cruise costs included in
 WHOI’s annual workplan, which is a savings of approximately $       per year, or $         over the 5-year cooperative
 agreement.




 11       NSF.GOV/OIG | 20-2-004
to conduct turn cruises.




Reduction in Refurbishment Materials Costs

WHOI’s initial annual operation plan includes approximately $     million per year in refurbishment
materials costs, or $    million over the 5-year cooperative agreement. WHOI uses these materials to
prepare the arrays for deployment.
                                                           WHOI plans to achieve significant savings
by




Table 1. Required Refurbishment Materials Cost Savings
  Year Percent Cost Reduction to Cover Inflation Savings Amount to Cover Inflation
   1*                                            0                                 $0
   2
   3
   4
   5
  Total
Source: NSF
*
  There is no inflation in year 1. We assume efficiency cost savings will not begin until year 2.




12     NSF.GOV/OIG | 20-2-004
About NSF OIG
We promote effectiveness, efficiency, and economy in administering the Foundation’s programs; detect
and prevent fraud, waste, and abuse within NSF or by individuals who receive NSF funding; and
identify and help to resolve cases of research misconduct. NSF OIG was established in 1989, in
compliance with the Inspector General Act of 1978, as amended. Because the Inspector General reports
directly to the National Science Board and Congress, the Office is organizationally independent from the
Foundation.

Obtaining Copies of Our Reports
To view this and any of our other reports, please visit our website at www.nsf.gov/oig.

Connect with Us
For further information or questions, please contact us at OIGpublicaffairs@nsf.gov or 703.292.7100.
Follow us on Twitter at @nsfoig. Visit our website at www.nsf.gov/oig.

Report Fraud, Waste, Abuse, or Whistleblower Reprisal
   • File online report: https://www.nsf.gov/oig/report-fraud/form.jsp
   • Anonymous Hotline: 1.800.428.2189
   • Email: oig@nsf.gov
   • Mail: 2415 Eisenhower Avenue, Alexandria, VA 22314 ATTN: OIG HOTLINE




13    NSF.GOV/OIG | 20-2-004
                              NATIONAL SCIENCE FOUNDATION
                              OFFICE OF INSPECTOR GENERAL




14   NSF.GOV/OIG | 20-2-004