National Science Foundation • Office of Inspector General
4201 Wilson Boulevard, Suite I-1135, Arlington, Virginia 22230
MEMORANDUM
DATE: September 30, 2014
TO: Dr. Pramod Khargonekar
Assistant Director, Directorate for Engineering
THRU: Allison Lerner /s/
Inspector General
FROM: Dr. Brett M. Baker
Assistant Inspector General for Audit
SUBJECT: NSF OIG Alert Memo Report No. 14-3-002, NSF’s Management of Costs
Proposed for the Large Synoptic Survey Telescope Construction Project
The purpose of this memo is to document our continued concern about NSF’s management of
large cooperative agreements. In light of our 2012 alert memo on this topic, 1 we have been
carefully following NSF’s actions with respect to the Large Synoptic Survey Telescope (LSST)
project, which is to be constructed in Chile. The LSST project was awarded at $467.7 million
(including $27.5 million obligated for Fiscal Year 2014) in August of 2014, and will be
conducted under a cooperative agreement with the Association of Universities for Research in
Astronomy (AURA). As discussed in this alert memo, we have serious concerns that NSF does
not have sufficient information to establish a reasonable basis for the cost of this project. In light
of those concerns, it is critical that NSF take strong action to ensure robust oversight of the costs
associated with this project as it proceeds.
One of our recommendations in the alert memo on NSF’s management of cooperative
agreements was that NSF obtain proposal and accounting system audits for high-risk cooperative
agreements in excess of $50 million to ensure that cost estimates are fair and reasonable and that
proposers’ accounting systems are adequate to bill the government properly. We also
recommended that NSF obtain incurred cost submissions and audits of these large projects.
Since the LSST project is the first construction project to be considered by the agency since we
made those recommendations, we watched to see what NSF would do to ensure the adequacy of
proposed costs. We found that an internal review completed at the Preliminary Design Review
stage by NSF’s Cost Analysis and Audit Resolution (CAAR) branch in June 2013 could not
independently verify costs for any of the 136 proposed expenditures sampled, including
1
NSF OIG Report No. 12-6-001, NSF’s Management of Cooperative Agreements, dated September 28, 2012.
1
approximately $145 million in direct materials, nearly $20 million for contingencies and more
than $6 million in direct labor costs. CAAR reported that without further documentation, it was
unable to determine if the methodology used to estimate the cost is appropriate, consistently
applied, or reasonable.
After this critical report, independent proposal and accounting system audits were clearly
warranted to ensure the adequacy and proper accounting of the proposed costs. Instead of
obtaining those, NSF had a contractor perform a “sufficiency review,” which is a less rigorous
option in the GAO Cost Estimating and Assessment Guide, GAO-09-3SP, that would not look at
information in sufficient detail to determine if the problems identified by CAAR had been
remedied. Subsequently, NSF developed a Cost Proposal Review Document (CPRD) to provide
more detail and follow-up on concerns raised by the CAAR review. However, the CPRD lacked
documentation and analysis to demonstrate that NSF performed a full review of two of the most
significant costs in the project’s proposed budget-- $ million in subcontracts/ subawards,
which comprise percent of the proposed costs, and $79 million in contingency costs. These
two elements—subcontracts and contingency-- comprise % of LSST’s total proposed costs,
and lack of support for these costs constitutes a fundamental accountability risk.
Given the concerns we had about the analysis of the project’s proposed costs, we reviewed the
CPRD to assess the extent to which it addressed the need for incurred cost submissions and
audits—tools which provide critical insight to both NSF management and NSF OIG after an
award is made. The CPRD did not address the matter of incurred cost submissions, so we cannot
determine the extent to which NSF will require preparation of these vital documents. With
respect to incurred cost audits, the CPRD notes that, based on NSF’s assessment of the project’s
risks, NSF determined that it would not require an incurred cost audit at the end of the first year
of performance. Based on post-award monitoring and risk assessment, NSF will decide if
incurred cost audits are needed at intermediate points during the agreement’s term; at minimum,
NSF will require an incurred cost audit at the expiration of the award. In light of the risks
detailed in this memo and the large amount of funding devoted to this project, we believe that
annual incurred cost submissions and audits are clearly warranted for this project.
Finally, as noted previously, the LSST project will be conducted under a cooperative agreement
with AURA. For four years, audits have documented significant estimating deficiencies and
concluded that AURA does not have an effective process for preparing adequate proposals. The
most recent audit of an AURA proposal, completed in August 2014, concluded that AURA’s re-
baselined cost proposal for the Advanced Technology Solar Telescope (newly renamed the
Daniel K. Inouye Solar Telescope (DKIST), was so deficient the auditors could not affirm that
the proposed costs were acceptable as a basis for a fair and reasonable price to the government.
In light of the known and continuing deficiencies with AURA’s estimating practices and cost
proposals and the lingering uncertainties about the reasonableness, accuracy, and currency of
many of the costs proposed for the LSST project, NSF should take immediate and strong action
to ensure that costs proposed for and incurred under the project comply with federal and NSF
requirements.
2
Background
For four years, the OIG has recommended that NSF strengthen accountability over its high-
dollar, high-risk cooperative agreements for its large facility construction projects. These
recommendations grew out of a large body of audit work examining awardees’ proposed costs
for such projects. Audits of three of these projects—the Ocean Observatories Initiative, the
Advanced Technology Solar Telescope, and the National Ecological Observatory Network
questioned a total of $305 million in unallowable or unsupported costs (out of $1.1 billion in
total costs for the three projects), $223 million of which was due to questioned amounts for
contingencies. 2
Our September 2012 alert memo on NSF’s management of high-risk, high-dollar cooperative
agreements outlined serious weaknesses in NSF’s management of these awards and
recommended that NSF improve both its pre-and post-award management. Appropriate controls
at the pre-award phase include audits of awardees’ proposed budgets and accounting systems to
ensure cost estimates are reasonable and accounting systems are adequate to bill the government
properly. A strong post-award process should include incurred costs submissions and audits to
help ensure that unallowable costs are not charged to the government, and should include a
requirement that awardees adequately track and identify contingency expenditures in their
accounting systems.
We recommended that, at a minimum, NSF implement such increased monitoring (obtaining
updated cost estimates; audits of proposed budgets and determinations of accounting system
adequacy at the pre-award stage; and incurred cost submissions and audits post-award) for
cooperative agreements valued at over $50 million. In response, in August 2013, NSF proposed
an alternate approach to strengthen its procedures for analysis of recipients’ cost proposals.
Among other things, it indicated that it would undertake a minimum of one of the following
actions: an independent audit; one of the actions in GAO’s Cost Estimating and Assessment
Guide; or project the organization’s cost trends for elements of cost on the basis of current and
historical information available to the Grants and Agreements Officer and to NSF or outside
experts.
While some of the actions NSF proposed appear to be sufficient for smaller, less risky projects,
we did not find them all to be adequate for high-dollar, high-risk projects. Accordingly, during
conversations with NSF management about their proposed response during the summer of 2013,
we urged NSF to conduct the type of stepped-up actions we identified in the alert memo for
cooperative agreements valued at over $200 million, or to suggest another threshold at which it
would strengthen its process. NSF disagreed with the $200 million threshold and has not
suggested a different one. The risk of NSF continuing its current practice is high: as of August
2013, it had 23 cooperative agreements worth over $50 million each and totaling over $4.2
billion.
2
Reports issued were Consortium of Ocean Leadership’s (COL) Ocean Observatories Initiative proposal in
September 2010 and in March 2012 (OIG Report Nos. 10-1-012 and 12-3-001); Association of Universities for
Research (AURA) in Astronomy’s Advance Technology Solar Telescope proposal and AURA’s accounting system
in March 2011 (OIG 11-1-001 and 11-1-010); and National Ecological Observatories Network’s (NEON) proposal
in September 2011 (OIG 11-1-021) and September 2012 (OIG 12-1-008).
3
Cost Estimates and the LSST Project
NSF’s Internal Review of LSST Costs at the Preliminary Design Stage
LSST was the first large facility project to make its way through the NSF approval process after
we escalated our recommendations on the need for NSF to strengthen its processes and thus was
an opportunity for NSF to demonstrate its commitment to accountability over hundreds of
millions of dollars in proposed costs.
In March of 2013, NSF asked staff in its CAAR division to conduct a cost analysis of the LSST
project at the preliminary design stage. CAAR’s review identified serious problems with the
LSST proposal. After sampling 136 transactions totaling 38 percent of the project’s costs,
CAAR was unable to independently verify the costs requested for any of the sampled costs. In
addition, through its sampling, CAAR also sought to gain more insight into the methodology
used to construct the cost estimate. When asked for further justification of the costs, the
organization often referred CAAR back to the established cost estimating plan or existing Work
Breakdown Structure data. Without further documentation, CAAR was not able to determine if
the methodology used to estimate the cost is appropriate, consistently applied, or reasonable.
CAAR also identified many problematic costs in the estimate, including:
• Salaries and wages: Median annual salaries for various positions were calculated based
upon the rates of pay for various positions at participating institutions. NSF’s internal
review found that most of the sampled salary expenditures could not be identified with
the median rates of pay provided and that salary amounts requested far exceeded pay
rates in supporting documentation provided to NSF.
• Equipment and associated indirect costs: Vehicles were proposed for both LSST and
AURA’s NOAO projects for use at the summit site, but the same supporting
documentation was provided for all proposed expenditures. NSF could not determine
whether the proposed costs represented duplicate requests. Additionally, indirect costs,
which normally are not allowed to be charged against equipment, had been proposed for
these purchases as had unnecessary amounts for contingency and risk.
• Management fees: Funds were requested for several different fees that may be
inconsistent with AURA’s current rate structure and/or NSF’s cooperative agreements.
CAAR also identified problems with subcontracts and associated indirect costs, with the
calculation and application of contingency costs and with the fringe benefit methodology,
escalation costs, indirect costs, fringe benefit pools, and LSST labor burden rate. In addition,
CAAR determined that the cost estimate for LSST was based on 2011 estimates. In May 2014,
NSF informed us that AURA had not provided NSF with current vendor quotes to support the
estimated costs.
The use of unsupported estimates and lack of current quotes, which are critical to developing
reliable cost estimates, is a serious and ongoing problem with cost proposals for large facility
construction projects at NSF, as documented in audits of the initial proposed budget for the $298
million Advanced Technology Solar Telescope (another project run by AURA) and the $434
million National Ecological Observatory Network project. Significant deficiencies rendered the
initial proposed budget for ATST unacceptable for audit and the auditors issued two inadequacy
4
memos, in March and October 2010 stating that direct material estimates were not current and
that direct labor and indirect costs were insufficiently supported, among other things. In June
2012, NSF requested AURA to re-baseline the project with a current cost estimate. In the case of
NEON, the flaws were so extensive that the auditors reached an adverse opinion, concluding that
the proposal was not an adequate basis on which to negotiate a fair and reasonable price.
Given the number of serious problems CAAR identified in the LSST proposal and the fact the
project was projected to cost NSF close to half a billion dollars, the agency should have taken
strong action to ensure that the issues identified by CAAR were addressed and that the final
proposed costs were reasonable. In keeping with the 2012 alert memo’s recommendations, a
proposal audit or independent cost estimate was clearly warranted.
Booz Allen Hamilton Review of LSST Cost Issues Identified in NSF’s Review
Instead of pursuing either of the options mentioned above, NSF contracted with Booz Allen
Hamilton for a sufficiency review—one of the less rigorous options in GAO’s Guide, and one
which would not examine the final proposal in sufficient detail to determine if the matters raised
in CAAR’s review had been rectified. Although the Booz Allen Hamilton report is written at a
much higher level than CAAR’s, in some areas it identified problems similar to those found in
CAAR’s review. In particular, the Booz Allen review found that AURA’s estimating ground
rules and assumptions only partially met requirements. It also concluded, in its assessment of the
extent to which the cost estimate was well developed and traceable, that the estimate only
partially met requirements.
NSF’s Cost Proposal Review Document for LSST
In the summer of 2014, NSF documented its analysis of the final LSST cost estimate in a Cost
Proposal Review Document (CPRD). In light of the issues raised by CAAR, we examined the
CPRD to determine the extent to which it addressed CAAR’s concerns. The CPRD provided
additional detail about some issues, such as labor, equipment, and escalation rates, about which
CAAR had raised concerns in its review.
As noted previously, one of the most significant issues CAAR raised was its inability to find
support for any of the 136 transactions it sampled. When we examined the CPRD to determine
the extent to which this issue had been addressed, we found that the CPRD lacked documentation
and analysis to demonstrate that NSF performed a full review of two of the most significant costs
in the project’s proposed budget-- million in subcontracts (called subawards), which
comprise percent of the proposed costs, and $79 million in contingency costs. These two
elements—subcontracts and contingency-- comprise % of LSST’s total proposed costs, and
lack of support for these costs constitutes a fundamental accountability risk.
The CPRD review of subcontract costs was limited to approximately $ million of labor and
$40 million of equipment, materials, and supplies. Even for the vast majority of those items,
there was little or no documentation to evidence that proposed subcontract costs were supported
by current vendor quotes. In addition, NSF has not reviewed or tested nearly $ million in
additional subcontract costs and thus has no visibility over these costs. As a result, the agency
has limited insight into the makeup of these costs and little, if any, assurance that they are
reasonable.
5
With respect to the $79 million in contingency, although the CAAR review identified problems
with the calculation and application of contingency costs, there was no evidence in the CPRD
review that AURA had provided documentation to support the amounts proposed for
contingencies. NSF acknowledged the lack of support for the full amount of the contingencies
proposed and has only authorized AURA to use $6.1 million in contingencies until further
documentation is provided, further NSF review is performed, and the award is modified
accordingly.
In light of these limitations in its analysis, NSF does not have a sufficient basis for concluding
that almost three-fourths of the amounts budgeted for the LSST project were reasonable and
adequately supported.
In light of the questions about the costs proposed for the project, we reviewed the CPRD to
assess the extent to which it addressed the need for incurred cost submissions and audits—tools
which provide critical insight to both NSF management and NSF OIG after an award is made.
The CPRD was silent on the matter of incurred cost submissions, so we cannot determine the
extent to which NSF will require preparation of these documents, which are vital for proper cost
monitoring and for the timely initiation of incurred cost audits. With respect to incurred cost
audits, the CPRD notes that NSF, based on its assessment of the project’s risks, determined not
to request an incurred cost audit subsequent to the completion of the first year of performance.
NSF indicated that, at a minimum, an incurred cost audit will be accomplished subsequent to
award expiration, and that NSF would monitor various matters (including the outcome of single
and project-specific audits) to determine if incurred cost auditing is needed at any point prior to
award expiration. In light of the risks detailed in this memo and the large amount of funding
devoted to this project, we believe that annual incurred cost submissions and audits are clearly
warranted.
Serious Flaws in AURA’s Cost Proposals
As noted previously, NSF will conduct the LSST project under a cooperative agreement with the
Association of Universities for Research in Astronomy (AURA). For four years, our audits have
repeatedly documented significant estimating deficiencies and concluded that AURA does not
have an effective process for preparing adequate proposals. A 2011 accounting system and
estimating practices audit of AURA identified eight significant deficiencies in the design of
AURA’s accounting system and in its estimating practices used on NSF awards. 3 A subsequent
preaward accounting system follow-up audit, performed in 2013, found that AURA had
developed procedures to resolve the accounting system deficiencies. However, there has been no
post-award accounting system audit or estimating system audit performed to verify that the
procedures have been satisfactorily implemented to ensure that 1) the accounting system is
adequate for accumulating and billing costs to the government and 2) estimating deficiencies
have been corrected and will result in current, accurate, and complete cost proposals and annual
program plans.
3
NSF OIG Audit Report No. OIG-11-1-010, Audit of Association of Universities for Research in Astronomy,
Inc.’s/National Optical Astronomy Observatories’ (NOAO) Accounting System and Proposal Estimating Practices,
dated March 31, 2011.
6
The most recent audit of AURA’s $344.1 million cost proposal for the Advanced Solar
Technology Telescope (i.e., DKIST) found that the majority of costs in the proposed budget
were not supported by current, accurate, and complete data. The auditors also found that AURA
did not use actual costs in the re-baseline of the proposal (even though actual costs were
available for 2010-2013) and that AURA included contingency costs that were explicitly
unallowable per OMB regulations. The auditors disclaimed an opinion on the proposal and
could not affirm that the proposed costs were acceptable as a basis for a fair and reasonable
price. We will be making recommendations addressing these concerns in our transmittal of the
re-baselined DKIST audit.
In light of the known deficiencies in AURA’s estimating practices and the serious questions
raised by NSF’s internal review of the LSST costs, it is important to ensure that AURA has in
place adequate accounting and estimating practices to manage this nearly $500 million project.
AURA’s estimating deficiencies greatly heighten the risk associated with the project and makes
it even more important to have strong visibility into and controls over costs.
Conclusion
NSF management received the results of CAAR’s review in June 2013. It had an opportunity at
that time to ensure a thorough review of project costs was conducted in a timely fashion. Instead,
it chose to proceed with a review that was not sufficiently in-depth to ensure that the problems
CAAR identified were rectified. Consequently, in May 2014 (the period in which the National
Science Board (NSB) approval to make the award was sought) we questioned whether NSF had
sufficient information to establish a reasonable basis for the cost of this project. As previously
noted, in the summer of 2014 NSF documented its analysis of the final LSST cost estimate in the
Cost Proposal Review Document, but that review provided little or no documentation to support
$267 million in subcontract costs and $79 million in contingency.
We have been urging NSF for the past four years to strengthen accountability of its high-dollar,
high-risk cooperative agreements for its large facility construction projects. NSF applies its
highest level of attention and scrutiny to determine the scientific merit of the projects it decides
to fund. It is imperative that NSF apply the same rigorous attention and scrutiny to its financial
management of these projects, prior to requesting NSB approval for award. The stakes are too
high for the Foundation to continue its current practice of requesting NSB approval and making
awards before it ensures that project costs are reasonable, are supported by adequate
documentation, and will use taxpayer dollars efficiently.
In light of the known and continuing deficiencies with AURA’s estimating practices and cost
proposals and the lingering uncertainties about the reasonableness, accuracy and currency of
many of the costs proposed for the LSST project, NSF should take immediate and strong action
to ensure that costs proposed for and incurred under the project comply with federal and NSF
requirements.
7
Recommendations
We recommend that NSF take appropriate action to ensure the reasonableness and integrity of
the costs proposed for and incurred under the LSST project, and modify the award accordingly.
Such actions would include:
• Ensuring that LSST total budgeted costs (especially subaward/subcontract and
contingency costs) are necessary, reasonable and adequately supported prior to
providing additional funding or finalizing LSST total project costs;
• Obtaining an audit of AURA’s estimating system and related internal controls, and a
post-award audit of AURA’s accounting system that includes the accounting and
related functions in Chile, and ensuring that all deficiencies are corrected; and
• Performing sufficient cost surveillance, including obtaining incurred cost submissions
and audits of the LSST project on an annual basis, to ensure that costs are reasonable,
allocable, and allowable.
We provided NSF with a draft copy of this memo on August 1, 2014 and on September 23, 2014.
NSF, in its responses on August 7th and September 29th provided comments to our draft memos
and submitted a CPRD on August 8th to document its review of the LSST. We fully considered
NSF’s responses and its CPRD in preparing this alert memo and made adjustments to our alert
memo where appropriate.
We conducted this inspection in accordance with the Quality Standards for Inspection and
Evaluation, January 2012. This memo is related to previously cited OIG reports (OIG Report
Nos. 10-1-012, 11-1-001,11-1-010,11-1-021,12-3-001, 12-1-008) and to OIG Alert Memo,
Report No. 12-6-001, and brings to NSF’s attention issues identified during that work that
warrant corrective action.
In accordance with OMB Circular A-50, NSF and OIG should agree on a corrective action plan
for resolution of all findings. Please provide us with you proposed corrective action plan within
sixty calendar days.
If you have any questions about this alert memo, please contact Jannifer Jenkins at (703) 292-
4996, or David Willems at (703) 292-4979.
cc: Fae Korsmo, OD
Martha Rubenstein, BFA
Jeffrey Lupis, DACS
Mary Santonastasso, DIAS
Bart Bridwell, DACS
Nigel Sharp, AST
Michael Van Woert, NSB
Ruth David, NSB
8
NSF's Management of Costs Proposed for the Large Synoptic Telescope
Published by the National Science Foundation, Office of Inspector General on 2014-09-30.
Below is a raw (and likely hideous) rendition of the original report. (PDF)