National Science Foundation • Office of Inspector General 4201 Wilson Boulevard, Suite I-1135, Arlington, Virginia 22230 MEMORANDUM DATE: September 30, 2014 TO: Dr. Pramod Khargonekar Assistant Director, Directorate for Engineering THRU: Allison Lerner /s/ Inspector General FROM: Dr. Brett M. Baker Assistant Inspector General for Audit SUBJECT: NSF OIG Alert Memo Report No. 14-3-002, NSF’s Management of Costs Proposed for the Large Synoptic Survey Telescope Construction Project The purpose of this memo is to document our continued concern about NSF’s management of large cooperative agreements. In light of our 2012 alert memo on this topic, 1 we have been carefully following NSF’s actions with respect to the Large Synoptic Survey Telescope (LSST) project, which is to be constructed in Chile. The LSST project was awarded at $467.7 million (including $27.5 million obligated for Fiscal Year 2014) in August of 2014, and will be conducted under a cooperative agreement with the Association of Universities for Research in Astronomy (AURA). As discussed in this alert memo, we have serious concerns that NSF does not have sufficient information to establish a reasonable basis for the cost of this project. In light of those concerns, it is critical that NSF take strong action to ensure robust oversight of the costs associated with this project as it proceeds. One of our recommendations in the alert memo on NSF’s management of cooperative agreements was that NSF obtain proposal and accounting system audits for high-risk cooperative agreements in excess of $50 million to ensure that cost estimates are fair and reasonable and that proposers’ accounting systems are adequate to bill the government properly. We also recommended that NSF obtain incurred cost submissions and audits of these large projects. Since the LSST project is the first construction project to be considered by the agency since we made those recommendations, we watched to see what NSF would do to ensure the adequacy of proposed costs. We found that an internal review completed at the Preliminary Design Review stage by NSF’s Cost Analysis and Audit Resolution (CAAR) branch in June 2013 could not independently verify costs for any of the 136 proposed expenditures sampled, including 1 NSF OIG Report No. 12-6-001, NSF’s Management of Cooperative Agreements, dated September 28, 2012. 1 approximately $145 million in direct materials, nearly $20 million for contingencies and more than $6 million in direct labor costs. CAAR reported that without further documentation, it was unable to determine if the methodology used to estimate the cost is appropriate, consistently applied, or reasonable. After this critical report, independent proposal and accounting system audits were clearly warranted to ensure the adequacy and proper accounting of the proposed costs. Instead of obtaining those, NSF had a contractor perform a “sufficiency review,” which is a less rigorous option in the GAO Cost Estimating and Assessment Guide, GAO-09-3SP, that would not look at information in sufficient detail to determine if the problems identified by CAAR had been remedied. Subsequently, NSF developed a Cost Proposal Review Document (CPRD) to provide more detail and follow-up on concerns raised by the CAAR review. However, the CPRD lacked documentation and analysis to demonstrate that NSF performed a full review of two of the most significant costs in the project’s proposed budget-- $ million in subcontracts/ subawards, which comprise percent of the proposed costs, and $79 million in contingency costs. These two elements—subcontracts and contingency-- comprise % of LSST’s total proposed costs, and lack of support for these costs constitutes a fundamental accountability risk. Given the concerns we had about the analysis of the project’s proposed costs, we reviewed the CPRD to assess the extent to which it addressed the need for incurred cost submissions and audits—tools which provide critical insight to both NSF management and NSF OIG after an award is made. The CPRD did not address the matter of incurred cost submissions, so we cannot determine the extent to which NSF will require preparation of these vital documents. With respect to incurred cost audits, the CPRD notes that, based on NSF’s assessment of the project’s risks, NSF determined that it would not require an incurred cost audit at the end of the first year of performance. Based on post-award monitoring and risk assessment, NSF will decide if incurred cost audits are needed at intermediate points during the agreement’s term; at minimum, NSF will require an incurred cost audit at the expiration of the award. In light of the risks detailed in this memo and the large amount of funding devoted to this project, we believe that annual incurred cost submissions and audits are clearly warranted for this project. Finally, as noted previously, the LSST project will be conducted under a cooperative agreement with AURA. For four years, audits have documented significant estimating deficiencies and concluded that AURA does not have an effective process for preparing adequate proposals. The most recent audit of an AURA proposal, completed in August 2014, concluded that AURA’s re- baselined cost proposal for the Advanced Technology Solar Telescope (newly renamed the Daniel K. Inouye Solar Telescope (DKIST), was so deficient the auditors could not affirm that the proposed costs were acceptable as a basis for a fair and reasonable price to the government. In light of the known and continuing deficiencies with AURA’s estimating practices and cost proposals and the lingering uncertainties about the reasonableness, accuracy, and currency of many of the costs proposed for the LSST project, NSF should take immediate and strong action to ensure that costs proposed for and incurred under the project comply with federal and NSF requirements. 2 Background For four years, the OIG has recommended that NSF strengthen accountability over its high- dollar, high-risk cooperative agreements for its large facility construction projects. These recommendations grew out of a large body of audit work examining awardees’ proposed costs for such projects. Audits of three of these projects—the Ocean Observatories Initiative, the Advanced Technology Solar Telescope, and the National Ecological Observatory Network questioned a total of $305 million in unallowable or unsupported costs (out of $1.1 billion in total costs for the three projects), $223 million of which was due to questioned amounts for contingencies. 2 Our September 2012 alert memo on NSF’s management of high-risk, high-dollar cooperative agreements outlined serious weaknesses in NSF’s management of these awards and recommended that NSF improve both its pre-and post-award management. Appropriate controls at the pre-award phase include audits of awardees’ proposed budgets and accounting systems to ensure cost estimates are reasonable and accounting systems are adequate to bill the government properly. A strong post-award process should include incurred costs submissions and audits to help ensure that unallowable costs are not charged to the government, and should include a requirement that awardees adequately track and identify contingency expenditures in their accounting systems. We recommended that, at a minimum, NSF implement such increased monitoring (obtaining updated cost estimates; audits of proposed budgets and determinations of accounting system adequacy at the pre-award stage; and incurred cost submissions and audits post-award) for cooperative agreements valued at over $50 million. In response, in August 2013, NSF proposed an alternate approach to strengthen its procedures for analysis of recipients’ cost proposals. Among other things, it indicated that it would undertake a minimum of one of the following actions: an independent audit; one of the actions in GAO’s Cost Estimating and Assessment Guide; or project the organization’s cost trends for elements of cost on the basis of current and historical information available to the Grants and Agreements Officer and to NSF or outside experts. While some of the actions NSF proposed appear to be sufficient for smaller, less risky projects, we did not find them all to be adequate for high-dollar, high-risk projects. Accordingly, during conversations with NSF management about their proposed response during the summer of 2013, we urged NSF to conduct the type of stepped-up actions we identified in the alert memo for cooperative agreements valued at over $200 million, or to suggest another threshold at which it would strengthen its process. NSF disagreed with the $200 million threshold and has not suggested a different one. The risk of NSF continuing its current practice is high: as of August 2013, it had 23 cooperative agreements worth over $50 million each and totaling over $4.2 billion. 2 Reports issued were Consortium of Ocean Leadership’s (COL) Ocean Observatories Initiative proposal in September 2010 and in March 2012 (OIG Report Nos. 10-1-012 and 12-3-001); Association of Universities for Research (AURA) in Astronomy’s Advance Technology Solar Telescope proposal and AURA’s accounting system in March 2011 (OIG 11-1-001 and 11-1-010); and National Ecological Observatories Network’s (NEON) proposal in September 2011 (OIG 11-1-021) and September 2012 (OIG 12-1-008). 3 Cost Estimates and the LSST Project NSF’s Internal Review of LSST Costs at the Preliminary Design Stage LSST was the first large facility project to make its way through the NSF approval process after we escalated our recommendations on the need for NSF to strengthen its processes and thus was an opportunity for NSF to demonstrate its commitment to accountability over hundreds of millions of dollars in proposed costs. In March of 2013, NSF asked staff in its CAAR division to conduct a cost analysis of the LSST project at the preliminary design stage. CAAR’s review identified serious problems with the LSST proposal. After sampling 136 transactions totaling 38 percent of the project’s costs, CAAR was unable to independently verify the costs requested for any of the sampled costs. In addition, through its sampling, CAAR also sought to gain more insight into the methodology used to construct the cost estimate. When asked for further justification of the costs, the organization often referred CAAR back to the established cost estimating plan or existing Work Breakdown Structure data. Without further documentation, CAAR was not able to determine if the methodology used to estimate the cost is appropriate, consistently applied, or reasonable. CAAR also identified many problematic costs in the estimate, including: • Salaries and wages: Median annual salaries for various positions were calculated based upon the rates of pay for various positions at participating institutions. NSF’s internal review found that most of the sampled salary expenditures could not be identified with the median rates of pay provided and that salary amounts requested far exceeded pay rates in supporting documentation provided to NSF. • Equipment and associated indirect costs: Vehicles were proposed for both LSST and AURA’s NOAO projects for use at the summit site, but the same supporting documentation was provided for all proposed expenditures. NSF could not determine whether the proposed costs represented duplicate requests. Additionally, indirect costs, which normally are not allowed to be charged against equipment, had been proposed for these purchases as had unnecessary amounts for contingency and risk. • Management fees: Funds were requested for several different fees that may be inconsistent with AURA’s current rate structure and/or NSF’s cooperative agreements. CAAR also identified problems with subcontracts and associated indirect costs, with the calculation and application of contingency costs and with the fringe benefit methodology, escalation costs, indirect costs, fringe benefit pools, and LSST labor burden rate. In addition, CAAR determined that the cost estimate for LSST was based on 2011 estimates. In May 2014, NSF informed us that AURA had not provided NSF with current vendor quotes to support the estimated costs. The use of unsupported estimates and lack of current quotes, which are critical to developing reliable cost estimates, is a serious and ongoing problem with cost proposals for large facility construction projects at NSF, as documented in audits of the initial proposed budget for the $298 million Advanced Technology Solar Telescope (another project run by AURA) and the $434 million National Ecological Observatory Network project. Significant deficiencies rendered the initial proposed budget for ATST unacceptable for audit and the auditors issued two inadequacy 4 memos, in March and October 2010 stating that direct material estimates were not current and that direct labor and indirect costs were insufficiently supported, among other things. In June 2012, NSF requested AURA to re-baseline the project with a current cost estimate. In the case of NEON, the flaws were so extensive that the auditors reached an adverse opinion, concluding that the proposal was not an adequate basis on which to negotiate a fair and reasonable price. Given the number of serious problems CAAR identified in the LSST proposal and the fact the project was projected to cost NSF close to half a billion dollars, the agency should have taken strong action to ensure that the issues identified by CAAR were addressed and that the final proposed costs were reasonable. In keeping with the 2012 alert memo’s recommendations, a proposal audit or independent cost estimate was clearly warranted. Booz Allen Hamilton Review of LSST Cost Issues Identified in NSF’s Review Instead of pursuing either of the options mentioned above, NSF contracted with Booz Allen Hamilton for a sufficiency review—one of the less rigorous options in GAO’s Guide, and one which would not examine the final proposal in sufficient detail to determine if the matters raised in CAAR’s review had been rectified. Although the Booz Allen Hamilton report is written at a much higher level than CAAR’s, in some areas it identified problems similar to those found in CAAR’s review. In particular, the Booz Allen review found that AURA’s estimating ground rules and assumptions only partially met requirements. It also concluded, in its assessment of the extent to which the cost estimate was well developed and traceable, that the estimate only partially met requirements. NSF’s Cost Proposal Review Document for LSST In the summer of 2014, NSF documented its analysis of the final LSST cost estimate in a Cost Proposal Review Document (CPRD). In light of the issues raised by CAAR, we examined the CPRD to determine the extent to which it addressed CAAR’s concerns. The CPRD provided additional detail about some issues, such as labor, equipment, and escalation rates, about which CAAR had raised concerns in its review. As noted previously, one of the most significant issues CAAR raised was its inability to find support for any of the 136 transactions it sampled. When we examined the CPRD to determine the extent to which this issue had been addressed, we found that the CPRD lacked documentation and analysis to demonstrate that NSF performed a full review of two of the most significant costs in the project’s proposed budget-- million in subcontracts (called subawards), which comprise percent of the proposed costs, and $79 million in contingency costs. These two elements—subcontracts and contingency-- comprise % of LSST’s total proposed costs, and lack of support for these costs constitutes a fundamental accountability risk. The CPRD review of subcontract costs was limited to approximately $ million of labor and $40 million of equipment, materials, and supplies. Even for the vast majority of those items, there was little or no documentation to evidence that proposed subcontract costs were supported by current vendor quotes. In addition, NSF has not reviewed or tested nearly $ million in additional subcontract costs and thus has no visibility over these costs. As a result, the agency has limited insight into the makeup of these costs and little, if any, assurance that they are reasonable. 5 With respect to the $79 million in contingency, although the CAAR review identified problems with the calculation and application of contingency costs, there was no evidence in the CPRD review that AURA had provided documentation to support the amounts proposed for contingencies. NSF acknowledged the lack of support for the full amount of the contingencies proposed and has only authorized AURA to use $6.1 million in contingencies until further documentation is provided, further NSF review is performed, and the award is modified accordingly. In light of these limitations in its analysis, NSF does not have a sufficient basis for concluding that almost three-fourths of the amounts budgeted for the LSST project were reasonable and adequately supported. In light of the questions about the costs proposed for the project, we reviewed the CPRD to assess the extent to which it addressed the need for incurred cost submissions and audits—tools which provide critical insight to both NSF management and NSF OIG after an award is made. The CPRD was silent on the matter of incurred cost submissions, so we cannot determine the extent to which NSF will require preparation of these documents, which are vital for proper cost monitoring and for the timely initiation of incurred cost audits. With respect to incurred cost audits, the CPRD notes that NSF, based on its assessment of the project’s risks, determined not to request an incurred cost audit subsequent to the completion of the first year of performance. NSF indicated that, at a minimum, an incurred cost audit will be accomplished subsequent to award expiration, and that NSF would monitor various matters (including the outcome of single and project-specific audits) to determine if incurred cost auditing is needed at any point prior to award expiration. In light of the risks detailed in this memo and the large amount of funding devoted to this project, we believe that annual incurred cost submissions and audits are clearly warranted. Serious Flaws in AURA’s Cost Proposals As noted previously, NSF will conduct the LSST project under a cooperative agreement with the Association of Universities for Research in Astronomy (AURA). For four years, our audits have repeatedly documented significant estimating deficiencies and concluded that AURA does not have an effective process for preparing adequate proposals. A 2011 accounting system and estimating practices audit of AURA identified eight significant deficiencies in the design of AURA’s accounting system and in its estimating practices used on NSF awards. 3 A subsequent preaward accounting system follow-up audit, performed in 2013, found that AURA had developed procedures to resolve the accounting system deficiencies. However, there has been no post-award accounting system audit or estimating system audit performed to verify that the procedures have been satisfactorily implemented to ensure that 1) the accounting system is adequate for accumulating and billing costs to the government and 2) estimating deficiencies have been corrected and will result in current, accurate, and complete cost proposals and annual program plans. 3 NSF OIG Audit Report No. OIG-11-1-010, Audit of Association of Universities for Research in Astronomy, Inc.’s/National Optical Astronomy Observatories’ (NOAO) Accounting System and Proposal Estimating Practices, dated March 31, 2011. 6 The most recent audit of AURA’s $344.1 million cost proposal for the Advanced Solar Technology Telescope (i.e., DKIST) found that the majority of costs in the proposed budget were not supported by current, accurate, and complete data. The auditors also found that AURA did not use actual costs in the re-baseline of the proposal (even though actual costs were available for 2010-2013) and that AURA included contingency costs that were explicitly unallowable per OMB regulations. The auditors disclaimed an opinion on the proposal and could not affirm that the proposed costs were acceptable as a basis for a fair and reasonable price. We will be making recommendations addressing these concerns in our transmittal of the re-baselined DKIST audit. In light of the known deficiencies in AURA’s estimating practices and the serious questions raised by NSF’s internal review of the LSST costs, it is important to ensure that AURA has in place adequate accounting and estimating practices to manage this nearly $500 million project. AURA’s estimating deficiencies greatly heighten the risk associated with the project and makes it even more important to have strong visibility into and controls over costs. Conclusion NSF management received the results of CAAR’s review in June 2013. It had an opportunity at that time to ensure a thorough review of project costs was conducted in a timely fashion. Instead, it chose to proceed with a review that was not sufficiently in-depth to ensure that the problems CAAR identified were rectified. Consequently, in May 2014 (the period in which the National Science Board (NSB) approval to make the award was sought) we questioned whether NSF had sufficient information to establish a reasonable basis for the cost of this project. As previously noted, in the summer of 2014 NSF documented its analysis of the final LSST cost estimate in the Cost Proposal Review Document, but that review provided little or no documentation to support $267 million in subcontract costs and $79 million in contingency. We have been urging NSF for the past four years to strengthen accountability of its high-dollar, high-risk cooperative agreements for its large facility construction projects. NSF applies its highest level of attention and scrutiny to determine the scientific merit of the projects it decides to fund. It is imperative that NSF apply the same rigorous attention and scrutiny to its financial management of these projects, prior to requesting NSB approval for award. The stakes are too high for the Foundation to continue its current practice of requesting NSB approval and making awards before it ensures that project costs are reasonable, are supported by adequate documentation, and will use taxpayer dollars efficiently. In light of the known and continuing deficiencies with AURA’s estimating practices and cost proposals and the lingering uncertainties about the reasonableness, accuracy and currency of many of the costs proposed for the LSST project, NSF should take immediate and strong action to ensure that costs proposed for and incurred under the project comply with federal and NSF requirements. 7 Recommendations We recommend that NSF take appropriate action to ensure the reasonableness and integrity of the costs proposed for and incurred under the LSST project, and modify the award accordingly. Such actions would include: • Ensuring that LSST total budgeted costs (especially subaward/subcontract and contingency costs) are necessary, reasonable and adequately supported prior to providing additional funding or finalizing LSST total project costs; • Obtaining an audit of AURA’s estimating system and related internal controls, and a post-award audit of AURA’s accounting system that includes the accounting and related functions in Chile, and ensuring that all deficiencies are corrected; and • Performing sufficient cost surveillance, including obtaining incurred cost submissions and audits of the LSST project on an annual basis, to ensure that costs are reasonable, allocable, and allowable. We provided NSF with a draft copy of this memo on August 1, 2014 and on September 23, 2014. NSF, in its responses on August 7th and September 29th provided comments to our draft memos and submitted a CPRD on August 8th to document its review of the LSST. We fully considered NSF’s responses and its CPRD in preparing this alert memo and made adjustments to our alert memo where appropriate. We conducted this inspection in accordance with the Quality Standards for Inspection and Evaluation, January 2012. This memo is related to previously cited OIG reports (OIG Report Nos. 10-1-012, 11-1-001,11-1-010,11-1-021,12-3-001, 12-1-008) and to OIG Alert Memo, Report No. 12-6-001, and brings to NSF’s attention issues identified during that work that warrant corrective action. In accordance with OMB Circular A-50, NSF and OIG should agree on a corrective action plan for resolution of all findings. Please provide us with you proposed corrective action plan within sixty calendar days. If you have any questions about this alert memo, please contact Jannifer Jenkins at (703) 292- 4996, or David Willems at (703) 292-4979. cc: Fae Korsmo, OD Martha Rubenstein, BFA Jeffrey Lupis, DACS Mary Santonastasso, DIAS Bart Bridwell, DACS Nigel Sharp, AST Michael Van Woert, NSB Ruth David, NSB 8
NSF's Management of Costs Proposed for the Large Synoptic Telescope
Published by the National Science Foundation, Office of Inspector General on 2014-09-30.
Below is a raw (and likely hideous) rendition of the original report. (PDF)