National Science Foundation • Office of Inspector General
4201 Wilson Boulevard, Arlington, Virginia 22230
MEMORANDUM
DATE: March 15, 2013
TO: Dr. Cora Marrett
Deputy Director
National Science Foundation
Martha P. Rubenstein
Chief Financial Officer
FROM: Dr. Brett M. Baker
Assistant Inspector General for Audit
SUBJECT: Audit of NSF’s FY 2012 Compliance with the Improper Payments Elimination and
Recovery Act, Report No. 13-2-007
This memo transmits Cotton and Company’s report for the audit of NSF’s compliance with the
Improper Payments Elimination and Recovery Act of 2010 (IPERA). The audit objective was to
review the improper payment reporting in NSF’s FY 2012 Annual Financial Report (AFR), and
accompanying materials, to determine whether the agency met the Office of Management and
Budget (OMB)’s criteria for compliance with IPERA (Public Law 111-204).
Results of Audit
The auditors found that NSF partially complied with the reporting requirements of IPERA in the
FY 2012 Annual Financial Report (AFR). Specifically, the auditors found that NSF did not fully
comply with 4 of the 7 IPERA reporting requirements. The auditors also identified issues related
to NSF’s risk assessment methodology and improper payment sample testing.
The auditors recommended that NSF take appropriate action to comply with the reporting
requirements of IPERA, to include improving its internal controls and reviewing procedures over
the improper payment risk assessment, sample testing methodology, and reporting details in
future AFRs. NSF responded in general to the findings and recommendations, but did not
specifically comment on each recommendation. Instead, NSF stated that it will consider
alternative procedures as part of its Corrective Action Plan to address the recommendations.
NSF’s response is included in its entirety in Appendix A.
Corrective Action Plans
To comply with OMB Circular A-50 requirements for audit followup, please provide us your
written corrective action plan to address the report recommendations within 60 calendar days.
This corrective action plan should detail specific actions and milestone dates.
In addition, OMB Memorandum M-11-16 instructs agencies that are not fully compliant with
IPERA to submit a plan to the Senate Homeland Security and Government Affairs Committee
and the House Committee on Oversight and Governmental Reform describing the actions that
the agency will take to become compliant. The plan must be submitted within 90 days of the date
of this final report, and include the following details:
• Measurable milestones to be accomplished in order to achieve compliance for each
program or activity;
• The designation of a senior agency official who shall be accountable for the progress of
the agency in coming into compliance for each program or activity; and
• The establishment of an accountability mechanism, such as a performance agreement,
with appropriate incentives and consequences tied to the success of the senior agency
official in leading agency efforts to achieve compliance for each program and activity.
OIG Oversight of Audit
To fulfill our responsibilities under Government Auditing Standards, the Office of Inspector
General:
• Reviewed the Cotton and Company team’s approach and planning of the audit;
• Evaluated the qualifications and independence of the auditors;
• Monitored the progress of the audit at key points;
• Coordinated periodic meetings with the Cotton and Company team and NSF officials,
as necessary, to discuss audit progress, findings, and recommendations;
• Reviewed the audit report, prepared by the Cotton and Company team to ensure
compliance with Government Auditing Standards; and
• Coordinated issuance of the audit report.
Cotton and Company is responsible for the attached auditor’s report on NSF’s compliance with
IPERA and the conclusions expressed in the report. We do not express any opinion on the
conclusions presented in the Cotton and Company team’s audit report.
We thank your staff for the assistance that was extended to our auditors during this audit. If you
have any questions regarding this report, please contact Thomas Moschetto at 703-292-7398.
Attachment
cc: Subra Suresh Shirl Ruffin
Allison Lerner G. P. Peterson
Rafael Cotto Thomas Moschetto
John Lynskey Cliff Gabriel
Rick Noll Susan Carnohan
Laura Hansen-Rainey Karen Scott
Catherine Walters
National Science Foundation
FY 2012 Improper Payments Elimination and
Recovery Act (IPERA)
Performance Audit
Final Report
March 12, 2013
Office of Inspector General
National Science Foundation
Arlington, Virginia
Cotton & Company LLP performed an audit of the National Science Foundation (NSF)’s fiscal year (FY)
2012 Improper Payment Reporting in the FY 2012 Agency Financial Report. We evaluated NSF’s
performance in complying with the Improper Payments Elimination and Recovery Act of 2010 (IPERA),
Public Law 111-204.
This performance audit, performed under Contract No. D13PD00160, was designed to meet the
objective identified in this report.
We conducted this performance audit in accordance with Government Auditing Standards, issued by the
Comptroller General of the United States. We communicated the results of our performance audit and
the related findings and recommendations to the NSF Office of Inspector General.
COTTON & COMPANY LLP
CPA, CFE
Partner
March 12, 2013
Alexandria, Virginia
CONTENTS
EXECUTIVE SUMMARY ........................................................................................................................................... 1
COMPLIANCE ISSUES .......................................................................................................................................................2
OTHER MATTERS TO REPORT............................................................................................................................................2
PERFORMANCE AUDIT REPORT.............................................................................................................................. 3
I. BACKGROUND.......................................................................................................................................................3
II. OBJECTIVE, SCOPE, AND METHODOLOGY ...................................................................................................................3
Objective ...........................................................................................................................................................3
Scope .................................................................................................................................................................4
Methodology .....................................................................................................................................................4
III. RESULTS ..............................................................................................................................................................5
Compliance Findings .........................................................................................................................................6
Other Matters to Report .................................................................................................................................11
Other Matters: NSF Management Updated Representation of IPERA Process ...............................................14
APPENDIX: MANAGEMENT COMMENTS .............................................................................................................. 17
Page | i
EXECUTIVE SUMMARY
This report presents the results of the work Cotton & Company LLP conducted to address performance
audit objectives related to the National Science Foundation (NSF)’s fiscal year (FY) 2012 improper
payment reporting in the FY 2012 Agency Financial Report (AFR). Our work was performed during the
period from January 9 through February 27, 2013. Our results are as of February 27, 2013.
We conducted this performance audit in accordance with generally accepted government auditing
standards (GAGAS). Those standards require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings based on the audit objectives. We
believe that the evidence obtained provides a reasonable basis for our findings based on the audit
objectives.
The objective of this performance audit was to review the improper payment reporting in NSF’s FY 2012
AFR and accompanying materials to determine whether the agency met the Office of Management and
Budget (OMB)’s criteria for compliance with the Improper Payments Elimination and Recovery Act of
2010 (IPERA), Public Law 111-204. As part of our compliance review, we evaluated the accuracy and
completeness of NSF’s reporting and its efforts to reduce and recapture improper payments. During the
course of the audit, NSF management provided an updated representation of the IPERA process that
differed from the information reported in the FY 2012 AFR. While OMB guidance limits the objective and
scope of our audit to the improper payment reporting in NSF’s FY 2012 AFR and accompanying
materials, we believe that this updated representation should be considered in NSF’s future
improvement efforts regarding improper payments. We therefore evaluated this updated information
separately and have included a discussion in the Other Matters to Report section of this report.
OMB Memorandum M-11-16, Issuance of Revised Parts I and II to Appendix C of OMB Circular A-123,
states that compliance with IPERA means that the agency has:
• Published a Performance and Accountability Report (PAR) or AFR for the most recent fiscal year
and posted that report and any accompanying materials required by OMB on the agency
website.
• Conducted a program specific risk assessment for each program or activity that conforms with
Section 3321 of Title 31 U.S.C. (if required).
• Published improper payment estimates for all programs and activities identified as susceptible
to significant improper payments under its risk assessment (if required).
• Published programmatic corrective action plans in the PAR or AFR (if required).
• Published, and has met, annual reduction targets for each program assessed to be at risk and
measured for improper payments.
• Reported a gross improper payment rate of less than 10 percent for each program and activity
for which an improper payment estimate was obtained and published in the PAR or AFR.
• Reported information on its efforts to recapture improper payments.
We found that NSF partially complied with the reporting requirements of IPERA in the FY 2012 AFR.
We identified the following compliance issues and other matters:
Page | 1
COMPLIANCE ISSUES
• NSF did not use a statistically valid method to calculate the estimated improper payment
rate published in the FY 2012 AFR.
• NSF did not publish separate improper payment rates for each program and activity for
which an improper payment estimate was obtained and published in the FY 2012 AFR.
• NSF did not properly report on its efforts to recapture improper payments in the FY 2012
AFR.
• NSF reported an FY 2012 improper payment reduction target in FY 2009. When we
compared this target to the FY 2012 improper payment estimate, we could not determine
whether NSF met the reduction target due to issues concerning the accuracy and
completeness of the FY 2012 improper payment estimate.
OTHER MATTERS TO REPORT
Our report also includes the following other matters relating to our evaluation of the completeness
and accuracy of NSF’s IPERA reporting details, its performance in reducing and recapturing improper
payments, and information represented by NSF during the course of the audit:
• NSF’s statistical sampling process for computing its improper payment estimate is based on
Federal Financial Report (FFR) expenditures and not the population of program outlays that
were identified as susceptible to significant risk of improper payments.
• NSF failed to develop a comprehensive testing methodology and was inconsistent in
applying its limited methodology to the test samples.
• NSF provided an updated representation that the FY 2012 IPERA procedures, including
statistical sampling, are a systematic method to satisfy the requirement to review all
programs and activities to identify those that are susceptible to significant improper
payments. Due to the identified issues and inconsistencies in NSF’s IPERA procedures,
management may be unable to rely on the results of this method.
We discussed the contents of this report in an exit conference with NSF management on February 27,
2013. In its response to the draft report (see Appendix), NSF management concurred with some of the
report’s five recommendations but did not specify which would be implemented. We reviewed NSF
management’s detailed response and determined that the contents of this report, including the findings
and recommendations, are supported based on our audit procedures.
Page | 2
NATIONAL SCIENCE FOUNDATION
IMPROPER PAYMENTS ELIMINATION AND RECOVERY ACT (IPERA)
PERFORMANCE AUDIT REPORT
FY 2012
I. BACKGROUND
The Improper Payments Elimination and Recovery Act of 2010 (IPERA), Public Law (PL) 111-204, dated
July 22, 2010, amended the Improper Payments Information Act of 2002 (IPIA), PL 107-300. IPERA
requires agencies to periodically review and identify programs and activities that may be susceptible to
significant improper payments, and to report on their actions to reduce and recover improper
payments. As directed under IPERA, the Office of Management and Budget (OMB) issued Memorandum
M-11-16, Issuance of Revised Parts I and II to Appendix C of OMB Circular A-123, on April 14, 2011. This
memorandum provides agencies with detailed guidance on the implementation of IPERA. Part II, Section
A, Subsection 4 of OMB Memorandum M-11-16 expands on the Inspector General (IG)’s responsibilities
as outlined in IPERA, including:
• Reviewing agency improper payment reporting in the agency’s annual Performance and
Accountability Report (PAR) or Agency Financial Report (AFR) and accompanying materials.
• Determining if the agency is in compliance with IPERA.
The IG is also directed to evaluate the accuracy and completeness of agency improper payment
reporting, as well as the agency’s performance in reducing and recapturing improper payments.
The Objective, Scope, and Methodology section of this report has been designed to address the IG’s
responsibilities as described in Part II, Section A, Subsection 4 of OMB Memorandum M-11-16.
II. OBJECTIVE, SCOPE, AND METHODOLOGY
Objective
The objective of our performance audit was to determine if the National Science Foundation (NSF) met
OMB’s criteria for compliance with IPERA as described in OMB Memorandum M-11-16, including:
• Publishing a PAR or AFR for the most recent fiscal year and posting that report, with
accompanying materials, to the agency website.
• Conducting a program specific risk assessment for each program or activity that conforms with
Section 3321 of Title 31 U.S.C. (if required).
• Publishing improper payment estimates for all programs and activities identified as susceptible
to significant improper payments under the agency’s risk assessment (if required).
• Publishing programmatic corrective action plans in the PAR or AFR (if required).
• Publishing, and meeting, annual reduction targets for each program assessed to be at risk and
measured for improper payments.
• Reporting a gross improper payment rate of less than 10 percent for each program and activity
for which an improper payment estimate was obtained and published in the PAR or AFR.
• Reporting information on the agency’s efforts to recapture improper payments.
Page | 3
We also evaluated the accuracy and completeness of agency improper payment reporting, and the
agency’s performance in reducing and recapturing improper payments.
Scope
As established in OMB Memorandum M-11-16, the scope of this performance audit included the
improper payment and reporting details in NSF’s FY 2012 AFR, Appendix 2. OMB approved a three-year
reporting cycle for NSF. In accordance with this cycle, NSF reported improper payments in its FY 2012
AFR. We designed procedures to gain an understanding of the risk assessment that NSF performed to
identify programs susceptible to significant risk of improper payments, as well as the statistical sampling
process that the agency performed to calculate its improper payment estimate. Our procedures
included having a subject matter expert (a statistician) evaluate the statistical validity of the improper
payment estimate.
We also designed procedures to evaluate the completeness and accuracy of the information reported in
Appendix 2, including re-performing testing of 45 randomly selected sample items, as well as one
additional sample that NSF identified as an improper payment, that NSF had tested in determining its
improper payment estimate. As discussed in the findings, we were unable to re-perform NSF’s testing
procedures; however, we evaluated the reasonableness of the conclusions and sufficiency of the
documentation provided to support NSF’s testing results.
In addition, we designed procedures to evaluate the agency’s performance in reducing and recapturing
improper payments.
Methodology
To verify compliance, evaluate completeness and accuracy, and evaluate the agency’s performance in
reducing and recapturing improper payments, we:
• Reviewed NSF’s FY 2012 AFR and confirmed that the report and any accompanying materials
were posted to the agency website.
• Reviewed NSF’s FY 2012 AFR and confirmed whether the presentation was in accordance with
the form and content requirements outlined in OMB Circular No. A-136, Financial Reporting
Requirements.
• Evaluated the completeness and accuracy of the IPERA reporting details presented in NSF’s FY
2012 AFR.
• Confirmed whether NSF conducted a program specific risk assessment and evaluated the results
of the assessment.
• Confirmed whether NSF published improper payment rate and dollar estimates for all programs
and activities identified as susceptible to significant improper payments under the agency’s risk
assessment.
• Evaluated the statistical sampling process that NSF used to obtain the improper payment rate
estimates published in its FY 2012 AFR.
• Evaluated the reasonableness of NSF’s conclusions and the sufficiency of documentation
supporting the results of testing procedures that NSF performed on sample items as part of the
statistical sampling process.
• Confirmed whether NSF was required to publish corrective action plans in its FY 2012 AFR.
Page | 4
• Confirmed whether NSF has published, and met, improper payment reduction targets for each
program assessed and measured to be at risk for improper payments.
• Evaluated whether NSF reported a gross improper payment rate of less than 10 percent for each
program and activity for which an improper payment estimate was obtained and published in
the AFR.
• Confirmed whether NSF reported on its efforts to recapture improper payments.
• Evaluated other activities performed by NSF to reduce and recapture improper payments.
In carrying out this methodology, we primarily applied audit techniques such as inquiry and observation
to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings related to the
audit objectives.
III. RESULTS
Based on the results of the audit performed, NSF did not fully meet four of the applicable OMB criteria
for compliance noted in the audit objectives. The following table identifies each criterion and states
whether NSF met the criterion. The five findings discussed below provide support for our conclusions.
OMB Criteria for IPERA Results Explanation of Results
Compliance
Published a PAR or AFR for the
most recent fiscal year and posted
that report and any accompanying Met N/A
materials required by OMB on the
agency website
Conducted a program specific risk
assessment for each program or
activity that conforms with Met N/A
Section 3321 of Title 31 U.S.C. (if
required)
Published improper payment NSF published improper payment estimates for
estimates for all programs and all programs and activities identified as
activities identified as susceptible susceptible to improper payments under the
Partially Met
to significant improper payments agency’s risk assessment; however, NSF did not
under the agency’s risk separately report these estimates for each
assessment (if required) program, as required. Refer to Finding 1.
Reporting of corrective actions is only required
for agencies reporting improper payments that
Published programmatic
Not exceed 2.5 percent of program outlays and $10
corrective action plans in the PAR
Applicable million, or $100 million. NSF’s estimated
or AFR (if required)
improper payments were below this level, and as
such, this criterion is not applicable.
Published, and has met, annual NSF published a combined annual reduction
reduction targets for each target for programs assessed to be at risk and
Partially Met
program assessed to be at risk and measured for improper payments; however, we
measured for improper payments could not determine whether NSF met the
Page | 5
OMB Criteria for IPERA Results Explanation of Results
Compliance
reduction target as we were unable to evaluate
the accuracy and completeness of the reported
improper payment estimate. Refer to Findings 2
and 5.
While NSF did report a combined improper
Reported a gross improper payment rate of less than 10 percent for the
payment rate of less than 10 programs for which an improper payment
percent for each program and estimate was obtained and published in the AFR,
Partially Met
activity for which an improper the estimate was a net value, and its accuracy
payment estimate was obtained and completeness could not be determined due
and published in the PAR or AFR to the lack of a statistically valid sampling
process. Refer to Findings 2 and 5.
While NSF did report information on the agency’s
efforts to recapture improper payments, the
Reported information on the agency did not include the specific elements
agency’s efforts to recapture Partially Met required by OMB Memorandum M-11-16 to
improper payments support its determination that a payment-
recapture audit program would not be cost
effective. Refer to Finding 3.
Compliance Findings
Our analysis identified four criteria for which NSF did not fully meet the IPERA compliance requirements
as outlined in OMB Memorandum M-11-16, noted as partial compliance in the table above. The specific
findings are discussed in detail below.
Finding 1: Improper Payment Rates Are Not Published for Each Program in Accordance with OMB A-
136 Form and Content Requirements
NSF did not publish improper payment rates for each program and activity for which an improper
payment estimate was obtained and published in the AFR, in accordance with the form and content
requirements outlined in OMB Circular No. A-136. IPERA requires NSF to publish improper payment
estimates for all programs and activities identified as susceptible to significant improper payments
under its risk assessment.
In Section IV of Appendix 2 to the FY 2012 AFR, NSF management presents a table entitled “Improper
Payment Reduction Outlook, FY 2012 - FY 2015 for R&RA and EHR Programs.” This table shows
combined FY 2012 appropriation outlays for the two programs identified (Research and Related
Activities (R&RA) and Education and Human Resources (EHR)), a combined FY 2012 improper payment
percentage, and a combined FY 2012 improper payment dollar amount. The FY 2012 improper payment
dollar amount is presented as a net amount based on a net population of Federal Financial Report (FFR)
expenditures. The table also displays estimates for these three items for FYs 2013-2015.
Page | 6
The table does not include the following required items:
• Separately identified outlays, improper payment percentages, and improper payment amounts
for each program or activity (i.e., R&RA and EHR) as identified in the FY 2011 Outlay Risk
Assessment.
• Prior-year (PY) information presented as a comparison for the current-year (i.e., FY 2012)
estimate.
• A gross improper payment amount and a list of the total overpayments and total
underpayments that make up the gross improper payment amount.
In addition, the table reports FY 2012 outlays of $5,769 million, which could not be tied to supporting
documentation.
OMB Memorandum M-11-16, Part I provides guidance to clarify the reporting requirements outlined in
PL 111-204. OMB Memorandum M-11-16 Part I, Section 7, Step 4 states:
a. Reporting. Agencies shall report to the President and Congress (through their annual PARs or
AFRs in the format required by OMB Circular A-136 for IPIA reporting) an estimate of the
annual amount of improper payments for all programs and activities determined to be
susceptible to significant improper payments under Step 1, regardless of the dollar amount
of the estimate, as further explained below.
OMB Circular No. A-136, Section II.5.8, Subsection IV, provides the following guidance with respect to
reporting improper payment estimates in the AFR:
a. The table that follows (Table 1) is required for each agency that has programs or activities
that are susceptible to significant improper payments. Agencies must include the following
information:
i. All risk-susceptible programs must be listed in this table whether or not an error
measurement is being reported;
ii. Where no measurement is provided, the agency should indicate the date by which a
measurement is expected;
iii. If the Current Year (CY) is the baseline measurement year, and there is no Previous
Year (PY) information to report, indicate by either note or by “n/a” in the PY column;
iv. If any of the dollar amount(s) included in the estimate correspond to newly
established measurement components in addition to previously established
measurement components, separate the two amounts to the extent possible;
v. Agencies are expected to report on CY activity, and if not feasible, then PY activity is
acceptable if approved by OMB. Agencies should include future outlay and improper
payment estimates for CY+1, +2, and +3 (future year outlay estimates should match
the outlay estimates for those years as reported in the most recent President’s
Budget).
b. Agencies should include the gross estimate of the annual amount of improper payments (i.e.,
overpayments plus underpayments) and should list the total overpayments and
underpayments that make up the current year amount. In addition, agencies are allowed to
calculate and report a second estimate that is a net total of both over and under payments
Page | 7
(i.e., overpayments minus underpayments). The net estimate is an additional option only,
and cannot be used as a substitute for the gross estimate. Agencies may include the net
estimate in Table 1 or in a separate table.
NSF management did not properly implement the guidelines outlined in IPERA, OMB Memorandum M-
11-16, and OMB Circular No. A-136. In addition, NSF management did not perform a detailed review of
the information reported to ensure that it was complete and accurate. Furthermore, NSF management
relied on the fact that OMB did not provide comments specific to the improper payment reporting
details in the FY 2012 AFR to infer that its reporting was consistent with OMB guidance.
NSF did not fully comply with IPERA requirements, as the agency did not separately determine and
report a gross improper payment rate for each program and activity identified as susceptible to
improper payments. In addition, the form and content of the information reported in Appendix 2 to the
FY 2012 AFR are not in full compliance with the requirements of OMB Circular No. A-136. Moreover, the
reported FY 2012 outlay amount does not tie to supporting records and may be incorrect.
We recommend that the NSF Chief Financial Officer take appropriate action to improve NSF compliance
with IPERA requirements for AFR reporting. Specifically, the agency should:
1.1 Fully report the elements required by IPERA, as outlined in OMB Circular No. A-136.
1.2 Implement internal controls and review procedures over the IPERA Reporting Details section
of Appendix 2 to the AFR to ensure that the required elements per OMB Circular No. A-136
are reported and the information is complete and accurate.
Finding 2: NSF Does Not Have a Statistically Valid Methodology for Estimating Improper Payments
NSF has not prepared a statistically valid estimate of improper payments, as required by IPERA. In
Appendix 2 to the FY 2012 AFR, Section II describes the statistical sampling process that NSF conducted
to estimate the improper payment rate for programs or activities identified as susceptible to improper
payments. Section IV presents the estimated improper payment percentage and dollar amount for the
R&RA and EHR programs.
The statistical sampling process that NSF management describes in Section II does not result in a
statistically valid estimate of improper payments due to the following:
• The formula that NSF used to determine the sample size is only appropriate for an attribute
sample, which would be used to determine whether an agency’s payments are correct or
incorrect. This approach yields an improper payment rate estimate expressed as a percentage of
the number of payments that were incorrect. It does not yield an improper payment dollar
estimate.
• The sample was selected using a multistage sample selection process. The sample size
determined for the attribute sample was not appropriate for the multistage sample selection
used.
• NSF projected the error rate to a universe of FFR expenditures that does not reconcile to the
universe of FFR expenditures from which the sample size was determined and sample items
were selected. The projection of the estimated error is also not consistent with the multistage
sample selection. Since NSF selected a multistage sample, two standard deviations must be
Page | 8
calculated: one at the sub-transaction level and the other at the quarter-FFR expenditure level.
The two should then be combined based on the selection probabilities.
• In assessing whether the results achieved the desired precision, NSF compared the sample error
rate to the stated precision. This comparison is not appropriate, as the error rate and the
precision are two independent measures.
IPERA Section 2, Subsection (b), Paragraphs (1) and (2) states the following with regard to the
requirements for the estimation of improper payments:
(b) Estimation of Improper Payments - With respect to each program and activity identified
under subsection (a), the head of the relevant agency shall –
(1) produce a statistically valid estimate, or an estimate that is otherwise appropriate
using a methodology approved by the Director of the Office of Management and Budget,
of the improper payments made by each program and activity; and
(2) include those estimates in the accompanying materials to the annual financial
statement of the agency required under section 3515 of title 31, United States Code, or
similar provision of law and applicable guidance of the Office of Management and
Budget.
NSF had originally planned to apply an attribute-level sampling methodology in determining its improper
payment error estimate, but instead it used a multistage sampling approach. NSF also did not perform
appropriate statistical analysis to ensure that it had achieved the required level of precision once the
testing was completed, as management relied on correspondence with OMB in 2005 as approval of the
statistical estimation methodology and therefore did not perform an adequate review to ensure that the
method was statistically valid.
The estimated improper payment rate and dollar amount presented in Section IV of Appendix 2 were
produced based on a statistical sampling methodology that was not statistically valid and therefore does
not comply with IPERA requirements. In addition, the estimated improper payment rate and dollar
amount may not be complete and accurate. As a result, we are unable to determine whether NSF met
previously reported reduction targets.
By presenting an improper payment estimate in Appendix 2, Section IV, NSF management is erroneously
implying to the user of the AFR that the estimate of improper payments for the two programs identified
as susceptible to improper payments is statistically valid under IPERA.
We recommend that the NSF Chief Financial Officer take appropriate action to improve NSF compliance
with IPERA requirements. Specifically, the agency should:
2.0 Utilize a statistically valid sampling methodology to develop improper payment estimates if,
during the course of its risk assessment process, it identifies programs susceptible to
significant risk of improper payments and is required to report an improper payment
estimate in the AFR.
Finding 3: NSF Did Not Properly Report on Its Efforts to Recapture Improper Payments
NSF did not properly report on its efforts to recapture improper payments in the FY 2012 AFR. Appendix
2, Section V of the FY 2012 AFR provides the following information:
Page | 9
In compliance with IPERA and OMB Circular A-123, Management’s Responsibility for Internal
Control, NSF evaluated its grants and contracts oversight processes. NSF determined that it was
not cost effective to establish a formal Recapture Audit Program. On January 14, 2011, NSF
submitted its plan for meeting the requirements of recapture audits to OMB and NSF OIG. The
plan included the reasons for a cost effective determination. On September 29, 2011, NSF sent a
follow-up to OMB reiterating its determination. NSF is leveraging its existing oversight policies
and procedures to meet the intent of OMB's requirements on improper payments.
NSF’s FY 2012 AFR does not include a list of programs and activities for which NSF determined that
conducting a payment recapture audit program would not be cost effective. NSF also did not provide
specific language in the FY 2012 AFR describing the justifications and analysis that it used to make this
determination.
OMB Memorandum M-11-16 Part I, Section B, Subsection 6 provides the following requirements for
reporting on efforts to recapture improper payments:
If an agency determines that it would be unable to conduct a cost-effective payment recapture
audit program for certain programs and activities that expend more than $1 million, then it must
notify OMB and the agency's Inspector General of this decision and include any analysis used by
the agency to reach this decision. OMB may review these materials and determine that the
agency should conduct a payment recapture audit program to review these programs and
activities. In addition, the agency shall report in its annual PAR or AFR:
1) a list of programs and activities where it has determined conducting a payment recapture
audit program would not be cost-effective; and
2) a description of the justifications and analysis that it used to determine that conducting a
payment recapture audit program for these programs and activities was not cost-effective.
NSF management did not properly implement the guidelines outlined in OMB Memorandum M-11-16.
Furthermore, NSF management relied on the fact that OMB did not provide comments specific to the
improper payment reporting details in the FY 2012 AFR to infer that its reporting on efforts to recapture
improper payments was consistent with OMB guidance.
NSF is not in full compliance with IPERA requirements as outlined in OMB Memorandum M-11-16. The
information reported in Section V of Appendix 2 to the FY 2012 AFR does not provide sufficient detail for
the users of the AFR to understand the justifications and analysis that NSF used in determining that it
was not cost effective to conduct a payment recapture audit program, or to which specific programs this
determination applies.
We recommend that the NSF Chief Financial Officer take appropriate action to improve NSF compliance
with IPERA requirements for AFR reporting. Specifically, the agency should:
1.1 Fully report the elements required by OMB Memorandum M-11-16 for supporting the
determination that a payment recapture audit program is not cost effective.
1.2 Implement internal controls and review procedures over the IPERA Reporting Details section of
Appendix 2 to the AFR to ensure that the required elements per OMB Memorandum M-11-16
are reported.
Page | 10
Other Matters to Report
As part of our review to determine NSF’s compliance with IPERA, we evaluated the accuracy and
completeness of reporting details and evaluated the agency’s efforts to reduce and recapture improper
payments. The findings noted below are related to these objectives of the performance audit.
Finding 4: NSF Did Not Base Its Improper Payment Estimate on the Population of Outlays Identified in
Its Risk Assessment
NSF’s statistical sampling process for computing its improper payment estimate does not cover all of the
elements of program outlays that are identified as susceptible to significant risk of improper payments
in the FY 2011 Outlay Risk Assessment. NSF’s FY 2011 Outlay Risk Assessment calculates 2.5 percent of
the total outlays for each NSF program and compares those calculated values to the $10 million
threshold. If the calculation yields results greater than $10 million, the program/activity is deemed
susceptible to risk of improper payments. As a result of its analysis, NSF classified the R&RA and EHR
programs as susceptible to significant risk of improper payments, noted these programs as “IPIA
Program – Research and Education Grants,” and performed a calculation to show that these programs
compose 90 percent of NSF’s outlays.
In its Sampling Plan for the Statistical Review of Federal Financial Reports for Improper Payments
Submitted for Payment of Awards, dated March 30, 2012, NSF used a population of FY 2011 FFR
expenditures to select and design a statistical sample to prepare the estimate of improper payments
(reported in Section IV of Appendix 2 to the FY 2012 AFR).
Total FY 2011 outlays for the R&RA and EHR programs include FFR expenditures, contract expenditures,
vendor payments, administrative costs, and other NSF accounting transactions; however, NSF only used
FFR expenditures in determining an estimate of improper payments. NSF’s Outlay Risk Assessment did
not include any additional analysis to determine whether the other outlay elements should be included
in the statistical sampling process. NSF also did not reconcile the FFR expenditure population used to
determine an estimate of improper payments to the supporting records to ensure that all appropriate
transactions were tested.
OMB Memorandum M-11-16, Part I, Section A, Subsection 7 states:
Unless an agency has specific written approval from OMB to deviate from the steps explained
below, agencies are required to follow these steps to determine whether the risk of improper
payments is significant and to provide valid annual estimates of improper payments.
Step 1: Review all programs and activities and identify those that are susceptible to significant
improper payments.
………………………………………………………………………………………………………………
Step 2: Obtain a statistically valid estimate of the annual amount of improper payments in
programs and activities (unless otherwise noted in this Guidance) for those programs that are
identified as susceptible to significant improper payments.
OMB Memorandum M-11-16, Part I, Section A, Subsection 4 states:
The law anticipates that agencies will examine the risk of, and feasibility of recapturing,
improper payment in all programs and activities they administer. The term “program” includes
Page | 11
activities or sets of activities recognized as programs by the public, OMB, or Congress, as well as
those that entail program management or policy direction. This definition includes, but is not
limited to, all grants including competitive grant programs and block/formula grant programs,
non-competitive grants such as single-source awards, regulatory activities, research and
development activities, direct Federal programs, all types of procurements (including capital
assets and service acquisition),4 and credit programs.
Footnote reference 4 states:
This is an important change for agencies to consider. Under the new law, agencies are required
to review vendor payments as part of their annual risk assessment process. If these risk
assessments determine that contract or vendor payments are susceptible to significant improper
payments (as defined in Part lA, Section 7, Step I), then agencies will be required to establish an
annual improper payment measurement for these vendor payments (as required by Part lA,
Section 7, Step 2).
NSF management considered its methodology (performing a risk assessment on outlays in order to
identify programs susceptible to significant risk of improper payments, then determining an improper
payment estimate on a sub-set of the outlays (FFR expenditures)) to be appropriate.
The statistical sampling process used to determine the improper payment estimate did not fully test
program outlays identified in the risk assessment. NSF’s estimated improper payment error rate
therefore cannot be applied to total program outlays. In addition, the improper payment estimate may
not be complete and accurate, as it is not based on the population of program outlays identified as
susceptible to significant risk of improper payments, and the population of FFR expenditures which the
estimate is based on is not reconciled to supporting records.
We recommend that the NSF Chief Financial Officer take appropriate action to improve its procedures
for performing the IPERA risk assessment. Specifically, the agency should:
2.1 Reevaluate the risk assessment process to ensure that it meets the requirements of IPERA and
the guidance outlined in OMB Memorandum M-11-16. NSF management should evaluate each
of the elements of program outlays (i.e., FFR expenditures, contract expenditures, vendor
payments, administrative costs, and other accounting transactions) separately during the risk
assessment process to identify which elements should be included for subsequent statistical
sampling processes.
2.2 Reconcile the FFR expenditures (and other expenditures, as required) included in the statistical
sampling process to supporting records in order to ensure the completeness of the population
used to calculate the improper payment estimate.
Finding 5: NSF Applied Testing Procedures Inconsistently and Maintained Insufficient Documentation
NSF did not perform testing procedures over the sample population in a consistent manner, and did not
retain sufficient documentation for the audit team to verify the criteria the agency followed in
identifying improper payments. NSF failed to develop a comprehensive testing methodology and applied
the limited methodology that was developed inconsistently in testing its samples. In addition, although
NSF’s Report for the Statistical Review of Federal Financial Reports for Improper Payment Awards, dated
June 30, 2012, states that NSF’s review was conducted in accordance with Government Auditing
Standards, NSF was unable to provide sufficiently detailed test plans, including the criteria for how each
sample item was tested against the attributes, to enable the audit team to re-perform the test work.
Page | 12
Furthermore, the work papers that NSF provided to the audit team did not contain documentation
supporting tests performed and conclusions reached.
Therefore, in lieu of re-performance, the audit team reviewed and evaluated the testing methodology
and supporting documentation provided for 45 randomly selected items of the 188 sample items
reviewed by NSF, as well as one additional sample NSF identified as an improper payment, in order to
evaluate the reasonableness of the conclusions reached. As a result of this review, we found that NSF’s
conclusions regarding the proper or improper nature of sample payments could not be supported by the
documentation in the sample item files and explanation of testing results. Specifically, we noted the
following:
• NSF determined that an expenditure was an improper payment based on the grantee’s
statement of inaccuracy in the account code. We determined that the workpapers did not
support the conclusion that the item was an improper payment.
• Expenditures were made outside of the period of performance identified in the grant letters. As
NSF did not provide additional documentation demonstrating how these items were determined
to be proper payments, we determined these samples should have been noted as improper
payments.
• An expenditure was determined to be an unallowable cost, but NSF concluded that the item was
a proper payment. We determined that the workpapers did not include explanation to support
the conclusion that the item was a proper payment.
In addition, we were unable to evaluate the reasonableness of NSF’s testing conclusions because the
criteria applied in test work and supporting workpapers were not sufficiently documented and
maintained.
• NSF’s workpapers did not contain sufficient documentation to support conclusions on testing
attributes.
• NSF’s workpapers did not contain adequate explanation to support attributes marked “N/A.”
Based on the reference in NSF’s Report for the Statistical Review of Federal Financial Reports for
Improper Payments of Awards, the following criteria apply. Generally accepted government auditing
standards (GAGAS) provide specific field work standards and requirements for non-financial-statement-
audit engagements, to include planning and documentation. These standards require agencies to
develop and document criteria specific to achieving audit objectives.
GAGAS also requires agencies to retain sufficient and appropriate documentation related to testing
audit objectives to enable re-performance. Specifically, GAGAS requires that documentation be
prepared in sufficient detail to enable an experienced auditor, having no previous connection with the
engagement, to understand from the documentation the nature, timing, extent, and results of
procedures performed; the evidence obtained and its source; and conclusions reached, including
evidence that supports the auditor’s significant judgments and conclusions.
Additionally, OMB Memorandum M-11-16, Part I, Section A, Subsection 2 states:
An improper payment is any payment that should not have been made or that was made in an
incorrect amount under statutory, contractual, administrative, or other legally applicable
requirements. Incorrect amounts are overpayments or underpayments that are made to eligible
Page | 13
recipients (including inappropriate denials of payment or service, any payment that does not
account for credit for applicable discounts, payments that are for the incorrect amount, and
duplicate payments). An improper payment also includes any payment that was made to an
ineligible recipient or for an ineligible good or service, or payments for goods or services not
received (except for such payments authorized by law). In addition, when an agency's review is
unable to discern whether a payment was proper as a result of insufficient or lack of
documentation, this payment must also be considered an improper payment.
NSF did not ensure that the testing methodology included defined attributes and criteria, or that
documentation was sufficient under GAGAS, as was represented in the report. Furthermore, NSF did not
ensure that sufficient review procedures were performed over the testing results.
As a result of the inconsistent application of the testing procedures, the improper payment estimate
may be inaccurate.
We recommend that the NSF Chief Financial Officer take appropriate action to improve its procedures
for conducting the testing necessary to identify improper payment estimates. Specifically, the agency
should:
5.1 Develop and consistently implement thorough test plans with detailed criteria for testing
attributes, which may include specific, unique guidance for evaluating each grant and/or each
type of grant expenditure.
5.2 Maintain testing documentation that contains sufficient, detailed explanations and supporting
documentation to support testing conclusions.
5.3 Develop internal controls and review procedures over the sample testing plans and results.
Other Matters: NSF Management Updated Representation of IPERA Process
During the fieldwork phase of the IPERA performance audit, NSF management represented to the audit
team that the agency’s approach, as documented in the FY 2012 AFR, complied with all aspects of IPERA,
including:
• Performing a risk assessment to identify programs susceptible to significant risk of improper
payments
• Performing sample testing to determine a statistically valid estimate of improper payments
• Reporting the appropriate results in the FY 2012 AFR
NSF performed an outlay risk assessment to calculate 2.5 percent of the total appropriation outlays for
each NSF program and compared those calculated values to the $10 million threshold to determine if
the program is susceptible to significant risk of improper payments. (NSF management describes the risk
assessment process in Subsection I of Appendix 2 to the FY 2012 AFR.) As a result of this analysis, the FY
2011 Outlay Risk Assessment identified the R&RA and EHR programs as susceptible to significant risk of
improper payments.
Subsection II of Appendix 2 describes the statistical sampling process that NSF conducts to estimate the
improper payment rate for each program identified as susceptible to a significant risk of improper
payments. Subsection IV includes an estimate of improper payments for the R&RA and EHR programs.
Furthermore, Section 1 of NSF’s Report for the Statistical Review of Federal Financial Reports for
Page | 14
Improper Payments of Awards states: “The ultimate purpose of the sampling was to estimate the value
of improper payments in accordance with P.L. 111-204.”
During fieldwork, the audit team determined that the statistical sampling methodology NSF utilized in
calculating its estimate of improper payments was not statistically valid, and presented its initial
evaluation to NSF management. The audit team also determined that the email correspondence
provided by NSF constituting the original approval of NSF’s statistical sampling methodology did not
demonstrate that the Director of OMB (Joshua Bolten, June 2003 through April 2006) approved the
appropriateness of NSF’s estimation methodology, as required per IPERA.
Subsequent to the communication of the audit team’s initial evaluation, NSF management had a
conversation with OMB to obtain its concurrence on the appropriateness of NSF’s statistical estimation
methodology. As a result of this conversation, NSF management provided the following updated
interpretation of the NSF IPERA reporting process and results.
NSF management represented to the audit team that all of the work performed in conjunction with the
NSF IPERA procedures was intended to satisfy OMB Memorandum M-11-16, Part I, Section A, Subsection
7, Step 1: Review all programs and activities and identify those that are susceptible to significant
improper payments, specifically:
a. Definition. For the purposes of this Guidance, “significant improper payments” are defined as
gross annual improper payments (i.e., the total amount of overpayments plus underpayments) in
the program exceeding (1) both 2.5 percent of program outlays and $10,000,000 of all program
or activity payments made during the fiscal year reported or (2) $100,000,000 (regardless of the
improper payment percentage of total program outlays)…
b. Systematic Method. All agencies shall institute a systematic method of reviewing all programs
and identify programs susceptible to significant improper payments. This systematic method
could be a quantitative evaluation based on a statistical sample or it could take into account risk
factors likely to contribute to significant improper payments…
NSF represented to the audit team that the risk assessment is a systematic method that includes two
steps:
(1) Comparing 2.5 percent of program outlays to the $10 million threshold to identify programs that
should be considered for subsequent evaluation.
(2) Performing a quantitative method based on statistical sampling on the identified programs’ FFR
expenditures to estimate an improper payment error rate and determine whether that rate is
less than 2.5 percent.
The statistical sampling that NSF performed in FY 2012 resulted in an estimated improper payment error
rate of .055 percent, which is less than the 2.5 percent threshold. NSF management maintained that
they may not be required to perform OMB Memorandum M-11-16, Part I, Section A, Subsection 7, Step
2: Obtain a statistically valid estimate of the annual amount of improper payments in programs and
activities (unless otherwise noted in this Guidance) for those programs that are identified as susceptible
to significant improper payments due to the second step they performed under the risk assessment
process. NSF management also stated that they believe the systematic method does not need to
produce a statistically valid estimate.
Page | 15
Although we took NSF management’s updated representation into account, we still identified a number
of issues with the ultimate calculation of the estimated improper payment error rate. As a result, we
determined that NSF management may be unable to rely upon the estimated improper payment error
rate because the sampling method is not statistically valid. Therefore, NSF cannot rely on the correlation
between the estimated improper payment rate and the improper payment threshold noted in OMB
guidance to support its statement that NSF is not susceptible to significant risk of improper payments.
Furthermore, the use of a systematic method such as a quantitative evaluation based on a statistical
sample only produces a reliable estimated improper payment rate if the entire process maintains
statistical validity. The findings noted above are the basis for this conclusion.
Although we make no recommendations regarding this matter, we suggest that the NSF Chief Financial
Officer ensure that the recommendations identified in findings 1 through 5 are implemented in order to
ensure that any future assessments of programs susceptible to improper payments are properly tested
and reported.
Page | 16
APPENDIX: MANAGEMENT COMMENTS
Compliance with the Improper Payments Elimination and Recovery Act
Published by the National Science Foundation, Office of Inspector General on 2013-03-15.
Below is a raw (and likely hideous) rendition of the original report. (PDF)