UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF INSPECTOR GENERAL AUDIT SERVICES February 11, 2016 Control Number ED-OIG/A06O0001 James Cole, Jr. General Counsel, Delegated the Duties of Deputy Secretary of Education U.S. Department of Education 400 Maryland Avenue, SW Washington, D.C. 20202 Dear Mr. Cole: This final audit report, “Management Certifications of Data Reliability,” presents the results of our audit. The objective of the audit was to determine what actions the U.S. Department of Education (Department) has taken, including making use of management certifications, to ensure the accuracy and reliability of select kindergarten through 12th grade (K–12) data reported in the Annual Performance Report (APR) and select Office of Elementary and Secondary Education (OESE) K–12 data. We judgmentally selected three data elements for case studies. These elements were either reported in the Department’s fiscal year (FY) 2013 1 APR, reported by State educational agencies (SEA) in their school year (SY) 2012–2013 Consolidated State Performance Report (CSPR) 2 or other annual reports, or used for determining program funding. The data elements selected were (1) Migrant Education Program Child Count (migrant child count), 3 (2) Prevention and Intervention Programs for Children and Youth Who Are Neglected, Delinquent, or At-Risk Child Count (ND child count), 4 and (3) Adjusted Cohort Graduation Rate (ACGR). BACKGROUND The Department is the smallest of the 15 Cabinet-level agencies in terms of Government staff, yet it has the third largest grant portfolio among the 26 Federal grant-making organizations. To demonstrate effective stewardship of these resources, the Department must have high-quality and timely data. Data quality is important to the Department because educators, researchers, 1 FY 2013 refers to October 1, 2012, through September 30, 2013. 2 For EDFacts reporting purposes including the CSPR, SY refers to a 12-month period defined by the State. 3 Category I migrant child count is the unduplicated statewide total number of eligible migrant children resident in a State for at least one day during the performance period. 4 Title I, Part D Subpart 1 provides formula grants to SEAs based on annual counts of ND children and youth, aged 20 or younger. The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational excellence and ensuring equal access. Final Report ED-OIG/A06O0001 Page 2 of 20 policymakers, and the public use information that the Department disseminates. The Department and other stakeholders also often use education data to make funding and policy decisions. The Government Performance and Results Act Modernization Act of 2010 (GPRA Modernization Act) serves as a foundation for helping agencies focus on their highest priorities and create a culture where data and empirical evidence play a greater role in policy, budget, and management decisions. Among the GRPA Modernization Act’s requirements are that the head of each agency discuss how the agency ensures the accuracy and reliability of data in its annual performance and planning reports. Office of Management and Budget (OMB) Circular A-11, “Preparation, Submission, and Execution of the Budget,” describes this and other requirements under the GPRA Modernization Act, including that agencies have verification and validation techniques that will ensure the completeness and reliability of all performance measurement data contained in their annual performance and planning reports as appropriate to the intended use of the data. OMB Memorandum M-13-17, “Next Steps in the Evidence and Innovation Agenda,” July 26, 2013, further emphasizes the President’s commitment to carrying out an aggressive management agenda that delivers a smarter, more innovative, and more accountable Federal Government for citizens. This agenda includes steps for harnessing data to improve agency results—specifically, data collected by Federal, State, or local agencies to run programs, which can be a valuable resource for program improvement and for helping agencies, consumers, and providers make more informed decisions. Finally, according to a 2013 report from the Association of Government Accountants Intergovernmental Partnership, as data become more important to monitoring the performance of Government programs and making policy decisions, the risk associated with using inaccurate data increases. Therefore performance data must be accurate and reliable when associated with specific metrics in the Department’s strategic plan, as well as overall grant performance. For the purposes of this audit, we are describing data quality by using terms defined in Government Accountability Office (GAO) guidance. 5 Data reliability refers to the accuracy and completeness of data, given the uses for which they are intended. Accuracy refers to the extent that recorded data reflect the actual underlying information. Completeness refers to the extent that relevant records are present and the fields in each record are populated appropriately. GAO does not define reasonableness; therefore, for the purposes of this audit, we refer to reasonableness as data that falls within the Department’s expected parameters. Data Reporting and Usage The data the Department receives are used in multiple ways and originates from various sources. For example, the Department uses the CSPR 6 to monitor States’ progress in implementing the Elementary and Secondary Education Act (ESEA), as amended, and to identify technical assistance needs and program management and policy needs. Part I of the CSPR collects data related to the five ESEA goals. 7 Part II of the CSPR collects information related to State activities 5 GAO, “Assessing the Reliability of Computer-Processed Data,” GAO-09-680G (July 2009). 6 The CSPR is the required annual reporting tool for each State, the District of Columbia, and Puerto Rico. 7 The five ESEA performance goals are: (1) by SY 2013–2014, all students will reach high standards in reading/language arts and mathematics; (2) all limited English proficient students will become proficient in English and reach high academic standards in reading/language arts and mathematics; (3) by SY 2005–2006, all students will be taught by highly qualified teachers; (4) all students will be educated in environments that are safe, drug free, and conducive to learning; and (5) all students will graduate from high school. Final Report ED-OIG/A06O0001 Page 3 of 20 and outcomes of specific ESEA programs needed for the programs’ GPRA Modernization Act indicators or other assessment and reporting requirements. The Department’s EDFacts data system is operated by the Institute of Education Sciences’ National Center for Education Statistics, and is used to centralize State-reported K–12 data in one federally coordinated data repository. EDFacts collects, analyzes, reports on, and promotes the use of high-quality, K–12 performance data for use in education planning, policy making, and management and budget decision-making to improve outcomes for students. EDFacts’ data quality program has four main areas of focus: (1) reviewing the data and providing feedback to the data suppliers; (2) refining business rules and improving guidance and instructions to the data suppliers; (3) managing the data dictionary, collection plan, and business rules; and (4) data governance regarding decision-making and performing data functions. The APR presents detailed information about progress in meeting the Department’s strategic goals and objectives and key performance measures. In the APR, the Secretary of Education (Secretary) asserts his confidence that the Department’s data verification and validation process and data sources used provide, to the extent possible, complete and reliable performance data pertaining to goals and objectives. The APR annual process begins in October and ends with the publication of the APR in February. The Office of the Deputy Secretary (ODS) serves as the lead office in the development of the APR. After program offices and the Executive Secretariat review and clear the APR, ODS is responsible for approving the final version of the APR for the Secretary’s signature. The Department’s Performance Improvement Officer, whose role was established by the GPRA Modernization Act, is responsible for coordinating and overseeing performance planning, measurement, and analysis activities, and must ensure that Department progress toward achieving all goals is communicated and made available on a public Web site of the agency. Numerous data elements in the Department’s APR originate at the school level. Schools submit data to local educational agencies (LEA), which then submit data to their SEA, which submits the data to the Department. The diagram below illustrates the flow of the ACGR data as it originates at the school level. The SEA certifies the ACGR data during its CSPR submission; it also certifies multiple data exchanges before it submits the CSPR. SEA Department School LEA • Certification via CSPR (EDFacts) Final Report ED-OIG/A06O0001 Page 4 of 20 Selected Data Elements Our audit focused on (1) the migrant child count that SEAs reported in their CSPR through EDFacts; (2) the ND child count that SEAs reported in their Annual Report of Children in Institutions for Neglected or Delinquent Children, Adult Correctional Institutions, and Community Day Programs for Neglected or Delinquent Children (ED Form 4376); and (3) the ACGR that SEAs reported in the CSPR through EDFacts and the Department reported in its APR. Migrant child count data are tied to program funding. SEAs submit these data annually through Part II of the CSPR, and the Department uses the data to calculate grantee award allocations. The statutory formula requires that, beginning with FY 2003, every State receives at least 100 percent of the amount that it received through the program in FY 2002. If Congress appropriates additional program funds above the FY 2002 amount, the Department uses the migrant child counts to calculate the additional funds allocated to each State. However, the appropriation has not exceeded the FY 2002 level of $396 million; in fact, it has decreased over time. The Migrant Education Program was appropriated about $375 million for FY 2014. SEAs report their ND child counts in ED Form 4376; those counts are tied directly to program funding and the Department uses them to calculate program grant allocations. Annual grant allocations are based on calculations using prior year child counts. For example, FY 2013 counts are used to determine FY 2014 awards. The ND program was appropriated almost $48 million for FY 2014. SEAs report the ACGR annually in Part II of the CSPR, and the Department uses it as a measure for its “Equitable Educational Opportunities” strategic goal, which aims to increase all students’ access to educational opportunities and decrease the number of high schools with persistently low graduation rates by 5 percent. The Department includes the ACGR in its APR via the CSPR annual submission. States calculate the ACGR based on the number of students who graduate in 4 years or less, from their first time in 9th grade, with a regular high school diploma divided by the number of students who form the adjusted cohort for the graduating class. To calculate and report the 4-year ACGR, States must follow the progress of each student in grades 9 through 12 over time and maintain documentation of students who enter or leave schools or districts within their States. AUDIT RESULTS The objective of the audit was to determine what actions the Department has taken, including making use of management certifications, to ensure the accuracy and reliability of select K–12 data it reports in the APR and select OESE K–12 data. We found that the Department took actions to ensure the completeness and reasonableness of select K–12 data it reported in its APR and select OESE K–12 data, including providing written guidance, using system edit checks, requiring management certifications, and conducting postsubmission data analysis. However, the Department needs to improve its controls to support the accuracy of data that SEAs report. Specifically, the Department could provide better oversight, including both technical assistance and monitoring, of SEAs’ controls over data quality for some Final Report ED-OIG/A06O0001 Page 5 of 20 of the elements reviewed and the verification and validation process for data it reports in its APR. The Department could also involve external auditors in the process by updating OMB Circular A-133 Compliance Supplement (Compliance Supplement) under which they conduct their audits. Such actions could result in improvements to the quality of data that SEAs submit. In its response to the draft report, ODS concurred with OIG’s view that the Department could continue to make improvements to its procedures to enhance data accuracy and reliability, and proposed corrective actions for each of our recommendations. However, ODS did not agree with OIG’s overall evaluation of the value and effectiveness of the Department’s current system of controls over SEA data accuracy and reliability, stating that it believed the draft report did not contain a sufficiently complete acknowledgement or description of its efforts in this area. We considered ODS’ comments, but did not change our finding or recommendations. We believe that the report includes appropriate recognition of the Department’s actions pertaining to data reliability while also identifying data quality issues, and that the recommendations—which the Department agreed to implement—will serve to further improve its system of controls. We summarized ODS’ comments at the end of the finding and included the full text of its comments as Attachment 3 of this report. FINDING – The Department Could Ensure More Accurate and Reliable Data by Promoting Better SEA Controls Over Data Quality The Department took actions to ensure the completeness of select K–12 data reported at various stages of the SEA data reporting processes. In addition, data submission processes were designed to require an authorized State official to certify to the quality of data submitted and allow for multiple submissions to encourage SEAs to self-report known data issues. Based on our review of these processes, we determined that the Department had adequate controls to provide reasonable assurance that the K–12 reported data are complete and reasonable, but that it could make improvements to its procedures to ensure data accuracy and reliability. Specifically, for migrant child count, ND child count, and ACGR data reported, the Department did not always require SEAs to provide reasonable assurance regarding the adequacy of their internal control systems in ensuring that the data the SEAs provided to the Department were accurate, reliable, and complete. In addition, the Department did not always require independent auditors to determine whether SEAs, LEAs, and other local operating agencies established, implemented, and accurately reported data quality control processes that ensure accurate eligible child counts and met the requirements of the Department’s regulations. According to OMB Circular A-123, “Management's Responsibility for Internal Control,” it is management’s responsibility to develop and maintain effective internal controls and to design management structures that help ensure accountability for results. SEA Submission When submitting data, SEAs followed the CSPR reporting process, business rules and file specifications, and annual program reporting requirements. For example, when SEAs submitted their annual CSPR data, an authorized SEA official certified that the reported data, to the best of the official’s knowledge, were true, reliable, and valid. EDFacts established a series of programmed system edit checks, whereby nonconforming data could not be submitted unless corrected or explained. Once the SEA finalized the data in the CSPR, it moved electronically into the EDFacts Data Warehouse. These CSPR procedures include the migrant child count and ACGR data, as well as performance information related to ND child count. SEAs report the ND Final Report ED-OIG/A06O0001 Page 6 of 20 child count data separately in the annual ED Form 4376, and the Department uses that data to determine funding allocations. In addition, the Department requires for some programs that SEA officials sign management certifications for information and data the SEA reports to the Department, which can be in addition to certifications SEAs provide for data collections like CSPR. Actions Taken by the Department For the three data elements we reviewed, the Department took actions before, during, and after SEAs submitted data to encourage data quality. Before SEAs submitted data, the Department established various regulations, guidance, instructions, and program directives, as well as annual business rules and file specifications to assist States in identifying, collecting, and reporting data, either through the CSPR or other required annual data submissions. The Department also offers SEAs program-specific technical assistance and training. When it received the SEAs’ data, the Department’s EDFacts system performed two automated edit checks and program staff assigned to EDFacts data groups assessed the data for completeness and reasonableness. The first edit check was a limited error check, in which data were tested to ensure that the data complied with automated business rules. If the data did not comply with the rules, the Department rejected the SEA data submission and required the SEA to resolve the deficiency before it resubmitted the data. The second edit check was a postsubmission check designed to detect and flag data that fall outside an expected range. These flags occurred when questions emerged about the reasonableness of the data or the Department needed more information from the SEA regarding the data it reported. The SEA either verified that the flagged data were correct or revised and resubmitted the corrected data. APR Process ODS serves as the lead office in developing the APR and is responsible for approving the final version to be signed by the Secretary, but it relies on program offices and others to verify the accuracy and reliability of reported data. Both ODS and program office officials stated that EDFacts staff assess data quality for data reported in the APR through EDFacts. EDFacts officials stated that program offices drive data quality and ultimately determine whether data are acceptable. Goal and Strategic Objective Leaders—agency officials who are responsible for the achievement of each performance goal, as well as objectives and subgoals under each strategic goal—provide ODS with quarterly updates on metrics (or identify when data are expected), analyze progress, and verify data sources and data limitations (an area in which we found some issues, as discussed in further detail below). EDFacts staff are also involved in the process through their use of diagnostic tests that are designed to assess the consistency and validity of data values. This reflects the shared responsibility across multiple Department offices for ensuring the quality of K–12 data, with ODS also playing an important role in the process. The GPRA Modernization Act requires that Federal agencies collect and validate performance data used to report progress toward performance goals. OMB Circular No. A-11 provides detailed guidance for the agency goal management process, as well as for the related performance data verification and validation process, which supports both reporting progress toward goals and data reliability efforts in general. In addition, OMB Circular A-123 and GAO’s “Standards for Internal Control in the Federal Government” both emphasize the importance of communication throughout Federal agencies to enable personnel to perform key roles in achieving objectives, addressing risks, and supporting systems of internal control. It is critical, then, that the Department’s system of Final Report ED-OIG/A06O0001 Page 7 of 20 internal control over the accuracy and reliability of reported data be universally understood and effectively implemented. Oversight of SEA Internal Controls The Department relies on the SEAs to report accurate data and requires the SEAs to sign certifications for the programs we reviewed. However, the Department does not always verify that the SEAs have adequate internal controls to ensure that data LEAs and other State agencies submitted are accurate and reliable, or tested to ensure the controls and systems are working as intended. From FY 2010 through FY 2015, the U.S. Department of Education Office of Inspector General (OIG) issued several audit reports pertaining to data quality in various programs (see Attachment 2). The Department responded to the recommendations made in these reports by modifying some monitoring processes, and providing additional training to program staff. For example, the Office of Migrant Education (OME) makes its State and local level monitoring protocols available on the Department’s Web site. These protocols, which OME uses when conducting on-site program reviews of migrant education programs, specifically require reviewers to look at the accuracy of certificate of eligibility documentation 8 and require a physical review of the certificate of eligibility form as well as follow up interviews of a sample of migrant families. OME’s State and local monitoring protocols also call for the review of the SEA’s child count procedures to ensure that they conform to the child count explanation OME accepted in the State’s CSPR. As a result of its monitoring reviews, OME has established grant conditions for repeated SEA data issues. However, we learned that OME conducted only three State monitoring visits due to staffing limitations in FYs 2013 through 2015. To ensure reliability, OME has been performing desk reviews of interview data related to child eligibility that States submit via the CSPR and has continued its efforts to resolve outstanding monitoring findings from prior years’ monitoring reports. OME also relies to some extent on the work of external auditors to determine whether entities receiving funds are submitting reliable data. The Compliance Supplement for 2014 describes the responsibility of external auditors with respect to the scope of single audits, and includes a section regarding the “counts of Migrant children eligible for funding purposes.” The Compliance Supplement suggests audit steps to determine whether SEAs, LEAs, and other local operating agencies established, implemented, and accurately reported in the CSPR Part I a quality control process that ensures an accurate eligible-child count and meets the requirements of the Department’s regulations. The Office of Safe and Healthy Students (OSHS), which is tasked with oversight of Title I, Part D programs, designed a monitoring plan for FY 2015 that includes a guiding question about an SEA’s process for data collection and a list of acceptable evidence, such as evidence that an SEA provided technical assistance to subgrantees on how to efficiently and effectively collect and use data. OSHS’s plan does not include tests specifically for the quality of the ND child count. During FY 2013, OSHS conducted three onsite monitoring visits that resulted in findings at two of 8 The certificate of eligibility includes eligibility information related to the qualifying move and work, and requires the parent, guardian, spouse, or worker to sign acknowledgement of the purpose of the form, and attest that the information to the best of their knowledge is true. The certificate of eligibility also includes an eligibility data certification that must be signed by the interviewer/recruiter and the designated SEA reviewer. Final Report ED-OIG/A06O0001 Page 8 of 20 the SEAs. OSHS found a lack of subgrantee monitoring by one SEA and the exclusion of the ND program from the SEA’s overall monitoring approach. Unlike OME, however, OSHS does not require external auditors to assess States’ or other entities’ controls over the quality of ND child count data. The three data elements we reviewed were all covered by management certifications the Department required program offices to complete using terminology shown in Table 1 below. To provide reasonable assurance for acceptable data submissions, the Department should amend the certification process and terminology to have the SEA assert that it has internal controls over data, which are tested and working as intended, or that the SEA has fully disclosed any data limitations and corrective actions. Including such language in the certification itself would not only provide criteria against which the Department or external auditors could monitor or audit, but also encourage States to think carefully about their data quality processes and promote a more open and honest dialogue between Federal and State officials. An OESE official stated that OESE is considering revising the SY 2014–2015 CSPR 9 management certification language to include a statement that the State has implemented internal controls and taken action to ensure data are true, reliable, and valid. OESE is also considering including a statement in the certification that allows for data submission with disclosure of known data limitations and the State’s corrective actions. Table 1. Management Certification Language Migrant Certificate of ED Form 4376 CSPR Part II Eligibility (ND child count) (Migrant child count and ACGR) (Migrant child count) I certify that based on the I certify that the State educational The CSPR Part II is now ready to be information provided to me, which agency has determined that the data certified and officially submitted to in all relevant aspects is reflected provided in Parts I and II meet the the U.S. Department of Education. above, I am satisfied that these requirements of Title I, ESEA. The By clicking the "Certify" button children are migratory children as information provided in this report below I acknowledge the following: defined in 20 United States Code is, to the best of my knowledge, (U.S.C.) 6399(2) and implementing complete and accurate. • That I am the proper certifying regulations, and thus eligible as such official for the SEA, and am for MEP [Migrant Education properly authorized by the SEA Program] services. to validate and submit all sections of this report. I hereby certify that, to the best of • That all data included in this my knowledge, the information is part of the SY 2012–2013 true, reliable, and valid and I CSPR, to the best of my understand that any false statement knowledge, are true, reliable, provided herein that I have made is and valid. subject to fine or imprisonment pursuant to 18 U.S.C. 1001. According to GAO’s “Standards for Internal Control in the Federal Government,” reliable internal and external data sources should provide data that are reasonably free from error and bias and faithfully represent what they purport to represent; thus, the Department should evaluate both internal and external sources of data for reliability. In addition, the Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards, Title 2 of the Code of 9 SY 2014-2015 CSPR reporting begins in mid-December 2015. Final Report ED-OIG/A06O0001 Page 9 of 20 Federal Regulations (C.F.R.), § 200.61, states internal controls are a process implemented and designed to provide reasonable assurance regarding reliability of reporting for internal and external use. We further noted that 2 C.F.R. § 200.208 states that, unless prohibited by Federal statutes or regulations, each Federal awarding agency or pass-through entity is authorized to require non- Federal entities to submit certifications and representations required by Federal statutes, or regulations, on an annual basis, and that submission may be required more frequently if a non- Federal entity fails to meet a requirement of a Federal award. Selected Data Element Corrections For SY 2012–2013, either EDFacts edit checks or program office data reviews identified inaccurate migrant child count, ND child count, and ACGR data that SEAs reported to the Department in the CSPR or other required annual report, resulting in data revisions and, in some cases, recalculation of award allocations. While the Department was able to identify errors and work with the SEAs for correction, the Department did not follow up with the SEAs to assess how those errors occurred when the SEA officials had provided certifications attesting the data’s accuracy. In addition, the spreadsheet maintained by EDFacts as evidence of SEA contact regarding reported data issues, was not detailed enough to consistently track information, such as, who identified the erroneous data (SEA, EDFacts staff or program staff), SEA response, timing or final resolution. Migrant Education Program Child Count In SY 2012–2013, one State had to revise its overall migrant child count. In April 2014, the State received comments as a result of EDFacts edit checks regarding information submitted for the CSPR Part I. EDFacts’ comments did not directly pertain to the child counts; however, corrections made to one file required a revision of the State’s overall migrant child count from 1,734 to 1,802 (as reported in CSPR Part II for SY 2012–2013). OESE’s EDFacts Data Steward found that the State provided contradictory information in its CSPR Part II data submission regarding its summer/intersession child count for SY 2012–2013. In its CSPR, the State explained that a wet growing season resulted in a decrease in its count of eligible migrant children in its summer/intersession. However, OESE noted that the State actually reported a 27.1 percent increase in this count. In response to OESE inquiries, the State identified a system coding error, and ultimately recertified its corrected migrant child count for SY 2012–2013 summer/intersession downward from 1,496 to 613. However, OESE did not determine why the SEA’s controls did not detect the coding error prior to certifying to the accuracy of the data. Neglected or Delinquent Program Child Count During FY 2013, OESE required two States to correct their reported child counts. The two SEAs had to correct ND child count data after certifying the data to be complete and accurate. As a result, OESE recalculated FY 2014 funding allocations for those SEAs. OESE staff noted a calculation discrepancy in one State’s submission used for the FY 2014 allocation. The State confirmed that the FY 2013 count, as originally submitted and certified as complete and accurate, was overstated due to a clerical error. The State submitted and certified a revised count, requiring the recalculation of its FY 2014 allocation and resulting in a downward adjustment to its FY 2015 award. Final Report ED-OIG/A06O0001 Page 10 of 20 The other State’s FY 2015 funding allocation data submission included a statement explaining the difference between the FY 2015 child count and the FY 2014 child count. This explanation indicated that the State might have made an error in its FY 2013 child count. After OESE contacted the State, the State confirmed that the count originally submitted and certified as complete and accurate for FY 2013 was incorrect. OESE required the State to submit a revised child count for FY 2013, resulting in a downward adjustment to its FY 2015 award because the FY 2014 allocation had already been made. Adjusted Cohort Graduation Rate In SY 2012–2013, after certifying that the data in their original submissions were true, reliable, and valid, 13 SEAs had to submit corrected ACGR data. One of these SEAs reported prior year ACGR data to meet the current year’s required CSPR reporting deadline, and provided a management certification that certified to the Department that this data represented the current year data. OESE contacted the SEA and requested that the SEA resubmit the data during the reopen correction period, which ended in May 2014. However, the SEA was not able to provide the corrected data until August 2014, which indicates that its SY 2012–2013 ACGR data were not available to report in January 2014 as required. The Department may be hampered in its ability to oversee the quality of ACGR data due to gaps created by an OESE reorganization. The Data Steward told us during the audit that the ACGR is not currently being monitored by the Office of State Support, a new office within OESE that resulted from the merging of a number of previously separate offices with various responsibilities. This reorganization was cited as the primary reason for the office’s lack of monitoring of States’ ACGRs. However, the Office of State Support plans to develop a monitoring process and pilot it in fall 2015. Additionally, the Office of State Support does not currently provide guidance to SEAs on establishing controls over the accuracy and reliability of ACGR data that the SEAs receive from the LEAs and subsequently report to the Department. Lastly, the Office of State Support also does not require external auditors to assess States’ or other entities’ controls over the quality of ACGR data. Although EDFacts edit checks and program office reviews identified data errors, the Department failed to follow up with the States to determine how these errors occurred when the State officials certified to the Department that the data were accurate and complete. The fact that State official certified to the quality of the submitted data in spite of data quality issues raises questions about the adequacy of State controls over the quality of their data. The Department’s reliance on inaccurate or unreliable data necessitated revisions to the Neglected and Delinquent Program award allocations for two SEAs as noted above. The Department also risks using inaccurate or unreliable data when describing its progress toward meeting performance goals. If the Department requires State officials to certify to the quality of data, it must follow up when data problems arise to ensure that the causes of misreported data are identified and corrected and, if necessary, hold certifying officials accountable if they did not meet the conditions of the certification statement. The management certification process provides the Department a level of assurance that SEAs have assessed the strength or weakness of data controls—both at the SEA and LEA levels—before they submit required data. When States submit data with significant errors, it is important for the Department to obtain some level of assurance that the SEA’s data quality procedures are adequate to ensure that those errors are not repeated, or, if there are known weaknesses to require full disclosure of any data issues found during testing. An SEA’s assertion of data quality enables the Department and auditors to assess the SEA’s controls supporting its data submissions. When there Final Report ED-OIG/A06O0001 Page 11 of 20 are known data quality weaknesses, an SEA’s full disclosure is necessary for the Department to meet its disclosure requirements on the APR and to ensure that data are used appropriately for critical decision-making. Recommendations We recommend that the Deputy Secretary of Education— 1.1 Review how management certifications are used across all programs in the Department, and ensure that certification language provides an assurance of the quality of data, clear accountability, and adequate disclosure of known data limitations. 1.2 Review how monitoring protocols are used across all programs in the Department to ensure that they contain steps to assess how well SEA or LEA procedures are working to provide accurate data for key data used in performance reporting or funding decisions. 1.3 Develop a formal tracking procedure for SEA data issues for use by EDFacts and principal offices that identifies how the data issue was identified, why it occurred, and resolution actions and timeframes. 1.4 Require all principal offices to follow up on known data errors where the SEA has signed a management certification to determine the causes of the error and ensure that the SEA develops procedures to ensure that it provides accurate information in accordance with the management certification. 1.5 Revise the Compliance Supplement, as appropriate, to address areas where external auditors should determine whether SEAs have controls in place to ensure that data collected from LEAs and other State agencies are accurate and reliable and support the management certifications they sign. ODS Comments In a written response to the draft report, ODS did not agree with OIG’s overall evaluation of the value and effectiveness of the Department’s current system of controls over SEA data accuracy and reliability, stating that it believed the draft report did not contain a sufficiently complete acknowledgement or description of its efforts in this area. However, ODS agreed to take action in accordance with each recommendation and stated it would continue to encourage more accurate and reliable reporting of data by SEAs. In response to Recommendation 1.1, ODS agreed to review management certification language for K–12 data submitted by SEAs to EDFacts and other major K–12 collections and to update certification language as appropriate. In response to Recommendation 1.2, ODS stated it will review and enhance, as necessary, monitoring protocols for major programs that require data to be submitted by State agencies to EDFacts. In response to Recommendations 1.3 and 1.4, ODS agreed to build on its existing tracking procedures and data quality process, to include noting how data issues are identified and having principal office staff follow up with SEAs on the cause of any such data issues and appropriate corrective actions. In response to Recommendation 1.5, ODS stated it will work with OIG to revise the Compliance Final Report ED-OIG/A06O0001 Page 12 of 20 Supplement, for certain programs, to address areas where external auditors may assess SEA controls over the accuracy and reliability of data collected from LEAs and support the management certifications the SEAs sign. ODS also provided technical comments, which we considered and made changes as appropriate. OBJECTIVES, SCOPE, AND METHODOLOGY The original objective of the audit was to determine what actions the Department has taken, including the use of management certifications, to ensure the accuracy and reliability of key K–12 performance data. The audit objective was revised to determine what actions the Department has taken, including making use of management certifications, to ensure the accuracy and reliability of select K–12 data reported in the APR and select OESE K–12 data. We performed work at principal and program offices responsible for creating the APR and administering select programs to determine what the Department did to ensure the accuracy and reliability of three selected data elements for FY 2013. We performed audit fieldwork from March 2014 through July 2015 at principal and program offices in Washington, D.C., and at our offices in Dallas, Texas and New York, New York. We held an exit conference with Department officials on August 4, 2015, to discuss the results of the audit. Selection of Case Study Data Elements To accomplish our audit objective, we judgmentally selected three K–12 APR and CSPR data elements for case study that were reported in the FYs 2012 and 2013 APRs or used by OESE to make critical management or funding decisions. We considered approximately 30 K–12 data elements sourced from 16 major CSPR data categories and 14 data elements used to measure 4 of 6 Department strategic goals. We prioritized the 30 data elements based on the following factors: • APR data and data sources, • data reported by States in the CSPR, • demonstrated broad usage of the data as evidenced by data queries of EDFacts data users, • associated programs having large funding amounts, and • data element tied directly to funding or eligibility to receive funding. We judgmentally selected the three data elements (1) migrant child count, (2) ND child count, and (3) ACGR. The results of these case studies should not be projected to other data elements not included in our review. For the three data elements selected, we determined how the Department ensured that SEAs have internal controls and processes for collecting and verifying data they submitted to the Department. We also determined what data verification and validation processes the Department used for data received from the SEAs. We gained an understanding of the actions the Department took to ensure data accuracy and reliability before data collection, at the point of data reporting, and after data submission for FY 2013 and FY 2014. Due to the timing of reporting and the unavailability of some FY 2014 data, we concentrated our case studies on FY 2013. Final Report ED-OIG/A06O0001 Page 13 of 20 To achieve our objective, we performed the following procedures. • Interviewed management and staff from Institute of Education Sciences National Center for Education Statistics, ODS, Office of General Counsel, and Risk Management Service, as well as from principal and program offices with responsibility for ensuring reliable migrant child count, ND child count, and ACGR data. • Reviewed OESE’s March 10, 2014 Dear Colleague Letter regarding Fraud in Title I-Funded Tutoring Programs and compared the language used in management certifications for the three data elements selected for (a) consistency and quality of the certification terminology, (b) an assertion as to accuracy, (c) an acknowledgement of personal liability and that Federal funds are involved, and (d) inclusion of the relevant Federal statute and potential for penalties—key elements of an effective certification as previously identified by OIG. • Analyzed, for the three data elements selected, the number of times data were corrected 10 after SEA submission and whether the Department or the SEA identified the inaccuracy. We also determined what actions the Department took when SEAs reported inaccurate, unreliable, and/or incomplete data. Internal Controls We performed a review of internal controls applicable to the Department’s processes to ensure the accuracy and completeness of SEA-reported data. Specifically, we reviewed • Federal laws and regulations related to data quality and accountability for data quality; • prior audit reports from OIG and other agencies related to data quality and accountability for data quality; • EDFacts policies and procedures related to data submission, reporting, and validation; • ODS policies and procedures for collecting data reported in the APR; and • OESE and program office policies, procedures, monitoring protocols, instructions, guidance, Dear Colleague Letters, and technical assistance provided to SEAs. Data Reliability To achieve our objective, we assessed the Department’s controls over data reliability with regards to the three data elements selected. We did not do any additional testing beyond what the Department had done. The finding presents how the Department can improve its processes to better ensure the reliability of its data. We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. 10 The Department could not identify the number of times ACGR data were changed or who initiated the changes. Final Report ED-OIG/A06O0001 Page 14 of 20 ADMINISTRATIVE MATTERS Corrective actions proposed (resolution phase) and implemented (closure phase) by your office will be monitored and tracked through the Department’s Audit Accountability and Resolution Tracking System (AARTS). ED policy requires that you develop a final corrective action plan for our review in the automated system within 30 days of the issuance of this report. The corrective action plan should set forth the specific action items, and targeted completion dates, necessary to implement final corrective actions on the findings and recommendations contained in this final audit report. An electronic copy of this report has been provided to your Audit Liaison Officer(s). In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector General is required to report to Congress twice a year on the audits that remain unresolved after six months from the date of issuance. In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office of Inspector General are available to members of the press and general public to the extent information contained therein is not subject to exemptions in the Act. We appreciate the cooperation given us during this review. If you have any questions, please call Daniel Schultz at 646-428-3888. Sincerely, /s/ Patrick J. Howard Assistant Inspector General for Audit Electronic cc: Laura Ginns, Audit Liaison Officer, ODS Heather Acord, Audit Liaison Officer, ODS Teresa Cahalan, Audit Liaison Officer, Institute of Education Sciences Delores Warner, Audit Liaison Officer, OESE Attachments Final Report ED-OIG/A06O0001 Page 15 of 20 Attachment 1: Acronyms, Abbreviations, and Short Forms Used in This Report ACGR Adjusted Cohort Graduation Rate APR Annual Performance Report CAP Corrective Action Plan C.F.R. Code of Federal Regulations Compliance Supplement OMB Circular A-133 Compliance Supplement for 2014 CSPR Consolidated State Performance Report Department U.S. Department of Education ED Form 4376 Annual Report of Children in Institutions for Neglected or Delinquent Children, Adult Correctional Institutions, and Community Day Programs for Neglected or Delinquent Children ESEA Elementary and Secondary Education Act FY Fiscal Year GAO Government Accountability Office GPRA Modernization Act Government Performance and Results Act Modernization Act of 2010 K–12 Kindergarten Through 12th Grade LEA Local Educational Agency Migrant child count Migrant Education Program Child Count ND child count Prevention and Intervention Programs for Children and Youth Who are Neglected, Delinquent or At Risk ODS Office of the Deputy Secretary OESE Office of Elementary and Secondary Education OIG Office of Inspector General OMB Office of Management and Budget Final Report ED-OIG/A06O0001 Page 16 of 20 OME Office of Migrant Education OSHS Office of Safe and Healthy Students SEA State Educational Agency Secretary Secretary of Education SY School Year U.S.C. United States Code Final Report ED-OIG/A06O0001 Page 17 of 20 Attachment 2: Prior OIG Reports That Noted Data Quality Issues OIG Audit Report, ACN, Date Issued Audit Results “U.S. Department of Education’s The OIG found that the Department relied on SEAs to ensure the accuracy of the Implementation and Oversight of information but did not verify that the SEAs had policies and procedures to ensure Approved Elementary and Secondary accuracy. In addition, the Department did not require SEAs to provide an assurance Education Act Flexibility Requests” statement covering the accuracy of the data submitted and did not have procedures A04N0012 requiring SEAs to disclose any limitations of the information, data, or validation process. January 2015 “U.S. Department of Education’s and The OIG found that, although the Department and all five SEAs reviewed had systems of Five State Education Agencies’ Systems internal control designed to prevent and detect inaccurate, unreliable, or incomplete of Internal Control Over Statewide Test statewide test results, these systems did not always require corrective action if they found Results” indicators of inaccurate, unreliable, or incomplete statewide test results. The OIG also A07M0001 found the Department could improve its monitoring of States’ test results by requiring April 2014 SEAs to provide an explanation for data that the Department’s data collection system flagged as either incorrect or outside an anticipated range. It also could improve its monitoring of SEAs and LEAs by resuming reviews of test administration procedures during onsite monitoring visits and having SEAs’ systems of internal control over statewide test results evaluated during standards and assessment peer reviews. “U.S. Department of Education’s OIG found that the overall data verification and validation approach initially described by Implementation of Government the Department, in its APR, was not actually being used by its principal offices. Performance and Results Act” Additionally, we noted that the Department did not specifically identify the means used to A19M0005 verify and validate measured values; the level of accuracy required for the intended use of January 2014 data; any limitations to the data, including how the agency will compensate for such limitations if needed to reach the required level of accuracy; and did not always accurately identify the sources for the data. “U.S. Department of Education’s and The OIG found that the Department relied on the SEAs to report accurate, reliable, and Selected States’ Oversight of the 21st complete 21st Century Community Learning Centers Program performance data through Century Community Learning Centers the Profile and Performance Information Collection System. However, it did not ensure Program” that the SEAs validated the data that subgrantees submitted. A04L0004 June 2013 “Centers for Independent Living The OIG found that RSA had not provided required and consistent monitoring and Compliance, Performance, Recovery Act oversight of the Centers for Independent Living as specified in Title VII of the Reporting, and Monitoring” Rehabilitation Act of 1973, as amended. The OIG also found partially supported A06K0011 performance information reports and inadequately documented files at the Centers for September 2012 Independent Living reviewed. The OIG also found that the documentation provided by eight Centers for Independent Living to support numbers of consumers served was inconsistent with the numbers reported on the annual 704 Performance Reports. “The Effectiveness of the Department’s Although the OIG found that the Department’s processes to ensure the accuracy and Data Quality Review Processes” completeness of recipient reported data were generally effective, it also found instances of A19K0010 recipient-reported data that were inconsistent with data in the Grants Administration and August 2011 Payment System, contract file documentation, or other data elements within the recipient reports. These anomalies still existed after the Department had completed its formal data quality review processes and after the related recipient correction period. “Georgia Department of Education’s The OIG found that neither a Georgia LEA nor the Georgia Department of Education Controls Over Performance Data Entered established adequate systems of internal control to ensure that accurate, reliable, and in EDFacts” complete data were entered in EDFacts. As a result, inaccurate or unsupported data were A04J0003 reported, including dropouts, graduates, and discipline incidents. April 2010 Final Report ED-0 I G/ A060000 I Page 18 of20 Attachment 3: ODS Comments UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF THE DEPl!TY SECRETARY January 13, 2016 TO: Patrick J. Howard Assistant Inspector General for Audit Office of Inspector General FROM: Mark Washington Performance Improvement Officer Office ofthe Deputy Sccrotary SUBJECT: Comments on Draft Audit Report HManagc:.1m:nt Certifications of Data Reliability" (Control Number ED 01G/A0600001) Thank you fot: providing the Office ofthe Deputy Secretary with an opportunity to review and respond to the finding and recommendations in the Office oflnspector General's (OIG) draft audit report on "Management Certifications ofData Reliability" OIG Control Number ED OIG/A060000l. . In accordance with the Government Performance and Results Act (GPRA) Modernization Act of 2010 (GPRAMA), the Department's framework for perfonnance management starts with the · Strategic Plan, including its Agency Priority Goals, which serve as the foundation for establishing long-term priorities and developing performance goals, objectives., and metrics by which the Department can gauge achievement of its stated outcomes. Progress toward the Department's Strategic Plan is measured using data-driven review and analysis to promote active management engagement across the Department and is reported annually. GPRAMA requires agencies to describe the accuracy and reliability ofdata presented in the Annual Performance Report (APR). Details of how the Deportment assesses the completeness and reliability, including known limitatioo.s, of the data reported are presented in an Appendix to the APR. Through a process ofcontinuous improvement, the Department continues to assess its validation process and welcomes input from the OIG and others. We are pleased that this draft report acknowledges a number ofthe actions the Department has talcen to improve the completeness and reasonableness of State Education Agency (SEA) data reported in the Department's APR. This includes providing written guidance, using system edit checks, requiring management certifications, and conducting post-submission data analysis. The Department also offers various forms oftechnical assistance and training sessions. The draft report states that the Department docs, in fact, take actions to encourage data quality before, during, and after an SEA submits data. We appreciate that OIG commends and affirms that the 400 MARYLAND AV.E., SW, WASHINGTON, DC 20202 Tl>c Dcport11trnt ofEdua>tion '$ mission u to p romote stutlcrt at:ltinem~nt antiJXeporation /or g/<Jbal com~titi~n= by /ostmng educational acellt~~Ce and ensunng "'tual access. Final Report ED-OIG/A06O0001 Page 19 of 20 2 Department has adequate controls to provide reasonable assurance that the K-12 reported data are complete and reasonable. During 2015, the Department strengthened its approach to data verification and validation, and will continue to do so. We are committed to creating a culture of continuous improvement to ensure more accurate and reliable data and we work continuously to promote better SEA controls over data quality. As such, we agree with OIG's view that the Department could continue to make improvements to its procedures to enhance data accuracy and reliability as we have for a number of years, despite practical limitations. Our responses to the draft finding are set forth below. While there is information with which we agree in the draft report, it does not contain a sufficiently complete acknowledgement or description of the efforts of the Department has made and is making in this area, and it does not properly value the positive aspects ofthese efforts. Thus, we do not agree with the overall evaluation of the value and effectiveness of our current system of technical assistance, assurances, data checks, certifications, and encouragement to States to submit corrections and updates when appropriate. As such, we are providing also attached a set oftechnical comments to improve the accuracy of the report. Finding: The Department Could Ensu re More Accurate and Reliable Data by Pr omoting Better SEA Controls O ver Data Quality R esponse: As noted above, the OIG does not give sufficient credit to the successes of the Department's current mix of tools to support SEAs on providing valid and reliable data. In spite of that, the Department will continue to encourage more accurate and reliable reporting of data by SEAs. The Department will review management certification language, for K-12 data · submitted by SEAs to EDFacts and other major K-12 collections, to determine that the certifications provide clear statements that the SEA is responsible for having an effective internal control structure that provides reasonable assurance that the data submitted to the Department are reasonably free from error and bias and faithfully represent what they purport to represent and for disclosing any known data limitations. As appropriate, certification language will be updated. The Department will also review monitoring protocols for major programs that require data be submitted by State agencies to EDFacts, to see if enhancements should be made. For data submitted under a signed management certification, the Department plans to build on its existing tracking procedures to track data issues including how those issues are identified. Additionally, the Department will build on its data quality process, for data submitted under a signed management certification, for principal office staff to follow up with SEAs on the cause of data issues and appropriate corrective actions. The Department will work with the OIG to revise, for certain programs, the Compliance Supplement to address areas where external auditors could determine whether SEAs have controls in place to ensure that data collected from LEAs are accurate and reliable and support the management certifications SEAs sign. Final Report ED-OIG/A06O0001 Page 20 of 20 3 Please Jet us know if you have any questions or need further information about any of our comments and responses. We appreciate the effort that went into the field work and the report and thank you for the opportunity to review and respond to the draft. Mark Washington / Performance lmproJ~m~ t Officer Office of the Deputy Secretary Enclosures: Corrective Actions Technical Comments
Management Certifications of Data Reliability
Published by the Department of Education, Office of Inspector General on 2016-02-11.
Below is a raw (and likely hideous) rendition of the original report. (PDF)