oversight

Management Certifications of Data Reliability

Published by the Department of Education, Office of Inspector General on 2016-02-11.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                 UNITED STATES DEPARTMENT OF EDUCATION
                                                 OFFICE OF INSPECTOR GENERAL

                                                                                                                        AUDIT SERVICES


                                                            February 11, 2016

                                                                                                                  Control Number
                                                                                                                  ED-OIG/A06O0001

James Cole, Jr.
General Counsel, Delegated the Duties of Deputy Secretary of Education
U.S. Department of Education
400 Maryland Avenue, SW
Washington, D.C. 20202

Dear Mr. Cole:

This final audit report, “Management Certifications of Data Reliability,” presents the results of our
audit. The objective of the audit was to determine what actions the U.S. Department of Education
(Department) has taken, including making use of management certifications, to ensure the
accuracy and reliability of select kindergarten through 12th grade (K–12) data reported in the
Annual Performance Report (APR) and select Office of Elementary and Secondary Education
(OESE) K–12 data. We judgmentally selected three data elements for case studies. These
elements were either reported in the Department’s fiscal year (FY) 2013 1 APR, reported by State
educational agencies (SEA) in their school year (SY) 2012–2013 Consolidated State Performance
Report (CSPR) 2 or other annual reports, or used for determining program funding. The data
elements selected were (1) Migrant Education Program Child Count (migrant child count), 3 (2)
Prevention and Intervention Programs for Children and Youth Who Are Neglected, Delinquent, or
At-Risk Child Count (ND child count), 4 and (3) Adjusted Cohort Graduation Rate (ACGR).




                                                         BACKGROUND


The Department is the smallest of the 15 Cabinet-level agencies in terms of Government staff, yet
it has the third largest grant portfolio among the 26 Federal grant-making organizations. To
demonstrate effective stewardship of these resources, the Department must have high-quality and
timely data. Data quality is important to the Department because educators, researchers,


1
  FY 2013 refers to October 1, 2012, through September 30, 2013.
2
  For EDFacts reporting purposes including the CSPR, SY refers to a 12-month period defined by the State.
3
  Category I migrant child count is the unduplicated statewide total number of eligible migrant children resident in a
State for at least one day during the performance period.
4
  Title I, Part D Subpart 1 provides formula grants to SEAs based on annual counts of ND children and youth, aged 20
or younger.


    The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                      excellence and ensuring equal access.
Final Report
ED-OIG/A06O0001                                                                                        Page 2 of 20

policymakers, and the public use information that the Department disseminates. The Department
and other stakeholders also often use education data to make funding and policy decisions.

The Government Performance and Results Act Modernization Act of 2010 (GPRA Modernization
Act) serves as a foundation for helping agencies focus on their highest priorities and create a
culture where data and empirical evidence play a greater role in policy, budget, and management
decisions. Among the GRPA Modernization Act’s requirements are that the head of each agency
discuss how the agency ensures the accuracy and reliability of data in its annual performance and
planning reports. Office of Management and Budget (OMB) Circular A-11, “Preparation,
Submission, and Execution of the Budget,” describes this and other requirements under the GPRA
Modernization Act, including that agencies have verification and validation techniques that will
ensure the completeness and reliability of all performance measurement data contained in their
annual performance and planning reports as appropriate to the intended use of the data. OMB
Memorandum M-13-17, “Next Steps in the Evidence and Innovation Agenda,” July 26, 2013,
further emphasizes the President’s commitment to carrying out an aggressive management agenda
that delivers a smarter, more innovative, and more accountable Federal Government for citizens.
This agenda includes steps for harnessing data to improve agency results—specifically, data
collected by Federal, State, or local agencies to run programs, which can be a valuable resource for
program improvement and for helping agencies, consumers, and providers make more informed
decisions. Finally, according to a 2013 report from the Association of Government Accountants
Intergovernmental Partnership, as data become more important to monitoring the performance of
Government programs and making policy decisions, the risk associated with using inaccurate data
increases. Therefore performance data must be accurate and reliable when associated with specific
metrics in the Department’s strategic plan, as well as overall grant performance.

For the purposes of this audit, we are describing data quality by using terms defined in
Government Accountability Office (GAO) guidance. 5 Data reliability refers to the accuracy and
completeness of data, given the uses for which they are intended. Accuracy refers to the extent
that recorded data reflect the actual underlying information. Completeness refers to the extent that
relevant records are present and the fields in each record are populated appropriately. GAO does
not define reasonableness; therefore, for the purposes of this audit, we refer to reasonableness as
data that falls within the Department’s expected parameters.

Data Reporting and Usage
The data the Department receives are used in multiple ways and originates from various sources.
For example, the Department uses the CSPR 6 to monitor States’ progress in implementing the
Elementary and Secondary Education Act (ESEA), as amended, and to identify technical
assistance needs and program management and policy needs. Part I of the CSPR collects data
related to the five ESEA goals. 7 Part II of the CSPR collects information related to State activities


5
  GAO, “Assessing the Reliability of Computer-Processed Data,” GAO-09-680G (July 2009).
6
  The CSPR is the required annual reporting tool for each State, the District of Columbia, and Puerto Rico.
7
  The five ESEA performance goals are: (1) by SY 2013–2014, all students will reach high standards in
reading/language arts and mathematics; (2) all limited English proficient students will become proficient in English
and reach high academic standards in reading/language arts and mathematics; (3) by SY 2005–2006, all students will
be taught by highly qualified teachers; (4) all students will be educated in environments that are safe, drug free, and
conducive to learning; and (5) all students will graduate from high school.
Final Report
ED-OIG/A06O0001                                                                      Page 3 of 20

and outcomes of specific ESEA programs needed for the programs’ GPRA Modernization Act
indicators or other assessment and reporting requirements.

The Department’s EDFacts data system is operated by the Institute of Education Sciences’
National Center for Education Statistics, and is used to centralize State-reported K–12 data in one
federally coordinated data repository. EDFacts collects, analyzes, reports on, and promotes the
use of high-quality, K–12 performance data for use in education planning, policy making, and
management and budget decision-making to improve outcomes for students. EDFacts’ data
quality program has four main areas of focus: (1) reviewing the data and providing feedback to the
data suppliers; (2) refining business rules and improving guidance and instructions to the data
suppliers; (3) managing the data dictionary, collection plan, and business rules; and (4) data
governance regarding decision-making and performing data functions.

The APR presents detailed information about progress in meeting the Department’s strategic goals
and objectives and key performance measures. In the APR, the Secretary of Education (Secretary)
asserts his confidence that the Department’s data verification and validation process and data
sources used provide, to the extent possible, complete and reliable performance data pertaining to
goals and objectives. The APR annual process begins in October and ends with the publication of
the APR in February. The Office of the Deputy Secretary (ODS) serves as the lead office in the
development of the APR. After program offices and the Executive Secretariat review and clear the
APR, ODS is responsible for approving the final version of the APR for the Secretary’s signature.
The Department’s Performance Improvement Officer, whose role was established by the GPRA
Modernization Act, is responsible for coordinating and overseeing performance planning,
measurement, and analysis activities, and must ensure that Department progress toward achieving
all goals is communicated and made available on a public Web site of the agency.

Numerous data elements in the Department’s APR originate at the school level. Schools submit
data to local educational agencies (LEA), which then submit data to their SEA, which submits the
data to the Department. The diagram below illustrates the flow of the ACGR data as it originates
at the school level. The SEA certifies the ACGR data during its CSPR submission; it also certifies
multiple data exchanges before it submits the CSPR.


                                                           SEA                  Department
     School                      LEA                  • Certification
                                                          via CSPR                (EDFacts)
Final Report
ED-OIG/A06O0001                                                                      Page 4 of 20

Selected Data Elements
Our audit focused on (1) the migrant child count that SEAs reported in their CSPR through
EDFacts; (2) the ND child count that SEAs reported in their Annual Report of Children in
Institutions for Neglected or Delinquent Children, Adult Correctional Institutions, and Community
Day Programs for Neglected or Delinquent Children (ED Form 4376); and (3) the ACGR that
SEAs reported in the CSPR through EDFacts and the Department reported in its APR.

Migrant child count data are tied to program funding. SEAs submit these data annually through
Part II of the CSPR, and the Department uses the data to calculate grantee award allocations. The
statutory formula requires that, beginning with FY 2003, every State receives at least 100 percent
of the amount that it received through the program in FY 2002. If Congress appropriates
additional program funds above the FY 2002 amount, the Department uses the migrant child
counts to calculate the additional funds allocated to each State. However, the appropriation has not
exceeded the FY 2002 level of $396 million; in fact, it has decreased over time. The Migrant
Education Program was appropriated about $375 million for FY 2014.

SEAs report their ND child counts in ED Form 4376; those counts are tied directly to program
funding and the Department uses them to calculate program grant allocations. Annual grant
allocations are based on calculations using prior year child counts. For example, FY 2013 counts
are used to determine FY 2014 awards. The ND program was appropriated almost $48 million for
FY 2014.

SEAs report the ACGR annually in Part II of the CSPR, and the Department uses it as a measure
for its “Equitable Educational Opportunities” strategic goal, which aims to increase all students’
access to educational opportunities and decrease the number of high schools with persistently low
graduation rates by 5 percent. The Department includes the ACGR in its APR via the CSPR
annual submission. States calculate the ACGR based on the number of students who graduate in 4
years or less, from their first time in 9th grade, with a regular high school diploma divided by the
number of students who form the adjusted cohort for the graduating class. To calculate and report
the 4-year ACGR, States must follow the progress of each student in grades 9 through 12 over time
and maintain documentation of students who enter or leave schools or districts within their States.




                                      AUDIT RESULTS


The objective of the audit was to determine what actions the Department has taken, including
making use of management certifications, to ensure the accuracy and reliability of select K–12 data
it reports in the APR and select OESE K–12 data.

We found that the Department took actions to ensure the completeness and reasonableness of
select K–12 data it reported in its APR and select OESE K–12 data, including providing written
guidance, using system edit checks, requiring management certifications, and conducting
postsubmission data analysis. However, the Department needs to improve its controls to support
the accuracy of data that SEAs report. Specifically, the Department could provide better oversight,
including both technical assistance and monitoring, of SEAs’ controls over data quality for some
Final Report
ED-OIG/A06O0001                                                                       Page 5 of 20

of the elements reviewed and the verification and validation process for data it reports in its APR.
The Department could also involve external auditors in the process by updating OMB Circular
A-133 Compliance Supplement (Compliance Supplement) under which they conduct their audits.
Such actions could result in improvements to the quality of data that SEAs submit.

In its response to the draft report, ODS concurred with OIG’s view that the Department could
continue to make improvements to its procedures to enhance data accuracy and reliability, and
proposed corrective actions for each of our recommendations. However, ODS did not agree with
OIG’s overall evaluation of the value and effectiveness of the Department’s current system of
controls over SEA data accuracy and reliability, stating that it believed the draft report did not
contain a sufficiently complete acknowledgement or description of its efforts in this area. We
considered ODS’ comments, but did not change our finding or recommendations. We believe that
the report includes appropriate recognition of the Department’s actions pertaining to data
reliability while also identifying data quality issues, and that the recommendations—which the
Department agreed to implement—will serve to further improve its system of controls. We
summarized ODS’ comments at the end of the finding and included the full text of its comments as
Attachment 3 of this report.

FINDING – The Department Could Ensure More Accurate and Reliable
          Data by Promoting Better SEA Controls Over Data Quality

The Department took actions to ensure the completeness of select K–12 data reported at various
stages of the SEA data reporting processes. In addition, data submission processes were designed
to require an authorized State official to certify to the quality of data submitted and allow for
multiple submissions to encourage SEAs to self-report known data issues. Based on our review of
these processes, we determined that the Department had adequate controls to provide reasonable
assurance that the K–12 reported data are complete and reasonable, but that it could make
improvements to its procedures to ensure data accuracy and reliability. Specifically, for migrant
child count, ND child count, and ACGR data reported, the Department did not always require
SEAs to provide reasonable assurance regarding the adequacy of their internal control systems in
ensuring that the data the SEAs provided to the Department were accurate, reliable, and complete.
In addition, the Department did not always require independent auditors to determine whether
SEAs, LEAs, and other local operating agencies established, implemented, and accurately reported
data quality control processes that ensure accurate eligible child counts and met the requirements
of the Department’s regulations. According to OMB Circular A-123, “Management's
Responsibility for Internal Control,” it is management’s responsibility to develop and maintain
effective internal controls and to design management structures that help ensure accountability for
results.

SEA Submission
When submitting data, SEAs followed the CSPR reporting process, business rules and file
specifications, and annual program reporting requirements. For example, when SEAs submitted
their annual CSPR data, an authorized SEA official certified that the reported data, to the best of
the official’s knowledge, were true, reliable, and valid. EDFacts established a series of
programmed system edit checks, whereby nonconforming data could not be submitted unless
corrected or explained. Once the SEA finalized the data in the CSPR, it moved electronically into
the EDFacts Data Warehouse. These CSPR procedures include the migrant child count and
ACGR data, as well as performance information related to ND child count. SEAs report the ND
Final Report
ED-OIG/A06O0001                                                                           Page 6 of 20

child count data separately in the annual ED Form 4376, and the Department uses that data to
determine funding allocations. In addition, the Department requires for some programs that SEA
officials sign management certifications for information and data the SEA reports to the
Department, which can be in addition to certifications SEAs provide for data collections like
CSPR.

Actions Taken by the Department
For the three data elements we reviewed, the Department took actions before, during, and after
SEAs submitted data to encourage data quality. Before SEAs submitted data, the Department
established various regulations, guidance, instructions, and program directives, as well as annual
business rules and file specifications to assist States in identifying, collecting, and reporting data,
either through the CSPR or other required annual data submissions. The Department also offers
SEAs program-specific technical assistance and training.

When it received the SEAs’ data, the Department’s EDFacts system performed two automated edit
checks and program staff assigned to EDFacts data groups assessed the data for completeness and
reasonableness. The first edit check was a limited error check, in which data were tested to ensure
that the data complied with automated business rules. If the data did not comply with the rules, the
Department rejected the SEA data submission and required the SEA to resolve the deficiency
before it resubmitted the data. The second edit check was a postsubmission check designed to
detect and flag data that fall outside an expected range. These flags occurred when questions
emerged about the reasonableness of the data or the Department needed more information from the
SEA regarding the data it reported. The SEA either verified that the flagged data were correct or
revised and resubmitted the corrected data.

APR Process
ODS serves as the lead office in developing the APR and is responsible for approving the final
version to be signed by the Secretary, but it relies on program offices and others to verify the
accuracy and reliability of reported data. Both ODS and program office officials stated that
EDFacts staff assess data quality for data reported in the APR through EDFacts. EDFacts
officials stated that program offices drive data quality and ultimately determine whether data are
acceptable. Goal and Strategic Objective Leaders—agency officials who are responsible for the
achievement of each performance goal, as well as objectives and subgoals under each strategic
goal—provide ODS with quarterly updates on metrics (or identify when data are expected),
analyze progress, and verify data sources and data limitations (an area in which we found some
issues, as discussed in further detail below). EDFacts staff are also involved in the process
through their use of diagnostic tests that are designed to assess the consistency and validity of data
values. This reflects the shared responsibility across multiple Department offices for ensuring the
quality of K–12 data, with ODS also playing an important role in the process.

The GPRA Modernization Act requires that Federal agencies collect and validate performance data
used to report progress toward performance goals. OMB Circular No. A-11 provides detailed
guidance for the agency goal management process, as well as for the related performance data
verification and validation process, which supports both reporting progress toward goals and data
reliability efforts in general. In addition, OMB Circular A-123 and GAO’s “Standards for Internal
Control in the Federal Government” both emphasize the importance of communication throughout
Federal agencies to enable personnel to perform key roles in achieving objectives, addressing risks,
and supporting systems of internal control. It is critical, then, that the Department’s system of
Final Report
ED-OIG/A06O0001                                                                                          Page 7 of 20

internal control over the accuracy and reliability of reported data be universally understood and
effectively implemented.

Oversight of SEA Internal Controls
The Department relies on the SEAs to report accurate data and requires the SEAs to sign
certifications for the programs we reviewed. However, the Department does not always verify that
the SEAs have adequate internal controls to ensure that data LEAs and other State agencies
submitted are accurate and reliable, or tested to ensure the controls and systems are working as
intended. From FY 2010 through FY 2015, the U.S. Department of Education Office of Inspector
General (OIG) issued several audit reports pertaining to data quality in various programs (see
Attachment 2). The Department responded to the recommendations made in these reports by
modifying some monitoring processes, and providing additional training to program staff.

For example, the Office of Migrant Education (OME) makes its State and local level monitoring
protocols available on the Department’s Web site. These protocols, which OME uses when
conducting on-site program reviews of migrant education programs, specifically require reviewers
to look at the accuracy of certificate of eligibility documentation 8 and require a physical review of
the certificate of eligibility form as well as follow up interviews of a sample of migrant families.
OME’s State and local monitoring protocols also call for the review of the SEA’s child count
procedures to ensure that they conform to the child count explanation OME accepted in the State’s
CSPR. As a result of its monitoring reviews, OME has established grant conditions for repeated
SEA data issues. However, we learned that OME conducted only three State monitoring visits due
to staffing limitations in FYs 2013 through 2015. To ensure reliability, OME has been performing
desk reviews of interview data related to child eligibility that States submit via the CSPR and has
continued its efforts to resolve outstanding monitoring findings from prior years’ monitoring
reports.

OME also relies to some extent on the work of external auditors to determine whether entities
receiving funds are submitting reliable data. The Compliance Supplement for 2014 describes the
responsibility of external auditors with respect to the scope of single audits, and includes a section
regarding the “counts of Migrant children eligible for funding purposes.” The Compliance
Supplement suggests audit steps to determine whether SEAs, LEAs, and other local operating
agencies established, implemented, and accurately reported in the CSPR Part I a quality control
process that ensures an accurate eligible-child count and meets the requirements of the
Department’s regulations.

The Office of Safe and Healthy Students (OSHS), which is tasked with oversight of Title I, Part D
programs, designed a monitoring plan for FY 2015 that includes a guiding question about an
SEA’s process for data collection and a list of acceptable evidence, such as evidence that an SEA
provided technical assistance to subgrantees on how to efficiently and effectively collect and use
data. OSHS’s plan does not include tests specifically for the quality of the ND child count.
During FY 2013, OSHS conducted three onsite monitoring visits that resulted in findings at two of

8
  The certificate of eligibility includes eligibility information related to the qualifying move and work, and requires the
parent, guardian, spouse, or worker to sign acknowledgement of the purpose of the form, and attest that the
information to the best of their knowledge is true. The certificate of eligibility also includes an eligibility data
certification that must be signed by the interviewer/recruiter and the designated SEA reviewer.
Final Report
ED-OIG/A06O0001                                                                                           Page 8 of 20

the SEAs. OSHS found a lack of subgrantee monitoring by one SEA and the exclusion of the ND
program from the SEA’s overall monitoring approach. Unlike OME, however, OSHS does not
require external auditors to assess States’ or other entities’ controls over the quality of ND child
count data.

The three data elements we reviewed were all covered by management certifications the
Department required program offices to complete using terminology shown in Table 1 below. To
provide reasonable assurance for acceptable data submissions, the Department should amend the
certification process and terminology to have the SEA assert that it has internal controls over data,
which are tested and working as intended, or that the SEA has fully disclosed any data limitations
and corrective actions. Including such language in the certification itself would not only provide
criteria against which the Department or external auditors could monitor or audit, but also
encourage States to think carefully about their data quality processes and promote a more open and
honest dialogue between Federal and State officials. An OESE official stated that OESE is
considering revising the SY 2014–2015 CSPR 9 management certification language to include a
statement that the State has implemented internal controls and taken action to ensure data are true,
reliable, and valid. OESE is also considering including a statement in the certification that allows
for data submission with disclosure of known data limitations and the State’s corrective actions.

Table 1. Management Certification Language
         Migrant Certificate of                       ED Form 4376                             CSPR Part II
              Eligibility                             (ND child count)                  (Migrant child count and ACGR)
             (Migrant child count)
    I certify that based on the              I certify that the State educational   The CSPR Part II is now ready to be
    information provided to me, which        agency has determined that the data    certified and officially submitted to
    in all relevant aspects is reflected     provided in Parts I and II meet the    the U.S. Department of Education.
    above, I am satisfied that these         requirements of Title I, ESEA. The     By clicking the "Certify" button
    children are migratory children as       information provided in this report    below I acknowledge the following:
    defined in 20 United States Code         is, to the best of my knowledge,
    (U.S.C.) 6399(2) and implementing        complete and accurate.                 •     That I am the proper certifying
    regulations, and thus eligible as such                                                official for the SEA, and am
    for MEP [Migrant Education                                                            properly authorized by the SEA
    Program] services.                                                                    to validate and submit all
                                                                                          sections of this report.
    I hereby certify that, to the best of                                           •     That all data included in this
    my knowledge, the information is                                                      part of the SY 2012–2013
    true, reliable, and valid and I                                                       CSPR, to the best of my
    understand that any false statement                                                   knowledge, are true, reliable,
    provided herein that I have made is                                                   and valid.
    subject to fine or imprisonment
    pursuant to 18 U.S.C. 1001.


According to GAO’s “Standards for Internal Control in the Federal Government,” reliable internal
and external data sources should provide data that are reasonably free from error and bias and
faithfully represent what they purport to represent; thus, the Department should evaluate both
internal and external sources of data for reliability. In addition, the Uniform Administrative
Requirements, Cost Principles, and Audit Requirements for Federal Awards, Title 2 of the Code of

9
    SY 2014-2015 CSPR reporting begins in mid-December 2015.
Final Report
ED-OIG/A06O0001                                                                       Page 9 of 20

Federal Regulations (C.F.R.), § 200.61, states internal controls are a process implemented and
designed to provide reasonable assurance regarding reliability of reporting for internal and external
use. We further noted that 2 C.F.R. § 200.208 states that, unless prohibited by Federal statutes or
regulations, each Federal awarding agency or pass-through entity is authorized to require non-
Federal entities to submit certifications and representations required by Federal statutes, or
regulations, on an annual basis, and that submission may be required more frequently if a non-
Federal entity fails to meet a requirement of a Federal award.

Selected Data Element Corrections
For SY 2012–2013, either EDFacts edit checks or program office data reviews identified
inaccurate migrant child count, ND child count, and ACGR data that SEAs reported to the
Department in the CSPR or other required annual report, resulting in data revisions and, in some
cases, recalculation of award allocations. While the Department was able to identify errors and
work with the SEAs for correction, the Department did not follow up with the SEAs to assess how
those errors occurred when the SEA officials had provided certifications attesting the data’s
accuracy. In addition, the spreadsheet maintained by EDFacts as evidence of SEA contact
regarding reported data issues, was not detailed enough to consistently track information, such as,
who identified the erroneous data (SEA, EDFacts staff or program staff), SEA response, timing or
final resolution.

Migrant Education Program Child Count
In SY 2012–2013, one State had to revise its overall migrant child count. In April 2014, the State
received comments as a result of EDFacts edit checks regarding information submitted for the
CSPR Part I. EDFacts’ comments did not directly pertain to the child counts; however,
corrections made to one file required a revision of the State’s overall migrant child count from
1,734 to 1,802 (as reported in CSPR Part II for SY 2012–2013).

OESE’s EDFacts Data Steward found that the State provided contradictory information in its
CSPR Part II data submission regarding its summer/intersession child count for SY 2012–2013. In
its CSPR, the State explained that a wet growing season resulted in a decrease in its count of
eligible migrant children in its summer/intersession. However, OESE noted that the State actually
reported a 27.1 percent increase in this count. In response to OESE inquiries, the State identified a
system coding error, and ultimately recertified its corrected migrant child count for SY 2012–2013
summer/intersession downward from 1,496 to 613. However, OESE did not determine why the
SEA’s controls did not detect the coding error prior to certifying to the accuracy of the data.

Neglected or Delinquent Program Child Count
During FY 2013, OESE required two States to correct their reported child counts. The two SEAs
had to correct ND child count data after certifying the data to be complete and accurate. As a
result, OESE recalculated FY 2014 funding allocations for those SEAs.

OESE staff noted a calculation discrepancy in one State’s submission used for the FY 2014
allocation. The State confirmed that the FY 2013 count, as originally submitted and certified as
complete and accurate, was overstated due to a clerical error. The State submitted and certified a
revised count, requiring the recalculation of its FY 2014 allocation and resulting in a downward
adjustment to its FY 2015 award.
Final Report
ED-OIG/A06O0001                                                                       Page 10 of 20

The other State’s FY 2015 funding allocation data submission included a statement explaining the
difference between the FY 2015 child count and the FY 2014 child count. This explanation
indicated that the State might have made an error in its FY 2013 child count. After OESE
contacted the State, the State confirmed that the count originally submitted and certified as
complete and accurate for FY 2013 was incorrect. OESE required the State to submit a revised
child count for FY 2013, resulting in a downward adjustment to its FY 2015 award because the
FY 2014 allocation had already been made.

Adjusted Cohort Graduation Rate
In SY 2012–2013, after certifying that the data in their original submissions were true, reliable,
and valid, 13 SEAs had to submit corrected ACGR data. One of these SEAs reported prior year
ACGR data to meet the current year’s required CSPR reporting deadline, and provided a
management certification that certified to the Department that this data represented the current year
data. OESE contacted the SEA and requested that the SEA resubmit the data during the reopen
correction period, which ended in May 2014. However, the SEA was not able to provide the
corrected data until August 2014, which indicates that its SY 2012–2013 ACGR data were not
available to report in January 2014 as required.

The Department may be hampered in its ability to oversee the quality of ACGR data due to gaps
created by an OESE reorganization. The Data Steward told us during the audit that the ACGR is
not currently being monitored by the Office of State Support, a new office within OESE that
resulted from the merging of a number of previously separate offices with various responsibilities.
This reorganization was cited as the primary reason for the office’s lack of monitoring of States’
ACGRs. However, the Office of State Support plans to develop a monitoring process and pilot it
in fall 2015. Additionally, the Office of State Support does not currently provide guidance to
SEAs on establishing controls over the accuracy and reliability of ACGR data that the SEAs
receive from the LEAs and subsequently report to the Department. Lastly, the Office of State
Support also does not require external auditors to assess States’ or other entities’ controls over the
quality of ACGR data.

Although EDFacts edit checks and program office reviews identified data errors, the Department
failed to follow up with the States to determine how these errors occurred when the State officials
certified to the Department that the data were accurate and complete. The fact that State official
certified to the quality of the submitted data in spite of data quality issues raises questions about
the adequacy of State controls over the quality of their data. The Department’s reliance on
inaccurate or unreliable data necessitated revisions to the Neglected and Delinquent Program
award allocations for two SEAs as noted above. The Department also risks using inaccurate or
unreliable data when describing its progress toward meeting performance goals. If the Department
requires State officials to certify to the quality of data, it must follow up when data problems arise
to ensure that the causes of misreported data are identified and corrected and, if necessary, hold
certifying officials accountable if they did not meet the conditions of the certification statement.
The management certification process provides the Department a level of assurance that SEAs
have assessed the strength or weakness of data controls—both at the SEA and LEA levels—before
they submit required data. When States submit data with significant errors, it is important for the
Department to obtain some level of assurance that the SEA’s data quality procedures are adequate
to ensure that those errors are not repeated, or, if there are known weaknesses to require full
disclosure of any data issues found during testing. An SEA’s assertion of data quality enables the
Department and auditors to assess the SEA’s controls supporting its data submissions. When there
Final Report
ED-OIG/A06O0001                                                                      Page 11 of 20

are known data quality weaknesses, an SEA’s full disclosure is necessary for the Department to
meet its disclosure requirements on the APR and to ensure that data are used appropriately for
critical decision-making.


Recommendations

We recommend that the Deputy Secretary of Education—

1.1    Review how management certifications are used across all programs in the Department,
       and ensure that certification language provides an assurance of the quality of data, clear
       accountability, and adequate disclosure of known data limitations.

1.2    Review how monitoring protocols are used across all programs in the Department to ensure
       that they contain steps to assess how well SEA or LEA procedures are working to provide
       accurate data for key data used in performance reporting or funding decisions.

1.3    Develop a formal tracking procedure for SEA data issues for use by EDFacts and principal
       offices that identifies how the data issue was identified, why it occurred, and resolution
       actions and timeframes.

1.4    Require all principal offices to follow up on known data errors where the SEA has signed a
       management certification to determine the causes of the error and ensure that the SEA
       develops procedures to ensure that it provides accurate information in accordance with the
       management certification.

1.5    Revise the Compliance Supplement, as appropriate, to address areas where external
       auditors should determine whether SEAs have controls in place to ensure that data
       collected from LEAs and other State agencies are accurate and reliable and support the
       management certifications they sign.


ODS Comments

In a written response to the draft report, ODS did not agree with OIG’s overall evaluation of the
value and effectiveness of the Department’s current system of controls over SEA data accuracy
and reliability, stating that it believed the draft report did not contain a sufficiently complete
acknowledgement or description of its efforts in this area. However, ODS agreed to take action in
accordance with each recommendation and stated it would continue to encourage more accurate
and reliable reporting of data by SEAs. In response to Recommendation 1.1, ODS agreed to
review management certification language for K–12 data submitted by SEAs to EDFacts and other
major K–12 collections and to update certification language as appropriate. In response to
Recommendation 1.2, ODS stated it will review and enhance, as necessary, monitoring protocols
for major programs that require data to be submitted by State agencies to EDFacts. In response to
Recommendations 1.3 and 1.4, ODS agreed to build on its existing tracking procedures and data
quality process, to include noting how data issues are identified and having principal office staff
follow up with SEAs on the cause of any such data issues and appropriate corrective actions. In
response to Recommendation 1.5, ODS stated it will work with OIG to revise the Compliance
Final Report
ED-OIG/A06O0001                                                                     Page 12 of 20

Supplement, for certain programs, to address areas where external auditors may assess SEA
controls over the accuracy and reliability of data collected from LEAs and support the
management certifications the SEAs sign. ODS also provided technical comments, which we
considered and made changes as appropriate.



                  OBJECTIVES, SCOPE, AND METHODOLOGY


The original objective of the audit was to determine what actions the Department has taken,
including the use of management certifications, to ensure the accuracy and reliability of key K–12
performance data. The audit objective was revised to determine what actions the Department has
taken, including making use of management certifications, to ensure the accuracy and reliability of
select K–12 data reported in the APR and select OESE K–12 data.

We performed work at principal and program offices responsible for creating the APR and
administering select programs to determine what the Department did to ensure the accuracy and
reliability of three selected data elements for FY 2013. We performed audit fieldwork from
March 2014 through July 2015 at principal and program offices in Washington, D.C., and at our
offices in Dallas, Texas and New York, New York. We held an exit conference with Department
officials on August 4, 2015, to discuss the results of the audit.

Selection of Case Study Data Elements
To accomplish our audit objective, we judgmentally selected three K–12 APR and CSPR data
elements for case study that were reported in the FYs 2012 and 2013 APRs or used by OESE to
make critical management or funding decisions. We considered approximately 30 K–12 data
elements sourced from 16 major CSPR data categories and 14 data elements used to measure 4 of
6 Department strategic goals. We prioritized the 30 data elements based on the following factors:

   •   APR data and data sources,
   •   data reported by States in the CSPR,
   •   demonstrated broad usage of the data as evidenced by data queries of EDFacts data users,
   •   associated programs having large funding amounts, and
   •   data element tied directly to funding or eligibility to receive funding.

We judgmentally selected the three data elements (1) migrant child count, (2) ND child count, and
(3) ACGR. The results of these case studies should not be projected to other data elements not
included in our review.

For the three data elements selected, we determined how the Department ensured that SEAs have
internal controls and processes for collecting and verifying data they submitted to the Department.
We also determined what data verification and validation processes the Department used for data
received from the SEAs. We gained an understanding of the actions the Department took to ensure
data accuracy and reliability before data collection, at the point of data reporting, and after data
submission for FY 2013 and FY 2014. Due to the timing of reporting and the unavailability of
some FY 2014 data, we concentrated our case studies on FY 2013.
Final Report
ED-OIG/A06O0001                                                                                   Page 13 of 20

To achieve our objective, we performed the following procedures.

       •   Interviewed management and staff from Institute of Education Sciences National Center for
           Education Statistics, ODS, Office of General Counsel, and Risk Management Service, as
           well as from principal and program offices with responsibility for ensuring reliable migrant
           child count, ND child count, and ACGR data.
       •   Reviewed OESE’s March 10, 2014 Dear Colleague Letter regarding Fraud in
           Title I-Funded Tutoring Programs and compared the language used in management
           certifications for the three data elements selected for (a) consistency and quality of the
           certification terminology, (b) an assertion as to accuracy, (c) an acknowledgement of
           personal liability and that Federal funds are involved, and (d) inclusion of the relevant
           Federal statute and potential for penalties—key elements of an effective certification as
           previously identified by OIG.
       •   Analyzed, for the three data elements selected, the number of times data were corrected 10
           after SEA submission and whether the Department or the SEA identified the inaccuracy.
           We also determined what actions the Department took when SEAs reported inaccurate,
           unreliable, and/or incomplete data.

Internal Controls
We performed a review of internal controls applicable to the Department’s processes to ensure the
accuracy and completeness of SEA-reported data. Specifically, we reviewed

       •   Federal laws and regulations related to data quality and accountability for data quality;
       •   prior audit reports from OIG and other agencies related to data quality and accountability
           for data quality;
       •   EDFacts policies and procedures related to data submission, reporting, and validation;
       •   ODS policies and procedures for collecting data reported in the APR; and
       •   OESE and program office policies, procedures, monitoring protocols, instructions,
           guidance, Dear Colleague Letters, and technical assistance provided to SEAs.

Data Reliability
To achieve our objective, we assessed the Department’s controls over data reliability with regards
to the three data elements selected. We did not do any additional testing beyond what the
Department had done. The finding presents how the Department can improve its processes to
better ensure the reliability of its data.

We conducted this performance audit in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and conclusions based on our
audit objectives. We believe the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




10
     The Department could not identify the number of times ACGR data were changed or who initiated the changes.
Final Report
ED-OIG/A06O0001                                                                      Page 14 of 20



                             ADMINISTRATIVE MATTERS


Corrective actions proposed (resolution phase) and implemented (closure phase) by your office
will be monitored and tracked through the Department’s Audit Accountability and Resolution
Tracking System (AARTS). ED policy requires that you develop a final corrective action plan for
our review in the automated system within 30 days of the issuance of this report. The corrective
action plan should set forth the specific action items, and targeted completion dates, necessary to
implement final corrective actions on the findings and recommendations contained in this final
audit report. An electronic copy of this report has been provided to your Audit Liaison Officer(s).

In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector General
is required to report to Congress twice a year on the audits that remain unresolved after six months
from the date of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office
of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us during this review. If you have any questions, please call
Daniel Schultz at 646-428-3888.

                                             Sincerely,

                                              /s/

                                             Patrick J. Howard
                                             Assistant Inspector General for Audit


Electronic cc: Laura Ginns, Audit Liaison Officer, ODS
               Heather Acord, Audit Liaison Officer, ODS
               Teresa Cahalan, Audit Liaison Officer, Institute of Education Sciences
               Delores Warner, Audit Liaison Officer, OESE

Attachments
Final Report
ED-OIG/A06O0001                                                            Page 15 of 20

      Attachment 1: Acronyms, Abbreviations, and Short Forms Used in This
                                  Report

ACGR                     Adjusted Cohort Graduation Rate

APR                      Annual Performance Report

CAP                      Corrective Action Plan

C.F.R.                   Code of Federal Regulations

Compliance Supplement    OMB Circular A-133 Compliance Supplement for 2014

CSPR                     Consolidated State Performance Report

Department               U.S. Department of Education

ED Form 4376             Annual Report of Children in Institutions for Neglected or
                         Delinquent Children, Adult Correctional Institutions, and
                         Community Day Programs for Neglected or Delinquent Children

ESEA                     Elementary and Secondary Education Act

FY                       Fiscal Year

GAO                      Government Accountability Office

GPRA Modernization Act   Government Performance and Results Act Modernization Act
                         of 2010

K–12                     Kindergarten Through 12th Grade

LEA                      Local Educational Agency

Migrant child count      Migrant Education Program Child Count

ND child count           Prevention and Intervention Programs for Children and Youth
                         Who are Neglected, Delinquent or At Risk

ODS                      Office of the Deputy Secretary

OESE                     Office of Elementary and Secondary Education

OIG                      Office of Inspector General

OMB                      Office of Management and Budget
Final Report
ED-OIG/A06O0001                                         Page 16 of 20

OME               Office of Migrant Education

OSHS              Office of Safe and Healthy Students

SEA               State Educational Agency

Secretary         Secretary of Education

SY                School Year

U.S.C.            United States Code
       Final Report
       ED-OIG/A06O0001                                                                                     Page 17 of 20

                 Attachment 2: Prior OIG Reports That Noted Data Quality Issues
OIG Audit Report, ACN, Date Issued                                              Audit Results
“U.S. Department of Education’s           The OIG found that the Department relied on SEAs to ensure the accuracy of the
Implementation and Oversight of           information but did not verify that the SEAs had policies and procedures to ensure
Approved Elementary and Secondary         accuracy. In addition, the Department did not require SEAs to provide an assurance
Education Act Flexibility Requests”       statement covering the accuracy of the data submitted and did not have procedures
A04N0012                                  requiring SEAs to disclose any limitations of the information, data, or validation process.
January 2015
“U.S. Department of Education’s and       The OIG found that, although the Department and all five SEAs reviewed had systems of
Five State Education Agencies’ Systems    internal control designed to prevent and detect inaccurate, unreliable, or incomplete
of Internal Control Over Statewide Test   statewide test results, these systems did not always require corrective action if they found
Results”                                  indicators of inaccurate, unreliable, or incomplete statewide test results. The OIG also
A07M0001                                  found the Department could improve its monitoring of States’ test results by requiring
April 2014                                SEAs to provide an explanation for data that the Department’s data collection system
                                          flagged as either incorrect or outside an anticipated range. It also could improve its
                                          monitoring of SEAs and LEAs by resuming reviews of test administration procedures
                                          during onsite monitoring visits and having SEAs’ systems of internal control over
                                          statewide test results evaluated during standards and assessment peer reviews.

“U.S. Department of Education’s           OIG found that the overall data verification and validation approach initially described by
Implementation of Government              the Department, in its APR, was not actually being used by its principal offices.
Performance and Results Act”              Additionally, we noted that the Department did not specifically identify the means used to
A19M0005                                  verify and validate measured values; the level of accuracy required for the intended use of
January 2014                              data; any limitations to the data, including how the agency will compensate for such
                                          limitations if needed to reach the required level of accuracy; and did not always accurately
                                          identify the sources for the data.

“U.S. Department of Education’s and       The OIG found that the Department relied on the SEAs to report accurate, reliable, and
Selected States’ Oversight of the 21st    complete 21st Century Community Learning Centers Program performance data through
Century Community Learning Centers        the Profile and Performance Information Collection System. However, it did not ensure
Program”                                  that the SEAs validated the data that subgrantees submitted.
A04L0004
June 2013
“Centers for Independent Living           The OIG found that RSA had not provided required and consistent monitoring and
Compliance, Performance, Recovery Act     oversight of the Centers for Independent Living as specified in Title VII of the
Reporting, and Monitoring”                Rehabilitation Act of 1973, as amended. The OIG also found partially supported
A06K0011                                  performance information reports and inadequately documented files at the Centers for
September 2012                            Independent Living reviewed. The OIG also found that the documentation provided by
                                          eight Centers for Independent Living to support numbers of consumers served was
                                          inconsistent with the numbers reported on the annual 704 Performance Reports.

“The Effectiveness of the Department’s    Although the OIG found that the Department’s processes to ensure the accuracy and
Data Quality Review Processes”            completeness of recipient reported data were generally effective, it also found instances of
A19K0010                                  recipient-reported data that were inconsistent with data in the Grants Administration and
August 2011                               Payment System, contract file documentation, or other data elements within the recipient
                                          reports. These anomalies still existed after the Department had completed its formal data
                                          quality review processes and after the related recipient correction period.

“Georgia Department of Education’s        The OIG found that neither a Georgia LEA nor the Georgia Department of Education
Controls Over Performance Data Entered    established adequate systems of internal control to ensure that accurate, reliable, and
in EDFacts”                               complete data were entered in EDFacts. As a result, inaccurate or unsupported data were
A04J0003                                  reported, including dropouts, graduates, and discipline incidents.
April 2010
Final Report
ED-0 I G/ A060000 I                                                                                                                Page 18 of20

                                         Attachment 3: ODS Comments



                                                 UNITED STATES DEPARTMENT OF EDUCATION 

                                                        OFFICE OF THE DEPl!TY SECRETARY 





                                                              January 13, 2016

          TO:                Patrick J. Howard
                             Assistant Inspector General for Audit
                             Office of Inspector General

          FROM:              Mark Washington
                             Performance Improvement Officer
                             Office ofthe Deputy Sccrotary

          SUBJECT:           Comments on Draft Audit Report HManagc:.1m:nt Certifications of Data
                             Reliability" (Control Number ED ­ 01G/A0600001)


          Thank you fot: providing the Office ofthe Deputy Secretary with an opportunity to review and
          respond to the finding and recommendations in the Office oflnspector General's (OIG) draft
          audit report on "Management Certifications ofData Reliability" OIG Control Number ED­
          OIG/A060000l.                                        .

          In accordance with the Government Performance and Results Act (GPRA) Modernization Act of
          2010 (GPRAMA), the Department's framework for perfonnance management starts with the ·
          Strategic Plan, including its Agency Priority Goals, which serve as the foundation for
          establishing long-term priorities and developing performance goals, objectives., and metrics by
          which the Department can gauge achievement of its stated outcomes. Progress toward the
          Department's Strategic Plan is measured using data-driven review and analysis to promote
          active management engagement across the Department and is reported annually. GPRAMA
          requires agencies to describe the accuracy and reliability ofdata presented in the Annual
          Performance Report (APR). Details of how the Deportment assesses the completeness and
          reliability, including known limitatioo.s, of the data reported are presented in an Appendix to the
          APR. Through a process ofcontinuous improvement, the Department continues to assess its
          validation process and welcomes input from the OIG and others.

          We are pleased that this draft report acknowledges a number ofthe actions the Department has
          talcen to improve the completeness and reasonableness of State Education Agency (SEA) data
          reported in the Department's APR. This includes providing written guidance, using system edit
          checks, requiring management certifications, and conducting post-submission data analysis. The
          Department also offers various forms oftechnical assistance and training sessions. The draft
          report states that the Department docs, in fact, take actions to encourage data quality before,
          during, and after an SEA submits data. We appreciate that OIG commends and affirms that the



                                           400 MARYLAND AV.E., SW, WASHINGTON, DC 20202
             Tl>c Dcport11trnt ofEdua>tion '$ mission u to p romote stutlcrt at:ltinem~nt antiJXeporation /or g/<Jbal com~titi~n= by
                                           /ostmng educational acellt~~Ce and ensunng "'tual access.
Final Report
ED-OIG/A06O0001                                                                                        Page 19 of 20




        2


        Department has adequate controls to provide reasonable assurance that the K-12 reported data
        are complete and reasonable.

        During 2015, the Department strengthened its approach to data verification and validation, and
        will continue to do so. We are committed to creating a culture of continuous improvement to
        ensure more accurate and reliable data and we work continuously to promote better SEA controls
        over data quality. As such, we agree with OIG's view that the Department could continue to
        make improvements to its procedures to enhance data accuracy and reliability as we have for a
        number of years, despite practical limitations.

        Our responses to the draft finding are set forth below.

        While there is information with which we agree in the draft report, it does not contain a
        sufficiently complete acknowledgement or description of the efforts of the Department has made
        and is making in this area, and it does not properly value the positive aspects ofthese efforts.
        Thus, we do not agree with the overall evaluation of the value and effectiveness of our current
        system of technical assistance, assurances, data checks, certifications, and encouragement to
        States to submit corrections and updates when appropriate. As such, we are providing also
        attached a set oftechnical comments to improve the accuracy of the report.


        Finding: The Department Could Ensu re More Accurate and Reliable Data by Pr omoting
        Better SEA Controls O ver Data Quality

        R esponse: As noted above, the OIG does not give sufficient credit to the successes of the
        Department's current mix of tools to support SEAs on providing valid and reliable data. In spite
        of that, the Department will continue to encourage more accurate and reliable reporting of data
        by SEAs. The Department will review management certification language, for K-12 data           ·
        submitted by SEAs to EDFacts and other major K-12 collections, to determine that the
        certifications provide clear statements that the SEA is responsible for having an effective internal
        control structure that provides reasonable assurance that the data submitted to the Department are
        reasonably free from error and bias and faithfully represent what they purport to represent and
        for disclosing any known data limitations. As appropriate, certification language will be updated.
        The Department will also review monitoring protocols for major programs that require data be
        submitted by State agencies to EDFacts, to see if enhancements should be made.

        For data submitted under a signed management certification, the Department plans to build on its
        existing tracking procedures to track data issues including how those issues are identified.
        Additionally, the Department will build on its data quality process, for data submitted under a
        signed management certification, for principal office staff to follow up with SEAs on the cause
        of data issues and appropriate corrective actions. The Department will work with the OIG to
        revise, for certain programs, the Compliance Supplement to address areas where external
        auditors could determine whether SEAs have controls in place to ensure that data collected from
        LEAs are accurate and reliable and support the management certifications SEAs sign.
Final Report
ED-OIG/A06O0001                                                                                    Page 20 of 20




        3


        Please Jet us know if you have any questions or need further information about any of our
        comments and responses. We appreciate the effort that went into the field work and the report
        and thank you for the opportunity to review and respond to the draft.




                                                    Mark Washington /
                                                    Performance lmproJ~m~ t Officer
                                                    Office of the Deputy Secretary


        Enclosures:
        Corrective Actions
        Technical Comments