oversight

The Department's Implementation of the Government Performance and Results Act Modernization Act

Published by the Department of Education, Office of Inspector General on 2014-01-27.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

     The Department’s Implementation of the Government 

       Performance and Results Act Modernization Act



                                 FINAL AUDIT REPORT





                                    ED-OIG/A19M0005

                                      January 2014



Our mission is to promote the                          U.S Department of Education
efficiency, effectiveness, and                         Office of Inspector General
integrity of the Department's                          Washington, DC
programs and operations.
                                   NOTICE
Statements that managerial practices need improvements, as well as other conclusions
and recommendations in this report, represent the opinions of the Office of Inspector
General. Determinations of corrective action to be taken will be made by the appropriate
Department of Education Officials.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by
the Office of Inspector General are available to members of the press and general public
to the extent information contained herein is not subject to exemptions in the Act.
                                   UNITED STATES DEPARTMENT OF EDUCATION
                                                        OFFICE OF INSPECTOR GENERAL

                                                                                                                     AUDIT SERVICES


                                                          January 27, 2014
Memorandum
TO:	                 James H. Shelton
                     Acting Deputy Secretary
                     Office of the Deputy Secretary

FROM:	               Patrick J. Howard /s/
                     Assistant Inspector General for Audit

SUBJECT:	            Final Audit Report
                     The Department’s Implementation of the Government Performance and
                     Results Act Modernization Act
                     Control Number ED-OIG/A19M0005

Attached is the final audit report that covers the results of our audit of the Department’s
implementation of the Government Performance and Results Act Modernization Act. An electronic
copy has been provided to your Audit Liaison Officer (ALO). We received comments from the
Office of the Deputy Secretary (ODS) generally concurring with the findings in our draft report.

Corrective actions proposed (resolution phase) and implemented (closure phase) by your office will
be monitored and tracked through the Department’s Audit Accountability and Resolution Tracking
System. Department policy requires that you develop a final corrective action plan (CAP) for our
review in the automated system within 30 days of the issuance of this report. The CAP should set
forth the specific action items and targeted completion dates necessary to implement final corrective
actions on the findings and recommendations contained in this final audit report.

In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector General is
required to report to Congress twice a year on the audits that remain unresolved after 6 months from
the date of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office of
Inspector General are available to members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us during this review. If you have any questions, please call
Michele Weaver-Dugan at (202) 245-6941.

Enclosure

cc: Heather Acord, ALO, ODS




 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
                                              TABLE OF CONTENTS


                                                                                                                                 Page

EXECUTIVE SUMMARY ...........................................................................................................1


BACKGROUND ............................................................................................................................3


AUDIT RESULTS .........................................................................................................................4


          FINDING NO. 1 – The Department Generally Adequately Identified

                          and Clearly Articulated Agency Priority Goals, 

                          Milestones, and Performance Indicators .....................................4


          FINDING NO. 2 – The Department Has Not Accurately or Adequately

                           Disclosed Information Related to its Data 

                           Verification and Validation Process.............................................6 


          FINDING NO. 3 – Quarterly Performance Reviews Have Generally

                          Been Performed as Required ......................................................10 


OBJECTIVES, SCOPE, AND METHODOLOGY ..................................................................12


ENCLOSURE 1: Table of Agency Priority Goals


ENCLOSURE 2: Acronyms/Abbreviations/Short Forms Used in this Report


ENCLOSURE 3: Department Response to the Draft Audit Report

Final Audit Report
ED-OIG/A19M0005                                                                      Page 1 of 12



                                EXECUTIVE SUMMARY



The Government Performance and Results Act Modernization Act of 2010 (GPRA
Modernization Act) was signed into law in January 2011. Its intent was to create an enhanced
government-wide planning and reporting framework, retaining and amplifying aspects of the
GPRA of 1993, and introducing new requirements that include the establishment of agency
priority goals (APGs), additional data validity and reliability reporting, and quarterly
performance reporting. The GPRA Modernization Act is intended to serve as a foundation for
engaging leaders in performance improvement and creating a culture where data and empirical
evidence play a greater role in policy, budget, and management decisions.

The objectives of our audit were to determine whether the U.S. Department of Education
(Department) has 1) identified and clearly articulated priority goals, milestones, and performance
indicators; 2) established a process for ensuring the validity and reliability of data; and 3)
conducted and reported on the results of quarterly performance reviews to provide information to
the public on its performance and operations.

Overall, we found that the Department has generally implemented the GPRA Modernization Act
as required as related to our objectives, but improvements are needed with regard to disclosures
related to congressional input and data verification and validation. Specifically, we found that
the Department generally identified and clearly articulated APGs, milestones, and performance
indicators as required by the GPRA Modernization Act. It also generally conducted and reported
on quarterly performance reviews of its APGs, although supporting documentation could be
better maintained. However, we noted that the Department did not identify on the government-
wide website - Performance.gov - how congressional views were incorporated into the
establishment of its APGs. In addition, while the Department has established processes for
ensuring the accuracy and reliability of data used to measure progress towards its APGs, it has
not accurately or adequately disclosed relevant information in its Annual Performance Plan
(APP) or Annual Performance Report (APR) as required. As a result, the public may have less
confidence that Congress and the Department are in agreement on the immediate priorities of the
agency and that the data presented in performance reports is credible, and they may be unaware
of any limitations of the data that would provide important context for understanding it.

To correct the weaknesses identified, we recommend that the Deputy Secretary:

   •	 Develop and implement formal written internal procedures related to the GPRA process,
      to include applicable policy on congressional consultations, reporting on such
      consultations, and roles and responsibilities of involved Department staff.
   •	 Clearly describe in all applicable performance reports and plans the Department’s data
      verification and validation process for each APG and include complete and accurate
      disclosures related to data limitations and sources in accordance with GPRA
      requirements.
   •	 Establish and implement formal written internal policies and procedures regarding the
      quarterly performance review process, to include documentation requirements and
      submission timeframes.
Final Audit Report
ED-OIG/A19M0005                                                                      Page 2 of 12

In response to the draft audit report, the Office of the Deputy Secretary (ODS) generally
concurred with the findings in the report and provided information on progress made related to
recommendations. It did not concur with the wording used to describe Finding No. 2 and the
related recommendation. Specifically, while ODS generally agreed with the substance of the
finding and recommendation, it noted that the wording used in the finding and recommendation
could be misunderstood and may imply to a reader that the related performance data was not
accurate, or that ODS was not disclosing the data appropriately.

ODS’s comments are summarized at the end of each applicable finding. The full text of ODS’s
response is included as Enclosure 3 to this report. No changes were made to the report as a result
of the response.
Final Audit Report
ED-OIG/A19M0005                                                                     Page 3 of 12



                                      BACKGROUND



The Government Performance and Results Act Modernization Act of 2010 (GPRA
Modernization Act) was signed into law in January 2011. Its intent was to create an enhanced
government-wide planning and reporting framework aimed at taking a more crosscutting and
integrated approach to focusing on results and improving government performance. The GPRA
Modernization Act is intended to serve as a foundation for engaging leaders in performance
improvement and creating a culture where data and empirical evidence play a greater role in
policy, budget, and management decisions. Among other things, it requires agencies to develop
agency priority goals (APGs) every 2 years. These goals are to reflect the highest priorities of
the agency, as identified by the head of the agency, and be informed by broad cross-cutting
federal government priority goals as well as input from relevant congressional committees. The
APGs are intended to be short-term goals that advance progress towards longer-term outcome-
focused goals.

The GPRA Modernization Act also made a number of changes to agency performance
management roles, and provided the officials in these roles with specific duties to increase
accountability of senior agency leadership for performance and results. The agency head was
given broad responsibility for performance management, to include identifying APGs and, along
with the Chief Operating Officer (COO), conducting quarterly priority goal progress reviews. It
requires the deputy agency head to serve as the COO, charged with improvement of agency
management and performance, and formally established the role of the Performance
Improvement Officer who would report directly to the COO and advise on goal-setting and
measurement and reviewing progress toward APGs. Additionally, each APG is to have a clearly
identified agency official, known as a goal leader, who will be held accountable for leading
implementation efforts to achieve the goal.

Agencies were required to identify their APGs and related information in their strategic plans
and performance plans published concurrently with the President’s budget in February 2012. In
December 2012, the APGs and related performance information were made publicly available by
the Office of Management and Budget (OMB) on a government-wide website ­
Performance.gov. Agencies are required to submit updates to OMB on each APG at least
quarterly for publishing on Performance.gov.
Final Audit Report
ED-OIG/A19M0005                                                                                     Page 4 of 12



                                           AUDIT RESULTS



Overall, we found that the U.S. Department of Education (Department) has generally
implemented the GPRA Modernization Act as required as related to our objectives, but
improvements are needed with regard to disclosures related to congressional input and data
verification and validation. Specifically, we found that the Department generally identified and
clearly articulated APGs, milestones, and performance indicators as required by the GPRA
Modernization Act. It also generally conducted and reported on quarterly performance reviews
of its APGs, although supporting documentation could be better maintained. However, we noted
that the Department did not identify on Performance.gov how congressional views were
incorporated into the establishment of its APGs. In addition, we found that while the
Department has established processes for ensuring the accuracy and reliability of data used to
measure progress towards its APGs, it has not accurately or adequately disclosed relevant
information in its Annual Performance Plan (APP) or Annual Performance Report (APR) as
required. As a result, the public may have less confidence that Congress and the Department are
in agreement on the immediate priorities of the agency and that the data presented in
performance reports is credible, and they may be unaware of any limitations of the data that
would provide important context for understanding it.


FINDING NO. 1 – The Department Generally Adequately Identified and Clearly
                Articulated Agency Priority Goals, Milestones, and Performance
                Indicators

We found that the Department generally identified and clearly articulated APGs, milestones, and
performance indicators as required by the GPRA Modernization Act. Specifically, we noted that
each APG was objective, quantifiable, measurable, had identified goal leaders, and included
2-year targets as well as quarterly milestones. 1 However, we noted that the Department did not
identify how congressional views were incorporated into the establishment of its APGs.
Specifically, we found no description on its agency homepage or individual APG pages on
Performance.gov of how the APGs incorporated the views and suggestions of Congress. While
we did note some references to Congress on individual APG pages, according to goal leaders the
information provided was not from congressional input received during the development of the
APGs; rather, the references were with regard to updates provided to Congress on the progress of
various aspects of the APG, such as implementation of the underlying Department program.

OMB Circular A-11 (OMB A-11) Part 6, “Preparation and Submission of Strategic Plans,
Annual Performance Plans, and Annual Program Performance Reports,” Section 250.3, dated
August 2012, 2 identifies the primary criteria agencies must use in setting APGs. This includes
developing goal statements that clearly identify the problems or opportunities the agency is

1
  See Enclosure 1 for a listing of the APGs, milestones and performance indicators.

2
  The August 2012 version of OMB A-11 was in effect for the scope of this audit. It was updated in July 2013. All 

OMB A-11 citations within this report are from the 2012 version; however, each citation remains in effect with no

significant revisions in the 2013 version, albeit with slightly different section numbers in some instances.

Final Audit Report
ED-OIG/A19M0005                                                                        Page 5 of 12

trying to address and framed in a way that can be easily understood by the public. APGs are also
required to have clear completion dates, targets, and indicators, as well as quarterly milestones to
track progress. Additionally, OMB A-11 Section 210.14 states that all agencies must include a
description of how the APGs incorporate views and suggestions obtained through the agency’s
congressional consultations. This information should be included in the overview section of the
agency homepage or APG page on Performance.gov.

We found that the Department has no related internal policies and procedures with regard to
GPRA reporting, to include the required disclosures of congressional input, as well as defined
roles and responsibilities of Department staff in the reporting process. According to goal leaders,
they were not informed of any congressional consultations that may have occurred during the
development of the APGs, nor were they provided guidance for providing information on
congressional input as part of their reporting on APGs. In addition, the Department’s Office of
Legislation and Congressional Affairs (OLCA) could provide no evidence of congressional
consultations during the development of the APGs. However, an OLCA official noted that they
are involved in discussions with Congress on a daily basis and not every discussion is
documented. She added that if the Department is required to consult with Congress, those
discussions would have occurred.

Disclosure of congressional consultations can help to improve the confidence in the American
people that Congress and the Department are in agreement on the immediate priorities of the
agency and are working together to improve related performance.

Recommendations

We recommend that the Deputy Secretary

1.1	   Develop and implement formal written internal procedures related to the GPRA process,
       to include applicable policy on congressional consultations, reporting on such
       consultations, and roles and responsibilities of involved Department staff.

Department Comments

The Office of the Deputy Secretary (ODS) concurred with the finding. ODS agreed that it may
not have clearly identified on Performance.gov how Congressional views were incorporated into
the establishment of the 2012-2013 APGs, but noted that Congress was consulted and apprised
on the progress of various aspects of APGs. ODS noted that it made progress toward developing
and implementing internal guidance related to GPRA Modernization Act requirements, including
Congressional consultations, reporting on such consultations, and roles and responsibilities of
involved Department staff. ODS also noted that it intends to set up clearer internal processes
consistent with the draft recommendation.
Final Audit Report
ED-OIG/A19M0005                                                                       Page 6 of 12

FINDING NO. 2 – The Department Has Not Accurately or Adequately Disclosed
                Information Related to its Data Verification and Validation
                Process

We found that while the Department has established processes for ensuring the accuracy and
reliability of data used to measure progress towards its APGs, it has not accurately or adequately
disclosed relevant information in its APP or APR as required. We noted that the overall data
verification and validation approach initially described by the Department was not actually being
utilized by its principal offices. Additionally, we noted that the Department did not specifically
identify the means used to verify and validate measured values; the level of accuracy required for
the intended use of data; any limitations to the data, including how the agency will compensate
for such limitations if needed to reach the required level of accuracy; and did not always
accurately identify the sources for the data.

Disclosure of Verification and Validation Process

We noted that the Department initially included a section in its draft fiscal year (FY) 2012
Annual Performance Report and FY 2014 Annual Performance Plan titled “The Department’s
Approach to Data Collection and Analysis” that included a section on data verification and
validation. Specifically, the Data Verification and Validation disclosure stated:

       The Department has developed a protocol for verifying and validating data that it makes
       public in its annual performance plans and reports. To ensure data conformity and
       accuracy, the Department has developed a guidance document to assist its offices
       responsible for reporting data on strategic and program performance measures to address
       issues of data integrity, reliability, quality, and credibility. The document provides a
       framework for validating and verifying performance data before it is collected and
       reported and is used to evaluate data prior to publication. The guidance document is in
       the form of a worksheet that is used to assist offices in ensuring the completeness and
       reliability of the Department’s performance data.

Our discussions with the respective goal leaders or goal leader representatives revealed that only
one was familiar with the framework or associated worksheet but she did not use them. The
actual data verification and validation approaches employed varied by principal office and data
source. We found in many cases, data is compiled manually and is not always subject to a
formal validation process, particularly in cases where information is obtained from publicly
available sources. For example, the Department staff responsible for compiling data for Priority
Goal 6 related to college completion plans stated that he tracks the related performance indicator
by researching and checking various public sources of information such as news articles, states’
higher education policies and strategic plans, and governors’ announcements. He compiles the
information manually in a spreadsheet and provides the data directly to ODS. It is not
independently verified.

Performance data for Priority Goal 5 is compiled by checking the number of states noted on the
Common Core State Standards website, an initiative of the National Governors Association and
Council of Chief State School Officers, as having adopted the standards. For some priority
goals, data is manually obtained from grant applications, to include counting the number of
Final Audit Report
ED-OIG/A19M0005                                                                                    Page 7 of 12

school districts noted in successful Race to the Top Early Learning Challenge (ELC) grant
applications that were designated as having high-quality plans (Priority Goal 1) and counting
states and districts noted in Race to the Top grant documentation noted as having requirements
for teacher and principal evaluation systems (Priority Goal 2). For other goals, data is obtained
from the Department’s EDFacts system. 3 With the exception of the data pulled from public data
sources, Department officials noted that data is validated through grantee monitoring efforts and
review of grantee annual performance reports.

Our discussions with ODS staff indicated that they do not have a separate process for verifying
the data submitted by the goal leaders and that they rely on the goal leaders to ensure that the
submitted data has already been verified by program offices. ODS staff believed that the data
was being verified by the Department’s Data Strategy Team (DST) prior to submission.
However, a member of the DST stated that while there is some checking related to the accuracy
of numbers reported that were obtained from the Department’s EDFacts system, the DST does
not have a formal role in ensuring the validity of results. As a result of our discussions with
ODS on this matter, the section on Data Verification and Validation contained in the draft
APR/APP was subsequently removed from the final report posted on April 10, 2013.

Disclosure of Performance Data Limitations and Data Sources

We further noted that while some information related to performance data was included within
each individual APG section, the Department generally did not adequately disclose the level of
accuracy required for the intended use of the data or any performance data limitations, to include
how the agency will compensate for such limitations. In the final APR/APP, the Department
used general statements to describe the limitations to the performance data it relied on to
measure success. Specifically, the Department stated that data tied to a measure was influenced
by external factors, such as the actions taken by local or state educational agencies, grantees, or
states. While the Department’s disclosure of limitations of the APG data noted the reliance by
the Department on performance data submitted by external entities, it did not disclose what
specific limitations may be associated with the data as a result, such as the potential for
incomplete or inaccurate data, any plans to compensate for the limitations, and whether the
limitations are deemed acceptable in relation to the level of accuracy required for the intended
use of the data. For one indicator, we noted the Department used data from the ELC grant to
track the number of states implementing high-quality plans to collect and report disaggregated
data on the status of children at kindergarten entry. Department staff stated that this may not
provide an accurate depiction of the number of states as there may be non-ELC states
implementing high-quality plans, but that the data is not collected or used to measure progress
for the related APG. We found that the Department did not adequately disclose this limitation to
the data in the final APR/APP.

While the Department did note data sources for all of the APG performance data, in one case we
noted that the data source did not appear accurate. In this instance, the data source was noted as
Department of Education annual monitoring plans, while the individual that collected the data for


3
 EDFacts is a Department initiative designed to collect state-reported K through 12 education performance data and
other data within the Department, such as financial grant information, to encourage improved analysis and use of
data.
Final Audit Report
ED-OIG/A19M0005                                                                      Page 8 of 12

the measure noted the source as news articles, state policies and strategic plans. Department
officials could not explain the reason for the discrepancy.

The GPRA Modernization Act of 2010 and OMB A-11 Section 210.14 require that each update
on agency performance should describe how the agency ensures the accuracy and reliability of
the data used to measure progress towards its performance goals, including an identification of:

       (A) the means used to verify and validate measured values;
       (B) the sources for the data;
       (C) the level of accuracy required for the intended use of the data;
       (D) any limitations to the data at the required level of accuracy; and
       (E) how the agency has compensated for such limitations if needed to reach the required
           level of accuracy.

OMB A-11 allows agencies to develop a single data verification and validation appendix used to
communicate the agency’s approaches, and/or agencies may choose to provide information about
data quality wherever the performance information is communicated.

Department officials in ODS stated that they believed that the underlying statement of the
process for validating and verifying data included in the draft APR/APP was accurate, but that
the connection to the worksheet may not have been an accurate representation of the process or
of the Department’s efforts. After the statement was removed from the draft APR/APP, the
Department did not include new information in the final report regarding its process for ensuring
performance data accuracy and reliability. Department officials believed that the Department
had adequately disclosed data limitations since the final APR/APP noted that the Department
relied on data that comes from States and local educational agencies.

According to a Department official within the Office of the Chief Financial Officer (OCFO) that
is responsible for developing the APR/APP, the statement for data accuracy and reliability was
developed approximately 4 years ago to satisfy prior reporting requirements. The official
believed that the statement accurately reflected the Department’s process and noted that program
offices were provided the Validation and Verification Worksheet at the time it was completed.
The Department official believed that enough information remained in the final APR/APP to
satisfy the requirement related to data accuracy and reliability.

Overall, verification and validation of performance data and disclosure of data limitations
support the general accuracy and reliability of performance information and reduce the risk of
inaccurate performance data. Without adequate disclosure of relevant information, the
Department may not provide a sufficient level of confidence to Congress and the public that the
information presented is credible as appropriate for its intended use.

As part of our audit, we reviewed APRs prepared by other federal agencies to identify possible
examples of good practices on disclosures of information on data validity and verification in
accordance with GPRA requirements. We found that the APR from the Department of Housing
and Urban Development included an appendix that clearly described data sources, data
limitations and advantages, and data validation and verification for each of its goals. We noted
that the APR from the Department of Agriculture included specific information as part of its
discussion of each individual goal. The results reported for each goal included clearly labeled
Final Audit Report
ED-OIG/A19M0005                                                                         Page 9 of 12

sections on the completeness and source of the data, reliability of the data, as well as quality of
the data.

Recommendations

We recommend that the Deputy Secretary

2.1 	   Clearly describe in all applicable performance reports and plans the Department’s data
        verification and validation process for each APG and include complete and accurate
        disclosures related to data limitations and sources in accordance with GPRA
        requirements.

Department Comments

ODS generally agreed with the substance of Finding No. 2, but did not concur with the wording
used to describe the finding and the recommendation. ODS noted that the wording could be
misunderstood and may imply to a reader that the related performance data was not accurate, or
that ODS was not disclosing the data appropriately. ODS suggested that the wording of the
finding be changed to reflect that the Department could make process improvements to better
manage its data verification and validation process, and better describe the methodology used in
published documents. ODS also suggested the removal of “complete and accurate” from the
recommendation.

ODS noted that it has designed procedures for ensuring that the best quality data is available for
its planning and reporting purposes as well as a framework for principal offices to identify
issues in data verification and validation prior to data reporting. Further, ODS noted that actions
are planned to address shortfalls in data completeness, accuracy, and reliability of the data.

OIG Response

While the Department notes that it has designed procedures to ensure the best quality data, our
review did not examine the accuracy or appropriateness of the performance data itself. Rather
our review examined the accuracy and adequacy of the Department’s disclosure of the processes
it used to determine the accuracy and completeness of the performance data. The finding title
conveys that the weaknesses noted were related to the disclosure of information while the finding
provides details on weaknesses noted in Department disclosures as compared to GPRA
Modernization Act disclosure requirements. These weaknesses included inaccurate disclosures
of actual processes followed as well as incomplete disclosures of required information.
Consequently, the finding and related recommendation reflect that we specifically found that
disclosures related to the verification and validation process were not always accurate or
adequate, not the actual data itself.

After considering ODS’s comments, we have not made any changes to the finding or related
recommendation.
Final Audit Report
ED-OIG/A19M0005                                                                                  Page 10 of 12

FINDING NO. 3 – Quarterly Performance Reviews Have Generally Been
                Performed as Required

The Department has generally implemented quarterly performance reporting under the GPRA
Modernization Act of 2010 as required, although supporting documentation could be better
maintained. Specifically, we determined that the Department generally adequately completed
the required quarterly performance updates (QPUs) and likely submitted the QPUs for OMB
review timely. OMB guidance required that agencies provide QPUs starting with the 2nd quarter
of FY 2012. The 2nd quarter QPU was to be submitted to OMB as part of the APG draft action
plan, with subsequent QPUs to be submitted electronically through OMB’s electronic reporting
system. QPUs were made available to the public via Performance.gov beginning with the QPU
for the 4th quarter of FY 2012.

We noted ODS developed templates that were to be distributed to goal leaders, or those
designated by the goal leader, at the end of each quarter. The template requested information
such as an update on progress since the previous quarter; key barriers and challenges to
successful APG implementation; external factors that may impact the APG; and next steps and
future actions for the remaining quarters during the 2-year APG period, to include quarterly
milestones. The updated QPU template was returned to ODS via email, and the information
loaded into OMB’s Performance Reporting Entry Portal by a team within ODS. According to
Department officials, this information is then reviewed by OMB and they work with OMB to
resolve any issues with the QPU reporting, such as the adequacy of the information included in
the update.

As part of our review, we asked Department officials to provide us with copies of each of the
required QPUs that were to be completed as of the date of our request (April 2013). The
Department provided the completed templates for the 3rd and 4th quarters of FY 2012 and the
1st quarter of 2013. The Department also provided the APG draft action plan required by OMB
as noted above; however, we found it did not include the FY 2012 2nd quarter performance
updates. Department officials indicated that the updates were prepared, but they were unable to
provide supporting documentation.

The Department was initially unable to provide evidence of the actual submission date of the
QPUs to OMB. It subsequently obtained the publish history maintained in Performance.gov
from OMB, which provided the date the QPU was posted on Performance.gov for public review
after its submission for review and comment by OMB. This information did not clearly indicate
which quarterly reporting period the publish date was associated with. In order to determine
whether a QPU was completed timely, we looked for publish dates that fell on or around the
publish date timelines cited in OMB A-11 for each QPU. The results of our review indicated
that for the 3 QPUs that were published on Performance.gov, 4 all were published within 2 weeks
or less of the cited publish timelines, indicating the Department likely completed and submitted
the QPUs for OMB review timely.




4
 Performance updates for the 2nd quarter of FY 2012 were not submitted or published via Performance.gov as the
website was not operational at that time.
Final Audit Report
ED-OIG/A19M0005                                                                    Page 11 of 12

Section 250.19 of OMB A-11 states that agencies must report progress to OMB on each APG
6 weeks after the end of each quarter, covering the period of the most recent quarter closed.
Section 200.22 provides specific due dates for quarterly reporting. Additionally, Section 210.14
provides a table that establishes detailed content that must be addressed in QPUs. Required
content includes achievements during the last quarter as well as an identification of significant
challenges if any impeded progress on the APG. Updates should be provided if needed on future
actions, implementation strategies, indicators, contributing programs, external factors and
changes in goal leaders.

We found that the Department has no related internal policies and procedures with regard to the
QPU process. This was magnified by staff turnover within ODS, including those responsible for
managing the QPU process. Department officials had to search through documentation to
recreate the QPU process as part of the audit as this information was not always readily
available. Without established policies and procedures, the QPU process may not be as efficient
as it could be and results may not be published timely, delaying the public’s ability to review
Department progress on key goals.

Recommendations

We recommend that the Deputy Secretary

3.1 	   Establish and implement formal written internal policies and procedures regarding the
        quarterly performance review process, to include documentation requirements and
        submission timeframes.

Department Comments

ODS concurred with the finding and noted that supporting documentation of completed quarterly
updates could have been better maintained. ODS noted that it has made progress, consistent with
the draft recommendation, toward developing and implementing internal guidance related to the
quarterly performance review process to improve the documentation and submission timeframes.
Final Audit Report
ED-OIG/A19M0005                                                                    Page 12 of 12



                 OBJECTIVES, SCOPE, AND METHODOLOGY



The objectives of our audit were to determine whether the Department has 1) identified and
clearly articulated priority goals, milestones, and performance indicators; 2) established a
process for ensuring the validity and reliability of data; and 3) conducted and reported on the
results of quarterly performance reviews to provide information to the public on its performance
and operations.

To accomplish our objectives, we gained an understanding of internal control applicable to the
Department’s implementation and administration of the GPRA Modernization Act. We
reviewed applicable legislation, regulations and other guidance, to include the GPRA
Modernization Act of 2010, OMB Circular A-11 “Preparation, Submission, and Execution of the
Budget,” and OMB Management Procedures Memorandum No. 2012-01,“FY 2012-2013
Agency Priority Goal Planning, Implementation, and Review Process.” In addition, to identify
potential vulnerabilities, we reviewed prior Office of Inspector General and Government
Accountability Office reports relevant to our audit objectives.

We conducted discussions with ODS officials and goal leaders or goal leader designees for each
APG with specific knowledge of the Department’s APGs and related processes. We also held
discussions with officials from the DST, Data Quality Initiative, and OCFO to gain an
understanding of their role in the APG performance data validity and reliability process.

The scope of our review was limited to reviewing information related to all six of the
Department’s APGs developed for the 2-year period covering FYs 2012-2013. This included a
review of the Department’s FY 2011-2014 Strategic Plan, FY 2012 APR, FY 2014 APP,
performance information posted to Performance.gov, and all quarterly performance updates
prepared by the Department as of April 2013. We also reviewed APRs prepared by other federal
agencies to identify possible examples of good practices on disclosures of information on data
validity and verification in accordance with GPRA Modernization Act requirements.

We conducted fieldwork at Department offices in Washington, D.C., from January 2013 through
September 2013. We provided our preliminary audit results to Department officials during an
exit conference conducted on September 9, 2013.

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objectives. We believe that the evidence obtained provides a reasonable basis
for our findings and conclusions based on the audit objectives.
                                                                                     Enclosure 1

                       Table of FY 2012-2013 Agency Priority Goals

APG             Goal Statement/Milestone                            Performance Indicator(s)
1 - Early       Improvement outcomes for all children from          States implementing high-
Learning        birth through third grade. By September 30,         quality plans to collect and
                2013, at least nine states will implement a high-   report disaggregated data on
                quality plan to collect and report disaggregated    the status of children at
                data on the status of children at kindergarten      kindergarten entry.
                entry.
2 - Effective   Improve learning by ensuring that more students     States with approval for
Teacher         have an effective teacher. By September 30,         evaluation system
                2013, at least 500 school districts will have       guidelines.
                comprehensive teacher and principal evaluation      Participating school districts
                and support systems and the majority of states      with qualifying evaluation
                will have statewide requirements for                systems for teachers.
                comprehensive teacher and principal evaluation      Participating school districts
                and support systems.                                with qualifying evaluation
                                                                    systems for principals.
3 - School      Demonstrate progress in turning around the          Lowest-achieving schools
Turnaround      nation’s lowest-performing schools. By              demonstrating significant
                September 30, 2013, 500 of the nation’s             improvement and serving as
                persistently lowest-achieving schools will have     potential models for future
                demonstrated significant improvement and will       turnaround efforts.
                have served as potential models for future
                turnaround.
4 - State       Make informed decisions and improve                 Number of states
Longitudinal    instruction through the use of data. By             implementing K-12 data
Data Systems    September 30, 2013, all states and territories      systems.
                will implement comprehensive statewide
                longitudinal data systems (SLDS).
5 - College     Prepare all students for college and career. By     States adopting
and Career      September 30, 2013, all states will adopt           internationally-
Readiness       internationally-benchmarked college-and career-     benchmarked college- and
                ready standards.                                    career-ready standards.
6 - College     Improve students’ ability to afford and complete    Number of states with
Completion      college. By September 30, 2013, the                 college completion plans in
and             Department will develop a college scorecard         place.
Affordability   designed to improve decision-making and
                transparency about affordability for students and
                borrowers by streamlining information on all
                degree-gathering institutions into a single,
                comparable, and simplified format, while also
                helping all states and institutions develop
                college completion plans.
                                                                        Enclosure 2

               Acronyms/Abbreviations/Short Forms Used in this Report

ALO          Audit Liaison Officer

APG          Agency Priority Goal

APP          Annual Performance Plan

APR          Annual Performance Report

CAP          Corrective Action Plan

COO          Chief Operating Officer

Department   U.S. Department of Education

DST          Data Strategy Team

ELC          Early Learning Challenge

FY           Fiscal Year

GPRA         Government Performance and Results Act

OCFO         Office of the Chief Financial Officer

ODS          Office of the Deputy Secretary

OLCA         Office of Legislation and Congressional Affairs

OMB          Office of Management and Budget

QPU          Quarterly Performance Update
                            Department Response to the Draft Audit Report                                  Enclosure 3

                                            O FFICE OF THE D EPUTY S ECRETARY 





TO:              Patrick J. Howard
                 Assistant Inspector General for Audit
                 Office of Inspector General

FROM:            Nancy Poon Lue          ~
                 Acting    Performan~ement Officer
                 Office of the Deputy Secretary

SUBJECT: 	       Comments on Draft Audit Report "The Department' s Implementation of the
                 Government Performance and Results Act Modernization Act"
                 (Control Number ED-OIG/A l 9M0005)



I am writing in response to the Office of Inspector General's (OIG's) draft audit report on "The
Department' s Implementation of the GPRA Modernization Act". We appreciate the work that
went into the report and the opportunity you have provided us to respond to the draft findings
and recommendations.

We are pleased that your draft audit found the Department has generally implemented the
Government Performance and Results Act Modernization Act of2010 (GPRA Modernization
Act), as required by law and consistent with the purposes of your audit, and that we have
identified and clearly articulated agency priority goals (APGs), milestones, and performance
indicators as required, and conducted reviews and reported on quarterly performance of its APGs
as required.

The Department is committed to building on the planning and reporting framework created by
the GPRA Modernization Act. Since the enactment of the Act, the Department has intensified
efforts to continuously improve through an ongoing cycle of establishing goals, assessing
performance, examining data, and improving practices. Creating a culture of continuous
improvement is at the heart of our efforts to work with and support early childhood, elementary,
secondary, and postsecondary educators, policy makers, and beneficiaries at the federal, state,
and local levels.

Our responses to each of the draft findings and recommendations are set forth below. Once
again, thank you for your draft findings and recommendations and for the opportunity to review
them and to respond.

                                                      www.ed.gov

                               400 MARYLAND AVE., SW, WASHINGTON, DC 20202
   The Department ofEducation 's mission is to promote student achievement and preparationfor global competitiveness by
                              fostering educational excellence and ensuring equal access.
2


Finding No.1. The Department Generally Adequately Identified and Clearly Articulated
Agency Priority Goals, Milestones, and Performance Indicators

We concur with Finding No.I , that the Department generally identified and clearly articulated
APGs, milestones, and performance indicators as required by the GPRA Modernization Act and
that each APG was objective, quantifiable, measurable, had identified goal leaders, and included
2-year targets as well as quarterly milestones.

We agree that we may not have clearly identified on Performace.gov how Congressional views
were incorporated into the establishment of the 2012-2013 APGs, however, Congress was
consulted and apprised on the progress of various aspects of the APGs, such as implementation
of the underlying Departmental programs, and we took their views into account on these matters.
The Office of the Deputy Secretary made progress toward developing and implementing internal
guidance related to GPRA Modernization Act requirements, including congressional
consultations, reporting on such consultations, and the roles and responsibilities of involved
Department staff, and consistent with the draft recommendation, we intend to set up clearer
internal processes.

Finding No.2. The Department Has Not Accurately or Adequately Disclosed Information
Related to its Data Verification and Validation Process

While we generally agree with the substance of the draft finding that the Department can
improve its data verification and validation process, we do not concur with the wording used to
describe Finding No. 2 and the recommendation, as it may be misunderstood and may imply to a
reader that the performance data we provided was not accurate, or we were not disclosing it
appropriately. We suggest that the wording of the finding be changed to reflect that the
Department could make process improvements to better manage its data verification and
validation process, and better describe the methodology used in published documents. Our
suggested wording for the title and substance ofthe fmding would be more closely aligned with
the specific details of your findings that pertain to our verification and validation process,
performance data limitations, and data sources.

We concur with the substance of your draft recommendation with the suggested word changes in
the followi ng draft recommendation:

We recommend that the Deputy Secretary:
2.1 Clearly describe in all applicable performance reports andplans the Department's data
verification and validation process for APGs and include disclosures related to data limitations
and sources, in accordance with GPRA Modernization Act requirements.

The Department has designed a procedure for ensuring that the best quality data are available for
its planning and reporting purposes and has developed a framework for principal offices to
identify issues in data validation and verification for its strategic and program performance goals
and measures prior to data reporting. In addition, limitations of data collected by the Department
are noted and actions are planned to address shortfalls in data completeness, accuracy, and
reliability.
3



For example, the Department's Policy and Program Studies Services (PPSS) office assesses the
completeness and reliability of the data presented in the FY 14-18 Strategic Plan. The types of
data and methodologies include:

           • 	 Statistical data collections which contain documented studies methodologies that
               provide evidence of data completeness and reliability and identify data limitations
               that arise from a variety of sources, including sampling error. To identify their
               completeness and reliability, ED will rely upon associated methodology reports
               developed by NCES, Census, and other statistical agencies as applicable.
           • 	 Strategic Plan metrics obtained from Annual Performance Reports (APR) are
               submitted by grantees to program offices, or from program data submitted to
               the Department through the EDFacts Submission System. The Department's
               EDFacts team works with ED program offices on protocols to assess the
               completeness, reliability, and overall quality ofEDFacts data; identifying
               limitations specific to the data elements used to calculate public-facing metrics.
               Program offices were asked to identify procedures they follow to ensure the
               completeness and reliability of APR data, known limitations, and applicable plans
               for quality enhancement.
           • 	 Monitoring and Grant Applications data (e.g., Flex Applications) and
               Management Information Systems/Business Operations (such as Past
               Performance Information Retrieval System) are also used to calculate
               performance measures. Program offices were asked to identify the monitoring
               process, information system, or business operation that is the source of metric
               data, describe quality assurance of procedures in use, and identify data limitations.
           • 	 Nonstatistical data sources external to ED are used to support four public-facing
               performance indicators. The source for two metrics is the Department of Health
               and Human Services (HHS). ED will consult with HHS on the limitations of their
               data. The other external data providers were asked to provide evidence of data
               quality and known data limitations.

Finding No.3. Quarterly Performance Reviews Have Generally Been Performed as Required

We concur with Finding No.3, that the Department has met the requirements for quarterly
performance reporting under the GPRA Modernization Act of2010. We completed the required
quarterly updates and submitted them timely for OMB review. Although supporting
documentation could have been better maintained, the Office of the Deputy Secretary has already
made progress, consistent with the draft recommendation, toward developing and implementing
internal guidance regarding the quarterly performance review process, to improve the
documentation and submission timeframes, as applicable to OMB's A-11 guidance.

Please let us know if you have any questions or need further information about any of our
comments and responses. Thanks again for the opportunity to provide comments.