oversight

Department's Implementation of the Teacher Incentive Fund Grant Program

Published by the Department of Education, Office of Inspector General on 2011-12-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

    Department’s Implementation of the Teacher Incentive

                   Fund Grant Program



                                 FINAL AUDIT REPORT





                                     ED-OIG/A19I0007

                                      December 2011





Our mission is to promote the                           U.S Department of Education
efficiency, effectiveness, and                          Office of Inspector General
integrity of the Department's                           Washington, DC
programs and operations.
                     NOTICE
Statements that managerial practices need improvements, as well as
other conclusions and recommendations in this report represent the
opinions of the Office of Inspector General. Determinations of
corrective action to be taken will be made by the appropriate
Department of Education officials.

In accordance with the Freedom of Information Act (5 U.S.C. § 552),
reports issued by the Office of Inspector General are available to
members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.
                                    UNITED STATES DEPARTMENT OF EDUCATION
                                           OFFICE OF INSPECTOR GENERAL

                                                                                                                     AUDIT SERVICES
                                                        December 30, 2011

Memorandum
TO:                  Michael Yudin
                     Acting Assistant Secretary
                     Office of Elementary and Secondary Education

FROM:                Keith West /s/
                     Assistant Inspector General for Audit

SUBJECT:             Final Audit Report
                     Department’s Implementation of the Teacher Incentive Fund Grant Program
                     Control Number ED-OIG/A19I0007

Attached is the final audit report that covers the results of our review of the Department’s
implementation of the Teacher Incentive Fund Grant Program. An electronic copy has been
provided to your Audit Liaison Officer (ALO). We received the Office of Elementary and
Secondary Education’s comments on the contents of our draft report and the corrective action
plan for each of our recommendations.

Corrective actions proposed (resolution phase) and implemented (closure phase) by your office
will be monitored and tracked through the Department’s Audit Accountability and Resolution
Tracking System (AARTS). Department policy requires that you develop a final corrective
action plan (CAP) for our review in the automated system within 30 days of the issuance of this
report. The CAP should set forth the specific action items and targeted completion dates
necessary to implement final corrective actions on the findings and recommendations contained
in this final audit report.

In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector
General is required to report to Congress twice a year on the audits that remain unresolved after
6 months from the date of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office
of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given to us during this review. If you have any questions, please
call Michele Weaver-Dugan at (202) 245-6941.

Enclosure

cc:       Delores Warner, ALO, Office of Elementary and Secondary Education
 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
                                              TABLE OF CONTENTS



                                                                                                                                 Page

EXECUTIVE SUMMARY ...........................................................................................................1


BACKGROUND ............................................................................................................................4


AUDIT RESULTS .........................................................................................................................5


          FINDING NO. 1 –	 The Department Did Not Always Evaluate and Fund

                           TIF Grant Applications in Accordance With 

                           Applicable Criteria or Guidance .................................................5


          FINDING NO. 2 –	 The Department Did Not Always Effectively

                           Oversee the TIF Program to Ensure Funds Were 

                           Used as Intended and Anticipated Recipient 

                           Performance Was Achieved ........................................................22


          FINDING NO. 3 –                The Department's Ability to Assess Overall 

                                         Outcomes of Initial TIF Awards May be Limited ....................29


OTHER MATTER.......................................................................................................................32


OBJECTIVES, SCOPE, AND METHODOLOGY ..................................................................34


ENCLOSURE 1 – Acronyms/Abbreviations/Short Forms Used in this Report


ENCLOSURE 2 – Department Response to Draft Report
Final Audit Report
ED-OIG/A19I0007                                                                                         Page 1 of 35



                                         EXECUTIVE SUMMARY



The Teacher Incentive Fund grant program (TIF) was established to support efforts to develop
and implement performance-based compensation systems (PBCS) for teachers and principals in
high-need schools. The program was created by Congress as part of the Appropriations Law for
the Departments of Labor, Health and Human Services, Education, and Related Agencies for
fiscal year ending September 30, 2006. This appropriation made $100 million available for
competitive grants and related U.S. Department of Education (Department) provided activities
such as technical assistance, peer review of applications, and evaluation activities.

A PBCS established under TIF must consider gains in student academic achievement, include
classroom evaluations conducted multiple times during each school year, and provide educators
with incentives to take on additional responsibilities and leadership roles. The related
competition was open to: Local Educational Agencies (LEA), including charter schools that are
LEAs in their state; State Educational Agencies (SEA); or partnerships of an LEA and/or an SEA
along with at least one nonprofit organization.

The objectives of our audit were to: 1) determine whether grant proposals were evaluated and
awarded in accordance with authorizing statute and all applicable regulations; and 2) evaluate the
effectiveness of program oversight to ensure that funds were used as intended, anticipated
recipient performance was achieved, and overall programmatic goals were met. Overall, our
audit determined that the Department could improve the effectiveness of its program
implementation, grant proposal evaluation and awarding processes, and program oversight.

With respect to program implementation and grant proposal evaluation and awarding processes,
our audit found that the Department did not always effectively apply Congressional intent and
the Department’s internal guidance in its implementation of the TIF program. Although the
Department is not specifically required to implement programs in accordance with Congressional
committee report language, we believe it is a better practice for the Department to ensure that the
intent of Congress is effectively considered and applied in instances such as TIF where there is
limited detail in the authorizing statute. We also found the Department could more effectively
determine applicant competence and responsibility prior to making awards, funded applications
that did not fully meet the Department’s absolute priorities 1 for the competitions, and funded
applications containing weaknesses that could be detrimental to program implementation or
sustainability. As a result, the Department may have limited prospective program applicants,
funded applications that should have been considered ineligible, and may have awarded funds to
applicants with a heightened risk of misusing public funds and/or the inability to achieve
programmatic goals.

With respect to its program oversight processes, our audit found that the Department did not
always ensure that recipients met applicable requirements prior to making continuation awards
and that implementation of funded applications was consistent with recipient proposals. As a
result, there may be an increased risk that funds are not used as intended and anticipated

1
    An absolute priority is an established requirement that an application must meet for funding consideration.
Final Audit Report
ED-OIG/A19I0007                                                                       Page 2 of 35

recipient performance is not achieved, as well as an increased risk for fraud, waste, and abuse of
program funds. Implementation of programs in a manner that is inconsistent with what was
proposed in recipient applications may undermine the outcome of the competition process and
compromise its underlying purpose.

We also noted weaknesses in the processes established by the Department to assess program
results. A national evaluation that would have potentially assessed the impact of the initial TIF
awards on student achievement and teacher and/or principal recruitment and retention will not be
performed. As a result, the Department may have lost an opportunity to obtain data regarding
the performance of these recipients to include data on overall TIF program effectiveness.

Finally, during the course of our audit, we became aware that the Department’s implementation
of the 2010 competition for new TIF awards allowed TIF funds to be used for a planning period
of up to 1 year for an applicant that could not demonstrate in its application that it had each of
the five required core elements in place at the time the application was submitted. By its nature,
the allowance of a planning period creates inconsistencies in standards applied to applicants and
increases the risk that programmatic performance may be negatively impacted and potential
benefits delayed.

The Department has recently taken steps that may help to improve some of the weaknesses noted
during our audit. This includes the issuance of guidance to discretionary grant program staff that
requires the assessment of an entity’s risk prior to the issuance of a new or continuation award
and identifies actions that may be taken when staff identify evidence of risk. The Department
has also developed and is in the process of implementing automated tools that will strengthen its
ability to identify areas of potential risk associated with grant applicants. In addition to these
actions, we recommend that the Assistant Secretary for Elementary and Secondary Education,
among other things:

   •	 Develop and implement procedures to ensure that the intent of Congress and the
      Department’s internal guidance is effectively considered and applied in the establishment
      of criteria applicable to discretionary grant competitions.
   •	 Revisit the TIF program requirements in conjunction with legislative intent and other
      applicable criteria prior to the next competition and make any necessary changes.
   •	 Through training or another appropriate method, ensure that program staff performing
      grant proposal screening functions have expert knowledge of programmatic criteria and
      that documentation designed to assist in the screening process is complete and accurate
      with respect to the program requirements being evaluated.
   •	 In future TIF competitions, consider implementation of a scoring system that ensures
      greater weight is applied to key areas.
   •	 Reinforce the need to conduct reviews of A-133 audits to program staff responsible for
      recipient monitoring and develop and implement processes to ensure that appropriate
      procedures for A-133 reviews are included in program monitoring plans and guides.
   •	 Document the types of recipient proposal revisions that should be considered scope
      changes and reinforce to program staff the need to hold recipients of discretionary grant
      funding accountable for the achievement of performance as identified in their respective
      grant applications.
Final Audit Report
ED-OIG/A19I0007                                                                        Page 3 of 35

   •	 Develop and implement an appropriate process that will provide meaningful data
      regarding the overall performance of initial TIF grantees and the program’s overall
      effectiveness with respect to its stated goals.

In its response to the draft audit report, the Office of Elementary and Secondary Education
(OESE) disagreed with each of the three findings and one of the nine recommendations. OESE
did not agree with draft recommendation 1.2 related to revisiting TIF program requirements in
conjunction with legislative intent and its internal guidance as it stated it did properly take this
into account when implementing the program. OESE did not specifically indicate whether it
agreed or disagreed with recommendation 2.3 regarding documenting types of recipient proposal
revisions that should be considered scope changes, but indicated it did not think it should be
included in the final audit report while it is being carried out and given the disagreement with the
related section of the finding.

Overall, OESE expressed that it strives to continually improve the TIF program and used
information from the implementation of Cohorts 1 and 2 (the focus of this audit) to inform the
competition for FY 2010 and the implementation of the program for Cohort 3. It further
indicated that it planned to use data and information from the evaluation study of Cohorts 1 and
2 to help others more effectively implement key reforms. OESE also stated that it welcomed the
opportunity to conduct further discussion with OIG on issues identified in the draft audit report
and believed its comments would help to further clarify its ongoing efforts to ensure funds under
this program will substantially improve the quality of teaching and student achievement growth.

OIG recognizes there is inherent difficulty in implementing a new program and that the
Department devoted substantial effort towards appropriately and effectively implementing the
TIF program. However, during our fieldwork we identified areas where the Department could
improve the effectiveness of its program implementation, grant proposal evaluation and
awarding processes, and overall program oversight.

Subsequent to fieldwork completion, OIG solicited additional comments and information from
Department officials in response to our proposed findings and recommendations. This included
discussions on proposed findings prior to the exit conference, additional discussion at the exit
conference, and an informal written response to the proposed findings and recommendations that
was provided to us on May 18, 2011, after the exit conference was held. We considered all of
the information provided to us in developing the draft audit report.

In response to the Department’s comments to the draft audit report, we revised finding 1 to
clarify the non-binding nature of Congressional committee report language. OESE’s comments
are summarized at the end of each applicable finding. The full text of OESE’s response is
included as Enclosure 2 to this report.
Final Audit Report
ED-OIG/A19I0007                                                                        Page 4 of 35



                                       BACKGROUND



The Teacher Incentive Fund grant program (TIF) was established to support efforts to develop
and implement performance-based compensation systems (PBCS) for teachers and principals in
high-need schools. The program was created by Congress as part of the Appropriations Law for
the Departments of Labor, Health and Human Services, Education, and Related Agencies for
fiscal year (FY) ending September 30, 2006. This appropriation made $100 million available for
competitive grants and related activities such as technical assistance, peer review of applications,
and evaluation activities. The goals of the TIF program include:

   •	 Improving student achievement by increasing teacher and principal effectiveness;
   •	 Reforming teacher and principal compensation systems so that teachers and principals are
      rewarded for increases in student achievement;
   •	 Increasing the number of effective teachers teaching poor, minority, and disadvantaged
      students in hard-to-staff subjects; and
   •	 Creating sustainable performance-based compensation systems.

Two separate grant competitions were initiated during 2006 to make TIF awards with the initial
program appropriation. As part of the U.S. Department of Education’s (Department) grant
application review process, TIF program staff screened applications for eligibility and
completeness. Those considered as qualified were subsequently reviewed by a panel of three
non-Federal reviewers and individually scored by each reviewer. Applications could receive a
maximum of 110 points, with 100 points from 5 selection criteria and 10 additional points
through competitive preference criteria. The three non-Federal reviewer scores for each
application were averaged, sorted into a rank order from highest to lowest, and awards made to
those with the highest scores.

Overall, 16 applicants scoring more than 85 points in the first competition and 18 applicants
scoring more than 88 points in the second competition were selected for funding. The value of
the initial 34 TIF awards totaled $80.3 million and ranged from approximately $126,000 to
$7.5 million. The 34 initial TIF awards were made as 5 year grants that required annual
reporting of programmatic and fiscal progress. Funding in subsequent project years was
dependent on the review of the recipient’s annual performance report and the availability of
future appropriated funds. An additional $795 million was made available for the TIF program
from FY 2008 through FY 2010. This included $200 million in funding through the American
Recovery and Reinvestment Act of 2009. These funds were intended to be used to provide
continuation funding for grants awarded in prior years and to conduct an additional award
competition during FY 2010.

The Department developed and implemented monitoring plans as a framework for the evaluation
of TIF recipient performance, to include the use of numerous monitoring tools such as the review
of the recipient’s annual performance reports and quarterly self assessment reports, site visits,
periodic telephone and email contact, and review of relevant data in the Department’s Grants
Administration and Payment System (GAPS).
Final Audit Report
ED-OIG/A19I0007                                                                     Page 5 of 35



                                    AUDIT RESULTS



Overall, our audit determined that the Department could improve the effectiveness of TIF
program implementation, grant proposal evaluation and awarding processes, and program
oversight. Specifically, we noted that the Department did not always effectively apply
Congressional intent and the Department’s internal guidance in its implementation of the TIF
program, could more effectively determine applicant competence and responsibility prior to
making awards, funded applications that did not fully meet the Department’s absolute priorities
for the competitions, and funded applications containing weaknesses that could be detrimental to
program implementation or sustainability. In addition, we noted that the Department did not
always ensure that recipients met applicable requirements prior to making continuation awards
and that implementation of funded applications was consistent with recipient proposals. As a
result, the Department may have limited prospective program applicants, funded applications
that should have been considered ineligible, and awarded funds to applicants with a heightened
risk of misusing public funds and/or the inability to achieve programmatic goals. Further,
implementation of programs in a manner that is inconsistent with what was proposed in recipient
applications may undermine the outcome of the competition process and compromise its
underlying purpose.

We also noted that the Department may have lost an opportunity to obtain data regarding the
performance of initial TIF recipients to include data on overall program effectiveness and
outcomes. Finally, we determined that the Department’s allowance for the use of TIF funds as
part of a planning period for applicants under the 2010 competition created inconsistencies in
standards applied to applicants, increased program performance risk, and may delay potential
program benefits.


FINDING NO. 1 - The Department Did Not Always Evaluate and Fund TIF Grant
                Applications in Accordance With Applicable Criteria or Guidance

Improvements are needed in the Department’s processes to ensure that grant programs are
implemented in accordance with applicable criteria or guidance. We determined that the
Department:

   1.	 Did not effectively apply Congressional intent and the Department’s internal guidance in
       its implementation of the TIF program;
   2.	 Could more effectively determine applicant competence and responsibility prior to
       making awards;
   3.	 Funded applications that did not fully meet the Department’s absolute priorities for the
       competitions; and
   4.	 Funded applications containing weaknesses that could be detrimental to program

       implementation or sustainability.

Final Audit Report
ED-OIG/A19I0007                                                                                  Page 6 of 35

As a result, the Department may have funded applicants that do not best assist the TIF program
in achieving its purpose and goals.

Application of Congressional Intent and Internal Guidance in Program Implementation

The Department considered but did not effectively apply Congressional intent as expressed
through the applicable Committee Report 2 and the Department’s internal guidance in its
implementation of the TIF program. We determined that the high-need schools definition and
programmatic cost sharing requirements applied during the initial TIF competitions were not
consistent with expressed intent and its internal guidance. As a result, the Department may have
limited prospective program applicants and inappropriately funded applications.

The Department is not specifically required to implement programs in accordance with
committee report language. However, we believe it is a better practice for the Department to
ensure that the intent of Congress is effectively considered and applied in instances such as TIF
where there is limited detail in the authorizing statute. We identified six key aspects of intent
applicable to the TIF program within the Committee Report in areas such as eligibility, project
sustainability, and stakeholder support. When comparing the Committee Report to the
Department’s two Federal Register Notices (FRN) inviting applications for new FY 2006 TIF
awards 3 and its Frequently Asked Questions (FAQ), we identified discrepancies in two
(33 percent) of six areas applicable to TIF program implementation as summarized below:

    1. Definition of a high-need school

        The Committee Report stated the conferees intended for the TIF program to award
        competitive grants to design and implement differentiated compensation systems for
        public school teachers and principals in high-need schools. It further indicated the
        conferees intended for the definition of high-need schools to be the same as defined in
        Section 2312 of the Elementary and Secondary Education Act (ESEA). However, the
        Department’s definition in both the FRNs and FAQs included only subpart (3)(A) of
        Section 2312 of the ESEA. That subpart defines high-need as the percentage of students
        from families with incomes below the poverty line. In excluding subparts 3(B)(i) through
        3(B)(iv), the Department omitted potential high-need schools as defined by factors such
        as: percentage of out-of-field teachers; number of unfilled, available teacher positions at
        the schools; teacher turnover rate; and percentage of teachers who are not certified or
        licensed. The discrepancies are highlighted further in the table that follows.




2
  Committee Report 12 of 50 - House Report 109-300 - Making appropriations for the Departments of Labor, Health
and Human Service, and Education, and Related Agencies for the Fiscal Year Ending September 30, 2006, and for
other purposes.
3
  Federal Register Volume 71, Number 83 dated May 1, 2006, and Federal Register Volume 71, Number 219 dated
November 14, 2006.
Final Audit Report
ED-OIG/A19I0007                                                                          Page 7 of 35

                        Discrepancies Between Committee Report and

                         Department Definition of High-Need School


                     Committee Report                             Department

         Public Law 107-110, Chapter B,                  Federal Register
         Transition to Teaching Program, Section         Notices/Frequently Asked
         2312, Definitions, (3) High-Need School,        Questions, High-Need School:
         dated January 8, 2002:


         (A) is located in an area in which the          A school with more than
         percentage of students from families with       30 percent of its enrollment
         incomes below the poverty line is 30            from low-income families,
         percent or more; or                             based on eligibility for free
                                                         and reduced price lunch
                                                         subsidies or other poverty
                                                         measures that the State
                                                         permits the LEAs to use.


         (B)(i) is located in an area with a high        Not included.
         percentage of out-of-field teachers, as
         defined in section 2102;
         (ii) is within the top quartile of elementary
         schools and secondary schools statewide, Not included.
         as ranked by the number of unfilled,
         available teacher positions at the schools;
         (iii) is located in an area in which there is   Not included.
         a high teacher turnover rate; or
         (iv) is located in an area in which there is
         a high percentage of teachers who are not       Not included.
         certified or licensed.



   2. Cost Sharing

      The Committee Report further stated the conferees intended that each applicant
      demonstrate a significant investment in, and ensure the sustainability of, its project by
      committing to pay for an increasing share of the total cost of the project, for each year of
      the grant, with State, local, or other non-Federal funds. The Department’s FRNs and
      FAQs included requirements for cost sharing at an increasing share, but they did not
      specify that the cost share must be with non-Federal funds. During our review of grant
      files, we found instances where Department officials advised recipients during
      monitoring activities that cost sharing could be made from other Federal funds. We
      further noted that the Department’s definition of cost sharing for the TIF program was
Final Audit Report
ED-OIG/A19I0007                                                                                     Page 8 of 35

        inconsistent with its Office of the Chief Financial Officer (OCFO)-04 “Handbook for the
        Discretionary Grant Process” (Handbook) which defined cost sharing as the portion of
        program costs that are not paid with Federal funds.

In the summer of 2010, during the course of this audit, an additional competition for TIF funding
was conducted. While this competition was outside the scope of this audit, we did review the
related FRN, dated May 21, 2010, for significant revisions to programmatic requirements. We
noted this FRN also included language that allowed the use of Federal funds for cost sharing,
specifically stating that

        In determining the adequacy of the support for the proposed project, the Secretary considers the
        extent to which … [t]he applicant will support the proposed project with funds provided under
        other Federal or State programs and local financial or in-kind 4 resources ….

The Department’s Handbook, issued February 24, 2006, 5 Section 4.3.2, Grantee Cost-Sharing or
Matching (Non-Federal Share), states

        The policies governing grantee cost-sharing or matching are as follows:
        1.	 Cost-sharing or matching is the portion of project/program costs not borne by the Federal
            government. The terms are used interchangeably and refer to either:
            a.	 A statutorily-specified percentage of project/program costs that must be contributed by a
                grant applicant in order to be eligible for funding; or
             b. Any situation where the applicant voluntarily shares in the costs of a project.

We asked Department officials about these discrepancies. In response, they expressed their
belief that Congressional reports were not binding for the Department's purposes in establishing
policy. They further stated that officials who worked on the initial policy decisions regarding the
TIF program were no longer with the Department. However, current Department officials
provided documentation that indicated the previous officials had examined several possible
definitions of high-need schools, including the definition contained in the conference committee
language. The documentation indicated that Department officials had concerns with how they
would be able to verify eligibility if the full definition of a high-need school from the Committee
Report was used. During our exit conference held on May 11, 2011, Department officials
reiterated that they did not consider the information in committee reports to be requirements and
believed they appropriately considered the legislative history of the TIF program when
implementing the program.

Correspondence within the official competition file for the first competition, also referred to as
Cohort 1, provided some explanation regarding the Department’s rationale for not limiting cost-
sharing to non-Federal funds. This included an email from a Department official indicating that
the Department’s policy officials in place at that time wanted TIF grantees to be able to use
Federal funds for matching purposes if that particular Federal grant program authorized the use
of its grant funds for differentiated educator compensation systems under its statute and
regulations.

4
  The term in-kind refers to non-monetary contributions.

5
  This 2006 version of the Handbook was in effect during the audit’s scope period. It was superseded by a revision 

dated January 26, 2009. All Handbook citations within this report are from the 2006 version; however, each citation

remains in effect in the 2009 version with no significant revisions.

Final Audit Report
ED-OIG/A19I0007                                                                          Page 9 of 35

The official also cited guidance in Section 5413(c) of the ESEA which stated

       Matching Funds – The Secretary may require recipients of awards under this subpart to provide
       matching funds from non-Federal sources, and shall permit the recipients to match funds in whole
       or in part with in-kind compensation.

The official stated that the Department determined it could read Section 5413(c) to mean that
because the Secretary could require grantees to use their own funds for matching requirements,
the Secretary could also permit grantees to use Federal grant funds for matching purposes. The
official further stated that Section 5413(c) did not preclude the use of Federal funds to meet a
match requirement, and while other interpretations were possible, this interpretation was one “we
could live with.”

Overall, the Department’s decisions may have: (1) filtered out prospective applicants who were
eligible to seek funding under the ESEA’s high-need school definition, but were ineligible under
the FRNs; and (2) inappropriately funded applications that proposed to use other Federal funds to
meet the TIF cost sharing requirement.

With regard to the high-need school definition, it was not possible to quantify whether any
potential program applicants were impacted by the Department’s decision to exclude subparts
3(B)(i) through 3(B)(iv) of Section 2312 of the ESEA under the FRN. With regard to cost-
sharing, we conducted a review of grant files for a judgmentally selected sample of
14 (41 percent) of 34 funded TIF applications. Our sample included $26.1 million (32 percent)
of the $80.3 million awarded to recipients in their first program year. We found that 4
(29 percent) of 14 funded applications sampled proposed to meet cost sharing requirements with
funds from other Federal sources. For example, one applicant indicated that each of its four
Local Educational Agency Partners planned to use Title II Teacher Quality Enhancement (TQE)
funds to provide the required match. A second applicant stated it would meet the cost sharing
requirement primarily through redirecting Title I funds. Other applicants were less specific in
the source of the potential matching through use of Federal funds. One applicant indicated that
in addition to specific funds identified in its application it “… may consider reallocation of some
of the other federal and state funds to provide for an increasing share of the differentiated
compensation.” Another stated that the matching funds would be provided “through local, state,
and federal funds.”

We reviewed the legislation creating Title I and II and determined it did not expressly authorize
the use of these funds for cost sharing with other Federal programs. As part of this review, we
further noted that TQE grantees are actually required to provide a contribution to their projects
with non-Federal funds in order to enhance the level of project services and to have the grantee
and its partners take some significant ownership of the project through investment of their own
resources.

Determination of Applicant Competence and Responsibility Prior to Grant Award

The Department may not have fully ensured applicant competence and responsibility prior to
funding grant applications. Department policy indicates program staff should consider an
applicant’s past performance and financial stability to ensure that the applicant is competent,
responsible and committed to carrying out proposed project activities. During our grant file
Final Audit Report
ED-OIG/A19I0007                                                                           Page 10 of 35

review, we found documentation of specific pre-award determinations that included items such
as budget reviews and funding recommendations. For example, program officials considered the
funding requested by an applicant in its budget narrative and then documented amounts the
Department recommended for funding. However, we found no documentation in any of the
14 grant files reviewed that showed program officials considered an applicant’s past performance
and/or financial stability in its decision to fund the TIF application.

The Department’s Handbook, Section 4.2.2, Determining Applicant Competence and
Responsibility, states

       ED [Department] policy requires grant recipients to be competent, responsible and committed to
       achieving the objectives of the awards they receive.
       1.	 In reviewing applications being selected for funding, the program staff should consider the
           following factors in determining an applicant’s ability to carry out the proposed project
           activities responsibly:
           a.	 The financial stability of the applicant;
           b.	 The applicant’s previous experience, if any, in administering Federal grants;
           c.	 Whether the applicant has adequate internal, fiscal and administrative controls (EDGAR
                [Education Department General Administrative Regulations] §§ 74.21 and 80.21);
           d.	 The applicant’s performance under other ED awards [EDGAR § 75.217 (d)(3)(ii)]; and
           e.	 Any adverse information regarding the applicant’s officials or key employees that calls
                into question their ability to perform satisfactorily.

       2.	 If the program staff finds evidence of problems regarding item 1, they must recommend that
           the program official take one of the following actions:
           a.	 Delay the award until the conditions are corrected;
           b.	 Deny the award; or
           c. 	 Designate the applicant as “high-risk” and make the award with special award conditions.

Program officials indicated they were unable to determine applicant competence and
responsibility because the Department did not have mechanisms in place to share information
related to subparts 1.a through 1.d in Section 4.2.2 of the Handbook. They specifically indicated
that GAPS did not have the functionality to provide program offices access to information
described in those subparts of the Handbook.

Program officials also indicated that audit findings related to discretionary grant programs were
rarely shared with program staff and were instead managed directly by OCFO. They stated this
made it difficult for program offices across the Department to share information from previous
audit findings that may be critical to an applicant’s future success with a program. They
believed these elements were why the Handbook indicates a program office “should” rather than
“must” perform competence and responsibility determinations as part of the applicant review
process.

During our exit conference, Department officials stated that some processes used to identify
potential concerns with applicant competence and responsibility may not be documented in grant
files. The process as described included the checking of potential grantees against the Excluded
Parties List System and the involvement of Office of the General Counsel (OGC) officials in the
approval of funding slates. Department officials believed that because of OGC’s involvement in
the review of all A-133, Government Accountability Office, and Office of Inspector General
Final Audit Report
ED-OIG/A19I0007                                                                        Page 11 of 35

(OIG) audit reports it was likely that concerns with a potential recipient’s ability to effectively
perform would be identified by OGC and appropriately considered prior to award. We noted
correspondence in the TIF grant competition files reflecting OGC’s review and approval of the
proposed funding slates. However, it did not contain information regarding the scope of the
OGC review. We also noted that the Department’s Handbook did not provide requirements for
the scope of OGC’s review of the funding slates. As such we cannot discern if the OGC review
did in fact include a determination of applicant competence and responsibility.

We believe the lack of a formal review of an applicant’s history, including financial stability and
performance under other ED awards, decreases the Department’s assurance that it is awarding
funds to a reliable applicant. The lack of ensuring applicant competence and responsibility
increases the risk of misuse of public funds and the inability to achieve programmatic goals.

During the course of our review, we noted the Department made progress in the development
and implementation of a Decision Support System (DSS) and a related component referred to as
the Entity Risk Review (ERR). The DSS is a suite of software analysis tools whose
implementation goals include the formalization of the Department’s processes for identifying
areas of potential risk in the Department’s grant portfolio and determining if special conditions
should be placed on a grant in the pre-award phase. The ERR is intended to provide financial,
administrative and internal controls information on grant applicants. Its objectives include
facilitating program offices’ efforts to analyze grantee risk prior to making an award; making
A-133 audit findings and other information about organizations’ fiscal health available and
accessible to program staff; and facilitating data sharing across grant programs and among
Department offices. A March 25, 2011, memo announced the initiation of a pilot project
utilizing these tools under the coordination of the Department’s Risk Management Service.
During our exit conference, Department officials stated that the Office of Elementary and
Secondary Education (OESE) was actively participating in the pilot project for these tools.

We further noted that subsequent to our exit conference, the Department issued Grant Bulletin
#11-01, dated May 16, 2011, “Assessing Grant Applicant and Grant Recipient Risk Prior to
Making New and Continuation Awards.” This is intended to provide discretionary grant
program staff with guidance for assessing an entity’s risk prior to issuing a new or continuation
award and for actions that may be taken when staff identify evidence of risk. The guidance
requires program offices to assess an entity’s risk prior to making an award by reviewing items
such as financial and performance information; compliance with Federal audit requirements;
relevant findings in audit and monitoring reports; and progress on corrective actions to resolve
audit findings. It also includes related documentation requirements.

Limitation of Funding to Applications Meeting Absolute Priorities

Program officials did not always limit funding to applications that fully met criteria established
in the TIF FRNs and further explained in the TIF FAQs. We identified inconsistencies in funded
applications with respect to absolute priority requirements in 3 (21 percent) of the 14 awards
reviewed.

We noted that 1 (7 percent) of the 14 funded applications reviewed lacked sufficient information
to ensure funds would be targeted to schools that are high-need. This applicant included four
new schools as high-need in its proposal although no school specific data was available. This
Final Audit Report
ED-OIG/A19I0007                                                                            Page 12 of 35

was cited as a weakness in each of the three application reviewer’s comments. Correspondence
from the recipient in the grant file indicated that the Department verbally agreed to include the
four new schools in year 1 of the applicant’s program based on district free/reduced price lunch
counts, with funding in years 2 through 5 contingent on actual school values once available in
spring 2008.

In addition, 2 (14 percent) of the 14 funded applications reviewed proposed incentive payments
that were not based primarily on student achievement as required. In one instance, payments
were to be based on equal weight between student achievement and teacher’s knowledge, skills,
and abilities. In the second instance, the proposal anticipated that 50 percent of the bonus system
for both teachers of record for math and reading and school principals would be based on
classroom observation and leadership assessment results.

Federal Register Vol. 71, No. 83, dated May 1, 2006, and Federal Register Vol. 71, No. 219,
dated November 14, 2006, define the following requirements for high-need schools and
compensation systems:

   1. High-need schools and compensation systems

       Section I. Funding Opportunity Description:

       Purpose of Program: The purpose of the Teacher Incentive Fund, authorized as part of the FY
       2006 Department of Education Appropriations Act, Public Law 109-149, is to support programs
       that develop and implement performance-based teacher and principal compensation systems in
       high-need schools ….

       Absolute Priority … Consistent with the program purpose, the grantee must establish a system
       that provides teachers and principals, or principals only, serving in high-need schools with
       differentiated levels of compensation based primarily on student achievement gains at the school
       and classroom levels.

       Definitions … A high-need school means a school with more than 30 percent of its enrollment
       from low-income families, based on eligibility for free and reduced price lunch subsidies or other
       poverty measures that the State permits the LEAs to use.

The FAQs included the following additional information regarding high-need schools and
compensation systems:

   Can the project include a school if they are just opening and does not know whether their
   student body is more than 30 percent eligible for free or reduced-price lunch? (Emphasis in
   original)
   No. Clear documentation, including a list of those schools proposed to be served by the grant with
   each of their free or reduced-price lunch eligibility rates, must be included in the application.

   What is differentiated compensation? (Emphasis in original)
   … Differentiated compensation for this program refers to a system of pay that is not exclusively
   defined by a career ladder or longevity in a job but rather by quality and effectiveness of
   performance. The differentiated compensation must be based primarily on improved student
   achievement at the school and classroom levels …
Final Audit Report
ED-OIG/A19I0007                                                                       Page 13 of 35

The Department implemented a screening process that included the use of a checklist to validate
key requirements and absolute priority requirements relating to the TIF application. However,
the execution of this system of internal control was not always effective. For example, one
checklist indicated the application did meet the requirement for high-need schools; however, it
also contained a handwritten annotation that it included four new schools, which was contrary to
the high-need guidance provided in the FAQs. In addition, the checklist included only partial
descriptions of FRN “must” requirements, to include basing compensation systems “primarily on
student achievement gains.”

Funding applications that did not fully meet selection criteria increases the risk of improper use
of funds, inadequate project execution, and/or increases the likelihood of future performance
problems. Failure to effectively ensure applicants meet established criteria prior to award could
diminish the public trust in the Department and reduce the applicants’ faith that the Department
appropriately conducts its discretionary grant competitions.

Consideration of Application Weaknesses and Impact on Program Implementation or
Sustainability

We noted weaknesses in 6 (43 percent) of 14 sampled funded applications that impacted or could
impact the recipient’s ability to effectively implement and sustain their differential compensation
systems. While the non-Federal reviewers may have scored the selection criteria in accordance
with applicable procedures, the scoring system employed did not apply sufficient weight to
impact the funding of these applications. The areas of weakness included the following:

   1. Plans to Sustain the Performance-Based Compensation System

       We determined that 5 (36 percent) of the 14 funded applications sampled did not include
       sufficiently detailed plans for sustaining the PBCS, but received program funding. We
       further noted that non-Federal application reviewers generally noted concerns in this area
       among the strengths and weaknesses identified in their respective Technical Review
       Forms.

       One funded applicant’s proposal, dated July 26, 2006, stated “written plans will be
       developed and implemented” regarding program sustainability. The proposal further
       indicated that it would develop plans to secure state and Federal grants and work within
       the community to build support for increasing a tax. The application anticipated that the
       primary method for supporting the performance based compensation system would be
       through a tax increase that was previously rejected by voters. While one of the three
       non-Federal application reviewers noted the funding identified was not guaranteed and a
       second reviewer indicated there was no assurance the project would be sustained, the
       impact of this issue on the overall application score was not significant enough to exclude
       the applicant from award.

       Subsequently, the Department’s TIF monitoring report for this recipient, dated
       May 2, 2008, included a finding that identified concerns regarding the sustainability of
       the project after the grant funding ends. The report specifically indicated that educators,
       administrators, and project staff expressed doubt that the district would be able to sustain
       the project after the grant period ended and required the recipient to submit a plan to the
Final Audit Report
ED-OIG/A19I0007                                                                       Page 14 of 35

      Department to ensure project sustainability. The recipient’s response to the monitoring
      report included many of the same items as were identified in its original proposal, such as
      the use of unidentified sources of state and Federal funds and the potential for an
      additional tax with the revenue going towards incentive pay programs. The plan included
      other items such as interest revenue, application of savings through maximizing
      classroom student levels, and investigating the creation of a foundation. However, it did
      not specify the level of funding projected to be available from the sources identified. As
      such it was incomplete, making it difficult to assess the reasonableness of the plan.

      A second funded applicant’s proposal, dated February 12, 2007, indicated it had
      commitments from partner districts to sustain the program at the conclusion of the grant
      period. We noted letters included in the application package from partner districts that
      indicated support for the TIF program; however, they made no specific commitment to
      sustain the PBCS, nor did they identify their individual plans for program sustainability.
      The applicant’s proposal indicated that planned funding sources included funds from
      Federal Title I, II, and V programs to gain additional resources for TIF sustainability and
      expansion. One of the three non-Federal application reviewer’s comments included “It is
      difficult if not impossible … to tell what amounts and percentage of costs will be
      undertaken by the districts in each year of the grant. There are no letters from any
      Federal Program directors nor superintendents promising to pick up costs after the
      program is over.” A second reviewer noted, “Letters stating that their [directors of the
      Federal programs] funds can be used for this purpose and a letter from the
      Superintendents stating that their district funds can be used in this manner need to be
      presented to support the plan for sustainability. Without such letters of financial
      commitment, the plan for sustainability is weak.”

      In its August 2008 self evaluation form, the recipient reported that partner districts
      submitted draft budgets and sustainability plans, but many were vague about specific
      funding sources to sustain compensation systems during and beyond the life of the grant.
      The recipient further stated it was providing assistance to aid in analyzing current budgets
      submitted by partner districts and identifying potential funding sources.

      We noted a third applicant included a cost-sharing model to demonstrate the transition
      from grant funding to district funding during the 5 year grant period as evidence of
      sustainability after the grant period ends. However, we found that the sustainability
      section of its application included no further information such as the planned source of
      funds to continue the project once the grant period ended. Two of the three non-Federal
      application reviewers identified weaknesses within this section of the proposal. Specific
      comments included “The cost sharing model for sustainability was not clear regarding
      sustainability after the grant period … there is no evidence for or narrative describing
      how sustainability will occur [after] the grant period” and “… for a project of this
      magnitude, more detailed information and supporting documentation should have been
      included.”

      We further noted that the Department’s TIF monitoring report for this recipient, dated
      April 16, 2008, identified concerns regarding the sustainability of the project after the
      grant funding ends. The report stated that all four districts participating in the project
      expressed doubt that districts would be able to sustain the project after the grant period.
Final Audit Report
ED-OIG/A19I0007                                                                     Page 15 of 35

      The recipient stated in its April 30, 2008, response that it would hold telephone
      conferences with each partner district and require each to submit a plan by June 30, 2008,
      that identified potential funding sources or plans of action (emphasis added) to obtain
      funding sources necessary to sustain the project beyond the period of Federal funding.

      Another applicant provided an estimate of its 2006-2007 budget that was allocated for
      various existing incentive measures and indicated it may consider reallocation of other
      Federal and state funds to provide for an increasing share of the differentiation
      compensation. However, we noted that it provided no firm plan or details regarding the
      ultimate source and usage of funds to sustain the TIF program at the end of the grant
      period. Two of the three application reviewers identified the lack of information on how
      the program will be sustained as a weakness.

      A final applicant stated that the proposed project would be sustained after the grant
      period ends through financial commitments made by the participating charter schools. It
      added the project was designed to establish a foundation during the grant term that would
      support long-term operation of the differential compensation system. However, there
      were no other substantive details provided. We reviewed letters of commitment from the
      charter schools included with the application. While the letters acknowledged the need to
      sustain the program beyond the period of the grant and committed to provide “such
      resources as are necessary,” they provided no additional detail regarding the actual
      sources and availability of related funds. We further noted that the proposal indicated the
      applicant planned to assist the charter schools in identifying and securing grant and other
      supplemental funding to sustain their PBCS; however, potential sources and amounts
      were unknown. Two of three non-Federal application reviewers noted concerns with the
      sustainability of this proposal. Related comments included: “The proposal does not have
      a detailed sustainability plan” and “The applicant did not clearly articulate assurance of
      the sustained funding for their proposal. General statements of financial support were
      made … but no definite commitment was made.”

   2. Performance-Based Compensation Systems

      We determined that 2 (14 percent) of the 14 sampled funded applications did not include
      adequate information in areas such as the project’s scope and compensation system
      design. One applicant’s proposal specifically identified year 1 of the program to be an
      “intensive planning year” during which stakeholders would “collaborate intensively in
      finalizing details of the compensation system and incentives to be provided.” As such,
      we found that concrete details on implementation were not included in its proposal,
      including the actual participating schools, performance levels needed to earn differential
      pay, and ranges of bonus payouts. Non-Federal application reviewers also noted
      weaknesses that included the need for more specific information to assess the needs of
      the 20 participating schools, lack of details regarding the performance-based reward
      system, a project design that was not fully developed, a program plan that is dependent
      on the planning phase of the project, and failure to establish specific plans for bonuses.

      During its program implementation, the recipient reported numerous problems in
      execution. This included delays incurred from shifting target schools to charter schools
      due to stakeholder rejection and a lack of necessary performance data. It also identified
Final Audit Report
ED-OIG/A19I0007                                                                             Page 16 of 35

       an inability to achieve elements of its proposal such as classroom evaluations for
       incentive purposes and filling leadership and mentoring positions, potentially due to the
       quality and instructional knowledge of teachers.

       Another applicant’s proposal indicated the project would develop a performance-based
       compensation system at each impacted school that is appropriate to the needs and
       circumstances of the school. We found that the proposal outlined a system where targets
       for individual student and classroom achievement would be established cooperatively by
       teachers and principals and would serve as performance measures upon which
       differentiated compensation for teachers will be based. As such, the fundamental basis
       for the PBCS was subject to development and not clearly identified in the proposal itself.
       Non-Federal application reviewers noted weaknesses including: “the plan is described in
       vague terms”; “no targets for student improvement are included”; “the applicant gives
       examples that could be used but does not conclude what would be considered acceptable
       or unacceptable gains”; and “Based upon the information presented, I could not
       determine how the compensation would actually work.” Information contained in the
       grant file indicated the recipient subsequently experienced difficulty and delays in the
       development of a predictive model planned as an essential element in its framework for
       performance based initiatives. It further indicated it had to reevaluate the use of
       academic predictions to distribute incentives relating to the 2008-2009 school year.

Selection criteria was included in Federal Register Vol. 71, No. 83, dated May 1, 2006, and
Federal Register Vol. 71, No. 219, dated November 14, 2006, Section V, “Application Review
Information.”

   Subsection (c), “Adequacy of Resources,” stated

       (2) The extent to which the applicant provides a detailed plan, including documentation of
       resources, for sustaining its performance-based compensation system after the grant period ends.

   Subsection (a), “Need,” stated

       The extent to which the applicant describes the scope and size of the project and the need for the
       project, including information on student academic achievement and the quality of the teachers
       and principals in the LEA(s) and high-need schools that will be served by the project.

   Subsection (b), “Project Design,” stated

       (2) The extent to which the applicant describes the performance-based teacher and principal
       compensation system that the applicant proposes to develop, implement, or expand, including the
       extent to which the applicant will build the capacity of teachers and principals through activities
       such as professional development to raise student achievement …

The selection criteria and underlying scoring system for TIF applications did not eliminate
proposals with noted review weaknesses that could be detrimental to program success from
award. There were a total of 13 sub-elements within the 5 selection criteria. The overall total
value of the 5 selection criteria was 100 points, with none of the sub-elements accounting for
more than 12.5 points. As non-Federal reviewers take into account both the strengths and
weaknesses of any given sub-element, significant weaknesses noted may not end up having a
Final Audit Report
ED-OIG/A19I0007                                                                          Page 17 of 35

substantial impact on the total overall score, as was the case in some of the applications we
reviewed. For example, in the applications where we noted concern with sustainability, the
average score resulted in an overall deduction of only five points. In addition, applicants could
earn 10 extra points by adequately addressing two non-mandatory competitive priority areas,
which could help to offset points lost among the 5 selection criteria. As identified above,
funding applications with the types of weaknesses noted has and could continue to impact a
recipient’s ability to effectively implement and sustain their differential compensation systems.

The Department provided written comments to our draft findings subsequent to our exit
conference. In its informal response, the Department expressed concern that OIG took
non-Federal reviewer comments out of context and, therefore, did not accurately or completely
reflect the full nature of the commenter’s views. The Department further expressed that
strengths and weaknesses must be read together in order to have a full understanding of a
reviewer’s score. We found that related non-Federal reviewer comments generally included both
strengths and weaknesses within each selection criteria; however, the weaknesses described
above were deemed by OIG to be significant enough to impact the affected recipient’s ability to
effectively implement and sustain their differential compensation systems, and as noted, did
actually impact recipient progress and performance in several instances. We concluded that the
scoring system employed was not designed in a manner where significant weaknesses within
individual sub-elements would have a material impact on the applicant’s overall score.

As noted previously, an additional competition for TIF funding was conducted during 2010.
Through the review of the related FRN, we noted that the Department appeared to place greater
significance on the financial sustainability of the performance based compensation system by
making it an absolute priority rather than a part of scored selection criteria as in the first two TIF
competitions.

Recommendations

We recommend that the Assistant Secretary for OESE

1.1	   Develop and implement procedures to ensure that the intent of Congress and internal
       guidance is effectively considered and applied in the establishment of criteria for
       discretionary grant competitions.

1.2	   Revisit the TIF program requirements in conjunction with legislative intent and internal
       guidance prior to the next competition and make any necessary changes.

1.3	   Work with OCFO to establish and implement a process that ensures information
       necessary to effectively determine applicant competence and responsibility is available to
       program staff prior to making application funding decisions.

1.4	   Through training or another appropriate method, ensure that program staff performing
       grant application screening functions have expert knowledge of programmatic criteria
       and that documentation designed to assist in the screening process is complete and
       accurate with respect to the program requirements being evaluated.
Final Audit Report
ED-OIG/A19I0007                                                                        Page 18 of 35

1.5	   In future TIF competitions, consider implementation of a scoring system that ensures
       greater weight is applied to key areas.

Department Comments and OIG Response

OESE generally disagreed with Finding 1 overall based on its comments to the individual sub-
elements of the finding. It specifically expressed disagreement with three of the four finding
sub-elements. It did not specifically agree or disagree with the sub-element regarding
determination of applicant competence and responsibility prior to grant award. OESE disagreed
with one of the five recommendations made, recommendation 1.2, stating that it had already
properly taken into account the legislative history of the TIF statute. Comments specific to each
sub-element of Finding 1 and OIG’s response are provided below.

OESE Comments on Application of Congressional Intent and Internal Guidance

OESE disagreed with this sub-element of Finding 1. OESE stated that it properly took into
account the legislative history of the TIF statute and gave it due consideration in interpreting the
statute and implementing the program. It further stated that it considers information in
Congressional committee reports as a useful source of legislative history and while the reports
reflect the views of a committee they do not contain statutory language that reflects the enacted
views of the entire Congress. In addition, OESE stated that the TIF authorizing statute did not
address the definition of a high-need school or anything about grantee cost sharing contributions,
two specific matters that the Draft Audit Report claims were improperly implemented in
establishing rules for the first two TIF competitions. For these reasons, OESE stated it disagreed
with recommendation 1.2. OESE stated that because recommendation 1.1 is current practice, it
had no objection to it.

OIG Response

OIG believes that in the absence of detail in the authorizing statute for discretionary grant
competitions, as was the case with TIF, it is a better practice for the Department to ensure that
the intent of Congress as expressed in other vehicles such as committee reports is effectively
considered and applied in the establishment of criteria, even if the committee report and other
Congressional expressions are not binding. While we noted that the Department did consider the
applicable Committee Report, we also noted significant discrepancies between the expressed
intent and actual implementation in the two areas highlighted in the finding. The documentation
that was available for our review to support the Department’s decision making process
concerning the definition of high-need schools indicated that implementation decisions were
made to ease the eligibility verification processes for the Department. This decision potentially
filtered out prospective applicants who were eligible to seek funding under the intended
definition.

We also noted that the Committee Report expressed the intent that grantees should demonstrate a
significant investment in and ensure sustainability of their project by committing to pay for an
increasing share of the total cost of the project in each year of the grant with non-Federal funds.
We found that the actual implementation of the TIF program was inconsistent with this
expressed Congressional intent as well, since allowance was made for any Federal funding to be
used for matching. Implementation also contradicted the Department’s own internal guidance in
Final Audit Report
ED-OIG/A19I0007                                                                         Page 19 of 35

its Discretionary Grant Handbook which defined cost sharing as the portion of program costs that
are not paid with Federal funds.

Based on the reasons outlined above and in the related report finding, we disagree that the
Department’s practices ensured that Congressional intent and the Department’s internal guidance
were effectively considered and applied in the establishment of programmatic criteria. In
response to the Department’s, comments we revised the finding to clarify the non-binding nature
of the Committee Report.

OESE Comments on Determination of Applicant Competence and Responsibility

OESE did not specifically note whether it agreed or disagreed with this sub-element of the
finding, but did note that since the 2006 and 2007 competitions that were the subject of this
audit, the Department has significantly improved its process for systematically considering
applicants’ past performance and financial history prior to making awards. OESE specifically
referenced the development of the DSS and ERR systems and the issuance of a related grant
policy bulletin that were mentioned in the related finding. OESE indicated that further processes
had been implemented, to include discussions of risk assessment in each proposed funding slate
and pre-award conditions where relevant. OESE expected that as the Department gains more
experience with DSS and ERR and adopts even more systematic procedures for implementing
them, these tools will significantly assist staff in more effectively indentifying areas of potential
risk than the more informal procedures used in prior years, and will also provide information
about applicant competence and responsibility that is more useful for grant making and
determining the need for technical assistance than was available in the past.

OESE proposed rewording the statement of finding for this sub-element to read “Department
Officials Can More Comprehensively Consider Applicants’ Past Performance and/or Financial
History Prior to Making Awards.” It also noted agreement with recommendation 1.3 but
recommended that it be revised to include a reference to the DSS and ERR pilot currently being
conducted.

OIG Response

We considered the proposed rewording for the finding but have not modified the report
language, as it accurately reflects the cited criteria from the Department’s Handbook and
correctly characterizes the issues noted in the finding. We also considered the suggestion related
to the recommendation but concluded we should not be prescriptive in the manner in which the
Department ensures necessary information is available to program staff as that is a management
decision. As such the recommendation itself was not revised.

OESE Comments on Funded Applications That Did Not Fully Meet Absolute Priorities

OESE did not agree with this section of the finding, but agreed with the related recommendation
(1.4). OESE stated that the information cited in the report regarding the applicant’s proposal that
included four high-need schools with no specific data to confirm they met the poverty threshold
had little or no significance. OESE explained that this was because the applicant viewed Year 1
as a planning year to prepare for implementation and no project activities would have occurred in
these schools until Year 2, provided it could confirm that these schools met the definition of a
Final Audit Report
ED-OIG/A19I0007                                                                             Page 20 of 35

high-need school. OESE stated it agreed with the grantee’s proposal to provide project services
in these schools beginning in Year 2, after submission of the requisite poverty data, and therefore
had no need for this data before making the grant award. OESE further stated that the cited
FAQs are not regulatory and even if they were they would not be relevant in this circumstance.

OESE also disagreed with the draft report’s assertion that 2 of 14 funded applications reviewed
proposed incentive payments that were not based primarily on student achievement as required.
OESE’s response asserts that one of the applicants gave the same weight to student achievement
as it did to teacher’s knowledge, skills, and abilities (KSAs). However, OESE proceeds to
indicate the KSAs are further divided into four different sub-domains; thus student achievement
had more weight than any of the KSA sub-domains. With regard to the other applicant cited,
OESE noted that because 50 percent of the bonus system would be based on classroom
observation and leadership assessment results and the other 50 percent on gains in student
achievement, student achievement carried more weight than any other factor.

OIG Response

We disagree with OESE’s view that lack of poverty data for the cited schools had little or no
significance. The position taken by OESE is contrary to the requirement that the applicant
needed to provide assurance that its proposal was in compliance with the absolute priority to
serve high-need schools. The FRN relating to the competition under which this recipient applied
stated that the program must establish a system of differentiated compensation for teachers and
principals serving in high-need schools. This criterion was designated as an absolute priority.
The FRN specifically stated that only applications meeting this priority would be considered.
Further, the Department’s Handbook, Section 3.3.7, Eligible Applications, states the following:
       Under EDGAR § 75.216, applications are reviewed for funding only if:
           b.	 The applicant follows all of the procedural rules that govern submitting the application
               (such as the applicant proposes a funding amount that does not exceed the maximum
               award amount in the application notice)

In conjunction with the above, we also disagree with OESE’s view of the significance of the
FAQs. While we agree that the FAQs are nonregulatory, we noted that the FAQs for the
2010 TIF competition specifically stated that the guidance provided in the FAQs are viewed as
the Department’s interpretation of various programmatic statutory provisions and other
requirements governing the TIF program competition. As noted in the finding, the related FAQ
in this case specifically stated that a project could not include a school if it could not provide in
its application (OIG emphasis added) the related eligibility rates. Had other potential or actual
applicants been aware of the exception that the Department was willing to provide, as it did with
the cited applicant in the finding, more may have applied or those applying may have included
other schools. Guidance should be clear so that potential applicants understand what is required
of them in their applications. Arbitrarily deciding when or how to apply the guidance negates
the value of the guidance.

We also disagree with OESE’s position regarding the two applicants cited in the draft report that
proposed incentive payments that were not based primarily on student achievement as required.
In one case, the proposal clearly states that there are two teacher performance compensation
elements—50 percent of the proposed bonus relates to student achievement growth and the other
Final Audit Report
ED-OIG/A19I0007                                                                        Page 21 of 35

50 percent relates to teachers’ knowledge, skills, and responsibilities. While the latter lists four
domains that will be included in the element, to include instructional design and planning,
instructional implementation, classroom environment, and teacher responsibilities, one score is
given to the element as a whole. We do not consider the four sub-elements to be distinct
categories upon which incentive payments were going to be based, rather the overall cited
element of teacher KSAs. Further, this same applicant noted in its proposal that if principals
were considered for incentive payments, 75 percent of their bonuses would be based on fidelity
of implementation and 25 percent on school wide gains in Year 1. This would move to a 50/50
split in Year 2. Neither year would have incentive payments based primarily on student
achievement. OESE applies similar logic to the remaining applicant. As noted in the draft
report, that applicant indicated that 50 percent of the bonus system for both teachers of record for
math and reading and school principals would be based on the overall category of classroom
observation/leadership assessment results, with the remaining 50 percent proposed to come from
improved student achievement growth. There were no sub-elements noted for the former
element; the category as a whole was assigned a designated percentage of 50 percent upon which
a bonus would be based.

After considering OESE’s comments, we did not make any changes to the finding or related
recommendation.

OESE Comments on Funded Applications Containing Weaknesses that Could be Detrimental to
Program Implementation or Sustainability

OESE disagreed with this section of the finding and expressed its belief that it presented an
incomplete picture of the peer review process. OESE specifically believed that comments
quoted in the draft report were taken out of context and did not accurately or completely reflect
the full nature of the commenters’ views. OESE believed that both strengths and weaknesses
must be read together to have a full, complete, and appropriate understanding of the basis for a
reviewer’s score. OESE’s response further states that by selecting comments that point to
weaknesses in applications, while ignoring the comments that focus on strengths, the draft report
provides an incomplete, and therefore inaccurate, picture of the peer review process and
erroneously suggests that awards were improperly made.

OESE also noted that it understands that this section of the finding is intended to focus on
whether more weight should have been given to how well applicants addressed their
performance-based compensation systems’ sustainability plans after TIF project funding ends,
particularly in light of the fact that the draft report also pointed to similar concerns that were
noted in the context of program monitoring and grantee self-evaluations. OESE stated it
appreciates OIG’s concerns regarding the weight given to project sustainability, and noted that it
placed much emphasis on sustainability as a key requirement and selection criterion that
governed the TIF Cohort 3 competition and expects to continue this focus as it considers
proposals in the FY 2012 competition. OESE stated that it had changed the scoring system for
the FY 2010 competition to give more weight to the key area of sustainability and therefore
agrees with recommendation 1.5.



OIG Response
Final Audit Report
ED-OIG/A19I0007                                                                        Page 22 of 35


We agree that all reviewer comments must be considered to interpret the basis for a score within
any particular selection criterion. However, we do not agree with OESE’s assertions that the
draft report’s presentation within this area was incomplete, inaccurate, and erroneous. This
portion of the draft report finding was developed around the weaknesses OIG identified during a
review of sampled applications. During these reviews we identified concerns with respect to
some applications’ lack of sufficiently detailed plans for sustaining the PBCS or adequate
information in areas such as the project’s scope and compensation system design. The
weaknesses we identified were also recognized in some form by multiple non-Federal reviewers,
and, when considered in conjunction with noted strengths, were still deemed potentially
significant enough to impact the affected recipient’s ability to effectively implement and sustain
their differentiated compensation system. In several of these instances, we noted that recipient
progress and performance was actually impacted by these weaknesses. The draft report did not
in any way question the validity of reviewers’ scoring or state that the Department improperly
made awards. Rather it identified that the scoring system employed did not provide sufficient
weight to eliminate proposals from funding consideration that had noted weaknesses that could
be detrimental to program success.

After considering OESE’s comments, we did not make any changes to the finding or related
recommendation.


FINDING NO. 2 – The Department Did Not Always Effectively Oversee the TIF
                     Program to Ensure Funds Were Used as Intended and
                     Anticipated Recipient Performance Was Achieved

Improvements are needed in the Department’s processes to oversee recipient performance. We
determined the Department did not always effectively ensure recipients:

   1.	 Met applicable requirements prior to making continuation awards; and
   2.	 Implemented their programs consistently with major design elements of their approved
       proposals.

Overall, the monitoring of the TIF program may not always effectively identify and seek to
correct weaknesses in the recipient’s internal control structure or hold recipients accountable for
performance levels identified in their proposals. This may increase the risk that funds are not
used as intended and anticipated recipient performance is not achieved.

Ensuring Recipient Performance Prior to Making Continuation Awards

During our review of post award activity for 14 sampled funded applications, we noted instances
where continuation awards were made even though A-133 audit reports were not reviewed and
the Department’s review of annual grant performance reports (GPR) indicated a lack of
substantial progress. Specifically we noted:
Final Audit Report
ED-OIG/A19I0007                                                                                     Page 23 of 35

    1.	 Lack of A-133 Review

        We found 12 (86 percent) of 14 grants sampled lacked reviews of A-133 audits prior to
        continuation awards made during FY 2008. Documentation retained in the Department’s
        official files indicated that the value of these awards ranged from $1 6 to $7,145,714. The
        total value of the 12 awards was $25,712,481.

        We performed queries of the Federal Audit Clearinghouse’s Single Audit Database and
        reviewed related documentation for these 12 recipients as of the time that their
        continuation awards were made. We subsequently identified areas of concern for
        9 (75 percent) of the 12 recipients that received continuation awards totaling
        $22,906,887. Specifically:

             •	 3 (25 percent) of the 12 recipients had not submitted A-133 audits timely as of the
                date their continuation award was made. The most recent reports available for
                each at that time contained audit findings for Department programs. The related
                continuation awards totaled $4,787,272.
             •	 3 (25 percent) of the 12 recipients had not submitted A-133 audits timely. The
                most recent reports available at the time did not contain audit findings for
                Department programs. The related continuation awards totaled $4,863,046.
             •	 3 (25 percent) of the 12 recipients submitted their A-133 audit timely and the
                reports contained audit findings for Department programs. The related
                continuation awards totaled $13,256,569.

    2.	 Lack of Substantial Progress

        The Department made a continuation award for 1 (7 percent) of 14 funded applications
        reviewed even though a lack of substantial progress was noted in the grant files and other
        related documentation prior to the award. Specifically, the Department’s year 1
        monitoring checklist indicated the recipient had not made substantial progress and the
        project had a scope change due to conflicts with its teachers’ union. The Department’s
        September 24, 2007, memorandum, Approval of Funding Slate for FY 2007 Continuation
        Awards for the Teacher Incentive Program - Cohort 1, identified the recipient as failing
        to make substantial progress, citing the withdrawal of union support as a key factor and
        indicating the recipient had identified some charter schools that were willing to
        participate in the project. The memo further stated that the recipient would be provided
        with a letter with its continuation award documentation to outline specific milestones and
        tasks to be completed within a specified time. Similar information relating to the
        recipient’s performance was presented in the FY 2008 Monitoring Plan.

        The Department approved a continuation award of $1,048,208 on March 4, 2008. We did
        not locate documentation relating to tasks and milestones included with the related Grant
        Award Notification (GAN) or other correspondence retained in the recipient’s grant file.


6
 Two (14 percent) of the 14 recipients sampled received year 2 continuation awards of $1. Both of these recipients
had been identified by the Department as having significant year 1 grant funding balances available prior to the
awarding of the continuation funding.
Final Audit Report
ED-OIG/A19I0007                                                                            Page 24 of 35

Section 5.4.8 of the Handbook, OMB Circular A-133 Audit Review Prior to Issuing a
Continuation Award, states

   Program staff must review and document audit data, if applicable, before issuing a continuation
   award to a grantee.

       1.	 A grantee that expended $500,000 or more in Federal funds during the previous fiscal year
           must have an audit that meets the requirements of OMB Circular A-133. … The program staff
           can access the audit database maintained by the Federal Audit Clearinghouse, which has
           detailed information, at: http://harvester.census.gov/sac/
       2.	 If the review of the audit data reveals that the grantee failed to comply with the A-133 filing
           requirement, the program staff must contact PAG [Post Audit Group] to discuss any issues
           and coordinate follow-up with the grantee. If the grantee's audit report is missing, PAG
           contacts the grantee to inquire about the status of the missing audit report, establish a
           deadline for the grantee to submit the report to the Federal Audit Clearinghouse, and directs
           the grantee to the appropriate data collection forms. If the grantee fails to submit the report
           by the established deadline and all efforts to get the grantee to submit the report have failed,
           the program staff must consider not issuing a continuation grant. However, the program staff
           must consult with OGC before taking any final actions based on audit information. The
           program staff must document in the grant file all actions taken to bring the grantee in
           compliance with the A-133 requirement.
       3.	 If the A-133 audit data reveals that the grantee’s audit report had findings, the program staff
           should check with their ALO [Audit Liaison Officer] to determine whether the audit findings
           have been resolved or are being resolved and whether a Program Determination Letter (PDL)
           has been transmitted to the grantee. If a PDL has been transmitted, the ALO must obtain a
           copy for the official file. If the findings are serious, the program staff may need to:
                a.	 Recommend that the program official impose special conditions on the grant;
                b.	 Provide additional monitoring and technical assistance to the grantee; or
                c. 	Not make a continuation award.

Section 5.4.5 of the Handbook, Substantial Progress, states

   The project data included in a grantee’s annual Grant Performance Report must correspond to the
   scope and objectives that were established in the approved project application or any approved
   amendments. The determining factor in awarding a continuation grant is whether the recipient has
   made substantial progress within the scope of the approved application in attaining the objectives of
   the grant as evidenced by meeting the grant’s performance measures. The program staff must analyze
   each report to ensure that the grantee has made substantial progress toward reaching the objectives
   included in the grantee’s approved application …

   The program staff must recommend discontinuing funding to grantees that have not demonstrated
   substantial progress toward meeting project goals and objectives, unless the program official approves
   changes to the project that will enable the grantee to make substantial progress in succeeding budget
   periods [see EDGAR § 75.253 (a)(2)]. In such cases, the grantee must submit a plan describing how
   substantial progress will be made in the future to justify continued funding.
Final Audit Report
ED-OIG/A19I0007                                                                            Page 25 of 35

EDGAR, Part 75, Subpart D, Section 75.253, “Continuation of a multi-year project after the first
budget period,” revised as of January 1, 2008, states

   (a) The Secretary may make a continuation award for a budget period after the first budget period of
   an approved multi-year project if:
        (2) The recipient has either—
               (i) Made substantial progress toward meeting the objectives in its approved application;
               or
               (ii) Obtained the Secretary's approval of changes in the project that—
                    (A) Do not increase the cost of the grant; and
                    (B) Enable the recipient to meet those objectives in succeeding budget periods …

The Department’s Grant Award Notifications that were issued in conjunction with the initial
funding to TIF recipients, Terms and Conditions, states

   In accordance with 34 CFR [Code of Federal Regulations] 75.253 the Department of Education will
   consider continued funding if: … (3) the recipient has made substantial progress toward meeting the
   objectives in its approved application.

An OESE official indicated that A-133 audits were requested and reviewed as part of its on-site
monitoring process. We reviewed official file documentation as of November 3, 2008, for the 12
grants that did not have evidence of A-133 review and determined that only 2 (17 percent) of the
12 grant files had documented site visits prior to their FY 2008 continuation award. We noted
that five other files documented site visits after continuation awards were made. However, none
of the seven site visit reports indicated that A-133 reports were requested and reviewed within its
“Monitoring Activities” section. We reviewed the FY 2008 Monitoring Plan for TIF and noted
that A-133 audits were not specifically identified as a monitoring tool. We also reviewed the
TIF on-site Monitoring Guide and found the A-133 audits were not specifically identified in
either the “Documents Requested for On-Site Review” or the “Documents Provided in Packet
Prior to Visit” sections of the guide. In a related discussion, an OESE official believed that the
OESE front office received the A-133 reports and was supposed to provide them to the TIF
program group, but the TIF program group never received them.

Although the file documentation for a TIF recipient indicated a lack of substantial progress, the
program office recommended the recipient for continuation funding into the next performance
year. In a related discussion, OESE officials stated that implementing a differentiated
compensation system was a complicated and often difficult process for recipients.

Without a formal review of A-133 audit data, OESE lacks assurance that grantees are in
compliance with reporting requirements and are appropriately managing Federal funds.
Awarding funds to entities that are not showing substantial progress and/or are not providing
corrective action plans to previously noted deficiencies increases the chance that an award will
be made to an unqualified entity, and increases the potential risk to the Department of fraud,
waste and abuse of program funds.

As previously noted, subsequent to our exit conference, the Department issued Grant Bulletin
#11-01. Among other things, this requires program offices to assess an entity’s risk prior to
making an award to include a review of compliance with Federal audit requirements. The
guidance further requires funding slate memos to include a risk assessment section. This
Final Audit Report
ED-OIG/A19I0007                                                                      Page 26 of 35

assessment must include A-133 compliance and findings, if applicable, as part of its risk
identification.

Ensuring Implementation of Funded Applications Consistent with Recipient Proposals

The Department did not always effectively ensure that implementation of funded applications
was consistent with the recipient’s proposal. We found that 3 (21 percent) of 14 grants reviewed
included documentation in the related grant file that showed the execution of the TIF program
included significant revisions to proposed scope and/or objectives, to include differentiated
compensation systems and/or the size of the project. Specific examples of these occurrences are
annotated below.

   1. Changes to proposed differentiated compensation systems

       One recipient’s grant file included documentation outlining changes that impacted its
       differential compensation system. The original proposal stated a limited number of
       exemplary teachers would receive payouts as a high as $10,000. Other teachers at
       impacted schools would receive “nominal” incentive awards ranging from $750 to
       $1,000. The grantee actually ended up giving all instructional staff in award winning
       schools equal payouts of $8,000 and reduced planned maximum awards to principals by
       half and to assistant principals by 40 percent.

       Other examples included a recipient that revised proposed incentive amounts and payout
       schedules. It reduced administrator award amounts by as much as 33 percent and
       increased awards made to all teachers at participating schools, regardless of their
       individual performance, by 150 percent. The recipient further scheduled delays in awards
       to exemplary teachers by approximately 2 years. Another recipient in our sample
       originally proposed to utilize classroom level achievement growth as part of its payout
       calculations; however, the recipient subsequently eliminated student achievement growth
       at the classroom level as part of the factors for performance awards.

   2. Size of the Project

       We found one recipient’s proposal planned to implement the TIF program at 20 public
       schools, with an estimated impact on 400 teachers, 20 principals, and at least 12,000
       students. The recipient modified its planned implementation to 20 charter schools;
       however, we could not identify documentation in the grant file that quantified the number
       of impacted teachers and students at these charter schools. In September 2007, the
       recipient projected it would impact 5 charter schools with 175 teachers and 2,650
       students in year 2 of the program. In its undated Budget Period 2 GPR, the recipient
       indicated that it planned to expand participation to eight additional charter schools in
       year 3 of the grant.

       A second recipient implemented a revision that resulted in a smaller number of schools
       being eligible for awards when compared to the original program proposal. The recipient
       identified the overall impact as requiring slightly more funding to provide awards to one-
       third of the number of schools identified in the original model. The original proposal
       suggested the project would annually impact 25 schools and over 500 teachers.
Final Audit Report
ED-OIG/A19I0007                                                                           Page 27 of 35

       Documentation from the recipient dated October 24, 2008, indicated actual
       implementation would impact 6 schools and 171 “instructional staff.” We could not
       locate data in the grant file that identified the number of students impacted.

The Handbook Section 5.5.11, Prohibiting Changes to the Project Scope or Objectives of a
Grant, states

       EDGAR §§ 74.25 and 80.30 appear to give ED [Department of Education] program officials the
       authority to review and approve requests to change the scope or objectives of a discretionary
       grant. However, to ensure the integrity of ED’s competitive review process, ED does not permit
       changes to the scope or objectives of a grant except in rare cases where the competition for the
       grant was not truly competitive.

File documentation indicated the Department did not consider proposed revisions as changes in
scope and therefore approved the requests. In a related discussion, OESE officials stated that
they consider the requirements of EDGAR and the Discretionary Grant Handbook as part of their
review of proposed project revisions. They stated that OESE evaluates the degree to which the
proposed change impacts the overall outcome of the project and consults with an OGC attorney
if there is doubt regarding whether a proposed revision is a change in project scope. OESE
stated that they consider the integrity of the competition foremost when considering revision
requests and would not allow a grantee to maintain a grant when it is clear that the
scope/objectives will not be met.

As previously noted, the Department provided informal comments to our draft findings
subsequent to our exit conference. In this response, the Department indicated it is moving from a
focus on compliance to one that also focuses more on outcomes for the intended beneficiaries
served by the program. It further believed that in the Department’s programs that drive
innovation, it is especially difficult for applicants to anticipate programmatic changes that may
become necessary upon implementation at the local level. However, the items identified above
were considered significant enough by OIG to have potentially affected how the applications
were viewed during the application review and scoring process.

In discretionary grant programs, the Department awards funds on the basis of a competitive
process. Applications are subjected to a review process developed in consideration of legislative
and regulatory requirements and published selection criteria established for a program. The
purpose of the review process is to provide a basis for determination of which applications best
address the program requirements and are, therefore, most worthy of funding. Significant
deviations from proposals may undermine the outcome of the competition process and
compromise its underlying purpose, as well as program effectiveness.

Recommendations

We recommend that the Assistant Secretary for OESE

2.1	   Reinforce the need to conduct reviews of A-133 audits to program staff responsible for
       recipient monitoring and develop and implement processes to ensure that appropriate
       procedures for A-133 reviews are included in program monitoring plans and guides.
Final Audit Report
ED-OIG/A19I0007                                                                          Page 28 of 35

2.2	   Reiterate to staff that continuation funding should not be provided to grantees that are not
       showing substantial progress unless the specific terms allowing such funding as identified
       in its Handbook and EDGAR are appropriately documented, especially in light of the
       Department’s stated move from a focus on compliance to one that also focuses on
       outcomes for program beneficiaries.

2.3	   Document the types of recipient proposal revisions that should be considered scope
       changes and reinforce to program staff the need to hold recipients of discretionary grant
       funding accountable for the achievement of performance as identified in their respective
       grant applications.

Department Comments and OIG Response

OESE generally disagreed with Finding 2 overall based on its comments to the individual sub-
elements. OESE agreed with two of the three recommendations (2.1 and 2.2), but stated that
they reflect steps that have already been taken. OESE stated it is carrying out recommendation
2.3, but believed it was not appropriate for inclusion in the final audit report given its response to
the finding.

OESE Comments on Ensuring Applicants Met Requirements Prior to Making Continuation
Awards

OESE acknowledged that it made a continuation award to one recipient prior to ensuring that it
made substantial progress and that TIF program staff may not have reviewed A-133 reports prior
to making continuation awards. However, OESE stated that it disagrees with the finding
because: 1) the continuation award was an isolated incident and steps have been taken to prevent
this from recurring; and 2) program staff had substantial knowledge of the progress and
conditions of each TIF grantee and considered all information from monitoring reports and
contacts in its determination of whether to recommend continuation awards.

OIG Response

In its response, OESE acknowledges the conditions identified in the draft audit report existed.
As such certain internal control processes as required in Department guidance were not
effectively followed. These processes were intended to provide additional assurance that
grantees were in compliance with reporting requirements, adequately managing Federal funds,
and making substantial progress prior to receipt of additional funding. OESE stated that program
staff had substantial knowledge of the progress and conditions of each TIF grantee and
considered all information from monitoring reports and contacts in determining whether to
recommend continuation awards. However, as noted in the finding, we found that neither the
monitoring reports nor the TIF Monitoring Guide included a review of A-133 reports. Without
such a review, OESE staff lack assurance that grantees are in compliance with reporting
requirements and are appropriately managing Federal funds. Several of the grantees included in
our review had A-133 reports that noted audit findings for Department programs and/or had not
submitted reports timely as required.

After considering OESE’s comments, we did not make any changes to the finding or related
recommendations.
Final Audit Report
ED-OIG/A19I0007                                                                        Page 29 of 35

OESE Comments on Ensuring Recipients Implemented Programs Consistently with Approved
Proposals

OESE stated that it did not agree with the finding. The basis for OESE’s disagreement was that
in innovative programs such as TIF it may be difficult for applicants to anticipate some of the
changes that may become necessary when the program is implemented at the local level and
therefore may need to make changes to approved plans to achieve project goals and milestones.
OESE believed that an appropriate level of flexibility is necessary in order to promote successful
outcomes and can be provided so long as there is not a change in the scope and objectives of the
project. OESE further stated that it reviewed situations described in the draft audit report and
believed they were permissible changes of project design.

In addition, OESE did not consider recommendation 2.3 appropriate for inclusion in a final audit
report while it was being implemented. In its corrective action plan, OESE stated it created a
TIF policy team in January 2011 to work closely and regularly with the OGC TIF program
attorney to discuss individual grantee circumstances and determine whether changes in an
approved application reflect a change in scope. The TIF program office is working to create a
process to help guide TIF staff in identifying circumstances in which a grant recipient’s proposed
changes of project design would be acceptable and those in which changes would be considered
impermissible changes in scope. OESE stated this process will enhance the collaboration with
OGC on these matters.

OIG Response

OESE has not presented additional information beyond what was already considered by OIG in
developing the draft audit report. Our position remains that the items identified in this section of
the audit report may have potentially affected how the applications were viewed during the
application review and scoring process. This is because of the impact in key areas such as
differential payouts and the size of the project. In addition, we are encouraged by the actions
OESE notes it is taking in collaboration with OGC related to recommendation 2.3. However,
actions in process do not warrant removal of a recommendation from the audit report.

After considering OESE’s comments, we did not make any changes to the finding or related
recommendation.


FINDING NO. 3 – The Department’s Ability to Assess Overall Outcomes of Initial
                     TIF Awards May be Limited

During our review, we noted a weakness in the processes established by the Department to assess
program results related to the 34 initial TIF awards. We found that the Department had taken
steps to establish a process that would assess the program’s overall success in achieving its goals.
This included implementing selection criteria that evaluated the extent to which applicants
committed to participation in a national evaluation that would provide a common design
methodology, data collection instruments, and performance measures for all grantees funded
under this competition. Although the Department had contracted for a national evaluation of the
TIF program, we noted that parts of the evaluation that would have assessed the impact of
grantee awards on teacher and principal recruitment and retention and student achievement will
Final Audit Report
ED-OIG/A19I0007                                                                                   Page 30 of 35

not be performed. As a result, the Department’s ability to effectively quantify the progress and
impact of the initial 34 TIF awards towards the achievement of programmatic goals is likely
reduced.

In 2008, the Department contracted for a national evaluation of the TIF program with respect to
the 34 initial awards. 7 The selected contractor’s technical proposal included a feasibility study to
establish whether or not program related outcome studies were possible. The contractor’s
proposal noted that the feasibility study would determine whether performance pay systems had
been put in place as planned and if appropriate data was available to assess the program’s effects
on educators and students. If these items were in place, the contractor indicated that a related
outcome study could be performed that would include assessing the impact of the TIF program
on student achievement and teacher/principal recruitment and retention.

During 2010, the Department executed a new competition for the TIF program to award funds
made available through the American Recovery and Reinvestment Act and its FY 2010
appropriation. As part of this process, the Department conducted two separate competitions, a
TIF program competition and a TIF evaluation competition. TIF evaluation competition
applicants were required to meet additional criteria that included agreement for at least eight
schools to participate in a lottery process for implementing the differentiated effectiveness
incentive component of the performance-based compensation system, cooperation with data
collection, and commitment to evaluation. In return, these applicants were eligible for at least an
additional $1 million over their 5 year grant period.

The Department made 10 awards under the TIF evaluation competition. The Department
awarded a 5 year $7.9 million dollar contract to perform the related national evaluation. Its
purpose included an evaluation of performance-based compensation systems in districts
receiving 2010 TIF grants to determine the impact on student achievement, teacher/principal
effectiveness, and teacher/principal recruitment and retention.

A Department official indicated that the optional outcome studies related to the initial TIF
awards would not be pursued in order to avoid duplication of effort with the evaluation of the
2010 awards. The official indicated that the primary focus of the initial evaluation would
generally be limited to identifying aspects that facilitated or served as barriers and/or challenges
to the implementation of TIF at individual locations. Other Department officials indicated that
there would be difficulty in planning the study for the initial awards once the program was
underway and believed the study for the 2010 awards would be more comprehensive and a better
use of programmatic resources. At our exit conference, Department officials further indicated
that the outcome study was not pursued because of concerns that it may interfere with the
evaluation of 2010 awards for those recipients who received awards under both the initial and
2010 competitions. We reviewed the list of recipients for each and determined that 2 of the 34
initial recipients also appeared to receive funds under the 2010 evaluation competition.




7
 Public Law 109-149 -- Appropriations Law for the Departments of Labor, Health and Human Services, Education,
and Related Agencies for fiscal year ending September 30, 2006 – provided that five percent of the program funds
were available for activities that included program outreach and evaluation activities.
Final Audit Report
ED-OIG/A19I0007                                                                     Page 31 of 35

The Department has lost an opportunity to obtain meaningful data regarding the overall
performance of the 34 TIF grantees from the first 3 years of program implementation, to include
data on overall TIF program effectiveness. This data would provide valuable input for decisions
on future program funding as well as examples of aspects of implemented programs that have
proven successful at multiple sites, and complement the Department’s stated focus on program
outcomes.

Recommendation

We recommend that the Assistant Secretary for OESE

3.1	   Develop and implement an appropriate process that will provide meaningful data
       regarding the overall performance of initial TIF grantees and the program’s overall
       effectiveness with respect to its stated goals.

Department Comments

OESE stated that it did not agree with the negative implications of the finding, but agreed with
recommendation 3.1. OESE further stated its concern that pursuing the optional outcome studies
could impact an evaluation of 2010 awards in districts that had schools participating in multiple
TIF cohorts. OESE also believed that an interim study to be published in late fall 2011 may
provide information about the effectiveness and performance of grantees to date.

OIG Response

We were provided with an advance draft of the cited interim study dated June 2011 in response
to discussion at the exit conference. We acknowledge the final report may vary substantially
from the draft; however, we found no tangible information that specifically related to
programmatic performance and effectiveness in our review of the draft document. We identified
several comments in the draft report that indicated determination of program effectiveness could
not be made at this point in time, and any noted changes could not necessarily be attributed
solely to the TIF program.

We also noted that the limited comments relating to performance that were included in the draft
report related mostly to recruitment and retention and were based solely on opinions from
interviews conducted. Where limited attempt was made to quantify impact, it was based on
statements that had no data confirmation and at times had conflicting statements regarding the
level of impact.

After considering OESE’s comments, we did not make any changes to the finding or
recommendation.
Final Audit Report
ED-OIG/A19I0007                                                                               Page 32 of 35




                                       OTHER MATTER



The Department’s Implementation of the FY 2010 Competition May Increase
Recipient Performance Risk

The Department’s May 21, 2010, FRN Inviting Applications for New Awards for 2010 included
provisions that may increase performance risk among funded applicants. Specifically, the
Department required applicants to demonstrate in their applications that they had five core
elements in place. These elements consisted of:

       (a) A plan for effectively communicating to teachers, administrators, other school personnel, and
       the community-at-large the components of its [Performance Based Compensation System];

       (b) The involvement and support of teachers, principals, and other personnel (including input
       from teachers, principals, and other personnel in the schools and LEAs to be served by the grant)
       and the involvement and support of unions in participating LEAs (where they are the designated
       exclusive representatives for the purpose of collective
       bargaining) that is needed to carry out the grant;

       (c) Rigorous, transparent, and fair evaluation systems for teachers and principals that differentiate
       effectiveness using multiple rating categories that take into account student growth (as defined in
       this notice) as a significant factor, as well as classroom observations conducted at least twice
       during the school year. …

       (d) A data-management system that can link student achievement … data to teacher and principal
       payroll and human resources systems; and

       (e) A plan for ensuring that teachers and principals understand the specific measures of teacher
       and principal effectiveness included in the PBCS, and receive professional development that
       enables them to use data generated by these measures to improve their practice.

However, the FRN also allowed a potential planning period for an applicant that could not
demonstrate that it had each of the five core elements in place at the time of application. This
provision provided applicants with up to 1 year to use TIF funds to develop the core element(s)
that were lacking. Applicants were required to provide a plan for how they would establish the
lacking element(s) during the planning period and to further demonstrate in their performance
reporting that the lacking element(s) had been implemented.

Over review of the Department’s Handbook did not find this practice to be inherently
unallowable; however, Section 1.1, “Purpose,” states “… [the Handbook] is designed to create
consistent policies, standards and procedures for ED’s discretionary grant programs to ensure
that ED awards and administers Federal funds across every program in a fair and equitable
manner, for the benefit of all children and other learners.” By its nature, the allowance of a
planning period creates inconsistencies in standards applied to applicants and increases the risk
that programmatic performance may be negatively impacted and potential benefits delayed. One
Final Audit Report
ED-OIG/A19I0007                                                                         Page 33 of 35

of the 14 grantees reviewed as part of this audit had actually identified year 1 of its program to
be an “intensive planning year” during which stakeholders would “collaborate intensively in
finalizing details of the compensation system and incentives to be provided.” This grantee was
noted as experiencing significant problems in program execution in later years.

We suggest that the Assistant Secretary for OESE evaluate the potential benefits of funding
applicants that have undefined implementation plans for core program elements with respect to
the increased risk this presents and determine if this practice is in the best interests of the TIF
program.

Department Comments

OESE disagreed with the Other Matter but agreed with the related suggestion. OESE stated it
has used planning grants in a number of Department programs and has found this type of grant to
be an effective strategy to minimize risk in certain instances. It believed the planning period
provides an opportunity to obtain more detailed information about a grantee's potential
performance than would be available from review of a project application alone.

OIG Response

The Department provided no clear examples or evidence of the success of planning year usage in
mitigating risk. Because of this, no revisions will be made to the Other Matters section. Our
position remains that applicants without core elements in place present an inherently higher
performance risk after program award. This also creates potential for inconsistency in the
manner in which potential grantees are treated during the application review and award
processes.

In September 2011, OIG initiated an audit that included an objective to review and assess the
Department’s monitoring plans for funded applicants proposing a planning period, and determine
if the Department’s monitoring efforts ensured applicants made progress towards developing the
lacking core element(s) and mitigated related performance risk.
Final Audit Report
ED-OIG/A19I0007                                                                       Page 34 of 35



                 OBJECTIVES, SCOPE, AND METHODOLOGY



The objectives of our audit were to: 1) determine whether grant proposals are being evaluated
and awarded in accordance with authorizing statute and all applicable regulations; and
2) evaluate the effectiveness of program oversight to ensure that funds are used as intended,
anticipated recipient performance is achieved, and overall programmatic goals are met. To
accomplish our objectives we gained an understanding of internal control applicable to the
implementation of the TIF program. This included internal control related to: competition
planning; grant proposal receipt, review, rating, and initial award; and subsequent recipient
monitoring and continuation awards.

We reviewed the House Committee Report for the appropriation establishing the TIF program,
applicable laws and regulations, Department policies and procedures, Department competition
guidance, and the Government Accountability Office “Standards for Internal Control in the
Federal Government.” We reviewed the Department’s internal documentation to further our
understanding of the program’s implementation. We also reviewed documentation relating to
the Department’s TIF technical assistance contract and program evaluation contracts.

We conducted discussions with OESE officials and TIF program staff to obtain an understanding
of the history of the TIF program implementation and the competition and monitoring processes.
We also held discussions with officials in the Department’s Office of Planning, Evaluation, and
Policy Development and Institute of Education Sciences to gain additional understanding of the
program evaluation contracts.

The scope of our review was limited to TIF implementation related activities conducted during
the period October 1, 2005, through September 30, 2008. Our scope included two competitions
that were conducted in relation to FRNs inviting applications for new awards dated May 1, 2006,
and November 14, 2006. These competitions resulted in initial awards to 34 recipients worth
approximately $80.3 million. We also performed limited work pertaining to the 2010 TIF
competition, specifically with regard to significant changes in program implementation, to
include the related evaluation competition and any impact on the planned 2008 national program
evaluation.

To fulfill our first objective, we reviewed and assessed internal documentation made available by
OESE officials to determine whether grant proposals were evaluated and awarded in accordance
with authorizing statute and all applicable regulations. This included items such as grant
completion schedules, TIF competition guidance to include FRNs and FAQs, non-Federal
reviewer guidance, organizational charts and staff assignments, technical review plans, funding
slates and approval memoranda, and the contents of competition files.

We selected a sample of 14 (41 percent) of the 34 funded applications and reviewed the official
grant files maintained by the Department. Our sample consisted of 5 judgmentally selected
applications due to risk factors that included prior audit history, abnormal funding patterns, and
preliminary discussions with OESE officials; and 9 randomly selected applications. Our sample
included $26.1 million (32 percent) of the $80.3 million in initial awards. We reviewed the
Final Audit Report
ED-OIG/A19I0007                                                                        Page 35 of 35

contents of the official grant files with respect to applicable guidance to further assess the
application evaluation and award processes along with the effectiveness of the Department’s
oversight to ensure that funds were used as intended and anticipated recipient performance was
achieved. This included review of items such as applications and related attachments, non-
Federal reviewer Technical Review Forms, GANs, correspondence between the Department
and/or technical assistance contractor and the recipient, recipient prepared GPRs and quarterly
self assessments, and on-site monitoring reports.

We also selected and reviewed a random sample of 13 (27 percent) of 48 unfunded applications
from the competition completed in response to the FRN dated May 1, 2006. This was conducted
to determine if the related funding decisions appeared to be reasonable and in compliance with
established criteria. No exceptions were noted from this process and we did not expand our
review to unfunded applications from the second TIF competition.

To fulfill our second objective, we were provided supplemental information and conducted
additional discussions regarding the status of the Department’s program evaluation contracts
through May 2011 to ensure our understanding of related activity was complete and appropriate.
We also became aware of the Department’s third TIF competition conducted in relation to a FRN
dated May 21, 2010. We reviewed this information to gain an understanding of any significant
revisions to competition requirements.

Because there is no assurance that the judgmental sample used in this audit is representative of
the respective universe, the results should not be projected over the unsampled awards.

Use of computer-processed data for the audit was generally limited to assist the audit team in
addressing objective one. This data was specifically used to: establish the universe of applicants;
obtain applicant scoring and ranking information; and establish the universe of new and
continuation awards within our audit scope period. The computer processed data and its use in
addressing the first objective did not require an assessment of the reliability of information
systems. We utilized limited reports from GAPS and focused our data reliability testing to those
elements used in the audit. An alternate automated data source is not available to directly test the
accuracy or completeness of the data in GAPS. However, we conducted testing through
comparison of GAPS data to certain source documentation. This included validation of
computer-processed data against items such as non-Federal reviewer Technical Review Forms
and GANs for sampled applications. We further noted that GAPS is the official system of record
for the Department’s grants data and is widely used and relied on by Department officials. As a
result, we considered it to be the best available data for the purpose of our audit. Based on the
work conducted, we determined that the computer processed data were sufficiently accurate and
complete for purpose of this audit.

We conducted fieldwork at Department offices in Washington, D.C., from October 2008 through
May 2011. We provided our audit results to OESE officials during an exit conference conducted
on May 11, 2011. We conducted this performance audit in accordance with generally accepted
government auditing standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit objectives.
                                                                           Enclosure 1

             Acronyms/Abbreviations/Short Forms Used in this Report

A-133            An audit of a non-Federal entity that expends Federal awards above an
                 established annual threshold and is conducted in accordance with OMB
                 Circular A-133

ALO              Audit Liaison Officer

CAP              Corrective Action Plan

Department       U.S. Department of Education

DSS              Decision Support System

EDGAR            Education Department General Administrative Requirements

ERR              Entity Risk Review

ESEA             Elementary and Secondary Education Act

FAQ              Frequently Asked Questions

FRN              Federal Register Notice

FY               Fiscal Year

GAN              Grant Award Notification

GAPS             Grants Administration and Payment System

GPR              Grant Performance Report

Handbook         U.S. Department of Education, Office of Chief Financial Officer-04,
                 “Handbook for the Discretionary Grant Process”

LEA              Local Educational Agency

OCFO             Office of the Chief Financial Officer

OESE             Office of Elementary and Secondary Education

OGC              Office of the General Counsel

OIG              Office of Inspector General

OMB              Office of Management and Budget
PAG    Post Audit Group

PBCS   Performance-Based Compensation System

PDL    Program Determination Letter

TIF    Teacher Incentive Fund grant program

SEA    State Educational Agency

TQE    Teacher Quality Enhancement
                                                                                                      Enclosure 2

                       Department Response to Draft Report




                         WRITTEN COMMENTS
     FROM THE OFFICE OF ELEMENTARY AND SECONDARY EDUCATION IN
                          RESPONSE TO THE
        DRAFT AUDIT REPORT, DEPARTMENT'S IMPLEMENTATION OF
      THE TEACHER INCENTIVE FUND GRANT PROGRAM, ED-OIG/AI9JO007
                             October 7, 2011

The Office of Elementary and Secondary Education (OESE) appreciates the opportunity to
provide written comments to the Draft Audit Report, Department's Implementation of the
Teacher Incentive Fund Grant Program, ED-OIG/ AIOI0007, provided on August 11 ,201 1 (Draft
Audit Report).

As with all of the programs we administer, OESE strives to continuously improve the operation
of the Teacher Incentive Fund (TIF). We have used the information obtained from the
implementation of Cohorts I and 2 (the focus of this draft audit report) to infonn the competition
held in Fiscal Year (FY) 2010 and the implementation of that new program for Cohort 3. In
addition, we will use the data and information received from the evaluation study of Cohorts 1
and 2 (to be released in the fall 0(2011) to provide information to the Department and the "'field"
that should help others to morc effectively implement key refonns.

We recognize the challenges and the complexities of implementing such a new and innovative
program, and welcome the opportunity for further di scussions with OIG on some of the key
issues included in this Draft Audit Report. Our comments on the draft report should help to
clarify our ongoing efforts, as we strive to ensure that the funds Congress appropriates to the
Department for this program will significantly improve the quality of teaching and student
academic growth.

OESE's responses to this draft report are detailed below, organized by finding and
recommendation, and in the accompanying Corrective Action Plan. Any subsequent questions,
comments, or concerns should be addressed to:

Michael Yudin
Acting Assistant Secretary
U.S. Department of Education
Office of Elementary and Secondary Education
400 Maryland A venue, SW
Washington, DC 20202
FINDING NO. 1- The Department Did Not Always Evaluate and Fund TIF Grant
Applications in Accordance with Applicable Criteria or Guidance.

Finding No. 1.1 - The Department did not effectively apply Congressional intent and
relevant criteria in its implementation of the TIF program.

       Recommelldatiolls:
       1.1 Develop and implement procedures to ensure that the intent of Congress and
       relevant criteria is effectively considered and applied in the establishment of criteria
       for discretionary grant competitions.

       1.2 Revisit the TIF program requirements in conjunction with legislative intent and
       relevant criteria prior to the next competition and make any necessary changes.

Comments. OESE properly took into account the legislative history of the TIF statute and gave
it due consideration in interpreting the statute and implementing the program. OESE considers
the infonnation in Congressional committee reports appropriately when establishing criteria and
requirements for discretionary grant competitions; however, we do so with the understanding
that while Congressional committee reports provide a useful source oflegislative history and
reflect the views of a committee, they do not contain statutory language that reflects the duly
enacted views of the entire Congress. In this regard, we note that the TIF authorizing statute
addresses neither (1) the elements or definition of a "high-need school" nor (2) anything about a
grantee's cost-sharing contributions - the two specific matters the Draft Audit Report claims the
Department improperly implemented in establishing rules for the first two TIF competitions.
Thus, OESE disagrees with Finding No. 1.1. Since Recommendation 1.1 is our currcnt practice,
we have no objection to it. We disagree with Recommendation 1.2 for the reasons noted above.
At the same time, since our use oflegislative history is not deficient, there is no need for the
Recommendations.

In considering legislative history and committee reports, we consult with the Office of the
General Counsel (OGC) and rely on its advice. We understand that OGC will bc further
responding to the specific issue of the extent to which the Department needs to rely on
statemcnts contained in Congressional committee reports in its own response to the Draft Audit
Report.

Finding No. 1.2 - The Department could morc effectively dctcrmine applicant competence
and responsibility prior to making awards.

       ReCOlllmemiatioll :
       1.3 Work with OCFO to establish and implcment a process that ensures
       information necessary to effectively determine applicant competence and
       responsibility is available to program staff prior to making application funding
       decisions.



                                                2
Comments. Since the 2006 and 2007 TIF competitions that are the subject of this audit, the
Department has significantly improved its process for systematically considering applicants' past
perfonnance and financial history prior to making grant awards. Therefore, OESE agrees
generally with Recommendation 1.3.

OESE has had processes in place to identify potential grantees that may have a history of
financial instability or poor perfonnance and takes these and other factors into account before
grant awards are made. For example. TlF program staff have checked potential grantees against
the Excluded Parties List System (EPLS) for any adverse infonnation regarding an applicant ' s
officials or key employees that suggests an inability to perfonn satisfactorily. Additionally, slate
memos with the list of recommended applicants and further background information undergo
rigorous review through a Department clearance process before grants are awarded. This
clearance process includes, among other offices, OGC, which reviews A-l33 audits and OIG
audits as part of the Department's audit biage process, and when appropriate raises with program
offices significant capacity and performance issues of applicants noted in the audits and program
monitoring reviews.

OESE continuously works hard to improve its processes to more comprehensively consider
applicants' past perfonnance and financial history prior to making its grant awards. As noted
above, since the 2006 and 2007 TlF competitions, the Department has strengthened its ability to
more exhaustively examine risks in relation to applicants on program funding slates and now
makes relevant infonnation much more readily available to program and other offices as they
consider recommendations of applicants for award. These improvements include, as
acknowledged on pages 10- 11 of the Draft Audit Report, issuance of Grant BuUetin #11-01,
entitled "Assessing Grant Applicant and Grant Recipient Risk Prior to Making an Award," on
May 16,2011. Further improvements include the development of the Decision Support System
(DSS) and the Entity Risk Review (ERR), which OESE is piloting with a number of program
offices. In implementing this bulletin and DSS and ERR, OESE now requires a discussion of
risk assessments in each proposed funding slate, together with a discussion of relevant pre·award
special conditions for some grantees, and of relevant post-award conditions or other follow· up
for others. OGC and other revi ewing offices review those slate memos to ensure that areas of
risk are properly considered in determining whether applications recommended for fu nding
should be funded, and whether any proposed conditions are appropriate.

We firmly expect as the Department gains more experience with Grant Bulletin #11·01 and
systems like DSS and ERR, and adopts even morc systematic procedures for implementing them,
that these tools (I) will significantly assist OESE and other Department staff in more effectively
identifying areas of potential risk than the more infonnal procedures used in prior years, and (2)
will also provide infonnation about applicant competence and responsibility that is more useful
for grant making and determining the need for technical assistance than was available in the past.

Based on the considerations noted above, a more accurate statement of Finding No. 1.2 would
be: "Department Officials Can More Comprehensively Consider Applicants' Past Performance
and/or Financial History Prior to Making Awards." In addition, while we agree with


                                                 3
Recommendation 1.3, we recommend that it be revi sed to include a reference to the DSS and
ERR pilot currently being conducted by RMS.

Finding No. 1.3 - The Department funded applications that did not fully meet the
Department's absolute priorities for the competitions

       Recommendatioll :
       1.4 Through training or another appropriate method, ensure that program staff
       performing screening functions have expert knowledge of programmatic criteria
       and that documentation designed to assist in the screening process is complete and
       accurate with respect to the program requirements being evaluated.

Comments. OESE disagrees with Finding 1.3, but agrees with Recommendation 1.4. One of the
assertions in the Draft Audit Report is that "1 (7 percent) of the 14 funded applications reviewed
lacked sufficient infonnation to ensure funds would be targeted to schools that are high-need"
(page II). The report notes that the applicant included four high-need schools in its proposal,
although no school-specific data was available to confinn that they met the 30 percent poverty
threshold. The report also notes that "[ c]orrespondence from the recipient in the grant file
indicated that the Department verbally agreed to include the four new schools in year I of the
applicant' s program based on district free/reduced price lunch counts. with funding in years 2
through 5 contingent on actual school values once available in spring 2008."

All oftbis is true, but is of little or no significance. The project application clearly expressed that
the applicant viewed Year I as a year in which to prepare for implementation, and sought TIF
funding for that year at a fraction of what it sought for its implementation in Years 2 through 5.
As no project activities would have occurred in these four schools during Year I, the applicant
merely put the Department on notice of its desire to begin implementation of its project in Year
2, provided it could confinn that these schools (out ofa total of27 identified schools) met the
definition of "bigh-need school" provided in the notice inviting applications. The Frequently
Asked Questions (F AQ), which the Draft Audit Report cites (on page 12) for the proposition that
a TIF application had to contain the requi site poverty data for all participating schools, are not
regulatory; however, even ifrhe FAQ the OIG cites did reflect a regulatory requirement. the lack
of poverty data for these four schools is not relevant in this circumstance. As the Department
agreed with the grantee's proposal to provide project scrvices in these schools beginning in Year
2, after submission of the requisite poverty data, the Department had no need for this data before
making the grant award for Year I.

OESE also disagrees with the statement that "2 (14 percent) of the 14 funded applications
reviewed proposed incentive payments that were not based primarily on student achievement as
required" (page I I). As the Draft Audit Report notes, one application gave the same weight to
student achievement as it did to teacher's knowledge, skills, and abilities. However, this
application proposed that "teacher' s knowledge, skills, and abilities" be measured by indicators
in four different domains: instructional design and plann ing, classroom envi ronment,
instructional implementation, and teacher responsibilities. Thus, student achievement has morc
weight in determining the incentive payment than any of these four domains and thereforc

                                                  4
satisfies the requirement that TIF programs include incentive payments "based primarily on
student achievement gains at the school and classroom levels."

As the Draft Audit Report notes, the other application proposed that for perfonnance-based
compensation "50 percent of the bonus system for both teachers of record for math and reading
and school principals would be based on classroom observation and leadership assessment
results" and the other 50 percent based on gains in student achievement. But here, too, student
achievement had more weight in determining the incentive payment than any other factor, and so
the proposal satisfies the TlF program requirement.

We, of course, share the OIO's general concerns (Draft Audit Report, page 12) about the need to
ensure that funded applications meet the rules and requirements that govern the program;
however, this was no problem in any of the examples the oro cites. We respectfully request that
this section of the Draft Audit Report be revised appropriately.

Finding No. 1.4 - The Department funded applications containing weaknesses that could be
detrimental to program implementation or sustainability.

       Recommelldation:
       1.5 In future discretionary grant competitions, consider implementation of a
       scoring system that ensures greater weight is applied to key areas.

Comments. OESE believes that Finding No. 1.4 presents a very incomplete picture of the peer
review process, and, therefore, we disagree with the finding. With regard to Recommendation
1.5, prior to receiving this Draft Audit Report, OESE changed the scoring system for the FY
20 10 competition to give more weight to the key area of sustainability. Therefore, we agree to
the Recommendation in this context.

OESE has carefully reviewed the reviewers' comments on the applications identified in the Draft
Audit Report, and concludes that the comments quoted on pages 13-1 6 of the Draft Audit Report
are taken out of context and do not accurately or completely reflect the full nature of the
commenters'views. The comments excerpted in the Draft Audit Report focus solely on
weaknesses found in TIF applications and fail to acknowledge the strengths identified in these
same applications. OESE maintains that strengths and weaknesses must be read together in
order to have a full, complete, and appropriate understanding of the basis for a reviewer' s score.

For example, the Draft Audi t Report references the following quotes made by reviewers: "The
proposal does not have a detailed sustainability plan" and "[tJhe applicant did not clearl y
articu late assurance of sustained fund ing for their proposal. General statements of financial
support were made ... but no definite commitment was made" (page 15). However, on the
technical review form, this same reviewer noted the following as a strength for the same
selection criterion: "The applicant did a good job of illustrating how they will support the
program once the TlF has been exhausted. The letters of support from the community will also
ensure the success of the program."


                                                 5
During the peer review process, reviewers were instructed to note both the strengths and
weakncsses ofan applicant's response to each selection criterion and to ensure that their
numerical score was consistent with their comments. In fact, the technical review fonn that each
reviewer completed included a text box for "strengths" and a text box for " weaknesses." In all
the applications reviewed for this finding, OESE found that the numerical scores that reviewers
provided were consistent with their comments on the strengths and weaknesses. Thus, reviewers
appropriately balanced the weaknesses against the strengths and allotted points accordingly.

Additionally, we specifically infonn reviewers that they do not need to come to an agreement or
consensus on the final scores for an application. The Department uses external peer reviewers
because we want their expert judgments on applications and we expect diversity in judgments.
By selecting comments that point to the weaknesses in applications, while ignoring the
comments that focus on the strengths in applications, the Draft Audit Report provides an
incomplete, and therefore inaccurate, picture of the peer review process -- and one that
erroneously suggests that awards were improperly made. We request that the Draft Audit Report
be revised accordingly.

Moreover, OESE has worked aggressively with grantees post-award to address some of the
grantees' weaknesses noted in the peer review process and to assist in enhancing the chances for
the success of their projects. Recognizing that implementing these initial TIF projects would be
complicated and difficult for some grantees at first, OESE awarded a technical assistance
contract to the Center for Educator Compensation Refonn (CECR) and to Westat to help
grantees determine the level of assistance they would need to carry out their proposals and to
help us in our efforts to work with grantees on implementation. Subcontractorg to this contract
included Learning Point Associates, an affiliate of American Institutes for Research; Synergy
Enterprises; Vanderbilt University; and the University of Wisconsin-Madison. In accordance
with the contract's statement of work, CECR staff had expertise in providing the following areas
of technical assistance, which we anticipated that many grantees would need, and did need:

   •   Teacher/principal evaluation models
   •   Funding models for compensation systems
   •   Use of valid, reliable assessments by which projects can measure student achievement,
       particularly in grades and subjects where standardized State test data are not available
   •   Development and enhancement of staff and comm unity support for system reform
   •   Data management
   •   Project evaluation

Among other things, the contract required that, shortly after grants were awarded, the contractors
conduct needs assessments to determine grantees' areas of greatest need for technical assistance.
Pro&'Tam staff and technical assistance providers used these needs assessments to organize and
provide technical assistance to grantees throughout the project periods.

The Draft Audit Report noted instances (c.g., on pages 13-14) in which similar concerns about
sustainability also arose in the context of program monitoring and grantee self-evaluations. As
the report notes that project difticultics were si milar to ones identified in review of the original
                                                   6
proposal, we understand that Finding No. 1.4 is intended to focus on whether morc weight
should have been given to how well, in response to various sub-elements within the relevant
selection criteria, applicants addressed their perfonnance-based compensation systems'
sustainability plans after TLF project funding ends. For example, under the first two TIF
competitions, "sustainability" was included in selection criterion 3 (Adequacy of Resources) for
which applicants could receive up to a maximum of20 points. However, these 20 points were
divided among three factors, only one of which addressed sustainability.

OESE appreciates ~iG's concerns regarding the weight given to project sustainability. Indeed,
the Department placed much emphasis on sustainability as a key requirement and selection
criterion that governed the TfF Cohort 3 competition, and expects to continue this focus as it
considers proposals for what would be included in the FY 2012 competition. Similarly,OESE
acknowledges and understands OIG's concern (Draft Audit Report, pages 15-16) that some TlF
Cohort I and 2 grantce applications may have lacked desired specificity about the design of their
proposed performance-based compensation systems. Our discovery through program monitoring
of implementation challenges, such as those the draft report identifies, was a key factor in
prompting the Department to provide a required planning period for the Cohort 3 competition.
During thi s planning period, those awarded TIF grants would need to demonstrate to the
Department's satisfaction that they were implementing each of five·specified "core elements" of
their performance-based compensation system before the Department permitted those grantees to
use TIF funds to pay any performance-based compensation.

FINDING NO.2 - The Department Did Not Always Effectively Oversee the TtF Program
to Ensure Funds Were Used as lntended and Anticipated Recipient Performance Was
Achieved.

Finding No. 2.1 - The Department did not always effectively ensure recipients met
applicable requirements prior to making continuation awards.

       Recommelldatiolls:
       2.1 Reinforce the need to conduct reviews of A·133 audits to program staff
       responsible for recipient monitoring and develop and implement processes to ensure
       that appropriate procedures for A-l33 reviews are included in program monitoring
       plans and guides.

       2.2 Reiterate to staff that continuation funding should not be provided to grantees
       that arc not showing substantial progress unless the specific terms allowing such
       funding as identified in its Handbook and EDGAR are appropriately documented,
       especially in light of Department's stated move from a focus on compliance to one
       that also focuses on outcomes for program beneficiaries.

Comm ents. OESE believes thi s finding to be overl y broad, and agrees only in a very limited
degree with Finding No.2. Recommendations 2. 1 and 2.2 reflect steps that OESE has already
taken, and we agrcc with them.


                                                7
OESE acknowledges that we provided a continuation award to one recipient prior to ensuring
that it had made substantial progress in carrying out its project (compare Draft Audit Report,
page 18); however, this was an isolated incident that we have taken meaningful steps to correct
and to prevent from recurring. We also acknowledge that TIF program staff may not have
reviewed A-133 audit reports in the past prior to making continuation awards (please see our
response above to Finding No. 1.2 for steps taken to improve the process). Nevertheless, TIF
program staff, through monitoring and other ongoing work with TIF recipients, had substantial
knowledge of the progress and conditions of each TIF grantee, as documented in quarterly
reports and montWy call logs. OESE's TIF program staff considered all of the information
gleaned from monitoring reports and other recipient contacts when detennining whether to
recommend that OESE make continuation awards. Thus, we do not agree with Finding No. 2.1.
As we stated during the May II, 2011 exit conference, the extensive knowledge of OESE's TIF
program staff concerning the progress of grantees was extremely useful in making key decisions
in the process for considering continuation awards.

Finding No. 2.2 - The Department did not always effectively ensure recipients implemented
their programs consistently with major design elements of their approved proposals.

       Recommelldatioll:
       2.3 Document the types of recipient proposal revisions that should be considered
       scope changes and reinforce to program staff the need to hold recipients of
       discretionary grant funding accountable for the achievement of performance as
       identified in their respective grant applications.

Comments. OESE does not agree with Finding No. 2.2. Moreover, while we arc carrying out
Recommendation 2.3, we do not think it is appropriate for inclusion in the Final Audit Report
given our response to this Finding.

In an innovative program such as TIF, in which a grantee is committed to implementing
challenging reforms and, it may be difficult, ifnot impossible, for some applicants to anticipate
some of the changes that may become necessary when the program is implemented at the local
level. The grantees, therefore, may have to make some changes and adaptations to their original
approved plans in order to achieve their project goals and milestones. We strongly beli eve that
an appropriate levcl of flexibility is necessary for grantees implementing new and innovative
programs in order to help promote successful outcomes. This flexibility can be provided so that
it is consistent with the general requirement that there not be a change in the scope or objectives
of the project. OESE's program office staff work closely with OGC staff to ensure that only an
appropriate level of flexibility is pennitted in this context.

We have reviewed the vario us situations described in the Draft Audit Report at pages 21-22, and
find them to be pennissible changes of project design. Thus, we do not agree with Finding No.
2.2. And, as noted above, while we are implementing Recommendation 2.3, we do not think the
Recommendation is appropriate in the context of this Jack of support for the Finding.



                                                 8
FINDING NO.3 - The Department's Ability to Assess Overall Outcomes of Initial TIF
Awards May be Limited.

       Recommendation:
       3.1 Develop and implement an appropriate process that will provide meaningful
       data regarding the overall performance of initial TIF grantees and the program's
       overall effectiveness with respect to its stated goals.

Comments. OESE does not agree with the negative implications of this Finding, but agrees with
Recommendation 3.1 since it is a process that OESE already uses.

As we noted during the May II, 2011 exit conference, an important reason for not pursuing the
optional outcome sub-studies was concern that conducting the quasi-experimental study on
Cohorts 1 and 2 would potentially interfere with the conduct of the Institute of Education
Sciences' experimental study of Cohort 3 in districts that had schools participating in multiple
cohorts (e.g., Chicago Public Schools). OESE acknowledges and appreciates OIG's inclusion of
this rationale on pages 23-24 of the Draft Audit Report.

We also believe that the interim study on Cohorts I and 2, scheduled to be published in late fall
2011, will provide important information about the overall effectiveness and performance of the
grantees to date. We acknowledge that the data provided in the interim study may be primarily
qualitative; however, we are confident that these data will provide valuable information to the
field about the complexities, challenges, and approaches to successfully implementing teacher
incentive programs. OESE will use this infOimation, along with the informati on we obtain from
grantees' annual and final performance reports, to inform future TIF competitions, as well as
competitions for other innovative programs. The information in the interim stttdy report will
also be used to inform our work with current grantees. Thus, OESE believes that the correct
decision was made given the potential interference that sub-studies could have had on
conducting the experimental study on Cohort 3.

OTHER MATTER - The Department's Implementation of the FY 2010 Competition May
Increase Recipient Performance Risk.

       Suggestio,,:
       Evaluate the potential benefits of funding applicants that have undefined
       implementation plans for core program elements with respect to the increased risk
       this presents and determine if this practice is in the best interests of the TIF
       program.

Comments. The Department has used planning grants in a number of Department programs and
has found this type of grant to be an effective strategy to minimize ri sk in certain instances. The
planning period provides an opportunity to obtain morc detailed information about a grantee's
potential pcrfonnance than would be available from review of a project application alone. Thus,
OESE disagrees with the Other Matter presented here, but agrees with the suggestion.


                                                 9
As discussed at the exit conference on May 11,2011, OESE strongly believes 1hat there is often
great value in planning grants, particularly for new and innovative programs where there is
limited capacity in the field, and where we would not want to make a full investment in a
program at the outset. Indeed, in appropriate situations such as the TIF Cohort 3 grants, planning
grants have the potential to minimize recipient perfonnance risk, as they (I) provide OESE with
more detailed and comprehensive infonnation about a recipient's plans to implement project
activities prior to beginning full implementation that could ever be provided in a grant
application alone, and at the same time (2) provide grantees needed time to fully develop their
implementation plans in full consultation with key stakeholders. Furthennore, OESE believes
that planning periods have the potential to be of particularly high value in situations where the
success of a given grant is heavily dependent upon stakeholder buy·in. These situations include
the early iterations of the TIF competition, where there was little knowledge in the field at large
as to either the level of individual and collective teacher buy· in that could be expected or the
processes that were needed to secure that buy-in.

Given that the FY 2010 competition is already completed, a more informative statement of the
"Other Matter" would be: "The Potential Costs and Benefits of a Planning Period in the TLF
Program Should Be Further Evaluated."




                                                10