oversight

Wisconsin Department of Public Instruction's Reading First Program

Published by the Department of Education, Office of Inspector General on 2006-10-20.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                          UNITED STATES DEPARTMENT OF EDUCATION
                               OFFICE OF INSPECTOR GENERAL
                                               Chicago/Kansas City Audit Region

                   111 N. Canal St. Ste. 940                              8930 Ward Parkway, Ste 2401
                   Chicago, IL 60606-7297                                 Kansas City, MO 64114-3302
                   Phone (312) 886-6503                                   Phone (816) 268-0500
                   Fax (312) 353-0244                                     Fax (816) 823-1398



                                                             October 20, 2006
                                                                                                  Control Number
                                                                                                  ED-OIG/A05G0011

Ms. Elizabeth Burmaster
State Superintendent
Wisconsin Department of Public Instruction
P.O. Box 7841
125 South Webster Street
Madison, WI 53707-7841

Dear Ms. Burmaster:

This Final Audit Report, entitled Wisconsin Department of Public Instruction’s Reading First
Program, presents the results of our audit. The purpose of the audit was to determine whether
the Wisconsin Department of Public Instruction (WDPI) (1) developed and used criteria for
selecting scientifically based reading research (SBRR) programs in accordance with the law and
regulations, and (2) approved local education agencies’ (LEA) applications in accordance with
the law and regulations. Our audit covered the period May 1, 2002, through September 30,
2005.




                                                   BACKGROUND


The Elementary and Secondary Education Act of 1965 (ESEA) as amended by the No Child Left
Behind Act of 2001, signed into law on January 8, 2002, established the Reading First program.
Reading First is a focused nationwide effort to enable all students to become successful early
readers. The Reading First program increased the assistance to state educational agencies (SEA)
and LEAs to ensure that every student can read at grade level or above no later than the end of
grade three. Funds are dedicated to help states and LEAs eliminate the reading deficit by
establishing high-quality, comprehensive reading instruction in kindergarten through grade three.
The program is designed to select, implement, and provide professional development for teachers




          Our mission is to promote the efficiency, effectiveness, and integrity of the Department’s programs and operations.
Final Report
ED-OIG/ A05G0011                                                                                          Page 2 of 12

using SBRR programs 1 and to ensure accountability through ongoing, valid, and reliable
screening, diagnostic, and classroom-based assessment.

The total appropriation for the Reading First Program was $993.5 million for fiscal year (FY)
2003 and $1.024 billion for FY 2004. SEAs can receive funds for a six-year period. SEAs
award subgrants to LEAs on the basis of a competitive process.

Wisconsin’s Reading First grant application was approved by the U.S. Department of Education
(Department) in May 2003. For fiscal years 2002, 2003, 2004, and 2005 Wisconsin was
awarded $11,487,497, $10,548,061, $11,548,493, and $11,512,243, respectively. In its
application WDPI estimated that 139 LEAs and 382 schools were eligible to receive funding.
WDPI awarded $26.9 million of Reading First subgrants to 26 LEAs in two rounds of grant
competitions, (December 2003 and March 2004).




                                              AUDIT RESULTS


WDPI developed and used criteria for selecting the SBRR programs in accordance with the law
as interpreted by the Department. 2 WDPI used the research contained in the National Reading
Panel (2000) report as a guide for establishing its reading program. The National Reading Panel
(2000) report was provided as a reference in the Guidance for the Reading First Program issued
by the Department on April 2002. WDPI also identified the five essential components of an
effective reading program and required LEAs and schools to include these components in their
reading program. These five components are the “essential components of reading instruction”
in Section 1208(3) of the ESEA. WDPI required LEAs and schools to use A Consumer’s Guide
to Evaluating a Core Reading Program-Grades K-3: A Critical Elements Analysis to evaluate
instructional materials and reading programs for SBRR if they did not choose one of the five
recommended programs set forth in WDPI’s approved application.

However, WDPI did not always approve the LEAs’ applications in accordance with the law and
regulations related to implementing high-quality programs.

In its September 14, 2006, comments to the draft report, WDPI concurred with our
recommendations. The comments are summarized at the end of the finding. The full text of
WDPI’s comments on the draft report is included as an Attachment to this report.



1
  SBRR is defined as research that applies rigorous, systematic, and objective procedures to obtain valid knowledge
relevant to reading development, reading instruction, and reading difficulties; and includes research that employs
systematic, empirical methods that draw on observation or experiment; involves rigorous data analyses that are adequate to
test the stated hypotheses and justify the general conclusions drawn; relies on measurements or observational methods that
provide valid data across evaluators and observers and across multiple measurements and observations; and has been
accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous,
objective, and scientific review.
2
  The Department's interpretation of SBRR is under review in another OIG audit conducted by our Region III office in
Philadelphia.
Final Report
ED-OIG/ A05G0011                                                                        Page 3 of 12

FINDING – WDPI Did Not Ensure All Funded LEA and School Applications Met
High Standards

WDPI did not ensure that funded LEAs and schools could implement high-quality programs that
met the requirements of the Reading First program. For nine LEAs awarded $5,844,522 in
subgrants, WDPI did not maintain evidence that the applications met high standards. Those
applications were lacking either review panel recommendations to be funded (to indicate a
determination that the applications met the qualifications for high-quality listed in the legislation)
or lacking documentation of modifications to address questions or concerns the review panel had
with the applications.

States Must Ensure LEAs Could Implement High-Quality Reading Programs
Pursuant to 20 U.S.C. Chapter 31 (General Education Provisions Act) § 1232f, each recipient of
federal funds shall maintain records as will facilitate an effective financial or programmatic audit
for three years after the completion of the activity for which the funds were used. Title I, Part B,
Section 1202 (c)(4) of the ESEA states that, to be eligible to receive a subgrant, an LEA shall
submit an application to the SEA at such time, in such manner, and containing such information
as the SEA may reasonably require. The Department issued Guidance for the Reading First
Program, in April 2002, which states that the SEA must describe, in its application to the
Department, its process and selection criteria and how it will ensure that the instructional
materials, programs, strategies and approaches are based on SBRR. The SEA is responsible for
ensuring that Reading First funds will go only to LEAs that will implement high-quality
programs that meet the requirements of Reading First.

Section 1.D. “Selection Criteria For Awarding Subgrants” of WDPI’s state application sets out
the criteria for determining eligible LEAs and grant proposals. After defining eligible LEAs, the
application continues:

     Grant Proposals from eligible LEAs must meet high standards evidenced by
     meeting or exceeding the application criteria, and must coherently integrate:
      • State-required valid and reliable outcome assessments;
      • State-required progress monitoring assessment;
      • Screening and diagnostic instruments from Wisconsin’s short list (See Section
        1.B.ii.), which were selected from the classroom-based instruments reviewed for
        use in Wisconsin Reading First;
      • Instructional programs and aligned materials that include explicit and systematic
        instruction in the five essential components of reading instruction;
      • Aligned professional development plan; and
      • Dynamic instructional leadership.

     The quality of the program will be determined by an initial screening of the
     application and then through the external review panel. First and most important,
     each application must address the required components of the legislation. In
     addition, the review panel members will base their reviews on the benchmarks that
     the WDPI has developed in the technical review form.

Section 1.E.v. “Subgrant Review Process” of WDPI’s state application stated that the review
panel groups “must determine which applications address the criteria for high-quality listed in
Final Report
ED-OIG/ A05G0011                                                                                  Page 4 of 12

the legislation and make recommendations to the Reading Leadership Team.” The review panel
groups will provide the governor and the state superintendent a list of recommended LEAs that
should be funded and those that should not be funded. This section further states that phone
interviews will be conducted with LEAs to clarify any questions or concerns of the review panel.
A timeline will be provided to the LEA to supply additional information and/or to address any
budgetary concerns.

Review panels were to 1) score the applications based on how well they addressed the required
components in the Reading First legislation and 2) make recommendations for funding.
However, WDPI did not require panels to state whether each application met or exceeded
standard in each area. Therefore, the only assessment of whether the applications addressed the
required components and met the high standards referred to in WDPI’s application was the
review panel’s recommendation for funding.

The review panel recommended applications to WDPI for funding, but it was the responsibility
of WDPI to make the final decision. Officials from the Office of the State Superintendent told us
that the application score and recommendation by the panel indicating how well the program met
standard was not considered a basic requirement for eligibility. The score was just one of the
factors considered equally with geographic location, assessment scores, poverty levels, and
school “buy in.” The officials further stated that they believed applicants with high poverty
levels and low test scores should be funded despite submitting a weak application.

WDPI Did Not Maintain Evidence That Applications Met High Standards
WDPI conducted two rounds of grant competitions. The first round resulted in six eligible LEA
applications. WDPI funded five. The low number of LEA applications prompted WDPI to
solicit additional applications. In the second round, WDPI received 23 valid applications, 3
including a revised application from the one not funded in the first round. WDPI funded 21.

We reviewed the score sheets for all 28 LEAs that applied for a Reading First grant. Only 13 of
the 26 LEA applications funded received the review panel’s recommendation “Should be
Funded.”

                       Review Panel’s Recommendation
                       None         Should not be Fund With     Should Be
                       Documented Funded          Modifications Funded                       Total
         Funded LEAs    1            4             8             13                          26
         Unfunded LEAs 0             2             0             0                           2

While the review panel provided comments on each reviewed application, it did not make a
recommendation at all for one of the applications reviewed. Review panel’s comments also did
not always identify the exact modifications needed. Of the eight recommended with
modifications, only four identified the specific modification needed, and WDPI obtained
information prior to award to satisfy the specific modifications identified. 4 The review panel’s


3
  Two other applications were received but were not part of the competition: one LEA wasn’t eligible, and one LEA
applied but withdrew before the application was read by the review panel.
4
  One of the four LEAs was Madison, which subsequently withdrew from the program in October 2004 and returned
the funds it was awarded.
Final Report
ED-OIG/ A05G0011                                                                               Page 5 of 12

comments documented the specific concerns they had with the four applications not
recommended for funding.

WDPI did not document how it resolved the review panel’s concerns prior to award for 9 5 of the
13 applications that lacked the review panel’s unqualified recommendation for funding. WDPI
did not obtain amended applications. Even if it had obtained amended applications, it would not
have been possible, without resubmitting the application to the review panel, to determine if the
amendments would have changed the overall recommendation for these nine applications.

Why WDPI Did Not Maintain Evidence
WDPI officials indicated that the LEAs and schools in the state initially showed very little
interest in applying for the Reading First grant, which resulted in only six eligible LEAs
applying. According to WDPI officials, their approved Reading First application funding
process required a geographic distribution of approximately 60 Reading First grants throughout
the state. WDPI held a second round of grant competitions, personally called all of the eligible
districts that had not shown a previous interest in Reading First, provided a grant writing
workshop, and assisted the interested LEAs in the grant writing process. These applications
were due by March 5, 2004. WDPI officials stated they were running out of time to get the
awards made, which is why they addressed concerns and clarifications verbally and did not
require LEAs and schools to submit modified applications.

WDPI indicated it held discussions with funded LEAs and schools regarding concerns with the
quality of the program proposed and the actions needed to address those concerns. However,
these discussions were not always documented. WDPI’s application files contained handwritten
notes documenting phone conversations with only four schools and no LEAs. These notes
documented concerns only with the budget. The notes did not document a timeline for the
school to supply additional information as specified in the WDPI application.

We confirmed with officials at two LEAs, one recommended with modifications (Racine) and
the other no recommendation indicated (Menomonie) that WDPI did call and discuss concerns
with them prior to their receiving the Reading First grant. According to the officials at these two
LEAs, WDPI did not discuss specific review panel comments or scores. The discussions were
about concerns WDPI had with the proposal. With the Racine LEA, WDPI discussed the budget
and changes that needed to be made to assessment goals, parent involvement, and staff
development. The Menomonie official recalled discussing some changes but could not recall the
specifics of those changes. The LEA’s did not document the specifics of these discussions. Both
officials indicated that WDPI worked very closely with them the first year, helping to ensure the
program was implemented properly, and WDPI monitoring staff held numerous conversations
with LEA officials after the LEAs were awarded the Reading First grant.

WDPI officials indicated that WDPI Reading First staff reviewed the reading panel’s comments
prior to making awards; however, the documents provided by WDPI to support this were created
in response to our finding. The process was not documented at the time the awards were being
made.

5
 The nine LEAs and their Reading First awards are None Documented: Menomonie, $461,108; Should not be
Funded: Adams-Friendship Area School, $370,166; Racine, $936,029; Riverdale, $389,166; Monroe, $440,541;
Fund With Modifications: Wausau, $1,369,811; Alma, $414,694; 21st Century Preparatory School, $681,808; and
Tigerton, $781,199.
Final Report
ED-OIG/ A05G0011                                                                   Page 6 of 12


WDPI officials informed us that even after granting the awards it did not require the agreed to
modifications to be documented because it ensured the programs met Reading First requirements
through its monitoring process. According to WDPI monitors, they reviewed the LEA and
school application and score sheets before conducting the 2004 fall reviews. However, because
the exact modifications needed and indicated telephone discussions were not always
documented, the monitors were not always aware of what specific actions needed to be made.
WDPI officials insist that the monitors, using a detailed monitoring checklist, ensured that the
LEAs and schools implemented the program in accordance with the required criteria set forth in
the application. The checklist sections were tied to the same sections used in the scoring
worksheets. However, specific items within the scoring worksheet sections did not directly
correspond to items within the monitoring checklist sections. For example, under the
Instructional Leadership section of the monitoring checklist, the only item reviewed is the
Reading First Coordinator job description, but under this section of the scoring worksheet a
number of conditions for Meets Standard were listed.

Because WDPI did not adequately document how it determined funded applications met
standards, we were unable to determine whether all Reading First funds were awarded to LEAs
that had plans to implement high-quality programs that would comply with the Reading First
requirements. WDPI awarded $5,844,522 to the nine LEAs that did not receive unqualified
recommendations for funding. If those LEAs had not been funded, that amount could have been
used to either fund additional schools or increase awards to approved schools in LEAs which had
received recommendations for funding.

Recommendations

We recommend that the Assistant Secretary for Elementary and Secondary Education require
WDPI to

1.1    For each of the nine LEAs in question, either provide support to demonstrate that its
       reading programs meet eligibility requirements or return the funding provided to the LEA
       to the Department.

1.2    Implement and follow policies and procedures to provide reasonable assurance that, in
       future competitions, all funded LEA applications clearly will demonstrate that the
       requirements of Reading First are met prior to award and that adequate documentation of
       the application review and award process will be maintained.

WDPI Comments

WDPI concurs with the recommendations. WDPI is taking corrective action that includes
requiring the nine LEAs to update and amend their applications to ensure compliance with all
aspects of Reading First. In October and November of 2006 WDPI will meet with the nine
LEAs to review their applications to ensure the review panel’s concerns are addressed. WDPI
also agreed to review its grant award process to ensure that applications are uniform and
consistent, and that program managers understand that grant standards must be well documented
and met prior to award.
Final Report
ED-OIG/ A05G0011                                                                   Page 7 of 12




                 OBJECTIVES, SCOPE, AND METHODOLOGY


The objectives of our audit were to determine, for the period May 1, 2002, through September
30, 2005, whether WDPI (1) developed and used criteria for selecting the SBRR programs in
accordance with the law and regulations, and (2) approved the LEAs’ applications in accordance
with the law and regulations.

To achieve our objectives, we reviewed Title I, Part B, Subpart 1 §§ 1202, 1203 and 1208 of the
ESEA, as amended by the No Child Left Behind Act of 2001; Education Department General
Administrative Regulations (EDGAR) §§ 76, 77, 80, 82, and 85; 20 U.S.C. § 1232f; Reading
First Program Guidance issued by the U.S. Department of Education; and portions of the State
of Wisconsin’s Procurement Manual for state competition laws. We interviewed officials from
the Department, WDPI, four LEAs, and three schools. We also interviewed four Reading First
grant readers that participated in WDPI’s subgrant application review process.

We also reviewed documents provided by WDPI, including (1) the approved State of Wisconsin
Reading First Grant Proposal, Title I, Part B, Subpart 1, dated May 12, 2003; (2) five LEA
applications; (3) rubric scoring documents for all 28 LEAs that submitted applications; and (4)
WDPI’s organization chart. In addition, we reviewed Wisconsin's statewide single audit reports
for the fiscal years ending 2002, 2003, and 2004; and Reading First Monitoring Feedback Report
State: Wisconsin, produced by American Institute of Research, dated September 19, 2005.

We judgmentally selected a sample of 4 of the 26 funded LEA applications and one of the two
unfunded LEA applications, based on the reading programs used prior to and after Reading First,
to determine if WDPI awarded subgrants in accordance with the process identified in WDPI’s
approved application and the ESEA. We also judgmentally selected two LEAs for site visits
based on the first LEAs subsequent withdrawal from the program (Madison) and the second LEA
(Kenosha) based on amount of Title I funds received, changes in reading programs before and
after grant application approval, and approved program.

We performed our audit work at WDPI’s offices in Madison, Wisconsin, Kenosha School
District offices in Kenosha, Wisconsin, Madison School District offices in Madison, Wisconsin,
and our Chicago/Kansas City offices from November 2005 through July 2006. We discussed the
results of our audit with WDPI officials on June 23, 2006. We performed our audit in
accordance with generally accepted government auditing standards appropriate to the scope of
the review described above.
Final Report
ED-OIG/ A05G0011                                                                      Page 8 of 12



                            ADMINISTRATIVE MATTERS


Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken, including the recovery of funds, will be made by
the appropriate Department of Education officials, in accordance with the General Education
Provisions Act.

If you have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following Education Department
official, who will consider them before taking final Departmental action on this audit:

                              Dr. Henry L. Johnson
                              Assistant Secretary
                              U.S. Department of Education
                              Office of Elementary and Secondary Education
                              400 Maryland Avenue, SW
                              Washington, DC 20202

It is the policy of the U. S. Department of Education to expedite the resolution of audits by
initiating timely action on the findings and recommendations contained therein. Therefore,
receipt of your comments within 30 days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

                                             Sincerely,

                                             /s/
                                             Richard J. Dowd
                                             Regional Inspector General for Audit

Attachment
Final Report
ED-OIG/ A05G0011                                                                   Page 9 of 12




        ATTACHMENT: WDPI Comments on the Draft Report

The following three pages are the WDPI comments addressing the draft report findings and
recommendations. The final report finding was unchanged, however we did modify
recommendation 1.1: “require WDPI to for each of the nine LEAs in question, either provide
support to demonstrate that its reading programs meet eligibility requirements or return the
funding provided to the LEA to the Department.”

WDPI’s response addresses the steps WDPI is taking to ensure compliance with statutory
requirements.
     Final Report
     ED-OIG/ A05G0011                                                                                     Page 10 of 12



                                                    State of Wisconsin
                                              Department of Public Instruction
                                                   Elizabeth Burmaster, State Superintendent


     September 14, 2006


     Richard J. Dowd, Regional Inspector General for Audit
     U.S. Department of Education, Office of Inspector General
     111 North Canal Street, Suite 940
     Chicago, IL 60606-7297


     Dear Mr. Dowd:

     Thank you for the opportunity to comment on the findings and recommendations in your draft
     audit report of the Department of Public Instruction’s Reading First Program. We are committed
     to the implementation of high quality Reading First Programs in all funded districts.

     Finding

     The Wisconsin Department of Public Instruction (WDPI) Did Not Ensure All Funded LEA and
     School Applications Met High Standards

     The WDPI awarded grants to LEAs eligible for Reading First funding and all funded grants met
     minimum standards for funding. The WDPI acknowledged that the strengths of the proposal
     varied among the funded applicants. The department has worked extensively with all funded
     districts to ensure the implementation resulted in high quality programs consistent with
     requirements of the grant. The WDPI provided the following technical assistance to all funded
     projects.

                •    Monitoring throughout the school year to insure compliance with program
                     requirements (The WDPI monitoring checklists, as concurred by your audit team,
                     tie directly to the sections used by the grant review teams scoring worksheets. The
                     use of these checklists was to ensure that high quality programs are in place)
                •    Working with district/local leadership during the summer reading academies to
                     review grant requirements and standards
                •    Providing schools with a 4 day reading academy for Reading First staff on
                     implementing Scientifically-Based Reading Research (SBRR) methods in the
                     classroom and training in proper administration of the DIBELS and ERDA testing
                     instruments

    Mailing Address: P.O. Box 7841, Madison, WI 53707-7841 • Street Address: 125 South Webster Street, Madison, WI 53702
Telephone: (608) 266-3390 • Toll Free: (800) 441-4563 • FAX: (608) 267-1052 • TDD: (608) 267-2427 • Internet Address: dpi.wi.gov
Final Report
ED-OIG/ A05G0011                                                                    Page 11 of 12


         •   Providing a 2 week training for administration, not only in SBRR methods for the
             classroom and proper administration of the testing instruments, but also in grant
             management and the requirements of the Reading First grant
         •   Providing 5 in-the-classroom follow-up visits to schools from INSIGHT, WDPI’s
             approved professional development provider, to further ensure that (SBRR)
             methods were implemented in Reading First classrooms
         •   Providing at least 3 technical assistance workshops to Reading First
             building/district leadership
         •   Providing on-going professional development opportunities for Reading First staff
             in the SBRR on working with: special needs & English Language Learners (ELL),
             in the best practices on classroom management and scientifically based reading
             center activities, which cover the 5 essential elements of reading instruction
         •   Providing finance workshops to ensure proper use of funds and proper reporting

Recommendation

1.1 Provide support to the nine LEAs that resolves the review panel’s concerns with the
    applications or return $5,844,522 to the Department.

WDPI does concur with the recommendation made by your office to provide additional support
to the nine LEAs sited in the audit to ensure districts that submitted weaker proposals implement
high quality programs. These LEAs are: 21st Century Preparatory School, Adams-Friendship
School District, Alma School District, Menomonie Area School District, Monroe School
District, Racine Unified School District, Riverdale School District, Tigerton School District and
Wausau School District.

In the 2005 – 2006 school year the WDPI required the above LEAs to update and amend their
applications in order to ensure compliance with all aspects of the Reading First requirements. In
October and November WDPI will meet individually with each of these 9 LEAs to review their
applications to ensure the review panel’s concerns are addressed. In addition, WDPI will
provide the following services to ensure that high quality Reading First programs are
implemented.

         •   At least two additional classroom visits from INSIGHT consultants to target areas
             of weakness in the implementation of the 5 essential elements of reading
         •   Additional monitoring and technical assistance visits from WDPI staff to ensure
             high quality standards are being met through grant compliance
         •   Additional regional professional development opportunities for staff will be made
             available both this fall and next summer in the areas of differentiated instruction,
             vocabulary and assessment of ELLs, while still offering past trainings for new staff
             members
         •   Additional training for local leadership, to be provided by the Central Regional
             Reading First Technical Assistance Center on November 9, 2006
Final Report
ED-OIG/ A05G0011                                                                  Page 12 of 12


Recommendation

1.2 Implement and follow policies and procedures to provide reasonable assurance that, in
    future competitions, all funded LEA applications clearly will demonstrate that the
    requirements of Reading First are met prior to award and that adequate documentation of
    the application review and award process will be maintained.

We agree with this recommendation and have already taken steps to ensure all discretionary
grant programs administered by WDPI have consistent high quality standards in the application
review and approval process. In the spring of this year our agency gathered information from all
discretionary grant applications and reviewed:

         •   grant award processes to ensure that the applications are uniform and consistent
         •   grant review processes to ensure program managers understand that grant standards
             must be well documented and met to ensure the greatest chance for success

WDPI will maintain evidence that all funded LEA applications clearly demonstrate that the
requirements of Reading First are met prior to award and that documentation of the program
application review and the award process is maintained.

WDPI supports the Reading First program and will do whatever it takes to guarantee successful
implementation of all its programs.

Sincerely,

/s/
Julie Enloe
State Reading First Coordinator

c: Maxine Hough, Director, Title I