UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF INSPECTOR GENERAL Chicago/Kansas City Audit Region 111 N. Canal St. Ste. 940 8930 Ward Parkway, Ste 2401 Chicago, IL 60606-7297 Kansas City, MO 64114-3302 Phone (312) 886-6503 Phone (816) 268-0500 Fax (312) 353-0244 Fax (816) 823-1398 October 20, 2006 Control Number ED-OIG/A05G0011 Ms. Elizabeth Burmaster State Superintendent Wisconsin Department of Public Instruction P.O. Box 7841 125 South Webster Street Madison, WI 53707-7841 Dear Ms. Burmaster: This Final Audit Report, entitled Wisconsin Department of Public Instruction’s Reading First Program, presents the results of our audit. The purpose of the audit was to determine whether the Wisconsin Department of Public Instruction (WDPI) (1) developed and used criteria for selecting scientifically based reading research (SBRR) programs in accordance with the law and regulations, and (2) approved local education agencies’ (LEA) applications in accordance with the law and regulations. Our audit covered the period May 1, 2002, through September 30, 2005. BACKGROUND The Elementary and Secondary Education Act of 1965 (ESEA) as amended by the No Child Left Behind Act of 2001, signed into law on January 8, 2002, established the Reading First program. Reading First is a focused nationwide effort to enable all students to become successful early readers. The Reading First program increased the assistance to state educational agencies (SEA) and LEAs to ensure that every student can read at grade level or above no later than the end of grade three. Funds are dedicated to help states and LEAs eliminate the reading deficit by establishing high-quality, comprehensive reading instruction in kindergarten through grade three. The program is designed to select, implement, and provide professional development for teachers Our mission is to promote the efficiency, effectiveness, and integrity of the Department’s programs and operations. Final Report ED-OIG/ A05G0011 Page 2 of 12 using SBRR programs 1 and to ensure accountability through ongoing, valid, and reliable screening, diagnostic, and classroom-based assessment. The total appropriation for the Reading First Program was $993.5 million for fiscal year (FY) 2003 and $1.024 billion for FY 2004. SEAs can receive funds for a six-year period. SEAs award subgrants to LEAs on the basis of a competitive process. Wisconsin’s Reading First grant application was approved by the U.S. Department of Education (Department) in May 2003. For fiscal years 2002, 2003, 2004, and 2005 Wisconsin was awarded $11,487,497, $10,548,061, $11,548,493, and $11,512,243, respectively. In its application WDPI estimated that 139 LEAs and 382 schools were eligible to receive funding. WDPI awarded $26.9 million of Reading First subgrants to 26 LEAs in two rounds of grant competitions, (December 2003 and March 2004). AUDIT RESULTS WDPI developed and used criteria for selecting the SBRR programs in accordance with the law as interpreted by the Department. 2 WDPI used the research contained in the National Reading Panel (2000) report as a guide for establishing its reading program. The National Reading Panel (2000) report was provided as a reference in the Guidance for the Reading First Program issued by the Department on April 2002. WDPI also identified the five essential components of an effective reading program and required LEAs and schools to include these components in their reading program. These five components are the “essential components of reading instruction” in Section 1208(3) of the ESEA. WDPI required LEAs and schools to use A Consumer’s Guide to Evaluating a Core Reading Program-Grades K-3: A Critical Elements Analysis to evaluate instructional materials and reading programs for SBRR if they did not choose one of the five recommended programs set forth in WDPI’s approved application. However, WDPI did not always approve the LEAs’ applications in accordance with the law and regulations related to implementing high-quality programs. In its September 14, 2006, comments to the draft report, WDPI concurred with our recommendations. The comments are summarized at the end of the finding. The full text of WDPI’s comments on the draft report is included as an Attachment to this report. 1 SBRR is defined as research that applies rigorous, systematic, and objective procedures to obtain valid knowledge relevant to reading development, reading instruction, and reading difficulties; and includes research that employs systematic, empirical methods that draw on observation or experiment; involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn; relies on measurements or observational methods that provide valid data across evaluators and observers and across multiple measurements and observations; and has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review. 2 The Department's interpretation of SBRR is under review in another OIG audit conducted by our Region III office in Philadelphia. Final Report ED-OIG/ A05G0011 Page 3 of 12 FINDING – WDPI Did Not Ensure All Funded LEA and School Applications Met High Standards WDPI did not ensure that funded LEAs and schools could implement high-quality programs that met the requirements of the Reading First program. For nine LEAs awarded $5,844,522 in subgrants, WDPI did not maintain evidence that the applications met high standards. Those applications were lacking either review panel recommendations to be funded (to indicate a determination that the applications met the qualifications for high-quality listed in the legislation) or lacking documentation of modifications to address questions or concerns the review panel had with the applications. States Must Ensure LEAs Could Implement High-Quality Reading Programs Pursuant to 20 U.S.C. Chapter 31 (General Education Provisions Act) § 1232f, each recipient of federal funds shall maintain records as will facilitate an effective financial or programmatic audit for three years after the completion of the activity for which the funds were used. Title I, Part B, Section 1202 (c)(4) of the ESEA states that, to be eligible to receive a subgrant, an LEA shall submit an application to the SEA at such time, in such manner, and containing such information as the SEA may reasonably require. The Department issued Guidance for the Reading First Program, in April 2002, which states that the SEA must describe, in its application to the Department, its process and selection criteria and how it will ensure that the instructional materials, programs, strategies and approaches are based on SBRR. The SEA is responsible for ensuring that Reading First funds will go only to LEAs that will implement high-quality programs that meet the requirements of Reading First. Section 1.D. “Selection Criteria For Awarding Subgrants” of WDPI’s state application sets out the criteria for determining eligible LEAs and grant proposals. After defining eligible LEAs, the application continues: Grant Proposals from eligible LEAs must meet high standards evidenced by meeting or exceeding the application criteria, and must coherently integrate: • State-required valid and reliable outcome assessments; • State-required progress monitoring assessment; • Screening and diagnostic instruments from Wisconsin’s short list (See Section 1.B.ii.), which were selected from the classroom-based instruments reviewed for use in Wisconsin Reading First; • Instructional programs and aligned materials that include explicit and systematic instruction in the five essential components of reading instruction; • Aligned professional development plan; and • Dynamic instructional leadership. The quality of the program will be determined by an initial screening of the application and then through the external review panel. First and most important, each application must address the required components of the legislation. In addition, the review panel members will base their reviews on the benchmarks that the WDPI has developed in the technical review form. Section 1.E.v. “Subgrant Review Process” of WDPI’s state application stated that the review panel groups “must determine which applications address the criteria for high-quality listed in Final Report ED-OIG/ A05G0011 Page 4 of 12 the legislation and make recommendations to the Reading Leadership Team.” The review panel groups will provide the governor and the state superintendent a list of recommended LEAs that should be funded and those that should not be funded. This section further states that phone interviews will be conducted with LEAs to clarify any questions or concerns of the review panel. A timeline will be provided to the LEA to supply additional information and/or to address any budgetary concerns. Review panels were to 1) score the applications based on how well they addressed the required components in the Reading First legislation and 2) make recommendations for funding. However, WDPI did not require panels to state whether each application met or exceeded standard in each area. Therefore, the only assessment of whether the applications addressed the required components and met the high standards referred to in WDPI’s application was the review panel’s recommendation for funding. The review panel recommended applications to WDPI for funding, but it was the responsibility of WDPI to make the final decision. Officials from the Office of the State Superintendent told us that the application score and recommendation by the panel indicating how well the program met standard was not considered a basic requirement for eligibility. The score was just one of the factors considered equally with geographic location, assessment scores, poverty levels, and school “buy in.” The officials further stated that they believed applicants with high poverty levels and low test scores should be funded despite submitting a weak application. WDPI Did Not Maintain Evidence That Applications Met High Standards WDPI conducted two rounds of grant competitions. The first round resulted in six eligible LEA applications. WDPI funded five. The low number of LEA applications prompted WDPI to solicit additional applications. In the second round, WDPI received 23 valid applications, 3 including a revised application from the one not funded in the first round. WDPI funded 21. We reviewed the score sheets for all 28 LEAs that applied for a Reading First grant. Only 13 of the 26 LEA applications funded received the review panel’s recommendation “Should be Funded.” Review Panel’s Recommendation None Should not be Fund With Should Be Documented Funded Modifications Funded Total Funded LEAs 1 4 8 13 26 Unfunded LEAs 0 2 0 0 2 While the review panel provided comments on each reviewed application, it did not make a recommendation at all for one of the applications reviewed. Review panel’s comments also did not always identify the exact modifications needed. Of the eight recommended with modifications, only four identified the specific modification needed, and WDPI obtained information prior to award to satisfy the specific modifications identified. 4 The review panel’s 3 Two other applications were received but were not part of the competition: one LEA wasn’t eligible, and one LEA applied but withdrew before the application was read by the review panel. 4 One of the four LEAs was Madison, which subsequently withdrew from the program in October 2004 and returned the funds it was awarded. Final Report ED-OIG/ A05G0011 Page 5 of 12 comments documented the specific concerns they had with the four applications not recommended for funding. WDPI did not document how it resolved the review panel’s concerns prior to award for 9 5 of the 13 applications that lacked the review panel’s unqualified recommendation for funding. WDPI did not obtain amended applications. Even if it had obtained amended applications, it would not have been possible, without resubmitting the application to the review panel, to determine if the amendments would have changed the overall recommendation for these nine applications. Why WDPI Did Not Maintain Evidence WDPI officials indicated that the LEAs and schools in the state initially showed very little interest in applying for the Reading First grant, which resulted in only six eligible LEAs applying. According to WDPI officials, their approved Reading First application funding process required a geographic distribution of approximately 60 Reading First grants throughout the state. WDPI held a second round of grant competitions, personally called all of the eligible districts that had not shown a previous interest in Reading First, provided a grant writing workshop, and assisted the interested LEAs in the grant writing process. These applications were due by March 5, 2004. WDPI officials stated they were running out of time to get the awards made, which is why they addressed concerns and clarifications verbally and did not require LEAs and schools to submit modified applications. WDPI indicated it held discussions with funded LEAs and schools regarding concerns with the quality of the program proposed and the actions needed to address those concerns. However, these discussions were not always documented. WDPI’s application files contained handwritten notes documenting phone conversations with only four schools and no LEAs. These notes documented concerns only with the budget. The notes did not document a timeline for the school to supply additional information as specified in the WDPI application. We confirmed with officials at two LEAs, one recommended with modifications (Racine) and the other no recommendation indicated (Menomonie) that WDPI did call and discuss concerns with them prior to their receiving the Reading First grant. According to the officials at these two LEAs, WDPI did not discuss specific review panel comments or scores. The discussions were about concerns WDPI had with the proposal. With the Racine LEA, WDPI discussed the budget and changes that needed to be made to assessment goals, parent involvement, and staff development. The Menomonie official recalled discussing some changes but could not recall the specifics of those changes. The LEA’s did not document the specifics of these discussions. Both officials indicated that WDPI worked very closely with them the first year, helping to ensure the program was implemented properly, and WDPI monitoring staff held numerous conversations with LEA officials after the LEAs were awarded the Reading First grant. WDPI officials indicated that WDPI Reading First staff reviewed the reading panel’s comments prior to making awards; however, the documents provided by WDPI to support this were created in response to our finding. The process was not documented at the time the awards were being made. 5 The nine LEAs and their Reading First awards are None Documented: Menomonie, $461,108; Should not be Funded: Adams-Friendship Area School, $370,166; Racine, $936,029; Riverdale, $389,166; Monroe, $440,541; Fund With Modifications: Wausau, $1,369,811; Alma, $414,694; 21st Century Preparatory School, $681,808; and Tigerton, $781,199. Final Report ED-OIG/ A05G0011 Page 6 of 12 WDPI officials informed us that even after granting the awards it did not require the agreed to modifications to be documented because it ensured the programs met Reading First requirements through its monitoring process. According to WDPI monitors, they reviewed the LEA and school application and score sheets before conducting the 2004 fall reviews. However, because the exact modifications needed and indicated telephone discussions were not always documented, the monitors were not always aware of what specific actions needed to be made. WDPI officials insist that the monitors, using a detailed monitoring checklist, ensured that the LEAs and schools implemented the program in accordance with the required criteria set forth in the application. The checklist sections were tied to the same sections used in the scoring worksheets. However, specific items within the scoring worksheet sections did not directly correspond to items within the monitoring checklist sections. For example, under the Instructional Leadership section of the monitoring checklist, the only item reviewed is the Reading First Coordinator job description, but under this section of the scoring worksheet a number of conditions for Meets Standard were listed. Because WDPI did not adequately document how it determined funded applications met standards, we were unable to determine whether all Reading First funds were awarded to LEAs that had plans to implement high-quality programs that would comply with the Reading First requirements. WDPI awarded $5,844,522 to the nine LEAs that did not receive unqualified recommendations for funding. If those LEAs had not been funded, that amount could have been used to either fund additional schools or increase awards to approved schools in LEAs which had received recommendations for funding. Recommendations We recommend that the Assistant Secretary for Elementary and Secondary Education require WDPI to 1.1 For each of the nine LEAs in question, either provide support to demonstrate that its reading programs meet eligibility requirements or return the funding provided to the LEA to the Department. 1.2 Implement and follow policies and procedures to provide reasonable assurance that, in future competitions, all funded LEA applications clearly will demonstrate that the requirements of Reading First are met prior to award and that adequate documentation of the application review and award process will be maintained. WDPI Comments WDPI concurs with the recommendations. WDPI is taking corrective action that includes requiring the nine LEAs to update and amend their applications to ensure compliance with all aspects of Reading First. In October and November of 2006 WDPI will meet with the nine LEAs to review their applications to ensure the review panel’s concerns are addressed. WDPI also agreed to review its grant award process to ensure that applications are uniform and consistent, and that program managers understand that grant standards must be well documented and met prior to award. Final Report ED-OIG/ A05G0011 Page 7 of 12 OBJECTIVES, SCOPE, AND METHODOLOGY The objectives of our audit were to determine, for the period May 1, 2002, through September 30, 2005, whether WDPI (1) developed and used criteria for selecting the SBRR programs in accordance with the law and regulations, and (2) approved the LEAs’ applications in accordance with the law and regulations. To achieve our objectives, we reviewed Title I, Part B, Subpart 1 §§ 1202, 1203 and 1208 of the ESEA, as amended by the No Child Left Behind Act of 2001; Education Department General Administrative Regulations (EDGAR) §§ 76, 77, 80, 82, and 85; 20 U.S.C. § 1232f; Reading First Program Guidance issued by the U.S. Department of Education; and portions of the State of Wisconsin’s Procurement Manual for state competition laws. We interviewed officials from the Department, WDPI, four LEAs, and three schools. We also interviewed four Reading First grant readers that participated in WDPI’s subgrant application review process. We also reviewed documents provided by WDPI, including (1) the approved State of Wisconsin Reading First Grant Proposal, Title I, Part B, Subpart 1, dated May 12, 2003; (2) five LEA applications; (3) rubric scoring documents for all 28 LEAs that submitted applications; and (4) WDPI’s organization chart. In addition, we reviewed Wisconsin's statewide single audit reports for the fiscal years ending 2002, 2003, and 2004; and Reading First Monitoring Feedback Report State: Wisconsin, produced by American Institute of Research, dated September 19, 2005. We judgmentally selected a sample of 4 of the 26 funded LEA applications and one of the two unfunded LEA applications, based on the reading programs used prior to and after Reading First, to determine if WDPI awarded subgrants in accordance with the process identified in WDPI’s approved application and the ESEA. We also judgmentally selected two LEAs for site visits based on the first LEAs subsequent withdrawal from the program (Madison) and the second LEA (Kenosha) based on amount of Title I funds received, changes in reading programs before and after grant application approval, and approved program. We performed our audit work at WDPI’s offices in Madison, Wisconsin, Kenosha School District offices in Kenosha, Wisconsin, Madison School District offices in Madison, Wisconsin, and our Chicago/Kansas City offices from November 2005 through July 2006. We discussed the results of our audit with WDPI officials on June 23, 2006. We performed our audit in accordance with generally accepted government auditing standards appropriate to the scope of the review described above. Final Report ED-OIG/ A05G0011 Page 8 of 12 ADMINISTRATIVE MATTERS Statements that managerial practices need improvements, as well as other conclusions and recommendations in this report, represent the opinions of the Office of Inspector General. Determinations of corrective action to be taken, including the recovery of funds, will be made by the appropriate Department of Education officials, in accordance with the General Education Provisions Act. If you have any additional comments or information that you believe may have a bearing on the resolution of this audit, you should send them directly to the following Education Department official, who will consider them before taking final Departmental action on this audit: Dr. Henry L. Johnson Assistant Secretary U.S. Department of Education Office of Elementary and Secondary Education 400 Maryland Avenue, SW Washington, DC 20202 It is the policy of the U. S. Department of Education to expedite the resolution of audits by initiating timely action on the findings and recommendations contained therein. Therefore, receipt of your comments within 30 days would be appreciated. In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the Office of Inspector General are available to members of the press and general public to the extent information contained therein is not subject to exemptions in the Act. Sincerely, /s/ Richard J. Dowd Regional Inspector General for Audit Attachment Final Report ED-OIG/ A05G0011 Page 9 of 12 ATTACHMENT: WDPI Comments on the Draft Report The following three pages are the WDPI comments addressing the draft report findings and recommendations. The final report finding was unchanged, however we did modify recommendation 1.1: “require WDPI to for each of the nine LEAs in question, either provide support to demonstrate that its reading programs meet eligibility requirements or return the funding provided to the LEA to the Department.” WDPI’s response addresses the steps WDPI is taking to ensure compliance with statutory requirements. Final Report ED-OIG/ A05G0011 Page 10 of 12 State of Wisconsin Department of Public Instruction Elizabeth Burmaster, State Superintendent September 14, 2006 Richard J. Dowd, Regional Inspector General for Audit U.S. Department of Education, Office of Inspector General 111 North Canal Street, Suite 940 Chicago, IL 60606-7297 Dear Mr. Dowd: Thank you for the opportunity to comment on the findings and recommendations in your draft audit report of the Department of Public Instruction’s Reading First Program. We are committed to the implementation of high quality Reading First Programs in all funded districts. Finding The Wisconsin Department of Public Instruction (WDPI) Did Not Ensure All Funded LEA and School Applications Met High Standards The WDPI awarded grants to LEAs eligible for Reading First funding and all funded grants met minimum standards for funding. The WDPI acknowledged that the strengths of the proposal varied among the funded applicants. The department has worked extensively with all funded districts to ensure the implementation resulted in high quality programs consistent with requirements of the grant. The WDPI provided the following technical assistance to all funded projects. • Monitoring throughout the school year to insure compliance with program requirements (The WDPI monitoring checklists, as concurred by your audit team, tie directly to the sections used by the grant review teams scoring worksheets. The use of these checklists was to ensure that high quality programs are in place) • Working with district/local leadership during the summer reading academies to review grant requirements and standards • Providing schools with a 4 day reading academy for Reading First staff on implementing Scientifically-Based Reading Research (SBRR) methods in the classroom and training in proper administration of the DIBELS and ERDA testing instruments Mailing Address: P.O. Box 7841, Madison, WI 53707-7841 • Street Address: 125 South Webster Street, Madison, WI 53702 Telephone: (608) 266-3390 • Toll Free: (800) 441-4563 • FAX: (608) 267-1052 • TDD: (608) 267-2427 • Internet Address: dpi.wi.gov Final Report ED-OIG/ A05G0011 Page 11 of 12 • Providing a 2 week training for administration, not only in SBRR methods for the classroom and proper administration of the testing instruments, but also in grant management and the requirements of the Reading First grant • Providing 5 in-the-classroom follow-up visits to schools from INSIGHT, WDPI’s approved professional development provider, to further ensure that (SBRR) methods were implemented in Reading First classrooms • Providing at least 3 technical assistance workshops to Reading First building/district leadership • Providing on-going professional development opportunities for Reading First staff in the SBRR on working with: special needs & English Language Learners (ELL), in the best practices on classroom management and scientifically based reading center activities, which cover the 5 essential elements of reading instruction • Providing finance workshops to ensure proper use of funds and proper reporting Recommendation 1.1 Provide support to the nine LEAs that resolves the review panel’s concerns with the applications or return $5,844,522 to the Department. WDPI does concur with the recommendation made by your office to provide additional support to the nine LEAs sited in the audit to ensure districts that submitted weaker proposals implement high quality programs. These LEAs are: 21st Century Preparatory School, Adams-Friendship School District, Alma School District, Menomonie Area School District, Monroe School District, Racine Unified School District, Riverdale School District, Tigerton School District and Wausau School District. In the 2005 – 2006 school year the WDPI required the above LEAs to update and amend their applications in order to ensure compliance with all aspects of the Reading First requirements. In October and November WDPI will meet individually with each of these 9 LEAs to review their applications to ensure the review panel’s concerns are addressed. In addition, WDPI will provide the following services to ensure that high quality Reading First programs are implemented. • At least two additional classroom visits from INSIGHT consultants to target areas of weakness in the implementation of the 5 essential elements of reading • Additional monitoring and technical assistance visits from WDPI staff to ensure high quality standards are being met through grant compliance • Additional regional professional development opportunities for staff will be made available both this fall and next summer in the areas of differentiated instruction, vocabulary and assessment of ELLs, while still offering past trainings for new staff members • Additional training for local leadership, to be provided by the Central Regional Reading First Technical Assistance Center on November 9, 2006 Final Report ED-OIG/ A05G0011 Page 12 of 12 Recommendation 1.2 Implement and follow policies and procedures to provide reasonable assurance that, in future competitions, all funded LEA applications clearly will demonstrate that the requirements of Reading First are met prior to award and that adequate documentation of the application review and award process will be maintained. We agree with this recommendation and have already taken steps to ensure all discretionary grant programs administered by WDPI have consistent high quality standards in the application review and approval process. In the spring of this year our agency gathered information from all discretionary grant applications and reviewed: • grant award processes to ensure that the applications are uniform and consistent • grant review processes to ensure program managers understand that grant standards must be well documented and met to ensure the greatest chance for success WDPI will maintain evidence that all funded LEA applications clearly demonstrate that the requirements of Reading First are met prior to award and that documentation of the program application review and the award process is maintained. WDPI supports the Reading First program and will do whatever it takes to guarantee successful implementation of all its programs. Sincerely, /s/ Julie Enloe State Reading First Coordinator c: Maxine Hough, Director, Title I
Wisconsin Department of Public Instruction's Reading First Program
Published by the Department of Education, Office of Inspector General on 2006-10-20.
Below is a raw (and likely hideous) rendition of the original report. (PDF)