United States General Accounting Office Testimony For Release Impacts of Education Reform On Delivery Expected at lo:00 a.m. EDT Tuesday, March 7, 1989 Statement of Eleanor Chelimsky Assistant Comptroller General for Program Evaluation and Methodology Before the Subcommittee on Eiementary, Secondary and Vocational Education Committee on Education and Labor United States House of Representatives GAO/T-PEMD-89-2 Mr. Chairman and Members of the Subcommittee: I am very pleased to be here today to discuss GAO's work on education reforms and their impacts. Specifically, Mr. Chairman, you asked us for an evaluation of the effects of recent reforms on the achievement of disadvantaged students, on their dropout rates and on their enrollments in vocational education. My testimony today thus addresses a general concern about how the specific reforms that have imposed stricter new requirements in high schools might have affected the nation's most disadvantaged, at-risk students. On balance, our study shows that in the four large-city school districts we evaluated, education reEorm has been neither a disaster nor a boon for the performance of low-achieving students. Looking at the performance of all students in these districts, we did not find that the reforms we examined improved education outcomes a great deal. Finally, prior progress was not maintained for some disadvantaged groups, at least over the short term. The big question, then, is the longer term: whether performance will improve over time. It is important to note that the students we studied were the first to experience the reforms' effects. We are completing our analyses this month and the full findings will be included in a written report later this year. Our study offers quasi-experimental evidence on the local effects of state reforms in four major city school districts located in four geographically dispersed states. Its results cannot be generalized beyond these sites. 1 THE SCHOOL DISTRICTS WE STUDIED AND OUR APPROACH The four school districts in our study vary in enrollment from 40,000 to 230,000. All are in urban areas. One is located in the northeast, two are in the southeast, and the fourth is in the southwest. Black and Hispanic students are in the majority in each district. The legislative mandates these districts implemented included varying combinations of -- requiring specific tests for graduation, -- increasing academic course requirements, -- tightening attendance rules, -- setting Ilno pass/no play" rules governing participation in sports or other activities, and -- requiring or making available varying degrees of remedial classes for those in difficulty. Other details about the four school districts we studied can be found in table 1. The basic data for answering the questions came from school districts' computerized student records. Our design called for achievement tests and other data on students as far back as 1982, so that we could analyze students' high school careers before and after reform. Altogether, we traced the progress of 61,500 students as they moved through secondary schools. I would like to express my appreciation here to the four districts for their cooperation in assembling the large data files we asked for. We also visited each district and interviewed central office officials 2 ‘Pdl,le I : Characteclstlcs ot selected Scnool ulstricts Year post-rrtornr School Number ot cohort MAJOR reforms attectiny city district students entered nace/etnn1c enrolllnent Dlstrlct Region students populatlo" enrollment studied grade Y wnite ___ Ulack - tiispanlc A SW Protlclency test YUU ,uuu 13u.uuu 17,7nn IY84-d5 lY# 50% ‘!ns More academic courses Stricter attendance rules "U" no longer passing Hemedlatlon tar low test- SCOrtZS "NO pdss/no play" rule I3 NE Protlclency test 38U ,uuu 2u,uuo L,btiU IYtli-tlb 1U b4 Lb Remedial rrelp reyulred w c SE aore credit In ,lldttI b Science 70,uuu 4u,uuo 44 '1 5 Hetnedldl nelp reyulred tar low test-scores "No pass/no play" rule Fewec vocdtl"nd, education reyuirements D SE tiusf pass proficlrncy test 350,uoo 230,rJOU 36,000 1983-84 Addltmndl science credit required Extra period added III 3rd year Remedial "elp authorized but not kunded as well as principals and teachers in high schools. Before presenting our findings, let me first address the question of whether and how educators in our four school districts attended to the needs of disadvantaged students during the period in which the reforms were initiated. HOW SCHOOLS TRIED TO HELP LOW-ACHIEVING STUDENTS One hypothesis about current education reforms is that, although clearly intended to raise educational achievement for all students, they could actually be harmful for disadvantaged, low- achieving or at-risk students by creating additional barriers to school completion without providing resources and assistance for them to meet the new standards. Did we see evidence that this was happening in our four school districts? We did not. In describing their implementation of their states' new requirements, educators in these cities told us of a wide variety of initiatives to help students meet the higher standards. These included, for example: -- offering an optional additional period in the day even when the state did not fund it, to help students take all the required courses: -- altering teaching methods, class sizes, and the content covered, to help students learn enough to pass required tests; -- increasing individual attention by counselors to students who may not meet increased requirements; 4 -- offering special remedial classes during the regular year, or in special periods before and after school or on weekends for students who fail the required high school exit examination; -- offering summer work-study for students who failed the exit exam during the year, providing remedial classes in the morning and a job in the afternoons; -- requiring teachers to develop special individual improvement plans for low-achieving students to target instruction on specific skills needed: -- reorganizing school to provide special self-contained programs and extra attention for students needing extensive help in all basic skills. We did not evaluate the adequacy, in terms of quantity or quality, of each district's specific education programs in relation to students' needs. However, it appeared to us that the districts were in general making serious efforts to be fair in helping all students meet the new requirements. The effect of these efforts is the question I turn to next. IMPACTS OF EDUCATION REFORMS ON STUDENT ACHIEVEMENT As I have already noted, some people argue that disadvantaged students--those who reach high school already achieving poorly or having been held back, who are limited in their English ability, or who are members of minority racial or ethnic groups--may find the increased requirements of education reform an insuperable barrier to graduation. Thus, one guess about the likely effects of the higher hurdles set by reform would involve student failure and exit. Achievement, while perhaps increasing for some students, might not rise among those who are at-risk. For these students, more vivid failure in classes and on required tests may be accompanied by restricted opportunities to participate in the athletic and extracurricular life of the school, which together with other frustrations, could lead to increased drop-out rates. On the other hand, many legislators and school officials believe that the reforms could have their intended effect. According to this hypothesis, the changed legal framework of increased requirements would be translated by teachers into higher expectations for all students, better identification of learning gaps, and increased provision of extra help, such as the initiatives taken in our four school districts, to help low- achieving students bridge those gaps. In this view, at-risk students would be even more likely to benefit from reform than more academically advantaged students. The key question we asked in comparing student achievement before and after the introduction of the reforms is whether there are benefits for low-achieving and minority students as well as for higher achievers and white students. Data from our four districts showed that, in general, low-achievers did not universally fall behind after the reforms, as had been feared. Indeed, we found some test score trends showing students gaining more from school 6 after reform than before, and this was true for both low- achieving and for higher-achieving students. However, these gains, overall, were very modest. As for instances of markedly worse drops in achievements through high school after reform, these most negative results actually happened more often for the higher- achieving group. Let me now turn to a more detailed analysis of these findings. Impacts on At-Risk or Low-Achieving Students We defined at-risk students in all four districts as those at the 34th percentile or lower on grade 8 reading achievement tests. We tracked the progress of these eighth graders through high school in terms of their performance on reading and mathematics tests. We also tracked similar results for ~higher-achieving--that is, not at- risk--students. We examined the performances of two groups of at- risk students: a group that completed high school before the reforms were in effect (the pre-reform group) and the first group to pass through school under the full reform requirements (the post-reform group). Thus, the principal reforms (such as increased course requirements and graduation exams) were in effect throughout the high school years for the post-reform groups, but did not affect the pre-reform groups at all. We have results, then, across their high school careers to compare two cohorts of at-risk students (pre-reform and post- 7 reform) in four cities on two achievement measures each--reading and mathematics--or a total of eight achievement comparisons. Our findings are negative on five of these eight outcomes, as shown at the bottom of figure 1. The most negative of these is in District B, where we see a decline throughout high school by both the pre-reform and post-reform groups in reading achievement. Further, the decline is slightly sharper for the post-reform group. The group starts grade 8 slightly worse off than the pre-reform group (at the 22nd percentile versus the 23rd) and drops to the 15th percentile, thus declining 7 percentile points or one more than their predecessors. How do we assess the other four cases of no net gain? In District A, it is true that the post-reform group improved its percentile standing in both reading and mathematics relative to the norms. In addition, the post-reform group has higher test scores by grade 11 than the pre-reform group. But the growth rate of the post-reform group throughout high school was lower than that of the pre-reform group. (This finding is reflected by the somewhat steeper lines shown in the graph for the pre-reform group in District A.) Moreover, the post-reform group had a higher initial grade 8 performance than the pre-reform group. These initial differences may result from changes in the community or in testing practices, or could represent stronger performance at the elementary grades. In any case, the post-reform group's initial advantage fades in high school, despite reform efforts. 8 Figure 1: Achievement of At-Risk Students: Comparisons of Pre-reform and Post-reform Groups from Grade 8 Through High School Testing Pattern of Net Gain for Post-Reform Students Relative to Pre-Reform Students a. Reversal from negative to positive trend District B - Math District C - Math District C - Reading Patterns of No Net Gain for Post-Reform Students Relative to Pre-Reform Students a. Positive trends for b. Negative trends for both cohorts; less both cohorts; slightly gain after reform greater decline after reform 46 j6111is;_lij District D - Math o _____ District D - Reading , ;:q_l District B - Reading 46 46 ,____----- Median ___---- Percentiles _r_---* ‘1 I o- District A - Math District A - Reading - Post-reform cohort ------ Pre-reform cohort 9 In addition to the District A results, we found two more examples of positive trends for both the pre- and post-reform groups but lower rates of gain after reform. In District D, the post-reform group started slightly ahead of the pre-reform group in both mathematics and reading, but lost those advantages and ended up only equal to, or slightly lower than, its predecessor by the final testing year. On the other hand, we do have findings of net gain on some of the eight outcomes. The upper part of figure 1 shows three clear examples of positive impacts of education reform for at-risk students. Note that in all three cases, the post-reform group median percentile scores increased throughout high school, while the pre-reform group had had declines. In short, in these three cases, the post-reform group o f at-risk students both improved their performance relative to the national norms and improved more than the pre-reform group (whi ch, in fact, had declined rather than improved). These three positive impacts were found in District C in both reading and mathematics and in District B in mathematics. These achievement results do suggest that at-risk students have not suffered "disasters" as a result of education reform in these four cities. But our f ind ings on the eight outcomes are both mixed and modest. 10 Impacts on Black and Hispanic Students We paid special attention in our study to the impacts of education reforms on minority students. I move now to the results for all black and Hispanic students, not just those with lower test scores. For blacks, the results are about the same as I have reported for students at-risk due to low reading achievement: 3 cases of overall net gain (see figure 2). The only difference is .for reading in District D. Those results showed a slight positive trend (though no net gain) for at-risk students (see figure 1) but are increasingly negative for blacks after reform. Figure 2 thus shows two such cases of increasingly negative trends for black students, compared to only one for low-achieving students (in figure 1). For Hispanics, the results were different. We had data for only six comparisons instead of eight due to the very small Hispanic student population in District C. The results showed no instances of net gains for the post-reEorm group and two cases of increasing negative trends. Figure 3 shows the reading decline for Hispanics in Districts B and D, which worsened after reform. Two other results--mathematics in Districts B and D--show the Hispanic students' performance relative to national norms declined somewhat during high school for both the pre-reform and post-reform groups. The results for reading and mathematics in District A 11
Impacts of Education Reform
Published by the Government Accountability Office on 1989-03-07.
Below is a raw (and likely hideous) rendition of the original report. (PDF)