United States General Accounting Office Washington, D.C. 20548 Health, Education, and Human Services Division B-282033 March 31, 1999 The Honorable Ernest J. Istook Chairman, Subcommittee on the District of Columbia Committee on Appropriations House of Representatives Subject: District of Columbia Public Schools: Audit Contis Reasonableness of Enrollment Count, but Reuort’s Presentation Is Unclear Dear Mr. Chairman: The District of Columbia Public Schools (DCPS) is one of the largest public school districts in the country-ranking in the top 40 of the nation’s nearly 15,000 school districts in terms of enrollment. A portion of its funding is federal, with about $95 million of its $645 rnilhon budget coming from federal funds in fiscal year 1999. Over the past decade, DCPS’enrollment count has been controversial, with critics charging that the count has been overstated. We reviewed DCPS’enrollment policies and procedures in 1995 and 1997 and found a lack of documentation for enrollment information as well as other problems.’ Concerns about the accuracy of DCPS’enrollment count led the Congress to include a requirement in the District of Columbia School Reform Act of 1995 for an independent audit of the count. The most recent audit, completed in May 1998, used a sampling methodology to conduct its verification, and you asked us to review the firm’s approach. Specifically, we asked (1) Was the audit methodology reasonable? and (2) Was the audit report’s presentation clear and complete? To answer your request, we reviewed the audit report and supporting documentation and discussed the audit approach and results with officials responsible for the audit. We also compared the auditors’ sampling approach to ‘District of Columbia: Weaknesses in Personnel Records and Public Schools’ Management Information and Controls (GAO/T-AND-95-170, June 14, 1995) and District of Columbia Public Schools: Student Enrollment Count Remains Vulnerable to Errors (GAO/HEHS-97-161, Aug. 21, 1997). GAO/HEHS-9946R District of Columbia Schools Enrollment Audit - B-282033 generally accepted sampling methods.2 Our statisticians reviewed the report and supporting documentation, and one of our statisticians discussed the analytic techniques with the auditors’ statistician. We also reviewed previously completed work related to the District of Columbia schools’enrollment procedures. We conducted our review between September 1998 and February 1999 in accordance with generally accepted government auditing standards. In summary, we found the audit methodology to be reasonable given the limitations of DCPS’data. Both the sampling approach and the analysis of the data involved analytical techniques that were carried out by using generally accepted statistical methods. In presenting the results, the audit report determined that DCPS’enrollment count had been fairly stated. It did not, however, fully present supporting detail explaining its conclusion. For example, the report did not clearly explain that the sample data actualIy yielded a wider range of estimates than it presented. Better presentation would have made it easier to fully understand the basis for the report’s conclusion. BACKGROUND In the past, DCPS’enrollment count was not a basis for determining its funding. However, the school district’s funding for f%scaIyear 1997 and subsequent years was directly linked to school enrollment under the District of Columbia School Reform Act of 1995.3 The appropriations law that provided fiscal year 1999 federal funding for DCPS underscores continuing congressional concern about DCPS’enrollment information.4 It prohibits the use of federal funds to pay the salaries of any DCPS school teacher, principal, administrator, official, or employee who knowingly provides false enrollment or attendance information. It also prohibits the use of federal funds to subsidize the education of nonresidents of the District of Columbia at DCPS elementary or secondary schools unless they pay tuition to the District of Columbia. Our 1997 review of DCPS’enrollment count process found that even though DCPS had changed parts of the process to address past criticisms, it remained 2Using Statistical Samuling (GAO/PEMD-10.1.6, May 1992) and Richard L. Scheaffer, William MenderhaIl, and Lyman Ott, Elementarv Survev Samuling, 4th ed. (Boston, Mass.: TWS-KENT Publishing Co., 1990). 3Although federal appropriations for DCPS are not directly tied to enrollment, annual payments to DCPS from the District of Columbia’s general fund are related to student enrollment. 4Public Law 105-277(1998). 2 GAOLHEHS-99-66R District of Columl$a Schools Enrollment Audit B-282033 flawed in several areas, including problems with the procedures used to correct errors in the school district’s automated database and problems with the procedures used to verify student residency. We also determined that DCPS and the District of Columbia Financial Responsibility and Management Assistance Authority were not in compliance with certain requirements of the School Reform Act of 1995. We recommended that DCPS comply with the reporting requirements of the act and that the Authority comply with the auditing requirements of the act. Since then, DCPS has implemented various new procedures and requirements intended to improve the accuracy of its enrollment count, and the Authority has contracted for an audit of the DCPS enrollment count. In March 1998, the Authority awarded a contract to the firm of Thompson, Cobb, Bazilio, and Associates to conduct an audit of the official enrollment count that DCPS had established for October 30, 1997. The firm was charged with providing an opinion as to the accuracy of certain information in DCPS’official enrollment count. After familiarizing themselves with how the count had been conducted, the auditors found that the records on which the count was based contained errors. For example, in some cases school officials had not made corrections to the records although school district officials had identified duplicate student names in the system and had requested that the records be updated. In other cases, proper transfer procedures had not been followed, allowing students to be counted by the system more than once. Because of the problems the auditors encountered with data quality, they were unable to verify the student count by certain categories (such as by grade) but were able to report a count total. To assess the accuracy of the enrollment count, the audit staff developed a methodology that was based on a sample of DCPS’records. They concluded that despite data and process deficiencies, DCPS had managed to conduct an enrollment count that was fairly stated.5 THE AUDITORS’ METHODOLOGY WAS REASONABLE GIVEN DATA LIMITATIONS The auditors’ use of a sampling approach for verifying the accuracy of DCPS’ enrollment count was reasonable, given the limitations they identified in DCPS’ automated database. The size of the auditors’ data sample and the sampling methods they used were also reasonable for producing data that had statistical reliability. To develop their sample, the auditors first collected membership lists 5Thompson, Cobb, Bazilio, and Associates, Audit of the Official Membershiuf the District of Columbia Public Schools as of October 30, 1997 (Washington, D.C.: May 1998). 3 GAOLHEHS-99-66R District of Columbia Schools Enrollment Audit B-282033 f&m all the school homerooms that had been used to record the October 30, 1997, student enrollment count6 Using these lists as a base, the auditors selected a random sample of 237 homerooms from the total 3,557 homerooms. Next, the auditors collected enrollment information for all the students in their sample homerooms. To collect this information, the auditors visited the sample homerooms and verified their lists against actual students. In the final count, the auditors included only students for whom they had collected adequate evidence of enrollment.7 If no acceptable alternative evidence was provided for the students in DCPS’count, they were considered errors and were not included. The auditors then compared their homeroom totals to the official DCPS count and noted the differences to prepare for their data analysis. The auditors used a ‘difference estimation” approach to conduct their data analysis and develop their estimates of the school district’s enrollment. This standard analytical approach allowed the auditors to estimate the count for the homeroom sample by comparing the differences between DCPS’October 30, 1997, enrollment count and the corrected count that they had determined from their on-site visits. Using the results from the 237 sampled homerooms, they estimated the differences for all homerooms to obtain a corrected total homeroom estimate.8 The auditors used a different approach to calculate the number of DCPS tuition grant students. These students are paid for by funds available to DCPS but are ‘The auditors noted, however, that there were no standard policies throughout DCPS regarding what constituted a homeroom. As a result, schools used different groupings to report their counts, such as a “homeroom” class (in some schools), first period class, or group advisory. In all cases, these groupings were referred to as homerooms. 7They followed up on students noted as absent, transferred in or out, or withdrawn to determine their enrollment status and reviewed enrollment dates for students present for their site visits but not listed on the homeroom lists. They also reviewed files to verify student residency information. ‘The auditors reported that DCPS’count included 187 students who were not residents of the district and had withdrawn from school since the official count on October 30, 1997. The auditors subtracted these students from their final estimates. A more direct comparison with DCPS’count would have left these students in, just as they had been included in the district’s count. However, the effect on the auditors’ final conclusion was minimal, as the DCPS count would still be withm the confidence interval for the estimates adjusted on this basis. 4 GAO/HEHS-99-66R District of Columbia Schools Enrollment Audit B-282033 not enrolled in DCPS schools and thus were not included in the homeroom count. To count these students, the auditors surveyed each of the 73 schools identified as having tuition grant students. Using the confirmation information received through their survey, the auditors calculated this population separately fkom the homeroom count. Table 1: DCPS’and Auditors Renorted Totals (I Source DCPS’reported count Homeroom counta 76,035 Tuition grant counlf 1,076 Total count 77,111 76,268 1,302 77,570 76,076 1,195 77,271 I aNote that the homeroom count was obtained by a “difference estimation” approach while the tuition grant count was obtained by collecting survey information. Like DCPS’total figure, the auditors’ total figures included the homeroom count and the tuition grant students. As table 1 shows, DCPS reported a total student count of 77,111,while the auditors reported-as a range-two count estimates of 77,271 and 77,570. Although slightly lower, DCPS’number was within 1 percent of the auditors’ estimates. The auditors reported two numbers for their homeroom count, each based on a different assumption about the data they were using. For audit estimate 1, the auditors included in their base calculation a group of students who they thought DCPS might have excluded in error from its enrollment count. For audit estimate 2, the auditors excluded this same group of students, just as had been done in DCPS’calculations. This second estimate, calculated from the same base number that DCPS used, made it easier for the auditors to identify differences between DCPS’count and their count because the base number was held constant. The auditors also reported two numbers for tuition grant students. The low number (1,195) was based on verifying the presence of these students through the auditors’ school surveys and adjusting the school district lists accordingly. The higher number (1,302) reflects the survey total plus the potential addition of 107 students who either were unconfirmed (because their survey information was not returned) or may have had their tuition paid for by another agency. 5 GAO/HEHS-99-66R District of Columbia Schools Enrollment Audit B-282033 T%IE REPORT MET THE MAIN AUDIT OBJECTIVE BUT DID NOT FULLY PRESENT SUPPORTING DETAIL The audit report answered the audit’s main objective of determining whether DCPS’enrollment count was fairly stated; however, the analysis information the auditors provided to support their conclusion was not as complete and clear as it could have been. For example, the auditors presented their final results in a manner that appears as though the homeroom count estimates are the endpoints of a single range, stating, “we estimate the total DCPS students enrolled and attending at October 30, 1997, to be in the range of 76,07676,268.”With this presentation, readers could be led to believe that the projected number of DCPS students fell between these two point estimates. Instead, the number could have fallen outside the point estimates. Our follow-up with a member of the audit firm showed that these two estimates were based on the two different assumptions discussed earlier and that each one has its own confidence interval establishing a likely range for the true student enrollment. Figure 1 shows the difference between what readers might infer from the report’s presentation and what the statistical projections actually were. Under estimate 1, the homeroom count for DCPS is identified as 76,268with a precision of plus or minus 614 students. Under estimate 2, the homeroom count is identified as 76,076 with a precision of plus or minus 131.’ Figure 1 shows that DCPS’count of 76,035 is within the range of either estimate and that the school district’s and auditors’ counts are within about 1 percent of each other. Confidence intervals or other measures identifying the precision of the estimates were also missing &om the written report. Without this type of information, a reader is left unaware of the true range of the estimates. ‘The range under both assumptions reflects a 95percent cotidence interval. A confidence interval is a range of values that is expected (with a reasonable degree of confidence) to include the true value. For these two estimates, it may be interpreted that there is 95percent confidence that the true value is between 75,654 and 76,882 for estimate 1 and between 75,945 and 76,207for estimate 2. 6 GAO/HEHS-99-66R District of Columbia Schools Enrollment Audit B-282033 Fi&re 1: ReDort Presentation Compared With Auditors’ Actual Results What Readers Might Infer About the Estimates, Given the Report’s Presentation 76,5Cil-- Estimating Approach I /. ) 76,268 (High Estimate) Estimating : ::, .y + ‘. ‘1, District of Columbia Approach 2 * 76,076 (Low Estimate)_- 76,035 -- Public Schools Official Enrollment Count - 75,500 -- What the Estimates Actually Showed 76,882 (High Estimate) : 76,268 (Point Estimate) District of Columbia 76,035 4- Public Schools Official Enrollment Count In addition, if the auditors had more clearly linked the information in the report body with the information in the report appendix, it would have been easier to understand the report results. For example, the auditors did not provide a direct link between their analysis in the appendix and their subtraction of the 187 7 GAOIHEHS-99-66R District of Columbia Schools Enrollment Audit B-282033 students who were not district residents and had withdrawn from school since DCPS’October 30, 1997, count. Although the calculation related to these 187 students was noted in the report body, it is not apparent in the appendix. As a result, in our opinion, readers are left with the difficulty of interpreting results and making calculations to link the results in the body to the results in the appendix. It was not until we posed questions to one of the auditors that we were able to link the appendix numbers with the final results identified in the report body. lo AGENCY COMMENTS We provided a draft of this report to the Authority and DCPS for comment. Both the Authority and DCPS concurred with our finding that the audit methodology was reasonable and that the audit firm’s report presentation could have been more clear. The Authority commented that itwill request the current auditor of the DCPS enrollment count to include its analysis in the body of its audit report. We also discussed the contents of the draft with representatives of the audit firm and they concurred with our factual representation of the audit and generally supported the need for improvements in future audit reports. We are sending copies of this correspondence to appropriate congressional committees, the District of Columbia Financial Responsibility and Management Assistance Authority, and the District of Columbia Public Schools. We will make it available to others who request a copy. If you or your staff have any questions or wish to discuss this material further, please call me or Harriet Ganson, Assistant Director, at (202) 512-7014. Major contributors included Dawn Hoff, Wayne Dow, and Stan Stenersen. Sincerely yoxs, Marnie S. Shaul Associate Director, Education, Workforce, and Income Security Issues (104954) “We also identified other smaller presentation errors, where data had been incorrectly calculated or stated. However, none of these errors affected the final conclusion of the audit report. 8 GAO/HEHS-99-66R District of Columbia Schools Enrollment Audit Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the foliowing address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and Mastercard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please ca.B (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States General Accounting Office Washington, D.C. 20548-0001 I Permit No. GlOO I Official Business Penalty for Private Use $300 Address Correction Reauested
District of Columbia Public Schools: Audit Confirms Reasonableness of Enrollment Count, but Report's Presentation Is Unclear
Published by the Government Accountability Office on 1999-03-31.
Below is a raw (and likely hideous) rendition of the original report. (PDF)