UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF INSPECTOR GENERAL Information Technology Audits and Computer Crime Investigations FINAL MANAGEMENT INFORMATION REPORT DATE: 01/25/2010 TO: William J. Taggart Chief Operating Officer Federal Student Aid Daniel T. Madzelan Delegated the Authority to Perform the Functions and Duties of the Assistant Secretary for the Office of Postsecondary Education FROM: Charles E. Coe, Jr. /s/ Assistant Inspector General Information Technology Audits and Computer Crime Investigations Office of Inspector General SUBJECT: Weaknesses in the regulations and guidelines for Department of Education approved publishers of the Ability-to-Benefit Test (X11J0002) The purpose of this Final Management Information Report (MIR) is to provide the Office of Federal Student Aid (FSA) and the Office of Postsecondary Education (OPE) with information that might be beneficial in improving the Department of Education (ED) regulations and guidelines for ED-approved publishers of the Ability-to-Benefit (ATB) test. In FY 2009, FSA disbursed approximately $113 billion in Title IV funds. Of these funds, up to $12.8 billion (11.3%) were disbursed to students who qualified for Title IV aid on the basis of an ATB exam.1 A data analytics project conducted by the Office of Inspector General (OIG) has identified potential vulnerabilities in ED's regulations concerning the test score analysis submitted by ATB publishers every three years as part of their agreement with the Secretary. This project examined Independent Test Administrators (ITAs) who were decertified by one specific publisher after its triennial test score analysis revealed potentially compromised or invalid ATB examinations. Of the 106 ITAs decertified by the publisher, OIG identified 83 decertified ITAs who provided tests for approximately 5,619 students who received an estimated $51.4 million in disbursed Title IV funds at 133 post-secondary institutions.2 1 This figure represents the amount of Title IV aid disbursed to students who filed the Free Application for Federal Student Aid (FAFSA) and did not indicate they had a high school diploma, General Education Development (GED) credential, or were homeschooled. 2 This figure represents the total disbursements of Title IV funds to these 5,619 students. In order to determine the actual Title IV loss, additional investigative steps will be required. 550 12th St SW, Suite 8000 Washington, DC 20202 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational excellence and ensuring equal access. Page 2 Final Management Information Report Ability-to-Benefit Test BACKGROUND Under ED’s regulations, an applicant who has not earned a high school diploma or its recognized equivalent may qualify for Title IV funds by passing an independently administered test published by an ED-approved ATB publisher.3 An institution may use an approved ATB test to determine whether a student who has not earned a high school diploma or its recognized equivalent may be eligible to receive any Title IV funds. All institutions providing ATB testing (except those that qualify as an assessment center)4 must use the services of an ITA, who has been certified by the appropriate test publisher.5 In order to qualify a student to receive Title IV funds, the ATB test must be administered by an ITA who is not affiliated in any way with the school or its staff.6 ED regulations require the publishers of ATB examinations to decertify any ITA who violates or compromises approved ATB testing procedures.7 The publishers of the ATB examinations are required under 34 C.F.R. § 668.150(b)(8) to conduct an analysis of their ITAs test scores every three years and provide their findings to the Secretary. Previous Office of Inspector General Activities In 2002, OIG conducted audits of two approved ATB publishers: Wonderlic Inc. (Wonderlic) and American College Testing (ACT).8 During this time, OIG also conducted an audit to evaluate FSA’s methods for monitoring approved ATB test publishers from July 1, 1997, through June 30, 2000.9 These audits found: 1. FSA did not have an effective monitoring system in place to ensure that ED-approved ATB test publishers comply with applicable laws and regulations, 2. FSA’s procedures did not ensure that ATB test publishers comply with their agreements with the Secretary, and 3. Seven ITAs who were decertified from Wonderlic for suspicious testing activity subsequently received certification through ACT to administer ATB tests. In March 2006, Wonderlic submitted a report, “A Review of Ability-to-Benefit Test Administrations,” to FSA in compliance with 34 C.F.R. § 668.150(b)(8).10 The report covered the test administrations of Wonderlic's ITAs conducted between October 26, 2002, and October 3 34 C.F.R. § 668.32(e)(2) (2008). 4 34 C.F.R. § 668.142 (2008). 5 34 C.F.R. § 668.151 (2008). 6 34 C.F.R. § 668.151(b)(2) (2008). 7 34 C.F.R. § 668.150(b)(3) (2008). 8 "Wonderlic's Ability to Benefit Program", (A03-B0022, February 4, 2002) and "American College Testing's Career Programs Assessment Test (CPat) Ability-to-Benefit (ATB) Program" (A03-B0024, May 13, 2002). 9 "Audit of Procedures at Federal Student Aid for Monitoring the Ability-to-Benefit (ATB) Test Publishers Approved by the U.S. Department of Education(ED): American College Testing (ACT), The College Board, CTB/McGraw-Hill, and Wonderlic", (A03-B0001, August 8, 2002). 10 This report was prepared by a third-party company, Data Research Services Inc., and included an audit review of Wonderlic's test administrations using the A*Star® method of analysis. Page 3 Final Management Information Report Ability-to-Benefit Test 25, 2005. The report included an audit review of the ATB test administrations that used a method of analysis contrasting the patterns of test answers provided by the ITAs to normative patterns established for the specific test form. As a result of the contrast, it was determined that there was increased frequency of improper influence among certain ITAs, indicating that they administered potentially compromised or invalid ATB examinations. On the basis of this review, Wonderlic decertified 106 of its ITAs. These 106 ITAs tested approximately 39,300 students in the time period covered by the review. In March 2007, Wonderlic provided OIG with testing data for the 106 ITAs. From the group of 106 decertified ITAs, OIG identified 83 ITAs who provided potentially compromised or invalid ATB examinations for approximately 5,619 students at 133 Title IV post-secondary institutions.11 These 5,619 students received an estimated $51.4 million in disbursed Title IV funds between January 1, 2005, and January 31, 2009.12 At this time, OIG has not yet determined if the individual tests administered to these students were compromised or invalid. However, as discussed above, these students were tested during the period of Wonderlic's report and audit review. Based on the analysis of the data from Wonderlic, OIG identified several instances of potential criminal activity that we are considering for further investigative activity. Government Accountability Office Activities In August 2009 the Government Accountability Office (GAO) released the report, "Proprietary Schools: Stronger Department of Education Oversight Needed to Help Ensure Only Eligible Students Receive Federal Student Aid," which included recommendations to improve FSA's administration of the ATB program. At the start of GAO's audit, OIG met with the GAO auditors and provided them with information found during our analysis of the data; this is discussed in GAO's report. GAO's report included the following recommendations: Conduct regular follow-up of ATB test analyses submissions to ensure Federally approved test publishers provide complete submissions as required; Use data provided by test publishers on schools where test administrators improperly administered tests and were later decertified to target schools for further review; Require test publishers to conduct an interim or mid-point analysis—a supplement to the 3-year test score analysis and submission requirement—to provide a preliminary review of potential testing problems, and submit a copy of their results to the Secretary; or require test publishers to have a process to follow-up on identified test score irregularities, take action to decertify test administrators if test irregularities suggest improper test administration, report actions taken as a result of test score analysis to the 11 The OIG analyzed testing data for all 106 of the decertified ITAs. For the purposes of this project, ITAs who tested a relatively small number of students (in most cases fewer than 50 students) between October 26, 2002 and October 25, 2005 were excluded from the data. Students that were administered the ATB test and received disbursements of Title IV aid prior to January 1, 2005 were excluded in order to comply with the statute of limitations for criminal charges. 12 This figure represents the total disbursements of Title IV funds to these 5,619 students. In order to determine the actual Title IV loss and specific instances of fraudulent activity, additional investigative steps will be required. Page 4 Final Management Information Report Ability-to-Benefit Test Secretary and prohibit test publishers from using ATB test administrators who have been decertified by any test publisher IDENTIFIED PROGRAM DEFICIENCIES OIG identified several vulnerabilities in the regulations and guidelines regarding the approved publishers of the ATB test. Requirements for the Triennial Test Score Analysis To become an approved publisher of the ATB test, publishers must submit an application to ED and undergo a rigorous review process. As part of this, ED contracts with experts in the fields of educational testing and psychometrics to review each publisher’s ATB test and testing methods.13 Once publishers successfully complete the application process, the Secretary may approve the test for a period of up to five years. At least six months before the end of their approval period, publishers must resubmit their application for another five year approval period and repeat the review process.14 Publishers are required to maintain test records for at least three years. Three years from the date their test was approved and for each subsequent three-year period, the publishers must analyze the test scores to determine if any testing anomalies exist and provide a copy of their analysis to the Secretary.15 This triennial test score analysis is separate from the information submitted during the publisher's initial application or any subsequent applications for approval, and it is not subject to the same internal review process by FSA. Unlike the requirements for initial approval, the regulations provide limited instructions for the triennial analysis and require only that the publisher "analyze the test scores of students to determine whether the test scores produce any irregular pattern that raises an inference that the tests were not being properly administered, and provide the Secretary with a copy of this analysis."16 This lack of detail has resulted in significant variations among the test-score analyses submitted by the individual publishers. OIG's review of the submissions and conversations with FSA officials indicated that some publishers of the ATB test merely submit a list of numbers with little or no analysis, while others submit reports prepared by third-party reviewers and are several hundred pages in length.17 The regulations also do not specify when publishers must submit their triennial test. Conversations with FSA revealed several publishers do not submit their analysis in a timely fashion. To date, Wonderlic appears to be the only publisher who uses the test score analysis as a basis for identifying and decertifying ITAs that exhibit suspicious testing patterns. 13 34 C.F.R. § 668.145(a)(1) (2008). 14 34 C.F.R. § 668.145(d) (2008). 15 34 C.F.R. § 668.150(b)(7&8) (2008). 16 34 C.F.R. § 668.150(b)(8) (2008). 17 As previously discussed, the report submitted by Wonderlic in 2006 was prepared by a third-party company, Data Research Services Inc., and included an audit review of Wonderlic's test administrations using the A*Star® method of analysis. Page 5 Final Management Information Report Ability-to-Benefit Test FSA’s Internal Review Process The current regulations do not stipulate a process for review or evaluation of the triennial test analysis submitted by the publishers, and there has been little formal oversight or review of the test score analysis. Unlike data submitted during the five-year reapplication, data from the triennial test analysis are not required to be reviewed by experts in educational testing and psychometrics. During a meeting with FSA officials, FSA confirmed it did not have a formal internal process for reviewing or analyzing the publishers’ triennial test score analysis or for ensuring that the test score analysis submitted by the publishers complied with applicable laws and regulations. This is consistent with the findings in the 2002 OIG audit discussed above. FSA also confirmed it has not provided any of the publishers with feedback on the results of their analysis. FSA is aware of these issues and is working to address them. However, without clarification of the regulations, it may be difficult to establish long-term policies and procedures for the submission and review of the publisher’s analysis. Misaligned Reporting Periods The time periods for the five year reapplication for approval and the triennial test analysis are misaligned. Under the current system of regulations, FSA does not receive a report of the publisher’s test analysis for the final two years of an approval period until at least a full year after a new approval period has begun. Because the requirements for the five-year reapplication do not require publishers to include the data from their test-score analysis and publishers must apply to renew their approval six months prior to the end of their existing approval period, FSA may not be notified of any current anomalies or problems in the test analysis that may affect its decision to renew a publisher's approval period until after the renewal process is completed. Decertified ITAs As stated in 34 C.F.R. § 668.150, institutions must use a certified ITA to administer the ATB test. Publishers of the ATB test are responsible for certifying ITAs to administer their test and for decertifying any ITA who violates or compromises approved ATB testing procedures. However, the ATB publishers are not required to notify an institution when an ITA is decertified. They do not provide FSA or the institutions with a list of decertified ITAs nor do they share information on their decertified ITAs with other publishers. The 2002 OIG audits of Wonderlic and ACT found that ACT was employing seven ITAs that were previously decertified by Wonderlic for demonstrating testing patterns that differed significantly from the norm. It is possible for an ITA decertified by one publisher for testing fraud to become an ITA for another publisher or at another institution without the knowledge of either entity. SUGGESTIONS In order to specify the publishers' responsibilities under the regulations outlined in 34 C.F.R. § 668.141-156 and to ensure that adequate internal controls and guidelines exist, the following actions should take place: Page 6 Final Management Information Report Ability-to-Benefit Test 1. Revise the current regulations to specify the requirements publishers must follow in completing and submitting a triennial test score analysis, including establishing standards and a timeline for providing the Secretary with a copy of the completed analysis; 2. Revise the timelines for submitting the triennial test score analysis and the requirements for the five-year renewal application to make the reporting periods consistent and to ensure that FSA has results from the most current test analysis when considering the publisher's renewal application; 3. Revise the regulations to require publishers to immediately report to FSA and the institution, or both, when an ITA is decertified and have FSA disseminate this information as necessary; 4. Establish a formal and timely internal review process within FSA for collecting and reviewing the publisher’s triennial test score analysis, similar to the review process for the five-year renewal applications. DEPARTMENT RESPONSE A draft of this report was provided to FSA and OPE on November 19, 2009. After receiving the draft, FSA and OPE provided OIG with a formal, written response on January 11, 2010. (Attachment 1) In this response, the Assistant Secretary for OPE and the Chief Operating Officer for FSA stated that OIG suggestions numbered 1-3 are under consideration at the negotiated rulemaking that began in November 2009 and is scheduled for conclusion in January 2010. Until such time as the regulatory process is complete, they will be unable to provide specifics regarding the regulatory changes. When details are confirmed, a Corrective Action Plan can be developed. With regard to OIG suggestion 4, FSA and OPE responded that the Department now has systems in place to monitor and track the triennial test score analysis submitted by test publishers and is currently engaged in contracting for the services of independent psychometritions who will review the test publisher's submissions. In addition, the Department has begun planning for changes to its school reporting systems that will support student-specific ATB reporting to help the Department focus on institutions with a high number of ATB-eligible students. In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the OIG are available to members of the press and general public to the extent information contained therein is not subject to exemptions in the Act. If you have any questions, please contact Mark Smith, Special Agent in Charge for the Technology Crimes Division, at (202) 245-7019. cc: Martha J. Kanter, Under Secretary, Office of the Under Secretary
X11J0002 - Weaknesses in the regulations and guidelines for Department of Education approved publishers of the Ability-to-Benefit Test - Date Issued: 01/25/2010 PDF (950K)
Published by the Department of Education, Office of Inspector General on 2010-01-25.
Below is a raw (and likely hideous) rendition of the original report. (PDF)