UNITED STATES DEPARTMENT OF EDUCATION orFIn; OF INSPECTOR GENERAL MAY 13 2002 ED-OIOI A03,B0024 Dr. John Roth Principal Consultant Placement Programs Amelican College Testing, Inc, P,O. Box 168 Iowa City, IA 52243-0168 Dear Dr, Roth: This Final Audit Report (Control Number ED-OIO/A03-B0024) presents the results of our audit of American College Testing's AbilitY-lo-Benefit (ATB) policies and procedures for its Career Programs Assessment Test (CPAt). A draft of this reporl was provided to ACT, In its response, ACT concuned with our recommendations for Finding No.1 through Finding No,S, and noted that it has either already implemented or will be implementing actions to address the issues raised in these findings, ACT also appears to concur with the basis for Finding No, 6, but provided an alternate conective action plan for its resolution, We summarized ACT's response after each finding, and a copy of the complete response is provided as an attachment to this report. AUDIT RESULTS We found that American College Testing (ACT) was not in compliance with its agreement with the United States [)epartment of Education (ED), for the approved use of the CPAt, and with applicable laws and regulations. Under 34 c.F.R. § 668,145(e) The approval of a test may be withdrawn if the Secretary detem1incs that the publisher violated any terms of the agreement ... or that the information the publisher submitted as a basis for approval of the test was inaccurate, ' , , Based on the number and substance of our audit findings, if ACT does not implement all of this report's recommendations for findings 1 through 5, we recommend that ED withdraw its approval of the CPAt test for ATB testing purposes, We also found that the agreement and procedures for ACT's Assessment Test, which is also approved by ED for ATD testing purposes, do not meet Fedt'lal Jequirefllt'llls, 400 MARYLAND AVE., Sow. WASHINGTON, D,C. 20202-1510 Our mission is to ensure equal access to education and to promote educational excellence throughout the Nation. Finding No.1 ACT's Analysis of CPAt ATR Test S"nrps Was Not Submitted Timely We found that ACT did not submit its rC(juired three-year analysis o[CPAt ATE test scores to ED in a timely manner. ACT was required to submit the analysis to ED no latcr than October 1999. In November 2000, we were informed by an ACT official that ACT had not prepared an analysis of student CPAt ATE test scores and that it would begin to do so. ACT submitted its analysis to ED in January 2001. Under 34 C.F.R. § 668.150(b )(8) The agreement between a test publisher and the Secretary provides that the test publisher shall ... [tJhree years after the date the Secretary approves the test and for each subsequent three-year period, analyze the test scores of students to determine whether the test scores produce any irregular pattern that raises an inference that the tests were not being properly administered, and provide the Secretary with a copy of this analysis .... An ACT official stated that the report was not prepared timely because ED did not provide guidance for conducting the evaluation and reporting its results. Without timely analysis of CPAt ATR test scores, ACT cannot ensure that instances of improper test administrations are identified or that prompt corrective actions are taken. Consequently, a high volume of ATB testing abuses may occur that result in funds being awarded, under Title IV of the Higher Education Act of 1965, as amended (HEA), to ineligible students. Recommendation: 1. We recommend that the Chief Operating Officer (COO) for Federal Student Aid (FSA)! require ACT to strengthen its management controls to ensure that any future analysis of student CPAt ATB test scores required under 34 C.l'.R. § 608.150(b)(8) is prepared and submitted to ED in a timely manner. ACT's Reply ACT concurred with this recommendation. In its response, ACT asked about the date that it must submit the next analysis of student ATE test scores for the CPAt and about the peliod of testing dates that the next analysis should include. DIG's Response ED approved ACT'" CPAt test for ATB purposes on October 25, 1996. Consequently, under 34 C.ER. § 668.150(b)(8) I Student Financial Assistance (SFA) became Federal Student Aid (FSA) on March 6, 2002. 2 Audit of ACT's Career Program.::. A~~e~.::.ment Te~t ATR Propram ED-OIGIAm-B0024 • ACT's first analysis of students' CPAt ATB test scores should have covered the three year period ended October 24, 1999. • ACT's next analysis of students' CPAt ATE test .scores rllle to FD sholllrl rover the three year period from October 25, 1999, through October 24,2002. ACT should contact the appropriatc progrum officials at ED regarding the required submission date for this analysis. Finding No.2 ACT's Analysis of CPAt ATB Test Scures Did Nul Cuntain the True CPAt ATB Population ACT's Director ot StatIstIcal Research mformed us that ACT could not identify directly, based on its data, the universe of CPAt ATB test scores for its three-year analysis of CPAt ATB test results, covering the period January 1997 through December 1999. Instead, ACT identified the universe by identifying the universe of CPAt examinees who • Recei ved a valid test score for all three CPAt testing sections, and • Indicated on the CP At answer sheet that they had not graduated from high school. As a result, the universe used for ACT's three-year analysis improperly excluded CPAt ATB examinees who • Did not complete the educational background question on the CPAt answer sheet; • Indicated, on tbe educational background question, that they have at least a high school education; or • Did not receive a valid lest score on al1lhree testing sections. In addition, we found that the universe may have incorrectly included test results for non-ATB CPAt examinees with less than a high school education who received a valid test score on all three testing sections. Federal regulations applicable to this finding are in 34 C.P.R, § 668.150(b)(8), as cited above for Finding No.1. ACT's Director of Statistical Research stated that, at the time the universe for the three-year analysis was determined, the data needed to identify CPAt ATB examinees directly was not readily available. We concluded that, since there is not an appropriate universe, there is no assurance that ACT's three-year evaluation of CPAt ATB test records is reliable. 3 Audit of ACT's Career Progf::\ms Ass.es.sment Test ATB Program ED-OIG/A03-B0024 R""ommenrlation: 2. We recommend that the COO for FSA require ACT to initiate appropriate action to ensure that ACT bases any future analysis required under 34 C.F.R. § 668.150(b)(8) on a complete and accurate universe of test data. ACT's Reply ACT concurred with this recommendation. Finding No.3 ACT Does Not Monitor CPAt ATB Retest Administrations ACT does not have processes in place to identify and infOlID institutions of CPAt ATB administrations conducted in violation of its retesting procedures. Our review of ACT's file documentation for a sample of 25 examinees from the universe of 629 examinees who retested on the same CPAt form at the same institution in consecutive months during the period July 1, 1997, through June 30, 2000, revealed that 20 (80 percent) were retested improperly. Also, our analysis of the universe of 9,179 examinees who were administered a retest of the CPAt examination during the period july 1,1997, through June 30, 2000, identified approximately 400 instances in which examinees were improperly tested twice on the same CPAt form, during the same month, at the same institution 2 The CPAt Test Administration Manual allows an examinee to retest if his or her performance on the test was affected by conditions other than ability. For example, an examinee would be able to retest if his or her performance was affected by an illness or by an interrupted test session, or if there is reason to believe a significant change occurred in the examinee's level of knowledge ami skills. A<,;<.:on,lillg Lo Lhe CPA! Tes! AdminislrUfion Manual, if n:u;:sLing is warranted- An alternate form of the CPAt (a form other than the one most recently admll1Istered) IS used for the retest (I.e., retestmg must "cycle through" all CPAt fomls available from ACT). If all CPAt forms available from ACT have been administered to the examinee, a minimum of 30 days must pass between administrations of the same CPAt fonn. Federal regulations state • "The agreement between a test publisher and the Secretary provides that the test publisher shall ... [c]ertify test administrators who have ... [tJhe necessary trainin!?" knowledge, and skill to test students in accordance with the test publisher's testing requirements ...." 34 C.F.R. § 668.150(b)(2)(i). 2 Our review revealed 410 cases where this set of circumstances occurred. We did not report the exact number because the test date was not always accurate, as discussed in Finding No.4, and bt:<.:ause we did noL <,;onfirlll or sample the LeSL dales to the original records. Al!diJJ_l.LACT'~ rilreer Programs Assessment Test ATR PrOpf<lm FD-OfG/ A03-B0024 • 'The agreement between a test publisher and the Secretary provides that the test publisher shall ... [dlecertify a test administrator for a period that coincides with the period for which the publisher's test is approved if the test publisher finds that the test administrator ... [hlas repeatedly failed to give its test in accordance with the publisher's instructions ...." 34 C.P.R. § 668.150(b)(3)(i). "An institution may usc the results of an approved test to dctcnnine a student's eligibility to receive Title IV, HEA programs funds if the test was independently administered and properly administered." 34 C.F.R. § 668. [51 (a)(2). • "The Secretary considers that a test is properly administered if the test administrator ... [aldministers the test in accordance with instructions provided by the test publisher, and m a manner that ensures the mtegrity and security of the test ...." 34 C.F.R. § 668.1 S 1(d)(2). ACT's Manager of Production Services explained that ACT does not monitor CPAt ATB retesting activity because its certified independent test administrators (ITA's) are responsible for conducting retests in compliance with its rules. ACT's failure to identify and report retest administrations not conducted in accordance with its established procedures may result in invalid ATB determinations, improper admission of students, and disbursements of Title IV, HEA program funds to ineligible students. Also, because ACT does not monitor its certified ITA's retesting activity, ITA's who should be decertified may continue to perfonn CPA! ATB testing. Recommendations: We recommend that the COO for FSA require ACT to 3.1 Strengthen its management controls to ensure that ITA's follow the already estabhshed procedures for conducting a CPAt ATB retest. 3.2 Establish a system to monitor and review the testing practices of each certified ITA, and decertify any IT A who repeatedly violates or compromises its approved testing procedures. 3.3 Establish a system for identifying and reporting retest errors to ensure that institutions have accurate and timely information at the time that eligibility determinations are mmle ACT's Reply ACT concurred with the recommendations and stated that on May 1, 2002, it will activate a computerized process for applying retest rules to each CPAt ATB answer sheet it scores. ACT's reply described the process that ACT planned to implement, to identify and report retest errors. 5 Audit of ACT's Career Programs Assessment Test ATB Program OIG's Response ACT's planned actions must ensure it will establish a system for identifying and reporting violations of the retesting guidelines for CPAt ATB teslS described in Attachment A of its response to the Draft Audit Report. The CPAr ATB retest guideJines contained in Annchment A have not yet been reviewed and approved by ED. :Finding No.4 ACT's CPAt Database Contains Incomplete and Inaccurate Test Date Information The test date field maintained in ACT's CPAt database is a four digit numeric field that includes only the month and year data for test administrations. Our review of ACT's file documentation tor 1,15 ot 73,455 CPAt ATB tests conducted during the period July I, 1997, through June JO, 2000, revealed that the test date recorded on 9 CPAt ATB test answer sheets (9.5 percent) did not agree with the test date in the CP At database. In seven instances the wrong test month was recorded in the CPAt database, in one case the wrong test year was recorded, and on another occasion both the wrong test month and year were recorded. Data from CPAt test answer sheets that are scanned to file, for review and update to the CPAt database, includes the examinee's name, Social Security number, date of birth, race, educational hackgrOlmd, and item responses for the CPAt's Language Usage, Reading, and Numerical Skills Tests. Although each examinee records the full testing date on the CPAt (including the month, day, and year the test was administered), that full date is not recorded in a fonnat that can be scanned to file. ACT uses a "Site Header Sheet" to update the CPAt database with the general date (month and year unly) for each gruup ufCPAl ATB answer sheets received from an ITA. The "Site Header Sheet" is completed by ACT staff, and it is also used to collect the state code, test center code, number of answer sheets to be scored, and any applicablc special codes. Special codes are used to designate CPAt ATB tests, un-timed tests, and the lTA. The erroneous test date records in ACT's CPAt database for nine of the ATE tests we reviewed appear to be the result of human error on the part of ACT staff, when completing the "Site Header Sheets." Without complete (month, day, and year) and accurate leSI date records, ACT cannot adequately monitor CPAt ATE administrations for compliance with its retesting procedures. Recommendation: 4. We recommend that the COO for FSA require ACT to initiate appropriate action to ensure that complete and accurate CPAt ATB test dates arc maintained in its CPAt database. ACT's Reply ACT concUlTed with this recommendation. 6 AIH-iir of ArT'" rare~r Proerams Assessment T~st ATR Pmp:r:'lm ED-OIG/A03-B0024 Finding No.5 ACT Needs to Improve Procedures Used to Certify CPAt ITA's The process that ACT uses to certify CP At IT A's does not provide adequate assurance that the ITA's are not affiliated in any way with all of the institutions for which they may perform CPAt ATB testing. ACT's IT A certification process requires applicants to complete an "Ability-to Benefit Independent Test Administrator Certification Form." Applicants use the form to provide a list of names and addresses of the schools at which they plan to adminisler ATB tesls, and applicants sign an "ATB Test Administrator Agreement" on the form, attesting to the requirements of test administrator independence. The "ATB Test Administrator Agreement" docs not identify the institutions for which the ITA applicant is asserting independence. Also, ACT mforms each eertltied CPAtlTA, upon approval. that he or she "may administer CPAt tests for any institution currently registered with ACT as a CPAt participating institution." Federal regulations state • "An institution may use the results of an approved test to detelmine a student's eligibility to receive Title IV, REA programs funds if the test was independently administered and properly administered" 14 r.F.R. § 668.1 'Il(a)(2). • "The Secretary considers that a test is independently administered if the test is ... [g]iven by a test administrator who "(i) Ras no current or prior financial or ownership interest in the institution, its affiliates, or its parent corporation, other than the interest obtained through its agreement to administer the test, and has no controlling inten:st in allY uther educarional institution: ''OJ) Is not a current or former employee of or consultant to the institution, its affiliates, Of its parent corporation, a person in control of another msttlution, or a member of the family of any of these individuals: "(iii) Is not a current or former member of the board of directors, a current or former employee of or a consultant to a member of the board of directors, chief executive officer, chief fInancial officer of the institution or its parent corporation or at any other institution, or a member of the family of any of the above individuals: and "(iv) Is not a current or former student of the institution." 34 C.F.R. § 66S.1S1(b)(2). ACT's failure to ensure that its ITA's are independent from all institutions at which they can conduct CPAt ATB test administrations may result in invalid ATB determinations, improper admission of students, and disbursements of Title IV, HEA program funds to ineligible students. 7 !:A""u"ciilLt,"nfLA~C.LT-,'S"CC£"Wre",'e<!-f"-p"ro",g.!Jra!!Jrn""sw;A-,"s",~e",·~s""sm!.!.!!<Jenu.>I-,T>~e""st,-,AlTL1RLLP"ro",~1J.!ffi!.!1m-,-_ _ _ _ _ _ _ _-,F-".n~",-,O TGI A01- ROO?4 Recommendations: We recommend that the COO for FSA require ACT to 5.1 Strengthen its management controls to ensure that its certified ITA's are independent from all institutions at which they are approved for CP At ATB testing. 5.2 Estahlish a system to identify the institutions at which each certified ITA is approved for CPAt ATB testing. 5.3 Process and score CPAt ATB answer sheets for an institution only if the answer sheets are submitted by an IT A who is approved for testing at that institution. ACT's Reply ACT concurred with the recommendations. Finding No.6 ACT's ATB Administration Procedures for its Assessment Test Do Not Meet Federal Requirements The ACT Assessment tcst is a national college admi"ion examination designed to meaSlJre high school students' general educational development and their ability to complete college level work. It is administered on five national test dates each year and covers four content areas: English, mathematics, reading, and science reasoning. ED approved the ACT Assessment test for ATB testing purposes on October 27, 1998. ACT Revises the Assessment Test After Each Administration ACT officials informed us that. for security purposes, a new edition of the Assessment test is developed for each national test date. An applicatIOn for approval ot each new edition ot the test is not submitted to ED for approval. A test publisher that wishes to have its test approved by the Secretary for A TB purposes must submit an application to the Secretary. Under 34 C.F.R. § 66S.l44(c)(9) A test publisher shall include with its application ... [ilf a test has been revised from the most recent edition approved by the Secretary, an analysis of the revisions, including the reasons for the revisions, the impii"ntions of the. revisions for the comparability of scores on the current test to scores on the previous test, and data from validity studies of the test undertaken subsequent to the revisions Without this information, ED cannot determine whether each new edition of the Assessment test meets the ATB testing criteria in Subpart J of 34 c.P.R. Part 668. As a result, the new editions of the Assessment test are not approved by ED, and institutions may not use them to determine student eligibility. 8 A"rlit of ArT', rareer Programs Assessment Te.sLATIlJ'-I:Qg[.i1J"mllL_ _ _ _ _ _ _ _ _"'ED!Lo\-O.nIlJ.GJLIA"",032c-J2B..,OQ\L2",4 ACT Docs Not Maintain ATB Records for the Assessment Test The Assessment test is designed as an ;:Himl.~~inn~ ~r.rpp.nin8 tonI for trntiitlnn}ll P()~tsP,C()nOHry education at four-year institutions, and it is taken, primarily, by cunent high school students, The test is never administered specifically for ATB examinees. ACT officials told us that they do not identify Assessment tests that are administered for ATB purposes. As a result, ACT cannot determine whether any Assessment test data it maintains is related to ATB adminislraliuns. Therdun;, ACT willllul be ablt: lu prepan;; the lequired three year analysis of Assessment ATB test result data for indications of inegularities in the test administration process. This analysis is required under 34 C.F.R. § 66S.150(b)(S). On February 14, 2UU2. ACT submItted Its three-year analysis to FSA. Our review of the analysis revealed that its data is insufficient and that the submission does not meet the criteria in 34 C.F.R. § 668.150(b)(8). (The text of 34 C.F.R. § 668.150(b)(8) is provided above, for Finding No. I). ACT cannot identify instances of inegularities in the Assessment ATB test administration process. As a result, Assessment test administrators who should be decertified may continue to conduct tests, and FSA funds may be at risk. ACT's Assessment Test Record Retention Policy Needs Improvement According to the ACT Assessment User Handbook. "ACT keeps students' original [ACT Assessment] registration folders and answer documents for one year.... For research purposes, we retain student records on magnetic tape indefinitely." According to infolTl1ation for the Assessment test on ACT's website (www.act.org), and confilTl1ed by ACT's Assistant Vice President of Applied Research. examinecs may direct ACT to drop any or all of their Assessment records from ACT's files at any time. Under C.P.R. § 668.ISO(b)(7), approved ATB test publishers must- Keep for a period of three years each test answer sheet or electronic record forwarded for scoring and all other documents forwarded by the test administrator with regard to the test .... Since examinees may ask ACT to destroy their Assessment test records at any time. ACT cannot ensure that it has the complete universe of Assessment test results. Without a complete universe, ACT cannot prepare a reliable analysis of Assessment ATB test records. The Agreement between ED and ACT Does Not Meet Regulatory Criteria Under 34 C.F.R. § 668. 150(a). "If the Secretary approves a test under this subpart, the test publisher must enter into an agreement with the Secretary that contains the provisions set forth in paragraph (b) of this section before an institution may usc the test to detenninc a student's eligibility for Title IV, REA program funds." The agreement entered into by ED and ACT for 9 Allnit of ACT's C:ueerPrograms Assessment Test ATE ProGram ED-OlGt A03-B0024 the use of the Assessment test for ATB purposes does not meet all or the criteria required by 34 C.F.R. § 668.l50(b). • Under 34 CFR § 6f)R I ~()(h)(1), if an independent test administrator mmmits (Certain violations, he or she must be decertified by the test publisher "for a period that coincides with the period for which the publisher's test is approved ...." For the same violations. the agreement between ED and ACT stutes only thut ACT "will not ship test materials to [the] test administrator for a specified period .... Shipment of tests will resume only after corrections to procedures have been thoroughly documented and approved. " • Under 34 C.F.R. § 668.150(b)(6), the test publisher must "[p]romptly send to the student and the institution the student indicated he or she is attending or scheduled to attend a notice stating the student's score for the test and whet/leI' or not the student passed the test ...." The agreement between ED and ACT states only that ACT "will generate score reports for answer sheets that it receives from test administrators." • Under 34 C.F.R. § 668.150(b)(7), the test publisher must "[kJeep for a period of three years each test answer sheet or electronic record forwarded for scoring and all other documents forwarded by the test administrator with regard to the test ...." The agn.,ement hetween ED and ACT states only that ACT "will keep for a period of two years each test answer sheet or electronic record forwarded for scoring." • Under 3,1 C.F.R. § 668. 150(b)(8), a test publisher must, "[t]hree years after the datc the Secretary approves the test and for each subsequent three-year period, analyze the test scores of students to determine whether the test scores produce any irregular pattern that raises an inference that the tests were not being properly administered, and provide the Secretary with a copy of this analysis ...." The agreement between ED and ACT does not include this requirement. Since the agreement does not meet all ofthe required criteria in 34 C.F.R. § 668.150, it is not valid. Consequently, institutions may not use the ACT Assessment test to detennine a student's eligibility for Title IV REA funds. Recommendations: We recommend that the COO for FSA 6.1 Require ACT to use only approved versions of the Assessment lest for ATB purposes. 6.2 Reconsider ED's approval of the ACT Assessment test as an ATB test. 6.3 If ED's approval of the ACT Assessment test is not withdrawn, revise the agreement between ED and ACT to ensure that all criteria for test publishers in Subpart J of 34 C.F.R. Part 668 are met, including but not limited to the requirements in 34 C.F.R. § 668.150. 10 AllClit of ArT's rareer Programs Assessment Test ATB Program ACT's Reply ACT's response indicated that it concurs with the finding; however, ACT provided an alternate corrective action plan for the finding's resolution. ACT's response stated that its interest in having the Assessment test on ED's approved ATB test list was motivated by its potential service to "home school" students. To meet (he ATB re4uirefllcllts, ()ur previ()us understanding was lhallhese "HulIlc School" students would need to complete one of the approved ATB tests to qualify for federal financial aid; therefore, ACT submitted the ACT Assessment test tor ATB approval, to aVOId having "Home School" students who had completcd the ACT Assessment be required to test again on another approved test. ACT further stated that it recently learned that new options are available for home school students to qualify for Title IV, HEA funds, and ACT proposed that- Given this new information, we would like [to] review this new (to us) policy over the coming 30 days, with one of two possihilities likely to result: a. ACT would withdraw its request for ACT Assessment being included on the ATB list of approved tests (Home School students do not "need" the ACT Assessment as an ATB test) OR b. ACT would prepare information for DOE regarding why ACT believes the test should be retained on the ATB list of approved tests (Home School students or significant numbers of other students are well-served by having the ACT Assessment appear on the ATB list of approved tests). ACT asked if its proposal would be an acceptahle process with respect to determining whether to seek removal of its Assessment tcst from the Jist of ED-approved ATB tests. OIG's Response We reviewed ACT's comments but our recommendations remain unchanged. ACT's proposal and question should be addressed to the appropriate program officials at ED. OTHER MA TTERS Two other ACT tests, the Compass test and the Asset test, arc approved by ED for ATB testing purposes. ACT is required, under 34 C.ER. 668.150(b)(8), to submit three-year analyses to discover irregularities in the test administration process. ACT did not submit analyses for these 11 Audit nf ACT', Cnreer Pnwr3ms Assessment Test ATR Prnornm ED-OIG/A01-BOO?4 two tests in a timely manner. ACT was required to submit the analyses to ED no later than October 1999, but ED did not receive them until January 2001. (The text of 34 C.P.R. § 668.150(b)(8) is provided above, for Pinding No.1). BACKGROUND The Higher Education Technical Amendments of 1991 amended the lillA, requiring postsecondary students who do not have a high school diploma or its equivalent to pass an independently administered examination that has been approved by ED before receiving Title IV, HEA program funds. These examinations are intended to establish that studenls have the ability to benefit from postsecondary school training programs. This testing has become known as "Ability-to-Benefit (ATB) testing." On December 1, 1995, ED published final regulations, effective July 1, 1996, specifying the procedures and requirements for ATB testing which affect test publishers, schools, and ITA's, as Subpart J of 34 C.P.R. Part 668. Compliance with these regulations is mandatory in detennining the eligibility of applicants for Title IV, IIEA program funds. ED assesses tests submitted for ATB approval according to the requirements in Subpart J of 34 C.F.R. Part 668. ED approves a test for a period of no more than five years, although the approval ean he extended while a suhsequent review is conducted tn detennine fe-approval. A list of approved tests and passing scores are puhlished in the Federal Register. The initial list of approved tests was published in the Federal Register on October 25,1996. ACT's CPAt was approved by ED and appeared on the list published on October 25, 1996. ACT submitted an application for re-approval of the CPAt for ATB testing, and ED is currently conducting its review of the application. OBJECTIVE, SCOPE AND METHODOLOGY The objective of our audit was to detcnnine whether ACT properly administered its CPAt ATB program in accordance with its agreement with ED, for the approved use of the CPAt, and with applicable laws and regulations. To identify our ohjective, we reviewed • The agreements between ACT and ED for the use of ACT's CPAt, Compass, Asset, and Assessment tests for ATE testing purposes; • ACT's reports on CPAt, Compass, and Asset ATB testing. covering the period January 1, 1997. through December 31. 1999: and • ACT's CPAt, Compass, Asset, and Assessment test user manuals. To accomplish our objective • We worked with ACT officials and developed a process to identify the universe of CPAt ATB test administrations (73,455) conducted dUling the period July 1, 1997, through June 12 Audit of ACT's Career Programs Assessment Test ATcfBLP£'r""o$gr"'allm'--_ _ _ _ _ _ _ _ _~E"'D"'-"O'_'1GoE/~A",O,,3c':B"0",0,,,",24 ,0,2000. We analyzed a computer file, provided hy ACT, !h,,! contained the CPAt ATR test result data for this universe. We examined ACT's file documentation for 1, A random sample of 25 CPAt ATB test administrations selected from the universe of 73,455 administrations conducted during the period July 1, 1997, through June 30, 2000. 2. A random sample of 25 CPAt ATB applicants selected from the universe of 629 applicants who retested (54 administrations) on the same CPAt test form, at the same school, in consecutive months, during the period July 1, 1997, through June 30, 2000. 3. A random sample of 10 CPAt ATB test administrations selected from the universe of 1143 administrations conducted during the peliod July 1, 1997, through June 30, 2000, in which the applicant indicated on the CPAt answer sheet having obtained a 2 or 4 year college degree. 4. The universe of CPA! ATB test administrations (6) conducted during the pellod July 1. 1997, through June 30, 2000, for applicants with disabilities. • We analyzed a computer file, provided by ACT, covering the period July 1997 through June 2000, that contained data about the universe of certified (394), inactive (295), and decertified (5) CPA! ITA's. We examined the tIle documentation for 14 randomly selected certified ITA's and 11 randomly selected inactive IT A's. • We reviewed ACT's procedures for CPAt ATB test administration, scoring and reporting . We interviewed ACT personnel to obtain an understanding of these procedures. DUling our review at ACT, we also tested the reliability of computerized CPAt ATB data by comparing selected data records with the completed CPAt answer sheets. We concluded that the computerized informatIOn was sufficiently reliable for the purposes of our audit. We conducted our on-site fieldwork at ACT, in Iowa City, lA, from August 13, 2001, through August 17,2001. Our exit conference was held on August 17,2001. From August 2001 through Januaty 2002, we analyzed data that we received from the institution and at the Department. Our audit was performed in accordance with government auditing standards appropriate to the scope of the audit described above. STATF.MJ<~NT ON MANAGEMENT CONTROLS As part of our review, we assessed ACT's management control structure, as well as its policies, procedures, and practices applicahle to the scope of the audit. We assessed the level of control risk for determining the nature, extent, and timing of our substantive tests to accomplish the audit objective. 13 Audit of ACT's Career Proo-rams A"'~f'5;,~m~nt T~st ATR Prner:lm EO.OJG/ Am·R0024 For the purpose of this report, we assessed and classified the significant controls into the following categories: • ITA Training and Certification • CP At Administration • CPAt Reporting • CPA! Records l'v1anagcmcnt Because of inherent limitations, a study and evaluation made for the limited purpose described above would not necessarily disclose all material weaknesses in the management controls. However, our assessment disclosed management control weaknesses, which adversely affected ACT's ability to administer its CPAt ATB program. These weaknesses included inadequate reporting procedures, lack of monitoring procedures, deficient data collection and recordmg procedures, and inadequate IT A certification procedures. These weaknesses and their effects are fully discussed in the Audit Results section of this report, Audit of ACT'.s Career Program~ Assessment Test ATB Program ED-OIGI A03-BOO24 ADMINISTRATIVE MATTERS Slatements that managerial practices need improvements, as well as other conclusions and recommendations in this report represent the opinions of the Office of Inspector General. Determination of corrective action to be taken will be made by the appropriate Department of Education officials. If you have any additional comments or information that you believe may have a bearing on the resolution of this audit, you should send them directly to the following Department of Education official, who will consider them before taking final Department action on the audit: Mr. Greg Woods Chief Operating Officer Federal Student Aid Union Center Plaza Building, Rm. 112G 1 830 1sl Street, NE Washington, DC 20202 Office of Management and Budget Circular A-50 directs Federal agencies to expedite the resolution of audits by initiating timely action on the findings and recommendations contained therein~ Therefore, receipt of your comments within 30 days would be greatly appreciated. In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the Office of Inspector General arc available, if requested, to members of the press and general public to the extent infonnation contained therein is not subject to exemptions in the Act. We appreciate the cooperation given us in the review. Should you have any questions concerning this report please contact Bernard Tadley, Regional Inspector General for Audit, at 215-656-6279. Assistant Inspector General for Audit Attachment 15 Attachment - Auditee's Response ACT~ INFORMATION FOR LIFE'S TRANSITIONS April 16, 2002 Mr, Bernard Todley Regional Inspector General for Audit U,S, Department of Education Office of Inspector General The Wanamaker Building 100 Penn Square East. Suite 502 Philadelphia, PA 19107 Dear Mr, Tadley, On behalf of ACT, Inc., enclosed please find the document titled "ACT Responses to Draft Audit Report Regarding Ability-to-Benefit Policies and Procedures for Career Programs Assessment (CPAt)", This information is being provided in response to the Draft Audit Report (Control Number ED-OIG/A03-B0024), dated March 15, 2002. The information in the document includes responses to each of the various findings and recommendations identified in the Draft Audit Report, as well as several questions regarding your advice as to how ACT should proceed. We appreciate this opportunity to respond to the Draft Audit Report and look forward to continuing to serve students and institutions and to meet their individual and collective needs with respect to Ability-to-Benefit testing services. If additional questions should arise or additional information is needed, please do not hesitate to contact me regarding the CPAt, ASSET, COMPASS, ESL, or ACT Assessmp.nt Progr:::lm. Sincerely, John D. Roth, PllO ACT, Inc. Principal Consultant Placement Assessment Programs (ASSET, COMPASS, ESL, CPAt) Phone: 319/337-1030 FAX: 319/337-1790 Internet: firstname.lastname@example.org :no I No!-th Dodge Street po. Box 168 Iowa City, Iowa 52243-0168 319133/ 1000 wwwactorg ACT Responses to Draft Audit Report Regarding Ability-to-Benefit Policies and Procedures for Career Programs Assessment (CPAt) (Control Number ED-OIG/A03-B0024) April,2002 Finding No.1 ACT's Analysis of CPAt ATB Test Scores Was Not Submitted Timely Hecommendation: 1. We recommend that the Chief Operating Officer (COO) for FSA require ACT fa strengthen its management controls to ensure that any future analysis of student CPAt ATB test scores required under 34 C.F.R. § 668.150(b)(8) is prepared and submitted to ED in a timely manner. ACT Response: ACT will submit the next analysis of student ATB test scores on a schedule as specified by ED. Question: The analysis reviewed in the current audit was submitted by ACT in January 2001, covering student testing records for the period of January, 1997 through December 1999. What is the date for submission for the next analyses of student ATB test scores for CPAt and what period of testing dates should those analyses include? Finding No.2 ACT's Analysis of CPAt ATB Test Scores Did Not Contain the True CPAt ATB Population Recommendation: 2. We recommend lhal the COO lor FSA require ACT to iniliate appropriate action to ensure that ACT bases any future analysis required under 34 C.F.R. § 668.150(b)(8) on a complete and accurate universe of test data. ACT Response: Based en ATB indicators in ACT's batch tracking system, ACT has modified all CPAt records processed tor A I Cl purposes since July 1, 1997 so that the individual student records now carry an indicator that identifies the record as being scored for ATB purposes. All future CPAt scoring records will also contain an indicator as to whether the record was scored for ATB purposes" Finding No.3 ACT Does Not Monitor CPA! ATB Retest Administrations Recommendations: We recommend that the COO for I=SA require ACT to 1 3.1 Slren!llhen its mana!lement controls to ensure that ITA's follow the already established procedures for conducting a CPAt ATB retest. ACT Response: ACT will notify the institution, examinee, and ITA involved where CPAt ATB testing has violated ACT's retest policy. ACT will not report scores from ATB administrations in which the CPAt retest policy has been violated_ Institutions found to repeatedly violate ACT's retest policy or repeatedly fail to adhere to DOE or ACT policies and procedures will be disqualified as CPA! Participating institutions eligible to use CPAt. ITA's who repeatedly violate ACT's retest policy or repeatedly fail to adhere to DOE or ACT policies and procedures will be decertified through written notice to the ITA and to each institution for which the ITA is certified, ACT will not report scores from answer sheets submitted by a decertified ITA. ACT will process all CPAt ATB answer sheets submitted, even if scores are not reported, in order to retain a record of the test administration for future monitoring of the retest policy, ACT will revise sections of the CPAt Test Administration Manual to include clarification of the CPAt retest policy and its application by ACT as part of the ATB scoring and reporting process (to become effective 5/01/02), Additional procedures requiring institutions and ITA's to provide information to all examinees re~arding rele~t rul,,~ will also be added to the Manual. For an overview of the retest rules 10 be apfllied, see Attachment A: Retesting Conditions GuidelineslRequirements for CPAt ATB, 3.2 Establish a system to monitor and review the testing practices of each certified ITA, and decertify any IT A who repeatedly violates or compromises its approved testing procedures. ACT Response:ACT will monitor and review the testing practices of each certified Independent Test Admini"trator (ITA) through: 1) review of testing Irregularity Reports and other documentation submitted with answer sheets; 2) follow-up and documentation of comments and reports from examinees or school officials related to ITA performance; 3) review of management reports from the CPAt scoring system identifying records in violation of the CPAt retest policy; 4) issuance of written notice to ITA (copy to institution) of infractions and steps to be taken to correct procedures. Repeated infractions and notices will be documented in the ITA's file, Upon repeated infractions or repeated failure to adhere to DOE or ACT policies and procedures, an ITA will be decertified through written notice to the ITA and to each institution for which the ITA is certified, ACT maintains a file identifying decertified ITA's and will not report scores from answer sheets submitted by a decertified ITA, 3,3 Establish a system for identifying and reporting retest errors to ensure that institutions have accurate and timely information at the time that eligibility determinations aro made. ACT Response: On 5/01/02, ACT will activate a computerized process for applying retest rules to each CPAt ATB answer sheet scored by ACT, As part of this computerized routine, ACT can verify whether the appropriate retest requirements are met. The retest check will query the CPAt master history file and flag any record that marches by las! name or SSN and resr lorm wimln me past 9·1 days 01 me rest dare on me current answer sheet ACT staff will then determine whether or not the flagqed record meets the retest criteria. When a particular administration violates the CPAt retest policy, ACT will provide written notice to the Institution, to the examinee, and to the ITA that examinee scores are not being reported because of a retest violation. Periodically ACT will generate a report from the CPAt scoring and reporting system that will show the number of retest violations by institution and by ITA. ACT will contact institutions and ITA's that have continued to violate the retest policy after receiving notice of retest violations and review the policy to an~wer any qu"~liun~ afllj be ,;ur" II", in,;iiluliun '''IU til" ITA UnUJ;I,;taIlU II,,, f./uli"y. All CPAt Institutions and certified ITA's will receive a revised CPAI Test Administration Manual from ACT following th'" DOE's publication of the new passing score values in the Federal Registry Finding No.4 ACT's CPAt Database Contains Incomplete and Inaccurate Test Date Information Recommendation: 4. We recommend that the COO for FSA require ACT to initiate appropriate action to ensure that complete and accurate CPA! ATB test dates are maintained in its CPAt database. ACT Response: ACT will activate (as of 5/01/02) the insertion of a full test date ill each record anu a computerized process for applying retest rules to each CPA! ATB answer sheet scored by ACT. Finding No.5 ACT Needs to Improve Procedures Used to Certify CPAt ITA's Recommendations: We recommend that the COO for FSA require ACT to 5.1 Strengthen its management controls to ensure that its certified ITA's are independent from all institutions at which they are approved for CPAt ATB testing. ACT Response: ACT I,,,~ inlwuulO!;!u cuJuilio",,' .,1U"!;!dw!;!,, to ""rtify Ilt)w ITA's (Of s.,"'lOililO insliluliuns lu "nsurt; 1),,,1 lI'''' ITA is truly independent of each institution for which he/she is certified to administer CPAt tests for ATB purposes. In April 2002, ACT will initiate additional steps to collect and review credentials and test administration aGtivities of all previously certified ITA's to ensure that they are certified only for institutions from which they are independent as defined by the U.S. Department of Education. 5.2 Establish a system to identify thll! institutions at which each certified ITA is approved for CPAt ATB testing. ACT Response: ACT has expanded its ITA database to track specific institutions for which the ITA has been certified to administer ATB testing. 3 5.3 Process and score CPAt ATB answer sheets for an institution only if the answer sheets are submitted by an ITA who is approved for testing at that institution. ACT staff reviews the database of approved ITA's before CPAt ATB answer documents begin the scoring and reporting process. If the person listed on the CPAt ATB Answer Document ID Form is not on the approved ITA list for the specific institution. ACT notifies the institution, the examinees, and the ITA that no scores will be reported for that administration. Finding NO.6 ACT's ATB Administration Procedures for its Assessment Test 00 Not Meet Federal Requirements Recommendations: We recommend Ihat the COO for FSA 6.1 Require ACT 10 use only approved versions of the Assessment lest for ATB purposes. 6.2 Reconsider ED's approval of the ACT Assessment test as an ATB test. 6.3 If ED's approval of the ACT Assessmenllest is not wilhdrawn, revise the agreement between ED and ACT to ensure that all criteria for test publishers in Subpart J of 34 C.1=.R. Part 668 are met, including but not limited to the requirements in 34 C.F.R. § 668.150. ACT Response: ACT's interest in including the ACT Assessment on the ATB approved test list has been largely prompted by its potential service to "Home School" students (students who do not receive a standard High School diploma but wno often completed me ACT Assessment Instrument as part Of their application for COllege admission). To meet the ATB requiremento, our proviouG undomtanding wao thnt thone 'IHomc School" otudcntG would need to complete one of the approved ATB tests to qualify for federal financial aid; therefore, ACT submitted the ACT Assessment test for ATB approval, to avoid having "Home School" students who had completed the ACT Assessment be required to test again on another approved test. However, as part of the follow up to the Draft Audit Report, we have learned (earlier today) that July 1, 2001 revisions of the pertinent policies for ATB include new options for this for "Home SchOOl" students that would/could avoid unnecessary additional testing for these students. Given this new information, we would like review this new (to us) policy over the coming 30 days, with one ur twu fJu~~iLJiliti"~ likely tu re~ult. a. ACT would withdraw its request for ACT Assessment being included on the ATB list of approved tests (Home School students do not "need" the ACT Assessment as an ATB test) OR b. ACT would prepare information for DOE regarding why ACT believes the test should be retained on the ATB list of approved tests (Home School students or significant numbers of other students are well-served by having the ACT Assessment appear on the ATB list of approved tests). 4 Would this be an acceptable process with respect to determining whether ACT would seek to remove the ACT Assessment from the approvod tests for determining ATB eligibility? OTHER MATTERS Two other ACT tests, the COMPASS test and the ASSET test, are approved by ED for ATB testing purposes. ACT is required, under 34 C.F.R. 668.150(b)(8), to submit three-year analyses to discover irregularities in the test administration process. ACT did not submit analyses for these two tests in a timely manner. ACT was required to submit the analyses to ED no later than October 1999, but ED did not receive them until January 2001. (The text of 34 C.F.R. § 668.150(b)(8) is provided above, lor Finding No.1). ACT Response: ACT will submit the next analysis of student ATB test scores for ASSET and COMPASS on a schedule as specified by ED. Question: The analysis reviewed in the current audit was submitted by ACT in January 2001, covering student testing records for the period of January, 1997 through December 1999. What Is the dale tor submission for the next analyses of student ATB test scores for ASSET and COMPASS and what period of testing dates should those analyses include? 5 Attachment A: Retesting Conditions Guidelines/Requirements for CPAt ATB A. There is convincing evidence that the reason for retesting under either Situation 1 (interfering circumstances) or Situation 2 (additional learning has occurred) is true. B. If the retest occurs in less than 30 days from the previous testin(J, an alternate form (I.e., a form other than the one most recently administered) must be used for the retest. A student may be retested using the same form if a minimum of 30 days has passed since the previous administration of that form. However, no form may be administered more than twice to a given individual in a gO-day fJ"riuu. For ATB students tested with CPAt, the U.S. Department of Education requires that all three CPAt tests be administered during a retest, and prohibits a "mix and match" strategy. 6 REPORT DISTRIBUTION LIST CONTROL NO. ED-OIG/A03-B0024 No. of Auditee Copies Dr. John Roth I Principal Consultant. Placement Programs American College Testing. Inc. 2201 North Dodge Street P.O. Box 168 Iowa City, IA 52243-0168 ED Action Official Mr. Greg Woods Chief Operating Officer Federal Student Aid Other ED Officials/Staff (electronic copy) Audit Liaison Officer, Federal Student Aid I Assistant General Counsel, Office of the General Counsel Correspondence Control, Office of the General Counsel Chief of Staff, Office of the Secretary 1 Under Secretary. Officc of the Under Secretary 1 Deputy Secretary, Office of the Deputy Secretary 1 Director. Office of Public Affairs 1 Press Secretary. Office of Public Affairs 1 Assistant Secretary, Office of Legislation and Congressional Affairs 1 Chief Financial Officer, Office of the Chief Financial Officer 1 Director, Financial Improvement and Post Audit Operations, Otfice I of the Chief Financial Officer Post Audit Group Supervisor. Financial Improvemenl and Post I Audit Operations, Office of the Chief Financial Officer Assistant Secretary, Office of Intergovernmental and Interagency Affairs Assistant Secretary, Office of Postsecondary Education I General Manager for Schools Channel, Federal Student Aid 1 Area Case Director for Ca~e Management nne! ()v","<ight. F"e!eral Student Aid
American College Testing's Career Programs Assessment Test Ability-to-Benefit (ATB) Program.
Published by the Department of Education, Office of Inspector General on 2002-05-13.
Below is a raw (and likely hideous) rendition of the original report. (PDF)