oversight

American College Testing's Career Programs Assessment Test Ability-to-Benefit (ATB) Program.

Published by the Department of Education, Office of Inspector General on 2002-05-13.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                             UNITED STATES DEPARTMENT OF EDUCATION

                                               orFIn; OF INSPECTOR GENERAL




                                                         MAY 13 2002                                         ED-OIOI A03,B0024

Dr. John Roth
Principal Consultant Placement Programs
Amelican College Testing, Inc,
P,O. Box 168
Iowa City, IA 52243-0168

Dear Dr, Roth:

This Final Audit Report (Control Number ED-OIO/A03-B0024) presents the results of our audit
of American College Testing's AbilitY-lo-Benefit (ATB) policies and procedures for its Career
Programs Assessment Test (CPAt).

A draft of this reporl was provided to ACT, In its response, ACT concuned with our
recommendations for Finding No.1 through Finding No,S, and noted that it has either already
implemented or will be implementing actions to address the issues raised in these findings, ACT
also appears to concur with the basis for Finding No, 6, but provided an alternate conective
action plan for its resolution, We summarized ACT's response after each finding, and a copy of
the complete response is provided as an attachment to this report.

                                                 AUDIT RESULTS

We found that American College Testing (ACT) was not in compliance with its agreement with
the United States [)epartment of Education (ED), for the approved use of the CPAt, and with
applicable laws and regulations.

Under 34 c.F.R. § 668,145(e)­

       The approval of a test may be withdrawn if the Secretary detem1incs that the
       publisher violated any terms of the agreement ... or that the information the
       publisher submitted as a basis for approval of the test was inaccurate, ' , ,

Based on the number and substance of our audit findings, if ACT does not implement all of this
report's recommendations for findings 1 through 5, we recommend that ED withdraw its
approval of the CPAt test for ATB testing purposes,

We also found that the agreement and procedures for ACT's Assessment Test, which is also
approved by ED for ATD testing purposes, do not meet Fedt'lal Jequirefllt'llls,




                                    400 MARYLAND AVE., Sow. WASHINGTON, D,C. 20202-1510 


            Our mission is to ensure equal access to education and   to   promote educational excellence throughout the Nation. 

Finding No.1            ACT's Analysis of CPAt ATR Test S"nrps Was Not Submitted Timely

We found that ACT did not submit its rC(juired three-year analysis o[CPAt ATE test scores to
ED in a timely manner. ACT was required to submit the analysis to ED no latcr than October
1999. In November 2000, we were informed by an ACT official that ACT had not prepared an
analysis of student CPAt ATE test scores and that it would begin to do so. ACT submitted its
analysis to ED in January 2001.

Under 34 C.F.R. § 668.150(b )(8) ­

         The agreement between a test publisher and the Secretary provides that the test
         publisher shall ... [tJhree years after the date the Secretary approves the test and
         for each subsequent three-year period, analyze the test scores of students to
         determine whether the test scores produce any irregular pattern that raises an
         inference that the tests were not being properly administered, and provide the
         Secretary with a copy of this analysis ....

An ACT official stated that the report was not prepared timely because ED did not provide
guidance for conducting the evaluation and reporting its results.

Without timely analysis of CPAt ATR test scores, ACT cannot ensure that instances of improper
test administrations are identified or that prompt corrective actions are taken. Consequently, a
high volume of ATB testing abuses may occur that result in funds being awarded, under Title IV
of the Higher Education Act of 1965, as amended (HEA), to ineligible students.

Recommendation:

      1. 	 We recommend that the Chief Operating Officer (COO) for Federal Student Aid (FSA)!
           require ACT to strengthen its management controls to ensure that any future analysis of
           student CPAt ATB test scores required under 34 C.l'.R. § 608.150(b)(8) is prepared and
           submitted to ED in a timely manner.

ACT's Reply

ACT concurred with this recommendation. In its response, ACT asked about the date that it
must submit the next analysis of student ATE test scores for the CPAt and about the peliod of
testing dates that the next analysis should include.

DIG's Response

ED approved ACT'" CPAt test for ATB purposes on October 25, 1996. Consequently, under 34
C.ER. § 668.150(b)(8)




I   Student Financial Assistance (SFA) became Federal Student Aid (FSA) on March 6, 2002.


                                                   2

Audit of ACT's Career Program.::.   A~~e~.::.ment Te~t   ATR Propram 	              ED-OIGIAm-B0024


•	        ACT's first analysis of students' CPAt ATB test scores should have covered the three­
          year period ended October 24, 1999.

•	        ACT's next analysis of students' CPAt ATE test .scores rllle to FD sholllrl rover the three­
          year period from October 25, 1999, through October 24,2002.

ACT should contact the appropriatc progrum officials at ED regarding the required submission
date for this analysis.

Finding No.2 	             ACT's Analysis of CPAt ATB Test Scures Did Nul Cuntain the True
                           CPAt ATB Population

ACT's Director ot StatIstIcal Research mformed us that ACT could not identify directly, based
on its data, the universe of CPAt ATB test scores for its three-year analysis of CPAt ATB test
results, covering the period January 1997 through December 1999. Instead, ACT identified the
universe by identifying the universe of CPAt examinees who­

     • 	 Recei ved a valid test score for all three CPAt testing sections, and

     • 	 Indicated on the CP At answer sheet that they had not graduated from high school.

As a result, the universe used for ACT's three-year analysis improperly excluded CPAt ATB
examinees who ­

     •	    Did not complete the educational background question on the CPAt answer sheet;

     • 	 Indicated, on tbe educational background question, that they have at least a high school
           education; or

     • 	 Did not receive a valid lest score on al1lhree testing sections.
In addition, we found that the universe may have incorrectly included test results for non-ATB
CPAt examinees with less than a high school education who received a valid test score on all
three testing sections.

Federal regulations applicable to this finding are in 34 C.P.R, § 668.150(b)(8), as cited above for
Finding No.1.

ACT's Director of Statistical Research stated that, at the time the universe for the three-year
analysis was determined, the data needed to identify CPAt ATB examinees directly was not
readily available.

We concluded that, since there is not an appropriate universe, there is no assurance that ACT's
three-year evaluation of CPAt ATB test records is reliable.




                                                              3

Audit of ACT's Career Progf::\ms Ass.es.sment Test ATB Program 	                     ED-OIG/A03-B0024



R""ommenrlation:

2. 	      We recommend that the COO for FSA require ACT to initiate appropriate action to
          ensure that ACT bases any future analysis required under 34 C.F.R. § 668.150(b)(8) on a
          complete and accurate universe of test data.

ACT's Reply

ACT concurred with this recommendation.

Finding No.3             ACT Does Not Monitor CPAt ATB Retest Administrations

ACT does not have processes in place to identify and infOlID institutions of CPAt ATB
administrations conducted in violation of its retesting procedures. Our review of ACT's file
documentation for a sample of 25 examinees from the universe of 629 examinees who retested
on the same CPAt form at the same institution in consecutive months during the period July 1,
1997, through June 30, 2000, revealed that 20 (80 percent) were retested improperly. Also, our
analysis of the universe of 9,179 examinees who were administered a retest of the CPAt
examination during the period july 1,1997, through June 30, 2000, identified approximately 400
instances in which examinees were improperly tested twice on the same CPAt form, during the
same month, at the same institution 2

The CPAt Test Administration Manual allows an examinee to retest if his or her performance on
the test was affected by conditions other than ability. For example, an examinee would be able
to retest if his or her performance was affected by an illness or by an interrupted test session, or
if there is reason to believe a significant change occurred in the examinee's level of knowledge
ami skills. A<,;<.:on,lillg Lo Lhe CPA! Tes! AdminislrUfion Manual, if n:u;:sLing is warranted-

          An alternate form of the CPAt (a form other than the one most recently
          admll1Istered) IS used for the retest (I.e., retestmg must "cycle through" all CPAt
          fomls available from ACT). If all CPAt forms available from ACT have been
          administered to the examinee, a minimum of 30 days must pass between
          administrations of the same CPAt fonn.

Federal regulations state ­

       • 	 "The agreement between a test publisher and the Secretary provides that the test
           publisher shall ... [c]ertify test administrators who have ... [tJhe necessary trainin!?"
           knowledge, and skill to test students in accordance with the test publisher's testing
           requirements ...." 34 C.F.R. § 668.150(b)(2)(i).




2 Our review revealed 410 cases where this set of circumstances occurred. We did not report the
exact number because the test date was not always accurate, as discussed in Finding No.4, and
bt:<.:ause we did noL <,;onfirlll or sample the LeSL dales to the original records.
Al!diJJ_l.LACT'~   rilreer Programs Assessment Test ATR PrOpf<lm 	                    FD-OfG/ A03-B0024



        • 	 'The agreement between a test publisher and the Secretary provides that the test
            publisher shall ... [dlecertify a test administrator for a period that coincides with the
            period for which the publisher's test is approved if the test publisher finds that the test
            administrator ... [hlas repeatedly failed to give its test in accordance with the
            publisher's instructions ...." 34 C.P.R. § 668.150(b)(3)(i).

            "An institution may usc the results of an approved test to dctcnnine a student's
            eligibility to receive Title IV, HEA programs funds if the test was independently
            administered and properly administered." 34 C.F.R. § 668. [51 (a)(2).

        • 	 "The Secretary considers that a test is properly administered if the test administrator ...
            [aldministers the test in accordance with instructions provided by the test publisher, and
            m a manner that ensures the mtegrity and security of the test ...."
            34 C.F.R. § 668.1 S 1(d)(2).

ACT's Manager of Production Services explained that ACT does not monitor CPAt ATB
retesting activity because its certified independent test administrators (ITA's) are responsible for
conducting retests in compliance with its rules.

ACT's failure to identify and report retest administrations not conducted in accordance with its
established procedures may result in invalid ATB determinations, improper admission of
students, and disbursements of Title IV, HEA program funds to ineligible students. Also,
because ACT does not monitor its certified ITA's retesting activity, ITA's who should be
decertified may continue to perfonn CPA! ATB testing.

Recommendations:

We recommend that the COO for FSA require ACT to­

3.1 	     Strengthen its management controls to ensure that ITA's follow the already estabhshed
          procedures for conducting a CPAt ATB retest.

3.2 	     Establish a system to monitor and review the testing practices of each certified ITA, and
          decertify any IT A who repeatedly violates or compromises its approved testing
          procedures.

3.3 	     Establish a system for identifying and reporting retest errors to ensure that institutions
          have accurate and timely information at the time that eligibility determinations are mmle

ACT's Reply

ACT concurred with the recommendations and stated that on May 1, 2002, it will activate a
computerized process for applying retest rules to each CPAt ATB answer sheet it scores. ACT's
reply described the process that ACT planned to implement, to identify and report retest errors.




                                                         5

Audit of ACT's Career Programs Assessment Test ATB Program



OIG's Response

ACT's planned actions must ensure it will establish a system for identifying and reporting
violations of the retesting guidelines for CPAt ATB teslS described in Attachment A of its
response to the Draft Audit Report. The CPAr ATB retest guideJines contained in Annchment A
have not yet been reviewed and approved by ED.

:Finding No.4 	        ACT's CPAt Database Contains Incomplete and Inaccurate Test Date
                       Information

The test date field maintained in ACT's CPAt database is a four digit numeric field that includes
only the month and year data for test administrations. Our review of ACT's file documentation
tor 1,15 ot 73,455 CPAt ATB tests conducted during the period July I, 1997, through June JO,
2000, revealed that the test date recorded on 9 CPAt ATB test answer sheets (9.5 percent) did not
agree with the test date in the CP At database. In seven instances the wrong test month was
recorded in the CPAt database, in one case the wrong test year was recorded, and on another
occasion both the wrong test month and year were recorded.

Data from CPAt test answer sheets that are scanned to file, for review and update to the CPAt
database, includes the examinee's name, Social Security number, date of birth, race, educational
hackgrOlmd, and item responses for the CPAt's Language Usage, Reading, and Numerical Skills
Tests. Although each examinee records the full testing date on the CPAt (including the month,
day, and year the test was administered), that full date is not recorded in a fonnat that can be
scanned to file.

ACT uses a "Site Header Sheet" to update the CPAt database with the general date (month and
year unly) for each gruup ufCPAl ATB answer sheets received from an ITA. The "Site Header
Sheet" is completed by ACT staff, and it is also used to collect the state code, test center code,
number of answer sheets to be scored, and any applicablc special codes. Special codes are used
to designate CPAt ATB tests, un-timed tests, and the lTA.

The erroneous test date records in ACT's CPAt database for nine of the ATE tests we reviewed
appear to be the result of human error on the part of ACT staff, when completing the "Site
Header Sheets." Without complete (month, day, and year) and accurate leSI date records, ACT
cannot adequately monitor CPAt ATE administrations for compliance with its retesting
procedures.

Recommendation:

4. 	   We recommend that the COO for FSA require ACT to initiate appropriate action to
       ensure that complete and accurate CPAt ATB test dates arc maintained in its CPAt
       database.

ACT's Reply

ACT concUlTed with this recommendation.



                                                  6
AIH-iir of ArT'"   rare~r   Proerams Assessment   T~st   ATR Pmp:r:'lm 	              ED-OIG/A03-B0024




Finding No.5                  ACT Needs to Improve Procedures Used to Certify CPAt ITA's

The process that ACT uses to certify CP At IT A's does not provide adequate assurance that the
ITA's are not affiliated in any way with all of the institutions for which they may perform CPAt
ATB testing. ACT's IT A certification process requires applicants to complete an "Ability-to­
Benefit Independent Test Administrator Certification Form."

Applicants use the form to provide a list of names and addresses of the schools at which they
plan to adminisler ATB tesls, and applicants sign an "ATB Test Administrator Agreement" on
the form, attesting to the requirements of test administrator independence. The "ATB Test
Administrator Agreement" docs not identify the institutions for which the ITA applicant is
asserting independence. Also, ACT mforms each eertltied CPAtlTA, upon approval. that he or
she "may administer CPAt tests for any institution currently registered with ACT as a CPAt
participating institution."

Federal regulations state

     •	   "An institution may use the results of an approved test to detelmine a student's
          eligibility to receive Title IV, REA programs funds if the test was independently
          administered and properly administered" 14 r.F.R. § 668.1 'Il(a)(2).

     •	   "The Secretary considers that a test is independently administered if the test is ...
          [g]iven by a test administrator who
                  "(i) Ras no current or prior financial or ownership interest in the institution, its
          affiliates, or its parent corporation, other than the interest obtained through its
          agreement to administer the test, and has no controlling inten:st in allY uther educarional
          institution:
                   ''OJ) Is not a current or former employee of or consultant to the institution, its
          affiliates, Of its parent corporation, a person in control of another msttlution, or a
          member of the family of any of these individuals:
                   "(iii) Is not a current or former member of the board of directors, a current or
          former employee of or a consultant to a member of the board of directors, chief
          executive officer, chief fInancial officer of the institution or its parent corporation or at
          any other institution, or a member of the family of any of the above individuals: and
                   "(iv) Is not a current or former student of the institution." 

          34 C.F.R. § 66S.1S1(b)(2). 


ACT's failure to ensure that its ITA's are independent from all institutions at which they can
conduct CPAt ATB test administrations may result in invalid ATB determinations, improper
admission of students, and disbursements of Title IV, HEA program funds to ineligible students.




                                                              7

!:A""u"ciilLt,"nfLA~C.LT-,'S"CC£"Wre",'e<!-f"-p"ro",g.!Jra!!Jrn""sw;A-,"s",~e",·~s""sm!.!.!!<Jenu.>I-,T>~e""st,-,AlTL1RLLP"ro",~1J.!ffi!.!1m-,-_ _ _ _ _ _ _ _-,F-".n~",-,O TGI A01- ROO?4



Recommendations:

We recommend that the COO for FSA require ACT to­

5.1 	         Strengthen its management controls to ensure that its certified ITA's are independent
              from all institutions at which they are approved for CP At ATB testing.

5.2 	         Estahlish a system to identify the institutions at which each certified ITA is approved for
              CPAt ATB testing.

5.3 	         Process and score CPAt ATB answer sheets for an institution only if the answer sheets
              are submitted by an IT A who is approved for testing at that institution.

ACT's Reply

ACT concurred with the recommendations.

Finding No.6 	                             ACT's ATB Administration Procedures for its Assessment Test Do
                                           Not Meet Federal Requirements

The ACT Assessment tcst is a national college admi"ion examination designed to meaSlJre high
school students' general educational development and their ability to complete college level
work. It is administered on five national test dates each year and covers four content areas:
English, mathematics, reading, and science reasoning. ED approved the ACT Assessment test
for ATB testing purposes on October 27, 1998.

ACT Revises the Assessment Test After Each Administration

ACT officials informed us that. for security purposes, a new edition of the Assessment test is
developed for each national test date. An applicatIOn for approval ot each new edition ot the test
is not submitted to ED for approval.

A test publisher that wishes to have its test approved by the Secretary for A TB purposes must
submit an application to the Secretary. Under 34 C.F.R. § 66S.l44(c)(9)

              A test publisher shall include with its application ... [ilf a test has been revised
              from the most recent edition approved by the Secretary, an analysis of the
              revisions, including the reasons for the revisions, the impii"ntions of the. revisions
              for the comparability of scores on the current test to scores on the previous test,
              and data from validity studies of the test undertaken subsequent to the revisions


Without this information, ED cannot determine whether each new edition of the Assessment test
meets the ATB testing criteria in Subpart J of 34 c.P.R. Part 668. As a result, the new editions
of the Assessment test are not approved by ED, and institutions may not use them to determine
student eligibility.



                                                                                           8
A"rlit of ArT', rareer Programs Assessment Te.sLATIlJ'-I:Qg[.i1J"mllL_ _ _ _ _ _ _ _ _"'ED!Lo\-O.nIlJ.GJLIA"",032c-J2B..,OQ\L2",4




ACT Docs Not Maintain ATB Records for the Assessment Test

The Assessment test is designed as an              ;:Himl.~~inn~ ~r.rpp.nin8    tonI for trntiitlnn}ll   P()~tsP,C()nOHry
education at four-year institutions, and it is taken, primarily, by cunent high school students,
The test is never administered specifically for ATB examinees.

ACT officials told us that they do not identify Assessment tests that are administered for ATB
purposes. As a result, ACT cannot determine whether any Assessment test data it maintains is
related to ATB adminislraliuns. Therdun;, ACT willllul be ablt: lu prepan;; the lequired three­
year analysis of Assessment ATB test result data for indications of inegularities in the test
administration process. This analysis is required under 34 C.F.R. § 66S.150(b)(S). On February
14, 2UU2. ACT submItted Its three-year analysis to FSA. Our review of the analysis revealed that
its data is insufficient and that the submission does not meet the criteria in 34 C.F.R.
§ 668.150(b)(8). (The text of 34 C.F.R. § 668.150(b)(8) is provided above, for Finding No. I).

ACT cannot identify instances of inegularities in the Assessment ATB test administration
process. As a result, Assessment test administrators who should be decertified may continue to
conduct tests, and FSA funds may be at risk.

ACT's Assessment Test Record Retention Policy Needs Improvement

According to the ACT Assessment User Handbook. "ACT keeps students' original [ACT
Assessment] registration folders and answer documents for one year.... For research purposes,
we retain student records on magnetic tape indefinitely." According to infolTl1ation for the
Assessment test on ACT's website (www.act.org), and confilTl1ed by ACT's Assistant Vice
President of Applied Research. examinecs may direct ACT to drop any or all of their Assessment
records from ACT's files at any time.

Under C.P.R. § 668.ISO(b)(7), approved ATB test publishers must-

         Keep for a period of three years each test answer sheet or electronic record
         forwarded for scoring and all other documents forwarded by the test administrator
         with regard to the test ....

Since examinees may ask ACT to destroy their Assessment test records at any time. ACT cannot
ensure that it has the complete universe of Assessment test results. Without a complete universe,
ACT cannot prepare a reliable analysis of Assessment ATB test records.

The Agreement between ED and ACT Does Not Meet Regulatory Criteria

Under 34 C.F.R. § 668. 150(a). "If the Secretary approves a test under this subpart, the test
publisher must enter into an agreement with the Secretary that contains the provisions set forth in
paragraph (b) of this section before an institution may usc the test to detenninc a student's
eligibility for Title IV, REA program funds." The agreement entered into by ED and ACT for




                                                               9
Allnit of ACT's C:ueerPrograms Assessment Test ATE ProGram 	                            ED-OlGt A03-B0024


the use of the Assessment test for ATB purposes does not meet all or the criteria required by
34 C.F.R. § 668.l50(b).

        •	    Under 34 CFR § 6f)R I ~()(h)(1), if an independent test administrator mmmits (Certain
              violations, he or she must be decertified by the test publisher "for a period that
              coincides with the period for which the publisher's test is approved ...." For the same
              violations. the agreement between ED and ACT stutes only thut ACT "will not ship test
              materials to [the] test administrator for a specified period .... Shipment of tests will
              resume only after corrections to procedures have been thoroughly documented and
              approved. "

        • 	 Under 34 C.F.R. § 668.150(b)(6), the test publisher must "[p]romptly send to the
            student and the institution the student indicated he or she is attending or scheduled to
            attend a notice stating the student's score for the test and whet/leI' or not the student
            passed the test ...." The agreement between ED and ACT states only that ACT "will
            generate score reports for answer sheets that it receives from test administrators."

        •	    Under 34 C.F.R. § 668.150(b)(7), the test publisher must "[kJeep for a period of three
              years each test answer sheet or electronic record forwarded for scoring and all other
              documents forwarded by the test administrator with regard to the test ...." The
              agn.,ement hetween ED and ACT states only that ACT "will keep for a period of two
              years each test answer sheet or electronic record forwarded for scoring."

        •	    Under 3,1 C.F.R. § 668. 150(b)(8), a test publisher must, "[t]hree years after the datc the
              Secretary approves the test and for each subsequent three-year period, analyze the test
              scores of students to determine whether the test scores produce any irregular pattern that
              raises an inference that the tests were not being properly administered, and provide the
              Secretary with a copy of this analysis ...." The agreement between ED and ACT does
              not include this requirement.

Since the agreement does not meet all ofthe required criteria in 34 C.F.R. § 668.150, it is not
valid. Consequently, institutions may not use the ACT Assessment test to detennine a student's
eligibility for Title IV REA funds.

Recommendations:

We recommend that the COO for FSA­

6.1 	        Require ACT to use only approved versions of the Assessment lest for ATB purposes.

6.2 	        Reconsider ED's approval of the ACT Assessment test as an ATB test.

6.3 	        If ED's approval of the ACT Assessment test is not withdrawn, revise the agreement
             between ED and ACT to ensure that all criteria for test publishers in Subpart J of 34
             C.F.R. Part 668 are met, including but not limited to the requirements in 34 C.F.R.
             § 668.150.



                                                      10 

AllClit of ArT's rareer Programs Assessment Test ATB Program




ACT's Reply

ACT's response indicated that it concurs with the finding; however, ACT provided an alternate
corrective action plan for the finding's resolution.

ACT's response stated that its interest in having the Assessment test on ED's approved ATB test
list was motivated by its potential service to "home school" students.

        To meet (he ATB re4uirefllcllts, ()ur previ()us understanding was lhallhese "HulIlc
        School" students would need to complete one of the approved ATB tests to
        qualify for federal financial aid; therefore, ACT submitted the ACT Assessment
        test tor ATB approval, to aVOId having "Home School" students who had
        completcd the ACT Assessment be required to test again on another approved
        test.

ACT further stated that it recently learned that new options are available for home school
students to qualify for Title IV, HEA funds, and ACT proposed that-

        Given this new information, we would like [to] review this new (to us) policy
        over the coming 30 days, with one of two possihilities likely to result:

                a. ACT would withdraw its request for ACT Assessment being included
                on the ATB list of approved tests (Home School students do not "need"
                the ACT Assessment as an ATB test)

                        OR

                b. ACT would prepare information for DOE regarding why ACT believes
                the test should be retained on the ATB list of approved tests (Home
                School students or significant numbers of other students are well-served
                by having the ACT Assessment appear on the ATB list of approved tests).

ACT asked if its proposal would be an acceptahle process with respect to determining whether to
seek removal of its Assessment tcst from the Jist of ED-approved ATB tests.

OIG's Response

We reviewed ACT's comments but our recommendations remain unchanged. ACT's proposal
and question should be addressed to the appropriate program officials at ED.

                                         OTHER MA TTERS

Two other ACT tests, the Compass test and the Asset test, arc approved by ED for ATB testing
purposes. ACT is required, under 34 C.ER. 668.150(b)(8), to submit three-year analyses to
discover irregularities in the test administration process. ACT did not submit analyses for these



                                                    11 

Audit nf ACT', Cnreer Pnwr3ms Assessment Test ATR Prnornm 	                       ED-OIG/A01-BOO?4



two tests in a timely manner. ACT was required to submit the analyses to ED no later than
October 1999, but ED did not receive them until January 2001. (The text of 34 C.P.R.
§ 668.150(b)(8) is provided above, for Pinding No.1).

                                          BACKGROUND

The Higher Education Technical Amendments of 1991 amended the lillA, requiring
postsecondary students who do not have a high school diploma or its equivalent to pass an
independently administered examination that has been approved by ED before receiving Title
IV, HEA program funds. These examinations are intended to establish that studenls have the
ability to benefit from postsecondary school training programs. This testing has become known
as "Ability-to-Benefit (ATB) testing."

On December 1, 1995, ED published final regulations, effective July 1, 1996, specifying the
procedures and requirements for ATB testing which affect test publishers, schools, and ITA's, as
Subpart J of 34 C.P.R. Part 668. Compliance with these regulations is mandatory in detennining
the eligibility of applicants for Title IV, IIEA program funds.

ED assesses tests submitted for ATB approval according to the requirements in Subpart J of 34
C.F.R. Part 668. ED approves a test for a period of no more than five years, although the
approval ean he extended while a suhsequent review is conducted tn detennine fe-approval. A
list of approved tests and passing scores are puhlished in the Federal Register. The initial list of
approved tests was published in the Federal Register on October 25,1996. ACT's CPAt was
approved by ED and appeared on the list published on October 25, 1996. ACT submitted an
application for re-approval of the CPAt for ATB testing, and ED is currently conducting its
review of the application.

                        OBJECTIVE, SCOPE AND METHODOLOGY

The objective of our audit was to detcnnine whether ACT properly administered its CPAt ATB
program in accordance with its agreement with ED, for the approved use of the CPAt, and with
applicable laws and regulations. To identify our ohjective, we reviewed

•	   The agreements between ACT and ED for the use of ACT's CPAt, Compass, Asset, and
     Assessment tests for ATE testing purposes;

•	   ACT's reports on CPAt, Compass, and Asset ATB testing. covering the period January 1,
     1997. through December 31. 1999: and

•	   ACT's CPAt, Compass, Asset, and Assessment test user manuals.

To accomplish our objective

•	   We worked with ACT officials and developed a process to identify the universe of CPAt
     ATB test administrations (73,455) conducted dUling the period July 1, 1997, through June




                                                  12 

Audit of ACT's Career Programs Assessment Test ATcfBLP£'r""o$gr"'allm'--_ _ _ _ _ _ _ _ _~E"'D"'-"O'_'1GoE/~A",O,,3c':B"0",0,,,",24



       ,0,2000. We analyzed a computer file, provided hy ACT, !h,,! contained the CPAt ATR
       test result data for this universe. We examined ACT's file documentation for­

        1, 	   A random sample of 25 CPAt ATB test administrations selected from the universe of
               73,455 administrations conducted during the period July 1, 1997, through June 30,
               2000.

        2. 	   A random sample of 25 CPAt ATB applicants selected from the universe of 629
               applicants who retested (54 administrations) on the same CPAt test form, at the same
               school, in consecutive months, during the period July 1, 1997, through June 30, 2000.

        3. 	   A random sample of 10 CPAt ATB test administrations selected from the universe of
               1143 administrations conducted during the peliod July 1, 1997, through June 30,
               2000, in which the applicant indicated on the CPAt answer sheet having obtained a 2
               or 4 year college degree.

        4. 	   The universe of CPA! ATB test administrations (6) conducted during the pellod July
               1. 1997, through June 30, 2000, for applicants with disabilities.

•	     We analyzed a computer file, provided by ACT, covering the period July 1997 through
       June 2000, that contained data about the universe of certified (394), inactive (295), and
       decertified (5) CPA! ITA's. We examined the tIle documentation for 14 randomly selected
       certified ITA's and 11 randomly selected inactive IT A's.

•	     We reviewed ACT's procedures for CPAt ATB test administration, scoring and reporting .
       We interviewed ACT personnel to obtain an understanding of these procedures.

DUling our review at ACT, we also tested the reliability of computerized CPAt ATB data by
comparing selected data records with the completed CPAt answer sheets. We concluded that the
computerized informatIOn was sufficiently reliable for the purposes of our audit.

We conducted our on-site fieldwork at ACT, in Iowa City, lA, from August 13, 2001, through
August 17,2001. Our exit conference was held on August 17,2001. From August 2001 through
Januaty 2002, we analyzed data that we received from the institution and at the Department. Our
audit was performed in accordance with government auditing standards appropriate to the scope
of the audit described above.

                            STATF.MJ<~NT          ON MANAGEMENT CONTROLS

As part of our review, we assessed ACT's management control structure, as well as its policies,
procedures, and practices applicahle to the scope of the audit. We assessed the level of control
risk for determining the nature, extent, and timing of our substantive tests to accomplish the audit
objective.




                                                               13 

Audit of ACT's Career Proo-rams   A"'~f'5;,~m~nt T~st   ATR   Prner:lm            EO.OJG/ Am·R0024



For the purpose of this report, we assessed and classified the significant controls into the
following categories:

        •    ITA Training and Certification
        •    CP At Administration
        •    CPAt Reporting
        •    CPA! Records l'v1anagcmcnt

Because of inherent limitations, a study and evaluation made for the limited purpose described
above would not necessarily disclose all material weaknesses in the management controls.
However, our assessment disclosed management control weaknesses, which adversely affected
ACT's ability to administer its CPAt ATB program. These weaknesses included inadequate
reporting procedures, lack of monitoring procedures, deficient data collection and recordmg
procedures, and inadequate IT A certification procedures. These weaknesses and their effects are
fully discussed in the Audit Results section of this report,
Audit of ACT'.s Career Program~ Assessment Test ATB Program                       ED-OIGI A03-BOO24



                                 ADMINISTRATIVE MATTERS 


Slatements that managerial practices need improvements, as well as other conclusions and
recommendations in this report represent the opinions of the Office of Inspector General.
Determination of corrective action to be taken will be made by the appropriate Department of
Education officials.

If you have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following Department of Education
official, who will consider them before taking final Department action on the audit:

                        Mr. Greg Woods
                        Chief Operating Officer
                        Federal Student Aid
                        Union Center Plaza Building, Rm. 112G 1
                        830 1sl Street, NE
                        Washington, DC 20202

Office of Management and Budget Circular A-50 directs Federal agencies to expedite the
resolution of audits by initiating timely action on the findings and recommendations contained
therein~   Therefore, receipt of your comments within 30 days would be greatly appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General arc available, if requested, to members of the press and general
public to the extent infonnation contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us in the review. Should you have any questions
concerning this report please contact Bernard Tadley, Regional Inspector General for Audit, at
215-656-6279.




                                                Assistant Inspector General for Audit

Attachment




                                                   15
                                                                               Attachment - Auditee's Response

                                          ACT~                                 

                                          INFORMATION FOR LIFE'S TRANSITIONS




April 16, 2002




 Mr, Bernard Todley 

 Regional Inspector General for Audit 

 U,S, Department of Education 

 Office of Inspector General 

 The Wanamaker Building 

 100 Penn Square East. Suite 502 

 Philadelphia, PA 19107 



 Dear Mr, Tadley,

 On behalf of ACT, Inc., enclosed please find the document titled "ACT Responses to Draft Audit Report
 Regarding Ability-to-Benefit Policies and Procedures for Career Programs Assessment (CPAt)", This
 information is being provided in response to the Draft Audit Report (Control Number ED-OIG/A03-B0024),
 dated March 15, 2002.

 The information in the document includes responses to each of the various findings and recommendations
 identified in the Draft Audit Report, as well as several questions regarding your advice as to how ACT should
 proceed.

 We appreciate this opportunity to respond to the Draft Audit Report and look forward to continuing to serve
 students and institutions and to meet their individual and collective needs with respect to Ability-to-Benefit
 testing services. If additional questions should arise or additional information is needed, please do not
 hesitate to contact me regarding the CPAt, ASSET, COMPASS, ESL, or ACT Assessmp.nt Progr:::lm.

 Sincerely,




John D. Roth, PllO 

ACT, Inc. 

Principal Consultant 

 Placement Assessment Programs (ASSET, COMPASS, ESL, CPAt) 

Phone: 319/337-1030 

FAX: 319/337-1790 

Internet: rothj@act.org 





:no I No!-th Dodge Street      po. Box 168       Iowa City, Iowa 52243-0168        319133/ 1000    wwwactorg
                           ACT Responses to Draft Audit Report Regarding 

        Ability-to-Benefit Policies and Procedures for Career Programs Assessment (CPAt) 

                                 (Control Number ED-OIG/A03-B0024) 

                                             April,2002 




Finding No.1 	          ACT's Analysis of CPAt ATB Test Scores Was Not Submitted Timely

Hecommendation:

1. 	    We recommend that the Chief Operating Officer (COO) for FSA require ACT fa strengthen
        its management controls to ensure that any future analysis of student CPAt ATB test
        scores required under 34 C.F.R. § 668.150(b)(8) is prepared and submitted to ED in a timely
        manner.

ACT Response:

ACT will submit the next analysis of student ATB test scores on a schedule as specified by ED.

Question: The analysis reviewed in the current audit was submitted by ACT in January 2001, covering
student testing records for the period of January, 1997 through December 1999. What is the date for
submission for the next analyses of student ATB test scores for CPAt and what period of testing dates
should those analyses include?




Finding No.2 	          ACT's Analysis of CPAt ATB Test Scores Did Not Contain the True CPAt
                        ATB Population

Recommendation:

2. 	    We recommend lhal the COO lor FSA require ACT to iniliate appropriate action to ensure
        that ACT bases any future analysis required under 34 C.F.R. § 668.150(b)(8) on a complete
        and accurate universe of test data.

ACT Response:

Based en ATB indicators in ACT's batch tracking system, ACT has modified all CPAt records processed
tor A I Cl purposes since July 1, 1997 so that the individual student records now carry an indicator that
identifies the record as being scored for ATB purposes. All future CPAt scoring records will also contain
an indicator as to whether the record was scored for ATB purposes"




Finding No.3 	          ACT Does Not Monitor CPA! ATB Retest Administrations

Recommendations:

We recommend that the COO for I=SA require ACT to ­




                                                    1
3.1 	   Slren!llhen its mana!lement controls to ensure that ITA's follow the already established
        procedures for conducting a CPAt ATB retest.

ACT Response:

ACT will notify the institution, examinee, and ITA involved where CPAt ATB testing has violated ACT's
retest policy. ACT will not report scores from ATB administrations in which the CPAt retest policy has
been violated_

Institutions found to repeatedly violate ACT's retest policy or repeatedly fail to adhere to DOE or ACT
policies and procedures will be disqualified as CPA! Participating institutions eligible to use CPAt. ITA's
who repeatedly violate ACT's retest policy or repeatedly fail to adhere to DOE or ACT policies and
procedures will be decertified through written notice to the ITA and to each institution for which the ITA is
certified, ACT will not report scores from answer sheets submitted by a decertified ITA.

ACT will process all CPAt ATB answer sheets submitted, even if scores are not reported, in order to
retain a record of the test administration for future monitoring of the retest policy,

ACT will revise sections of the CPAt Test Administration Manual to include clarification of the CPAt retest
policy and its application by ACT as part of the ATB scoring and reporting process (to become effective
5/01/02), Additional procedures requiring institutions and ITA's to provide information to all examinees
re~arding rele~t rul,,~ will also be added to the Manual. For an overview of the retest rules 10 be apfllied,
see Attachment A: Retesting Conditions GuidelineslRequirements for CPAt ATB,



3.2 	   Establish a system to monitor and review the testing practices of each certified ITA, and
        decertify any IT A who repeatedly violates or compromises its approved testing
        procedures.

ACT Response:ACT will monitor and review the testing practices of each certified Independent Test
Admini"trator (ITA) through:
        1) review of testing Irregularity Reports and other documentation submitted with answer sheets;
        2) follow-up and documentation of comments and reports from examinees or school officials
           related to ITA performance;
        3) review of management reports from the CPAt scoring system identifying records in violation
           of the CPAt retest policy;
        4) issuance of written notice to ITA (copy to institution) of infractions and steps to be taken to
            correct procedures.

Repeated infractions and notices will be documented in the ITA's file, Upon repeated infractions or
repeated failure to adhere to DOE or ACT policies and procedures, an ITA will be decertified through
written notice to the ITA and to each institution for which the ITA is certified, ACT maintains a file
identifying decertified ITA's and will not report scores from answer sheets submitted by a decertified ITA,



3,3 Establish a system for identifying and reporting retest errors to ensure that institutions have
accurate and timely information at the time that eligibility determinations aro made.

ACT Response:

On 5/01/02, ACT will activate a computerized process for applying retest rules to each CPAt ATB answer
sheet scored by ACT, As part of this computerized routine, ACT can verify whether the appropriate retest
requirements are met. The retest check will query the CPAt master history file and flag any record that
marches by las! name or SSN and resr lorm wimln me past 9·1 days 01 me rest dare on me current answer
sheet ACT staff will then determine whether or not the flagqed record meets the retest criteria. When a
particular administration violates the CPAt retest policy, ACT will provide written notice to the Institution, to
the examinee, and to the ITA that examinee scores are not being reported because of a retest violation.

Periodically ACT will generate a report from the CPAt scoring and reporting system that will show the
number of retest violations by institution and by ITA. ACT will contact institutions and ITA's that have
continued to violate the retest policy after receiving notice of retest violations and review the policy to
an~wer   any qu"~liun~ afllj be ,;ur" II", in,;iiluliun '''IU til" ITA UnUJ;I,;taIlU II,,, f./uli"y.

All CPAt Institutions and certified ITA's will receive a revised CPAI Test Administration Manual from ACT
following th'" DOE's publication of the new passing score values in the Federal Registry




Finding No.4                 ACT's CPAt Database Contains Incomplete and Inaccurate Test Date
                             Information

Recommendation:

4. 	     We recommend that the COO for FSA require ACT to initiate appropriate action to ensure
         that complete and accurate CPA! ATB test dates are maintained in its CPAt database.

ACT Response:

 ACT will activate (as of 5/01/02) the insertion of a full test date ill each record anu a computerized
process for applying retest rules to each CPA! ATB answer sheet scored by ACT.



Finding No.5                 ACT Needs to Improve Procedures Used to Certify CPAt ITA's

Recommendations: 


We recommend that the COO for FSA require ACT to­

5.1 	    Strengthen its management controls to ensure that its certified ITA's are independent from
         all institutions at which they are approved for CPAt ATB testing.


ACT Response:

ACT I,,,~ inlwuulO!;!u cuJuilio",,' .,1U"!;!dw!;!,, to ""rtify Ilt)w ITA's   (Of   s.,"'lOililO insliluliuns lu "nsurt; 1),,,1   lI''''
ITA is truly independent of each institution for which he/she is certified to administer CPAt tests for ATB
purposes. In April 2002, ACT will initiate additional steps to collect and review credentials and test
administration aGtivities of all previously certified ITA's to ensure that they are certified only for institutions
from which they are independent as defined by the U.S. Department of Education.

5.2 	    Establish a system to identify thll! institutions at which each certified ITA is approved for
         CPAt ATB testing.

ACT Response:

ACT has expanded its ITA database to track specific institutions for which the ITA has been certified to
administer ATB testing.




                                                              3
5.3 	   Process and score CPAt ATB answer sheets for an institution only if the answer sheets
        are submitted by an ITA who is approved for testing at that institution.

ACT staff reviews the database of approved ITA's before CPAt ATB answer documents begin the scoring
and reporting process. If the person listed on the CPAt ATB Answer Document ID Form is not on the
approved ITA list for the specific institution. ACT notifies the institution, the examinees, and the ITA that
no scores will be reported for that administration.




Finding NO.6 	           ACT's ATB Administration Procedures for its Assessment Test 00 Not
                         Meet Federal Requirements

Recommendations:

We recommend Ihat the COO for FSA­

6.1 	   Require ACT 10 use only approved versions of the Assessment lest for ATB purposes.

6.2 	   Reconsider ED's approval of the ACT Assessment test as an ATB test.

6.3 	   If ED's approval of the ACT Assessmenllest is not wilhdrawn, revise the agreement
        between ED and ACT to ensure that all criteria for test publishers in Subpart J of 34 C.1=.R.
        Part 668 are met, including but not limited to the requirements in 34 C.F.R. § 668.150.

ACT Response:

ACT's interest in including the ACT Assessment on the ATB approved test list has been largely prompted
by its potential service to "Home School" students (students who do not receive a standard High School
diploma but wno often completed me ACT Assessment Instrument as part Of their application for COllege
admission).

To meet the ATB requiremento, our proviouG undomtanding wao thnt thone 'IHomc School" otudcntG
would need to complete one of the approved ATB tests to qualify for federal financial aid; therefore, ACT
submitted the ACT Assessment test for ATB approval, to avoid having "Home School" students who had
completed the ACT Assessment be required to test again on another approved test.

However, as part of the follow up to the Draft Audit Report, we have learned (earlier today) that July 1,
2001 revisions of the pertinent policies for ATB include new options for this for "Home SchOOl" students
that would/could avoid unnecessary additional testing for these students.

Given this new information, we would like review this new (to us) policy over the coming 30 days, with one
ur twu fJu~~iLJiliti"~ likely tu re~ult.

        a. ACT would withdraw its request for ACT Assessment being included on the ATB list of
        approved tests (Home School students do not "need" the ACT Assessment as an ATB test)

                 OR

        b. ACT would prepare information for DOE regarding why ACT believes the test should be
        retained on the ATB list of approved tests (Home School students or significant numbers of other
        students are well-served by having the ACT Assessment appear on the ATB list of approved
        tests).


                                                     4
Would this be an acceptable process with respect to determining whether ACT would seek to remove the
ACT Assessment from the approvod tests for determining ATB eligibility?



                                          OTHER MATTERS

Two other ACT tests, the COMPASS test and the ASSET test, are approved by ED for ATB testing
purposes. ACT is required, under 34 C.F.R. 668.150(b)(8), to submit three-year analyses to
discover irregularities in the test administration process. ACT did not submit analyses for these
two tests in a timely manner. ACT was required to submit the analyses to ED no later than
October 1999, but ED did not receive them until January 2001. (The text of 34 C.F.R. §
668.150(b)(8) is provided above, lor Finding No.1).

ACT Response:

ACT will submit the next analysis of student ATB test scores for ASSET and COMPASS on a schedule as
specified by ED.

Question: The analysis reviewed in the current audit was submitted by ACT in January 2001, covering
student testing records for the period of January, 1997 through December 1999. What Is the dale tor
submission for the next analyses of student ATB test scores for ASSET and COMPASS and what period
of testing dates should those analyses include?




                                                  5

            Attachment A: Retesting Conditions Guidelines/Requirements for CPAt ATB

A. 	 There is convincing evidence that the reason for retesting under either Situation 1 (interfering
    circumstances) or Situation 2 (additional learning has occurred) is true.

B. 	 If the retest occurs in less than 30 days from the previous testin(J, an alternate form (I.e., a form other
     than the one most recently administered) must be used for the retest. A student may be retested
     using the same form if a minimum of 30 days has passed since the previous administration of that
     form. However, no form may be administered more than twice to a given individual in a gO-day
     fJ"riuu.

For ATB students tested with CPAt, the U.S. Department of Education requires that all three CPAt
tests be administered during a retest, and prohibits a "mix and match" strategy.




                                                       6

                         REPORT DISTRIBUTION LIST 

                        CONTROL NO. ED-OIG/A03-B0024 


                                                                               No. of
Auditee                                                                        Copies
      Dr. John Roth                                                                 I
       Principal Consultant. Placement Programs
       American College Testing. Inc.
       2201 North Dodge Street
       P.O. Box 168
       Iowa City, IA 52243-0168

ED Action Official
      Mr. Greg Woods
      Chief Operating Officer
      Federal Student Aid

Other ED Officials/Staff (electronic copy)

       Audit Liaison Officer, Federal Student Aid                                   I
       Assistant General Counsel, Office of the General Counsel
       Correspondence Control, Office of the General Counsel
       Chief of Staff, Office of the Secretary                                      1
       Under Secretary. Officc of the Under Secretary                               1
      Deputy Secretary, Office of the Deputy Secretary                              1
      Director. Office of Public Affairs                                            1
      Press Secretary. Office of Public Affairs                                     1
      Assistant Secretary, Office of Legislation and Congressional Affairs          1
      Chief Financial Officer, Office of the Chief Financial Officer                1
      Director, Financial Improvement and Post Audit Operations, Otfice             I
             of the Chief Financial Officer
      Post Audit Group Supervisor. Financial Improvemenl and Post                  I
             Audit Operations, Office of the Chief Financial Officer
      Assistant Secretary, Office of Intergovernmental and Interagency
             Affairs
      Assistant Secretary, Office of Postsecondary Education                       I
      General Manager for Schools Channel, Federal Student Aid                     1
       Area Case Director for   Ca~e   Management nne! ()v","<ight. F"e!eral
              Student Aid