oversight

X11J0002 - Weaknesses in the regulations and guidelines for Department of Education approved publishers of the Ability-to-Benefit Test - Date Issued: 01/25/2010 PDF (950K)

Published by the Department of Education, Office of Inspector General on 2010-01-25.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                     UNITED STATES DEPARTMENT OF EDUCATION
                                          OFFICE OF INSPECTOR GENERAL

                                                      Information Technology Audits and Computer Crime Investigations


FINAL MANAGEMENT INFORMATION REPORT

DATE:           01/25/2010
TO:            William J. Taggart
               Chief Operating Officer
               Federal Student Aid

               Daniel T. Madzelan
               Delegated the Authority to Perform the Functions and Duties of the
               Assistant Secretary for the Office of Postsecondary Education

FROM:          Charles E. Coe, Jr. /s/
               Assistant Inspector General
               Information Technology Audits and Computer Crime Investigations
               Office of Inspector General

SUBJECT: Weaknesses in the regulations and guidelines for Department of Education
         approved publishers of the Ability-to-Benefit Test (X11J0002)

The purpose of this Final Management Information Report (MIR) is to provide the Office of
Federal Student Aid (FSA) and the Office of Postsecondary Education (OPE) with information
that might be beneficial in improving the Department of Education (ED) regulations and
guidelines for ED-approved publishers of the Ability-to-Benefit (ATB) test.

In FY 2009, FSA disbursed approximately $113 billion in Title IV funds. Of these funds, up to
$12.8 billion (11.3%) were disbursed to students who qualified for Title IV aid on the basis of an
ATB exam.1

A data analytics project conducted by the Office of Inspector General (OIG) has identified
potential vulnerabilities in ED's regulations concerning the test score analysis submitted by ATB
publishers every three years as part of their agreement with the Secretary. This project examined
Independent Test Administrators (ITAs) who were decertified by one specific publisher after its
triennial test score analysis revealed potentially compromised or invalid ATB examinations. Of
the 106 ITAs decertified by the publisher, OIG identified 83 decertified ITAs who provided tests
for approximately 5,619 students who received an estimated $51.4 million in disbursed Title IV
funds at 133 post-secondary institutions.2

1
  This figure represents the amount of Title IV aid disbursed to students who filed the Free Application for Federal
Student Aid (FAFSA) and did not indicate they had a high school diploma, General Education Development (GED)
credential, or were homeschooled.
2
  This figure represents the total disbursements of Title IV funds to these 5,619 students. In order to determine the
actual Title IV loss, additional investigative steps will be required.
                                                     550 12th St SW, Suite 8000
                                                      Washington, DC 20202

                      The Department of Education's mission is to promote student achievement and preparation
                      for global competitiveness by fostering educational excellence and ensuring equal access.
Page 2 Final Management Information Report Ability-to-Benefit Test


BACKGROUND

Under ED’s regulations, an applicant who has not earned a high school diploma or its recognized
equivalent may qualify for Title IV funds by passing an independently administered test
published by an ED-approved ATB publisher.3 An institution may use an approved ATB test to
determine whether a student who has not earned a high school diploma or its recognized
equivalent may be eligible to receive any Title IV funds.

All institutions providing ATB testing (except those that qualify as an assessment center)4 must
use the services of an ITA, who has been certified by the appropriate test publisher.5 In order to
qualify a student to receive Title IV funds, the ATB test must be administered by an ITA who is
not affiliated in any way with the school or its staff.6

ED regulations require the publishers of ATB examinations to decertify any ITA who violates or
compromises approved ATB testing procedures.7 The publishers of the ATB examinations are
required under 34 C.F.R. § 668.150(b)(8) to conduct an analysis of their ITAs test scores every
three years and provide their findings to the Secretary.

Previous Office of Inspector General Activities

In 2002, OIG conducted audits of two approved ATB publishers: Wonderlic Inc. (Wonderlic)
and American College Testing (ACT).8 During this time, OIG also conducted an audit to
evaluate FSA’s methods for monitoring approved ATB test publishers from July 1, 1997,
through June 30, 2000.9 These audits found:

    1. FSA did not have an effective monitoring system in place to ensure that ED-approved
       ATB test publishers comply with applicable laws and regulations,
    2. FSA’s procedures did not ensure that ATB test publishers comply with their agreements
       with the Secretary, and
    3. Seven ITAs who were decertified from Wonderlic for suspicious testing activity
       subsequently received certification through ACT to administer ATB tests.

In March 2006, Wonderlic submitted a report, “A Review of Ability-to-Benefit Test
Administrations,” to FSA in compliance with 34 C.F.R. § 668.150(b)(8).10 The report covered
the test administrations of Wonderlic's ITAs conducted between October 26, 2002, and October

3
  34 C.F.R. § 668.32(e)(2) (2008).
4
  34 C.F.R. § 668.142 (2008).
5
  34 C.F.R. § 668.151 (2008).
6
  34 C.F.R. § 668.151(b)(2) (2008).
7
  34 C.F.R. § 668.150(b)(3) (2008).
8
  "Wonderlic's Ability to Benefit Program", (A03-B0022, February 4, 2002) and "American College Testing's
Career Programs Assessment Test (CPat) Ability-to-Benefit (ATB) Program" (A03-B0024, May 13, 2002).
9
  "Audit of Procedures at Federal Student Aid for Monitoring the Ability-to-Benefit (ATB) Test Publishers
Approved by the U.S. Department of Education(ED): American College Testing (ACT), The College Board,
CTB/McGraw-Hill, and Wonderlic", (A03-B0001, August 8, 2002).
10
   This report was prepared by a third-party company, Data Research Services Inc., and included an audit review of
Wonderlic's test administrations using the A*Star® method of analysis.
Page 3 Final Management Information Report Ability-to-Benefit Test


25, 2005. The report included an audit review of the ATB test administrations that used a
method of analysis contrasting the patterns of test answers provided by the ITAs to normative
patterns established for the specific test form. As a result of the contrast, it was determined that
there was increased frequency of improper influence among certain ITAs, indicating that they
administered potentially compromised or invalid ATB examinations. On the basis of this
review, Wonderlic decertified 106 of its ITAs. These 106 ITAs tested approximately 39,300
students in the time period covered by the review.

In March 2007, Wonderlic provided OIG with testing data for the 106 ITAs. From the group of
106 decertified ITAs, OIG identified 83 ITAs who provided potentially compromised or invalid
ATB examinations for approximately 5,619 students at 133 Title IV post-secondary
institutions.11 These 5,619 students received an estimated $51.4 million in disbursed Title IV
funds between January 1, 2005, and January 31, 2009.12

At this time, OIG has not yet determined if the individual tests administered to these students
were compromised or invalid. However, as discussed above, these students were tested during
the period of Wonderlic's report and audit review. Based on the analysis of the data from
Wonderlic, OIG identified several instances of potential criminal activity that we are considering
for further investigative activity.

Government Accountability Office Activities

In August 2009 the Government Accountability Office (GAO) released the report, "Proprietary
Schools: Stronger Department of Education Oversight Needed to Help Ensure Only Eligible
Students Receive Federal Student Aid," which included recommendations to improve FSA's
administration of the ATB program. At the start of GAO's audit, OIG met with the GAO
auditors and provided them with information found during our analysis of the data; this is
discussed in GAO's report. GAO's report included the following recommendations:

        Conduct regular follow-up of ATB test analyses submissions to ensure Federally
         approved test publishers provide complete submissions as required;
        Use data provided by test publishers on schools where test administrators improperly
         administered tests and were later decertified to target schools for further review;
        Require test publishers to conduct an interim or mid-point analysis—a supplement to the
         3-year test score analysis and submission requirement—to provide a preliminary review
         of potential testing problems, and submit a copy of their results to the Secretary; or
         require test publishers to have a process to follow-up on identified test score
         irregularities, take action to decertify test administrators if test irregularities suggest
         improper test administration, report actions taken as a result of test score analysis to the

11
   The OIG analyzed testing data for all 106 of the decertified ITAs. For the purposes of this project, ITAs who
tested a relatively small number of students (in most cases fewer than 50 students) between October 26, 2002 and
October 25, 2005 were excluded from the data. Students that were administered the ATB test and received
disbursements of Title IV aid prior to January 1, 2005 were excluded in order to comply with the statute of
limitations for criminal charges.
12
   This figure represents the total disbursements of Title IV funds to these 5,619 students. In order to determine the
actual Title IV loss and specific instances of fraudulent activity, additional investigative steps will be required.
Page 4 Final Management Information Report Ability-to-Benefit Test


        Secretary and prohibit test publishers from using ATB test administrators who have been
        decertified by any test publisher

IDENTIFIED PROGRAM DEFICIENCIES

OIG identified several vulnerabilities in the regulations and guidelines regarding the approved
publishers of the ATB test.

Requirements for the Triennial Test Score Analysis

To become an approved publisher of the ATB test, publishers must submit an application to ED
and undergo a rigorous review process. As part of this, ED contracts with experts in the fields of
educational testing and psychometrics to review each publisher’s ATB test and testing
methods.13 Once publishers successfully complete the application process, the Secretary may
approve the test for a period of up to five years. At least six months before the end of their
approval period, publishers must resubmit their application for another five year approval period
and repeat the review process.14

Publishers are required to maintain test records for at least three years. Three years from the date
their test was approved and for each subsequent three-year period, the publishers must analyze
the test scores to determine if any testing anomalies exist and provide a copy of their analysis to
the Secretary.15 This triennial test score analysis is separate from the information submitted
during the publisher's initial application or any subsequent applications for approval, and it is not
subject to the same internal review process by FSA. Unlike the requirements for initial approval,
the regulations provide limited instructions for the triennial analysis and require only that the
publisher "analyze the test scores of students to determine whether the test scores produce any
irregular pattern that raises an inference that the tests were not being properly administered, and
provide the Secretary with a copy of this analysis."16 This lack of detail has resulted in
significant variations among the test-score analyses submitted by the individual publishers.
OIG's review of the submissions and conversations with FSA officials indicated that some
publishers of the ATB test merely submit a list of numbers with little or no analysis, while others
submit reports prepared by third-party reviewers and are several hundred pages in length.17

The regulations also do not specify when publishers must submit their triennial test.
Conversations with FSA revealed several publishers do not submit their analysis in a timely
fashion. To date, Wonderlic appears to be the only publisher who uses the test score analysis as
a basis for identifying and decertifying ITAs that exhibit suspicious testing patterns.




13
   34 C.F.R. § 668.145(a)(1) (2008).
14
   34 C.F.R. § 668.145(d) (2008).
15
   34 C.F.R. § 668.150(b)(7&8) (2008).
16
   34 C.F.R. § 668.150(b)(8) (2008).
17
   As previously discussed, the report submitted by Wonderlic in 2006 was prepared by a third-party company, Data
Research Services Inc., and included an audit review of Wonderlic's test administrations using the A*Star® method
of analysis.
Page 5 Final Management Information Report Ability-to-Benefit Test


FSA’s Internal Review Process

The current regulations do not stipulate a process for review or evaluation of the triennial test
analysis submitted by the publishers, and there has been little formal oversight or review of the
test score analysis. Unlike data submitted during the five-year reapplication, data from the
triennial test analysis are not required to be reviewed by experts in educational testing and
psychometrics. During a meeting with FSA officials, FSA confirmed it did not have a formal
internal process for reviewing or analyzing the publishers’ triennial test score analysis or for
ensuring that the test score analysis submitted by the publishers complied with applicable laws
and regulations. This is consistent with the findings in the 2002 OIG audit discussed above.
FSA also confirmed it has not provided any of the publishers with feedback on the results of
their analysis.

FSA is aware of these issues and is working to address them. However, without clarification of
the regulations, it may be difficult to establish long-term policies and procedures for the
submission and review of the publisher’s analysis.

Misaligned Reporting Periods

The time periods for the five year reapplication for approval and the triennial test analysis are
misaligned. Under the current system of regulations, FSA does not receive a report of the
publisher’s test analysis for the final two years of an approval period until at least a full year after
a new approval period has begun. Because the requirements for the five-year reapplication do
not require publishers to include the data from their test-score analysis and publishers must apply
to renew their approval six months prior to the end of their existing approval period, FSA may
not be notified of any current anomalies or problems in the test analysis that may affect its
decision to renew a publisher's approval period until after the renewal process is completed.

Decertified ITAs

As stated in 34 C.F.R. § 668.150, institutions must use a certified ITA to administer the ATB
test. Publishers of the ATB test are responsible for certifying ITAs to administer their test and
for decertifying any ITA who violates or compromises approved ATB testing procedures.
However, the ATB publishers are not required to notify an institution when an ITA is decertified.
They do not provide FSA or the institutions with a list of decertified ITAs nor do they share
information on their decertified ITAs with other publishers. The 2002 OIG audits of Wonderlic
and ACT found that ACT was employing seven ITAs that were previously decertified by
Wonderlic for demonstrating testing patterns that differed significantly from the norm. It is
possible for an ITA decertified by one publisher for testing fraud to become an ITA for another
publisher or at another institution without the knowledge of either entity.

SUGGESTIONS

In order to specify the publishers' responsibilities under the regulations outlined in 34 C.F.R.
§ 668.141-156 and to ensure that adequate internal controls and guidelines exist, the following
actions should take place:
Page 6 Final Management Information Report Ability-to-Benefit Test


   1. Revise the current regulations to specify the requirements publishers must follow in
      completing and submitting a triennial test score analysis, including establishing standards
      and a timeline for providing the Secretary with a copy of the completed analysis;
   2. Revise the timelines for submitting the triennial test score analysis and the requirements
      for the five-year renewal application to make the reporting periods consistent and to
      ensure that FSA has results from the most current test analysis when considering the
      publisher's renewal application;
   3. Revise the regulations to require publishers to immediately report to FSA and the
      institution, or both, when an ITA is decertified and have FSA disseminate this
      information as necessary;
   4. Establish a formal and timely internal review process within FSA for collecting and
      reviewing the publisher’s triennial test score analysis, similar to the review process for
      the five-year renewal applications.

DEPARTMENT RESPONSE

A draft of this report was provided to FSA and OPE on November 19, 2009. After receiving the
draft, FSA and OPE provided OIG with a formal, written response on January 11, 2010.
(Attachment 1) In this response, the Assistant Secretary for OPE and the Chief Operating
Officer for FSA stated that OIG suggestions numbered 1-3 are under consideration at the
negotiated rulemaking that began in November 2009 and is scheduled for conclusion in January
2010. Until such time as the regulatory process is complete, they will be unable to provide
specifics regarding the regulatory changes. When details are confirmed, a Corrective Action
Plan can be developed.

With regard to OIG suggestion 4, FSA and OPE responded that the Department now has systems
in place to monitor and track the triennial test score analysis submitted by test publishers and is
currently engaged in contracting for the services of independent psychometritions who will
review the test publisher's submissions. In addition, the Department has begun planning for
changes to its school reporting systems that will support student-specific ATB reporting to help
the Department focus on institutions with a high number of ATB-eligible students.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
OIG are available to members of the press and general public to the extent information contained
therein is not subject to exemptions in the Act.

If you have any questions, please contact Mark Smith, Special Agent in Charge for the
Technology Crimes Division, at (202) 245-7019.

cc: Martha J. Kanter, Under Secretary, Office of the Under Secretary