oversight

Direct Assessment Programs: Processes for Identifying Risks and Evaluating Applications for Title IV Eligibility Need Strengthening to Better Mitigate Risks Posed to the Title IV Programs

Published by the Department of Education, Office of Inspector General on 2014-09-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                       UNITED STATES DEPARTMENT OF EDUCATION
                                            OFFICE OF INSPECTOR GENERAL

                                                                                                                  AUDIT SERVICES


                                                                                               Control Number
                                                                                               ED-OIG/A05N0004


September 30, 2014


Dr. Ted Mitchell
Under Secretary
U.S. Department of Education
400 Maryland Avenue, SW, Room 7E304
Washington, D.C. 20202

Dear Dr. Mitchell:

This final audit report, “Direct Assessment Programs: Processes for Identifying Risks and
Evaluating Applications for Title IV Eligibility Need Strengthening to Better Mitigate Risks
Posed to the Title IV Programs,” presents the results of our audit.1 The objectives of our audit
were to determine whether the U.S. Department of Education (Department) has addressed the
risks that schools offering direct assessment programs pose to Title IV of the Higher Education
Act of 1965, as amended (Title IV), programs and established processes to ensure that only
programs meeting Federal regulatory requirements are approved as Title IV eligible. We
evaluated the Department’s operations as of January 23, 2014.

We found that the Department did not adequately address the risks that schools offering
direct assessment programs pose to the Title IV programs and did not establish sufficient
processes to ensure that only programs meeting Federal regulatory requirements are approved as
Title IV eligible. Not adequately addressing risks increases the likelihood that schools might
create direct assessment programs that are not Title IV eligible, such as those that are really
correspondence programs. Not establishing sufficient processes to ensure that only programs
meeting Federal regulatory requirements are approved as Title IV eligible increases the risk that
the Department will not obtain enough information to sufficiently evaluate the merits of all
direct assessment program applications. During our audit, we identified two applications for
which the Department could have obtained additional information from the school or the
accrediting agency before making decisions about whether the programs were Title IV-eligible
direct assessment programs.


1
 Throughout this report, we use the term “direct assessment program” to refer to a competency-based education
program that measures a student’s learning through direct assessment, not credit or clock hours.

     The Department of Education’s mission is to promote student achievement and preparation for global competitiveness by fostering
                                           educational excellence and ensuring equal access.
Final Audit Report
ED-OIG/A05N0004                                                                       Page 2 of 20

We provided the draft of this report to the Department for comment. In its comments on the
draft report, the Department did not explicitly agree or disagree with the finding. However, it
provided comments on each of the seven draft audit report recommendations, agreeing with four
(Recommendations 1.1, 1.2, 1.4, and 1.7), partially agreeing with two (Recommendations 1.5
and 1.6), and disagreeing with one (Recommendation 1.3). Although the Department partially
disagreed with two recommendations and disagreed with one recommendation, it proposed
reasonable, alternative corrective actions that should eliminate the causes of the issues that we
reported. We include the Department’s comments on the draft audit report in their entirety as
Attachment 2 to this final audit report.

We considered the Department’s comments and its proposed corrective action plan and
determined that the proposed actions were sufficient to address all seven recommendations.
Where appropriate, we clarified the report and recommendations based on the Department’s
comments. Specifically, we clarified that responsibility for approving direct assessment
programs for Title IV purposes lies solely with the Department. We revised Recommendation
1.2 to make it clear that the recommendation to create and maintain records sufficient to
adequately document the review and approval of direct assessment program applications was
directed toward employees who are responsible for reviewing direct assessment program
applications. We also revised the subsection about involving Federal Student Aid school
participation division managers in the application review process. We changed the
corresponding recommendation (Recommendation 1.3) to emphasize that Federal Student Aid
school participation division managers should be fully informed about, not necessarily actively
involved with, the review of applications, before making decisions about a direct assessment
program’s Title IV eligibility. In addition, we revised Recommendation 1.5, clarifying that
employees should gain an understanding of the processes used by accrediting agencies when
evaluating a school’s application to offer Title IV-eligible direct assessment programs. Finally,
we modified Recommendation 1.6, using the language that the Department proposed because it
better explains that employees should refer any accrediting agency that violates Department
requirements or fails to follow its standards or fulfill its responsibilities to the Office of
Postsecondary Education’s Accreditation Group.



                                      BACKGROUND


Section 8020 of the Higher Education Reconciliation Act of 2005 (HERA) amended the
Higher Education Act of 1965, as amended (HEA), so that students enrolled in direct assessment
programs could receive Title IV of the Higher Education Act of 1965, as amended (Title IV),
funds. The HERA states that instructional programs that use direct assessment instead of credit
or clock hours to measure student learning may qualify as a Title IV-eligible program if the
assessment is consistent with the school’s or program’s accreditation. The HERA also states that
the U.S. Department of Education (Department) must initially determine the Title IV eligibility
of each program for which a school proposes to use direct assessment.

The Department published an interim final rule, effective September 8, 2006, implementing the
HERA provisions. The interim final rule defined a direct assessment program, identified the
information a school must include in its application for the program to be approved as a
Final Audit Report
ED-OIG/A05N0004                                                                          Page 3 of 20

Title IV-eligible program, and limited the use of Title IV funds to learning that results from
instruction that the school provides or oversees. According to 34 Code of Federal Regulations
(C.F.R.) § 668.10,2 direct assessment is a measure—such as a paper, exam, or portfolio—that
shows what a student knows and can do and provides evidence that a student has command of
a specific subject, content area, or skill.

A school that offers a direct assessment program must apply to the Department before the
program can be considered eligible for Title IV purposes. The regulations require the school’s
application to include a description of

               the credential offered and field of study,

               how the assessment of student learning will be done,

               how the program is structured,

               how and when the school determines what each student needs to learn,

               how the school assists students in gaining knowledge needed to pass assessments,

               the number of credit or clock hours to which the program is equivalent, and

               the methodology used to determine credit- or clock-hour equivalencies for the
                program and portions of the program students complete.

In addition, a school must include documentation from its accrediting agency that indicates the
agency evaluated the program and has accredited the program. The school must also provide
documentation showing that the accrediting agency or State licensing body agreed with the
school’s claim of credit- or clock-hour equivalency.

In September 2012, the Department created a direct assessment workgroup (workgroup) to
review schools’ direct assessment program applications and provide technical assistance to
schools and other entities that had questions related to Title IV eligibility for direct assessment
programs. After reviewing a school’s application, the workgroup recommends approval or
denial of the program for Title IV eligibility. The workgroup was also tasked with developing
an application review process to ensure that all schools’ applications are and will be consistently
reviewed. The workgroup consists of employees from the Department’s office of Federal
Student Aid (FSA), Office of the General Counsel, and Office of Postsecondary Education
(OPE). The Department selected the employees based on their prior Title IV experiences and
areas of expertise. According to members of the workgroup, the Department plans to keep the
workgroup in place as long as necessary but will eventually transfer responsibility for reviewing
applications and recommending approval or denial to FSA’s school participation divisions.

Before recommending approval of an application, the workgroup must evaluate whether a school
provided a factual basis for its claim that a direct assessment program is equivalent to a specific
number of credit or clock hours, including how the program meets the minimum requirement for

2
    All regulatory citations are to the July 1, 2013, version.
Final Audit Report
ED-OIG/A05N0004                                                                                           Page 4 of 20

weeks of instructional time.3 According to Title IV requirements, educational activity in
a direct assessment program includes regularly scheduled learning sessions; faculty-guided
independent study; consultations with a faculty mentor; development of an academic action plan
addressed to the competencies identified by the school; or, in combination with any of the
foregoing, assessments. For purposes of direct assessment programs, independent study occurs
when a student follows a course of study with predefined objectives but works with a faculty
member to decide how the student is going to meet those objectives. The student and faculty
member agree on what the student will do (for example, required readings, research, and work
products), how the student’s work will be evaluated, and what the relative timeframe for
completion of the work will be. According to 34 C.F.R. § 668.10(a)(3), “The student must
interact with the faculty member on a regular and substantive basis to assure progress within
the course or program.”

In a Dear Colleague Letter issued on March 19, 2013,4 the Department provided guidance
to schools that want to have their direct assessment programs approved for Title IV eligibility
under the direct assessment regulations. According to the letter, the Department planned
to collaborate with both accrediting agencies and the higher education community to encourage
the use of direct assessment programs, to identify promising practices, and to gather information
to inform future policy. The Department also intended to use what it learned from participating
schools to inform future discussions regarding the reauthorization of the HEA.

As of January 23, 2014, only five schools (Argosy University, Capella University,
Northern Arizona University, Southern New Hampshire University, and University of
Wisconsin Colleges) had submitted direct assessment program applications to the Department.
Capella University and Southern New Hampshire University were the only two schools whose
applications the Department had approved. According to members of the workgroup,
Capella University and Southern New Hampshire University planned to start awarding
Title IV funds to students enrolled in the approved programs in fall 2013 and January 2014,
respectively.

Northern Arizona University submitted an application to the Department in July 2013 but
subsequently withdrew it. (We discuss this situation in the finding under “Credit-Hour
Equivalencies.”) As of January 23, 2014, the Department had not approved the applications
received from Argosy University and University of Wisconsin Colleges.




                                               AUDIT RESULTS


As of January 23, 2014, the Department had not adequately addressed the risks that schools
offering direct assessment programs pose to the Title IV programs and had not established
sufficient processes to ensure that only programs meeting Federal regulatory requirements are
approved as Title IV eligible. Not adequately addressing risks increases the likelihood that
schools might create direct assessment programs that are not Title IV eligible, such as those that

3
    A week of instructional time is defined as any 7-day period in which at least 1 day of educational activity occurs.
4
    GEN-13-10, “Applying for Title IV Eligibility for Direct Assessment (Competency-Based) Programs.”
Final Audit Report
ED-OIG/A05N0004                                                                       Page 5 of 20

are really correspondence programs. Not establishing sufficient processes to ensure that only
programs meeting Federal regulatory requirements are approved as Title IV eligible increases
the risk that the Department will not obtain enough information to sufficiently evaluate
the merits of all direct assessment program applications.

Without a thorough assessment of the specific risks associated with schools offering
direct assessment programs, the Department cannot implement control activities to ensure
management’s directives are carried out and actions are taken to address the risks. If its control
activities are not robust and operating as intended, the Department might approve direct
assessment programs that do not meet Federal regulatory requirements, putting Title IV funds at
risk. The Department also might improperly deny a program, which would restrict students’
access to Title IV funds and keep the Department from achieving its goal of being a supporter
of innovative education methods.

The Department provided an overall comment along with its comments on all seven
recommendations. The Department stated that it established a pilot workgroup to develop
processes and procedures for approving direct assessment program applications because direct
assessment was a new model for both the Department and schools. The Department further
stated that, as is common to many pilot projects, the pilot workgroup’s activities included testing
processes and procedures; therefore, rules and management decisions evolved and will evolve as
experience dictates. Rather than create policies up front, the workgroup was to identify and
resolve principal direct assessment policy issues as they arose. In addition, the workgroup was
to gain sufficient experience with the programs and applications to develop support tools
to ensure consistency in the future review and approval processes that FSA’s school participation
divisions will use.

FINDING — The Department Could Better Manage the Risks That Direct Assessment
          Programs Pose to the Title IV Programs

The Department did not thoroughly assess the risks or develop and implement control activities
sufficient to mitigate the risks that schools offering direct assessment programs pose to the
Title IV programs. Furthermore, the Department did not ensure that it adequately communicated
with and obtained information from accrediting agencies. Not adequately addressing risks
increases the likelihood that schools might create direct assessment programs that are not
Title IV eligible, such as those that are really correspondence programs. Not establishing
sufficient processes to ensure that only programs meeting Federal regulatory requirements are
approved as Title IV eligible increases the risk that the Department will not obtain enough
information to sufficiently evaluate the merits of all direct assessment program applications.

We identified two applications for which the Department could have obtained additional
information from the school or the accrediting agency before making decisions about whether
the programs were Title IV-eligible direct assessment programs. In one instance, the Department
and the accrediting agency arrived at different conclusions about whether the school had
developed the required credit-hour equivalencies for direct assessment programs or was simply
offering credit-hour programs. The Department followed up with the school to obtain more
information but did not contact the accrediting agency to discuss why the agency considered
the school’s programs to be direct assessment. The school withdrew its application before
the Department could determine whether the programs were Title IV-eligible direct assessment
programs. In another instance, the Department did not follow up with either the school or
Final Audit Report
ED-OIG/A05N0004                                                                                  Page 6 of 20

the accrediting agency about the expected level of interaction between students and faculty but
approved the school’s direct assessment program as Title IV eligible, even though the program
might not have satisfied all the requirements in 34 C.F.R. § 668.10.

The Department did not establish an effective system of internal control because, according to
members of the workgroup and FSA’s Chief Risk Officer and Chief Compliance Officer, few
schools have submitted applications to offer direct assessment programs, so the direct assessment
program does not pose a significant risk. According to “Standards for Internal Control in the
Federal Government,”5 management should comprehensively identify risks and then analyze
those risks for their possible effects. Management should then implement sufficient control
activities to ensure that employees carry out management’s directives and address the risks
identified. Examples of control activities include creation and maintenance of relevant records
and appropriate documentation and reviews by management at the functional or activity level.
The standards further state that management should ensure there are adequate means of
communicating with, and obtaining information from, external stakeholders that might have
a significant impact on the Department achieving its goals.

Significant Risks Not Thoroughly Assessed
In its 2014 risk assessment, FSA identified only two risk areas associated with direct assessment
programs: the Department might approve a direct assessment program that should not be
approved, and schools might not implement approved programs in accordance with Title IV
requirements. To mitigate the two identified risk areas, FSA planned to review each school’s
application and have a supervisor review the application file using a checklist to ensure that
the file contained the required documents and information before final approval. In addition,
FSA planned to conduct onsite program reviews of schools with approved programs.

Although the two risk areas that the Department identified are legitimate risk areas and require
mitigation, the areas are too broad to allow the Department to implement specific control
activities to mitigate all significant risks. For example, the two risk areas identified by
the Department do not address the following risks that we considered significant:

           Students might receive Title IV funds for life experience. Title IV funds may be used
            only for learning that results from instruction the school provides or oversees.
            In certain programs, students might receive credit for a course or competency that
            they successfully completed based on their prior knowledge, without engaging
            the learning resources that the school offered.

           A direct assessment program might really be a correspondence program. For a
            school’s direct assessment program to be eligible for Title IV purposes, Department
            regulations require a faculty member to work with a student to design a program of
            study and to interact with the student on a regular and substantive basis. To satisfy
            the requirement for regular interaction with students, a school might use employees
            who are not faculty members. Because direct assessment programs are self-paced,
            the school might not require regular and substantive interaction between students and
            faculty members. Therefore, the programs might actually be correspondence
            programs. Students enrolled in programs that are improperly defined as direct

5
 U.S. Government Accountability Office, “Standards for Internal Control in the Federal Government,”
GAO/AIMD-00-21.3.1, November 1999.
Final Audit Report
ED-OIG/A05N0004                                                                         Page 7 of 20

           assessment programs will receive more aid than they are allowed because cost of
           attendance for students enrolled in correspondence programs is generally limited
           to tuition and fees. Additionally, a school might be ineligible to receive Title IV
           funds if it offers more than 50 percent of its courses by correspondence or if it enrolls
           50 percent or more of its students in correspondence courses.

          A school might develop credit- or clock-hour equivalencies for the program that are
           not based on the regulatory definition of a credit or clock hour. Schools must equate
           the length of their direct assessment programs to credit or clock hours using the
           definition of credit or clock hours established by Title IV regulations. This
           equivalency helps determine whether the program meets the minimum Title IV
           requirements for an academic year and is used as the basis for defining a payment
           period and calculating the amount of Title IV awards. Direct assessment programs
           with improperly calculated credit- or clock-hour equivalencies could result in students
           receiving more Title IV funds than allowed.

Sufficient Control Activities Not Developed and Implemented
As of January 23, 2014, we identified two significant control activity weaknesses directly related
to the Department’s direct assessment program application review process. Specifically,
the Department had not created and maintained documentation of relevant discussions the
workgroup held and conclusions it reached during the review of applications and had not ensured
that FSA school participation division managers were kept fully informed of issues raised during
the application review process.

Not Documenting the Basis for Approval or Denial of an Application
At the time of our audit, the workgroup did not create or maintain records of relevant discussions
it held, conclusions it reached, and common issues it identified during the reviews of
applications. These records could provide valuable information to guide future decisions about
the Title IV eligibility of direct assessment programs or the process the Department could use for
future reviews of applications. Such historical records would be especially helpful as the
employees reviewing applications change over time. According to the Department’s
Administrative Communications System Departmental Directive, “Records and Information
Management Program,” OM: 6-103, the Department must create and maintain official records
that are sufficient to adequately document all of its functions, policies, decisions, procedures, and
essential transactions.

Since we completed our fieldwork, the workgroup told us that it has started to retain
documentation of the questions and answers the workgroup sends back and forth with schools
during the application review process, and the workgroup will begin maintaining records of
its discussions and conclusions.

FSA School Participation Division Managers Not Fully Informed of Issues Raised During
the Application Review Process
FSA school participation division managers are responsible for approving the Title IV eligibility
of direct assessment programs. However, FSA school participation division managers rely on
the workgroup to review applications and provide recommendations regarding approval or
denial. Through interviews with FSA school participation division managers, we learned that
they did not receive specific information about issues that the workgroup identified during its
review of direct assessment program applications. The FSA school participation division
Final Audit Report
ED-OIG/A05N0004                                                                      Page 8 of 20

managers received information only about the progress of the review and notification when
the workgroup recommended approval. The FSA school participation division managers then
approved the programs without being fully informed about the issues raised during the
workgroup’s review of the application or knowing how the workgroup resolved any issues. If
the FSA school participation division managers were kept informed about issues identified
during the review of applications and how any issues were resolved, they would be in a better
position to make well-informed approval decisions. They also would gain experience that will
be necessary to perform a thorough review of applications once the workgroup is disbanded and
full responsibility for approving direct assessment programs is turned over to FSA’s school
participation divisions.

Communication With External Entities Involved in the Application Process Not Adequate
to Make Well-Informed Decisions
The workgroup has not communicated with accrediting agencies to determine the agencies’
standards for approvals of direct assessment programs—specifically, the agencies’ evaluations of
credit-hour equivalencies and faculty involvement. An accrediting agency is an external
stakeholder that has a significant impact on a school’s application to have its direct assessment
programs approved for Title IV eligibility. The accrediting agency evaluates the school’s
programs and must include the programs as part of the school’s accreditation. In addition,
the accrediting agency must indicate agreement with the school’s claim of the direct assessment
program’s equivalence in terms of credit or clock hours for Title IV purposes.

During our audit, we identified two situations, one involving credit-hour equivalencies and
one involving student-faculty interaction, in which the workgroup could have communicated
with and obtained additional information from accrediting agencies. Although approval of direct
assessment programs for Title IV purposes lies solely with the Department, communication with
accrediting agencies about credit-hour equivalencies and student-faculty interaction would
ensure that the workgroup has sufficient information to make well-informed decisions about
the Title IV eligibility of direct assessment programs.

Credit-Hour Equivalencies
For one school’s application, the Department and the accrediting agency arrived at different
conclusions about whether the school had developed the required credit-hour equivalencies for
direct assessment programs or was simply offering credit-hour programs. The accrediting
agency concluded that the school was offering direct assessment programs. However, the
Department concluded that the programs as described in the application appeared to be
credit-hour programs, not direct assessment programs, and asked the school for additional
information. The Department did not contact the accrediting agency to discuss why the agency
considered the school’s programs to be direct assessment.

Without gaining an understanding of how and why the accrediting agency made its
determination, the Department cannot be sure that it has made the correct decision to approve or
deny the school’s application or whether the accrediting agency is a reliable authority for making
Title IV-related direct assessment determinations. In this case, the school withdrew its
application based on questions posed by the Department. Because the school’s application
contained information that caused the Department to question whether the programs were
direct assessment, the Department could have contacted the accrediting agency to obtain
information about the accrediting agency’s conclusion that the school’s programs were direct
assessment. It is essential for the Department to communicate with accrediting agencies as well
Final Audit Report
ED-OIG/A05N0004                                                                        Page 9 of 20

as the schools. Such communication will help the Department make a better-informed decision
about whether a program is a direct assessment program or a credit- or clock-hour program and
to ensure that only students enrolled in Title IV-eligible programs receive Title IV funds.
Further, if communications with the accrediting agency lead one component of the Department
to question the reliability of the agency’s assessment of a school’s program, that component
should raise its concerns with OPE’s Accreditation Group. Those concerns might include an
agency violating the Department’s requirements for recognition; the accrediting agency failing to
follow its own standards, policies, or procedures; or the accrediting agency not fulfilling its
responsibilities under the Title IV programs and approving a school’s involvement of faculty
below a level generally accepted in the higher education community or unreasonable
methodology for establishing credit- or clock-hour equivalencies.

Student-Faculty Interaction
The Department approved another school’s application to operate a Title IV-eligible direct
assessment program even though it did not have sufficient information to determine whether the
program met the requirement for the student to interact with a faculty member on a regular and
substantive basis to ensure progress within the course or program (34 C.F.R. § 668.10(a)(3)(iii)).
Rather than communicating with and obtaining additional information from the school and
accrediting agency, the workgroup chose to rely on the accrediting agency’s determination that
the program was direct assessment and met the requirement for student-faculty interaction.

As described in the school’s application, the program was to meet the requirement for weeks of
instructional time through the use of faculty-guided independent study. The application stated
that the program would use a “coach” to guide the student’s independent study, and the
application’s glossary defined a coach as “a trained professional, typically with counseling or
coaching experience, who works with the student to establish goals and set pace and who asks
questions and recommends resources or support tools, as necessary.” The application did not
state that the coach was a faculty member or that a faculty member would guide students through
independent study; the glossary was silent on the terms “faculty” and “faculty member.”
The application also did not state whether the coach was required to have subject matter
expertise in the area the student was pursuing through independent study.

According to the accrediting agency’s standards, the responsibilities of teaching faculty include
instruction and the systematic understanding of effective teaching and learning processes and
outcomes in courses and programs for which they share responsibility. Additional duties may
include functions such as student advisement and academic planning. The accrediting agency’s
standards also explain that faculty must be allowed adequate time to provide effective
instruction; advise and evaluate students; contribute to program and institutional assessment and
improvement; continue professional growth; and participate in scholarship, research, creative
activities, and service compatible with the mission and purposes of the institution. The
accrediting agency’s definition of faculty and the definition of a coach in the school’s application
did not match.

The workgroup discussed whether the position of coach, as described in a school’s application,
met the requirement for the student to interact with the faculty member on a regular and
substantive basis. The workgroup decided that it would rely on the approval of the program
given by the accrediting agency. We saw no evidence in the Department’s records indicating
that the workgroup sought additional information from the accrediting agency, even though the
Final Audit Report
ED-OIG/A05N0004                                                                     Page 10 of 20

accrediting agency documentation that the school included with its application did not mention
any evaluation of faculty.

Reliance on the accrediting agency is not sufficient to evaluate whether a school is in compliance
with Title IV requirements. The regulations might be interpreted in different ways by different
entities. For example, not every entity will have the same interpretation of the specific level of
faculty involvement needed to satisfy the regular and substantive interaction requirement in
34 C.F.R. § 668.10(a)(3)(iii). If the Department does not communicate with and obtain
sufficient information from the accrediting agency, it cannot fully understand how the agency
evaluated a school’s direct assessment program, what standards the agency used, what aspects of
the program the agency looked at, or how the agency reached its conclusions about the program.
Therefore, it is incumbent on the Department to obtain sufficient information to make a
well-informed determination about whether a direct assessment program meets Title IV
requirements, including the requirement that students interact with faculty members on a regular
and substantive basis. In this case, the Department decided not to make its own determination
about whether the school provided regular and substantive interaction with a faculty member.
Without such interaction, a program might not be Title IV eligible. Or, the lack of interaction
with a faculty member could make the program a correspondence program and could affect
the school’s determination of enrollment status, student attendance, and other Title IV-related
areas.

Title IV Funds Are at Risk
We agree with the Department that few schools offering direct assessment programs have
applied to have their programs deemed Title IV eligible; so, the amount of Title IV funds
currently at risk is relatively low. However, the program eligibility decisions the Department is
making about these early-implementing schools could set a precedent for future direct
assessment programs and have a lasting, negative impact on the Title IV programs. Approving
programs that do not meet the Federal regulatory requirements puts Title IV funds at risk.
Without documentation that is sufficient and appropriate for making decisions and clear and
continuous communication between all involved with the application review and approval
processes, the Department cannot effectively mitigate the risks posed to the Title IV programs
and ensure that it approves only applications meeting Federal regulatory requirements as Title IV
eligible.

For example, because a program’s credit- or clock-hour equivalency is used as the basis for
defining payment periods, a school that improperly establishes the equivalency might improperly
define its payment periods. If the Department approves such a program, Title IV-related areas,
such as disbursements and the determination of satisfactory academic progress, will be affected.
Additionally, schools might use activities that are not academic in nature as evidence that
students began, continued, or ceased attending. For instance, if the Department approves
a program that uses coaches as faculty, the school might assume that it can accept a student’s
discussion with a coach as evidence of attendance even though that discussion might not pertain
to the subject matter of a particular course or competency. Improperly determining student
attendance might cause a school to incorrectly determine a change in enrollment status,
satisfactory academic progress, disbursements, and return of Title IV aid calculations.

Additionally, without appropriate documentation and communication, the Department might
improperly deny a school’s application. Improperly or inadvertently denying a program that
Final Audit Report
ED-OIG/A05N0004                                                                       Page 11 of 20

meets Title IV requirements restricts students’ access to Title IV funds and could keep
the Department from achieving its goal of being a supporter of innovative education methods.

Recommendations

We recommend that the Under Secretary—

1.1    Reassess the risks that direct assessment programs pose to the Title IV programs,
       communicate the results of that risk assessment to Department employees, and develop
       additional control activities to mitigate any newly identified risks.

1.2    Require employees responsible for reviewing direct assessment program applications
       to create and maintain records that are sufficient to adequately document all of
       the functions, procedures, and decisions that are relevant to the review and approval of
       direct assessment program applications.

1.3    Ensure that FSA school participation division managers responsible for approving
       direct assessment programs are fully informed about the issues raised during
       the workgroup’s review of school applications.

1.4    Require employees involved in the review and approval of applications to obtain
       adequate evidence to support their conclusions about a school’s compliance with
       the direct assessment program requirements.

1.5    Require employees to gain an understanding of the processes that each accrediting
       agency used to evaluate a school’s offering of direct assessment programs, including the
       level of student-faculty interaction and the methodology used to evaluate credit- or
       clock-hour equivalencies.

1.6    Require employees to refer to OPE’s Accreditation Group any accrediting agency that
       (a) violates the Department’s requirements for recognition; (b) fails to follow accrediting
       agency standards, policies, or procedures; or (c) fails to fulfill its responsibilities under
       the Title IV programs, such as approving a level of faculty involvement below that
       generally accepted in the higher education community or approving an unreasonable
       methodology for establishing credit- or clock-hour equivalencies.

1.7    Develop guidance on how schools meet the Federal regulatory requirement that
       direct assessment programs include regular and substantive interaction between students
       and faculty.

Department Comments

The Department agreed with Recommendation 1.1, stating that it considered the risks that it has
identified to date to be comprehensive in nature. The Department added that it has been aware of
the additional risks that the Office of Inspector General cited in the draft audit report and will
have FSA reassess the risks that direct assessment programs pose to the Title IV programs;
communicate the results of that risk assessment to employees; and develop additional control
activities, as applicable, to mitigate any newly identified risks. In addition, the Department
stated that, as it reviews additional applications, FSA will continue to evaluate potential risks. If
Final Audit Report
ED-OIG/A05N0004                                                                     Page 12 of 20

it identifies any additional risks, FSA will provide that information to employees and develop
control activities to mitigate the newly identified risks.

The Department agreed with Recommendation 1.2, stating that the workgroup has been
recording its activities and retaining documentation on the workgroup’s SharePoint site. In
addition, FSA will develop procedures for creating and maintaining records and will develop
application review guidelines and a checklist for approval of applications based on the statutory
and regulatory requirements for direct assessment.

The Department disagreed with Recommendation 1.3, stating that FSA school participation
division managers are responsible for approving thousands of eligibility, compliance audit,
financial statement, and program review actions every year and, therefore, cannot be “actively”
involved in the review of every single one of these actions. For direct assessment program
applications, the Department established a workgroup with subject matter experts. The
workgroup is responsible for reviewing each application and recommending to the respective
manager whether to approve or deny the application. FSA stated that as part of the procedures
and review guidelines developed in response to Recommendation 1.2, it will document its
practice that the workgroup include an employee from the respective school participation
division as a member of the workgroup while a direct assessment application from the region is
being reviewed. The Department also proposed that the workgroup provide a copy of the
application along with copies of the questions and answers exchanged between the workgroup
and the school. In addition, the workgroup will provide to the school participation division its
recommendation, including any issues, concerns, or other information to assist the FSA school
participation division managers and analysts in making a final decision on the direct assessment
application.

The Department agreed with Recommendation 1.4, emphasizing that FSA has always required
employees to obtain adequate evidence to support their conclusions about a school’s compliance
with the direct assessment program requirements. The Department further stated that the
application review guidelines that will be developed in response to Recommendation 1.2 will be
amended, as needed, when new issues are noted. The Department also informed us that a
member of the workgroup provided training on direct assessment to school participation division
managers and appropriate headquarters employees on July 8, 2014.

The Department partially agreed with Recommendation 1.5, stating that the workgroup should
look beyond accrediting agency approval when reviewing direct assessment programs. Such
review should include looking at the level of faculty involvement and the methodology used for
evaluating credit- and clock- hour equivalencies. The Department also stated that the application
review guidelines developed in response to Recommendation 1.2 will include information on this
topic.

The Department partially agreed with Recommendation 1.6, stating that approval of a direct
assessment program lies solely with the Department. The Department further stated that FSA
will include in its application review guidelines a statement that the workgroup will refer any
violations by an accrediting agency of the requirements for recognition, agency standards,
policies, or procedures, and accrediting agency responsibilities under the Federal student aid
programs, including any agency approval of faculty involvement below that generally accepted
in the higher education community, and any agency approval of an unreasonable methodology
Final Audit Report
ED-OIG/A05N0004                                                                         Page 13 of 20

for establishing credit or clock hour equivalencies, to OPE’s Accreditation Group, through
the workgroup members who work in the Accreditation Group.

The Department agreed with Recommendation 1.7, stating that OPE will publish a
Dear Colleague Letter to provide information on the differences between direct assessment and
credit- or clock-hour competency-based education. The Dear Colleague Letter will also provide
guidance on how schools can meet the Federal regulatory requirement that direct assessment
programs include regular and substantive interaction between students and faculty.

Office of Inspector General Response

Because we agree with the Department’s proposed corrective action(s) for draft audit report
Recommendations 1.2, 1.4, and 1.7, we do not provide specific responses to the Department’s
comments for each of those recommendations.

We agree with the Department’s proposed corrective actions for Recommendation 1.1 that it
should reassess the risks that direct assessment programs pose to the Title IV programs,
communicate the results of the risk assessment to employees, and develop additional control
activities to mitigate any newly identified risks. However, we believe that the Department still
needs to specifically address the three significant risks that we identified in the draft audit report:
(1) students might receive Title IV funds for life experience, (2) a direct assessment program
might really be a correspondence program, and (3) a school might develop credit- or clock-hour
equivalencies for the program that are not based on the regulatory definition of a credit or
clock hour.

We acknowledge that FSA school participation division managers cannot be actively involved
with every action for which he or she is responsible and agree with the Department’s proposed
corrective actions for Recommendation 1.3. Therefore, we revised the subsection and
corresponding recommendation to reflect that FSA school participation division managers should
be fully informed of issues raised during the review of direct assessment program applications,
not necessarily actively involved with the review process.

We agree with the Department’s proposed corrective action that the workgroup continue to look
beyond accrediting agency approvals and look at the level of faculty involvement and the
methodology used for evaluating equivalencies. However, the Department also needs to
consider the role of the accrediting agency when evaluating a school’s direct assessment program
application. As part of the application package, the Department requires a school to submit
documentation from its accrediting agency indicating that the agency has evaluated the school’s
offering of direct assessment program(s), has included the program(s) in the school’s grant of
accreditation, and indicated agreement with the school’s claim of the direct assessment
program’s equivalence in terms of credit or clock hours. We revised Recommendation 1.5 to
clarify that employees need to gain an understanding of the processes used by accrediting
agencies when evaluating a school’s application to offer Title IV-eligible direct assessment
programs, not that employees should evaluate the sufficiency of accrediting agencies’ approvals
of direct assessment programs.

Finally, we agree that approval of a school’s application to offer a direct assessment program for
Title IV purposes lies solely with the Department. We also agree with the Department’s
proposed corrective action for Recommendation 1.6. Accordingly, we revised the
Final Audit Report
ED-OIG/A05N0004                                                                       Page 14 of 20

recommendation, using the Department’s proposed language because it better explains that
employees should refer any accrediting agency that violates Department requirements or fails to
follow its standards or fulfill its responsibilities to the OPE’s Accreditation Group.




                  OBJECTIVES, SCOPE, AND METHODOLOGY


The objectives of our audit were to determine whether the Department had addressed the risks
that schools offering direct assessment programs pose to the Title IV programs and established
processes to ensure that only programs meeting the Federal regulatory requirements were
approved as Title IV eligible. We evaluated the Department’s operations as of January 23, 2014.
The purpose of this report is to assist the Department in taking a proactive approach to
identifying and mitigating the unique risks that schools offering direct assessment programs
could pose to the Title IV programs.

To achieve our objectives, we gained an understanding of selected provisions of the HEA,
the HERA, Title IV regulations, and the Department’s guidance, policies, procedures, and
practices that were in place as of January 23, 2014. We also reviewed and gained an
understanding of the “Standards for Internal Control in the Federal Government,” and used
the standards as criteria for evaluating the Department’s system of internal control related to
direct assessment programs.

To understand how the Department assessed risk and handled the review and approval of
direct assessment program applications, we conducted interviews with workgroup members from
FSA, the Office of the General Counsel, and OPE; officials from FSA’s Program Compliance
and Risk Management offices; an FSA school participation division employee who was involved
with the application review process for one school; and FSA school participation division
managers who were responsible for approving two applications. To help us understand
management’s attitude toward direct assessment programs and the risks associated with them,
we also spoke to the former Acting Assistant Secretary for Postsecondary Education before his
departure from the Department. Additionally, we reviewed copies of correspondence among
members of the workgroup, FSA school participation division teams, and schools regarding
the review of the applications. We also reviewed applications submitted to the Department for
consideration by Capella University, Northern Arizona University, and Southern New
Hampshire University—three of the five schools that had applied as of January 23, 2014—and
compared them with the requirements set forth in 34 C.F.R. § 668.10. Finally, we reviewed and
analyzed the Department’s comments on the draft of this report and revised the report and
recommendations as necessary.

We conducted our audit from March 2013 through January 2014 at the Department’s offices in
Washington, D.C., and our offices in Chicago, Illinois, and Kansas City, Missouri. We
conducted interviews with Department employees in regional offices by teleconference.
We discussed the results of our work with Department officials on August 29, 2013, and
May 20, 2014.
Final Audit Report
ED-OIG/A05N0004                                                                     Page 15 of 20

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objectives. We believe that the evidence obtained provides a reasonable basis
for our finding and conclusions based on our audit objectives.




                            ADMINISTRATIVE MATTERS


Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department of
Education officials.

Corrective actions proposed (resolution phase) and implemented (closure phase) by your office
will be monitored and tracked through the Department’s Audit Accountability and Resolution
Tracking System (AARTS). Department policy requires that you develop a final corrective
action plan (CAP) for our review in the automated system within 30 days of the issuance of this
report. The CAP should set forth the specific action items, and targeted completion dates,
necessary to implement final corrective actions on the finding and recommendations contained in
this final audit report.

In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector
General is required to report to Congress twice a year on the audits that remain unresolved after
6 months from the date of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us during this review. If you have any questions or require
additional information, please do not hesitate to contact me at (202) 245-6900 or Gary D.
Whitman, Regional Inspector General for Audit, at (312) 730-1620.

Sincerely,

/s/

Patrick J. Howard
Assistant Inspector General for Audit
Final Audit Report
ED-OIG/A05N0004                                                                     Page 16 of 20

                                                                                    Attachment 1



             Acronyms, Abbreviations, and Short Forms Used in this Report

C.F.R.                 Code of Federal Regulations

Department             U.S. Department of Education

Direct Assessment      Competency-based education program that measures a student’s
Program                learning through direct assessment, not credit or clock hours

FSA                    Federal Student Aid

HEA                    Higher Education Act of 1965, as amended

HERA                   Higher Education Reconciliation Act of 2005

OPE                    Office of Postsecondary Education

Title IV               Title IV of the Higher Education Act of 1965, as amended

Workgroup              Direct assessment workgroup created by the Department to review
                       schools’ applications and provide technical assistance to schools and
                       other entities that had questions related to Title IV eligibility for direct
                       assessment programs
Final Audit Report
ED-OIG/A05N0004                                                                    Page 17 of 20

                                                                                   Attachment 2

MEMORANDUM


DATE:          July 17, 2014

TO:            Gary D. Whitman
               Regional Inspector General for Audit Services
               Office of Inspector General

               Pat Howard
               Assistant Inspector General for Audit Services
               Office of Inspector General

FROM:          Ted Mitchell
               Under Secretary

SUBJECT:       Draft Audit Report
               Direct Assessment Programs: Processes for Identifying Risks and Evaluating
               Applications for Title IV Eligibility Need Strengthening to Better Mitigate Risks
               Posed to the Title IV Programs
               Control No. ED-OIG/A05N0004


Thank you for the opportunity to comment on the Office of Inspector General’s (OIG) draft audit
report, Direct Assessment Programs: Processes for Identifying Risks and Evaluating
Applications for Title IV Eligibility Need Strengthening to Better Mitigate Risks Posed to the
Title IV Programs, dated June 6, 2014. The objectives of the audit were to determine whether the
U.S. Department of Education has addressed the risks that schools offering direct assessment
programs pose to Title IV of the Higher Education Act of 1965 (as amended) programs and
established processes to ensure that only programs meeting Federal regulatory requirements are
approved as Title IV eligible.

This Memorandum provides the Department’s comments and responses to the seven (7)
recommendations in OIG’s draft audit report. Federal Student Aid, the Office of Postsecondary
Education, and the Office of the Under Secretary collaborated to provide the comments in this
Memorandum.

The Department has created a Direct Assessment Workgroup (Workgroup) to review direct
assessment program applications and to provide technical assistance to schools and other entities
that have questions related to direct assessment. The Workgroup is comprised of subject matter
experts from FSA’s office of Program Compliance, OPE, and the Office of General Counsel.

As of January 23, 2014, only five applications for direct assessment programs had been received,
and only two had been approved. The Workgroup’s authority and responsibility is to provide
recommendations to School Participation Divisions as to whether an application for the
Department’s approval of a direct assessment program satisfies the criteria in 34 C.F.R. 668.10.
Final Audit Report
ED-OIG/A05N0004                                                                     Page 18 of 20

Memorandum from Mitchell to Whitman and Howard
Response to Draft Audit Report, Direct Assessment
Page 2 of 4

In addition, the Workgroup intends to identify and resolve principal direct assessment policy
issues as they arise and to gain sufficient experience with these programs and applications to
develop support tools to ensure consistency in the review/approval process by FSA’s regional
staff.

Because determining Title IV eligibility for direct assessment programs is a new model for both
institutions and the Department, the Department chose to establish a pilot Workgroup to develop
processes and procedures for approving direct assessment programs. As is common to many pilot
projects, activities include testing processes/procedures, and rules and management decisions
evolve as experience dictates.

FINDING — The Department Could Better Manage the Risks That Direct Assessment
Programs Pose to the Title IV Programs

RECOMMENDATION 1.1 — Reassess the risks that direct assessment programs pose to
the Title IV programs, communicate the results of that risk assessment to Department
employees, and develop additional control activities to mitigate any newly identified risks.

Although we consider the risks identified to date to be comprehensive in nature, and have been
aware of the two more specific risks cited in the draft report since inception, FSA will reassess
the risks that direct assessment programs pose to the Title IV programs, communicate the results
of that risk assessment to employees, and develop additional control activities as applicable to
mitigate any newly identified risks. Also, as additional applications are reviewed, FSA will
continue to evaluate potential risks and, if any are identified, provide that information to
employees and develop control activities to mitigate the newly identified risks.

RECOMMENDATION 1.2 — Require employees to create and maintain records that are
sufficient to adequately document all of the functions, procedures, and decisions that are
relevant to the review and approval of applications.

We agree with this recommendation. The Direct Assessment Workgroup has been recording the
activities of the group and retaining this documentation on the Workgroup’s SharePoint site.
FSA will develop procedures for creating and maintaining records and will develop application
review guidelines and a checklist for approval of applications based on the statutory and
regulatory requirements for direct assessment and will use that information in the review process.

RECOMMENDATION 1.3 — Ensure that regional FSA managers responsible for
approving direct assessment programs are actively involved in the review of school
applications.

We disagree with this recommendation. FSA managers are responsible for approving thousands
of eligibility, compliance audit, financial statement, and program review actions every year.
They cannot be “actively” involved in the review of every single one of these actions. Rather,
FSA has subject matter experts who do the reviews and recommend approval or disapproval to
Final Audit Report
ED-OIG/A05N0004                                                                      Page 19 of 20

Memorandum from Mitchell to Whitman and Howard
Response to Draft Audit Report, Direct Assessment
Page 3 of 4

the managers. For direct assessment applications, a Workgroup was established with subject
matter experts from across the Department, with the responsibility to review and make
recommendations to the respective manager on approval or denial of the application.

In addition, a member of FSA’s New York/Boston School Participation Division (SPD) and a
member of the Chicago/Denver School Participation Division were in close contact with the
Workgroup when their respective school applications were under review, and were fully
informed as to any issues. All issues were resolved prior to the Workgroup decision to
recommend approval to the SPD. As part of the SPD approval process, a Team meeting is held to
discuss the school's application. This discussion includes all relevant data and information about
the school’s current status, e.g., funding, default rates, financial scores, audit findings,
accreditation, licensing, complaints, eligibility, media reports, etc. This discussion included any
issues from the direct assessment Workgroup process.

That said, as part of the procedures and review guidelines developed in response to
Recommendation 1.2, FSA will document its consistent practice that the Workgroup include a
staff member from the respective SPD as a member of the Workgroup while a direct assessment
application from the region is being reviewed. FSA will also include in the procedures that the
Workgroup will provide a copy of the application and copies of the questions/answers exchanged
between the workgroup and the institution. In addition, the Workgroup will provide to the SPD
its recommendation, as well as any issues, concerns, or other information to assist the SPD
manager and analysts in making a final decision on the direct assessment application.

RECOMMENDATION 1.4 — Require employees involved in the review and approval of
applications to obtain adequate evidence to support their conclusions about a school’s
compliance with the direct assessment program requirements.

We agree with this recommendation and, in fact, FSA always required employees to obtain
adequate evidence to support their conclusions about a school’s compliance with the direct
assessment program requirements. The regulations clearly state what institutions are required to
provide as part of the application process. The application review guidelines that will be
developed in response to Recommendation 1.2 will be amended and revised as the Workgroup
reviews new applications and notes new issues. The Workgroup has expanded the review process
as it has learned more about the varied ways that direct assessment programs are being
implemented. In addition, training on direct assessment was provided to the regional managers
and appropriate headquarters staff on July 8, 2014, by a member of the Direct Assessment
Workgroup.

RECOMMENDATION 1.5 — Require employees to evaluate the sufficiency of accrediting
agencies’ approvals of direct assessment programs, including the level of faculty
involvement in the program and the methodology used for evaluating the credit- or clock-
hour equivalencies.
Final Audit Report
ED-OIG/A05N0004                                                                      Page 20 of 20

Memorandum from Mitchell to Whitman and Howard
Response to Draft Audit Report, Direct Assessment
Page 4 of 4

We agree that the Workgroup should continue to look beyond accrediting agency approvals in
reviewing direct assessment programs, including looking at the level of faculty involvement and
the methodology used for evaluating equivalencies. The application review guidelines developed
in response to Recommendation 1.2 will include information on this topic.

RECOMMENDATION 1.6 — Require employees to refer to the Office of Postsecondary
Education’s Accreditation Group any accrediting agency that they believe may have
incorrectly concluded that a school is offering a direct assessment program, particularly
when the decision pertains to evaluating the level of faculty involvement or the
methodology used for establishing credit- or clock-hour equivalencies.

We disagree with this recommendation to the extent it indicates that determining whether a
program is a “direct assessment” program is an accrediting agency function. We have been
advised by the Office of General Counsel that this is a determination to be made by the
Department. FSA will include in the application review guidelines a statement that the
Workgroup will refer any violations by an accrediting agency of the requirements for
recognition, agency standards, policies, or procedures, and/or accrediting agency responsibilities
under the federal student aid programs, including any agency approval of faculty involvement
below that generally accepted in the higher education community, and any agency approval of an
unreasonable methodology for establishing credit or clock hour equivalencies, to OPE’s
Accreditation Group, through the Workgroup members who work in the Accreditation Group.

RECOMMENDATION 1.7 — Develop guidance on how schools meet the Federal
regulatory requirement that direct assessment programs include regular and substantive
interaction between students and faculty.

OPE will publish a Dear Colleague Letter (DCL) to provide information on the differences
between direct assessment and credit or clock hour competency-based education. The DCL will
also provide guidance on how schools can meet the Federal regulatory requirement that direct
assessment programs include regular and substantive interaction between students and faculty. In
addition, OPE has developed and scheduled training for accreditors on direct assessment to
address this finding.

Again, thank you for the opportunity to comment on the draft audit report. If you have further
questions, please feel free to contact Dawn Dawson, the audit liaison officer for this audit, at
(202) 377-3468 or by email at dawn.dawson@ed.gov.