oversight

The Higher Learning Commission Could Improve Its Evaluation of Competency-Based Education Programs to Help the Department Ensure the Programs Are Properly Classified for Title IV Purposes

Published by the Department of Education, Office of Inspector General on 2015-09-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                               UNITED STATES DEPARTMENT OF EDUCATION
                                           OFFICE OF INSPECTOR GENERAL

                                                                                                           AUDIT SERVICES
                                                                                              Chicago/Kansas City Audit Region

September 30, 2015

                                                                                                          Control Number
                                                                                                          ED-OIG/A05O0010


Dr. Barbara Gellman-Danley
President
The Higher Learning Commission
230 South LaSalle Street, Suite 7-500
Chicago, Illinois 60604

Dear Dr. Gellman-Danley:

This final audit report, “The Higher Learning Commission Could Improve Its Evaluation of
Competency-Based Education Programs to Help the Department Ensure the Programs Are
Properly Classified for Title IV Purposes,” presents the results of our audit. The objective of our
audit was to determine whether the Higher Learning Commission established a system of
internal control that provided reasonable assurance that schools’ classifications of delivery
methods and measurements of student learning for competency-based education programs,
including direct assessment programs, were sufficient and appropriate to help the
U.S. Department of Education (Department) ensure that it properly classified the schools’
programs for Title IV of the Higher Education Act of 1965, as amended (Title IV), purposes. 1

We evaluated operations related to the Higher Learning Commission’s reviews of competency-
based education programs, including direct assessment programs, as of December 31, 2014. We
concluded that the Higher Learning Commission did not establish a system of internal control
that provided reasonable assurance that schools’ classifications of delivery methods and
measurements of student learning for competency-based education programs, including direct
assessment programs, were sufficient and appropriate to help the Department ensure that it
properly classified the schools’ programs for Title IV purposes. Specifically, we found that
the Higher Learning Commission did not

        •    consistently apply its standards when reviewing competency-based education
             programs, including direct assessment programs, and determining the proposed
             programs’ delivery methods and measurements of student learning;


1
 Throughout this report, we use the term “direct assessment program” to refer to a competency-based education
program that measures a student’s learning through direct assessment, not credit or clock hours.

     The Department of Education’s mission is to promote student achievement and preparation for global competitiveness by fostering
                                           educational excellence and ensuring equal access.
Final Audit Report
ED-OIG/A05O0010                                                                           Page 2 of 33
           •    obtain sufficient information to determine the delivery methods and measurements
                of student learning of proposed competency-based education programs that it
                determined were not direct assessment programs;

           •    sufficiently evaluate the credit-hour equivalencies of direct assessment programs; or

           •    maintain complete records supporting its decisions regarding the credit-hour
                equivalencies of direct assessment programs.

Schools must obtain approval for direct assessment programs from the Department before they
may provide Title IV funds to students who enroll in those programs. Title 34 Code of Federal
Regulations (C.F.R.) Section (§) 668.10 requires a school to include in its direct assessment
program application to the Department (1) documentation from the school’s accrediting agency
indicating that the accrediting agency has evaluated the school’s offering of the direct assessment
program and has included the program in the school’s grant of accreditation and
(2) documentation from the accrediting agency indicating that the accrediting agency agreed with
the school’s claim of the direct assessment program’s equivalency in terms of credit or
clock hours. 2

Without an appropriate evaluation by the Higher Learning Commission of the classification of
proposed competency-based education programs, the Department might not receive sufficient
information about a school’s proposed competency-based education programs, including
direct assessment programs, to make fully informed decisions about the Title IV eligibility of the
programs. Further, because of the limits that Title IV places on programs offered by
correspondence, weaknesses in the Higher Learning Commission’s review process that result in
schools’ misclassifying competency-based education programs offered by correspondence as
competency-based education credit-hour or direct assessment programs offered by distance
education could result in overpayments of Title IV funds to students or disbursements of Title IV
funds to students enrolled in ineligible programs. Additionally, if the Higher Learning
Commission does not properly determine the type of programs schools are offering, it might not
be able to meet all of the Department’s criteria for recognition of accrediting agencies.

Finally, if the Higher Learning Commission does not effectively and consistently apply its
policies, it might create an atmosphere of uncertainty among schools planning to develop
competency-based education programs, including direct assessment programs. As a result,
schools might forgo plans to create innovative and effective competency-based education
programs that comply with Title IV requirements and meet the needs of students.

The Higher Learning Commission did not consistently apply its standards for reviewing
competency-based education programs and determining the proposed programs’ delivery
methods and measurements of student learning because its policies and procedures for
substantive change applications needed strengthening. All Higher Learning Commission
peer reviewers used a standard form to document their evaluations of and recommendations to
approve or deny applications for direct assessment substantive changes from schools. The form

2
    All regulatory citations are to the July 1, 2014, version.
Final Audit Report
ED-OIG/A05O0010                                                                                        Page 3 of 33
that the peer reviewers used did not include information specific to the level of interaction
between faculty and students, and the substantive change applications for direct assessment
programs did not clearly indicate that the interaction would be regular and substantive. Also,
Higher Learning Commission policy did not require vice presidents for accreditation relations
(staff liaisons) to determine whether a school’s initial offering of a competency-based education
credit-hour program represented a significant departure from the school’s existing education
program offerings and needed to be reviewed as a substantive change pursuant to 34 C.F.R.
§ 602.22(a)(2)(iii). 3 Additionally, Higher Learning Commission staff liaisons, who were
responsible for determining whether proposed programs were direct assessment programs, did
not ensure that they consistently applied Higher Learning Commission policies because they did
not meet with each other to discuss proposed programs or monitor decisions that they had made
independently. Finally, according to its policy, the Higher Learning Commission reviewed
whether a school’s assignment of credit hours met the Federal definition only during
comprehensive reviews for reaffirmation of accreditation, which occurred as infrequently as once
every 10 years. Such a policy, as it relates to the evaluation of the credit-hour equivalencies for
direct assessment programs, is inadequate for Title IV gatekeeping purposes because the
Higher Learning Commission must have a current baseline for evaluating whether the
equivalencies that the school is assigning to the direct assessment programs are appropriate at the
time when the Higher Learning Commission evaluates the proposed programs through the
substantive change process.

We recommend that the Assistant Secretary for Postsecondary Education require the
Higher Learning Commission to revise its policies and procedures to ensure that it obtains
sufficient information about the proposed competency-based education program’s expected level
of interaction between faculty and students; consistently applies its procedures for reviewing
competency-based education programs, including direct assessment programs; and requires
all programs that meet the definition of a substantive change to go through the substantive
change process. We also recommend that the Assistant Secretary require the Higher Learning
Commission to review the assignment of credit hours for any school that submitted a substantive
change application for direct assessment programs but has not had its assignment of credit hours
reviewed since July 1, 2011 (the effective date of the Federal regulation that defined a credit
hour for Title IV purposes). Finally, we recommend that the Assistant Secretary require the
Higher Learning Commission to reevaluate competency-based education programs previously
proposed by schools to determine whether interaction between faculty and students will be
regular and substantive, and, if not, determine whether the programs should have been classified
as correspondence programs.

We provided the draft of this report to the Higher Learning Commission for review and comment
on July 15, 2015. We received the Higher Learning Commission’s comments and additional
documentation on August 13, 2015. The Higher Learning Commission generally agreed with the
matters presented in the finding. However, the Higher Learning Commission stated that we
evaluated its operations against regulatory interpretations, specifically Dear Colleague Letter

3
  The Higher Learning Commission assigns each accredited school to one of its vice presidents of accreditation
relations, referred to as staff liaisons. Staff liaisons serve as the primary contact between the Higher Learning
Commission and the schools it accredits. Throughout this report, we use the term staff liaison instead of
vice president of accreditation relations.
Final Audit Report
ED-OIG/A05O0010                                                                     Page 4 of 33
GEN–14–23 (December 18, 2014), that were not in existence in September 2014 when we
started our audit. The Higher Learning Commission stated that it should not be held accountable
for expectations related to competency-based education programs offered in a credit-hour
framework or interaction between faculty and students that were not made known until after the
audit began. Regardless, the Higher Learning Commission proposed corrective actions to
address all eight recommendations.

We considered the Higher Learning Commission’s comments and made changes where
appropriate. We removed all references to Dear Colleague Letter GEN–14–23
(December 18, 2014) as criteria for the finding because we evaluated the Higher Learning
Commission’s operations against the regulations that existed when we started our audit, not the
guidance that the Department issued after we started our audit. We also clarified
Recommendations 1.1 through 1.6. Instead of recommending that all competency-based
education programs be required to go through the substantive change approval process, we are
recommending that only those competency-based education programs meeting the definition of a
substantive change should be subject to the substantive change approval process. Finally, in the
section “Objectives, Scope, and Methodology,” we clarified that our evaluation of the
Higher Learning Commission’s operations was limited to the schools for which the
Higher Learning Commission had received at least a screening form and that we did not evaluate
its overall accreditation standards, other substantive change processes, or general operations.

We summarized the Higher Learning Commission’s written comments on the draft audit report
at the end of the finding. We included the comments and the Higher Learning Commission’s
proposed corrective actions in their entirety as Attachment 2 of this report. We did not include
the documentation with Attachment 2. However, copies of the documentation, less any
personally identifiable information protected under the Privacy Act of 1974 (5 U.S.C. § 552a) or
other information exempt under the Freedom of Information Act (5 U.S.C. § 552(b)), are
available on request.




                                      BACKGROUND


Schools may deliver programs that are eligible for participation in Title IV programs through
various delivery methods: campus-based, distance education, and correspondence.
Campus-based programs deliver instruction to students in an on-campus setting where the
instructor and students meet in person. Distance education programs use technology to deliver
instruction to students who are separated from the instructor but are supported by regular and
substantive interaction between the instructor and student. Correspondence programs deliver
instructional materials by mail or electronic transmission to students who are separated from the
instructor. Correspondence programs are typically self-paced. Interaction between the instructor
and student is limited, not regular and substantive, and primarily initiated by the student.

Similarly, schools may offer Title IV-eligible programs that measure student learning in various
ways: credit hours, clock hours, or direct assessment. The regulations at 34 C.F.R. § 600.2
Final Audit Report
ED-OIG/A05O0010                                                                                   Page 5 of 33
define credit hours and clock hours for Title IV purposes. A credit hour is generally based on
the amount of work that the student is expected to complete and is equivalent to a minimum of
1 hour of classroom or direct faculty instruction each week and 2 hours of out-of-class student
work each week. A clock hour is directly based on the amount of time in which the student is
in a class, lecture, recitation, faculty-supervised laboratory, shop training, internship, or
preparation in a correspondence course. According to 34 C.F.R. § 668.10, direct assessment
measures academic progress through an evaluation that is not based on the amount of time a
student spends on coursework; it measures what a student knows and can do in terms of the body
of knowledge making up the educational program. These direct assessment measures should
provide evidence that a student has command of a specific subject, content area, or skill or that
the student demonstrates a specific quality, such as creativity, analysis, or synthesis, associated
with the subject matter of the program. Examples of direct assessment measures include
projects, papers, examinations, presentations, performances, and portfolios.

Like other programs, a competency-based education program—a program that organizes
academic content according to competencies (what a student knows and can do)—may be
offered as a campus-based, distance education, or correspondence program and may measure
student learning either by credit hours, clock hours, or direct assessment. Accordingly, as a type
of competency-based education program, a program that measures student progress through
direct assessment may be offered as a campus-based, distance education, or correspondence
program.

Accrediting agencies play a critical role in the Department’s processes for determining the
Title IV eligibility of competency-based education programs, including direct assessment
programs. According to 34 C.F.R. § 668.10, as part of its application to the Department for
approval of a direct assessment program, a school must include documentation from its
accrediting agency that indicates the agency evaluated the program and included the program in
the school’s grant of accreditation. The school must also provide documentation showing that
the accrediting agency agreed with the school’s claim of the program’s credit- or clock-hour
equivalency. 4 More generally, 34 C.F.R. § 668.8(m) states that a program offered in whole or in
part through telecommunications is Title IV eligible if the program is offered by a school that has
been evaluated and is accredited for its effective delivery of distance education programs by a
recognized accrediting agency that has accreditation of distance education within the scope of its
recognition.

On December 18, 2014, the Department issued Dear Colleague Letter GEN–14–23,
“Competency-Based Education Programs—Questions and Answers,” to provide clarification
regarding schools’ proposed offerings of competency-based education programs, including
direct assessment programs. The Department explained the distinction between competency-
based education credit-hour and competency-based education direct assessment programs;
the requirements for establishing credit-hour equivalencies in direct assessment programs; and
the requirements for regular and substantive interaction between faculty and students. The
Dear Colleague Letter specified the following:


4
 In March 2013, the Department reiterated these requirements for accrediting agency approval through
Dear Colleague Letter GEN–13–10.
Final Audit Report
ED-OIG/A05O0010                                                                       Page 6 of 33
       •   Pursuant to the regulations at 34 C.F.R. § 602.22, an accrediting agency should
           evaluate a school’s first offering of a competency-based education program, including
           a direct assessment program, as a substantive change, and the accrediting agency
           should approve such a program before the school may provide Title IV funds to
           students enrolled in those competency-based education programs.

       •   Any competency-based education program, including a direct assessment program,
           that does not include regular and substantive interaction between students and
           school employees who meet accrediting agency standards for providing instruction in
           the subject matter being discussed is considered a correspondence program.

       •   Pursuant to 34 C.F.R. § 602.24(f), an accrediting agency must include a review of a
           school’s policy for determining credit hours for competency-based education
           programs when the accrediting agency reviews the school for initial accreditation,
           renewal of accreditation, or for a substantive change under 34 C.F.R. § 602.22.

Proper classification of a program’s delivery method and measurement of student learning is
important for Title IV purposes because certain Federal regulations are specific to the particular
delivery method or measurement of student learning. For example, to be an eligible
Title IV program, a direct assessment program must meet the requirements in 34 C.F.R.
§ 668.10. Also, the Higher Education Act of 1965, as amended, and Title IV regulations treat
correspondence programs differently than other educational programs. For example,
section 472(5) of the Higher Education Act of 1965, as amended, generally limits cost of
attendance to tuition and fees for students enrolled in correspondence programs. Additionally,
students enrolled only in correspondence programs may receive only a half-time Federal Pell
Grant award (34 C.F.R. § 690.66), and a school may not award Title IV funds to any students if
more than 50 percent of its courses are correspondence courses or if 50 percent or more of its
students are enrolled in correspondence courses (34 C.F.R. § 600.7(a)(1)).

The Higher Learning Commission and Its Processes for Evaluating Proposed Programs
To carry out its accrediting activities, the Higher Learning Commission relied on the work of its
employees and peer reviewers. The peer reviewers were a group of about 1,300 faculty and
administrators from the more than 1,000 schools that the Higher Learning Commission
accredited. These peer reviewers’ duties included evaluating schools’ applications for initial
accreditation, conducting comprehensive reviews of schools for reaffirmation of accreditation,
and reviewing and recommending approval or denial of substantive changes.

If a school wanted to offer a new program that was a significant departure from the school’s
existing programs, the Higher Learning Commission required the school to apply for approval
through a substantive change process. The Higher Learning Commission considered
direct assessment programs a significant departure from a school’s existing programs. After the
Department issued Dear Colleague Letter GEN–13–10, the Higher Learning Commission
established a pilot program to understand and identify best practices in direct assessment,
develop its policies and procedures for evaluating direct assessment programs, and train its
employees and peer reviewers to evaluate direct assessment programs. In April and May 2013,
the Higher Learning Commission accepted substantive change applications from four schools as
Final Audit Report
ED-OIG/A05O0010                                                                       Page 7 of 33
part of its direct assessment pilot program. The Higher Learning Commission approved the
substantive change applications for two of the four schools in May 2013 and approved the
remaining two in July 2013. In December 2013, the Higher Learning Commission began
accepting substantive change applications from all its accredited schools interested in offering
direct assessment programs.

According to the Higher Learning Commission’s policies, a direct assessment program was a
program that measured student learning by competencies but did not follow a course-based,
credit-hour framework. As the first step in its process for evaluating a school’s proposed
program, the Higher Learning Commission required the interested school to submit a
Direct Assessment Competency-Based Programs Screening Form (screening form) consisting of
five yes-or-no questions about the proposed program. The screening form directed schools not
to provide any additional information. The staff liaison assigned to that school then used the
answers on the form, as well as discussions with school officials, to determine whether the
proposed program was a direct assessment program. If the staff liaison determined that a
school’s proposed program was a direct assessment program, the Higher Learning Commission
notified the school by email that the school should submit a substantive change application.

If the staff liaison determined that a program was not a direct assessment program, the
Higher Learning Commission did not require a substantive change application unless that
program represented another type of change that was considered substantive. Other types of
substantive changes included a school’s first offering of a program (for example, its
first bachelor’s degree in business administration) or the school’s first offering of a
distance education or correspondence course. In that case, the Higher Learning Commission
would evaluate the program using its normal accreditation process for whatever type of change
the program represented. The Higher Learning Commission did not require approval for a
school’s initial offering of a competency-based education version of a traditional program that
was already included in the school’s grant of accreditation if that program was not considered a
direct assessment program.

The Higher Learning Commission’s Accreditation Services employees managed the substantive
change approval process and coordinated the review of applications for direct assessment
programs. When a school first submitted a substantive change application, Accreditation
Services employees forwarded the application to the school’s assigned staff liaison, who
reviewed the application for completeness and returned it to Accreditation Services.
Accreditation Services employees then set up a change panel consisting of two or three
peer reviewers whom the Higher Learning Commission had trained to evaluate direct assessment
programs.

The change panel reviewed the substantive change application and recommended approval or
denial. The substantive change application and the change panel’s recommendation then moved
to the Institutional Actions Council, which made the final decision to approve or deny the
proposed program. The Higher Learning Commission documented the decision in the
Institutional Action Council’s meeting minutes and sent a letter to the school notifying the school
of the decision.
Final Audit Report
ED-OIG/A05O0010                                                                                     Page 8 of 33
As of December 31, 2014, the Higher Learning Commission had received at least a screening
form from 15 schools expressing interest in offering direct assessment programs. Staff liaisons
who reviewed the screening forms determined that six of the schools’ proposed programs did not
meet the Higher Learning Commission’s definition of direct assessment programs and, therefore,
according to the Higher Learning Commission’s policy, did not need to go through the
substantive change process. For three other schools, the staff liaisons who reviewed the
screening forms requested that the schools submit substantive change applications, but, as of
December 31, 2014, the three schools had not submitted substantive change applications.
The other six schools submitted a total of eight substantive change applications to the
Higher Learning Commission. 5 The Higher Learning Commission approved seven of the
eight applications. However, after receiving a direct assessment program application, the
Department was not certain that one of the seven programs approved by the Higher Learning
Commission met the definition of a Title IV-eligible, direct assessment program. The
Department requested additional information from the school, but the school withdrew its
application before the Department made a final determination as to the type of program being
proposed. The Higher Learning Commission suspended its review of the eighth application
while it revised its substantive change processes.




                                           AUDIT RESULTS


The objective of our audit was to determine whether the Higher Learning Commission
established a system of internal control that provided reasonable assurance that schools’
classifications of delivery methods and measurements of student learning for competency-based
education programs, including direct assessment programs, were sufficient and appropriate to
help the Department ensure that it properly classified the schools’ programs for Title IV
purposes. We concluded that, as of December 31, 2014, the Higher Learning Commission had
not established a system of internal control that provided reasonable assurance that schools’
programs were properly classified for Title IV purposes. Specifically, we found that the
Higher Learning Commission did not

        •    consistently apply its standards when reviewing competency-based education
             programs, including direct assessment programs, and determining the proposed
             programs’ delivery methods and measurements of student learning;

        •    obtain sufficient information to determine the delivery methods and measurements
             of student learning of proposed competency-based education programs it determined
             were not direct assessment programs or whether those programs represented a
             substantive change that required approval through the substantive change process;


5
 One school submitted three applications; the other five schools submitted one application each. All six schools
subsequently submitted direct assessment applications to the Department, and, as of July 13, 2015, the Department
had approved four of the six schools’ applications.
Final Audit Report
ED-OIG/A05O0010                                                                             Page 9 of 33
       •   sufficiently evaluate the credit-hour equivalencies of direct assessment programs; or

       •   maintain complete records supporting its decisions regarding the credit-hour
           equivalencies of direct assessment programs.

FINDING — The Higher Learning Commission Did Not Consistently Apply Its Standards
          or Adequately Evaluate Schools’ Proposed Programs

When considering schools’ proposed direct assessment programs, the Higher Learning
Commission did not consistently apply its standards or consistently evaluate the schools’
classifications of delivery methods and measurements of student learning. The Higher Learning
Commission also did not obtain sufficient information to appropriately evaluate whether
proposed programs were distance education or correspondence programs, whether the programs
measured student learning by credit hours or direct assessment, and whether the programs met
the definition of a substantive change. In addition, for four of the six schools for which it now
includes direct assessment programs in the schools’ grants of accreditation, the Higher Learning
Commission agreed with the schools’ assessments of the programs’ credit-hour equivalencies
without knowing whether the credit hours for the programs from which the direct assessment
programs were derived met the Federal definition of a credit hour. Lastly, the Higher Learning
Commission did not document its agreement with the credit-hour equivalencies of the
direct assessment programs it approved.

The Higher Learning Commission Did Not Consistently Apply Its Standards
The Higher Learning Commission’s Elements of Good Practice provided the following
guidelines that the Higher Learning Commission expected schools to follow when developing
direct assessment programs:

       Faculty or instructors with subject matter expertise in the student’s discipline and in
       general education play a formative role in the direct assessment competency-based
       student’s academic program. While faculty with subject matter expertise design the
       curriculum, this faculty or other similarly qualified faculty or instructors also regularly
       engage with students during the course of the program, provide expert assistance and
       support to students in the program, and have a meaningful role in directing and reviewing
       the assessment of competencies. . . . While mentors or counselors may have an
       important role in direct assessment competency-based programs in supporting or assisting
       students, they should not replace faculty or instructors with subject-matter expertise.

We found that the Higher Learning Commission approved six schools’ substantive change
applications even though the applications described the proposed programs as self-paced
programs and did not clearly indicate that the programs would include regular and substantive
interaction between students and school employees who met the Higher Learning Commission’s
definition of faculty members. For example, according to one school’s substantive change
application, the school’s proposed program would involve both coaches and subject matter
experts. The school’s application stated that the coaches would connect with students once
each week, on average, and serve as academic advisors, coaches, and mentors. The coaches
would track student progress and reach out to students if the students were struggling within
any area of the program. The school’s application further defined subject matter experts.
Final Audit Report
ED-OIG/A05O0010                                                                        Page 10 of 33
Subject matter experts would provide tutoring and academic support, helping students with
any questions or challenges the students might have in acquiring the necessary knowledge or
skills needed to demonstrate proficiency in the program’s competencies. The application did not
indicate whether students would regularly interact with subject matter experts if the students
were not struggling.

We also found that the Higher Learning Commission determined that the programs proposed
by three separate schools represented three different types of competency-based education
programs—a competency-based education credit-hour program offered by distance education,
a direct assessment program offered by distance education, and a credit-hour program offered
by correspondence—even though the screening forms, substantive change applications, and other
documents provided by the three schools similarly described the three schools’ programs. Each
school described its program as self-paced, measuring student progress through the students’
mastery of concepts and involving student interaction with both faculty members and coaches or
mentors. According to the Higher Learning Commission, the initial review of the proposed
programs focused on determining whether the programs represented direct assessment programs
that required approval as a substantive change. Staff liaisons based their decisions on whether
the programs measured student progress in credit hours or assessment of competencies. The
Higher Learning Commission’s documentation did not indicate whether staff liaisons considered
other factors when determining the types of programs being proposed.

Additionally, for one of the nine schools that submitted only a screening form, a staff liaison
initially determined that the school’s proposed programs were competency-based education
credit-hour programs, not direct assessment programs. In such a case, Higher Learning
Commission policy did not require the programs to go through the substantive change process.
However, despite the school submitting only a screening form, the Higher Learning Commission
employed its approval process as if the school had submitted a substantive change application.
The Institutional Actions Council approved the programs as direct assessment programs, and the
Higher Learning Commission notified the school that the programs were approved and would be
included in the school’s grant of accreditation. The Higher Learning Commission did not catch
the error until, according to the Higher Learning Commission’s vice president for legal and
governmental affairs, the Department’s Office of Postsecondary Education learned that the
school might submit one or more programs to the Department for a determination of Title IV
eligibility as direct assessment programs. The Department contacted the Higher Learning
Commission for more information about the school’s proposed programs. At that point, the
Higher Learning Commission’s vice president for legal and governmental affairs reviewed the
school’s file, identified the error, and notified the school that it was not approved to offer
direct assessment programs.

Federal Regulations Require Consistent Application and Enforcement of Standards
According to 34 C.F.R. § 602.18, an accrediting agency must consistently apply and enforce its
standards to ensure that the education or training a school offers is of sufficient quality to achieve
the stated objective of the education or training. To meet this requirement, an accrediting agency
must have effective controls against the inconsistent application of its standards, base its
decisions on its published standards, and have a reasonable basis for determining that the
information it relies on for decision-making is accurate.
Final Audit Report
ED-OIG/A05O0010                                                                      Page 11 of 33
According to 34 C.F.R. § 602.3, correspondence education is defined as typically self-paced, and
the school provides by mail or electronic transmission instructional materials, including
examinations on the materials, to students who are separated from the instructor. Interaction
between the instructor and the student is limited, not regular and substantive, and primarily
initiated by the student.

Substantive Change Policy and Processes Need Strengthening
The Higher Learning Commission did not consistently apply its standards because its substantive
change policy and processes needed strengthening. The Higher Learning Commission’s
substantive change application had a question asking schools to describe the roles of typical
faculty and school employees involved in the proposed direct assessment programs. However,
schools’ responses to this question on the substantive change application did not clearly indicate
that students would regularly and substantively interact with employees who met the
Higher Learning Commission’s definition of faculty members. Further, the Change Panel
Substantive Change Recommendation Form, used by the Higher Learning Commission’s
peer reviewers to document their evaluation of the application and recommendation to approve
or deny it, did not require peer reviewers to evaluate the level of interaction between faculty and
students. Such information could help the Higher Learning Commission evaluate whether the
interaction between faculty and students should be considered regular and substantive—as
required under its standards as well as Department rules—and identify the types of programs
being proposed.

Additionally, the Higher Learning Commission staff liaisons did not meet with each other to
discuss proposed direct assessment programs or monitor decisions that they had made
independently. Policies requiring such actions could help ensure that staff liaisons consistently
determine what type of programs are being proposed.

When we presented our concerns about the inconsistent application of its standards, the president
of the Higher Learning Commission told us that the Higher Learning Commission was unsure
how direct assessment programs and correspondence programs intersected until the Department
issued Dear Colleague Letter GEN–14–23. The president indicated that the Higher Learning
Commission would revise its substantive change application for direct assessment programs to
include questions designed to obtain more information about the self-paced nature of the
programs, the nature and frequency of the engagement between faculty and students, and other
considerations necessary to reach a determination about whether the proposed direct assessment
programs are correspondence programs.

The president also informed us that the Higher Learning Commission would revise its
Change Panel Substantive Change Recommendation Form to include additional questions that
will prompt peer reviewers to more directly evaluate the interaction between faculty and students
and determine whether the proposed programs are correspondence programs. Further, the
president stated that the Higher Learning Commission would ask all schools that previously
submitted a screening form, a substantive change application, or both, for proposed
direct assessment programs for more information about the level of interaction between faculty
and students. The Higher Learning Commission would use that additional information to decide
whether such programs should be classified as correspondence. Lastly, to ensure consistency in
Final Audit Report
ED-OIG/A05O0010                                                                            Page 12 of 33
its responses to schools, the president stated that Higher Learning Commission employees would
review every substantive change application for proposed direct assessment programs at a
case review meeting before forwarding the application to a change panel.

The Higher Learning Commission Did Not Obtain Sufficient Information About
the Types of Programs Being Proposed
The Higher Learning Commission did not obtain enough information from schools to make a
well-informed decision about the types of programs being proposed. For the nine schools that
submitted only a screening form, eight screening forms indicated that the programs would be
self-paced. However, the eight screening forms did not provide enough information for the
Higher Learning Commission to accurately determine the proposed programs’ level of
interaction between faculty and students. Each screening form required the school to indicate,
with only a “yes” or “no” answer, whether the following five attributes applied to its proposed
program(s):

       1. Direct assessment is used to demonstrate student competencies irrespective of
          course structures, faculty-student relationships, and institution-designed curricula.

       2. The program is self-paced and can be completed in a flexible timeframe based on the
          student’s progress.

       3. Student work performed, student learning outcomes and goals, and the application of
          student learning assessments in direct assessment, competency-based programs are
          equivalent or superior to those in traditional academic courses or programs.

       4. Competencies, student work, and achieved learning are not defined exclusively in
          terms of credit or credit hour equivalencies.

       5. Title IV student financial aid will be offered for these programs.

The screening form directed schools not to provide any information other than a yes or no
answer for each of the five questions.

Despite the lack of any descriptive information beyond what was on the screening forms,
staff liaisons determined that six of the nine schools’ programs were not direct assessment
programs. According to the staff liaisons we interviewed, they would often discuss proposed
programs with schools to help them determine whether the program was a direct assessment
program. However, the staff liaisons did not always obtain sufficient additional information
from the schools to correctly determine whether the programs might be correspondence
programs. For example, for one of the six schools, a staff liaison determined that the proposed
program was a competency-based education credit-hour program, but a subsequent program
review conducted by the Department’s office of Federal Student Aid identified the program as
a potential correspondence program. The school then submitted to the Higher Learning
Commission a substantive change application proposing to offer a correspondence program, and
the Higher Learning Commission approved the program as a correspondence program. The
Higher Learning Commission’s procedures were not sufficient to identify the school’s program
as correspondence when the school initially submitted the screening form.
Final Audit Report
ED-OIG/A05O0010                                                                               Page 13 of 33
According to 34 C.F.R. § 602.22(a), an accrediting agency must maintain substantive change
policies that ensure any substantive changes to the educational mission, program, or programs of
a school after the agency has accredited or preaccredited the school do not adversely affect the
capacity of the school to continue to meet the agency’s standards. An accrediting agency meets
this requirement if its definition of substantive change includes at least “the addition of courses
or programs that represent a significant departure from the existing offerings of educational
programs, or method of delivery, from those that were offered when the agency last evaluated
the institution.” Therefore, a school must first obtain its accrediting agency’s approval of the
change before Title IV funds may be provided to students enrolled in the program.

We concluded that the Higher Learning Commission did not obtain sufficient information to
make well-informed decisions because its policy did not consider a school’s initial offering of
a competency-based education credit-hour program to be a substantive change. During the
screening process, staff liaisons were not required to determine whether the design of the
proposed programs represented a significant departure from a school’s existing programs, which
would require the programs to be approved through the substantive change process. Even
though the regulation requires the initial offering of a competency-based education program that
is a significant departure from the school’s existing offerings of educational programs or
methods of delivery to be subject to the substantive change process, the Higher Learning
Commission’s screening form stated

        The institution should not complete this form or seek approval if any of the following
        apply:

            •   The competency-based program, whether on ground or offered via distance or
                correspondence education, is structured within standard courses and credit
                offerings of the institution; i.e., the competencies are part of a program that is
                also defined in terms of required credits and courses.

            •   The competency-based program uses assessment to establish achievement of
                competencies; however, the program also requires student work, student time,
                and student learning equivalent to that of parallel courses and credits in non-
                competency-based offerings and awards a degree, diploma or certificate based on
                the award of credits as well as the achievement of competencies.

According to the president of the Higher Learning Commission, until the Department issued
Dear Colleague Letter GEN–14–23, the Department had not provided guidance to indicate that
it considered all competency-based education programs substantive changes that would require
the approval of the accrediting agency under 34 C.F.R. § 602.22. 6 Nonetheless, according to the
president, the Higher Learning Commission will change its policies so that it handles proposed
competency-based education programs as substantive changes. The Higher Learning
Commission will discontinue use of its screening form related to direct assessment programs and
will not accept new substantive change applications related to direct assessment programs while
it revises its substantive change processes. In the future, a school seeking approval of either
new competency-based education credit-hour or direct assessment programs will need to

6
 Dear Colleague Letter GEN–14–23 was issued December 18, 2014, and we evaluated the Higher Learning
Commission’s operations as of December 31, 2014.
Final Audit Report
ED-OIG/A05O0010                                                                                    Page 14 of 33
complete a multipart application that includes questions that will provide more information about
the nature and frequency of the engagement between faculty and students.

The Higher Learning Commission Did Not Adequately Evaluate Credit-Hour
Equivalencies
The Higher Learning Commission did not adequately evaluate proposed direct assessment
programs’ equivalencies in terms of credit hours. Since July 1, 2011, the effective date of the
Federal regulation defining a credit hour for Title IV purposes, the Higher Learning Commission
has not completed comprehensive reviews for reaffirmation of accreditation of four of the
six schools that it approved to offer direct assessment programs. Therefore, the Higher Learning
Commission did not have an adequate basis on which to compare the credit-hour equivalencies
of any of those four schools’ proposed direct assessment programs.

According to 34 C.F.R. § 602.3, for Title IV purposes, a school must obtain approval for a
direct assessment program from the Department. As part of that approval, an accrediting agency
must evaluate the program and include the program in the school’s grant of accreditation or
preaccreditation; an accrediting agency also must review and approve the school’s claim of
each program’s equivalency in terms of credit or clock hours. Also, 34 C.F.R. § 602.24(f) states
that an accrediting agency, as part of its review of a school for initial accreditation or
preaccreditation or renewal of accreditation, must demonstrate that it has established and used
credit-hour policies that include an effective review and evaluation of the reliability and accuracy
of the school’s assignment of credit hours.

The Higher Learning Commission’s evaluations of proposed direct assessment programs’
equivalencies in terms of credit hours were inadequate because the Higher Learning Commission
did not review a school’s assignment of credit hours for existing programs frequently enough.
Of the six schools that submitted direct assessment substantive change applications, five schools’
applications indicated that the credit-hour equivalencies for their programs were derived from
existing credit-hour programs. However, according to its policy, the Higher Learning
Commission would review whether a school’s assignment of credit hours for the existing
program met the Federal definition only during comprehensive reviews for reaffirmation of
accreditation, which could occur as infrequently as once every 10 years. When we presented the
Higher Learning Commission with our concern about its review of credit hours, its president
informed us that the Higher Learning Commission will require schools seeking approval of
direct assessment programs to complete the Credit Hour Worksheet, which is normally
completed during comprehensive reviews, and submit it along with the school’s substantive
change application. 7 The Higher Learning Commission will also require the change panels
evaluating direct assessment programs to be trained on reviewing the Credit Hour Worksheet and
reaching a determination about the appropriateness of the school’s credit-hour policy.




7
  The Credit Hour Worksheet provided information about the school’s term lengths; the school’s credit-hour
policies; the number and length of course meetings (or instructional time for distance education or
correspondence courses) for a 1 credit-hour course, 2 credit-hour course, and so on; and the number of courses
for which the school awarded each number of credit hours.
Final Audit Report
ED-OIG/A05O0010                                                                      Page 15 of 33
The Higher Learning Commission Did Not Document Its Agreement With
Credit-Hour Equivalencies
The Higher Learning Commission did not maintain complete records related to decisions it
made regarding direct assessment programs’ equivalencies in terms of credit or clock hours.
For six of the seven substantive change applications for direct assessment programs that the
Higher Learning Commission approved, the Higher Learning Commission did not maintain
documentation explicitly stating its agreement with the schools’ determinations about their
direct assessment programs’ equivalencies in terms of credit hours.

According to 34 C.F.R. § 602.3, for Title IV purposes, a school must obtain approval for a
direct assessment program from the Department. As part of that approval, an accrediting agency
must review and approve the school’s claim of each program’s equivalency in terms of credit or
clock hours. Additionally, 34 C.F.R. § 602.15(b)(2) requires an accrediting agency to maintain
complete and accurate records of all decisions made throughout a school’s or program’s
affiliation with the agency regarding the accreditation and preaccreditation of the school or
program and substantive changes, including all correspondence that is significantly related to
those decisions.

According to the Higher Learning Commission’s president, for previously approved direct
assessment programs, the Change Panel Substantive Change Recommendation Form and the
school’s substantive change application provided documentation that the credit-hour
equivalencies were reviewed. However, those documents and the letters sent to schools
notifying them of the approval of the programs did not contain the Higher Learning
Commission’s explicit agreement with those credit-hour equivalencies.

The president of the Higher Learning Commission stated that the Higher Learning Commission
revised the standard forms used to document its review of direct assessment programs to ensure
that, in future cases, it will maintain documentation of its agreement with the programs’
credit-hour equivalencies. The president also indicated that the Higher Learning Commission
will update the files of schools that it previously approved to offer direct assessment programs to
indicate the Higher Learning Commission’s explicit agreement with the credit-hour
equivalencies.

Title IV Funds Could Be At Risk, and the Higher Learning Commission Might Not Comply
With Requirements
If the Higher Learning Commission does not effectively and consistently apply its policies, it
might create an atmosphere of uncertainty among schools planning to develop competency-based
education programs, including direct assessment programs. As a result, schools might forgo
plans to create innovative and effective competency-based education programs that comply with
Title IV requirements and meet the needs of students. Also, without an appropriate evaluation
by the Higher Learning Commission of a school’s classification of proposed competency-based
education programs, the Department might not receive sufficient information about a school’s
proposed direct assessment programs to make fully-informed decisions about the Title IV
eligibility of the programs. Further, because of the limits that the Higher Education Act of 1965,
as amended, places on correspondence programs, schools’ misclassifying correspondence
programs as competency-based education credit-hour or direct assessment programs simply
Final Audit Report
ED-OIG/A05O0010                                                                       Page 16 of 33
offered by distance education could result in overpayments of Title IV funds to students or
disbursement of funds to students enrolled in ineligible programs.

Additionally, if the Higher Learning Commission does not properly determine the type of
programs being offered, it might not know whether a school is meeting Title IV requirements
specific to different types of programs. As a result, the Higher Learning Commission might not
comply with 34 C.F.R. § 602.27(a)(6), which states that “an accrediting agency must submit
to the Department the name of any school or program it accredits that the accrediting agency has
reason to believe is failing to meet its Title IV responsibilities or is engaged in fraud or abuse.”

Recommendations

We recommend that the Assistant Secretary for Postsecondary Education require the
Higher Learning Commission to—

1.1    Revise its policies and procedures for performing substantive change reviews to ensure
       that it obtains sufficient information about the interaction between faculty and students in
       competency-based education programs, including direct assessment programs, to
       determine whether the interaction will be regular and substantive. If not, classify the
       programs as correspondence programs.

1.2    Develop procedures that will ensure that it consistently evaluates proposed programs,
       including competency-based education programs, to determine whether they should be
       subject to substantive change review or direct assessment review.

1.3    Revise its procedures to ensure that is has an adequate mechanism to determine that all
       programs, including competency-based education programs, that meet the definition of a
       substantive change go through the substantive change process.

1.4    Inform all schools that it accredits that no programs, including competency-based
       education programs, that are subject to the substantive change policy are Title IV-eligible
       programs until the programs have been approved through the required substantive change
       process.

1.5    Reevaluate proposed programs for which it has previously received at least a screening
       form since April 2013 to determine whether interaction between faculty and students will
       be substantially different from the school’s prior offerings of programs using the same
       delivery method. If so, determine whether the interaction between faculty and students
       will be regular and substantive. If not, classify the programs as correspondence
       programs.

1.6    Inform any schools that it accredits and that are currently providing Title IV funds to
       students in programs subject to the substantive change policy but have not been approved
       through the substantive change approval process to immediately cease providing
       Title IV funds to those students.
Final Audit Report
ED-OIG/A05O0010                                                                         Page 17 of 33
1.7    Develop procedures that will ensure that it has reviewed a school’s assignment of
       credit hours since July 1, 2011, for any school that it reviews and approves a substantive
       change application for competency-based education programs.

1.8    Ensure that the documentation it maintains for schools that it has approved to offer
       direct assessment programs clearly indicates that the Higher Learning Commission
       agreed with the school’s determination of the programs’ equivalencies in terms of
       credit or clock hours.

Higher Learning Commission’s Comments
The Higher Learning Commission stated that it generally agreed with the issues discussed in the
finding but did not agree with aspects of the draft audit report in which we evaluated its
processes against guidance (Dear Colleague Letter GEN–14–23, issued on December 18, 2014)
that was not in effect when the audit started. The Higher Learning Commission stated that it
should not be held accountable for expectations related to competency-based education
credit-hour programs or interaction between faculty and students that were not made known until
after the audit began. The Higher Learning Commission also stated that it should not be
expected to fulfill the Department’s role of determining whether a school’s programs are Title IV
eligible. Regardless, the Higher Learning Commission provided a corrective action plan
describing actions that it has already taken, or is in the process of taking, in response to the issues
highlighted in the report.

OIG Response
Although not issued until after we began our audit, Dear Colleague Letter GEN–14–23 only
clarified what the existing regulations required. Regarding the review of competency-based
education credit-hour programs, the regulations in effect before December 18, 2014, required an
accrediting agency to have procedures in place to identify whether any program represented a
significant departure from a school’s existing offerings. The Higher Learning Commission did
not have procedures to determine whether competency-based education credit-hour programs
were a substantive change, even when such programs could closely resemble direct assessment
programs.

We agree that the Department is ultimately responsible for ensuring that education programs are
properly classified for Title IV purposes. However, schools must first be accredited by an
accrediting agency recognized by the Department before they may be eligible to receive Title IV
funds. The Department relies on the recognized accrediting agencies to provide it assurances on
the quality of the education programs offered by schools. Therefore, the Department needs the
Higher Learning Commission to provide it with relevant information so the Department can
ensure that it properly classifies the schools’ educational programs and Title IV funds are
appropriately awarded.
Final Audit Report
ED-OIG/A05O0010                                                                                  Page 18 of 33


                     OBJECTIVE, SCOPE, AND METHODOLOGY


The objective of our audit was to determine whether the Higher Learning Commission
established a system of internal control that provided reasonable assurance that schools’
classifications of delivery methods and measurements of student learning for competency-based
education programs, including direct assessment programs, were sufficient and appropriate to
help the Department ensure that it properly classified the schools’ programs for Title IV
purposes. 8 We evaluated the Higher Learning Commission’s operations related to its reviews of
competency-based education programs, including direct assessment programs, as of
December 31, 2014.

To accomplish our objective, we first gained an understanding of selected provisions of the
Higher Education Act of 1965, as amended; the Higher Education Reconciliation Act of 2005;
Title IV regulations in 34 C.F.R. § 600.2, 34 C.F.R. Part 602, and 34 C.F.R. § 668.10; and
Dear Colleague Letters GEN–11–06, GEN–13–10, and GEN–14–23. We then gained an
understanding and assessed the adequacy of the Higher Learning Commission’s system of
internal control, including information systems control, over evaluating schools’ applications for
approval to offer competency-based education programs, including direct assessment programs.
To do so, we interviewed the following Higher Learning Commission employees:

        •    vice president for legal and governmental affairs, who led the Higher Learning
             Commission’s development and implementation of new policies, including those
             related to the evaluation of direct assessment programs;

        •    five of the Higher Learning Commission’s nine staff liaisons, who were responsible
             for initially reviewing proposed programs and determining the types of programs that
             schools are proposing to offer;

        •    vice president and chief operating officer, who oversaw internal operations, including
             the substantive change process;

        •    associate vice president for accreditation processes, associate director for institutional
             change, and accreditation processes coordinator for institutional change, who all
             managed the substantive change approval process and coordinated the review of
             applications for direct assessment programs;

        •    associate director for decision-making, who coordinated the Institutional Actions
             Council’s reviews of proposed substantive changes; and

8
  By sufficient, we mean that the Higher Learning Commission’s processes allowed it to obtain enough information
to make a fully informed decision about the classification of programs’ delivery methods and measurements
of student learning. By appropriate, we mean that the Higher Learning Commission’s evaluation processes worked
as intended, as evidenced by proper classification of programs’ delivery methods and measurements of
student learning.
Final Audit Report
ED-OIG/A05O0010                                                                                Page 19 of 33
        •   director of information technology, who provided an overview of the information
            systems that the Higher Learning Commission used for its accreditation processes.

We also reviewed the Higher Learning Commission’s policies and procedures for evaluating
competency-based education programs (credit-hour and direct assessment), credit- or clock-hour
equivalencies of direct assessment programs, and credit hours that schools assigned to the
courses and programs from which direct assessment programs were derived. Lastly, we
reviewed the Higher Learning Commission’s documentation for all 15 schools that submitted at
least a screening form expressing interest in offering direct assessment programs as of
December 31, 2014. 9 The documentation included the following:

        •   Direct Assessment Competency-Based Programs Screening Form,

        •   Internal Higher Learning Commission Processing and Desk Review Substantive
            Change forms,

        •   substantive change applications submitted during the Higher Learning Commission’s
            pilot program and after the pilot program,

        •   Change Panel Substantive Change Recommendation Form,

        •   final minutes from Institutional Actions Council meetings that involved approval
            of substantive changes for direct assessment programs and letters notifying schools
            of program approval or denial, and

        •   other correspondence between the Higher Learning Commission and schools.

We reviewed the documentation to learn how the Higher Learning Commission determined
whether schools’ proposed programs were direct assessment programs and whether the
documentation clearly showed how the Higher Learning Commission made its determinations.
In addition, we reviewed the substantive change applications to determine (1) how schools
defined faculty; (2) how schools planned to measure student learning (credit hour, clock hour, or
direct assessment); (3) whether schools clearly described whether their proposed programs
would include regular and substantive interaction between faculty and students; (4) whether
schools clearly described the planned programs’ delivery methods (campus-based, distance
education, or correspondence); and (5) how schools established the credit- or clock-hour
equivalencies of the proposed direct assessment programs.

We next evaluated whether the Higher Learning Commission’s documentation included enough
information to allow the Higher Learning Commission to appropriately determine the types of
programs being proposed. We compared the information in the Higher Learning Commission’s
documentation with the applicable definitions of direct assessment, distance education, and
correspondence in 34 C.F.R. Part 602. We also compared the schools’ descriptions of faculty
9
 The Higher Learning Commission did not have an approval process or separate documents specifically for
competency-based credit-hour programs. Therefore, the documentation we reviewed in the Higher Learning
Commission’s files only included documents related to schools’ proposed direct assessment programs.
Final Audit Report
ED-OIG/A05O0010                                                                      Page 20 of 33
positions and descriptions of the level of interaction between faculty and students with the
Higher Learning Commission’s definition of faculty to determine whether the proposed
programs clearly would involve regular and substantive interaction between faculty and students.
Additionally, we reviewed the Higher Learning Commission’s Web site to find the most recent
date of reaffirmation of accreditation for each of the six schools for which the Higher Learning
Commission approved direct assessment programs to determine whether it reviewed each
school’s assignment of credit hours since July 1, 2011, the effective date of the Federal definition
of a credit hour. Lastly, we identified and reviewed all the accreditation documentation that the
Higher Learning Commission provided us to determine whether the Higher Learning
Commission followed its own procedures for the evaluation of competency-based education
programs, including direct assessment programs.

We reviewed the Higher Learning Commission’s documentation for 15 (100 percent) of the
schools that, as of December 31, 2014, submitted at least a screening form that indicated their
interest in offering direct assessment programs. The results of our audit cannot be projected
to the Higher Learning Commission’s review of any other programs at the schools for which we
reviewed documentation or any programs at any other schools that the Higher Learning
Commission accredits.

Weaknesses in Internal Control
We identified weaknesses in the Higher Learning Commission’s system of internal control over
evaluating schools’ proposed competency-based education programs, including direct
assessment programs, that were significant to our audit objective. We found that the
Higher Learning Commission’s policies and procedures were not sufficient and appropriate to
ensure that schools’ proposed programs were properly classified in terms of delivery methods
and measurements of student learning (see Finding, The Higher Learning Commission Did
Not Consistently Apply Its Standards or Adequately Evaluate Schools’ Proposed
Programs, on page 9 of this report).

Data Reliability
To achieve our objective, we relied, in part, on accreditation documentation that the
Higher Learning Commission maintained in its information systems. To assess the reliability of
the accreditation documentation, we first gained an understanding of the accreditation
documentation contained in the Higher Learning Commission’s information systems and the
relevant information systems controls. We interviewed the Higher Learning Commission’s
director of information technology and the vice president and chief operating officer. From our
interviews, we learned that the Higher Learning Commission maintained the official records
for the schools it accredits in one information system. According to the director of information
technology, only three Higher Learning Commission employees can upload documents directly
into that system. We also interviewed two Higher Learning Commission employees—a staff
liaison and the associate vice president for accreditation processes—who were knowledgeable
about the accreditation documentation contained in the information system.

After gaining an understanding of the accreditation documentation contained in the
Higher Learning Commission’s information system and the relevant information system
controls, we applied logic tests on the documentation that we received for two schools that
Final Audit Report
ED-OIG/A05O0010                                                                      Page 21 of 33
submitted substantive change applications and two schools that submitted only screening forms.
We looked for (1) missing data, (2) the relationship of one data element to another, and (3) dates
outside of the expected range or following an illogical progression. Based on our interviews and
logic tests, we determined that the accreditation documentation that the Higher Learning
Commission maintained in its information system were sufficiently reliable for the purposes
of our audit.

We conducted our audit at the Higher Learning Commission’s offices in Chicago, Illinois, and
our offices from July 2014 through March 2015. We discussed the results of our work with
Higher Learning Commission officials on June 11, 2015, and provided them with a copy of the
draft of this report on July 15, 2015. After reviewing the draft audit report, the Higher Learning
Commission provided us with comments on the draft audit report on August 13, 2015. We
reviewed and analyzed the Higher Learning Commission’s comments on the draft of this report
and revised the report as necessary.

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objectives. We believe that the evidence obtained provides a reasonable basis
for our finding and conclusions based on our audit objective.




                            ADMINISTRATIVE MATTERS


Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department of
Education officials.

This report incorporates the comments you provided in response to the draft audit report. If you
have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following Department of Education
official, who will consider them before taking final Departmental action on this audit:

                      Jamienne S. Studley
                      Deputy Under Secretary Delegated Duties of the Assistant Secretary
                      for Postsecondary Education
                      U.S. Department of Education
                      Office of Postsecondary Education
                      1990 K Street, N.W.
                      Washington, D.C. 20006
Final Audit Report
ED-OIG/A05O0010                                                                     Page 22 of 33
It is the policy of the U.S. Department of Education to expedite the resolution of audits by
initiating timely action on the findings and recommendations contained therein. Therefore,
receipt of your comments within 30 calendar days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation and assistance extended by your employees during our audit.
If you have any questions or require additional information, please do not hesitate to contact me
at 312-730-1620 or Lisa F. Robinson, Assistant Regional Inspector General for Audit, at
816-268-0500.

Sincerely,

/s/

Gary D. Whitman
Regional Inspector General for Audit


Attachments
Final Audit Report
ED-OIG/A05O0010                                                                    Page 23 of 33


                                                                             ATTACHMENT 1



                Acronyms, Abbreviations, and Short Forms Used in This Report



C.F.R.                            Code of Federal Regulations

Department                        U.S. Department of Education

Direct assessment                 Competency-based education program that measures a
program                           student’s learning through direct assessment, not credit or
                                  clock hours

Screening form                    Direct Assessment Competency-Based Programs Screening
                                  Form

Staff liaison                     Vice president for accreditation relations, a Higher Learning
                                  Commission employee who served as the primary point of
                                  contact between the Higher Learning Commission and the
                                  schools it accredits

Title IV                          Title IV of the Higher Education Act of 1965, as amended
Final Audit Report
ED-OIG/A05O0010                      Page 24 of 33


                                ATTACHMENT 2




             AUDITEE COMMENTS ON
            THE DRAFT AUDIT REPORT
Final Audit Report
ED-OIG/A05O0010                                                                   Page 25 of 33




August 13, 2015


Mr. Gary Whitman
Regional Inspector General for Audit
Office of Inspector General
U.S. Department of Education
Citigroup Center
500 W. Madison St., Suite 1414
Chicago, IL 60661

Dear Mr. Whitman:

I am writing in response to the draft audit report issued by the Office of Inspector General
(“OIG”) regarding the Higher Learning Commission’s (“the Commission’s”) handling of
competency-based education, including direct assessment. In particular, the purpose of the
audit was initially defined in an e-mail from Ms. Lisa Robinson on September 3, 2014 as
“to determine whether the Higher Learning Commission is evaluating the appropriateness
of institutional classification of delivery method and measurement of student learning for
competency-based programs, including direct assessment programs, to ensure that
programs are properly classified for Title IV purposes.” The draft audit report recently
issued further expanded on this purpose by indicating that the audit was intended to
examine the Commission’s internal controls relative to this work. The draft audit report
identifies a number of deficiencies in the Commission’s internal processes related to review
of direct assessment and competency-based education. While the Commission agrees with
many of the findings identified by the OIG team in its recent draft audit, it disagrees with
those aspects of the report that evaluate the Commission against regulatory interpretations
that were not extant on September 3, 2014 when the audit was initiated particularly to the
extent that they raise implications regarding the Commission’s overall capacity to act as a
responsible gatekeeper for Title IV federal regulations.

Development of Statute and Regulations Related to Direct Assessment and Application to
Recognized Accrediting Organizations

Congress initially authorized the eligibility of programs that utilize the direct assessment of
student learning in lieu of measuring student learning in credit hours in the Higher
Education Reconciliation Act of 2005 (Section 8020). This statutory language was said to
have been specifically intended to assure that one particular new institution sponsored by
Final Audit Report
ED-OIG/A05O0010                                                                                   Page 26 of 33
several governors would have access to Title IV funds (Senator Mike Enzi [R-Wy],
Remarks on the U.S. Senator Floor, July 23, 2007 regarding Western Governors
University). However, more broadly, the authorization appears to have been intended to
provide new avenues through which students could gain a certificate or degree from an
accredited college and that such avenues would fulfill multiple purposes including
recognizing skills or learning students attained outside of higher education, thereby
reducing the cost of the academic program for students and institutions, and providing
flexibility for students that might result in some cases in a more expeditious route to
completion. Thus the initiation of direct assessment programs appears to have fulfilled
public policy goals related to these areas.

In November 2006 the U.S. Department of Education (“the Department”) subsequently
issued new regulations related to direct assessment. These regulations not only defined
direct assessment and established approval mechanisms by the Department (34 CFR
§668.10) for new direct assessment programs seeking Title IV aid, but also required
recognized accrediting agencies to complete certain responsibilities with regard to direct
assessment programs:

        (T)he accrediting agency must— (1) Evaluate the program(s) and include them in
        the institution's grant of accreditation or preaccreditation; and (2) Review and
        approve the institution's claim of each direct assessment program's equivalence in
        terms of credit or clock hours. (34 CFR §602.2)

As this language makes clear, accreditors have two very specific tasks related to direct
assessment: 1) evaluate the proposed direct assessment program based on the standards and
procedures related to the agencies’ oversight of the quality of institutions and their
academic programs; and 2) approve the proposed direct assessment program’s credit hour
equivalencies.

For several years the Department was silent regarding direct assessment, and no institutions
approached the Commission about developing a direct assessment program. It was not until
2012-13 that a few institutions contacted the Commission about developing a direct
assessment program in the future. At that time there was extremely limited understanding
among institutions generally about what constituted a direct assessment program or how
credit hours were to be handled for Title IV purposes within a direct assessment program.
Even the Department seemed unsure how to apply its own regulatory language to specific
institutional situations. 1 In March 2013 the Department issued its first guidance related to
direct assessment programs (Dear Colleague Letter GEN 13-10, “Applying for Title IV
Eligibility for Direct Assessment [Competency-Based] Programs,” issued by the
Department on March 19, 2013). Relying on this nascent guidance from the Department the
Commission initiated an in-depth program to understand and identify best practices in

1
  Northern Arizona University was told by the Department that its new “personalized learning program” was a direct
assessment program, but the Department ultimately concluded that the program was not a direct assessment program
despite approval of the program by the Commission as a direct assessment program. In addition, this case
demonstrates that it is the Department, not the Commission, which makes the final decision on the classification of
the program for Title IV purposes.
Final Audit Report
ED-OIG/A05O0010                                                                                     Page 27 of 33
direct assessment, to develop an evaluation protocol for institutions seeking approval for a
direct assessment program, and to train staff and peer reviewers regarding evaluation of
direct assessment. 2 The Commission reviewed and approved its first direct assessment
programs in June and July of 2013.

In December of 2014 the Department issued its second guidance related to direct
assessment programs (Dear Colleague Letter GEN 14-23, “Competency-Based Education
Programs-Questions and Answers,” issued by the Department on December 19, 2014). In
that document it also clarified regulatory expectations with regard to both competency-
based education operating within a credit-hour framework 3 and faculty-student
engagement. Prior to late 2014, nothing in the statute, regulations or guidance clearly
established any federal obligation for accrediting agencies related to competency-based
education that operated within a credit hour framework. Such education has existed at
many institutions for many years. At such institutions students earn credits and grades as
well as demonstrate specific competencies to qualify for a certificate or degree.
Competencies provide a supplemental or alternative approach to establishing learning
objectives, one focused not only on what students know but what they can do, and provide
an alternative framework for evaluating whether students have met the objectives of the
course. In other words, they are not a delivery mode; they are an evaluation methodology.
Nothing in statute, regulations or guidance prior to 2014 required the Commission to grant
prior approval for the initiation or continuation of competency-based programs that
maintained a credit hour framework, or to grant prior approval to any other evaluative
methodology undertaken by an accredited institution.

In addition, prior to late 2014 nothing in the statute, regulations or guidance clearly laid out
expectations for application of regulatory concepts related to faculty-student engagement in
the direct assessment environment. While these concepts were carefully laid out in
regulations with regard to (credit-based) distance education, they were not carefully stated
for the direct assessment environment where the instructional time requirements inherent in
credit hour-based education were absent.

The Appropriateness of the Scope of the Audit

The OIG initiated its audit related to the Commission’s handling of competency-based
education in September 2014, almost four months before any regulatory expectations were
set with regard to competency-based education within a credit-hour framework or faculty-
student engagement in direct assessment or competency-based programs. The audit
evaluated the Commission in areas in which federal expectations were clear, meaning
specifically the appropriate evaluation of the quality of direct assessment programs and of
credit-hour equivalencies, as stated in the regulations and reiterated in the March 2013 Dear

2
  The Commission refers to this phase as its “pilot program” recognizing that direct assessment and evaluation of
direct assessment were new to higher education and to the federal regulatory environment; the Commission
anticipated that changes to its direct assessment approval protocols would likely be necessary as all parties,
including the Department, learned from their initial experiences and made improvements.
3
  Note that, as defined in federal regulations, direct assessment is a form of competency-based education that
measures student learning in competencies instead of credit hours.
Final Audit Report
ED-OIG/A05O0010                                                                   Page 28 of 33
Colleague letter (Dear Colleague Letter GEN 13-10, “Applying for Title IV Eligibility for
Direct Assessment [Competency-Based] Programs,” issued by the Department on March
19, 2013). However, as noted in the scope of the audit, the audit also purported to evaluate
the Commission in areas where no clear federal expectations had been set previously,
including its evaluation of competency-based education operating within a credit-hour
framework.

In addition, the scope of the audit set the expectation that it is the Commission that must
classify programs for Title IV purposes. While the Commission may classify programs in
order to ensure that they are appropriately evaluated, it is the Department that classifies
programs to determine their eligibility for Title IV funds. The requirements for eligibility
for Title IV purposes are laid out in 34 CFR §668. These requirements specifically
reference the federal definition of an eligible institution, concepts of instructional time, and
certain other factors including those factors defining a direct assessment program.
Recognized accrediting agencies have no defined role in determining whether an institution
or its programs meet the requirements in §668; it is the Department that determines whether
an institution or its program meets these requirements. Recognized accrediting agencies
determine the quality of education offered by the institutions they accredit following
expectations in federal regulations related to recognition (34 CFR §602.1 ff). The agencies’
responsibilities with regard to direct assessment are laid out in 34 CFR §602.2, as noted
above. To the extent that the audit held the Commission accountable for adherence to
regulatory requirements not published until the audit was complete, and for responsibilities
that exceed the scope of authority awarded the Commission in regulations related to the
recognition of accrediting agencies, the scope of the audit was overly broad.

In addition, as previously noted, at the time of the inception of the audit, there was no
established expectation that a recognized accrediting agency had a specific obligation to
review and approve competency-based education, which is not direct assessment but retains
a credit-hour framework. While the Department may have been within the scope of its
authority to extend in December 2014 existing regulations to require certain approvals by
accreditors of such programs, it was not reasonable for the OIG to apply these requirements
to the Commission’s activities in September 2014 when no such federal expectation had
been set.

Specific Findings in the Draft Audit Report

The draft audit report makes several specific findings related to the Commission’s review
of direct assessment programs (Office of Inspector General, Draft Audit Report, July 15,
2015, p. 7ff). These findings relate to its evaluation of direct assessment programs in areas
such as the following:

   •   whether the Commission properly evaluated credit-hour equivalencies for direct
       assessment programs when it had not yet reviewed an institution’s overall allocation
       of credit hours;
Final Audit Report
ED-OIG/A05O0010                                                                                    Page 29 of 33
    •   whether the Vice Presidents for Accreditation Relations properly ensured consistent
        application of Commission policies when they did not review as a group each
        change application seeking approval of a new direct assessment program; or

    •   whether the Commission had appropriately emphasized faculty-student engagement
        when it had not asked specific questions or made explicit findings in the review
        related to faculty-student engagement.

The Commission takes very seriously the identification by the OIG of these and other
deficiencies in the Commission’s process of reviewing direct assessment programs. The
Commission took immediate action, as outlined in its February 26, 2015 letter to the OIG,
when deficiencies were first identified by the OIG. It suspended its review of direct
assessment programs and put off any review of new competency-based programs as
required in the December 2014 Dear Colleague letter while it made improvements to its
review processes. 4 When the Department issued additional guidance in 2015 related
particularly to faculty-student engagement in these programs and other aspects of
competency-based education, the Commission worked to incorporate this new guidance in
its protocols (Dear Accrediting Agency Executive Directors Letter, June 9, 2015, regarding
experimental sites and other direct assessment guidance). The Commission anticipates re-
opening the application process for new direct assessment and competency-based programs
on or before August 31, 2015 once the revisions to these protocols are complete. The
enclosed document identifies progress the Commission has made since March of 2015 in
making appropriate improvements in its processes based on findings from the OIG and
from the most recent guidance provided by the Department.

The Commission began its review of direct assessment programs in 2012-13 with limited
guidance from the Department in good faith to ensure that its processes for review of direct
assessment programs comported with (if not exceeded) federal expectations. The
Commission has requested and awaited additional guidance from the Department with
regard to direct assessment and, more recently, competency-based credit-based education,
and moved quickly to follow that developing guidance. While the OIG draft audit report
identifies various areas where improvements in Commission processes are necessary, it is
important to note that the draft audit report does not call into question the thoroughness of
the Commission’s review regarding compliance with its standards or the Commission’s
capacity to ensure the overall quality of direct assessment programs, which is the central
task of recognized accrediting agencies. In addition, it is important to note that the OIG
audit team looked at one small subsection of the Commission’s operations in which it had
approximately only 18 months experience through reviews of five institutions. The OIG did
not audit more generally the Commission’s review of substantive change, its standards, or
its basic operations. Therefore any conclusion that the Commission is not generally in
compliance with federal requirements for recognition of accrediting agencies or is not a
reliable authority on institutional quality is not supported by means of the limited review
conducted by the OIG team.


4
 While these improvements were pending, institutions could not proceed with the initiation of direct assessment or
new competency-based programs.
Final Audit Report
ED-OIG/A05O0010                                                                 Page 30 of 33
Finally, as noted with regard to the scope of the audit, the Commission should not be held
accountable either for federal expectations related to competency-based education offered
in a credit-hour framework or faculty-student engagement that were not available in
September 2014 when the audit was initiated or for fulfilling a role that is appropriately
assigned to the U.S. Department of Education. It is the Department that ultimately
determines whether an institution or its programs, including direct assessment or
competency-based programs, are eligible for Title IV federal financial aid. The
Commission’s role is to ensure the quality of these programs so that students are well
served. We assert that we have been good stewards in carrying out this charge.

Please let me know if you have any questions. As you know, the Commission is committed
to ensuring its ongoing compliance with regulations that define its federal responsibilities.


Sincerely,

/s/

Barbara Gellman-Danley
President


Enclosures


cc:
Lisa Robinson, Assistant Regional Inspector General for Audit, U.S. Department of
Education Office of Inspector General
Gregory Bernert, Auditor, U.S. Department of Education Office of Inspector General,
Region V
April White, Auditor, U.S. Department of Education Office of Inspector General, Region V
Final Audit Report
ED-OIG/A05O0010                                                                       Page 31 of 33

                                                                       Higher Learning Commission
                                                                                          8/13/15


              Work Plan Status Related to Revisions to Approvals of

               Direct Assessment and Competency-Based Education
A. Increase focus on faculty-student engagement to determine whether proposed direct assessment
   or competency-based education is correspondence education.
    Status                                              Task
                  1. Revise substantive change application to include additional questions related to
In progress          faculty-student engagement. This revision is in process, and the completed form
                     will be available by August 31, 2015.
                  2. Revise panel review form to require findings related to faculty-student
                     engagement and correspondence education. This revision will be complete by
                     October 31, 2015.
                  3. Train peer reviewers and IAC members regarding revised questions. IAC
                     members were trained on this topic on June 9, 2015 and July 21, 2015. Peer
                     reviewers will be trained on this topic by October 31, 2015.
                  4. Begin implementation of revised review process for new proposed direct
                     assessment or competency-based education programs including focus on faculty-
                     student engagement and correspondence education. HLC will release information
                     on new direct assessment and competency-based education protocols by August
                     31, 2015.


B. Initiate approval of competency-based education (“CBE”)
    Status                                              Task
                  1. Update HLC policies and procedures to include CBE pending additional
In progress          guidance from the Department of Education. If guidance is available by October
                     9, 2015, the HLC Board of Trustees will review proposed policy changes
                     requiring institutions to seek prior approval for competency-based education
                     programs at its meeting on November 5-6, 2015.
                  2. Revise substantive change application to include questions related to CBE
                     (includes new questions on faculty-student engagement and correspondence
                     education). The revised form is in process and will be available by August 31,
                     2015; HLC will further update the form to address competency-based education
                     if USDE provides additional guidance.
                  3. Train peer reviewers and IAC members regarding protocols for approval of
                     CBE. HLC will complete this task by October 31, 2015.
                  4. Begin implementation of review process of CBE. HLC will undertake this task
                     with the submission of CBE applications in fall 2015.
Final Audit Report
ED-OIG/A05O0010                                                                         Page 32 of 33
C. Contact institutions previously approved for direct assessment to review faculty-student
   engagement.
     Status                                               Task
                   1. Develop supplement (revised application) that includes only those questions
Complete              recently developed related to faculty-student engagement and correspondence
                      education and release supplement to institutions previously approved to offer
                      direct assessment programs. HLC prepared this supplement and released it on
                      April 29, 2015.
                   2. Receive and review supplement; identify any institutions whose direct
                      assessment program is correspondence education. HLC received supplements for
                      institutions approved for direct assessment programs. The Institutional Actions
                      Council reviewed these materials and acted on them at their meetings of June 16,
                      2015 and July 27, 2015.


D. Contact institutions that HLC previously determined were not offering direct assessment to
   update approvals based on new USDE requirements.
     Status                                               Task
                   1. Notify institutions about revised HLC procedures and protocols related to direct
In progress.          assessment. HLC notified institutions on March 20, 2015 that new procedures
                      would be forthcoming.
                   2. Require institutions to seek approval of new direct assessment and CBE
                      programs and provide approximately 30 days to complete and submit
                      application. HLC sent a survey to these institutions on April 29, 2015 to seek
                      more information about programs offered that could be direct assessment or
                      competency-based. HLC received responses to these questions. HLC has revised
                      its questions on this topic and will reach out to institutions to seek more
                      information when the revised application is available, by August 31, 2015.
                   3. Receive, review, and act on applications for approval of competency-based or
                      direct assessment programs that may arise from this group of institutions.


E. Revise internal protocols for staff review of proposed direct assessment programs
     Status                                               Task
                   1. Discontinue use of screening form; initiate use of revised application form, as
In progress           noted above. HLC discontinued use of the screening form on February 24, 2015.
                   2. HLC receives revised application from institution for direct assessment program.
                      HLC will release information on new direct assessment and competency-based
                      education protocols by August 31, 2015 and will receive applications thereafter.
                   3. Bi-monthly Case Review Meeting to conduct initial review of all new
                      applications to ensure consistent handling of all applications prior to review by
                      change panel. Case review meetings are already in place; applications will be
                      reviewed once received after the August 31, 2015 release of new application
                      materials and protocols.
Final Audit Report
ED-OIG/A05O0010                                                                          Page 33 of 33
F. Include credit hour review in the protocol for approval of new direct assessment and
   competency-based programs
     Status                                               Task
                   1. Require submission of Credit Hour Worksheet with all applications for approval
In progress           of direct assessment or CBE programs. HLC will reflect this change on the
                      revised application, by August 31, 2015.
                   2. Revise substantive change panel review form to include questions related to the
                      credit hour when direct assessment or CBE programs are under review. HLC
                      will reflect this change on the revised application, by August 31, 2015.
                   3. Train panel and IAC members for direct assessment and CBE regarding
                      appropriate review of institutional assignment of credits. Peer reviewers will be
                      trained on direct assessment and CBE by October 31, 2015.
                   4. HLC will build a review of the credit hour into the protocol for review of
                      competency-based education.


G. Miscellaneous
     Status                                                Task
                   1. HLC ceased review of direct assessment programs as of February 16, 2015. Any
In progress           institution at that time under review or pending action for approval will need to
                      reinitiate the approval process by submitting the revised application form after
                      HLC publishes it (by August 31, 2015) and submit the credit hour worksheet if it
                      has not already done so in conjunction with a comprehensive evaluation.
                   2. HLC will notify institutions in March 2015 about the anticipated changes in HLC
                      requirements related to direct assessment and CBE. HLC sent this notification to
                      all institutions on March 17, 2015 and to certain institutions on March 19, 2015
                      by email.
                   3. HLC previously updated its substantive change panel review form and action
                      letter template to require a finding on credit hour equivalencies for new direct
                      assessment programs. HLC will update the files of institutions previously
                      approved to indicate that the approval included an approval of the credit hour
                      equivalencies by September 30, 2015.
                   4. HLC will continue working to get more timely and clearer guidance from the
                      Department about its expectations for approval of competency-based education.