oversight

The Western Association of Schools and Colleges Senior College and University Commission

Published by the Department of Education, Office of Inspector General on 2016-08-02.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                            UNITED STATES DEPARTMENT OF EDUCATION
                                                 OFFICE OF INSPECTOR GENERAL

                                                                                                                  AUDIT SERVICES
                                                                                                     Chicago/Kansas City Audit Region




                                                              August 2, 2016

                                                                                                               Control Number
                                                                                                               ED-OIG/A05P0013


Dr. Mary Ellen Petrisko
President
Western Association of Schools and Colleges
Senior College and University Commission
985 Atlantic Avenue, Suite 100
Alameda, California 94501

Dear Dr. Petrisko:

This final audit report, “The Western Association of Schools and Colleges Senior College and
University Commission Could Improve Its Evaluation of Competency-Based Education
Programs to Help the Department Ensure Programs Are Properly Classified for Title IV
Purposes,” presents the results of our audit. The objective of our audit was to determine whether
the Western Association of Schools and Colleges Senior College and University Commission
(Commission) established control activities that provided reasonable assurance that schools
properly classified, for Title IV of the Higher Education Act of 1965, as amended, (Title IV)
purposes, the methods of delivery and measurements of student learning for competency-based
education programs. 1 We evaluated the Commission’s processes over reviewing schools’
proposed competency-based education programs as of November 13, 2015.

We concluded that the Commission’s control activities did not provide reasonable assurance that
schools properly classified the methods of delivery for competency-based education programs.
As a result, the Commission’s evaluations of the schools’ classifications of the methods of
delivery will not help the U.S. Department of Education (Department) ensure that proposed
competency-based education programs are properly classified for Title IV purposes. We found
that the Commission did not evaluate whether proposed competency-based education programs
were designed to ensure faculty-initiated, regular, and substantive interaction between faculty
and students. According to Title IV regulations, programs that are not designed to ensure such
interaction should be classified as programs delivered via correspondence, not distance
education. The Commission did not implement processes to provide reasonable assurance that
schools properly classified competency-based education programs’ methods of delivery because
correspondence education was not within its scope of recognition by the Secretary. After the
Department notified the Commission that correspondence was no longer included under its scope
of recognition, the Commission removed references to correspondence education from its

1
 Control activities include the policies, procedures, and mechanisms that help ensure that the organization’s
objectives are met.
    The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                      excellence and ensuring equal access.
Final Audit Report
ED-OIG/A05P0013                                                                     Page 2 of 28
accreditation handbook and other documents. Therefore, the Commission did not require
peer reviewers, those who reviewed schools’ proposed competency-based education programs,
to determine whether proposed programs should be more appropriately considered
correspondence education and outside the Commission’s scope of recognition.

We concluded that the Commission’s control activities over reviewing schools’ proposed
competency-based education programs should be sufficient to provide reasonable assurance
that schools’ measurements of student learning for proposed competency-based education
programs are properly identified as clock hours, credit hours, or direct assessment.
The Commission reviewed all proposed competency-based education programs the same,
regardless of whether schools planned to offer the proposed programs as competency-based
education clock- or credit-hour programs or direct assessment programs. We did not identify
any proposed programs for which schools incorrectly reported the program’s measurement of
student learning (as clock hour, credit hour, or direct assessment). Although we did not identify
any significant weaknesses in the Commission’s processes for ensuring that proposed programs’
measurements of student learning were properly identified as clock hours, credit hours, or
direct assessment, we did identify a deficiency in the Commission’s processes for evaluating
the accuracy and reliability of schools’ credit-hour assignments. We found that the Commission,
during substantive change reviews, did not always ensure that the credit hours assigned to
the programs from which schools derived competency-based education programs met the
Federal definition of a credit hour. As a result, the Commission did not always have an adequate
basis against which to evaluate direct assessment programs’ credit-hour equivalencies. We also
found that the Commission did not always follow its own policy relevant to the review of
credit hours. As a result, the Commission approved one school’s proposed programs without
reviewing the school’s credit-hour policy.

We provided the draft of this report to the Commission for review and comment on
May 20, 2016. We received the Commission’s comments on June 20, 2016. The Commission
did not explicitly agree or disagree with either finding. However, the Commission stated that
it has taken or plans to take action to correct the issues identified. We considered the
Commission’s comments and made changes to the report where appropriate. We summarized
the Commission’s written comments on the draft audit report at the end of each finding and
included the comments in their entirety as Attachment 3 of this report.




                                      BACKGROUND


Schools may deliver education programs that are eligible for participation in Title IV programs
through various delivery methods: campus-based, distance education, and correspondence.
Campus-based education delivers instruction to students in an on-campus setting where the
instructor and students meet in person. Distance education uses technology to deliver instruction
to students who are separated from the instructor but are supported by regular and substantive
interaction between the instructor and student. Correspondence education delivers instructional
materials by mail or electronic transmission to students who are separated from the instructor.
Correspondence education is typically self-paced. Interaction between the instructor and student
is limited, not regular and substantive, and primarily initiated by the student.
Final Audit Report
ED-OIG/A05P0013                                                                                   Page 3 of 28
Similarly, schools may offer Title IV-eligible education that measures student learning in various
ways: clock hours, credit hours, or direct assessment. Title 34 of the Code of Federal
Regulations, Section (34 C.F.R. §) 600.2 defines clock hours and credit hours for Title IV
purposes. 2 The regulation states that a clock hour is directly based on the amount of time
in which the student is in a class, lecture, recitation, faculty-supervised laboratory, shop training,
internship, or preparation in a correspondence course. The regulation does not mandate the
classroom hours or “seat time” required of every program. Instead, the regulation defines
a credit hour as

        an amount of work represented in intended learning outcomes and verified by evidence
        of student achievement that is an institutionally established equivalency that reasonably
        approximates not less than . . . [o]ne hour of classroom or direct faculty instruction and
        a minimum of two hours of out of class student work each week for approximately
        fifteen weeks for one semester or trimester hour of credit, or ten to twelve weeks for
        one quarter hour of credit, or the equivalent amount of work over a different amount of
        time.” [Italics added for emphasis.]

According to 34 C.F.R. § 668.10, direct assessment measures academic progress through an
evaluation of what a student knows and can do in terms of the body of knowledge making up
the educational program rather than the amount of time a student spends on coursework.
Direct assessment measures should provide evidence that a student has command of a specific
subject, content area, or skill or that the student demonstrates a specific quality, such as
creativity, analysis, or synthesis, associated with the subject matter of the program. Examples of
direct assessment measures include projects, papers, examinations, presentations, performances,
and portfolios. Like other programs, a competency-based education program—a program that
organizes academic content according to competencies (what a student knows and can do)—may
be offered as campus-based, distance education, or correspondence and may measure student
learning either by clock hours, credit hours, or direct assessment.

Accrediting agencies play a critical role in the Department’s processes for determining the
Title IV eligibility of competency-based education programs. According to 34 C.F.R. § 668.10,
as part of its application to the Department for approval of a direct assessment program, a school
must include documentation from its accrediting agency that indicates the agency evaluated
the program and included the program in the school’s grant of accreditation. The school must
also provide documentation showing that the accrediting agency agreed with the school’s claim
of the program’s clock- or credit-hour equivalency. 3 More generally, 34 C.F.R. § 668.8(m)
states that a program offered in whole or in part through telecommunications is Title IV eligible
if the program is offered by a school that has been evaluated and is accredited for its effective
delivery of distance education programs by a recognized accrediting agency that has
accreditation of distance education within the scope of its recognition.

Proper classification of a program’s delivery method and measurement of student learning is
important for Title IV purposes because certain Federal regulations are specific to the particular
delivery method or measurement of student learning. For example, to be an eligible Title IV
program, a direct assessment program must meet the requirements in 34 C.F.R. § 668.10.

2
 All regulatory citations are to the July 1, 2015, version.
3
 In March 2013, the Department reiterated these requirements for accrediting agency approval through
Dear Colleague Letter GEN–13–10.
Final Audit Report
ED-OIG/A05P0013                                                                                        Page 4 of 28
Also, Title IV and Federal regulations treat correspondence programs differently than other
educational programs. Section 472(5) of the Higher Education Act of 1965, as amended,
generally limits cost of attendance to tuition and fees for students enrolled in correspondence
programs. According to 34 C.F.R. § 668.38(a), students enrolled in correspondence courses are
Title IV-eligible only if the courses are part of a program that leads to an associate, bachelor’s, or
graduate degree. Additionally, according to 34 C.F.R. §§ 690.66 and 600.7(a)(1), students
enrolled solely in correspondence programs may receive only a half-time Federal Pell Grant
award, and a school may not award Title IV funds to any students if, during the school’s latest
complete award year, more than 50 percent of the school’s courses were correspondence courses
or 50 percent or more of the school’s regular students were enrolled in correspondence courses.

As of August 12, 2015, the Commission had accredited or preaccredited about 190 schools.
As of November 13, 2015, five of the schools had submitted substantive changes proposing
seven competency-based education programs. 4 Two schools sought approval for two programs
each, and three schools sought approval for one program each. The Commission approved
four of the programs but did not accept three. 5 The Commission told us that the schools had
implemented three of the four approved programs.

The Commission’s Process for Evaluating Competency-Based Education Programs
Figure 1 on the next page illustrates the Commission’s general procedures for reviewing
substantive changes. See Attachment 2 of this report for a narrative description of the
Commission’s procedures.




4
  In response to the draft of this report, the Commission told us that, as of June 20, 2016, one additional school had
submitted one program for approval but the Commission had not yet approved the program.
5
  One of the three schools whose proposed programs the Commission did not accept resubmitted its proposal. As of
November 13, 2015, the Commission still had not accepted the school’s proposed program.
Final Audit Report
ED-OIG/A05P0013                                                                  Page 5 of 28
Figure 1. The Commission’s Substantive Change Review Process




       Source: Commission’s Substantive Change Manual

The Commission considered all proposed competency-based education programs to be
substantive changes and considered a school's first proposed competency-based education
program to also be a structural change. The Commission’s process for reviewing substantive
changes involving proposed competency-based education programs was the same process used
for reviewing other types of substantive changes.

The Commission used a pool of about 1,500 peer reviewers to conduct its work, including the
reviews of substantive changes. The Commission assigned about 40 peer reviewers to its
substantive change committee, the committee tasked with reviewing substantive changes for
proposed competency-based education programs. For proposed programs that were considered
structural changes, the Commission required a preimplementation site visit at the school and
a review of the programs by both the Commission’s substantive change committee and its
structural change committee.
Final Audit Report
ED-OIG/A05P0013                                                                      Page 6 of 28


                                     AUDIT RESULTS


We concluded that the Commission’s control activities over reviewing schools’ proposed
competency-based education programs did not provide reasonable assurance that schools
properly classified the methods of delivery for competency-based education programs. As a
result, the Commission’s evaluations of the schools’ classifications of the methods of delivery
will not help the Department ensure that proposed competency-based education programs are
properly classified for Title IV purposes. We found that the Commission did not evaluate
whether proposed competency-based education programs were designed to ensure faculty-
initiated, regular, and substantive interaction between faculty and students. According to
Title IV regulations, programs that are not designed to ensure such interaction should be
classified as programs delivered via correspondence, not distance education. The Commission
did not implement procedures to provide reasonable assurance that schools properly classified
competency-based education programs’ methods of delivery because correspondence education
was not within its scope of recognition by the Secretary. Therefore, the Commission did not
require peer reviewers, those who reviewed schools’ proposed competency-based education
programs, to determine whether proposed programs should be more appropriately considered
correspondence education.

We concluded that the Commission’s control activities over reviewing schools’ proposed
competency-based education programs should be sufficient to provide reasonable assurance that
schools’ measurements of student learning for proposed competency-based education programs
are properly identified as clock hours, credit hours, or direct assessment. The Commission
reviewed all proposed competency-based education programs the same, regardless of whether
schools planned to offer the proposed programs as competency-based education clock- or credit-
hour programs or direct assessment programs. If a school indicated its intention to submit the
proposed program to the Department for approval as a Title IV-eligible direct assessment
program, the Commission reviewed the clock- or credit-hour equivalencies of the proposed
program and documented whether it agreed with the assigned equivalencies. We did not identify
any proposed programs for which schools incorrectly reported the program’s measurement of
student learning (as clock hour, credit hour, or direct assessment).

Although we did not identify any significant weaknesses in the Commission’s control activities
over ensuring that proposed programs’ measurements of student learning were properly
identified as clock hours, credit hours, or direct assessment, we did identify a deficiency in the
Commission’s processes for evaluating the accuracy and reliability of schools’ credit-hour
assignments. We found that the Commission, during substantive change reviews, did not always
ensure that the credit hours assigned to the programs from which schools derived competency-
based education programs met the Federal definition of a credit hour (effective July 1, 2011).
The Commission’s substantive change policies and procedures did not require either a thorough
review of a school’s assignment of credit hours to existing programs or a confirmation that such
a review had taken place since July 1, 2011. We also found that the Commission did not always
follow its own policy relevant to the review of credit hours. The Commission’s policy required
peer reviewers to review a school’s credit-hour policy as part of the substantive change review
process. However, the Commission approved one school’s proposed programs without
reviewing the school’s credit-hour policy.
Final Audit Report
ED-OIG/A05P0013                                                                                   Page 7 of 28
In comments on the draft of this report, the Commission acknowledged both of the findings but
did not state whether it agreed or disagreed with either one. The Commission stated that it was
not required to classify a program’s method of delivery as discussed in Finding No. 1.
Nevertheless, the Commission explained that it has updated the substantive change proposal
template to emphasize the Federal expectations related to faculty-initiated, regular, and
substantive interaction with students. In addition, the Commission stated that its peer reviewer
guide will prompt peer reviewers to review proposals for such regular and substantive
interaction. Finally, the Commission noted its responsibility to review credit hours and evaluate
the credit-hour equivalencies of direct assessment programs and agreed to change the policies
and procedures to address the issues presented in Finding No. 2.

FINDING NO. 1 – The Commission Did Not Sufficiently Evaluate Whether
                Proposed Competency-Based Education Programs Included
                Faculty-Initiated, Regular, and Substantive Interaction

We concluded that the Commission did not sufficiently evaluate whether schools designed
proposed competency-based education programs to include faculty-initiated, regular, and
substantive interaction between faculty and students. 6 As of November 13, 2015, the
Commission had received seven substantive change proposals for competency-based education
programs from five schools. Based on the descriptions of the programs in the schools’
substantive change proposals, all seven proposed programs would be offered completely online
and be self-paced. For five of the seven proposals, the Commission’s records did not show that
peer reviewers evaluated whether interaction between faculty and students would be faculty-
initiated, regular, and substantive. The descriptions of three of these five proposed programs did
not show whether the programs would include faculty-initiated, regular, and substantive
interaction between faculty and students. 7 If a program is not designed to provide such
interaction, the program should be classified as correspondence education, not distance
education. However, the Commission did not even consider whether the three proposed
programs should have been considered competency-based education programs delivered via
correspondence and outside the Commission’s scope of recognition.

According to 34 C.F.R. § 602.3, correspondence education is defined as typically self-paced,
and the school provides by mail or electronic transmission instructional materials, including
examinations, to students who are separated from the instructor. Interaction between the
instructor and the student is limited, not regular and substantive, and primarily initiated by
the student. On December 18, 2014, the Department issued Dear Colleague Letter GEN–14–23
“Competency-Based Education Programs—Questions and Answers.” In Q&A #9, the
Department stated—

        All Title IV eligible programs, except correspondence programs, must be designed to
        ensure that there is regular and substantive interaction between students and instructors.
        Such interaction must occur as a required part of the program. Therefore, any

6
  Faculty are school employees who met the Commission’s definition of faculty.
7
  We based our conclusion that the descriptions of the proposed programs did not show whether the programs would
involve faculty-initiated, regular, and substantive interaction solely on the information available in the
Commission’s records. Determining whether the programs should have been classified as correspondence for
Title IV purposes would require a thorough analysis of the schools’ records showing how the programs were or
would be implemented.
Final Audit Report
ED-OIG/A05P0013                                                                           Page 8 of 28
       [competency-based education] program, including a direct assessment program, that does
       not include regular and substantive interaction between students and instructors would be
       considered to be a correspondence program.

Further, Q&A #15 of the Dear Colleague Letter stated that an accrediting agency should ensure
that the instructors used in competency-based education programs meet the agency’s standards.
The Commission’s 2013 Handbook of Accreditation defined faculty as “instructional staff of
an institution responsible for the design, delivery, and assessment of academic programs,” and
stated that faculty “does not typically include administrators, counselors, or other campus
educators” such as student service employees.

The Commission did not consider whether proposed competency-based education programs
might be correspondence because the Commission did not have a process to determine whether
the proposed programs were within its scope of recognition. When correspondence education
was removed from the Commission's scope of recognition for Title IV purposes, the Commission
removed the definition of correspondence education from its substantive change manual and
removed references to correspondence education from other Commission documents. Therefore,
the Commission’s substantive change proposal template and peer reviewer guide did not require
schools to explain whether the proposed program would include faculty-initiated, regular, and
substantive interaction between faculty and students or require peer reviewers to determine
whether the program included such interaction. The only information relevant to faculty
interaction with students that schools were required to provide in the substantive change proposal
template was the manner and frequency of mentoring engagement, faculty-to-student ratios, how
faculty would monitor student progress, and faculty qualifications and training. The peer
reviewer guide prompted peer reviewers to consider only the number, roles, and qualifications
of faculty, not whether proposed programs might be more appropriately considered
correspondence programs and outside the Commission’s scope of recognition.

Without a sufficient evaluation by the Commission of a school’s classification of proposed
competency-based education programs’ methods of delivery, the Department might not receive
sufficient information about a school’s proposed competency-based education programs to make
fully informed decisions about the Title IV eligibility of the programs. Programs outside the
Commission’s scope of recognition would not be Title IV-eligible programs, so the Department
needs sufficient information to take appropriate action with respect to programs that are not
Title IV eligible.

Recommendations

We recommend that the Assistant Secretary for Postsecondary Education require the
Commission to—

1.1    Revise its policies and procedures for performing substantive change reviews to ensure
       that the Commission obtains sufficient information about the interaction between faculty
       and students in competency-based education programs to determine whether the
       interaction will be faculty-initiated, regular, and substantive. If substantive change
       reviews find that proposed programs are not designed to ensure faculty-initiated, regular,
       and substantive interaction, notify the schools that the Commission is not authorized to
       approve the programs for Title IV purposes because correspondence education is not
       within the Commission’s scope of recognition.
Final Audit Report
ED-OIG/A05P0013                                                                            Page 9 of 28
1.2     Reevaluate the competency-based education programs that schools previously proposed
        to determine whether interaction between faculty and students was designed to be
        faculty-initiated, regular, and substantive.

1.3     For programs the Commission determines were not designed to provide faculty-initiated,
        regular, and substantive interaction between faculty and students, notify the schools
        offering the programs that the programs are not within the Commission’s scope of
        recognition.

1.4     Provide the Department with the names of the schools that the Commission has
        determined are offering programs that are not within the Commission’s scope of
        recognition and the date of the Commission’s initial approval of those programs.

Commission Comments
The Commission did not state whether it agreed or disagreed with the finding. The Commission
did state that the regulations and guidance do not require the Commission to classify a program’s
delivery modality, report program or modality approvals to the Department, or track faculty-
initiated contact with students. The Commission also stated that it is not authorized to make
decisions regarding the classification of a program’s delivery method. Such decisions are the
school’s and Department’s responsibility. Nevertheless, the Commission stated that it has taken
steps to update the substantive change proposal template to emphasize the Federal expectations
related to faculty-initiated, regular, and substantive interaction with students. Additionally,
the Commission stated that the peer reviewer guide will now prompt peer reviewers to review
proposals for information relevant to regular and substantive interaction.

OIG Response
While the Department has the final decision on the classification of programs for Title IV
purposes, the Commission still needs to take steps to ensure it reviews only programs that it is
recognized to approve. Because correspondence education is not included in the scope of the
Commission’s recognition by the Secretary, the Commission has a specific obligation to ensure
that it is not unknowingly approving programs as distance education when the programs’
methods of delivery do not include regular and substantive interaction between faculty and
students and should be considered correspondence education. Schools must first be accredited
by an accrediting agency recognized by the Secretary before students enrolled in the schools’
programs may receive Title IV funds. Because correspondence education is not within the
Commission’s scope of recognition by the Secretary, the Commission must establish procedures
to identify programs that it is not authorized to approve. If the Commission “has reason to
believe [an institution] is failing to meet its title IV, HEA responsibilities,” it is required to notify
the Department (34 C.F.R. § 602.27(a)(6)). Without sufficient procedures for assessing faculty-
student interaction, the Commission increases the risk that it will approve competency-based
education programs that are outside the scope of its recognition and programs that would
therefore not be Title IV eligible.

The Commission stated that it has taken steps to update the substantive change proposal template
to emphasize the Federal expectations related to faculty-initiated, regular, and substantive
interaction with students. Additionally, the Commission stated that it updated the peer reviewer
guide to prompt peer reviewers to review proposals for information relevant to regular and
substantive interaction. Although we did not receive or review the revised documents, the
Final Audit Report
ED-OIG/A05P0013                                                                                   Page 10 of 28
Commission’s actions are partially responsive to recommendation 1.1. We revised
recommendation 1.1 to clearly explain what else the Commission’s policies and procedures
should cover.

FINDING NO. 2 – The Commission Did Not Ensure That Credit Hours
                Assigned to Competency-Based Education Programs Met
                Federal Requirements

During a comprehensive review, the Commission required peer reviewers to evaluate whether
a school’s credit-hour policy adhered to Federal regulations and to test the school’s application
of the credit-hour policy for a sample of its courses. For schools that had not undergone
a comprehensive review since July 1, 2011, we concluded that the Commission’s substantive
change reviews did not effectively ensure that the schools’ assignments of credit hours to
proposed competency-based education programs were accurate and reliable. The substantive
change reviews were not effective because the Commission did not ensure that the credit hours
assigned to the programs from which schools derived the competency-based education programs
complied with the Federal definition of a credit hour (effective July 1, 2011).

From July 1, 2011, through November 13, 2015, the Commission had not conducted
comprehensive reviews at two of the five schools that submitted substantive change proposals
for competency-based education programs. One of the two schools submitted one program to
the Department for consideration as a Title IV-eligible direct assessment program. 8 In
compliance with regulatory requirements, the Commission provided a letter to the school stating
that the Commission agreed with the school’s credit-hour equivalencies for the program.
However, because the Commission did not evaluate whether the credit hours assigned to the
program from which the school derived the direct assessment program complied with the
Federal definition of a credit hour, the Commission did not have an adequate basis against which
to evaluate the direct assessment program’s credit-hour equivalencies.

The Commission also did not consistently apply its credit-hour policy to reviews of all proposed
competency-based education programs. The Commission’s policy required peer reviewers to
review a school’s credit-hour assignments (through review of the school’s credit-hour policy) as
part of the Commission’s substantive change review process. However, the Commission
approved one school’s proposed programs without reviewing the school’s credit-hour policy.

According to 34 C.F.R. § 602.24(f), an accrediting agency must include, as part of its review of
a school for initial accreditation or preaccreditation or renewal of accreditation, an effective
review and evaluation of the reliability and accuracy of the school’s assignment of credit hours.
Also, according to 34 C.F.R. § 602.3, for Title IV purposes, a school must obtain approval for
a direct assessment program from the Department. As part of that approval, an accrediting
agency must evaluate the program and include the program in the school’s grant of accreditation
or preaccreditation. An accrediting agency also must review and approve the school’s claim of
each program’s equivalency in terms of clock or credit hours. Dear Colleague Letter
GEN–14–23, Q&A #15, states that, when an accrediting agency reviews a school for initial
accreditation, renewal of accreditation, or for a substantive change under 34 C.F.R. § 602.22,
8
 The school informed the Commission that, as of September 16, 2015, a second proposed program had not been
submitted to the Department for consideration as a Title IV-eligible program. The other school’s substantive change
proposal indicated that the school did not anticipate seeking Title IV eligibility for its proposed program.
Final Audit Report
ED-OIG/A05P0013                                                                                  Page 11 of 28
“the agency must include in that review, the institution’s policy for determining credit hours for
its [competency-based education] programs to ensure that those policies conform to commonly
accepted practice in higher education.”

We concluded that the Commission did not evaluate schools’ assignments of credit hours to
proposed competency-based education programs because the substantive change policies and
procedures did not require peer reviewers to complete a thorough review of a school’s
assignment of credit hours to existing programs or confirm that such a review had taken place
since July 1, 2011. According to the Commission’s policy, peer reviewers evaluated whether a
school’s assignment of credit hours for an existing program met the Federal definition of a credit
hour during comprehensive reviews for reaffirmation of accreditation, which could occur as
infrequently as once every 10 years. During a comprehensive review, the Commission’s policy
required peer reviewers to evaluate whether the school’s credit-hour policy adhered to Federal
regulations and test the school’s application of the credit-hour policy for a sample of courses.
The peer reviewers would indicate the number, degree level, and discipline of courses for which
they reviewed syllabi, and their determination whether the materials reviewed for online courses
showed that students would be required to do the amount of work equivalent to the credit hours
assigned to the course. 9

The Commission’s policy further stated that credit-hour assignments would be included as part
of substantive change reviews. However, the Commission’s records for substantive change
reviews for competency-based education programs did not show that peer reviewers evaluated
the assignment of credit hours, including whether the assigned credit hours met the
Federal definition of a credit hour, to the extent that would be required during a comprehensive
review. During a substantive change review for competency-based education programs, the
policy required peer reviewers only to review a school’s credit-hour policy and indicate the
extent to which the policy was applied or mapped to competency-based modules or courses. The
Commission’s policy did not require peer reviewers to ensure that the school had undergone a
comprehensive review since July 1, 2011, or conduct an in-depth evaluation of credit-hour
assignments if the school had not undergone a comprehensive review since July 1, 2011.

Without an effective review by the Commission of the reliability and accuracy of a school’s
assignment of credit hours, the Department might not receive sufficient information about a
school’s proposed competency-based education programs to make fully-informed decisions
about the Title IV eligibility of the programs. Additionally, if a school does not assign credit
hours in accordance with the Federal definition of a credit hour, the school might award and
disburse more Title IV aid to students than allowed.




9
  We did not design our audit work to evaluate the Commission’s comprehensive review process or the credit-hour
reviews completed during comprehensive reviews. Therefore, we cannot express an opinion on whether the work
completed during comprehensive reviews would be sufficient for the peer reviewers to determine whether a school’s
credit-hour assignments complied with the Federal definition of a credit hour.
Final Audit Report
ED-OIG/A05P0013                                                                     Page 12 of 28
Recommendations

We recommend that the Assistant Secretary for Postsecondary Education require the
Commission to—

2.1    Develop procedures that will ensure the Commission has reviewed, since July 1, 2011,
       a school’s assignment of credit hours for compliance with the Federal definition before
       approving a substantive change for competency-based education programs.

2.2    Review the assignment of credit hours for compliance with the Federal definition of
       a credit hour for any school that submitted a substantive change proposal for a
       competency-based education program but has not had its assignment of credit hours
       reviewed since July 1, 2011.

Commission Comments
The Commission did not state whether it agreed or disagreed with the finding. While
acknowledging its responsibility to review schools’ credit-hour policies and evaluate schools’
assignment of credit-hour equivalencies to direct assessment competency-based education
programs, the Commission stated that evaluating a proposed program using the credit hour as a
proxy for learning in an outcomes-based program has been a challenge. Additionally, the
Commission stated that schools and accrediting agencies face a challenge introducing and
encouraging innovative programs when existing policy and requirements have not been
appropriately adjusted.

While noting the above challenges, the Commission agreed to make changes based on the
finding. The Commission stated that recommendation 2.1 has already been accomplished
through the existing documents. The Commission explained that the substantive change
proposal template requires a school to submit its credit-hour policy, which the Commission “will
review and make a reasonable determination as to alignment with” the Commission’s policy and
Federal regulations. The Commission further explained that its policy requires peer reviewers
to review this information. Finally, the Commission stated that the necessity to review the
credit-hour policy and credit-hour equivalencies has been reinforced with those conducting the
reviews and also would be addressed through training. For recommendation 2.2, the
Commission stated that it will review the assignment of credit hours for compliance with the
Federal definition for any school for which the Commission approved a substantive change for a
competency-based education program but had not reviewed the assignment of credit hours since
July 1, 2011.

OIG Response
We do not agree that the Commission’s existing documents, specifically the current peer
reviewer guide, satisfactorily address recommendation 2.1. The current peer reviewer guide asks
peer reviewers to indicate the extent to which the school’s credit-hour policy is applied or
mapped to competency-based modules or courses. The guide does not prompt peer reviewers to
evaluate whether the school’s assignment of credit hours complies with the Commission’s policy
or with the Federal definition of a credit hour. We revised the finding to clarify the information
that the guide required.
Final Audit Report
ED-OIG/A05P0013                                                                    Page 13 of 28


                  OBJECTIVE, SCOPE, AND METHODOLOGY


The objective of our audit was to determine whether the Commission established control
activities that provided reasonable assurance that schools properly classified, for Title IV
purposes, the methods of delivery and measurements of student learning for competency-based
education programs. We evaluated the Commission’s processes over reviewing schools’
proposed competency-based education programs as of November 13, 2015.

To accomplish our objective, we first gained an understanding of selected provisions of the
Higher Education Act of 1965, as amended; the Higher Education Reconciliation Act of 2005;
regulations in 34 C.F.R. Part 600, 34 C.F.R. Part 602, and 34 C.F.R. § 668.10; and
Dear Colleague Letters GEN–11–06, GEN–13–10, and GEN–14–23. We then gained an
understanding of the Commission’s system of internal control, including information systems
control, over evaluating schools’ substantive change proposals for competency-based education
programs. We determined that the control activities standard of internal control was significant
to answering our audit objective. Accordingly, we gained an understanding and assessed the
adequacy of the control activities that the Commission designed to evaluate schools’ proposed
competency-based education programs. To do so, we interviewed the following Commission
employees:

       •   president of the Commission;

       •   director of substantive change, who was the primary manager of the substantive
           change process and helped develop the Commission’s policies and procedures related
           to competency-based education programs;

       •   two associates, who acted as the Commission’s liaisons to schools for all substantive
           changes, and who reviewed, along with peer reviewers, schools’ substantive change
           proposals for competency-based education programs;

       •   accreditation resource coordinator, who coordinated the review of substantive change
           proposals for competency-based education programs and managed the flow of records
           between schools, the Commission, and peer reviewers;

       •   four of the Commission’s five vice presidents, who acted as the Commission’s
           liaisons to the schools for all matters other than substantive changes; and

       •   chief information officer, who provided an overview of the information systems that
           the Commission used for its substantive change review process.

From the interviews, we gained an understanding of the Commission’s policies and procedures
for evaluating competency-based education programs, clock- or credit-hour equivalencies of
competency-based education programs, and credit hours that schools assigned to the courses and
programs from which competency-based education programs were derived. We also reviewed
Final Audit Report
ED-OIG/A05P0013                                                                     Page 14 of 28
the Commission’s accreditation handbook, substantive change manual, credit-hour policy, and
correspondence related to the policies and procedures.

We also reviewed the Commission’s records for all five schools that had submitted substantive
change proposals for competency-based education programs as of November 13, 2015. The
records included the following:

       •   substantive change proposals for competency-based education programs, including
           resubmissions of proposals;

       •   evaluation forms and worksheets completed by peer reviewers for their reviews of
           proposals and resubmissions of proposals for competency-based education programs;

       •   action letters sent by the Commission to schools notifying the schools of the
           Commission’s decisions;

       •   site-visit reports for the preimplementation and postimplementation site visits the
           Commission required for a school’s first competency-based education program;

       •   final minutes from Commission meetings that involved approval of substantive
           changes for competency-based education programs; and

       •   notes taken by Commission associates during peer reviewers’ discussions of
           substantive change proposals for competency-based education programs.

We reviewed the records to (1) determine whether the Commission implemented its control
activities over evaluating schools’ substantive change proposals for competency-based education
programs, (2) determine whether the Commission followed its own procedures for the evaluation
of competency-based education programs, and (3) learn how the Commission determined that
schools’ proposed programs were competency-based education programs and whether the
records clearly showed how the Commission made its determinations.

To determine whether schools’ proposed competency-based education programs should have
been classified as distance education or correspondence education, we reviewed the schools’
substantive change proposals. We reviewed the proposals to determine (1) how schools defined
faculty; (2) how schools planned to measure student learning (clock hour, credit hour, or direct
assessment); (3) whether schools clearly described whether their proposed programs would
include faculty-initiated, regular, and substantive interaction between faculty and students;
(4) whether schools clearly described the planned programs’ delivery methods (campus-based,
distance education, or correspondence); and (5) how schools established the clock- or credit-hour
equivalencies of the proposed direct assessment programs.

Next, we determined whether the Commission’s records would provide the Commission with
enough information to determine the types of programs being proposed. For substantive change
proposals for competency-based education programs that the Commission reviewed before
December 18, 2014, we compared the information in the Commission’s records with the
definitions of correspondence, direct assessment, and distance education in 34 C.F.R. § 602.3
and the definition of a clock hour and credit hour in 34 C.F.R. § 600.2. For substantive change
Final Audit Report
ED-OIG/A05P0013                                                                     Page 15 of 28
proposals for competency-based education programs that the Commission reviewed after
December 18, 2014, we compared the information in the Commission’s records with the
applicable definitions in 34 C.F.R. § 602.3 and 34 C.F.R. § 600.2 and the guidance that the
Department provided in Dear Colleague Letter GEN–14–23. To determine whether the proposed
programs would involve faculty-initiated, regular, and substantive interaction between faculty
and students, we compared the schools’ descriptions of faculty positions and descriptions of the
planned level of interaction between faculty and students with the Commission’s definition of
faculty.

To determine whether the Commission reviewed each of the five school’s assignment of
credit hours since July 1, 2011 (the effective date of the Federal definition of a credit hour),
we obtained from Commission employees and the Commission’s Web site the date of the
Commission’s most recent comprehensive review for each school. For each comprehensive
review conducted since July 1, 2011, we reviewed the Commission’s reports and the worksheets
used to document the peer reviewers’ evaluations of the school’s credit-hour assignments. We
reviewed the worksheets to determine whether the Commission, as required by 34 C.F.R.
§ 602.24(f), reviewed schools’ (1) policies and procedures for assigning credit hours, as defined
in 34 C.F.R. § 600.2, to their courses and programs and (2) application of the policies and
procedures to their courses and programs. Because the comprehensive review process was not
within the scope of our audit, we did not review records supporting the Commission’s
comprehensive review reports. We also did not evaluate whether the information reviewed by
peer reviewers during the comprehensive reviews was sufficient for the Commission to
determine whether the schools’ credit-hour assignments complied with the Federal definition of
a credit hour.

Although we reviewed the Commission’s records for all five (100 percent) of the schools that,
as of November 13, 2015, had submitted a substantive change proposal for competency-based
education programs, the results of our audit cannot be projected to the Commission’s review of
any other programs at the schools or any programs at other schools that the Commission
accredits.

Internal Control
To achieve our objective, we gained an understanding of the Commission’s system of internal
control over evaluating schools’ proposed competency-based education programs. We then
determined that the control activities standard was the only standard of internal control
significant to our audit objective. Accordingly, we gained an understanding and assessed the
adequacy of the processes that the Commission designed to evaluate schools’ proposed
competency-based education programs, including the proposed programs’ methods of delivery
and measurements of student learning. We also gained an understanding and assessed the
adequacy of the Commission’s processes designed to evaluate the clock- or credit-hour
equivalencies of competency-based education programs and the credit hours assigned to the
courses and programs from which competency-based education programs were derived. We
identified weaknesses in the Commission’s control activities over evaluating proposed programs’
methods of delivery and credit hours assigned to the courses and programs from which proposed
competency-based education programs were derived (see Finding No. 1 and Finding No. 2).
Final Audit Report
ED-OIG/A05P0013                                                                       Page 16 of 28
Data Reliability
To achieve our objective, we relied, in part, on accreditation records that the Commission
maintained in electronic format. To assess the reliability of these accreditation records, we first
learned what accreditation records were maintained in electronic format and the relevant
computer systems controls. We interviewed the chief information officer and director of
substantive change for the Commission. From our interviews, we learned that the Commission
used a third-party, cloud-based Web site to maintain the official records for the accredited
schools. The chief information officer controlled access to the system’s files by assigning
appropriate permissions to the people, such as school officials, peer reviewers, commissioners,
and Commission employees, involved in the Commission’s substantive change review process.

After gaining an understanding of the accreditation records maintained in electronic format and
the relevant computer systems controls, we applied logic tests on the records for two proposed
competency-based education programs. We looked for (1) missing data, (2) the relationship of
one data element to another, and (3) dates outside of the expected range or following an illogical
progression. Based on our interviews and logic tests, we determined that the accreditation
records that the Commission maintained in electronic format were sufficiently reliable for the
purposes of our audit.

We conducted our audit at the Commission’s offices in Alameda, California, and our offices
from September 2015 through February 2016. We discussed the results of our audit with
Commission officials on April 25, 2016, and provided them with a copy of the draft of this report
on May 20, 2016. The Commission provided us with comments on the draft of this report on
June 20, 2016. We reviewed the Commission’s comments and made minor revisions to the
report as necessary.

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objective. We believe that the evidence obtained provides a reasonable basis
for our findings and conclusions based on our audit objective.




                             ADMINISTRATIVE MATTERS


Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department
officials.

This report incorporates the comments you provided in response to the draft audit report. If you
have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following Department official, who
will consider them before taking final Departmental action on this audit:
Final Audit Report
ED-OIG/A05P0013                                                                      Page 17 of 28
                      Lynn B. Mahaffie
                      Deputy Assistant Secretary for Policy, Planning and Innovation, Delegated
                      the Duties of the Assistant Secretary for Postsecondary Education
                      U.S. Department of Education
                      Office of Postsecondary Education
                      LBJ Building, 400 Maryland Avenue, S.W.
                      Washington, D.C. 20202

The Department’s policy is to expedite the resolution of audits by initiating timely action on the
findings and recommendations contained therein. Therefore, receipt of your comments within
30 calendar days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation and assistance extended by your employees during our audit.
If you have any questions or require additional information, please do not hesitate to contact me
at 312-730-1620 or Lisa F. Robinson, Assistant Regional Inspector General for Audit, at
816-268-0519.

Sincerely,

/s/

Gary D. Whitman
Regional Inspector General for Audit

Attachments
Final Audit Report
ED-OIG/A05P0013                                                                   Page 18 of 28
                                                                          ATTACHMENT 1



              Acronyms, Abbreviations, and Short Forms Used in This Report


34 C.F.R. §                Title 34 of the Code of Federal Regulations, Section


Commission                 Western Association of Schools and Colleges Senior College and
                           University Commission


Department                 U.S. Department of Education


Title IV                   Title IV of the Higher Education Act of 1965, as amended
Final Audit Report
ED-OIG/A05P0013                                                                     Page 19 of 28
                                                                              ATTACHMENT 2

    The Commission’s Process for Evaluating Competency-Based Education Programs
                              (as of November 13, 2015)

When a school wanted approval to offer a competency-based education program, the
Commission required the submission of a one-page application. On the application, the
Commission asked for the name of the proposed program, type of substantive change
represented, date the school wanted to first offer the program, and date the school wanted the
substantive change proposal to be reviewed. Upon receipt of the school’s application, the
Commission’s accreditation resources coordinator would send an email to the school. The email
confirmed receipt of the application, included a template for the specific substantive change
proposal, and included instructions for completing the template and submitting the proposal.
The email also included the date by which the school was to submit the proposal and the
tentative date of the substantive change committee’s review.

After a school submitted a substantive change proposal to the Commission, the director of
substantive change assigned an associate to review the proposal. (One associate reviewed
all seven substantive change proposals for competency-based education programs that the
Commission received as of November 13, 2015.) The associate’s initial review focused on
whether all sections of the substantive change proposal were completed and fully developed.
After the associate’s initial review of the substantive change proposal, the proposal moved to
the substantive change committee.

Next, the director of substantive change selected two or three peer reviewers from the
substantive change committee to review the substantive change proposal. For a substantive
change proposal for competency-based education programs, the peer reviewers typically were
selected from a group of five or six members of the substantive change committee who had
experience in adult education and distance education or worked for a school that had submitted
a substantive change proposal for competency-based education programs.

The peer reviewers had about 4 weeks to review the proposal and complete evaluation forms.
Each peer reviewer individually completed an evaluation form. The accreditation resource
coordinator then compiled the individually completed forms into one document and provided
one consolidated form to the peer reviewers to guide their discussion about the substantive
change proposal.

The peer reviewers and the assigned associate participated in a conference call to discuss
the substantive change proposal. The associate took notes during the call. During the first
portion of the call, the peer reviewers discussed the questions they wanted to ask representatives
of the school regarding the proposal. Representatives from the school joined the discussion
about 30 minutes into the call to answer any questions from the peer reviewers. After answering
the peer reviewers’ questions, the school’s representatives left the call and the peer reviewers
reached a decision on the substantive change proposal. For competency-based education
programs, the peer reviewers would reach one of four decisions: (1) defer action, (2) not accept
the proposal, (3) recommend interim approval, or (4) proceed to a site visit (for a school’s first
competency-based education program).
Final Audit Report
ED-OIG/A05P0013                                                                                     Page 20 of 28
Defer Action
If the peer reviewers decided that the substantive change proposal was missing information but
the school could provide the missing information within 1 or 2 weeks, the peer reviewers would
defer action.

Not Accept
If the peer reviewers decided that the school's substantive change proposal was not sufficiently
developed or needed significant revisions, then they would not accept the proposal.
The assigned associate would draft a letter on behalf of the peer reviewers. The letter indicated
the areas in which the school needed to revise the substantive change proposal and the steps the
school needed to take to resubmit the substantive change proposal. The associate would provide
the letter to the Commission's accreditation resource coordinator, who would forward the letter
to the director of substantive change for review. Once the director of substantive change
approved the letter, the accreditation resource coordinator would send the letter to the school.

If a school chose to resubmit a substantive change proposal that was not initially accepted, the
Commission’s accreditation resource coordinator received the resubmission and then forwarded
it to the assigned associate and peer reviewers who reviewed the original submission. Each peer
reviewer would independently review the resubmitted substantive change proposal to determine
whether the school adequately addressed the issues identified in the letter the school received.
Each peer reviewer would then document his or her determination on a worksheet. If the
peer reviewers determined the school adequately addressed the issues, then they would either
recommend interim approval or, if the proposed program was the school’s first competency-
based education program, the Commission would schedule a preimplementation site visit at
the school.

If the peer reviewers thought that the school did not adequately address the issues in the
substantive change proposal, they would not accept the proposal and the school would be invited
to resubmit again.

Interim Approval
For a proposed program that was not a school’s first competency-based education program, if the
peer reviewers decided that a substantive change proposal was adequate and well-developed,
they would give interim approval. The assigned associate would provide the accreditation
resource coordinator with the details of the interim approval. The accreditation resource
coordinator then would prepare an action report on behalf of the peer reviewers, citing the
reviewers’ commendations and recommendations, and send the action report to the school.

After a substantive change proposal received interim approval from the peer reviewers, the
proposal would be added to the agenda for a meeting of the Commission’s executive committee
or the full Commission for final approval. 10 The Commission would then notify the school by
email when the program was approved or denied.




10
  If a program was up for approval in 1 of the 3 months that the full group of commissioners met, the full group
of commissioners would approve the program. Otherwise, the executive committee would approve the program
on behalf of the full group of commissioners.
Final Audit Report
ED-OIG/A05P0013                                                                    Page 21 of 28
Proceed to Site Visit (for a School’s First Competency-Based Education Program)
For a school’s first proposed competency-based education program, if the peer reviewers decided
that the substantive change proposal was adequate, the Commission would schedule a
preimplementation site visit to assess how the school would implement the proposed program
and the overall impact of the program on the school. The Commission's accreditation resource
coordinator would send the school an action letter on behalf of the peer reviewers. The letter
outlined the expectations for the visit. The visit would typically last 1 day and be conducted
by one or two of the peer reviewers who reviewed the substantive change proposal. Based on
the site visit, the peer reviewers would recommend either forwarding the proposal to the
Commission’s structural change committee or not accepting the proposal.

If the peer reviewers recommended forwarding the proposal to the structural change committee,
the director of substantive change would select two members of the structural change committee
to review the proposed substantive change. The accreditation resource coordinator would set up
a teleconference with the assigned associate, at least one of the peer reviewers who reviewed the
proposal, and the two selected members of the structural change committee. The accreditation
resource coordinator would provide the participating individuals with the necessary materials,
including the substantive change proposal, preimplementation site visit report, and the school's
response to the site visit report, if applicable. If recommended for approval by the structural
change committee, the proposed program would be included on the agenda for a meeting of the
Commission’s executive committee or the full Commission. If the program was approved by the
executive committee or full Commission, the accreditation resource coordinator would then send
an action letter to the school, informing the school of the program’s approval.

If the proposal was not accepted by the structural change committee, the Commission would
send the school an action letter specifying the reasons for the decision and explaining how
to resubmit the substantive change proposal if the school chose to do so.
Final Audit Report
ED-OIG/A05P0013                       Page 22 of 28


                                 ATTACHMENT 3




              AUDITEE COMMENTS ON
             THE DRAFT AUDIT REPORT
Final Audit Report
ED-OIG/A05P0013                                                                    Page 23 of 28
             Senior College and
             University Commission


June 20, 2016


Mr. Gary D. Whitman
Regional Inspector General for Audit
Office of the Inspector General, Chicago/Kansas City Audit Region
United States Department of Education

Dear Mr. Whitman:

Thank you for the opportunity to respond to the draft audit report, “The Western Association of
Schools and Colleges Senior College and University Commission Could Improve Its Evaluation
of Competency –Based Education programs to Help the Department Ensure Programs Are
Properly Classified for Title IV Purposes.” On behalf of the Commission staff, I want to thank
you and your staff for the professional and understanding manner in which the audit was
conducted. Mr. Bernert and Ms. White were clear in their expectations and communications and
we appreciated their approach to the audit. Any disagreements or observations I share with you
in this letter are not meant in any way as a negative reflection on the audit process.

I would also note that the Commission staff and peer reviewers have always made good faith
efforts to comply with the expectations of the Department regarding competency-based
education programs. This response is in furtherance of those efforts.

Our understanding of the Department’s expectations is driven by our interpretations of the plain
language of the Department’s regulations governing the Secretary’s Recognition of Accrediting
Agencies set forth in 34 CFR 602 et seq. (“Regulations”).

The Regulations and available guidance (i.e. GEN-14-23, Q.15) reveal that the Commission is
not required to classify a program by delivery modality and is not required to report program or
modality approvals to the Department upon approval by the Commission. Further, the
Commission is not authorized to make decisions on behalf of the Department regarding the
classification of modalities based on the Department’s definitions of distance education,
correspondence education, or otherwise. Nor is the Commission required to track faculty-
initiated contact with students. While the Commission does review the faculty’s role in new
programs that it approves, it does not assess faculty involvement with an eye toward classifying
the program as distance or correspondence education. To the extent that obligations exist to
classify a program by modality under the Department’s standards and to assess the level of
faculty-initiated contact for the purpose of such classification, they are obligations of the
institution and the Department.

The Commission’s peer reviewers are uniquely positioned to and capable of collecting data that
would assist the Department in determining the appropriate classification of a competency-based
education program. However, given the Commission’s desire to offer a full and fair opportunity
to innovative educational pedagogies and modalities, a program’s ultimate classification by the
Department will ultimately have no bearing on whether the same program is included within the
scope of an institution’s accreditation by the Commission.
Final Audit Report
ED-OIG/A05P0013                                                                         Page 24 of 28
The above position of the Commission is in no way meant to be construed as an attempted
avoidance of its obligations in section 602.27(a)(6). The Commission will inform the
Department of any member institution or institutional program that is participating in federal
student aid that the Commission believes may not meet the requirements for participation.
However, at the time an institution seeks approval of a program through the Commission, there
are not students enrolled in the program and no federal student aid is being distributed, so there
would be no basis to inform the Department if, in the opinion of the Commission, a program
could be a correspondence program and therefore beyond the Commission’s scope of
recognition.

Such classifications, if offered by the Commission, could not be more than mere opinions of the
Commission made in its professional judgment given the subjective and ambiguous nature of the
definitions of distance and correspondence education provided in 34 CFR 602.3. The
Commission does not recommend that the Department rely solely on any such opinions of
classification: it cannot stand in place of the institution as the appropriate reporting entity or the
Department as the entity charged with making the ultimate classification determinations.

All of the above notwithstanding: as a Commission, we want to ensure that our work is done in
such a way as to give the public confidence in our authority as a quality assurance agency. We
are fortunate to have a strong group of reviewers who take their work seriously and dedicate
many hours, outside of their campus positions, to the work of the Commission. It is our intention
to make any adjustments necessary to provide these evaluators with the information and training
they need in order to complete the reviews of competency-based programs in a way that will
fulfill our role as part of the Triad – quality assurance.

The swiftly changing landscape of higher education presents many challenges to accreditors,
especially in areas where regulations have not kept pace with that change. The desire to support
innovation among our member institutions is sometimes in conflict with current regulations and
Departmental guidance related to them. The external pressure to support, or at least not impede,
innovation is strong from many sectors. During a conference call related to the experimental
sites, US Department of Education Undersecretary Ted Mitchell stated that competency-based
education is at the heart of its innovation agenda. Despite this, clear and timely guidance related
to determining what is expected and, more importantly, regulations to define what is required
have not been forthcoming, which presents challenges to accreditors who review these programs.
Indeed, the guidance regarding competency-based (particularly direct assessment) education
programs, and therefore reviews, has trickled out. Based on the understanding of current
regulation as described above, it is our intent to use the guidance that is provided in order to
inform our quality assurance process, as we do with information from other sources.

The report refers to the number of reviews of competency-based education programs conducted
by WSCUC and identifies issues with some of them. It is important to understand the full
context of these reviews. At the time of the audit, seven proposals from five institutions had
been reviewed. All of the proposals were for direct assessment programs offered through
distance education. Four of those proposals, from two institutions, had been approved through
the review process. Since then, one additional program from a sixth institution has been
submitted and is proceeding through the approval process. Of the four programs approved by the
Commission, three have been implemented.
Final Audit Report
ED-OIG/A05P0013                                                                               Page 25 of 28


                        Timeline of CBE Program Reviews and Guidance Received

                         Statement of      Committee        Commission           Result        Current Status
                           Intent to         Review           Review
                            Submit          (Proposal
                           Proposal       Submitted 6 –
                           Received          8 Weeks
                                             Before)

Institution A BSBA        3/29/2012        11/19/2013         12/9/2013         Approved      Not Implemented
Institution A MBA         3/29/2012         9/13/2012            N/A           Not Accept
Institution A MBA         12/4/2012         1/16/2013         6/20/2013        Approved         Implemented
resubmission
Institution B BBA         12/21/2012        7/24/2013            N/A           Not Accept
Institution B BBA                           9/25/2013         12/9/2013        Approved         Implemented
resubmission
Institution B BS          5/30/2014         7/20/2015         8/10/2015         Approved        Implemented
Information
Technology
Institution C BA          9/25/2014         4/29/2015            N/A           Not Accept
Management
Institution D MA          6/12/2015         8/19/2015            N/A           Not Accept
Executive Leadership

Institution E MS           9/7/2015         6/13/2016          Pending           Pending
Instructional Design
and Technology


Institution F BA          10/12/2015        11/3/2015            N/A           Not Accept
Leadership
Dear Colleague Letter       Issued           Addresses direct assessment and credit hour
GEN-13-10                 3/19/2013                        equivalencies
Federal Register          7/31/2014
Notification of
Experimental Site
Initiative
Dear Colleague Letter       Issued        Addresses faculty-student interaction and further
GEN-14-23                 12/19/2014                 guidance on credit hours
OIG Exception Report    Issued 2/4/2016   Changes made to proposal template in response to
                                                         exception report


Findings and Recommendations
Generally, given the current state of regulations, WSCUC acknowledges the two findings by the
auditors and has taken, and will take, steps to make any adjustments to our policies and processes
that we believe will improve the reviews to meet our obligation as described above. There is
limited value in responding point by point to the identified issues as the overall goal of WSCUC
is to use the information provided to us to improve our quality review process. However, we do
think it important to respond to the following specific findings and recommendations.
Final Audit Report
ED-OIG/A05P0013                                                                        Page 26 of 28
Finding No. 1
Finding No. 1 – The Commission Did Not Sufficiently Evaluate Whether Proposed Competency-
Based Education Programs Included Faculty-Initiated, Regular, and Substantive Interaction.

The report notes that “for five of the seven proposals, the Commission’s records did not show
that peer reviewers evaluated whether interaction between students and faculty would be faculty-
initiated, regular and substantive.” The auditors found that “three of these five proposed
programs would not include faculty-initiated, regular and substantive interaction between
students and faculty.” Further, it was noted that a program not showing this type of interaction
should properly be classified as a correspondence program, which is outside of the
Commission’s current scope. It is ultimately the Department’s responsibility to classify
programs.

The report cites the December 18, 2014 Dear Colleague Letter as providing guidance related to
“regular and substantive interaction.” As shown in the chart provided above, the reviews of
competency-based programs by WSCUC occurred both before and after this letter was
published. It is an example of the challenge we, and I presume other accreditors, have had in
taking information that is presented after institutions have been encouraged to move forward
with competency-based education programs and making adjustments to our processes to assure
quality. A great deal of time and energy is invested by institutions in developing these programs
and proposals. Similarly, the development of review processes and the training of evaluators is a
process that happens over time. When the guidance related to these reviews comes piecemeal, it
is unreasonable to expect everything to align perfectly.

All of the above notwithstanding, we have taken steps to address the guidance given in order to
make more robust the reviews of faculty-student interaction. Steps taken, or to be taken, to
improve our quality assurance processes are outlined below.

Upon receipt of the initial exception report outlining these potential findings, WSCUC staff took
immediate steps to update our proposal template for competency-based education to emphasize
to institutions and reviewers the federal expectations related to regular and substantive faculty-
initiated interaction with students. The template cites the appropriate Dear Colleague letters
(GEN-13-10 issued March 19, 2013 and GEN-14-23 issued December 19, 2014) as well as the
Commission’s definition of “faculty.” The rubric used by reviewers as they evaluate proposals
similarly calls for specific review of this element of the proposal. This effort is one that supports
our institutions as they prepare their programs for our review and the subsequent review by the
Department for the purposes of classification.

Finding No. 2
Finding No. 2 – The Commission Did Not Ensure That Credit Hours Assigned to Competency-
Based Education Programs Met Federal Requirements.

The auditors noted that the reviews “did not effectively ensure that the schools’ assignment of
credit hours to proposed competency-based education programs were accurate and reliable.” It
is our understanding that it is our obligation to review an institution’s policies and procedures in
relation to 602.2 and its application of the same and make a “reasonable determination” as to
whether the assignment “conforms to commonly accepted practice in higher education.” To
make a reasonable determination is to provide an opinion on a matter for which no commonly
Final Audit Report
ED-OIG/A05P0013                                                                      Page 27 of 28
accepted practice has been developed in higher education. In light of our responsibility and the
expressed concerns of your office we have tried to address them, noting the challenge of using
the credit hour as a proxy for learning when evaluating an outcomes-based learning program.

This is an excellent example of the challenges faced by institutions and accreditors asked to
introduce and encourage (or at least not discourage) innovative educational programs to meet the
many different educational needs in our country. The problem here lies not with those
examining our practices in light of this policy but with the failure to adjust policy in recognition
of the modern educational context. Regardless, we, as accreditors, will continue to do our best to
reconcile this expectation with the proposals that we review. We recognize our responsibility to
review the credit hour policies of our institutions as part of the initial accreditation,
reaccreditation, and substantive change processes, as applicable; we further recognize that we are
expected to examine institutions’ descriptions of how they assign credit hour equivalencies to
direct assessment competency-based education programs, despite the challenges presented by
this expectation.

We agree to make adjustments based on this finding and to address any perceived deficiencies in
prior processes or documentation as outlined below.

Recommendation 2.1
This recommendation has already been accomplished. The templates for all types of substantive
change proposals require submission of the institution’s credit-hour policy, which WSCUC will
review and make a reasonable determination as to alignment with WSCUC’s credit-hour policy
and the federal regulations. The reported failure to examine this policy in at least one case is
unfortunate, but upon receiving the exception report, the template for competency-based
education programs was reviewed and found to request sufficient information to make a
determination about the institution’s compliance. The rubric for reviewers similarly asks that
this information be reviewed. The necessity to review the credit hour policy and equivalencies
has been reinforced with those conducting the reviews and will be addressed in the training of
our reviewers and staff.

Recommendation 2.2
We will review the assignment of credit hours to make a reasonable determination of compliance
with the federal definition of a credit hour for any school that had a substantive change proposal
for a competency-based education program approved but has not had its assignment of credit
hours reviewed since July 1, 2011. If necessary, the institution will be asked to provide
information to make this determination. We will work with the institution to bring it into
compliance if it is determined it is not. The review of the institution’s records and subsequent
contact with the institution, if required, will occur within the next 60 days.

Conclusion
The audit of the WASC Senior College and University Commission’s process for the review of
competency-based education programs has presented the opportunity to review and adjust our
processes as we review new programs of all types, but specifically for competency-based direct
assessment programs offered through distance education. As we have experienced turnover in
key roles related to substantive change during the period of the audit, it has also provided
guidance for the new staff. There are adjustments that have been made to the process that will,
we believe, only improve our quality assurance processes. There are steps we will take to ensure
Final Audit Report
ED-OIG/A05P0013                                                                       Page 28 of 28
we are meeting our regulatory obligations with respect to institutions currently offering programs
approved through our substantive change process. In addition, we will put forth reasonable
efforts to incorporate Departmental expectations into our process as indicated throughout this
response, while also recognizing the challenges inherent in such expectations. The report also
provides us with other feedback that, while not at the level of a Finding or Recommendation, will
enable us to further improve our processes as we move forward.

While we acknowledge that there are improvements that can be made to our review process, it
should also be recognized that the Department has, by issuing incomplete guidance in a
piecemeal manner, made it exceedingly difficult for institutions and accreditors to meet
expectations. The announcement of Experimental Sites and encouragement for institutions to
find creative means to meet the needs of today’s learners came well before guidance to
institutions was provided. Further, regulations have yet to catch up with this new pedagogy.

As educators, we celebrate the efforts by our institutions to find new and innovative ways to
serve learners in new and more accessible ways. As accreditors, we want to support this
innovation while ensuring that quality is maintained in order to avoid harming the institution
and, more importantly, students. As stated previously, the insistence on using a credit-hour
policy that not only does not fit the direct assessment model of competency-based education but
is, in fact, in direct opposition to it does not correspond to the encouragement of institutions to
innovate. We recognize that accreditors such as WSCUC play a role in the Department’s
determination of program eligibility for Title IV aid. The staff and peer evaluators of WSCUC
take very seriously their responsibility in this regard. However, it is ultimately the Department’s
decision and responsibility to determine Title IV eligibility. Pushing it down to the accreditors
further clouds our role and takes us away from the important work of quality assurance and
institutional improvement.

Thank you again for the professionalism with which this audit was conducted. Despite our
significant concerns about policy and guidance from the Department on key issues, we
acknowledge the information that has been presented with the intention to use it to improve our
processes and to do our best to fulfill our obligation to our institutions. We take seriously the
responsibilities that we have to the Department but, again, stress that we should only be
responsible for those items clearly outlined in the Regulations. Ultimately, it is the desire to
facilitate high-quality education for students, in whatever form it may take, that motivates all of
us. We believe that adhering to our traditional role and responsibilities as one member of the
Triad, while depending on the other members to do the same, is the best way to achieve this
purpose.

Sincerely,

/s/

Mary Ellen Petrisko
President